# Defacement Detection with Passive Adversaries
## 5 Results: Experiments, Full-Scale Deployment and Discussion
### Adversarial Newspaper Scenario  
We now describe an experimental setting similar to the one of the previous subsection, but where defacements are chosen by an adversary. Since the focus of this section is about studying the effect of adversarial action, only the adaptive classifier \(T_{L}\) was used, and the custom tests previously described were bypassed. The test system has been used to monitor just one selected URL from the same news website. A set P of \"normal\" page examples and a set N of \"defaced\" page examples were generated, where normal pages are relatively simple variants of the newspaper page, and defaced examples are obtained by injecting content from Zone-H [16] defacement content into variants of the newspaper pages. P is divided into a training set (PTr) and a test set (PTs). N is divided into a training set (NTr), a test set (NTs), and an adversarial defacement set (ADS). A random forest classifier \(RFC_{k}\) is learned using the training set \(\text{PTr}\cup\text{NTr}\), after selecting a secret key \(k\), based on the previously described keyed learning methodology. To simulate an attack, we suppose that when our adversary knows \(k\), she can learn a classifier \(RFC_{k}\), starting from the same learning set \(\text{PTr}\cup\text{NTr}\). Assuming the adversary has access to the same training set we use amounts to giving her a lot of information. In many practical cases, the adversary does not have access to this, or we could use the key to hide it. In this experiment we have decided to simulate a powerful adversary, having access to the same training data we use. However, the classifier \(RFC_{k}\) we have previously generated and \(RFC_{k}\) will not normally be functionally equivalent, because of the variability instrinsic in the random forest generation and in other system components (e.g., timeouts, hyper-parameters). The key was used in this experiment only to select the features to be used during traning. We used again the Nginx web server, as in the previous subsection. The browser will in this case always contact the local Nginx server, that will behave in two possible ways:  
1. download the real URL content from the newspaper site, and return a variant of it (then log \"page tag = normal\"), or
2. in some randomly selected cases, use a symbolic link in the file system, pointing to a substitution page (a defacement), return this page, and log \"page tag = defaced\". The link to the substitution page is changed by a script that is run repeatedly, and is chosen from the adversarial defacement set ADS. This choice can be done in two ways: 1. pick a random page from ADS (when the adversary does not know \(k\)), or 2. run through the ADS set and pick a page that, using \(RFC_{k}\), is tagged as a \"normal\". After the page is returned as just described, we classify it with \(RFC_{k}\) as either normal or defaced, and log this classification (\"class = normal\" or \"class = defaced\"). Finally, we compute FAR and UDR as follows:  
\[\text{FAR}=\left\{\right.\text{logs with \"page tag = normal\" and \"class = defaced\"}\left\}\right\rvert/\left\{\right.\text{logs with \"class = defaced\"}\left\}\right\rvert\]  
\[\text{UDR}=\left\{\right.\text{logs with \"page tag = defaced\" and \"class = normal\"}\left\}\right\rvert/\left\{\right.\text{logs with \"page tag = defaced\"}\left\}\right\rvert\]  
In case 2(b), the adversary tries to pick from ADS a defacement page that will escape detection. More precisely, knowing the key \(k\), she learns a classifier \(RFC_{k}\), and uses this to select a defacement that is likely to be classified as \"normal\" by our detector. If \(RFC_{k}\) and \(RFC_{k}\) are semantically close, the adversary will be likely to achieve her goal, and cause a high UDR. The results are reported in Table 3, and the experiment can be replicated using the software and data made available in www.di.unito.it/~fpb/KeyedLearningData/. As expected, if the adversary knows the key, she can simulate learning, thus obtaining a classifier that is similar to ours.