# Defacement Detection with Passive Adversaries
## 5 Results: Experiments, Full-Scale Deployment and Discussion
### Newspaper Scenario
#### 5.1.2 Performance and Results  
To assess the accuracy of the tool it is useful to evaluate its behavior in the context of the response mechanisms as previously discussed, where our combined evaluation produces an overall alarm level: No_alarm, Information, Warning, Critical. We also consider this experimental setting, where two possible cases are at hand: (1) the original URL content is downloaded or (2) a replacement page is obtained. The replacement page can be (2.1) a defacement instance D, (2.2) a page R from a similar Web Site or (2.3) a modified version R of the same page. We then found it useful to define three kinds of errors:  
* False positive: there has been no page replacement of any kind, as defined in case (1) above, but the system evaluated the URL as critical. * False warning: there has been no page replacement, again as defined in case (1) above, but we obtained a warning because an ordinary change in the original page was detected. * False negative: there has been a page replacement including some defacement instance D, as in case (2.1) above, but the evaluation is either \"Information\", \"No_alarm\", or \"Warning\"--since this is a defacement we would have wanted a \"Critical\" evaluation. The percentage of false positives corresponds to the _False Alarm Rate_ (FAR) as defined in Section 2.1, and the percentage of false negatives corresponds to the _Undetected Defacement Rate_ (UDR). We have run the system over a three-day period, injecting a total of 150 defacements, obtaining the results reported in Table 1.  
The results are also represented graphically in Figure 7. The defacement detection system was run every 10 minutes during the three-day period. For every execution, three quantities were measured:  
1. The total number of \"Critical\" alerts that were generated for the 10 URLs being monitored
2. The total number of \"Warning\" alerts that were generated for the 10 URLs being monitored
3. The value of the combined numerical evaluation computed in the alerting subsystem, averaged over the 10 URLs being monitored. The above three values were thus obtained every 10 minutes, yielding a total of 174 critical messages and 272 warnings over the three days. The average evaluation value and the total number of warning and critical alerts over one hour periods are plotted in Figure 7. Since there is a total of 10 URLs, a maximum of 60 alerts can be obtained every hour (10 \(\times\) 6 executions per hour). The graph of Figure 7 highlights four semantically meaningful events that occurred during those three days, as reported in Table 2.  
Taking into account the dynamic nature of the target Web Site, the experiment produced the expected results. We obtained a moderate number of false positives and a low number of false negatives. As reported in Table 1, a total of 266 false warning were obtained--these do not necessarily require human intervention and can be easily dealt with in a running installation. They are due to significant changes on the target pages, as is it common for a news web site. This can include major text and keyword changes, as well as abrupt changes in the page size, text to image ratio, and number of images. The learned component of the classifier produced no False Alarms (FAR = 0%), and an Undetected Defacement Rate (UDR) of 18%. The overall accuracy is better for UDR, as reported in Table 1, because the global evaluation combines \(T_{L}\) with the fixed tests of Figure 6. It is interesting to note that the result of the classification depends on the selected key k. Changing this, in terms of features and selected examples, causes an unpredictable change in the classifier. When the selected page is not a defacement from Zone-H,