# Defacement Detection with Passive Adversaries
## 5 Results: Experiments, Full-Scale Deployment and Discussion
### Limitations  
Defacement detection is an important, but very particular case of anomaly detection:  
1. it is extremely unbalanced, because in normal operating conditions all data is non-anomalous, but just one defacement instance has catastrophic consequences  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Period** & **Critical Events** & **Warning Events** & **Notification Events** \\ \hline MM2–MM9 & 4 & 4 & 2 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Total monitoring period.  
Figure 10: Aggregated evaluation graph and generated alerts (maintenance months MM2 till MM9).  
Figure 9: Aggregated evaluation graph and relevant events (production scenario—M1 till M4).  
2. once a defacement attempt is detected, it is game over for the adversary, because the vulnerability will be fixed and no additional attempts will be possible
3. the adversary normally has available all the non-anomalous examples, when the Web Site is public, as it normally is
4. defacement examples are available (e.g., Zone-H [16]), but they are normally unrelated to the target Web site and should not be considered as representative of possible defacements that will occur in the future in the target organization.  
As a consequence, we did not simulate adversarial behaviour in the production scenario. This was done, e.g., in [18] for the case of anti-SPAM applications. In that case, the above enumerated items do not apply. Yet, a number of simplifying assumptions had to be done: (a) a limited number of equi-probable randomization choices were possible (100 different sets of weights), (b) the adversary could arbitrarily reduce the score of each anti-spam module, and (c) the cost of such reduction was assumed to be proportional to its value. In our production environment even more assumptions would have been needed, due to the particular application characteristics as enumerated above.  
We did, however, apply a similar strategy in the newspaper scenario (Section 5.2), and the results show that keyed learning is effective: if the key is not known, the measured UDR is significantly lower, when compared to the value obtained when the key is known to an adversary.  
In the real world application context described in Section 5.3, our detection component was integrated with a SOC service and a CDN. We measured success based on FAR and UDR, and modeled the presence of an adversary as a _constraint_: the adversary should have as little information as possible about the learning procedure and the overall detection system. In fact, an adversary could avoid detection if this information were available. Moreover, we wanted to avoid so-called _security through obscurity_, and all hidden information is derived from an \(n\)-bit secret key, where \(2^{n}\) choices are possible. As a consequence, not only does the adversary lack information, as in the case of the introduction of a limited number of randomization choices, but she will also face a computational difficulty, as not all key values can be feasibly tried.  
A question remains unanswered: could an adversary predict classifier behaviour by some other means, i.e., without trying all possible key values? This is an intriguing area for future work. Moreover, we suggest that keyed learning should be applied to other areas of anomaly detection, where it is necessary to hide classifier details in a way that is both systematic and unpredictable.