# Securing Machine Learning Algorithms

Federica Granese

► To cite this version:

Federica Granese. Securing Machine Learning Algorithms. Artificial Intelligence [cs.AI]. Institut Polytechnique de Paris; Università degli studi La Sapienza (Rome), 2023. English. NNT: 2023IPPAX029. tel-04407139v2

HAL Id: tel-04407139

https://hal.science/tel-04407139v2

Submitted on 14 Mar 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# Vers la Sécurisation des Algorithmes d'Apprentissage Automatique par Misclassification Detection et Adversarial Attack Detection

Thèse de doctorat de l'Institut Polytechnique de Paris préparée à École Polytechnique et Sapienza Università di Roma

École doctorale n°626 École doctorale de l'Institut Polytechnique de Paris (EDIPP)
Spécialité de doctorat : Informatique

Thèse présentée et soutenue à Palaiseau, le 21 Avril 202, par

FEDERICA GRANESE

## Composition du Jury :

<table><tr><td>Marc Schoenauer<br>Directeur de recherche, Inria (TAU)</td><td>Président</td></tr><tr><td>Emanuele De Cristofaro<br>Professeur, University College London</td><td>Rapporteur</td></tr><tr><td>Mario Fritz<br>Professeur, CISPA Helmholtz Center for Information Security</td><td>Rapporteur</td></tr><tr><td>Ismail Ben Ayed<br>Professeur, École de Technologie Superieure (Lio-Livia)</td><td>Examinateur</td></tr><tr><td>Elena Marchiori<br>Professeur, Radboud Universiteit (iCIS)</td><td>Examinateur</td></tr><tr><td>Maks Ovsjanikov<br>Professeur, École Polytechnique (GeoViC)</td><td>Examinateur</td></tr><tr><td>Catuscia Palamidessi<br>Directeur de recherche, Inria, École Polytechnique & IPP, (COMÈTE)</td><td>Directeur de thèse</td></tr><tr><td>Daniele Gorla<br>Professeur, Sapienza Università di Roma</td><td>Co-directeur de thèse</td></tr><tr><td>Pablo Piantanida<br>Professeur, CentraleSupélec, International Laboratory on Learning Systems CNRS</td><td>Invité</td></tr></table>
Institute Polytechnique de Paris

Thése de Doctorat

Spécialité Informatique

# Towards Securing Machine Learning Algorithms through Misclassification Detection and Adversarial Attack Detection

## Federica Granese

**Rapporteurs:** Emanuele DE CRISTOFARO
Mario FRITZ

**Directeur de thése:** Catuscia PALAMIDESSI

**Co-Directeur de thése:** Daniele GORLA

**Advisor:** Pablo PIANTANIDA

**Examinateurs:** Ismail BEN AYED
Elena MARCHIORI
Maks OVSJANIKOV
Marc SCHOENAUER (Président du Jury)
To past adventures and those yet to come, to Montella and Italy where my roots
reside, and to the world that I have yet to explore. To the people who have
crossed my path, to those who have been a source of inspiration, and to those I
have yet to meet. May we always strive to take a step forward and never step
back.
Abstract

Deep Neural Networks (DNNs) have seen significant advances in recent years and are nowadays widely used in a variety of applications. When it comes to safety-critical systems, developing methods and tools to make these algorithms reliable, particularly for non-specialists who may treat them as “black boxes” with no further checks, constitutes a core challenge. The purpose of this thesis is to investigate various methods that can enable the safe use of these technologies.

In the first part, we tackle the problem of *identifying whether the prediction of a DNN classifier should (or should not) be trusted* so that, consequently, it would be possible to accept or reject it. In this regard, we propose a new detector which approximates the most powerful (Oracle) discriminator based on the probability of classification error with respect to the true class posterior probability. Two scenarios are investigated: Totally Black Box (TBB), where only the soft-predictions are available and Partially Black Box (PBB) where gradient-propagation to perform input pre-processing is allowed. The proposed detector can be applied to any pre-trained model, it does not require prior information about the underlying dataset and is as simple as the simplest available methods in the literature.

We address in the second part the problem of *multi-armed adversarial attacks detection*. The detection methods are generally validated by assuming a single implicitly known attack strategy, which does not necessarily account for real-life threats. Indeed, this can lead to an overoptimistic assessment of the detectors’ performance and may induce some bias in comparing competing detection schemes. We propose a novel multi-armed framework for evaluating detectors based on several attack strategies to overcome this limitation. Among them, we make use of three new objectives to generate attacks. The proposed performance metric is based on the worst-case scenario: detection is successful if and only if all different attacks are correctly recognized. Moreover, following this setting, we for-
mally derive a simple yet effective method to aggregate the decisions of multiple
trained detectors, possibly provided by a third party. While every single detec-
tor tends to underperform or fail at detecting types of attack that it has never
seen at training time, our framework successfully aggregates the knowledge of
the available detectors to guarantee a robust detection algorithm. The proposed
method has many advantages: it is *simple* as it does not require further training
of the given detectors; it is *modular*, allowing existing (and future) methods to
be merged into a single one; it is *general* since it can simultaneously recognize
adversarial examples created according to different algorithms and training (loss)
objectives.
# Résumé

Les réseaux de neurones profonds ont connu des progressions significatives ces dernières années et sont aujourd'hui largement utilisés dans une variété d'applications. Lorsqu'il s'agit de systèmes critiques pour la sécurité, le développement de méthodes et d'outils pour rendre ces algorithmes fiables constitue un défi central, en particulier pour les non-spécialistes qui peuvent les traiter comme des "boîtes noires" sans autre vérification. L'objectif de cette thèse est d'étudier différentes méthodes qui peuvent permettre l'utilisation sécuritaire de ces technologies.

D'abord, nous devons identifier si la prédiction d'un classificateur devrait (ou ne devrait pas) être fiable afin que il soit possible de l'accepter ou de la rejeter. A cet égard, nous proposons un nouveau détecteur qui approxime le discriminateur le plus puissant (Oracle) basé sur la probabilité d'erreur de classification calculée par rapport à la vraie probabilité postérieure du classificateur. Deux scénarios sont étudiés : Totally Black Box (TBB), où seules les soft-predictions sont disponibles et Partially Black Box (PBB) où la propagation du gradient est autorisée pour effectuer le input pre-processing. Le détecteur proposé peut être appliqué à n'importe quel modèle pre-trained, il ne nécessite pas d'informations préalables sur le dataset et est aussi simple que les méthodes les plus basiques disponibles dans la littérature.

Nous poursuivons en abordant le problème de multi-armed adversarial example detection. Les méthodes de détection sont généralement validées en supposant une seule stratégie d'attaque implicitement connue, ce qui ne réalise pas nécessairement des menaces réelles. En effet, cela peut conduire à une évaluation trop optimiste des performances des détecteurs et peut induire un certain biais dans la comparaison des schémas de détection concurrents. Nous proposons un nouveau framework multi-armed pour évaluer les détecteurs sur la base de plusieurs stratégies d'attaques afin de surmonter cette limitation. Parmi celles-
ci, nous utilisons trois nouvelles fonctions objectifs pour générer des attaques. La mesure de performance proposée est basée sur le scénario du worst case : la détection est réussie si et seulement si toutes les différentes attaques sont correctement reconnues. De plus, en suivant ce framework nous dérivons formellement une méthode simple mais efficace pour agréger les décisions de plusieurs détecteurs entraînés éventuellement fournis par une tierce partie. Alors que chaque détecteur a tendance à sous-performer ou à échouer dans la détection de types d'attaques qu'il n'a jamais vus au moment de l'entraînement, notre framework permet d'agréger avec succès les connaissances des détecteurs disponibles pour garantir un algorithme de détection robuste. La méthode proposée présente de nombreux avantages : elle est simple car elle ne nécessite pas d'entraînement supplémentaire des détecteurs donnés ; elle est modulaire, permettant aux méthodes existantes (et futures) d'être fusionnées en une seule ; elle est générale car elle peut reconnaître simultanément des exemples adverses créés selon différents algorithmes et objectifs d'entraînement.
Acknowledgements

I express my deepest gratitude and appreciation to all those who have supported
and guided me throughout my doctoral journey and the completion of this thesis.

First, I would like to thank my advisor, Prof. Pablo Piantanida, for his
unwavering guidance, expertise, and continuous support throughout this research
endeavor. His valuable insights, constructive feedback, and encouragement have
been instrumental in shaping this work.

I am deeply grateful to Prof. Catuscia Palamidessi who financed me through
the HYPATIA project of the European Research Council (ERC) under the Hori-
zon 2020 research and innovation program of the European Union (Grant agree-
ment N. 835294). Financial assistance has played a crucial role in the execution
of this research work.

I would like to underline my gratitude to Prof. Daniele Gorla, with whom I
started this journey in the research world nearly six years ago and who enabled
me to embark on my academic path in France.

I acknowledge the reviewers of this thesis, Prof. Emanuele De Cristofaro and
Prof. Mario Fritz, for their feedback and evaluation, which have helped improve
the quality of this work. I also extend my appreciation to all the members of
the jury. The questions posed during the evaluation have sparked my curiosity
to extend this work into new areas, and I believe this is the essence of research –
to constantly explore and push the boundaries of knowledge.

I would like to express a special thanks to Dr. Marco Romanelli, who has been
not only a colleague but also a friend. The works presented in this thesis have
emerged from a fruitful collaboration with him. I am grateful for the stimulating
discussions and for his encouragement during times when my motivation wavered
due to unexpected experimental results. Marco, along with Pablo, has been a
true mentor to me.
I extend my sincere appreciation to the entire COMÈTE team at Inria Saclay for fostering a stimulating academic environment. The collaborative opportunities and academic discussions within the team have been invaluable to my research journey. I am grateful for the friendships I have formed within the team. In particular, I would like to express my gratitude to Sayan Biswas, Carlos Pinzon, Santiago Quintero, Sergio Ramirez, Ganesh del Grosso, Renan Spencer Trindade, and Fabricio Morales for their support. Each of them has played a role in my journey. I appreciate the meaningful connections we have formed and the experiences we have shared. Lastly, I would like to thank Ruben Santisteban Salazar, with whom I am rediscovering France and Paris through a different lens and who is making my current days happier.

Finally, I am deeply grateful to my family for their unwavering encouragement, and understanding throughout my academic pursuits. The constant support and belief in my abilities of my parents, Alfonso Granse and Nissia Figliuolo, have been my driving force. I am grateful for the profound reflections and insightful conversations I shared with my sister Libera Granese and Giovanni Recupido. Their guidance and wisdom have not only contributed to my professional growth but also enriched me as an individual. I value their support and the meaningful impact they have had on my personal and professional development.

To everyone who has contributed, directly or indirectly, to this thesis, I express my heartfelt appreciation. Thank you all.

PS: Finally, I would like to thank ChatGPT for helping me write these acknowledgments.
# Contents

<table><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td>1</td><td>Introduction</td><td>1</td></tr><tr><td>1.1</td><td>Motivation</td><td>1</td></tr><tr><td>1.2</td><td>From the general learning problem to detection</td><td>3</td></tr><tr><td>1.2.1</td><td>The general learning problem</td><td>3</td></tr><tr><td>1.2.2</td><td>Detection as hypothesis testing</td><td>4</td></tr><tr><td>1.3</td><td>Misclassification detection</td><td>5</td></tr><tr><td>1.4</td><td>Multi-armed adversarial attacks detection</td><td>7</td></tr><tr><td>1.5</td><td>Plan of the thesis and contributions</td><td>10</td></tr><tr><td>2</td><td>Preliminaries</td><td>13</td></tr><tr><td>2.1</td><td>Multiclass classification</td><td>13</td></tr><tr><td>2.1.1</td><td>Basic definitions</td><td>13</td></tr><tr><td>2.1.2</td><td>Error variable</td><td>14</td></tr><tr><td>2.2</td><td>Attacking neural networks</td><td>15</td></tr><tr><td>2.2.1</td><td>Adversarial problem</td><td>15</td></tr><tr><td>2.2.2</td><td>Why do adversarial examples exist?</td><td>16</td></tr><tr><td>2.2.3</td><td>Crafting adversarial attacks</td><td>18</td></tr><tr><td>2.2.4</td><td>Protecting from adversarial attacks</td><td>19</td></tr><tr><td>I</td><td>Misclassification Detection</td><td>21</td></tr><tr><td>3</td><td>DOCTOR: A Simple Method for Detecting Misclassification Errors</td><td>23</td></tr><tr><td>3.1</td><td>The Optimal Discriminator</td><td>24</td></tr><tr><td>3.1.1</td><td>Statistical model for detection</td><td>24</td></tr><tr><td>3.1.2</td><td>Performance metrics and optimal discriminator</td><td>24</td></tr></tbody></table>
<table><tr><td>3.2</td><td>The Proposed Discriminator: DOCTOR</td><td>26</td></tr><tr><td>3.2.1</td><td>DOCTOR discriminator</td><td>26</td></tr><tr><td>3.2.2</td><td>Discussion</td><td>27</td></tr><tr><td>3.2.3</td><td>From the theory to a practical discriminator</td><td>27</td></tr><tr><td>3.3</td><td>Evaluation</td><td>28</td></tr><tr><td>3.4</td><td>Experimental Results</td><td>28</td></tr><tr><td>3.4.1</td><td>Review of related methods</td><td>29</td></tr><tr><td>3.4.2</td><td>Detection of misclassification errors, experimental setup and evaluation metrics</td><td>31</td></tr><tr><td>3.4.3</td><td>Experimental results: comparison between different discriminators</td><td>33</td></tr><tr><td>3.5</td><td>Final remarks</td><td>36</td></tr></table>

# II Multi-Armed Adversarial Attack Detection 39

## 4 MEAD 41

<table><tr><td>4.1</td><td>Generating adversarial examples according to different objectives.</td><td>42</td></tr><tr><td>4.2</td><td>A case study: ACE vs. Gini Impurity</td><td>43</td></tr><tr><td>4.3</td><td>Evaluation with a Multi-Armed Attacker</td><td>45</td></tr><tr><td>4.4</td><td>Experiments</td><td>46</td></tr><tr><td>4.4.1</td><td>Experimental setting</td><td>46</td></tr><tr><td>4.4.2</td><td>Experimental results</td><td>49</td></tr><tr><td>4.5</td><td>Final remarks</td><td>52</td></tr></table>

## 5 A Minimax Approach Against MEAD 53

<table><tr><td>5.1</td><td>Formalization of the Problem of Detecting multi-armed Adversarial Attacks</td><td>54</td></tr><tr><td>5.1.1</td><td>Statistical model</td><td>54</td></tr><tr><td>5.1.2</td><td>A novel objective for detection under simultaneous attacks</td><td>54</td></tr><tr><td>5.2</td><td>Experimental Results</td><td>58</td></tr><tr><td>5.2.1</td><td>Evaluation framework</td><td>58</td></tr><tr><td>5.2.2</td><td>Discussion</td><td>61</td></tr><tr><td>5.3</td><td>Final remarks</td><td>67</td></tr></table>
# 6 Conclusion of the Thesis 71

## References

### A Appendix to Chapter 3

<table><tr><td>A.1</td><td>Proofs</td></tr><tr><td>A.1.1</td><td>Proof of Proposition 1</td></tr><tr><td>A.1.2</td><td>Proof of Proposition 2</td></tr><tr><td>A.1.3</td><td>Proof of Inequalities in (2.3)</td></tr><tr><td>A.2</td><td>Logistic Regression and Gaussian Model</td></tr><tr><td>A.2.1</td><td>Theoretical analysis</td></tr><tr><td>A.2.2</td><td>Experiments</td></tr><tr><td>A.3</td><td>Supplementary Results of Section 3.4</td></tr><tr><td>A.3.1</td><td>Experimental environment</td></tr><tr><td>A.3.2</td><td>On the input pre-processing in DOCTOR</td></tr><tr><td>A.3.3</td><td>On the effect the intervals considered for γ, δ and ζ have on the AUROC computation</td></tr><tr><td>A.3.4</td><td>Additional plots and results</td></tr><tr><td>A.3.5</td><td>DOCTOR for pure OOD detection</td></tr><tr><td>A.3.6</td><td>Some observations on the white-box scenario (WB)</td></tr></table>

### B Appendix to Chapter 4

<table><tr><td>B.1</td><td>Additional results</td></tr><tr><td>B.1.1</td><td>Additional Results on CIFAR10</td></tr><tr><td>B.1.2</td><td>Additional Results on MNIST</td></tr></table>

### C Appendix to Chapter 5

<table><tr><td>C.1</td><td>On the optimization of Eq. (5.5)</td></tr><tr><td>C.2</td><td>Supplementary Results of Section 5.2</td></tr><tr><td>C.2.1</td><td>On the MEAD framework</td></tr><tr><td>C.2.2</td><td>The proposed aggregator against the adaptive-attacks in the MEAD scenario</td></tr><tr><td>C.2.3</td><td>AutoAttack</td></tr><tr><td>C.2.4</td><td>Additional plots</td></tr></table>
<figure><img src="image_1.png" /></figure>
List of Figures

<table><tbody><tr><td>1.1</td><td>General Learning Problem [Vap95]</td><td>3</td></tr><tr><td>1.2</td><td>Misclassification detection</td><td>6</td></tr><tr><td>1.3</td><td>Multi-armed adversarial attacks detection</td><td>8</td></tr><tr><td>2.1</td><td>The effects of FGSM [GSS15]</td><td>15</td></tr><tr><td>3.1</td><td>Detectors' distribution: comparison between DOCTOR and the competitors</td><td>32</td></tr><tr><td>3.2</td><td>ROC curves: comparison between DOCTOR and the competitors</td><td>34</td></tr><tr><td>4.1</td><td>A case study: ACE vs. Gini Impurity</td><td>43</td></tr><tr><td>4.2</td><td>MEAD workflow</td><td>45</td></tr><tr><td>5.3</td><td>Box-plot: the shallow detectors, the aggregator and NSS</td><td>61</td></tr><tr><td>5.1</td><td>Performance of the shallow detectors grouped by L<sub>p</sub>-norm and perturbation magnitude &epsilon; on CIFAR10</td><td>62</td></tr><tr><td>5.2</td><td>Discrimination performances</td><td>63</td></tr><tr><td>A.1</td><td>Toy example: ROCs</td><td></td></tr><tr><td>A.2</td><td>The effect of varying T and &epsilon;</td><td></td></tr><tr><td>A.3</td><td>Overall results when varying T and &epsilon;</td><td></td></tr><tr><td>C.1</td><td>Box-plot: the shallow detectors, NSS and FS</td><td></td></tr><tr><td>C.2</td><td>The aggregator against the adaptive-attacks under MEAD</td><td></td></tr><tr><td>C.3</td><td>Additional plots</td><td></td></tr></tbody></table>
<figure><img src="image_2.png" /></figure>
List of Tables

<table><tbody><tr><td>3.1</td><td>Misclassification detection: overall results</td><td>36</td></tr><tr><td>3.2</td><td>Misclassification detection in presence of OOD samples</td><td>36</td></tr><tr><td>4.1</td><td>MEAD: summary of the results on CIFAR10</td><td>47</td></tr><tr><td>4.2</td><td>MEAD: summary of the results on MNIST</td><td>47</td></tr><tr><td>4.3</td><td>MEAD: comparison average performance</td><td>48</td></tr><tr><td>5.1</td><td>Group of simultaneous attacks</td><td>59</td></tr><tr><td>5.2</td><td>Comparison between the aggregator and NSS on CIFAR10 and SVHN</td><td>65</td></tr><tr><td>5.3</td><td>Comparison between Ours (the aggregator) and Ours+NSS on CIFAR10</td><td>66</td></tr><tr><td>5.4</td><td>The aggregator and NSS in the non-simultaneous setting</td><td>68</td></tr><tr><td>A.1</td><td>Toy example: accuracy of f<sub>θ<sub>i</sub></sub> on the test set.</td><td></td></tr><tr><td>A.2</td><td>Toy example: AUROCs</td><td></td></tr><tr><td>A.3</td><td>The effects of varying the number of thresholds</td><td></td></tr><tr><td>A.4</td><td>DOCTOR for pure OOD detection</td><td></td></tr><tr><td>A.5</td><td>DOCTOR for OOD detection of samples similar to the in-distribution ones.</td><td></td></tr><tr><td>A.6</td><td>White-box setting</td><td></td></tr><tr><td>A.7</td><td>Misclassification detection in presence of OOD samples: overall results</td><td></td></tr><tr><td>B.1</td><td>Average number of successful attacks per natural sample considered in the single-armed setting and MEAD (CIFAR10)</td><td></td></tr><tr><td>B.2</td><td>Performances on NSS per objective and in MEAD on CIFAR10</td><td></td></tr></tbody></table>
CHAPTER 1

Introduction

1.1 Motivation

Machine learning (ML) has rapidly become integral to various industries, includ-
ing healthcare, finance, transportation, and entertainment. The goal is to enable
machines to learn from data and make predictions or decisions without explicit
instructions using algorithms and statistical models. Even if ML has revolution-
ized how we process and analyze data, it poses new risks and challenges, par-
ticularly as these technologies are quickly being applied to critical systems, such
as autonomous driving vehicles or industrial robots, including-but not limited
to-classification and decision-making tasks.

Therefore, a major concern with ML is its safety or the possibility of unintended consequences. Developing methods and tools to make these algorithms reliable, particularly for non-specialists who may treat them as “black boxes” with no further checks, constitutes a core challenge. Some of the risks associated with using ML technologies include bias and discrimination in decision-making, vulnerabilities to cyber-attacks, and unintended outcomes due to poorly designed or trained algorithms: according to the Washington Post, in 2021, Tesla vehicles equipped with autopilot software were involved in 273 reported crashes, some of which were fatal<fnref n="1" />; in 2014 the Amazon AI-based experimental hiring tool was discovered to be biased against women²; in 2021 IBM's Watson started provid-

<footnote n="1">¹https://www.washingtonpost.com/technology/2022/06/15/tesla-autopilot-crashes/
²https://www.reuters.com/article/amazon-com-jobs-automation/insight-amazon-scrapse-cret-ai-recruiting-tool-that-showed-bias-against-women-idINKCN1MK0AH?edition-redirec
t=in</footnote>
ing incorrect and several unsafe recommendations for the treatment of cancer patients<fnref n="3" />.

Researchers and practitioners have made several efforts to improve the safety of ML systems, including creating transparent and interpretable ML models, devising resilient and secure algorithms, and considering ethical concerns during the development and implementation of ML systems. In particular, Hendrycks et al. [HCSS21] categorize the study of machine learning (ML) safety into four main areas. *Robustness*, involving both long-tail and adversarial robustness. Long-tail robust models are required to be resilient to long-tail events, i.e., events that are harder to predict (e.g., 9/11, the financial crisis of 2008, and COVID-19), and generally result in catastrophic outcomes. These events are characterized by long-tail distributions, i.e., probability distributions whose tails are not exponentially bounded [Asm03]. On the other hand, adversarial robustness requires models to be resilient to carefully crafted and deceptive threats rather than unpredictable ones. *Monitoring* highly correlated with anomaly detection, i.e., detecting data instances that significantly deviate from the majority of data instances [PSCvdH22]. One research topic in which anomaly detection is actively studied is out-of-distribution detection (OOD) [HMD19], where the main goal is to prevent errors by identifying potential drifts of the testing distribution. Monitoring also concerns confidence calibration, i.e., the problem of predicting probability estimates representative of the true correctness likelihood [GPSW17]. *Alignment* which, as the name implies, refers to aligning the objective functions used to drive system behavior with human values. For instance, one definition of fairness in machine learning considers discrimination against a specified sensitive attribute in supervised learning [HPS16]. *Systematic safety*, which includes cybersecurity and tools to help decision-makers handle ML systems in highly uncertain, quickly evolving, turbulent situations [HCSS21].

<footnote n="3">³https://www.nytimes.com/2021/07/16/technology/what-happened-ibm-watson.html</footnote>
<figure><img src="image_3.png" /><figcaption>**Figure 1.1: General Learning Problem [Vap95].** In light blue is the training phase, and in red is the testing phase.</figcaption></figure>

## 1.2 From the general learning problem to detection

### 1.2.1 The general learning problem

To help the comprehension of the next sections, we briefly recall in Fig. 1.1 the *general learning problem* [Vap95]. As shown in Fig. 1.1, the general model consists of three components: (i) the *nature* namely the fixed but unknown distribution $p_X$ over a $\mathcal{X} \subseteq \mathbb{R}^d$, where $\mathcal{X}$ is also known as input space; (ii) the *object* namely the fixed but unknown conditional distribution $p_{Y|\mathcal{X}}$ which returns an output value $y \in \mathcal{Y}$ (the concept) to every input vector $\mathbf{x} \in \mathcal{X}$, where $\mathcal{Y}$ is the label space that can be either discrete or continuous; (iii) the *learning machine* capable of implementing a set of functions $g_\theta(\mathbf{x}), \theta \in \Theta$, where $\Theta$ is a set of parameters. Classical machine learning aims to select the function that best approximates the object's output: given a training set of $n$ i.i.d. observations $\{(\mathbf{x}_i, y_i)\}_{i=1}^n$ drawn according to $p_{XY} = p_X p_{Y|\mathcal{X}}$, we wish to identify a function $g_\theta$, namely the predictor (e.g., neural network), minimizing the empirical risk, i.e., the discrepancy between the object's response $y$ to the input $\mathbf{x}$ and the response provided by the learning machine. Based on this approach, if the object's output and the learning machine algorithm's output are highly likely to be the same, we say the predictor performs well. In general, this is called the *philosophical instrumentalism approach* (imitation of the object): science's role is to predict the concept, regardless of the law of nature. This is slightly different from the *philosophical realism approach* (approximation of the object) where science's role is to try to approximate the object itself.
### 1.2.2 Detection as hypothesis testing

After addressing the initial learning problem depicted in Fig. 1.1, we now proceed to the next phase by creating a decision rule that employs the acquired distribution $p_{\hat{Y}|X}$ to resolve a specific detection task.

Detection problems are usually cast as binary (or M-ary) hypothesis testing. Suppose it is given to us a random variable (r.v.) $X$ following one of the two probability density functions (pdf) $p_0$ and $p_1$ defined on a finite set $\mathcal{X}$. The true distribution is unknown to us, but we want to distinguish between the following two hypotheses, i.e., $\mathbb{H}_0$: $X \sim p_0 \equiv p_{X|H}(\cdot|0)$ (also known as *null hypothesis*); $\mathbb{H}_1$: $X \sim p_1 \equiv p_{X|H}(\cdot|1)$. The objective of binary hypothesis testing is then to develop a decision rule $d: \mathcal{X} \to \{0, 1\}$ for making the best guess about which hypothesis is correct. This context can lead to four possible outcomes. Generally, when $\mathbb{H}_1$ is true, and we choose it, we refer to that situation as a *detection*. Likewise when $\mathbb{H}_0$ is true but we select $\mathbb{H}_1$ we call that a *false alarm* (*Type-I error*); when $\mathbb{H}_1$ is true but we choose $\mathbb{H}_0$ we call that a *miss* (*Type-II error*). The decision rule will partition the input space $\mathcal{X}$, into two regions: $\mathcal{X}_0$ where the observations are consistent with $\mathbb{H}_0$; $\mathcal{X}_1$ where the observations are consistent with $\mathbb{H}_1$. In particular, if we characterize the probability of detection of $d$ as

$$P_D(d) \stackrel{\text{def}}{=} \Pr[\mathcal{X}_1 | \mathbb{H}_1] = \int_{x \in \mathcal{X}_1} P_1(x)dx$$

and, likewise, the probability of false alarm of $d$ as

$$P_F(d) \stackrel{\text{def}}{=} \Pr[\mathcal{X}_1 | \mathbb{H}_0] = \int_{x \in \mathcal{X}_1} P_0(x)dx,$$

we ideally would like to have $P_D(d) \to 1$ and $P_F(d) \to 0$. Clearly, the main aspiration would be to derive a decision rule splitting the input space optimally, e.g., if we suppose $\Pr[\mathbb{H}_0]$ and $\Pr[\mathbb{H}_1]$ to be known (a priori probabilities), the optimal decision rule will be, for every $x \in \mathcal{X}$:

$$ \begin{align} p_{H|X}(1|x) &\overset{\mathcal{X}_1}{\ge} p_{H|X}(0|x) \nonumber \\ \Pr[\mathbb{H}_1] p_{X|H}(x|1) &\overset{\mathcal{X}_1}{\ge} \Pr[\mathbb{H}_0] p_{X|H}(x|0). \tag{1.1} \end{align} $$
That is, we assign to $\mathcal{X}_1$ all the samples for which the left side of Eq. (1.1) is greater or equal than the left side; viceversa we assign to $\mathcal{X}_0$ all the samples for which the left side of Eq. (1.1) is smaller than the right side. However, this ideal scenario cannot occur when the two underlying distributions overlap. Thus, to increase the detection probability, we must also allow for the probability of false alarm to increase. This represents the fundamental tradeoff in hypothesis testing and detection theory. We can rewrite Eq. (1.1) as follows

$$ \Pr[\mathbb{H}_1] p_{X|H}(x|1) \underset{\mathcal{X}_0}{\overset{\mathcal{X}_1}{\gtrless}} \gamma \cdot \Pr[\mathbb{H}_0] p_{X|H}(x|0), \quad (1.2) $$

where $\gamma \in \mathbb{R}$ is the threshold regulating the tradeoff. We can assess the performance of the proposed decision rule in terms of *Receiver operating characteristic* (ROC) representing the upper boundary between achievable and un-achievable regions in the $(P_F, P_D)$-square. The detectors' goal will be to partition the input space accordingly.

The main aim of this thesis is to explore the behavior of neural networks when presented with input samples that are either ‘clean’ (or ‘natural’) but may be wrongly classified (*misclassification detection*), as well as samples ‘adversarial’ that have been intentionally manipulated to deceive the model (*adversarial detection*). Our objective is, therefore, to develop a decision rule that can identify, during testing, when an input sample is likely to cause the network to exhibit a specific behavior and make decisions based on this information.

## 1.3 Misclassification detection

The scheme in Fig. 1.2 refers to the setting of *misclassification detection*. The goal of the detector is to check whether the prediction made by the classifier is correct and accept or reject it accordingly. Let $E(\mathbf{x}) \stackrel{\text{def}}{=} \mathbb{1}[Y \neq g_{\theta}(\mathbf{x})]$ denote the error variable for a given $\mathbf{x} \in \mathcal{X}$ with respect to (w.r.t.) $g_{\theta}$, where $\mathbb{1}[\mathcal{E}]$ is the indicator vector which outputs 1 if $\mathcal{E}$ is true and 0 otherwise. The idea is to model the data distribution as a mixture of two pdfs, one representing the distribution of the wrongly classified samples and the other of the correctly classified samples (see upper left box of Fig. 1.2). More formally, $p_{X|E}(\mathbf{x}|1)$ is the pdf truncated
<figure><img src="image_4.png" /><figcaption>**Figure 1.2: Misclassification Detection.** DETECTOR in cyan represents either $D_\alpha$ or $D_\beta$. The initial learning process is not shown, indicating that it will not be repeated.</figcaption></figure>

to the error event $\{E = 1\}$ (i.e., the hard decision fails) and $p_{X|E}(\mathbf{x}|0)$ is the pdf truncated to the success event $\{E = 0\}$ (i.e., the hard decision succeeds). The problem is therefore cast as in Section 1.2.2 by first identifying our hypothesis

$$H_0: X \sim p_{X|E}(\mathbf{x}|0)$$

and

$$H_1: X \sim p_{X|E}(\mathbf{x}|1).$$

The most powerful (Oracle) discriminator at threshold $\gamma \in \mathbb{R}$ is defined as

$$D(\mathbf{x}, \gamma) = \begin{cases} 1, & \text{if } p_{X|E}(\mathbf{x}|1) \ge \gamma \cdot p_{X|E}(\mathbf{x}|0) \\ 0, & \text{otherwise,} \end{cases} \qquad (1.3)$$

where $D(\mathbf{x}, \gamma) = 1$ denotes the sample's prediction is going to be rejected. In Chapter 3, we will show that by applying Bayes theorem Eq. (1.3) can be rewritten in terms of probability of classification error $\mathbf{Pe}(\cdot)$ w.r.t. $p_{Y|X}$. Indeed $p_{E|X}(1|\mathbf{x}) = 1 - p_{Y|X}(g_\theta(\mathbf{x})|\mathbf{x}) = \mathbf{Pe}(\mathbf{x})$ and therefore

$$D(\mathbf{x}, \gamma) = \begin{cases} 1, & \text{if } \mathbf{Pe}(\mathbf{x}) \ge \gamma' \cdot (1 - \mathbf{Pe}(\mathbf{x})) \\ 0, & \text{otherwise.} \end{cases} \qquad (1.4)$$

To conclude, in Chapter 3 we will present two practical detectors based on the approximation of $\mathbf{Pe}(\mathbf{x})$. We recall the detector in Eq. (1.4) supposes to have
access to all the involved distributions that are typically unknown. Consequently, the practical detectors can rely only on the model posterior distribution $p_{\hat{Y}|X}$

$$D_{\alpha}(\mathbf{x}, \gamma) = \begin{cases} 1, & \text{if } \text{Gini}(\mathbf{x}) \ge \gamma' \cdot (1 - \text{Gini}(\mathbf{x})) \\ 0, & \text{otherwise.} \end{cases} \qquad (1.5)$$

and

$$D_{\beta}(\mathbf{x}, \gamma) = \begin{cases} 1, & \text{if } \hat{\text{Pe}}(\mathbf{x}) \ge \gamma' \cdot (1 - \hat{\text{Pe}}(\mathbf{x})) \\ 0, & \text{otherwise.} \end{cases} \qquad (1.6)$$

where $\text{Gini}(\mathbf{x}) \stackrel{\text{def}}{=} \sum_{y \in \mathcal{Y}} p_{\hat{Y}|X}(y|\mathbf{x}; \theta)(1 - p_{\hat{Y}|X}(y|\mathbf{x}; \theta))$ is the probability of incorrectly classifying the feature $\mathbf{x}$ if it was randomly labeled according to the model distribution and $\hat{\text{Pe}}(\mathbf{x}) \stackrel{\text{def}}{=} 1 - p_{\hat{Y}|X}(g_{\theta}(\mathbf{x})|\mathbf{x}; \theta)$ is the probability of classification error w.r.t. the predicted distribution.

The aforementioned detectors will be evaluated on either image or textual datasets in two alternatives scenario depending on the amount of information about the underlying classifier available. In the *totally black box* (TBB) we suppose only the soft-predictions are available in the *partially black box* (PBB) the gradient-propagation to perform input pre-processing is allowed. In particular, input pre-processing is a common technique used to slightly modify some patterns of an input sample in the direction of an objective loss to maximize/minimize. The concept is the same as when training a neural network where the weights are modified step-by-step to minimize the training loss. In our context, the modification will affect the pixel of the images in the direction where the loss in Eqs. (1.5) and (1.6) maximizes. This technique will show up to be particularly effective for our goal.

## 1.4 Multi-armed adversarial attacks detection

We recall that adversarial examples are carefully crafted input patterns designed to deceive a target classifier into making an incorrect decision while remaining as similar as possible to the original sample. More formally, let us consider a natural sample, denoted by $\mathbf{x} \in \mathcal{X}$, along with its true label, $y \in \mathcal{Y}$. An attacker aims to
<figure><img src="image_5.png" /><figcaption>**Figure 1.3: Multi-armed adversarial attacks detection.** The double arrows mean that the outputs are for each $\mathbf{x}^{(k)}$. Note that $K, J \in \mathbb{N}$ but $J$ can be different from $K$. $\mathcal{B}_\mathcal{E}(p)$ denotes the Bernoulli distribution of parameter $p \in [0, 1]$. The initial learning process is not shown, indicating that it will not be repeated.</figcaption></figure>

deceive the model $g_\theta$ by crafting an adversarial example, $\mathbf{x}'_\ell \in \mathcal{I} \subseteq \mathbb{R}^d$, where $\mathcal{I}$ is a held-out set of images that is distributed according to $p_{XY}$ but that was not used during training. The symbol $\ell$ denotes the objective loss function $\ell(\mathbf{x}, \mathbf{x}'_\ell; \theta)$ optimized by the attacker; $\varepsilon$ is perturbation magnitude, and $L_p, p \in \{1, 2, \infty\}$ is the norm constraint. The goal of the attack is to obtain an $\mathbf{x}'_\ell$ such that $g_\theta(\mathbf{x}'_\ell) \neq g_\theta(\mathbf{x})$, in order to force the target model to make a prediction error. As thoroughly investigated in [SZS$^{+14}$], the adversarial generation problem is difficult to tackle and it is commonly relaxed as follows

$$ \mathbf{x}'_\ell \equiv \mathbf{x}'_\ell(\mathbf{x}) = \underset{\mathbf{x}'_\ell \in \mathbb{R}^d : \|\mathbf{x}'_\ell - \mathbf{x}\|_p < \varepsilon}{\operatorname{argmax}} \ell(\mathbf{x}, \mathbf{x}'_\ell; \theta), \quad (1.7) $$

where $\mathbf{x}'_\ell$ is updated iteration by iteration starting from an initial given value.

The scheme in Fig. 1.3 refers to the setting of *multi-armed adversarial attacks detection*. In this case, the goal of the detector is to check whether the input sample is natural or has been adversarially perturbed according to *some* strategy. We refer to this setting as ‘multi-armed’ as in the classical detection setting (i.e., ‘single-armed’) the methods are generally validated by assuming a *single* attack strategy at a time. The proposed setting, as reported in Fig. 1.3, considers the evaluation of multiple instances at the same time. Consequently, in Chapter 4 we suggest an alternative framework for evaluating the performance
of the existing state-of-the-art detectors when the attacks at the evaluation time can be simultaneously crafted according to various algorithms and objective loss functions. The detection will be successful if and only if all different attacks are correctly recognized.

We provide a solution to the problem of multi-armed adversarial attack detection in Chapter 5 by dealing with the following worst-case scenario. Assume it is given a distribution for every attack strategy. We could group all such distributions in a set $\mathcal{M} = \{p_{X|Z}^{(k)} : k \in \mathcal{K}\}$, where $k \in \mathcal{K}$ represents the index and $\mathcal{Z} = \{0, 1\}$ indicates a binary space label for the adversarial example detection task. At the evaluation time, the attackers select an arbitrary strategy $i$ and then sample an input $\mathbf{x}$ according to $p_{X|Z}^{(i)}(\mathbf{x}|z)$, where $p_{X|Z}^{(i)}(\mathbf{x}|1)$ is the probability density function induced by the chosen attack $i$ and $p_{X|Z}^{(i)}(\mathbf{x}|0) = p_X(\mathbf{x})$ almost surely is the probability distribution of the natural samples. The defender is asked to choose between the following two hypotheses

$$ \mathbb{H}_0 : X \sim p_{X|Z}^{(k)}(\mathbf{x}|0) \text{ for some } k \in \mathcal{K} $$

and

$$ \mathbb{H}_1 : X \sim p_{X|Z}^{(k)}(\mathbf{x}|1) \text{ for some } k \in \mathcal{K}. $$

Moreover, suppose the defender has at her/his disposal a set of soft-detectors one for each of the possible distributions in $\mathcal{M}$,

$$ \mathcal{Q} = \left\{ q_{\hat{Z}|\mathbf{u}}^{(k)} : \mathcal{U} \mapsto [0, 1]^2 \right\}_{k \in \mathcal{K}}, $$

with $\mathbf{u} \in \mathcal{U} = \{g_{\theta}^{l}(\mathbf{x}) | \mathbf{x} \in \mathbb{R}^d\}$ denotes the space of logits. It is also important to keep in mind that the defender does not know what the attacker's strategy will be. The optimal detector will be the one performing simultaneously well over all the possible attacks in $\mathcal{M}$. In Chapter 5 we will show that this can be formalized as the solution to the following minimax problem

$$ \mathcal{L}(\mathcal{Q}, \mathbf{x}) = \min_{q_{\hat{Z}|\mathbf{u}}} \max_{k \in \mathcal{K}} \mathbb{E}_{q_{\hat{Z}|\mathbf{u}}^{(k)}} \left[ -\log q_{\hat{Z}|\mathbf{u}} \right], \quad (1.8) $$

where the minimization is performed over all (detectors) distributions $q_{\hat{Z}|\mathbf{u}}$, including elements that are not part of the set $\mathcal{Q}$. Thus, the optimal detector will
be as follows

$$D(\mathbf{x}, \gamma) = \begin{cases} 1, & \text{if } \mathcal{L}(\mathcal{Q}, \mathbf{x}) \ge \gamma \\ 0, & \text{otherwise.} \end{cases} \qquad (1.9)$$

Notice that, Eq. (1.8) is not tractable computationally. Therefore, in Chapter 5, we show how to derive a surrogate function that can be computationally optimized.

The aforementioned detector will be evaluated in the context of multi-armed adversarial attack detection (as well as in the ‘single-armed’ setting) by assuming that a third party provides us with four simple supervised detectors (i.e., the detectors in $\mathcal{Q}$) each of them trained to detect a single specific kind of attack. Indeed, in practical setting, besides not knowing the attack strategy, we also do not have access to all the possible detectors which adds to the difficulty.

# 1.5 Plan of the thesis and contributions

## Publications from this dissertation

The content of this dissertation is based on the following publications:

**Part I** is based on the results presented in **DOCTOR: A Simple Method for Detecting Misclassification Errors** [GRG$^{+21}$] (Chapter 3), that appeared as *Spotlight* in the proceedings of the 35th *Conference on Neural Information Processing Systems (NeurIPS2021)*. Federica Granese, Marco Romanelli, Daniele Gorla, Catuscia Palamidessi, Pablo Piantanida.

**Part II** is based on the results presented in: i) **MEAD: A Multi-Armed Approach for Evaluation of Adversarial Examples Detectors** [GPR$^{+22}$] (Chapter 4), that appeared in the proceedings of the 33rd *European Conference on Machine Learning and Data Mining (ECMLPKDD2022)*. Federica Granese, Marine Picot, Marco Romanelli, Francisco Messina, Pablo Piantanida. ii) **A Minimax Approach Against Multi-Armed Adversarial Attacks Detection** [GRGP23] (Chapter 5), that has been submitted to the 36th *IEEE Computer Security Foundations Symposium*
(CSF). Federica Granese, Marco Romanelli, Siddharth Garg, Pablo Pi-
antanida.

Other publications

Other works I have contributed to during my PhD (at the time I am writing the
thesis):

A) Works in Social Network Privacy/Utility and Differential-Privacy.

On the one hand, we studied how to control information propagation in social networks. Users want to communicate and interact freely with their peers. However, if misused, the information they spread can have harmful consequences. There is, therefore, a trade-off between utility, i.e., reaching as many intended nodes as possible, and privacy, i.e., avoiding the unintended ones. In [GGP21], we adapt the basic framework of Backes et al. [BGMS17] to include more realistic features, that in practice influence the way in which information is passed around. More specifically, we consider: (a) the topic of the shared information, (b) the time spent by users to forward information among them, and (c) the user's social behavior. Furthermore, we propose an enhanced formulation of the utility/privacy policies, to maximize the expected number of reached users among the intended ones, while minimizing this number among the unintended ones, and we show how to adapt the basic techniques to these enhanced policies. On the other hand, in cite [GJG$^{+22}$], we look at the problem of data protection. Differential privacy is nowadays one of the best established and theoretically solid tools to ensure data protection. Intuitively, given a set of databases, differential privacy requires that databases that only slightly differ one from the other (e.g., in one individual record) are mapped to the obfuscated values with similar probabilities; this provides privacy to the changed record because statistical functions run on the database should not overly depend on the data of any individual. In this work, we analyze to what extent final users can infer information about the level of protection of their data when the data obfuscation mechanism is a priori unknown to them.

Papers:

- Enhanced models for privacy and utility in continuous-time diffusion
networks, that appeared in the proceedings of the *International Journal of Information Security* [GGP21]. Federica Granese, Daniele Gorla, Catuscia Palamidessi.

- On the (Im)Possibility of Estimating Various Notions of Differential Privacy [GJG$^{+22}$], that has been submitted to 36th *IEEE Computer Security Foundations Symposium*. Daniele Gorla, Louis Jalouzot, Federica Granese, Catuscia Palamidessi, Pablo Piantanida.

B) Additional works in Machine Learning. On the one hand, we propose a new method (HAMPER) to detect adversarial examples by leveraging the concept of data depths, a statistical notion that provides center-outward ordering of points w.r.t. a probability distribution. In particular, the halfspace-mass (HM) depth exhibits attractive properties such as computational efficiency, which makes it a natural candidate for adversarial attack detection in high-dimensional spaces. Additionally, HM is non-differentiable making it harder for attackers to attack HAMPER via gradient based-methods directly. On the other hand, inspired by the work in [GRG$^{+21}$], we present a simple yet effective hyperparameter free method to implement the rejection option for a pre-trained classifier. The method is lightweight since it does not require any re-training of the network, and it is flexible since it can be used with any model that outputs soft-probabilities.

## Papers:

- A Halfspace-Mass Depth-Based Method for Adversarial Attack Detection, that has been accepted (with minor revision) to *Transactions on Machine Learning Research (TMLR)*. Marine Picot*, Federica Granese*, Guillaume Staerman, Marco Romanelli, Francisco Messina, Pablo Piantanida, Pierre Colombo.

- Trusting the Untrustworthy: A Cautionary Tale on the Pitfalls of Training-based Rejection Option, that has been submitted to 40th *International Conference on Machine Learning (ICML)*. Eduardo Dadalto Câmara Gomes*, Marco Romanelli*, Federica Granese, Siddharth Garg, Pablo Piantanida.

<footnote>*Equal contribution.</footnote>
# Preliminaries

This chapter aims to recall the fundamental concepts necessary to understand the thesis' content. We indicate for each section which chapter of the thesis it is related to.

## 2.1 Multiclass classification

In the following section, we begin by recalling some basic notions of machine learning that will be useful in the course of Chapters 3 and 4.

### 2.1.1 Basic definitions

Let $\mathcal{X} \subseteq \mathbb{R}^d$ be the (possibly continuous) feature space and let $\mathcal{Y} = \{1, \dots, C\}$ denote the concept of the label space related to some task of interest. We denote by $p_{XY}$ the unknown data distribution over $\mathcal{X} \times \mathcal{Y}$. A predictor $g_{\theta}: \mathcal{X} \to \mathcal{Y}$ uses the inferred model $p_{\hat{Y}|X}(y|\mathbf{x}; \theta)$ where $y \in \mathcal{Y}$ and $\theta \in \Theta$ are the learnt parameters,

$$g_{\theta}(\mathbf{x}) \stackrel{\text{def}}{=} \arg \max_{y \in \mathcal{Y}} p_{\hat{Y}|X}(y|\mathbf{x}; \theta),$$

and tries to approximate the optimal (Bayes) decision rule $g^*(\mathbf{x}) \stackrel{\text{def}}{=} \arg \max_{y \in \mathcal{Y}} p_{Y|X}(y|\mathbf{x})$. Notice that $p_{\hat{Y}|X}$ can be interpreted as the prediction of the class (label) posterior probability given a sample (e.g., $p_{\hat{Y}|X}(y|\mathbf{x}; \theta) \equiv \text{softmax}(\mathbf{x})_y$), while $p_{Y|X}$ is the true (unknown) probability. In several practical scenarios $p_{\hat{Y}|X}$ does not perfectly match $p_{Y|X}$ and still $g_{\theta} \approx g^*$ (cf. [GPSW17]).
### 2.1.2 Error variable

Let $E(\mathbf{x}) \stackrel{\text{def}}{=} 1[Y \neq g_{\theta}(\mathbf{x})]$ denote the error variable for a given $\mathbf{x} \in \mathcal{X}$ corresponding to $g_{\theta}$, i.e., where we denote with $1[\mathcal{E}]$ the indicator vector which outputs 1 if the event $\mathcal{E}$ is true and 0 otherwise. Similarly, we can define the self-error variable $\hat{E}(\mathbf{x}) \stackrel{\text{def}}{=} 1[\hat{Y} \neq g_{\theta}(\mathbf{x})]$ also corresponding to the inferred predictor $g_{\theta}$ but based on the prediction model $p_{\hat{Y}|X}$ of the class posterior probability. Notice that $\hat{E}(\mathbf{x})$ is observable since the underlying distribution is known. However, $E(\mathbf{x})$ cannot be observed and in general these binary variables do not coincide.

At this stage, it is convenient to introduce the notions of probability of classification error for a given $\mathbf{x} \in \mathcal{X}$ with respect to (w.r.t.) both the true class posterior and the predicted probabilities:

$$ \mathrm{Pe}(\mathbf{x}) \stackrel{\text{def}}{=} \mathbb{E}[E(\mathbf{x})|\mathbf{x}] = 1 - p_{Y|X}(g_{\theta}(\mathbf{x})|\mathbf{x}), \qquad (2.1) $$

$$ \widehat{\mathrm{Pe}}(\mathbf{x}) \stackrel{\text{def}}{=} \mathbb{E}[\widehat{E}(\mathbf{x})|\mathbf{x}] = 1 - p_{\widehat{Y}|X}(g_{\theta}(\mathbf{x})|\mathbf{x}; \theta). \qquad (2.2) $$

Notice that $\widehat{\mathrm{Pe}}(\mathbf{x})$ represents the probability of misclassification of the sample $\mathbf{x}$ with respect to the softmax probability $p_{\widehat{Y}|X}$, which can be interpreted as the model's approximation of nature $p_{Y|X}$. Such approximation is close when the model is well-calibrated. Obviously, $\mathrm{Pe}^*(\mathbf{x}) \le \mathrm{Pe}(\mathbf{x})$ for all $\mathbf{x} \in \mathcal{X}$, where $\mathrm{Pe}^*(\mathbf{x})$ corresponds to the minimum error of the Bayes classifier: $\mathrm{Pe}^*(\mathbf{x}) = 1 - p_{Y|X}(g^*(\mathbf{x})|\mathbf{x})$. It is worth mentioning that, by averaging (2.1) over the data distribution, we obtain the error rate of the classifier $g_{\theta}$. Although $\widehat{\mathrm{Pe}}(\mathbf{x})$ provides a valuable candidate to infer the unknown error variable $E(\mathbf{x})$, it is easy to check that

$$ \max \{\mathrm{Pe}(\mathbf{x}), \widehat{\mathrm{Pe}}(\mathbf{x})\} - \mathrm{Pr}(\hat{Y} = Y|\mathbf{x}) \le \mathrm{Pr}\{\hat{E}(\mathbf{x}) \ne E(\mathbf{x})|\mathbf{x}\} \le \mathrm{Pr}(\hat{Y} \ne Y|\mathbf{x}), \qquad (2.3) $$

which in particular implies that the error incurred in using $\hat{E}(\mathbf{x})$ to predict $E(\mathbf{x})$ is lower bounded by the classification error per sample (2.1) [GRG+21].

Note that, $\mathrm{Pe}(\mathbf{x})$ is not available in practical scenarios and the direct estimation (e.g., based on pairs of inputs and labels) of the true class posterior probability $p_{Y|X}$ cannot be performed. Notice that it is not possible to sample the
<figure><img src="image_6.png" /><figcaption>**Figure 2.1: The effects of FGSM.** The (well known) demonstration of fast adversarial example generation applied to GoogLeNet on ImageNet (from [GSS15]).</figcaption></figure>

conditional pdf $p_{Y|X}$ for each input $\mathbf{x} \in \mathcal{X}$. As a matter of fact, it is well-known that the application of direct methods for this estimation will lead to ill-posed problems, as shown in [VI20].

## 2.2 Attacking neural networks

In the following section, we provide the background necessary to understand Chapters 4 and 5. Note that this thesis is not fully devoted to studying the adversarial problem. As such, we refer to the survey in [AHFD22] and references therein for a comprehensive discussion of this topic.

### 2.2.1 Adversarial problem

*Adversarial examples* were first introduced in 2014 by Szegedy et al. [SZS+14] as a counter-intuitive property of deep neural networks.

Let us consider a natural sample $\mathbf{x} \in \mathcal{X}$ together with its true label $y \in \mathcal{Y}$. An attacker targets the model $g_{\theta}$ by crafting a sample $\mathbf{x}'_{\ell} \in \mathcal{I} \subseteq \mathbb{R}^d$ according to an objective loss function $\ell(\mathbf{x}, \mathbf{x}'_{\ell}; \theta)$ which is denoted by $\ell$, perturbation magnitude $\epsilon$, and norm constraint $L_p, p \in \{1, 2, \infty\}$. The goal of the attack is to obtain an $\mathbf{x}'_{\ell}$ such that $g_{\theta}(\mathbf{x}')_{\ell} \neq g_{\theta}(\mathbf{x})$, in order to force the target model to make a prediction error. An example is shown in Section 2.2.1).

Formally, Szegedy et al. [SZS+14] define the adversarial generation problem
as:

$$
\begin{array}{ll}
\underset{\mathbf{x}'_\ell \in \mathcal{I}}{\operatorname{argmin}} & \|\mathbf{x} - \mathbf{x}'_\ell\|_p \\
\text{s.t.} & g_\theta(\mathbf{x}'_\ell) \neq y,
\end{array}
$$

where $\mathcal{I}$ is a held-out set of images from the data distribution that the network was
not trained. As thoroughly investigated in [MMS$^{+18}$], the adversarial generation
problem as above is difficult to tackle and it is commonly relaxed as follows

$$
\mathbf{x}'_{\ell} \equiv \mathbf{x}'_{\ell}(\mathbf{x}) = \underset{\mathbf{x}_{\ell'} \in \mathcal{I}: \|\mathbf{x}_{\ell'} - \mathbf{x}\|_p < \epsilon}{\operatorname{argmax}} \ell(\mathbf{x}, \mathbf{x}_{\ell'}, y; \theta), \quad (2.4)
$$

where $\mathbf{x}'_\ell$ is updated iteration by iteration starting from an initial given value
and $\ell(\mathbf{x}, \mathbf{x}'_\ell, y; \theta)$ is the objective of the attacker, representing a surrogate of the
constraint to fool the target classifier, i.e., $g_\theta(\mathbf{x}'_\ell) \neq y$. The objective function $\ell$
traditionally used is the Cross-Entropy (CE) [SZS$^{+14}$, MMS$^{+18}$]:

$$
\ell_{\text{ACE}}(\mathbf{x}, \mathbf{x}'_\ell, y; \theta) = - \log p_{\hat{Y}|X}(y|\mathbf{x}'_\ell; \theta). \quad (2.5)
$$

2.2.2 Why do adversarial examples exist?

Although adversarial examples are easy to grasp, there is much speculation as to why they exist. Below is a brief overview of the most popular hypotheses of the moment. As a starting point, it is worth recalling that natural images are believed to exist in a low-dimensional manifold embedded in a high-dimensional space [CBB19].

Originally, Szegedy et al. [SZS+14] argued that adversarial examples represent densely populated “pockets” in the input space with a low probability of being observed and correctly classified. Later, Goodfellow et al. [GSS15] found the existence of adversarial examples in the linear behavior of classifiers rather than the non-linear behavior of classifiers in high-dimensional spaces.

The *off-manifold assumption* is supported by numerous papers (e.g., [CBB19, FCSG17, MC17] to cite a few), and it is based on the idea that adversarial perturbations push the sample off of the natural data manifold. Interestingly Feinmann et al. [FCSG17] describe three different situations depending on the position of
the adversarial example w.r.t. the decision boundary of the classifier and the submanifold of the adversarial example's class. The adversarial example in the first scenario is close to the decision boundary but far from the submanifolds for the original and new predicted classes; the adversarial example lies in the pocket of the submanifold of the class assigned to the adversarial example in the second scenario; in the third scenario, the adversarial example lies near both the decision boundary and both submanifolds. Note that the off-manifold hypothesis may not be the final explanation on the reason behind the adversarial phenomenon as subsequent papers (e.g. [SHS19, XYF+22]) have shown the existence of on-manifold adversarial examples.

As one of the most widely accepted hypotheses suggests, adversarial examples may be directly related to *non-robust features*, i.e., features that are highly predictive but are brittle, making them incomprehensible to humans. According to this hypothesis, adversarial vulnerabilities are caused by non-robust features, and are not inherently related to standard training frameworks [IST+19]. For example, consider the case of a classifier trained to distinguish *dogs* from *cats*. In addition, suppose the two animals are differentiated based on the direction of the hair rather than the shape itself (features). In this scenario, furry can be considered a non-robust feature, whereas shape can be considered robust. Particularly, the authors demonstrate that training a *non-robust* classifier on the original dataset based on exclusively robust features can achieve both good standard and robust accuracy, which is not the case when training the classifier using *non-robust* features.

By following a different perspective, the *dimpled manifold model* is a new mental model proposed in 2021 by Shamir et al. [SMB21] in which classifiers place their decision boundaries right next to data manifolds and only slightly curve around them. By going perpendicular to the manifold, adversarial examples can then be found.

Even though there are several possible other explanations for the existence of the adversarial examples (e.g., insufficient data [SST+18] and over-fitting [TG16]), a definitive answer has not yet been found. We would like to highlight that these assumptions are not necessarily mutually exclusive, suggesting (maybe) a unifying theory.
### 2.2.3 Crafting adversarial attacks

Over the years, a plethora of algorithms to generate adversarial samples has been proposed and, overall, we can group them into two main categories: *white-box* and *black-box* attacks.

#### White-box attacks

We talk about *white-box* attacks when the adversary knows everything about the target model (its architecture and weights). *Gradient-based* attacks belong to this category. They rely on finding the perturbation direction, i.e., the sign of gradient at each pixel of the input, that maximizes the attacker's objective value. Examples of gradient-based attacks are the *Fast Gradient Sign Method* (FGSM) [GSS15], the *Basic Iterative Method* (BIM) [KGB] and the *Projected Gradient Descent* method (PGD) [MMS$^{+18}$. BIM and PGD can be seen as iterative versions of FGSM (one-step perturbation). Unlike BIM, PGD attacks start from a random perturbation in $L_p$-ball around the input sample. Another powerful attack is the *Carlini-Wagner* attack (CW) [CW17c], which directly minimizes the additive noise constrained by a function which assure the misclassification of the perturbed sample. We conclude the list of white-box attacks by mentioning the *DeepFool* attack (DF) [MFF16], which is an iterative method based on a local linearization of the targeted classifier, and the resolution of the resulting simplified adversarial problem.

#### Black-box attacks

In the case of *black-box* attacks, the adversary has no access to the internals of the target model, hence it creates attacks by querying the model and monitoring outputs of the model to attack. Examples of black-box attacks are the *Square Attack* (SA) [ACFH20], which iteratively searches for a random perturbation, and checks if it increases the attacker's objective at each step; the *Hop Skip Jump* attack (HOP) [CJW20] which estimates the gradient direction to perturb, and the *Spatial Transformation Attack* (STA) [ETT$^{+19}$] which transforms the original samples by applying small translations and rotations to them.

It is worth to mention that there also exists *gray-box* attacks, i.e. when the adversary knows the training data but not the internals of the model. These
attacks rely on the transferability property of the adversarial examples: to create attacks these methods build a substitute model that performs the same task as the target model. A special class of attacks are the so-called *adaptive attacks* [ACW18, TCBM20, CW17c, YBTV21] where attacks are specifically designed to target a given defence. In this scenario, the attacker is supposed to have full knowledge of both the targeted classifier and the underlying defence.

### 2.2.4 Protecting from adversarial attacks

Methods to defend deep models against adversarial attacks can be grouped into two main families: methods that aim to increase the targeted model's robustness by re-training it [GSS15, MMS$^{+}$18, PMB$^{+}$22, XTG$^{+}$20, TKP$^{+}$18, AF21], and methods engineered to detect adversarial examples at evaluation time [KFHD20, MLW$^{+}$18, FCSG17, XEQ18, MC17, LLLS18a]. The work in [AHFD22] provides a recent and thorough survey about the state-of-the-art detection methods, which fall under two main categories: *supervised* and *unsupervised*. Below is a brief description of the methods analysed in the thesis.

#### Supervised methods

Detectors within this category extract features either directly from the targeted network's layer [KFHD20, FCSG17] or by using statistical tools [MLW$^{+}$18, LLLS18a]. To do so, both natural and adversarial examples are necessary. Generally, the adversarial samples are created according to a single fixed algorithm and a given loss function, which are then also used to create the examples at evaluation time.

*Natural Scene Statistics (NSS) [KFHD20]*. The approach is based on the hypothesis that natural images possess certain regular statistical properties (i.e., natural scene statistics [SLSZ03, Rud94]) that are altered by adversarial perturbations. Thus, by characterizing these deviations from the regularity of natural statistics using NSS, it is possible to determine whether the input is benign or malicious.

*Kernel Density and Bayesian Uncertainty (KD-BU) [FCSG17]*. Using the intuition that adversarial samples lie off the true data manifold, the method consists in extracting two types of features: *density estimates* that are computed starting from the logits of the network, and *bayesian uncertainty estimates* avail-
able in dropout neural networks. The first metric is meant to check how far an input sample is from the original manifold; the second one to identify if the input sample lies in low-confidence regions of the input space.

*Local Intrinsic Dimensionality (LID) [MLW+18]*. The approach aims to explore the subspace surrounding adversarial examples. The *local intrinsic dimensionality* directly measures the expansion rate of the local distance distribution from a reference point to its nearest neighbors. The idea is that the expansion of the adversarial subspace is higher than that of the normal data subspace and hence the metric can be used to detect whether the input sample is adversarial or not. Differently from the methods before, LID features of the input samples are extracted at each output layer of the network of the pre-trained classifier.

## Unsupervised methods

Methods falling under this category only rely on the features of natural samples that can be extracted using different techniques.

*Feature Squeezing (FS) [XEQ18]*. In this approach, the model's prediction on the original sample is compared to its prediction on the sample after *squeezing* (i.e., reducing the color depth of images, and using smoothing to reduce the variation among the pixels). If the original and squeezed inputs produce substantially different outputs, the input is likely to be adversarial.

*MagNet [MC17]*. As in [FCSG17], the initial assumption is that the adversarial examples lie off the true data manifold. A detector consists of an autoencoder that reconstructs input samples on the original manifold from input samples. The sample is classified as an adversary if the reconstruction error between the reconstructed input and the original exceeds a certain threshold. An additional autoencoder is used to improve performance, which calculates the Jensen's divergence between the original and reconstructed conditional distributions.
# Part I

## Misclassification Detection
<figure><img src="image_7.png" /></figure>
# DOCTOR: A Simple Method for Detecting Misclassification Errors

In this chapter we tackle the problem of identifying whether the prediction of a classifier should (or should not) be trusted.

From the theoretical point of view, we derive the trade-off between two types of error probabilities: Type-I, that refers to the rejection of the classification for an input that would be correctly classified, and Type-II, that refers to the acceptance of the classification for an input that would not be correctly classified (Proposition 1). The characterization of the optimal discriminator in Eq. (3.7) allows us to devise a feasible implementation of it, based on the softmax probability (Proposition 2).

From the algorithmic point of view, inspired by our theoretical analysis, we propose DOCTOR a new discriminator (Definition 2), which yields a simple and flexible framework to detect whether a decision made by a model is likely to be correct or not. We distinguish two scenarios under which DOCTOR can be deployed: Totally Black Box (TBB) where only the soft-predictions are available, hence gradient-propagation to perform input pre-processing is not allowed, and Partially Black Box (PBB) where we further allow method-specific inputs perturbations.

From the experimental point of view, we show that DOCTOR outperforms comparable state-of-the-art methods (e.g., ODIN [LLS18], softmax response [GE17] and Mahalanobis distance [LLLS18b]) on datasets including both in-distribution and out-of-distribution samples, and different architectures with various expressibilities, under both TBB and PBB. A key ingredient of DOCTOR is to fully
exploit all available information contained in the soft-probabilities of the predictions (not only their maximum).

## 3.1 The Optimal Discriminator

### 3.1.1 Statistical model for detection

Given a data sample $\mathbf{x} \in \mathcal{X}$ and an unobserved random label $y \in \mathcal{Y}$ drawn from the unknown distribution $p_{XY}$, we wish to predict the realization of the unobserved error variable $E \stackrel{\text{def}}{=} 1[Y \neq g_{\theta}(\mathbf{X})]$. To this end, we will model the data distribution as a mixture pdfs,

$$p_{XY}(\mathbf{x}, y) = p_E(1)p_{XY|E}(\mathbf{x}, y|1) + p_E(0)p_{XY|E}(\mathbf{x}, y|0),$$

where $p_{XY|E}(\mathbf{x}, y|1)$ denotes the pdf truncated to the error event $\{E = 1\}$ (i.e., the hard decision fails) and $p_{XY|E}(\mathbf{x}, y|0)$ is the pdf truncated to the success event $\{E = 0\}$ (i.e., the hard decision succeeds). By taking the marginal of $p_{XY}$ over the labels, we obtain: $p_X(\mathbf{x}) = p_E(1)p_{X|E}(\mathbf{x}|1) + p_E(0)p_{X|E}(\mathbf{x}|0)$. First, observe that the problem at hand is to infer $E$ from $(\mathbf{x}, p_{\hat{Y}|X})$ since $Y$ is not observed. Second, we further emphasize that in the present framework we assume that there are no available (extra) samples for training a discriminator to distinguish between $p_{X|E}(\mathbf{x}|0)$ and $p_{X|E}(\mathbf{x}|1)$. It is worth mentioning that a well-trained classifier would imply $p_E(1) \ll p_E(0)$, since in that case we should have very few classification errors. However, this also implies that it would be very unlikely to have enough samples available to train a good enough discriminator.

### 3.1.2 Performance metrics and optimal discriminator

We aim to distinguish between samples for which the predictions cannot be trusted and samples for which predictions should be trusted. We first state the optimal rejection region, given by Eq. (3.1), where we suppose the existence of an oracle who knows all the involved probability distributions.

**Definition 1** (Most powerful discriminator). For any $0 < \gamma < \infty$, define the de-
cision region:

$$ \mathcal{A}(\gamma) \stackrel{\text{def}}{=} \{ \mathbf{x} \in \mathcal{X} : p_{X|E}(\mathbf{x}|1) > \gamma \cdot p_{X|E}(\mathbf{x}|0) \}. \quad (3.1) $$

The most powerful (Oracle) discriminator at threshold $\gamma$ is defined by setting $D(\mathbf{x}, \gamma) = 1$ for all $\mathbf{x} \in \mathcal{A}(\gamma)$ for which the prediction is rejected (i.e., $\hat{E} = 1$) and otherwise $D(\mathbf{x}, \gamma) = 0$ for all $\mathbf{x} \notin \mathcal{A}(\gamma)$ for which the prediction is accepted.

In Proposition 1, we establish the characterization of the fundamental performance of the most powerful (Oracle) discriminator by providing a lower bound on the error achieved by any discriminator and show that this bound is achievable by setting $\gamma = 1$. Furthermore, we connect this result to the Bayesian error rate of this optimal discriminator.

**Proposition 1 (Performance of the discriminator).** For any given decision region $\mathcal{A} \subset \mathcal{X}$, let

$$ \epsilon_0(\mathcal{A}) \stackrel{\text{def}}{=} \int_{\mathcal{A}} p_{X|E}(\mathbf{x}|0)d\mathbf{x}, \quad \text{and} \quad \epsilon_1(\mathcal{A}^c) \stackrel{\text{def}}{=} \int_{\mathcal{A}^c} p_{X|E}(\mathbf{x}|1)d\mathbf{x}, \quad (3.2) $$

be the Type-I (rejection of the class prediction of an input $\mathbf{x}$ that would be correctly classified) and Type-II (acceptance of the class prediction of an input $\mathbf{x}$ that would not be correctly classified) error probability, respectively. Then,

$$ \epsilon_0(\mathcal{A}) + \epsilon_1(\mathcal{A}^c) \ge 1 - \|p_{X|E=1} - p_{X|E=0}\|_{\text{TV}} \quad (3.3) $$

$$ = 1 - \frac{1}{2} \int_{\mathcal{X}} |p_{X|E=1}(\mathbf{x}) - p_{X|E=0}(\mathbf{x})| d\mathbf{x}. \quad (3.4) $$

Equality is achieved by choosing the optimal decision region $\mathcal{A}^* \equiv \mathcal{A}(1)$ in Definition 1. If the hypotheses are equally distributed, the minimum Bayesian error satisfies:

$$ 2 \Pr\{D(\mathbf{X}) \neq E(\mathbf{X})\} \ge 1 - \|p_{X|E=1} - p_{X|E=0}\|_{\text{TV}}. \quad (3.5) $$

Equality is achieved by using the optimal decision region.

Expressions Eq. (3.4) and Eq. (3.5) provide lower bounds for the total error of an arbitrary discriminator. The proof of this proposition is relegated to the Supplementary material (Appendix A.1). Using Bayes we can rewrite Eq. (3.1)
via the posteriors as:

$$ \mathcal{A}(\gamma) = \{ \mathbf{x} \in \mathcal{X} : p_{E|X}(1|\mathbf{x})p_E(0) > \gamma \cdot (1 - p_{E|X}(1|\mathbf{x})) p_E(1) \}. \quad (3.6) $$

From Eq. (3.6), it is easy to check that $p_{E|X}(1|\mathbf{x}) = 1 - p_{Y|X}(g_{\theta}(\mathbf{x})|\mathbf{x}) = \text{Pe}(\mathbf{x})$, and hence, the decision region $\mathcal{A}(\gamma)$ can be reformulated as:

$$ \mathcal{A}(\gamma') = \left\{ \mathbf{x} \in \mathcal{X} : \frac{\text{Pe}(\mathbf{x})}{1 - \text{Pe}(\mathbf{x})} > \gamma' \right\} = \left\{ \mathbf{x} \in \mathcal{X} : \text{Pe}(\mathbf{x}) > \frac{\gamma'}{(\gamma' + 1)} \right\}, \quad (3.7) $$

where $\gamma' \stackrel{\text{def}}{=} \gamma \cdot \frac{p_E(1)}{p_E(0)}$ and $0 < \gamma' < \infty$. According to Eq. (3.7) and Proposition 1, the optimal discriminator is given by $D^*(\mathbf{x}, \gamma') = 1$, whenever $\mathbf{x} \in \mathcal{A}(\gamma')$, and $D^*(\mathbf{x}, \gamma') = 0$, otherwise. The main difficulty arises here since the error probability function of an input: $\mathbf{x} \mapsto \text{Pe}(\mathbf{x})$ is not known and in general cannot be learned from training samples.

## 3.2 The Proposed Discriminator: DOCTOR

### 3.2.1 DOCTOR discriminator

We start by deriving an approximation to the unknown function $\mathbf{x} \mapsto \text{Pe}(\mathbf{x})$ which can be used to devise the decision region in expression (3.7). First, we state the following:

**Proposition 2.** Let $\hat{g}(\mathbf{x})$ be defined by

$$ 1 - \hat{g}(\mathbf{x}) \stackrel{\text{def}}{=} \sum_{y \in \mathcal{Y}} p_{\hat{Y}|X}(y|\mathbf{x}; \theta) \Pr(\hat{Y} \neq y|\mathbf{x}) = 1 - \sum_{y \in \mathcal{Y}} p_{\hat{Y}|X}^2(y|\mathbf{x}; \theta), \quad (3.8) $$

for each $\mathbf{x} \in \mathcal{X}$, which indicates the probability of incorrectly classifying a feature $\mathbf{x}$ if it was randomly labeled according to the model distribution $p_{\hat{Y}|X}$ trained based on the dataset. Then,

$$ (1 - \sqrt{\hat{g}(\mathbf{x})}) - \Delta(\mathbf{x}) \leq \text{Pe}(\mathbf{x}) \leq (1 - \hat{g}(\mathbf{x})) + \Delta(\mathbf{x}), \quad (3.9) $$

where $\Delta(\mathbf{x}) \stackrel{\text{def}}{=} 2\sqrt{2 D_{\text{KL}}(p_{Y|X}(\cdot|\mathbf{x})\|p_{\hat{Y}|X}(\cdot|\mathbf{x}; \theta))}$ and denotes the Kullback-Leibler (KL) divergence (further details are provided in Supplementary material).
terial Appendix A.1.2).

### 3.2.2 Discussion

It is worth emphasizing that expressions in (3.9) provide bounds to the unknown function $\mathbf{x} \mapsto \text{Pe}(\mathbf{x})$ using a known statistics $\mathbf{x} \mapsto 1 - \hat{g}(\mathbf{x})$, which is based on the soft-probability of the predictor. On the other hand, $0 \le \hat{g}(\mathbf{x}) \le \sqrt{\hat{g}(\mathbf{x})} \le 1$, for all $\mathbf{x} \in \mathcal{X}$, which simply follows using the subadditive of the function $t \mapsto \sqrt{t}$ and the definition of $\hat{g}(\mathbf{x})$. By Markov's inequality,

$$ \Pr(\Delta(\mathbf{X}) \ge \varepsilon(\eta)) \le \eta \quad \text{with} \quad \varepsilon(\eta) = 2\sqrt{2\mathbb{E}_{\mathbf{X},Y}[\log p_{\hat{Y}|\mathbf{X}}(Y|\mathbf{X};\theta)]}/\eta, \quad (3.10) $$

for any $\eta > 0$, where $\mathbb{E}_{\mathbf{X},Y}[\log p_{\hat{Y}|\mathbf{X}}(Y|\mathbf{X};\theta)]$ in (3.10) is the cross-entropy risk. The latter is expected to be small provided that the model generalizes well. Thus, $\varepsilon(\eta)$ can be expected to be small for a desired confidence $\eta > 0$. Interestingly, (3.8) turns out to be related to the uncertainty of the classifier via the quadratic Rényi entropy [vEH14]: $-\log_2(\hat{g}(\mathbf{x})) = 2H_2(\hat{Y}|\mathbf{x}) \le 2H(\hat{Y}|\mathbf{x})$, where the latter is the Shannon entropy, i.e., the self-uncertainty of the classifier.

### 3.2.3 From the theory to a practical discriminator

Our previous discussion suggests that $\hat{\text{Pe}}(\mathbf{x})$ in (2.2) may be a valuable candidate to approximate $\text{Pe}(\mathbf{x})$ in the definition of the optimal discriminator (3.7). On the other hand, Proposition 2 suggests that $1 - \hat{g}(\mathbf{x})$ can also be a valuable candidate yielding another discriminator. These discriminators, referred to as DOCTOR, are introduced below.

**Definition 2 (DOCTOR).** For any $0 < \gamma < \infty$ and $\mathbf{x} \in \mathcal{X}$, define the following discriminators:

$$ D_{\alpha}(\mathbf{x}, \gamma) \stackrel{\text{def}}{=} 1[\hat{g}(\mathbf{x}) > \gamma \cdot \hat{g}(\mathbf{x})], \quad (3.11) $$

$$ D_{\beta}(\mathbf{x}, \gamma) \stackrel{\text{def}}{=} 1[\hat{\text{Pe}}(\mathbf{x}) > \gamma \cdot (1 - \hat{\text{Pe}}(\mathbf{x}))]. \quad (3.12) $$

Notice that because of Definition 2 and (3.8), $D_{\alpha}(\mathbf{x}, \gamma) = 1[1 - \sum_{y \in \mathcal{Y}} \text{softmax}^2(\mathbf{x})_y > \gamma \cdot \sum_{y \in \mathcal{Y}} \text{softmax}^2(\mathbf{x})_y]$; similarly because of Definition 2 and eq. (2.2), $D_{\beta}(\mathbf{x}, \gamma) = 1[1 - \max_{y \in \mathcal{Y}} \text{softmax}(\mathbf{x})_y > \gamma \cdot \max_{y \in \mathcal{Y}} \text{softmax}(\mathbf{x})_y]$.
The performance of these discriminators will be investigated and compared to state-of-the-art methods in the next section. In the Supplementary material (Appendix A.2), we illustrate how DOCTOR and the optimal discriminator (Definition 1) work on a synthetic data model that is a mixture of two spherical Gaussians with one component per class.

## 3.3 Evaluation

## 3.4 Experimental Results

In this section we present a collection of experimental results to investigate the effectiveness of DOCTOR, by applying it to several benchmark datasets. We provide publicly available code<fnref n="1" /> to reproduce our results, and we give further details on the environment, the parameter setting and the experimental setup in the Supplementary material (Appendix A.3). We propose a comparison with state-of-the-art methods using similar information. Though we are not concerned with the OOD detection problem, we are confident it is appropriate to compare DOCTOR to methods which use soft-probabilities or at most the output of the latent code, e.g., ODIN [LLS18], softmax response (SR) [GE17] and Mahalanobis distance (MHLNB) [LLLS18b]. Since we are focusing on misclassification detection, it is expected that OOD samples should be also detected as classification errors.

**Totally Black Box (TBB) and Partially Black Box (PBB).** We address two different scenarios with respect to the available information about the network. In the TBB only the output of the last layer of the network is available, hence gradient-propagation to perform input pre-processing is not allowed. In the PBB we allow method-specific inputs perturbations. When considering DOCTOR in PBB, for each testing sample $\mathbf{x}$, we calculate the pre-processed sample $\tilde{\mathbf{x}}$ by adding a small perturbation:

$$ \tilde{\mathbf{x}}^{\alpha} = \mathbf{x} - \epsilon \times \text{sign} \left[ -\nabla_{\mathbf{x}} \log \left( \frac{1 - \hat{g}(\mathbf{x})}{\hat{g}(\mathbf{x})} \right) \right], $$

$$ \tilde{\mathbf{x}}^{\beta} = \mathbf{x} - \epsilon \times \text{sign} \left[ -\nabla_{\mathbf{x}} \log \left( \frac{\hat{\mathbf{P}}\mathbf{e}(\mathbf{x})}{1 - \hat{\mathbf{P}}\mathbf{e}(\mathbf{x})} \right) \right]. $$

<footnote n="1">¹https://github.com/doctor-public-submission/DOCTOR/</footnote>
We will write directly $\tilde{\mathbf{x}}$ when it is clear from the context which input pre-processing we are referring to. In Supplementary material (Appendix A.3.2) we further analyze the equations above. When ODIN or MHLNB are used, we pre-process the inputs as in [LLS18] and in [LLLS18b], respectively.

### 3.4.1 Review of related methods

**PBB.** We compare DOCTOR (using input pre-processing and temperature scaling) with ODIN and MHLNB. ODIN [LLS18] comprises temperature scaling and input pre-processing via perturbation. Temperature scaling is applied to its scoring function, which has $f_i(\tilde{\mathbf{x}})$ for the logit of the i-th class. Formally, given an input sample $\mathbf{x}$:

$$ \begin{aligned} \text{SODIN}(\tilde{\mathbf{x}}) &= \max_{i=[1:C]} \frac{\exp(f_i(\tilde{\mathbf{x}})/T)}{\sum_{j=1}^{C} \exp(f_j(\tilde{\mathbf{x}})/T)}, \\ \text{ODIN}(\tilde{\mathbf{x}}; \delta, T, \epsilon) &= \begin{cases} \text{out,} & \text{if SODIN}(\tilde{\mathbf{x}}) \le \delta \\ \text{in,} & \text{if SODIN}(\tilde{\mathbf{x}}) > \delta, \end{cases} \end{aligned} $$

where $\tilde{\mathbf{x}}$ represents a magnitude $\epsilon$ perturbation of the original $\mathbf{x}$; $T$ is the temperature scaling parameter; $\delta \in [0, 1]$ is the threshold value; $in$ indicates the acceptance decision while out indicates the rejection decision. Notice, however, $\gamma$ in DOCTOR and $\delta$ in ODIN, respectively, are defined over two different domains: if $\delta$ denotes a probability, $\gamma$ is a ratio between probabilities. Although ODIN originally required tuning the hyper-parameter $T$ with out-of-distribution data, it was also shown that a large value for $T$ is generally desirable, suggesting that this gain is achieved at 1000. Anyway, in this framework, we notice an improvement of ODIN in performance for low values of $T$. Thus we report the best results obtained by ODIN considering the range of hyper-parameters values tested also for DOCTOR (cf. Section 3.4.3). ENERGY [LWOL20] comprises the denominator
of the softmax activation:

$$ 
\begin{aligned}
\mathrm{ES}(\mathbf{x}) &= -T \cdot \log \sum_{j=1}^{C} \exp(f_j(\mathbf{x})/T), \\
\mathrm{ENERGY}(\mathbf{x}; \xi, T) &= \begin{cases} \text{out, if } -\mathrm{ES}(\mathbf{x}) \le \xi \\ \text{in, if } -\mathrm{ES}(\mathbf{x}) > \xi \end{cases},
\end{aligned}
$$

where $\xi \in \mathbb{R}$ is the threshold value. Unlike all the methods considered in this paper, MHLNB [LLLS18b] requires the knowledge of the training set $\mathcal{D}_n$ which the pre-trained network was trained on to compute its empirical class mean $\hat{\mu}_c$ for each class c and its empirical covariance $\hat{\Sigma}$:

$$ \hat{\mu}_c = \frac{1}{n_c} \sum_{i: y_i=c} f(\tilde{\mathbf{x}}_i); \quad \hat{\Sigma} = \frac{1}{n} \sum_{c \in \mathcal{Y}} \sum_{\forall i: y_i=c} (f(\tilde{\mathbf{x}}_i) - \hat{\mu}_c)(f(\tilde{\mathbf{x}}_i) - \hat{\mu}_c)^{\top}, $$

where $n_c$ denotes the number of training samples with label c and $f(\tilde{\mathbf{x}})$ the log-its vector. As MHLNB directly uses the vector of logits, it does not comprise temperature scaling. Given an input sample $\mathbf{x}$:

$$ 
\begin{aligned}
M(\tilde{\mathbf{x}}) &= \max_{c \in \mathcal{Y}} - (f(\tilde{\mathbf{x}}) - \hat{\mu}_c)^{\top} \hat{\Sigma}^{-1} (f(\tilde{\mathbf{x}}) - \hat{\mu}_c), \\
\mathrm{MHLNB}(\tilde{\mathbf{x}}; \zeta, \epsilon) &= \begin{cases} \text{out, if } M(\tilde{\mathbf{x}}) > \zeta \\ \text{in, if } M(\tilde{\mathbf{x}}) \le \zeta \end{cases},
\end{aligned}
$$

as mentioned above, $\tilde{\mathbf{x}}$ represents a magnitude $\epsilon$ perturbation of the original $\mathbf{x}$; $\zeta \in \mathbb{R}_+$ is the threshold value; in indicates the acceptance decision while out indicates the rejection decision.

**TBB.** We compare DOCTOR (without input pre-processing and temperature scaling) with MHLNB (without input pre-processing and with the softmax output layer in place of the logits) and SR. Although both DOCTOR and SR have access to the softmax output of the predictor, a fundamental difference is that, while the former utilizes the softmax output in its entirety, the latter only uses the maximum value, therefore discarding potentially useful information. As it will be shown, this leads to better results for DOCTOR on several datasets (see Table 3.1). We emphasize that, by setting $T = 1$ and $\epsilon = 0$, ODIN reduces to softmax
response [GE17] since $SR(\mathbf{x}) \equiv SODIN(\mathbf{x})$.

### 3.4.2 Detection of misclassification errors, experimental setup and evaluation metrics

Before digging into the detailed discussion of our numerical results, we present an empirical analysis of the behavior of DOCTOR, ODIN, SR and MHLNB when faced with the task of choosing whether to accept or reject the prediction of a given classifier for a certain sample. In Figure 3.1, we propose a graphical interpretation of the discrimination performance, considering the labeled samples in the dataset TinyImageNet and the ResNet network as the classifier. We separate correctly and incorrectly classified samples according to their true labels in blue and in red, respectively. We remind that the label information is *not* necessary for the discriminators to define acceptance and rejection regions. Then, for each sample we compute the corresponding discriminators’ output. These values are binned and reported on the horizontal axis of Figure 3.1a and Figure 3.1b for $D_\alpha$, Figure 3.1c and Figure 3.1d for $D_\beta$, Figure 3.1e for SR, Figure 3.1f for ODIN, Figure 3.1g and Figure 3.1h for MHLNB. In each each plot, and according to the corresponding discriminator, the bins’ heights represent the frequency of the samples whose value falls within that bin. The intuition is that, if moving along the horizontal axis it is possible to pick a threshold value such that, w.r.t. this value, blue bars are on one side of the plot and red bars on the other, this threshold would correspond to the optimal discriminator, i.e. the discriminator that chooses the optimal acceptance and rejection regions.

In Figure 3.1g through Figure 3.1h, we observe that, for MHLNB, no matter how well we choose the threshold value, it is hard to fully separate red and blue bars both in TBB and PBB, i.e. the discriminator fails at defining acceptance and rejection regions so that all the hits can be assigned to the first one and all the mis-classification to the second one. The samples distribution for SR and ODIN in Figure 3.1e and Figure 3.1f, respectively, does not look significantly different from the one related to $D_\alpha$ and $D_\beta$ in TBB (Equation (3.11)). However, the discrimination between samples becomes evident in PBB. This is shown in Figure 3.1d for $D_\beta$ (eq. (3.11)) and even more in Figure 3.1b for $D_\alpha$ (eq. (3.11)) where, quite clearly, rightly classified samples are clustered on the left-end side of
the plot and incorrectly classified samples tend to cluster on the right-end side.
This intuition is supported by the results in Table 3.1.

<figure><img src="image_8.png" /><figcaption>**Figure 3.1: DOCTOR, ODIN, SR and MHLNB to split data samples in TinyImageNet both under TBB and PBB.** (a) - (b) show the results for expressions (2.2); (c) - (d) show the results for (3.8); (e) shows the results for SR; (f) shows the results for ODIN; (g) - (h) show the results for MHLNB. Histograms for wrongly classified samples (red) and correctly classified samples (blue).</figcaption></figure>

**Datasets and pre-trained networks.** We run experiments on both image and textual datasets. We use CIFAR10 and CIFAR100 [Kri09], TinyImageNet [JWX17] and SVHN [NWC+11] as image datasets; IMDb [MDP+11], AmazonFashion and AmazonSoftware [NLM19] as textual datasets. Note that, for all the aforementioned datasets, we consider only the test set since we rely on pre-trained models. Along the same lines of [LLS18], we use the pre-trained DenseNet models [HLW16] for CIFAR10, CIFAR100 and SVHN. In addition, we use a pre-trained ResNet model [HZRS16] for TinyImageNet, and BERT [DCLT19, WDS+20] for the Amazon datasets and IMDb. The accuracy achieved by the aforementioned networks on the test sets is showed in Table 3.1. According to the invariant properties of the discriminator (see Def. 2) with respect to the soft-probability of the underlying model, permutations of the posterior probabilities vector, due different initialization of the models before the training, do not change the output of Eq. (3.7), as it is a sum of squared values of the softmax probabilities. This variety of models/datasets characterizes the performance of the proposed method in scenarios with different accuracy levels.

**Evaluation metrics.** We will evaluate the performance according to Propo-
sition (1) via the empirical estimates of Type-I and Type-II errors in expressions (3.2). Throughout this section, when the model’s decision for a sample is correct (hit) but is rejected by the discriminator, we refer to such event as *false rejection*; when the model’s decision for a sample is not correct (miss) and is rejected by the discriminator, we refer to such event as *true rejection*. Similarly, we refer to a *false acceptance* when a miss is not rejected and to a *true acceptance* when a hit is not rejected. More specifically, let $\mathcal{T}_m = \{(\mathbf{x}_1, y_1), \dots, (\mathbf{x}_m, y_m)\} \sim p_{XY}$ be the testing set, where $\mathbf{x}_i \in \mathcal{X}$ is the input sample, $y_i \in \{1, \dots, C\}$ is the true class of $\mathbf{x}_i$, and $m$ denotes the size of the testing set. With $j \in \{\alpha, \beta\}$:

$$ \mathcal{FR}_j(\gamma) = \{(\mathbf{x}, y) \in \mathcal{T}_m : y = g_{\theta}(\mathbf{x}), D_j(\mathbf{x}, \gamma) = 1\}, \quad (3.13) $$

$$ \mathcal{TR}_j(\gamma) = \{(\mathbf{x}, y) \in \mathcal{T}_m : y \neq g_{\theta}(\mathbf{x}), D_j(\mathbf{x}, \gamma) = 1\}, \quad (3.14) $$

$$ \mathcal{FA}_j(\gamma) = \{(\mathbf{x}, y) \in \mathcal{T}_m : y = g_{\theta}(\mathbf{x}), D_j(\mathbf{x}, \gamma) = 0\}, \quad (3.15) $$

$$ \mathcal{TA}_j(\gamma) = \{(\mathbf{x}, y) \in \mathcal{T}_m : y = g_{\theta}(\mathbf{x}), D_j(\mathbf{x}, \gamma) = 0\}. \quad (3.16) $$

We measure the performance of the test in terms of:

* **FRR** versus **TRR**. The false rejection rate (FRR) represents the probability that a hit is rejected, while the true rejection rate (TRR) is the probability that a miss is rejected.

* **AUROC**. The area under the Receiver Operating Characteristic curve (ROC) [DG06a] depicts the relationship between TRR and FRR. The perfect detector corresponds to a score of 100%.

* **FRR at 95 % TRR**. This is the probability that a hit is rejected when the TRR is at 95 %.

### 3.4.3 Experimental results: comparison between different discriminators

**DOCTOR:** comparison between $D_\alpha$ and $D_\beta$. We compare the discriminators $D_\alpha$ and $D_\beta$ introduced in (3.11) to show how the AUROCs for CIFAR10, CIFAR100, TinyImageNet and SVHN change when varying the parameters $T$ and $\epsilon$. It is observed that $D_\alpha$ is less sensitive to the selection of $T$: for all the datasets, $D_\alpha$ outperforms $D_\beta$ achieving the best AUROCs by setting $T = 1$. Contrary to
$D_\alpha$, $D_\beta$ is more sensitive to the value selected for $T$ in the sense that small changes may result in very different values for the measured AUROCs (cf. Appendix A.3.4). In contrast, the best results are obtained for the same epsilon values of $D_\alpha$ and $D_\beta$ across all the datasets.

**Comparison in TBB.** We compare DOCTOR with MHLNB (without input pre-processing and with the softmax output in place of the logits) and SR. It is worth to emphasize that $D_\alpha$ does not coincide (in general) with SR since the former consists in the sum of squared values of all probabilities involved in the softmax. To complete the comparison, we include the results for both methods in Table 3.1.

<figure><img src="image_9.png" /><figcaption>**Figure 3.2: ROC curves.** Comparison between $D_\alpha$ ($T_\alpha = 1$ and $\epsilon_\alpha = 0.00035$), $D_\beta$ ($T_\beta = 1.5$ and $\epsilon_\alpha = 0.00035$), ODIN ($T_{ODIN} = 1.3$ and $\epsilon_{ODIN} = 0$), MHLNB ($T_{MHLNB} = 1$ and $\epsilon_{MHLNB} = 0.0002$) and ENERGY ($T_{ENERGY} = 1$ and $\epsilon_{ENERGY} = 0$). Red dashed lines mark the 95% threshold of TRR.</figcaption></figure>

**Comparison in PBB.** We compare DOCTOR with ODIN, MHLNB and ENERGY. We keep the same parameter setting for all the methods. In the case of DOCTOR and ODIN where temperature scaling is allowed, we test, for each dataset, 24 different values of $\epsilon$ for each of the 11 different values of $T$, see (Appendix A.3.4) for the set of ranges. In the case of MHLNB, which directly uses the logits, we keep $T = 1$ and we vary $\epsilon$ for each dataset. In the case of ENERGY, where no perturbation is allowed, we keep $\epsilon = 0$ and we maintain $T = 1$ (as in [LWOL20]). According to our framework, no validation samples are available; consequently, in order to be consistent across the datasets, we only report the experimental settings and values for which, on average, we obtain favorable results for all the considered domains (cf. Figure 3.2). In order to be fair, we update ODIN's parameters from those in [LLS18] to new values which are more suitable to the task at hand (cf. plots in Appendix A.3.4).

DOCTOR's performance compared to ODIN's, MHLNB's and ENERGY's,
are collected in Table 3.1 and in Figure 3.2. The results in the table show that noise further improves the performance of DOCTOR (cf. PBB) up to 1% over our previous experiments without noise (cf. TBB) in terms of AUROC. The improvement is even more significant in terms of FRR at 95% TRR: *a 4% decrease is obtained in terms of predictions incorrectly rejected for DOCTOR when passing from TBB to PBB*. Note that only the softmax output is available when we consider the pre-trained models for AmazonFashion, AmazonSoftware and IMDb datasets; therefore, we cannot access any internal layer and test DOCTOR for values of *T* which differ from the default value *T* = 1. Consequently, temperature scaling and input pre-processing cannot be applied in these cases and thus these datasets cannot be tested in PBB. Moreover, even in TBB, these datasets cannot be tested through MHLNB and ENERGY since the dataset on which the network was trained is not available. We provide simulations on how the range of interval for the different thresholds can affect the results in Appendix A.3.3.

**Misclassification detection in presence of OOD samples.** We evaluate DOCTOR's performance in misclassification detection considering a mixture of both in (DATASET-IN) and out-of-distribution (OOD) samples (DATASET-OUT), i.e. input samples for which the decision should not be trusted. The results are compared with ODIN. We test the two methods when one sample to reject out of five (♣), three (♦) or two (♠) is OOD. The details of the simulations, the considered dataset, and the complete experimental results are relegated Appendix A.3.4. In Table 3.2 we report an extract of the results for the PBB scenario in terms of *mean / standard deviation*: DOCTOR achieves, and most of the time outperforms ODIN's performance. We emphasize that, even though DOCTOR is not tuned for the OOD detection problem, it represents the best choice for deciding whether to accept or reject the prediction of the classifier also on mixed data scenarios where the percentage of OOD samples, as long as it is not dominant, can sensitively vary.
<table><caption>**Table 3.1:** Overall results for misclassification detection. For all methods, in TBB, we set $T = 1$ and $\epsilon = 0$; in PBB we set: $\epsilon_{\alpha} = \epsilon_{\beta} = 0.00035$, $T_{\alpha} = 1$, $T_{\beta} = 1.5$, $\epsilon_{ODIN} = 0$ and $T_{ODIN} = 1.3$, $\epsilon_{MHLNB} = 0.0002$ and $T_{MHLNB} = 1$, $\epsilon_{ENERGY} = 0$ and $T_{ENERGY} = 1$. In TBB, ODIN and SR coincide ($T = 1$ and $\epsilon = 0$).</caption><thead><tr><th rowspan="2">DATASET</th><th rowspan="2">METHOD</th><th colspan="2">AUROC↑%</th><th colspan="2">FRR↓95%%</th><th rowspan="2">DATASET</th><th rowspan="2">METHOD</th><th colspan="2">AUROC↑%</th><th colspan="2">FRR↓95%%</th></tr><tr><th>TBB</th><th>PBB</th><th>TBB</th><th>PBB</th><th>TBB</th><th>PBB</th><th>TBB</th><th>PBB</th></tr></thead><tbody><tr><td rowspan="6">CIFAR10<br>Acc. 95%</td><td>D<sub>α</sub></td><td><b>94</b></td><td><b>95.2</b></td><td><b>17.9</b></td><td>13.9</td><td rowspan="6">SVHN<br>Acc. 96%</td><td>D<sub>α</sub></td><td><b>92.3</b></td><td><b>93</b></td><td><b>38.6</b></td><td><b>36.6</b></td></tr><tr><td>D<sub>β</sub></td><td>68.5</td><td>94.8</td><td>18.6</td><td><b>13.4</b></td><td>D<sub>β</sub></td><td>92.2</td><td>92.8</td><td>39.7</td><td>38.4</td></tr><tr><td>ODIN</td><td>93.8</td><td>94.2</td><td>18.2</td><td>18.4</td><td>ODIN</td><td>92.3</td><td>92.3</td><td>38.6</td><td>40.7</td></tr><tr><td>SR</td><td>93.8</td><td>-</td><td>18.2</td><td>-</td><td>SR</td><td><b>92.3</b></td><td>-</td><td><b>38.6</b></td><td>-</td></tr><tr><td>MHLNB</td><td>92.2</td><td>84.4</td><td>30.8</td><td>44.6</td><td>MHLNB</td><td>87.3</td><td>88</td><td>85.8</td><td>54.7</td></tr><tr><td>ENERGY</td><td>-</td><td>91.1</td><td>-</td><td>34.7</td><td>ENERGY</td><td>-</td><td>88.9</td><td>-</td><td>49.4</td></tr><tr><td rowspan="6">CIFAR100<br>Acc. 78%</td><td>D<sub>α</sub></td><td><b>87</b></td><td><b>88.2</b></td><td>40.6</td><td><b>35.7</b></td><td rowspan="6">AMAZON<br>FASHION<br>Acc. 85%</td><td>D<sub>α</sub></td><td><b>89.7</b></td><td>-</td><td><b>27.1</b></td><td>-</td></tr><tr><td>D<sub>β</sub></td><td>84.2</td><td>87.4</td><td>40.6</td><td>36.7</td><td>D<sub>β</sub></td><td><b>89.7</b></td><td>-</td><td><b>26.3</b></td><td>-</td></tr><tr><td>ODIN</td><td>86.9</td><td>87.1</td><td>40.5</td><td>40.7</td><td>SR</td><td>87.4</td><td>-</td><td>50.1</td><td>-</td></tr><tr><td>SR</td><td>86.9</td><td>-</td><td><b>40.5</b></td><td>-</td><td>SR</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>MHLNB</td><td>82.6</td><td>50</td><td>66.7</td><td>94</td><td>AMAZON<br>SOFTWARE<br>Acc. 73%</td><td>D<sub>α</sub></td><td><b>68.8</b></td><td>-</td><td><b>73.2</b></td><td>-</td></tr><tr><td>ENERGY</td><td>-</td><td>78.7</td><td>-</td><td>65.4</td><td>AMAZON<br>SOFTWARE<br>Acc. 73%</td><td>D<sub>β</sub></td><td><b>68.8</b></td><td>-</td><td><b>73.2</b></td><td>-</td></tr><tr><td rowspan="6">TINY<br>IMAGENET<br>Acc. 63%</td><td>D<sub>α</sub></td><td><b>84.9</b></td><td><b>86.1</b></td><td><b>45.8</b></td><td><b>43.3</b></td><td rowspan="6">IMDB<br>Acc. 90%</td><td>D<sub>α</sub></td><td><b>84.4</b></td><td>-</td><td><b>54.2</b></td><td>-</td></tr><tr><td>D<sub>β</sub></td><td><b>84.9</b></td><td>85.3</td><td><b>45.8</b></td><td>45.1</td><td>D<sub>β</sub></td><td><b>84.4</b></td><td>-</td><td>54.4</td><td>-</td></tr><tr><td>ODIN</td><td>84.9</td><td>84.9</td><td>45.8</td><td>45.3</td><td>SR</td><td>83.7</td><td>-</td><td>61.7</td><td>-</td></tr><tr><td>SR</td><td><b>84.9</b></td><td>-</td><td><b>45.8</b></td><td>-</td><td>SR</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>MHLNB</td><td>78.4</td><td>59</td><td>82.3</td><td>86</td><td>IMDB<br>Acc. 90%</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ENERGY</td><td>-</td><td>78.2</td><td>-</td><td>63.7</td><td>ENERGY</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>

<table><caption>**Table 3.2:** Misclassification detection in presence of OOD samples. Same parameter setting as in Table 3.1 (PBB) for $D_{\alpha}$, $D_{\beta}$, ODIN, ENERGY; as in [LLS18] for ODINOOD and as in [LLLS18b] for MHLNBWB. Results presented in terms of mean / standard deviation.</caption><thead><tr><th rowspan="2">DATASET-IN</th><th rowspan="2">DATASET-OUT</th><th colspan="7">AUROC↑%</th><th colspan="7">FRR↓95%%</th></tr><tr><th>D<sub>α</sub></th><th>D<sub>β</sub></th><th>ODIN</th><th>ODINOOD</th><th>ENERGY</th><th>MHLNBWB</th><th>D<sub>α</sub></th><th>D<sub>β</sub></th><th>ODIN</th><th>ODINOOD</th><th>ENERGY</th><th>MHLNBWB</th></tr></thead><tbody><tr><td rowspan="2">CIFAR10<br>♠</td><td>iSUN</td><td><b>95.4</b> / 0.1</td><td>95.1 / 0.1</td><td>94.6 / 0.1</td><td>89.6 / 0</td><td>92.4 / 0.1</td><td>54.5 / 0.1</td><td>14 / 0.5</td><td><b>13.5</b> / 0.4</td><td>17.2 / 0.3</td><td>38.9 / 0</td><td>32.2 / 0.1</td><td>92 / 0.1</td></tr><tr><td>TINY (RES)</td><td><b>95.2</b> / 0.1</td><td>94.9 / 0</td><td>94.6 / 0.1</td><td>89.6 / 0</td><td>92.3 / 0.1</td><td>56.2 / 0</td><td><b>14</b> / 0.4</td><td><b>14</b> / 0.5</td><td>17.8 / 0.4</td><td>38.9 / 0</td><td>32.2 / 0.1</td><td>90.3 / 0.2</td></tr><tr><td rowspan="2">CIFAR10<br>◇</td><td>iSUN</td><td><b>95.5</b> / 0.1</td><td>95.3 / 0.1</td><td>94.9 / 0.1</td><td>91.5 / 0</td><td>92.9 / 0</td><td>54.5 / 0.1</td><td>14.4 / 0.6</td><td><b>13.4</b> / 0.2</td><td>16.8 / 0.5</td><td>34 / 0.1</td><td>27 / 1</td><td>92 / 0.2</td></tr><tr><td>TINY (RES)</td><td><b>95.4</b> / 0.1</td><td>95 / 0.1</td><td>94.8 / 0.1</td><td>91.4 / 0</td><td>92.8 / 0</td><td>56.2 / 0.1</td><td>15 / 0.1</td><td><b>14.8</b> / 0.7</td><td>17 / 0.5</td><td>34.5 / 0.9</td><td>28.8 / 1.9</td><td>90 / 0.3</td></tr><tr><td rowspan="2">CIFAR10<br>♠</td><td>iSUN</td><td><b>95.6</b> / 0.1</td><td><b>95.6</b> / 0</td><td>95.4 / 0</td><td>93.5 / 0</td><td>93.6 / 0.1</td><td>54.6 / 0</td><td>15.1 / 0.1</td><td><b>13.6</b> / 0.5</td><td>16.1 / 0.2</td><td>30.6 / 0.4</td><td>25.1 / 0.2</td><td>92 / 0.2</td></tr><tr><td>TINY (RES)</td><td><b>95.5</b> / 0.1</td><td>95.2 / 0.1</td><td>95.1 / 0.1</td><td>93.2</td><td>93.5 / 0</td><td>56.2 / 0.2</td><td><b>14.7</b> / 0.3</td><td>14.8 / 0.5</td><td>17.1 / 0.4</td><td>31 / 0</td><td>25.6 / 0.3</td><td>90.2 / 0.1</td></tr></tbody></table>

## 3.5 Final remarks

### Related work

Recent works have shown that the accuracy of a classifier and its ability to output soft-predictions that represent the estimate of the true posterior can be totally disjointed [GPSW17, KE16, KL15]. Furthermore, models often tend to be over-
confident about their decision even when their predictions fail [HAB19, KHH20]. This motivates a novel research area that strives to develop methods to assess when decisions made by classifiers should or should not be trusted. Although the detection of OOD samples is a different (domain) problem, it is naturally expected that samples from a distribution that is significantly different from the training one cannot be correctly classified. In [LLS18], the authors propose a method that increases the peakiness of the softmax output by perturbing the input samples and applying temperature scaling [GPSW17, HVD15, Pla00] to the classifier logits to detect in-distribution samples better. It is worth noticing that this method requires additional information on the internal structure of the latent code of the model. A very different approach [HSJK20, LLLS18b] tackles OOD detection by using the *Mahalanobis distance*. Although this approach appears to be more powerful, it also requires additional samples to learn the mean by class and the covariance matrix of the in-distribution. In [DT18], classifiers are trained to output calibrated confidence estimates that are used to perform OOD detection. A related line of research is concerned with the problem of *selective predictions* (aka *reject options*) in deep neural networks. The main motivation for selective prediction is reducing the error rate by abstaining from prediction when in doubt while keeping the number of correctly classified samples as high as possible [GKS21, GE17, GE19]. The idea is to combine classifiers with *rejection functions* by observing the classifiers' output, without supervision, to decide whether to accept or reject the classification outcome. In [GE17], the authors introduce *softmax response*, a rejection function which compares the maximum soft-probability to a pre-determined threshold to decide whether to accept or reject the class prediction given by the model.

Summary

We introduced a simple and effective method to detect misclassification errors, i.e., whether a classifier's prediction should or should not be trusted. We provided theoretical results on the optimal statistical model for misclassification detection, and we presented our empirical discriminator DOCTOR. Experiments on real (textual and visual) datasets—including OOD samples and comparison to state-of-the-art methods—demonstrate the effectiveness of our proposed meth-
ods. Whilst methods for ODD frameworks do not necessarily perform well in
predicting misclassification errors, our result advances the state-of-the-art, and
the main takeaway is that DOCTOR can be applied to both partially black-box
(PBB) setups and totally black-box (TBB) ones. In the latter, information about
the model's architecture may be undisclosed for security reason when dealing with
sensitive data). DOCTOR uses all the information in the softmax output, which
results in equal or better performance with respect to the other methods: the
results in PBB, where we observe a reduction up to 4% in terms of predictions
incorrectly rejected with respect to the ones in TBB are particularly promising.
Moreover, DOCTOR does not require training data and, thanks to its flexibility,
it can be easily deployed in real-world scenarios. Currently, DOCTOR does not
exploit information across the layers yet. Only the soft-predictions are used. Be-
sides, the most important obstacle is the calibration of the threshold (γ) between
the desired fault rejection and acceptance rates, which would require additional
validation samples. However, quite often, the cost of collecting data for this op-
eration can be prohibitive, making it difficult or too expensive to perform such
calibration. As future work, we shall combine DOCTOR with other related lines of
research such as the extension to white-box incorporating additional information
across the different latent codes of the model. Moreover, we shall investigate the
possibility of combining the two proposed discriminators.
# Part II

## Multi-Armed Adversarial Attack Detection
<figure><img src="image_10.png" /></figure>
# MEAD: A Multi-Armed Approach for Evaluation of Adversarial Examples Detectors

In this chapter we tackle the problem of adversarial attacks detection in a multi-armed framework, i.e. when the attacks at the evaluation time can be simultaneously crafted according to a variety of algorithms and objective loss functions.

We propose MEAD, a novel multi-armed evaluation framework for adversarial examples detectors involving several attackers to ensure that the detector is not overfitted to a particular attack strategy. The proposed metric is based on the following criterion. Each adversarial sample is correctly detected if and only if all the possible attacks on it are successfully detected. We show that this approach is less biased and yields a more effective metric than the one obtained by assuming only a single attack at evaluation time (cf. Section 4.3).

We make use of three new objective functions which, to the best of our knowledge, have never been used for the purpose of generating adversarial examples at testing time. These are *KL divergence*, *Gini Impurity* and *Fisher-Rao distance*. Moreover, we argue that each of them contributes to jointly creating competitive attacks that cannot be created by a single function (cf. Section 4.1).

We perform an extensive numerical evaluation of state-of-the-art (SOTA) and uncover their limitations, suggesting new research perspectives in this research line (cf. Section 4.4).
## 4.1 Generating adversarial examples according to different objectives

We recall the adversarial problem introduced in Section 2.2.1. Let us consider a natural sample $\mathbf{x} \in \mathcal{X}$ together with its true label $y \in \mathcal{Y}$. An attacker targets the model $g_{\theta}$ by crafting a sample $\mathbf{x}'_{\ell} \in \mathbb{R}^d$ according to an objective loss function $\ell(\mathbf{x}, \mathbf{x}'_{\ell}; \theta)$ which is denoted by $\ell$, perturbation magnitude $\varepsilon$, and norm constraint $L_p, p \in \{1, 2, \infty\}$. The goal of the attack is to obtain an $\mathbf{x}'_{\ell}$ such that $g_{\theta}(\mathbf{x}'_{\ell}) \neq g_{\theta}(\mathbf{x})$, in order to force the target model to make a prediction error. The adversarial generation problem is commonly relaxed as follows

$$ \mathbf{x}'_{\ell} = \underset{\mathbf{x}'_{\ell} \in \mathbb{R}^d : \|\mathbf{x}'_{\ell} - \mathbf{x}\|_p < \varepsilon}{\operatorname{argmax}} \ell(\mathbf{x}, \mathbf{x}'_{\ell}; \theta), \quad (4.1) $$

where $\mathbf{x}'_{\ell}$ is updated iteration by iteration starting from an initial given value. The objective function $\ell$ traditionally used is the Adversarial Cross-Entropy (ACE) [SZS+14, MMS+18]:

$$ \ell_{\text{ACE}}(\mathbf{x}, \mathbf{x}'_{\ell}; \theta) = \mathbb{E}_{Y|\mathbf{x}}[-\log p_{\hat{Y}|X}(Y|\mathbf{x}'_{\ell}; \theta)], \quad (4.2) $$

where the expectation is understood to be over the ground true conditional distribution of $Y$ given $\mathbf{x}$. Inspired by recent development in the fields of robustness and misclassification detection [GRG+21, PMB+22, ZYJ+19], we include in our study recently proposed objective functions which generate diversified adversarial examples and that we briefly recall below.

* The Kullback-Leibler divergence (KL):

$$ \ell_{\text{KL}}(\mathbf{x}, \mathbf{x}'_{\ell}; \theta) = \mathbb{E}_{\hat{Y}|\mathbf{x}; \theta} \left[ \log \left( \frac{p_{\hat{Y}|X}(\hat{Y}|\mathbf{x}; \theta)}{p_{\hat{Y}|X}(\hat{Y}|\mathbf{x}'_{\ell}; \theta)} \right) \right]. \quad (4.3) $$

* The Fisher-Rao objective (FR) [PMB+22]:

$$ \ell_{\text{FR}}(\mathbf{x}, \mathbf{x}'_{\ell}; \theta) = 2 \arccos \left( \sum_{y \in \mathcal{Y}} \sqrt{p_{\hat{Y}|X}(y|\mathbf{x}; \theta) p_{\hat{Y}|X}(y|\mathbf{x}'_{\ell}; \theta)} \right). \quad (4.4) $$
<figure><img src="image_11.png" /><figcaption>**Figure 4.1:** ACE vs. Gini Impurity. Decision boundary for the binary classifier 4.1a-4.1c: the decision region for class 1 is green, the decision region of class 0 is pink. The natural testing samples belonging to class 0 are reported in blue, the corresponding adversarial examples crafted using ACE (4.1a) and Gini Impurity (4.1c) in red. Decision boundary of the detectors 4.1b-4.1d: $\mathcal{B}$, the decision region of the natural examples; $\mathcal{A}_\ell$, reported in red shades, the decision region of the adversarial examples when the detector is trained on data points crafted via $\ell \in \{\text{ACE, Gini}\}$ as objective. The darker shades stand for higher confidence. The red points represent the adversarial examples created with the opposite loss (respectively $\ell \in \{\text{Gini, ACE}\}$).</figcaption></figure>

* The Gini Impurity score (Gini) [GRG$^{+21}$]:

$$ \ell_{\text{Gini}}(\cdot, \mathbf{x}_{\ell}'; \theta) = 1 - \sqrt{\sum_{y \in \mathcal{Y}} p_{\tilde{Y}|X}^2(y|\mathbf{x}_{\ell}'; \theta)}. \quad (4.5) $$

Interestingly, $\text{Gini}(\mathbf{x})$ corresponds to the function $1 - \sqrt{\hat{g}(\mathbf{x})}$ presented in Chapter 3.

## 4.2 A case study: ACE vs. Gini Impurity

In Figure 4.1 we provide insights on why we need to evaluate the detectors on attacks crafted through different objectives. We create a synthetic dataset that consists of 300 data points drawn from $\mathcal{N}_0 = \mathcal{N}(\mu_0, \sigma^2\mathbf{I})$ and 300 data points
drawn from $N_1 = N(\mu_1, \sigma^2 I)$, where $\mu_0 = [1 \ 1]$, $\mu_1 = [-1 \ -1]$ and $\sigma = 1$. To each data point $\mathbf{x}$ is assigned true label 0 or 1 depending on whether $\mathbf{x} \sim N_0$ or $\mathbf{x} \sim N_1$, respectively. The data points have been split between the training set (70%) and the testing set (30%). We finally train a simple binary classifier with one single hidden layer and a learning rate of 0.01 for 20 epochs. We attack the classifier by generating adversarial examples with PGD under the $L_\infty$-norm constraints with $\epsilon = 1.2$ for the ACE attacks and $\epsilon = 5$ for the Gini Impurity attacks to have a classification accuracy (classifier performance) of 50% on the corrupted data points. In Figure 4.1a-4.1c we plot the decision boundary of the binary classifier together with the adversarial and natural examples belonging to class 0. As can be seen, ACE creates points that lie in the opposite decision region with respect to the original points (Fig. 4.1a). Conversely, Gini Impurity tends to create new data points in the region of maximal uncertainty of the classifier (Fig. 4.1c). Consider the scenario where we train a simple Radial Basis Function (RBF) kernel SVM on a subset of the testing set of the natural points together with the attacked examples, generated with the ACE or the Gini Impurity score depending on the case (Fig. 4.1b-4.1d). We then test the detector on the data points originating with the opposite loss, Figure 4.1b and Figure 4.1d respectively. The decision region of the detector for natural examples is in blue, and the one for the adversarial examples is in red. The intensity of the color corresponds to the level of certainty of the detector. The detector's accuracy on natural and adversarial data points decreases from 71% to 62% when changing to the opposite loss in Fig. 4.1b, and from 87% to 63% in Fig. 4.1d. Hence, testing on samples crafted using a different loss in Eq. (2.4) means changing the attack and, consequently, evaluating detectors without considering this possibility leads to a more biased and unrealistic estimation of their performance. When the detector is trained on the adversarial examples created with both the losses, the accuracy is 79.8% when testing on Gini and 66.3% when testing on ACE, which is a better trade-off in adversarial detection performances.

The aforementioned losses will be included in the following section to design MEAD, our *multi-armed evaluation framework*, a new method to evaluate the performance of adversarial detection with low bias.
## 4.3 Evaluation with a Multi-Armed Attacker

<figure><img src="image_12.png" /><figcaption>**Figure 4.2: MEAD workflow:** x is the natural example, $\varepsilon = 5$ is the perturbation magnitude, $L_1$ is the norm. From the set of all the possible existing attacks $\mathcal{A}$ we consider the ones using PGD. The sifter discards all the perturbed samples that do not fool the classifier $g_\theta$. d is the detector.</figcaption></figure>

The proposed evaluation framework, MEAD, consists in testing an adversarial examples detection method on a large collection of attacks grouped w.r.t. the $L_p$-norm and the maximal perturbation $\varepsilon$ they consider. Each given natural input example is perturbed according to the collection of attacks. Note that, for every attack, a perturbed example is considered adversarial *if and only if* it fools the classifier. Otherwise, it is discarded and will not influence the evaluation. We then feed all the natural and successful adversarial examples to the detector and gather all the predictions. Finally, based on the detection decisions, we evaluate the detector according to a worst-case scenario:

i) **Adversarial decision:** for each natural example, we gather all the successful adversarial examples. If the detector detects *all* of them, then the perturbed sample is considered *correctly detected* (i.e., it is a true positive). However, if the detector misses at least one of them, the noisy sample is considered *undetected* (i.e., it is a false negative).

ii) **Natural decision:** for each natural sample, if the detector does not detect it, then the sample is considered *correctly non-detected* (i.e., it is a true negative); otherwise it is *incorrectly detected* (i.e., it is a false positive).

Specifically, let $\mathcal{D}_m = \{(\mathbf{x}_i, y_i)\}_{i=1}^m \sim p_{XY}$ be the testing set of size $m$, where $\mathbf{x}_i \in \mathcal{X}$ is the natural input sample and $y_i \in \mathcal{Y}$ is its true label. Let $d: \mathcal{X} \times \mathbb{R} \to$
$\{0, 1\}$ be the detection mechanism and $a_{\ell}: \mathcal{X} \times \mathbb{R} \times \{1, 2, \infty\} \to \mathcal{X}$ the attack strategy according to the objective function $\ell \in \mathcal{L}$ within a selected collection of objectives $\mathcal{L}$ as described in Section 4.1. For every considered $L_p$-norm, $p \in \{1, 2, \infty\}$, maximal perturbation $\varepsilon \in \mathbb{R}$, and every threshold $\gamma \in \mathbb{R}^1$:

$$Tp_{\varepsilon,p}(\gamma) = \left\{ (\mathbf{x}, y) \in \mathcal{D}_m : \forall \ell \in \mathcal{L} \{g_{\theta}(a_{\ell}(\mathbf{x})) \neq y\} \wedge \{d(a_{\ell}(\mathbf{x}), \gamma) = 1\} \right\} \quad (4.6)$$

$$FN_{\varepsilon,p}(\gamma) = \left\{ (\mathbf{x}, y) \in \mathcal{D}_m : \exists \ell \in \mathcal{L} \{g_{\theta}(a_{\ell}(\mathbf{x})) \neq y\} \wedge \{d(a_{\ell}(\mathbf{x}), \gamma) = 0\} \right\} \quad (4.7)$$

$$TN_{\varepsilon,p}(\gamma) = \{(\mathbf{x}, y) \in \mathcal{D}_m : d(\mathbf{x}, \gamma) = 0\} \quad (4.8)$$

$$Fp_{\varepsilon,p}(\gamma) = \{(\mathbf{x}, y) \in \mathcal{D}_m : d(\mathbf{x}, \gamma) = 1\}. \quad (4.9)$$

In Fig. 4.2 we provide a graphical interpretation of MEAD when the perturbation magnitude and the norm are fixed.

## 4.4 Experiments

In this section, we assess the effectiveness of the proposed evaluation framework, MEAD. The code is available at https://github.com/meadsubmission/MEAD.

### 4.4.1 Experimental setting

#### Evaluation metrics.

For each $L_p$-norm and each considered $\varepsilon$, we apply our multi-armed detection scheme. We gather the global result considering all the attacks and all the objectives. Moreover, we also report the results per objective. The performance is measured in terms of the AUROC↑% [DG06b] and in terms of FPR↓$_{95\%}$. The first metric is the Area Under the Receiver Operating Characteristic curve and represents the ability of the detector to discriminate between adversarial and natural examples (higher is better). The second metric represents the percentage of natural examples detected as adversarial when 95 % of the adversarial examples are detected, i.e., FPR at 95 % TPR (lower is better).

<footnote n="1">¹With an abuse of notation, $\forall \ell \in \mathcal{L}$ stands for all the considered attack mechanisms for specific values of $\varepsilon, p$ within a collection of objectives $\mathcal{L}$.</footnote>
<table><caption>Table 4.1: Average performance on CIFAR10 of all the detectors per objective and in MEAD. The worst results among all the settings is in bold; the ones in the single-armed setting is underlined. No norm denotes the group of attacks that do not depend on the norm constraint.</caption><thead><tr><th>CIFAR10</th><th colspan="2">MEAD</th><th colspan="2">ACE</th><th colspan="2">KL</th><th colspan="2">Gini</th><th colspan="2">FR</th></tr></thead><tbody><tr><th>NSS</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th></tr><tr><td>L<sub>1</sub> Average</td><td><strong>62.9</strong></td><td><strong>81.6</strong></td><td>67.4</td><td>75.7</td><td><strong>67.1</strong></td><td>76.0</td><td><strong>67.8</strong></td><td><strong>78.2</strong></td><td>67.6</td><td>75.6</td></tr><tr><td>L<sub>2</sub> Average</td><td><strong>64.0</strong></td><td><strong>82.0</strong></td><td>68.7</td><td>71.0</td><td><strong>68.5</strong></td><td>70.9</td><td><strong>65.1</strong></td><td><strong>82.0</strong></td><td>68.6</td><td>71.1</td></tr><tr><td>L<sub>3</sub> Average</td><td><strong>71.9</strong></td><td><strong>62.0</strong></td><td>76.9</td><td>40.1</td><td><strong>77.2</strong></td><td>39.5</td><td><strong>73.7</strong></td><td><strong>59.6</strong></td><td>74.1</td><td>57.2</td></tr><tr><td>No norm</td><td><strong>88.5</strong></td><td><strong>38.8</strong></td><td><strong>88.5</strong></td><td><strong>38.8</strong></td><td><strong>88.5</strong></td><td><strong>38.8</strong></td><td><strong>88.5</strong></td><td><strong>38.8</strong></td><td><strong>88.5</strong></td><td><strong>38.8</strong></td></tr><tr><th>KD-BU</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th></tr><tr><td>L<sub>1</sub> Average</td><td><strong>50.9</strong></td><td><strong>95.7</strong></td><td>70.0</td><td>88.6</td><td><strong>70.0</strong></td><td>88.4</td><td><strong>74.3</strong></td><td><strong>92.3</strong></td><td>69.8</td><td>88.4</td></tr><tr><td>L<sub>2</sub> Average</td><td><strong>59.0</strong></td><td><strong>94.1</strong></td><td>71.6</td><td>71.9</td><td><strong>71.7</strong></td><td>71.6</td><td><strong>70.6</strong></td><td><strong>92.8</strong></td><td>71.7</td><td>71.8</td></tr><tr><td>L<sub>3</sub> Average</td><td><strong>36.8</strong></td><td><strong>96.9</strong></td><td>64.8</td><td>92.1</td><td><strong>68.1</strong></td><td>91.3</td><td><strong>53.7</strong></td><td><strong>95.6</strong></td><td>67.8</td><td>91.7</td></tr><tr><td>No norm</td><td><strong>65.4</strong></td><td><strong>94.2</strong></td><td><strong>65.4</strong></td><td><strong>94.2</strong></td><td><strong>65.4</strong></td><td><strong>94.2</strong></td><td><strong>65.4</strong></td><td><strong>94.2</strong></td><td><strong>65.4</strong></td><td><strong>94.2</strong></td></tr><tr><th>LID</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th></tr><tr><td>L<sub>1</sub> Average</td><td><strong>50.8</strong></td><td><strong>95.4</strong></td><td>69.6</td><td>82.1</td><td><strong>69.4</strong></td><td>82.9</td><td><strong>88.9</strong></td><td><strong>49.9</strong></td><td>69.1</td><td>83.7</td></tr><tr><td>L<sub>2</sub> Average</td><td><strong>63.5</strong></td><td><strong>83.1</strong></td><td>73.7</td><td>70.1</td><td><strong>73.4</strong></td><td>70.7</td><td><strong>82.5</strong></td><td><strong>61.3</strong></td><td>73.2</td><td>71.3</td></tr><tr><td>L<sub>3</sub> Average</td><td><strong>53.8</strong></td><td><strong>90.8</strong></td><td>75.7</td><td>56.8</td><td><strong>79.9</strong></td><td>57.6</td><td><strong>71.3</strong></td><td><strong>79.7</strong></td><td>82.0</td><td>51.4</td></tr><tr><td>No norm</td><td><strong>88.0</strong></td><td><strong>58.1</strong></td><td><strong>88.0</strong></td><td><strong>58.1</strong></td><td><strong>88.0</strong></td><td><strong>58.1</strong></td><td><strong>88.0</strong></td><td><strong>58.1</strong></td><td><strong>88.0</strong></td><td><strong>58.1</strong></td></tr><tr><th>FS</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th></tr><tr><td>L<sub>1</sub> Average</td><td><strong>75.4</strong></td><td><strong>64.8</strong></td><td>92.8</td><td>25.1</td><td><strong>92.9</strong></td><td>24.9</td><td><strong>73.5</strong></td><td><strong>67.6</strong></td><td>92.9</td><td>24.6</td></tr><tr><td>L<sub>2</sub> Average</td><td><strong>74.9</strong></td><td><strong>65.8</strong></td><td>87.4</td><td>31.2</td><td><strong>87.6</strong></td><td>36.9</td><td><strong>73.7</strong></td><td><strong>67.2</strong></td><td>87.4</td><td>37.5</td></tr><tr><td>L<sub>3</sub> Average</td><td><strong>52.7</strong></td><td><strong>81.1</strong></td><td>73.0</td><td>60.1</td><td><strong>77.5</strong></td><td>55.7</td><td><strong>58.2</strong></td><td><strong>78.8</strong></td><td>75.7</td><td>58.5</td></tr><tr><td>No norm</td><td><strong>62.7</strong></td><td><strong>82.5</strong></td><td><strong>62.7</strong></td><td><strong>82.5</strong></td><td><strong>62.7</strong></td><td><strong>82.5</strong></td><td><strong>62.7</strong></td><td><strong>82.5</strong></td><td><strong>62.7</strong></td><td><strong>82.5</strong></td></tr><tr><th>MagNet</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th></tr><tr><td>L<sub>1</sub> Average</td><td><strong>49.6</strong></td><td><strong>93.7</strong></td><td>49.8</td><td>93.5</td><td><strong>49.7</strong></td><td>93.3</td><td><strong>50.1</strong></td><td><strong>93.2</strong></td><td><strong>49.1</strong></td><td><strong>93.8</strong></td></tr><tr><td>L<sub>2</sub> Average</td><td><strong>50.9</strong></td><td><strong>93.1</strong></td><td>52.3</td><td>89.6</td><td><strong>52.3</strong></td><td>89.4</td><td><strong>50.5</strong></td><td><strong>93.3</strong></td><td>51.8</td><td>91.4</td></tr><tr><td>L<sub>3</sub> Average</td><td><strong>78.0</strong></td><td><strong>46.1</strong></td><td><strong>79.2</strong></td><td><strong>44.6</strong></td><td><strong>80.2</strong></td><td>44.1</td><td><strong>79.2</strong></td><td><strong>44.6</strong></td><td>80.0</td><td><strong>44.6</strong></td></tr><tr><td>No norm</td><td><strong>79.9</strong></td><td><strong>45.7</strong></td><td><strong>79.9</strong></td><td><strong>45.7</strong></td><td><strong>79.9</strong></td><td><strong>45.7</strong></td><td><strong>79.9</strong></td><td><strong>45.7</strong></td><td><strong>79.9</strong></td><td><strong>45.7</strong></td></tr></tbody></table>

<table><caption>Table 4.2: Average performance on MNIST of all the detectors per objective and in MEAD. The worst results among all the settings is in bold; the ones in the single-armed setting is underlined. No norm denotes the group of attacks that do not depend on the norm constraint.</caption><thead><tr><th>MNIST</th><th colspan="2">MEAD</th><th colspan="2">ACE</th><th colspan="2">KL</th><th colspan="2">Gini</th><th colspan="2">FR</th></tr></thead><tbody><tr><th>NSS</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th></tr><tr><td>L<sub>1</sub> Average</td><td><strong>96.8</strong></td><td><strong>9.4</strong></td><td>97.0</td><td>8.2</td><td>97.1</td><td>8.6</td><td>97.4</td><td>7.0</td><td>97.1</td><td>8.1</td></tr><tr><td>L<sub>2</sub> Average</td><td><strong>90.3</strong></td><td><strong>26.5</strong></td><td>90.7</td><td>25.8</td><td>90.8</td><td>25.4</td><td>91.4</td><td>23.7</td><td><strong>90.6</strong></td><td><strong>26.5</strong></td></tr><tr><td>L<sub>3</sub> Average</td><td><strong>88.7</strong></td><td><strong>23.5</strong></td><td>89.5</td><td>23.5</td><td>89.5</td><td><strong>23.6</strong></td><td>90.0</td><td><strong>23.6</strong></td><td>89.8</td><td><strong>23.5</strong></td></tr><tr><td>No norm</td><td><strong>87.1</strong></td><td><strong>57.8</strong></td><td><strong>87.1</strong></td><td><strong>57.8</strong></td><td><strong>87.1</strong></td><td><strong>57.8</strong></td><td><strong>87.1</strong></td><td><strong>57.8</strong></td><td><strong>87.1</strong></td><td><strong>57.8</strong></td></tr><tr><th>KD-BU</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th></tr><tr><td>L<sub>1</sub> Average</td><td><strong>45.6</strong></td><td><strong>95.7</strong></td><td>59.9</td><td>93.0</td><td>59.3</td><td>93.1</td><td>61.4</td><td>92.7</td><td>58.9</td><td>93.3</td></tr><tr><td>L<sub>2</sub> Average</td><td><strong>50.3</strong></td><td><strong>94.8</strong></td><td>59.9</td><td><strong>93.0</strong></td><td>59.7</td><td>93.1</td><td>59.3</td><td>93.2</td><td>59.8</td><td><strong>93.0</strong></td></tr><tr><td>L<sub>3</sub> Average</td><td><strong>34.1</strong></td><td><strong>96.7</strong></td><td><strong>42.8</strong></td><td><strong>96.0</strong></td><td>44.7</td><td>95.8</td><td>48.6</td><td>95.3</td><td>44.9</td><td>95.8</td></tr><tr><td>No norm</td><td><strong>76.0</strong></td><td><strong>88.2</strong></td><td><strong>76.0</strong></td><td><strong>88.2</strong></td><td><strong>76.0</strong></td><td><strong>88.2</strong></td><td><strong>76.0</strong></td><td><strong>88.2</strong></td><td><strong>76.0</strong></td><td><strong>88.2</strong></td></tr><tr><th>LID</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th></tr><tr><td>L<sub>1</sub> Average</td><td><strong>79.9</strong></td><td><strong>54.9</strong></td><td>83.7</td><td>48.2</td><td>84.0</td><td>50.0</td><td>90.4</td><td><strong>52.1</strong></td><td>84.1</td><td>50.2</td></tr><tr><td>L<sub>2</sub> Average</td><td><strong>85.6</strong></td><td><strong>46.2</strong></td><td>87.4</td><td>44.1</td><td>87.0</td><td>45.1</td><td>87.6</td><td>44.4</td><td><strong>86.1</strong></td><td><strong>45.4</strong></td></tr><tr><td>L<sub>3</sub> Average</td><td><strong>77.9</strong></td><td><strong>55.1</strong></td><td>83.3</td><td>46.3</td><td>83.6</td><td>47.8</td><td>88.7</td><td>38.8</td><td><strong>83.0</strong></td><td><strong>49.5</strong></td></tr><tr><td>No norm</td><td><strong>98.1</strong></td><td><strong>8.2</strong></td><td><strong>98.1</strong></td><td><strong>8.2</strong></td><td><strong>98.1</strong></td><td><strong>8.2</strong></td><td><strong>98.1</strong></td><td><strong>8.2</strong></td><td><strong>98.1</strong></td><td><strong>8.2</strong></td></tr><tr><th>FS</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th></tr><tr><td>L<sub>1</sub> Average</td><td><strong>79.8</strong></td><td><strong>66.8</strong></td><td>83.4</td><td>57.6</td><td>83.5</td><td>57.1</td><td><strong>83.2</strong></td><td>53.0</td><td>83.4</td><td>57.4</td></tr><tr><td>L<sub>2</sub> Average</td><td><strong>73.5</strong></td><td><strong>69.0</strong></td><td>75.6</td><td>65.0</td><td>75.5</td><td>65.4</td><td><strong>74.5</strong></td><td><strong>67.0</strong></td><td>74.7</td><td>65.7</td></tr><tr><td>L<sub>3</sub> Average</td><td><strong>76.4</strong></td><td><strong>63.5</strong></td><td>80.8</td><td>54.6</td><td>80.2</td><td>54.6</td><td><strong>79.0</strong></td><td><strong>58.7</strong></td><td>80.4</td><td>58.2</td></tr><tr><td>No norm</td><td><strong>61.5</strong></td><td><strong>85.9</strong></td><td><strong>61.5</strong></td><td><strong>85.9</strong></td><td><strong>61.5</strong></td><td><strong>85.9</strong></td><td><strong>61.5</strong></td><td><strong>85.9</strong></td><td><strong>61.5</strong></td><td><strong>85.9</strong></td></tr><tr><th>MagNet</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th><th>AUROC↑%%</th><th>FPR↓95%%</th></tr><tr><td>L<sub>1</sub> Average</td><td><strong>98.1</strong></td><td><strong>5.7</strong></td><td>98.2</td><td>5.4</td><td>98.3</td><td><strong>5.6</strong></td><td>98.3</td><td>5.2</td><td><strong>98.1</strong></td><td><strong>5.6</strong></td></tr><tr><td>L<sub>2</sub> Average</td><td><strong>90.0</strong></td><td><strong>28.7</strong></td><td>90.6</td><td>27.6</td><td>90.8</td><td>27.8</td><td>90.6</td><td><strong>29.1</strong></td><td><strong>89.7</strong></td><td>28.1</td></tr><tr><td>L<sub>3</sub> Average</td><td><strong>98.5</strong></td><td><strong>10.3</strong></td><td>98.5</td><td>10.3</td><td><strong>98.4</strong></td><td><strong>10.6</strong></td><td>98.5</td><td>10.5</td><td>98.5</td><td>10.4</td></tr><tr><td>No norm</td><td><strong>86.9</strong></td><td><strong>74.3</strong></td><td><strong>86.9</strong></td><td><strong>74.3</strong></td><td><strong>86.9</strong></td><td><strong>74.3</strong></td><td><strong>86.9</strong></td><td><strong>74.3</strong></td><td><strong>86.9</strong></td><td><strong>74.3</strong></td></tr></tbody></table>
<table><caption>**Table 4.3: Performances of each detection method under the MEAD framework on CIFAR10 and MNIST averaged over the norm-based constraint.**
The best results among all the methods is in **bold**; the ones per type of detection method (i.e. Supervised and Unsupervised) are **underlined**.</caption><thead><tr><th rowspan="3"></th><th colspan="6">Supervised Methods</th><th colspan="6">Unsupervised Methods</th></tr><tr><th colspan="2">NSS</th><th colspan="2">KD-BU</th><th colspan="2">LID</th><th colspan="2">FS</th><th colspan="2">MagNet</th></tr><tr><th>AUROC†%</th><th>FPR↓90%</th><th>AUROC†%</th><th>FPR↓90%</th><th>AUROC†%</th><th>FPR↓90%</th><th>AUROC†%</th><th>FPR↓90%</th><th>AUROC†%</th><th>FPR↓90%</th></tr></thead><tbody><tr><td>MNIST</td><td>90.7</td><td><strong>29.3</strong></td><td>51.5</td><td>93.9</td><td>85.4</td><td>41.1</td><td>72.8</td><td>71.3</td><td><strong>93.4</strong></td><td>29.8</td></tr><tr><td>CIFAR10</td><td><strong>71.8</strong></td><td><strong>66.1</strong></td><td>53.0</td><td>95.2</td><td>64.0</td><td>81.8</td><td>66.4</td><td>73.6</td><td>64.6</td><td>69.7</td></tr></tbody></table>

## Datasets and classifiers.

We run the experiments on MNIST [LCB10] and CIFAR10 [Kri09]. The underlying classifiers are a simple CNN for MNIST, consisting of two blocks of two convolutional layers, a max-pooling layer, one fully-connected layer, one dropout layer, two fully-connected layers, and ResNet-18 for CIFAR10. The training procedures involve 100 epochs with Stochastic Gradient Descent (SGD) optimizer using a learning rate of 0.01 for the simple CNN and 0.1 for ResNet-18; a momentum of 0.9 and a weight decay of $10^{-5}$ for ResNet-18. Once trained, these networks are fixed and never modified again.

## Grouping attacks.

We test the methods on the attacks presented in Section 2.2.3, and we present them based on the norm constraint used to construct the attacks. Under the $L_1$-norm fall PGD with $\epsilon$ in $\{5, 10, 15, 20, 25, 30, 40\}$. Under the $L_2$-norm fall PGD with $\epsilon$ in $\{0.125, 0.25, 0.3125, 0.5, 1, 1.5, 2\}$, CW with $\epsilon = 0.01$, HOP with $\epsilon = 0.1$, and DF which has no constraint on $\epsilon$. Under the $L_\infty$-norm fall FGSM, BIM and PGD with $\epsilon$ in $\{0.0315, 0.0625, 0.125, 0.25, 0.3125, 0.5\}$, CW with $\epsilon = 0.3125$, and SA with $\epsilon = 0.3125$ for MNIST and $\epsilon = 0.125$ for CIFAR10. Finally, ST is not constrained by a norm or a maximum perturbation, as it is limited in maximum rotation (30 for CIFAR10 and 60 for MNIST) and translation (8 for CIFAR10 and 10 for MNIST).

## Detection Methods.

We tested detection methods introduced in Section 2.2.4. In the supervised case, we train the detectors using adversarial examples created by perturbing the samples in the original training sets with PGD under $L_\infty$-norm and $\epsilon = 0.03125$. In
the unsupervised case, the detectors only need natural samples in the training sets. They are tested on all the previously mentioned attacks, generated on the testing sets.

### 4.4.2 Experimental results

In this section, we refer to *single-armed setting* when we consider the setup where the adversarial examples are generated w.r.t. one of the objectives in Section 4.1. We provide the average of the performances of all the detection methods on CIFAR10 in Table 4.1 and on MNIST in Table 4.2. The detailed tables for each detection method (i.e., *NSS*, *LID*, *KD-BU*, *MagNet*, and *FS*) and for each dataset (i.e., CIFAR10 and MNIST) are reported in Appendix B.1.

#### MEAD and the single-armed setting.

Table 4.1 shows a decrease in the performance of all the detectors when going from the single-armed setting to MEAD. NSS is the more robust among the supervised methods when passing from the single-armed setting to the proposed setting. Indeed, (cf Table 4.1), in terms of AUROC↑%, it registers a decrease of up 4.9 percentage points under the $L_1$-norm constraint, 4.7 under the $L_2$-norm constraint, and 5.3 under the $L_∞$-norm constraint. This can be explained by the fact that the network in NSS is trained on the natural scene statistics extracted from the trained samples differently from the other detectors. In particular, these statistical properties are altered by the presence of adversarial perturbations and hence are found to be a good candidate to determine if a sample is adversarial or not. By looking closely at the results for NSS in Table B.2, it comes out that it performs better when evaluated on $L_∞$ norm constraint. Indeed, in this case, the adversarial examples at testing time are similar to those used at training time. Not surprisingly, the performance decreases when evaluated on other kinds of attacks. Notice that, in the single-armed setting, all the supervised methods turn out to be much more inefficient than when presented in the original papers. Indeed, as already explained in Section 4.4.1, we train the detectors using adversarial examples created by perturbing the samples in the original training sets with PGD under $L_∞$-norm and $\varepsilon = 0.03125$, and then we test them on a variety of attacks. Hence, we do not train a different detector for each kind of attack seen
at testing time. On the other side, the unsupervised detector *MagNet* appears to be more robust than *FS* when changing from the single-armed setting to MEAD. Indeed, in terms of AUROC↑%, it loses at most 2.2 percentage points ($L_∞$ norm case). On average, *FS* is the unsupervised detector that achieves the best performance on CIFAR10, while *MagNet* is the one to achieve the best performance on MNIST.

**Remark:** Some single-armed setting results turn out to be worse than the corresponding results in MEAD (cf Table B.2-B.6 and Table B.8-B.12 in Appendix B.1). We provide here an explanation of this phenomenon. Given a natural input sample $\mathbf{x}$, let $\mathbf{x}_{\ell}$ denotes the perturbed version of $\mathbf{x}$ according to some fixed norm $p$, fixed perturbation magnitude $\varepsilon$ and objective function $\ell$ between ACE, KL, Gini and FR. Suppose $g_{\theta}(\mathbf{x}_{\text{ACE}}) = y$, where $y$ is the ground true label of $\mathbf{x}$, this means that $\mathbf{x}_{\text{ACE}}$ is a perturbed version of the natural example but not adversarial. Assume instead $g_{\theta}(\mathbf{x}_{\text{KL}}) \neq y$, $g_{\theta}(\mathbf{x}_{\text{Gini}}) \neq y$ and $g_{\theta}(\mathbf{x}_{\text{FR}}) \neq y$. If at testing time the detector is able to recognize all of them as being positive (i.e., adversarial), then under MEAD, $\mathbf{x}_{\text{KL}}, \mathbf{x}_{\text{Gini}}, \mathbf{x}_{\text{FR}}$ would be considered a *true positive*. This example, counting as a true positive under MEAD, would instead be discarded under the single-armed setting of ACE, as $\mathbf{x}_{\text{ACE}}$ is neither a clean example nor an adversarial one. Then, the larger amount of true positives in MEAD can potentially lead to an increase in the global AUROC↑%.

**Effectiveness of the proposed objective functions.**

In Table B.1 and Table B.7, relegated to the Appendix, we report the averaged number of successful adversarial examples under the multi-armed setting as well as the details per single-armed settings on CIFAR10 and MNIST, respectively. The attacks are most successful when the value of the constraint $\varepsilon$ for every $L_p$-norm increases. Generating adversarial examples using the ACE for each attack scheme creates more harmful (adversarial) examples for the classifier than using any other objective. However, using either the Gini Impurity score, the Fisher-Rao objective, or the Kullback-Leibler divergence seems to create examples that are either equally or more difficult to be detected by the detection methods. For this purpose, we provide two examples. First, by looking at the results in Table B.4, we can deduce that *LID* finds it difficult to recognize the attacks
based on KL and FR objective functions but not the ones created through Gini. For example, with PGD1 and $\varepsilon = 40$, we register a decrease in AUROC↑% of 9.5 percentage points when going from the single-armed setting of Gini to the one of FR. Similarly, the decrease is 8.3 percentage points in the case of KL. This behavior is even more remarkable when we look at the results in terms of FPR↓$_{95\%}$: the gap between the best FPR↓$_{95\%}$ values (obtained via Gini) and the worst (via FR) is 30.7 percentage points. On the other side, the situation is reversed if we look at the results in Table B.5 as FS turns out to be highly inefficient at recognizing adversarial examples generated via the Gini Impurity score. By considering the results associated to the highest value of $\varepsilon$ for each norm, namely $\varepsilon = 40$ for L₁-norm; $\varepsilon = 2$ for L₂-norm; $\varepsilon = 0.5$ for L∞-norm, the gap between best FPR↓$_{95\%}$ values (obtained via KL divergence) and the worst (via Gini Impurity score), varies from a minimum of 41.7 (L∞-norm) to a maximum of 64.4 (L₂-norm) percentage points. This example, in agreement with Section 4.2, testify on real data that testing the detectors without taking into consideration the possibility of creating attacks through different objective functions leads to a biased and unrealistic estimation of their performances.

**Comparison between supervised and unsupervised detectors.**

The unsupervised methods find it challenging to recognize attacks crafted using the Gini Impurity score. Indeed, according to Section 4.2, that objective function creates attacks on the decision boundary of the pre-trained classifier. Consequently, the unsupervised detectors can easily associate such input samples with the cluster of naturals. Supervised methods detect Adversarial Cross-Entropy loss-based attacks more and, therefore, more volatile when it comes to other types of loss-based attacks. Overall, by looking at the results in Table 4.3 on both the datasets, most of the supervised and unsupervised methods achieve comparable performances with the multi-armed framework, meaning that the current use of the knowledge about the specific attack is not general enough. The exception to this is NSS, which, as already explained, seems to be the most general detector.
# A Minimax Approach Against
Multi-Armed Adversarial Attack
Detection

In this chapter, we propose an aggregation framework to combine the expertise of different adversarial examples detectors and address the problem of simultaneous attack detection as highlighted in Chapter 4. This method can aggregate pre-trained detectors without the need for additional training.

From a theoretical perspective, we revisit the multi-armed attack detection problem formulated in Chapter 4 and formalize it as a minimax cross-entropy risk. Based on this formulation, we derive a surrogate loss function and use it to characterize our optimal soft-detector in Eq. (5.5), leading to our proposed solution.

Empirical evaluations of our proposed solution on popular datasets, such as CIFAR10 and SVHN, show that it leads to higher and more consistent performance compared to the state-of-the-art (SOTA) in the simultaneous attack setup, even when using simple detectors that individually perform worse than SOTA detectors, as demonstrated in Section 5.2.
## 5.1 Formalization of the Problem of Detecting multi-armed Adversarial Attacks

In this section, we begin by formalizing the problem of multi-armed attacks as proposed in Chapter 4. We then delve deeper into the topic of optimal detectors and demonstrate how to apply our proposed solution to practical use-cases.

### 5.1.1 Statistical model

Let $\mathcal{K}$ be the countable set of indexes corresponding to each possible attack, e.g., based on various attack algorithms and loss functions, as described in Section 2.2.1. Let $\mathcal{M} = \{P_{XZ}^{(k)} : k \in \mathcal{K}\}$ be the set of joint probability distributions on $\mathcal{X} \times \mathcal{Z}$ which are indexed with $k$, $\forall k \in \mathcal{K}$, where $\mathcal{X}$ is the input (feature) space and $\mathcal{Z} = \{0, 1\}$ indicates a binary space label for the adversarial example detection task. At the evaluation time, the attacker selects an arbitrary strategy $k \in \mathcal{K}$ and then samples an input according to $p_{X|Z}^{(k)}(\mathbf{x}|z=1)$ which corresponds to the probability density function induced by the chosen attack $k$ where $p_{X|Z}^{(k)}(\mathbf{x}|z=0) = p_X(\mathbf{x})$ almost surely corresponds to the probability distribution of the natural samples. The learner is given a set of soft-detectors models:

$$ \mathcal{Q} = \{q_{\hat{Z}|\mathbf{u}}^{(k)} : \mathcal{U} \mapsto [0, 1]^2\}_{k \in \mathcal{K}}, $$

which have possibly been trained to detect attacks according to each strategy $k \in \mathcal{K}$, e.g., $q_{\hat{Z}|\mathbf{u}}^{(k)} = p_{\hat{Z}|\mathcal{U}}(z|\mathbf{u}; \psi_k)$ with parameters $\psi_k$ and $\mathbf{u} \in \mathcal{U} = \{g_{\theta}^l(\mathbf{x}) | \mathbf{x} \in \mathbb{R}^d\}$ denotes the space of logits. The set of possible detectors $\mathcal{Q}$ is available to the defender. However, the specific attack chosen by the attacker at the test time is unknown. In the remainder of this section, we formally devise an optimal detector that exploits full knowledge of the set $\mathcal{Q}$.

### 5.1.2 A novel objective for detection under simultaneous attacks

Consider a fixed input sample $\mathbf{x}_0$ and let $\mathbf{u}_0 = g_{\theta}^l(\mathbf{x}_0)$. Clearly, the problem at hand consists in finding an optimal soft-detector $q_{\hat{Z}|\mathbf{u}_0}^*$ that performs well simultaneously over all possible attacks in $\mathcal{K}$. This can be formalized as the solution
to the following minimax problem:

$$ \mathcal{L}(\mathcal{Q}, \mathbf{x}_0) = \min_{q_{\hat{Z}|\mathbf{u}_0}} \max_{k \in \mathcal{K}} \mathbb{E}_{q_{\hat{Z}|\mathbf{u}_0}^{(k)}} [-\log q_{\hat{Z}|\mathbf{u}_0}], \quad (5.1) $$

which requires to solve (5.1) for $\mathcal{Q}$ and for each given input sample $\mathbf{x}_0$. It is important to note that the minimization is performed over all (detectors) distributions $q_{\hat{Z}|\mathbf{u}_0}$, including elements that are not part of the set $\mathcal{Q}$.

That being said, the objective in Eq. (5.1) is not tractable computationally. To overcome this issue, we derive a surrogate (an upper bound) that can be computationally optimized. For any arbitrary choice of $q_{\hat{Z}|\mathbf{u}_0}$, we have

$$ \max_{k \in \mathcal{K}} \mathbb{E}_{q_{\hat{Z}|\mathbf{u}_0}^{(k)}} [-\log q_{\hat{Z}|\mathbf{u}_0}] \le \underbrace{\max_{k \in \mathcal{K}} \mathbb{E}_{q_{\hat{Z}|\mathbf{u}_0}^{(k)}} [-\log q_{\hat{Z}|\mathbf{u}_0}^{(k)}]}_{=\text{constant term}} + \max_{k \in \mathcal{K}} \mathbb{E}_{q_{\hat{Z}|\mathbf{u}_0}^{(k)}} \left[ \log \left( \frac{q_{\hat{Z}|\mathbf{u}_0}^{(k)}}{q_{\hat{Z}|\mathbf{u}_0}} \right) \right]. \quad (5.2) $$

Proof of Eq. (5.2).

$$ \begin{aligned} \max_{k \in \mathcal{K}} \mathbb{E}_{q_{\hat{Z}|\mathbf{u}_0}^{(k)}} [-\log q_{\hat{Z}|\mathbf{u}_0}] &= \max_{k \in \mathcal{K}} \left[ \mathbb{E}_{q_{\hat{Z}|\mathbf{u}_0}^{(k)}} [-\log q_{\hat{Z}|\mathbf{u}_0}^{(k)}] + \mathbb{E}_{q_{\hat{Z}|\mathbf{u}_0}^{(k)}} \left[ \log \left( \frac{q_{\hat{Z}|\mathbf{u}_0}^{(k)}}{q_{\hat{Z}|\mathbf{u}_0}} \right) \right] \right] \\ &\le \max_{k \in \mathcal{K}} \mathbb{E}_{q_{\hat{Z}|\mathbf{u}_0}^{(k)}} [-\log q_{\hat{Z}|\mathbf{u}_0}^{(k)}] + \max_{k \in \mathcal{K}} \mathbb{E}_{q_{\hat{Z}|\mathbf{u}_0}^{(k)}} \left[ \log \left( \frac{q_{\hat{Z}|\mathbf{u}_0}^{(k)}}{q_{\hat{Z}|\mathbf{u}_0}} \right) \right]. \end{aligned} \quad \square $$

Observe that the first term in (5.2) of the upper bound is constant w.r.t. the choice of $q_{\hat{Z}|\mathbf{u}_0}$ and the second term is well-known as being equivalent to the average worst-case regret [BRY98]. This upper bound provides a surrogate to our intractable objective in (5.1) that can be minimized over all $q_{\hat{Z}|\mathbf{u}_0}$. We can formally state our problem as follows:

$$ \tilde{\mathcal{L}}(\mathcal{Q}, \mathbf{x}_0) = \min_{q_{\hat{Z}|\mathbf{u}_0}} \max_{k \in \mathcal{K}} \mathbb{E}_{q_{\hat{Z}|\mathbf{u}_0}^{(k)}} \left[ \log \left( \frac{q_{\hat{Z}|\mathbf{u}_0}^{(k)}}{q_{\hat{Z}|\mathbf{u}_0}} \right) \right] = \min_{q_{\hat{Z}|\mathbf{u}_0}} \max_{P_{\Omega}} \mathbb{E}_{\Omega} \left[ D_{\text{KL}} \left( q_{\hat{Z}|\mathbf{u}_0}^{(\Omega)} \| q_{\hat{Z}|\mathbf{u}_0} \right) \right], \quad (5.3) $$
where the min is taken over all the possible distributions $q_{\hat{Z}|u_0}$; and $\Omega$ is a discrete random variable with $P_\Omega$ denoting a generic probability distribution whose probabilities are $(\omega_1, \dots, \omega_{|\mathcal{K}|})$, i.e., $P_\Omega(k) = \omega_k$; and $D_{\text{KL}}(\cdot|\cdot)$ is the Kullback-Leibler divergence, representing the expected value of regret of $q_{\hat{Z}|U}$ w.r.t. the worst-case distribution in $\mathcal{Q}$.

*Proof of Eq. (5.3).* The equality hold by noticing that

$$ \max_{P_\Omega} \mathbb{E}_\Omega \left[ D_{\text{KL}} \left( q_{\hat{Z}|u_0}^{(\Omega)} \| q_{\hat{Z}|u_0} \right) \right] \le \max_{k \in \mathcal{K}} \mathbb{E}_{q_{\hat{Z}|u_0}^{(k)}} \left[ \log \left( \frac{q_{\hat{Z}|u_0}^{(k)}}{q_{\hat{Z}|u_0}} \right) \right], $$

and moreover,

$$ \max_{k \in \mathcal{K}} \mathbb{E}_{q_{\hat{Z}|u_0}^{(k)}} \left[ \log \left( \frac{q_{\hat{Z}|u_0}^{(k)}}{q_{\hat{Z}|u_0}} \right) \right] = \mathbb{E}_{\bar{\Omega}} \left[ D_{\text{KL}} \left( q_{\hat{Z}|u_0}^{(\bar{\Omega})} \| q_{\hat{Z}|u_0} \right) \right], $$

by choosing the random variable $\bar{\Omega}$ with uniform probability over the set of maximizers $\bar{\mathcal{K}} = \text{argmax}_{k \in \mathcal{K}} \mathbb{E}_{q_{\hat{Z}|u_0}^{(k)}} \left[ \log \left( \frac{q_{\hat{Z}|u_0}^{(k)}}{q_{\hat{Z}|u_0}} \right) \right]$, zero otherwise. $\square$

The convexity of the KL-divergence allows us to rewrite Eq. (5.3) as follows:

$$ \min_{q_{\hat{Z}|u_0}} \max_{P_\Omega} \mathbb{E}_\Omega \left[ D_{\text{KL}} \left( q_{\hat{Z}|u_0}^{(\Omega)} \| q_{\hat{Z}|u_0} \right) \right] = \max_{P_\Omega} \min_{\hat{q}_{\hat{Z}|u_0}} \mathbb{E}_\Omega \left[ D_{\text{KL}} \left( q_{\hat{Z}|u_0}^{(\Omega)} \| q_{\hat{Z}|u_0} \right) \right]. \quad (5.4) $$

*Proof of Eq. (5.4).* We consider a zero-sum game with a concave-convex mapping defined on a product of convex sets. The sets of all probability distributions $q_{\hat{Z}|u_0}$ and $P_\Omega$ are two nonempty convex sets, bounded and finite dimensional. On the other hand, $(P_\Omega, q_{\hat{Z}|u_0}) \to \mathbb{E}_\Omega \left[ D_{\text{KL}} \left( q_{\hat{Z}|u_0}^{(\Omega)} \| q_{\hat{Z}|u_0} \right) \right]$ is a concave-convex mapping, i.e., $P_\Omega \to \mathbb{E}_\Omega \left[ D_{\text{KL}} \left( q_{\hat{Z}|u_0}^{(\Omega)} \| q_{\hat{Z}|u_0} \right) \right]$ is concave and $q_{\hat{Z}|u_0} \to \mathbb{E}_\Omega \left[ D_{\text{KL}} \left( q_{\hat{Z}|u_0}^{(\Omega)} \| q_{\hat{Z}|u_0} \right) \right]$ is convex for every $(P_\Omega, q_{\hat{Z}|u_0})$. Then, by classical min-max theorem [vN28] we have that Eq. (5.4) holds. $\square$

The solution to Eq. (5.4) provides the optimal distribution $P_\Omega^*$, i.e. the col-
lection of weights {$w_k^*$}, which leads to our soft-detector [BRY98]:

$$ \hat{q}_{\hat{Z}|u_0}^* = \sum_{k \in \mathcal{K}} w_k^* \cdot q_{\hat{Z}|u_0}^{(k)}, \quad \text{with} \quad P_{\Omega}^* = \underset{\{\omega_k\}}{\operatorname{argmax}} I_{u_0}(\Omega; \hat{Z}), \qquad (5.5) $$

where $I_{u_0}(\cdot; \cdot)$ denotes the Shannon mutual information between the random variable $\Omega$, distributed according to {$\omega_k$}, and the binary soft-prediction variable $\hat{Z}$, distributed according to $q_{\hat{Z}|u_0}^{(k)}$ and conditioned on the particular test example $u_0$.

*Proof of Eq. (5.5).* It is enough to show that

$$ \min_{\hat{q}_{\hat{Z}|u_0}} \mathbb{E}_{\Omega} \left[ D_{\text{KL}} \left( q_{\hat{Z}|u_0}^{(\Omega)} \| q_{\hat{Z}|u_0} \right) \right] = I_{u_0}(\Omega; \hat{Z}), \qquad (5.6) $$

for every random variable $\Omega$ distributed according to an arbitrary probability distribution $P_{\Omega}$ and each distribution $q_{\hat{Z}|u_0}^{(\Omega)}$. We begin by showing that

$$ \mathbb{E}_{\Omega} \left[ D_{\text{KL}} \left( q_{\hat{Z}|u_0}^{(\Omega)} \| q_{\hat{Z}|u_0} \right) \right] \geq I_{u_0}(\Omega; \hat{Z}), $$

for any arbitrary distributions $P_{\Omega}$ and $q_{\hat{Z}|u_0}^{(\Omega)}$. To this end, we use the following identities:

$$ \begin{aligned} \mathbb{E}_{\Omega} \left[ D_{\text{KL}} \left( q_{\hat{Z}|u_0}^{(\Omega)} \| q_{\hat{Z}|u_0} \right) \right] &= \mathbb{E}_{\Omega} \mathbb{E}_{q_{\hat{Z}|u_0}^{(\Omega)}} \left( \log \frac{q_{\hat{Z}|u_0}^{(\Omega)}}{q_{\hat{Z}|u_0}} \right) \\ &= \mathbb{E}_{\Omega} \mathbb{E}_{q_{\hat{Z}|u_0}^{(\Omega)}} \left( \log \frac{q_{\hat{Z}|u_0}^{(\Omega)}}{P_{\hat{Z}}} \right) + D_{\text{KL}} \left( P_{\hat{Z}} \| q_{\hat{Z}|u_0} \right) \\ &= I_{u_0}(\Omega; \hat{Z}) + D_{\text{KL}} \left( P_{\hat{Z}} \| q_{\hat{Z}|u_0} \right) \geq I_{u_0}(\Omega; \hat{Z}), \quad (5.7) \end{aligned} $$

where $P_{\hat{Z}}$ denotes the marginal distribution of $q_{\hat{Z}|u_0}^{(\Omega)}$ w.r.t. $P_{\Omega}$ and the last inequality follows since the KL divergence is positive. Finally, it is easy to check that by selecting $q_{\hat{Z}|u_0} = P_{\hat{Z}}$ the lower bound in (5.7) is achieved which proves the identity in expression (5.6). By taking the maximum overall probability distributions $P_{\Omega}$ at both sides of expression (5.6) the claim follows. $\square$

**From theory to our practical detector.** According to our derivation in Eq. (5.5), the optimal detector turns out to be given by a mixture of the
$|K|$ detectors belonging to the class $Q$, with weights carefully optimized to maximize the mutual information between $\Omega$ and the predicted variable $\hat{Z}$ for each detector in the class $Q$. Using this key ingredient, it is straightforward to devise our optimal detector.

**Definition 3.** For any $0 \le \gamma \le 1$ and a given $\mathbf{x}_0 \in \mathcal{X}$, let us define the following detector $D: \mathbb{R}^d \to \{0, 1\}$:

$$D(\mathbf{x}_0) \stackrel{\text{def}}{=} 1 \left[ q_{\hat{Z}|u_0}^*(\hat{z} = 1 | g_\theta^l(\mathbf{x}_0)) > \gamma \right], \quad (5.8)$$

where $1[\cdot]$ is the indicator function.

## 5.2 Experimental Results

We test our proposed solution by deploying it against the multi-armed adversarial attacks framework introduced in Chapter 4, and by evaluating its detection performance<fnref n="1" />.

In our empirical evaluation, we assume that a third party provides us with four simple supervised detectors. Each of them is trained to detect a single specific kind of attack. This is a reasonable assumption, as many methods in the literature can successfully detect at least one type of attack and fail at detecting others. In addition, to emphasize the role played by the proposed method, these detectors are merely shallow networks (3 fully-connected layers with 256 nodes each), which are only allowed to observe the logits of the target classifier to distinguish between natural and adversarial samples. Due to their specifics, these individual shallow detectors are bound to perform very poorly, i.e. much worse than SOTA detectors, against attacks they have not been trained on, as shown in Fig. 5.3. This aspect enhances the value of our solution, which attains favorable performance by aggregating detectors that individually exhibit subpar performance w.r.t. SOTA adversarial examples detection methods.

### 5.2.1 Evaluation framework

**Evaluation setup:** MEAD. We consider all the attack algorithms mentioned in MEAD Chapter 4, and we group them by the corresponding norm and the

<footnote n="1">¹The source code will be released upon acceptance of the paper this chapter is based on.</footnote>
perturbation magnitude. For each natural sample and each gradient-based attack algorithm (i.e., FGSM, PGD or BIM), we create four adversarial examples, each corresponding to one of the loss functions described in Section 4.1.

Table 5.1 reports all the attacks in the multi-armed setting. Each cell corresponds to a group of attacks crafted according to the algorithm (reported in the cell), the associated norm (indicated by the column label) and perturbation magnitude (indicated by the row label) and one of the considered four loss functions. Thus, for example, when we consider $L_\infty$ norm and $\varepsilon = 0.125$, the detector is evaluated on $4 + 4 + 4 + 1 = 13$ simultaneous adversarial attacks. Note that we discard the perturbed examples that do not fool the classifier as, by definition, they are neither natural nor adversarial.

**Evaluation metrics.**

Following the evaluation setup described above, for each sample and for each

<table><caption>**Table 5.1: MEAD.** Each cell corresponds to attacks simultaneously executed on the targeted classifier. Attacks created using all the losses in Section 4.1 are marked with *. Attacks such as SA and DF are not dependent on the choice for the loss but are equally considered as part of the multi-armed framework. Empty cells correspond to combinations of perturbation magnitude and norm constraint that are usually not considered in the literature.</caption><thead><tr><th></th><th>L<sub>1</sub></th><th>L<sub>2</sub></th><th>L<sub>∞</sub></th><th>No norm</th></tr></thead><tbody><tr><td>ε = 0.01</td><td>-</td><td>CW2</td><td>-</td><td>-</td></tr><tr><td>ε = 0.03125</td><td>-</td><td>-</td><td>PGDi*,FGSM*,BIM*</td><td>-</td></tr><tr><td>ε = 0.0625</td><td>-</td><td>-</td><td>PGDi*,FGSM*,BIM*</td><td>-</td></tr><tr><td>ε = 0.1</td><td>-</td><td>HOP</td><td>-</td><td>-</td></tr><tr><td>ε = 0.125</td><td>-</td><td>PGD2*</td><td>PGDi*,FGSM*,BIM*,SA</td><td>-</td></tr><tr><td>ε = 0.25</td><td>-</td><td>PGD2*</td><td>PGDi*,FGSM*,BIM*</td><td>-</td></tr><tr><td>ε = 0.3125</td><td>-</td><td>PGD2*</td><td>PGDi*,FGSM*,BIM*,CWi</td><td>-</td></tr><tr><td>ε = 0.5</td><td>-</td><td>PGD2*</td><td>PGDi*,FGSM*,BIM*</td><td>-</td></tr><tr><td>ε = 1</td><td>-</td><td>PGD2*</td><td>-</td><td>-</td></tr><tr><td>ε = 1.5</td><td>-</td><td>PGD2*</td><td>-</td><td>-</td></tr><tr><td>ε = 2</td><td>-</td><td>PGD2*</td><td>-</td><td>-</td></tr><tr><td>ε = 5</td><td>PGD1*</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ε = 10</td><td>PGD1*</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ε = 15</td><td>PGD1*</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ε = 20</td><td>PGD1*</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ε = 25</td><td>PGD1*</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ε = 30</td><td>PGD1*</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ε = 40</td><td>PGD1*</td><td>-</td><td>-</td><td>-</td></tr><tr><td>No ε</td><td>-</td><td>DF</td><td>-</td><td>-</td></tr><tr><td>max. rotation = 30</td><td>-</td><td>-</td><td>-</td><td>STA</td></tr><tr><td>max. translation = 8</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>

group of attacks corresponding to each cell in Table 5.1 we consider a detection
successful, i.e. a true positive, if and only if all the adversarial attacks are detected. Otherwise, we report a false negative. We use the classical definitions of true negative and false positive for the natural samples detection. This means that a true negative is a natural sample detected as natural, and a false positive is a natural sample detected as adversarial. We measure the performance of the detectors in terms of i) AUROC↑% [DG06b] (the Area Under the Receiver Operating Characteristic curve) which represents the ability of the detector to discriminate between adversarial and natural examples (higher is better); ii) FPR at 95 % TPR (FPR↓$_{95\%}$%), i.e., the percentage of natural examples detected as adversarial when 95 % of the adversarial examples are detected (lower is better).

**Datasets and pre-trained classifiers.** We run our experiments on CIFAR10 [Kri09] and SVHN [NWC$^{+}$11] image datasets. For both, the pre-trained target classifier is a ResNet-18 models that has been trained for 100 epochs, using SGD optimizer with a learning rate equal to 0.1, weight decay equal to $10^{-5}$, and momentum equal to 0.9. The accuracy achieved by the classifiers on the original clean data is 99% for CIFAR10 and 100% for SVHN over the train split; 93.3% for CIFAR10 and 95.5% for SVHN over the test split.

**Detectors.** The proposed method aggregates four simple pre-trained detectors. The detectors are four fully-connected neural networks composed of 3 layers of 256 nodes each. All the detectors are trained for 100 epochs, using SGD optimizer with a learning rate of 0.01 and weight decay 0.0005. They are trained to distinguish between natural and adversarial examples created according to the PGD algorithm, under $L_\infty$ norm constraint and perturbation magnitude $\varepsilon = 0.125$ for CIFAR10 and $\varepsilon = 0.25$ for SVHN. Each detector is trained on natural and adversarial examples generated using one of the loss functions mentioned in Section 4.1 (i.e., ACE Eq. (4.2), KL Eq. (4.3), FR Eq. (4.4), or Gini Eq. (4.5)) to craft its adversarial training samples. We want to point out that the purpose of this paper is not creating a new supervised detector but rather to show a method to aggregate a set of pre-trained detectors. Moreover, it is important to notice that either supervised and unsupervised methods can be added to or pool of experts (cf. Appendix C.2.1), provided that they output a
confidence on the input sample being or not an adversarial example. We further expand on the selection of the $\varepsilon$ parameter of the adversarial examples used at training time in Appendix C.2.1 (cf. Tables C.3 and C.5).

**NSS [KFHD20].** We compare the proposed method with NSS, which is the best among the supervised SOTA methods against multi-armed adversarial attacks (cf. Section 4.4). NSS characterizes the adversarial perturbations through the use of *natural scene statistics*, i.e., statistical properties that can be altered by the presence of adversarial perturbations. NSS is trained by using PGD algorithm, $L_\infty$ norm constraint and perturbation magnitude $\varepsilon = 0.03125$ for CIFAR10 and $\varepsilon = 0.0625$ for SVHN. We further expand on the selection of the $\varepsilon$ parameter of the adversarial examples used at training time in Tables C.2 and C.4 and Appendix C.2.1.

**On the optimization of Eq. (5.5).** For the optimization of Eq. (5.5), we rely on the SciPy [VGO+20] library, the `optimize` package, and the `minimize` function which uses the *Sequential Least Squares Programming* (SLSQP) algorithm to find the optimum. Further details can be found in Appendix C.1.

### 5.2.2 Discussion

We present the main experimental results to show the effectiveness of the proposed method for adversarial attack detection. Further discussion on these results, as well as additional experiments, can be found in Appendix C.2.

#### The shallow detectors

Figs. 5.1 to 5.3 provides a graphical interpretation of

<figure><img src="image_13.png" /><figcaption>**Figure 5.3:** The *shallow* detectors are named after the loss function used to craft the attacks they are trained to detect. The SOTA method NSS outperforms all the individual shallow detectors. By aggregating shallow models, we can achieve a detector with comparable or better performance than SOTA.</figcaption></figure>
<figure><img src="image_14.png" /><figcaption>**Figure 5.1:** Performance of the various detectors grouped by $L_p$-norm and perturbation magnitude $\varepsilon$ on CIFAR10. Each *shallow* detector is named after the loss function used to craft the attacks they are trained to detect. The plot shows how our method consistently attains better performance than the single one on all the different adversarial attacks, supporting the claim of optimality in Section 5.1.</figcaption></figure>

the detection performance when ResNet18, trained on CIFAR10, is the target classifier. The single detectors are named after the loss function used to craft the adversarial examples on which each detector is trained along with the natural samples. The main takeaway from Fig. 5.3 is the observation that, when considered individually, the shallow detectors are clearly subpar w.r.t. the state-of-the-art adversarial attacks detection mechanism. On the contrary, the aggregation provided by our method results in detection performance comparable to SOTA performance and, in some cases, outperforms well-established detection mechanisms. Figure 5.1 sheds light on the fact that the mixture of experts attained by our proposed method can consistently improve the detection of adversarial examples over several multi-armed attacks mounted using different norms and perturbation magnitudes.
<figure><img src="image_15.png" /><figcaption>(a) Attacks crafted with PGD algorithm, the FR loss, $\epsilon = 40$, and norm constraint $L_1$</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>(c) Ours against attacks crafted with PGD algorithm, the FR loss, $\epsilon = 40$, and norm constraint $L_1$</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>(b) Attacks crafted with FGSM algorithm, the FR loss, $\epsilon = 40$, and norm constraint $L_\infty$</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>(d) NSS against attacks crafted with PGD algorithm, the FR loss, $\epsilon = 40$, and norm constraint $L_1$</figcaption></figure>

**Figure 5.2: Discrimination performances.** In Fig. 5.2a and Fig. 5.2b, the accuracies of the detectors on natural and adversarial examples; in Fig. 5.2c and Fig. 5.2d we show how the proposed method and NSS split the data samples. We report the results for detecting adversarial examples in pink and the results for detecting natural examples in blue.

One main takeaway of this paper is that, if we are provided with generally non-robust detectors whose performance is good only against a limited amount of attacks (as it is confirmed by Figs. 5.1 and 5.3), we can successfully aggregate them through the proposed method to obtain a consistently better detection.

In Fig. 5.2 we consider attacks crafted according to the PGD algorithm, the FR loss, $\epsilon = 40$, and norm constraint $L_1$ (cf. Figs. 5.2a, 5.2c and 5.2d), and attacks crafted according to the FGSM algorithm, FR loss, $\epsilon = 0.5$, and $L_\infty$
norm in Fig. 5.2b. We also report the performance of the considered detectors in terms of detection accuracy over the natural examples in blue and the adversarial examples in pink. As we can observe, the individual detectors, which are named after the loss functions ACE, FR, KL, and Gini, exhibit different behaviors for the specific attack. In Fig. 5.2a, the Gini detector drastically fails at detecting the attack as its accuracy plummets to 0% on the adversarial examples. In the same way, the FR and KL detectors but mostly the ACE detector, perform poorly against FGSM (cf. Fig. 5.2b). On the contrary, our method, benefiting from the aggregation, obtains favorable results in both cases, confirming what we had previously observed.

The histograms in Figs. 5.2c and 5.2d show how the method we propose and NSS separate natural (blue) and adversarial examples (pink), respectively. The values along the horizontal axis represent the probability of being classified as adversarial, and the vertical axis represents the frequency of the samples within the bins. The detection error is proportional to the area of overlap between the blue and the pink histograms. Fig. 5.2c and Fig. 5.2d suggest that the proposed method achieves lower detection error on the considered attack, as it is confirmed in Table 5.2 where our proposed method attains 92.1 AUROC↑%, while NSS only achieves 76.1 AUROC↑% and. Additional plots are provided in Appendix C.2.4.

In particular, the performance attained by the proposed method is consistent across the largest part of the considered multi-armed adversarial attacks, as confirmed in Table 5.2 and Fig. 5.3.

## Evaluation of the proposed aggregator in MEAD

On CIFAR10, our aggregator achieves maximum AUROC improvement w.r.t. NSS is 79.5 percentage points and happens for attacks under $L_\infty$-norm constraint, $\varepsilon = 0.125$ and PGD*, FGSM*, BIM*, SA, i.e. when as many as 13 different simultaneous adversarial attacks are mounted. Similarly, for our proposed method the maximum attained FPR at 95% TPR improvement w.r.t. NSS is 90.3 percentage points and happens for attacks under $L_\infty$-norm constraint, $\varepsilon = 0.5$ and PGD*, FGSM*, BIM*, i.e., when as many as 12 different simultaneous adversarial attacks are mounted. Our aggregator outperforms NSS in the case of the attacks with $L_1$
<table><caption>Table 5.2: Comparison between the proposed method and NSS on CIFAR10 and SVHN. The * symbol means the perturbation mechanism is executed in parallel four times starting from the same original clean sample, each time using one of the objective losses between ACE Eq. (4.2), KL Eq. (4.3), FR Eq. (4.4), Gini Eq. (4.5).</caption><thead><tr><th rowspan="3"></th><th colspan="6">CIFAR10</th><th colspan="6">SVHN</th></tr><tr><th colspan="2">NSS</th><th colspan="2">Ours</th><th colspan="2">NSS</th><th colspan="2">Ours</th></tr><tr><th>AUROC↑%</th><th>FPR↓95%</th><th>AUROC↑%</th><th>FPR↓95%</th><th>AUROC↑%</th><th>FPR↓95%</th><th>AUROC↑%</th><th>FPR↓95%</th></tr></thead><tbody><tr><td>Norm L<sub>1</sub><br>PGD1*</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>ε = 5</td><td>48.5</td><td>94.2</td><td>62.1</td><td>87.1</td><td>40.2</td><td>91.3</td><td>76.9</td><td>79.0</td></tr><tr><td>ε = 10</td><td>54.0</td><td>90.3</td><td>56.8</td><td>90.6</td><td>36.9</td><td>91.3</td><td>73.0</td><td>82.5</td></tr><tr><td>ε = 15</td><td>58.8</td><td>86.8</td><td>69.3</td><td>84.4</td><td>35.6</td><td>91.3</td><td>78.9</td><td>72.5</td></tr><tr><td>ε = 20</td><td>63.5</td><td>82.3</td><td>78.7</td><td>73.1</td><td>36.1</td><td>91.3</td><td>83.6</td><td>60.7</td></tr><tr><td>ε = 25</td><td>67.7</td><td>77.2</td><td>87.1</td><td>50.8</td><td>37.8</td><td>91.3</td><td>87.0</td><td>48.6</td></tr><tr><td>ε = 30</td><td>71.4</td><td>73.4</td><td>90.3</td><td>35.4</td><td>39.8</td><td>91.3</td><td>89.3</td><td>37.2</td></tr><tr><td>ε = 40</td><td>76.1</td><td>67.3</td><td>92.1</td><td>26.4</td><td>43.1</td><td>91.3</td><td>92.6</td><td>20.0</td></tr><tr><td>Norm L<sub>2</sub><br>PGD2*</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>ε = 0.125</td><td>48.3</td><td>94.3</td><td>63.9</td><td>85.4</td><td>40.8</td><td>91.3</td><td>80.2</td><td>74.5</td></tr><tr><td>ε = 0.25</td><td>53.2</td><td>91.2</td><td>57.1</td><td>90.5</td><td>37.2</td><td>91.3</td><td>74.0</td><td>81.7</td></tr><tr><td>ε = 0.3125</td><td>55.8</td><td>89.2</td><td>61.0</td><td>88.9</td><td>36.1</td><td>91.3</td><td>75.2</td><td>79.4</td></tr><tr><td>ε = 0.5</td><td>63.3</td><td>82.6</td><td>79.4</td><td>73.2</td><td>35.9</td><td>91.3</td><td>82.5</td><td>64.4</td></tr><tr><td>ε = 1</td><td>76.4</td><td>67.5</td><td>91.4</td><td>26.4</td><td>42.5</td><td>91.3</td><td>92.3</td><td>24.7</td></tr><tr><td>ε = 1.5</td><td>81.0</td><td>63.0</td><td>91.9</td><td>24.2</td><td>46.3</td><td>91.3</td><td>94.1</td><td>7.5</td></tr><tr><td>ε = 2</td><td>82.6</td><td>62.3</td><td>91.9</td><td>24.1</td><td>49.8</td><td>91.3</td><td>94.9</td><td>5.3</td></tr><tr><td>DeepFool</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>No ε</td><td>57.0</td><td>91.7</td><td>81.9</td><td>54.8</td><td>41.3</td><td>91.3</td><td>94.9</td><td>12.0</td></tr><tr><td>CW2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>ε = 0.01</td><td>56.4</td><td>90.8</td><td>53.4</td><td>92.2</td><td>41.0</td><td>91.3</td><td>54.2</td><td>92.0</td></tr><tr><td>HOP</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>ε = 0.1</td><td>66.1</td><td>87.0</td><td>86.1</td><td>49.1</td><td>67.6</td><td>84.2</td><td>96.0</td><td>10.2</td></tr><tr><td>Norm L<sub>∞</sub><br>PGDi*, FGSM*, BIM*</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>ε = 0.03125</td><td>83.0</td><td>55.3</td><td>82.3</td><td>59.7</td><td>86.3</td><td>46.9</td><td>81.4</td><td>64.9</td></tr><tr><td>ε = 0.0625</td><td>96.0</td><td>17.2</td><td>92.0</td><td>29.6</td><td>88.9</td><td>0.7</td><td>89.1</td><td>33.3</td></tr><tr><td>ε = 0.25</td><td>97.3</td><td>0.6</td><td>95.9</td><td>8.8</td><td>51.6</td><td>88.9</td><td>92.3</td><td>16.4</td></tr><tr><td>ε = 0.5</td><td>82.5</td><td>100.0</td><td>94.6</td><td>9.7</td><td>46.7</td><td>86.7</td><td>92.9</td><td>14.4</td></tr><tr><td>PGDi*, FGSM*, BIM*, SA</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>ε = 0.125</td><td>9.4</td><td>99.9</td><td>88.9</td><td>40.8</td><td>32.9</td><td>91.3</td><td>89.2</td><td>29.1</td></tr><tr><td>PGDi*, FGSM*, BIM*, CWi</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>ε = 0.3125</td><td>63.2</td><td>99.1</td><td>80.0</td><td>61.1</td><td>41.3</td><td>91.3</td><td>88.2</td><td>33.1</td></tr><tr><td>No norm<br>STA</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>No ε</td><td>88.5</td><td>38.8</td><td>82.7</td><td>52.4</td><td>91.2</td><td>0.2</td><td>90.2</td><td>23.2</td></tr></tbody></table>
<table><caption>**Table 5.3: Comparison between Ours and Ours+NSS on CIFAR10.** The * symbol means the perturbation mechanism is executed in parallel four times starting from the same original clean sample, each time using one of the objective losses between ACE Eq. (4.2), KL Eq. (4.3), FR Eq. (4.4), Gini Eq. (4.5). We focus only in the cases in which the proposed method is outperformed from the corresponding competitors.</caption><thead><tr><th rowspan="3"></th><th colspan="4">CIFAR10</th></tr><tr><th colspan="2">Ours</th><th colspan="2">Ours+NSS</th></tr><tr><th>AUROC↑%</th><th>FPR↓<sub>95%</sub>%</th><th>AUROC↑%</th><th>FPR↓<sub>95%</sub>%</th></tr></thead><tbody><tr><td><strong>Norm L<sub>2</sub></strong><br><u>CW2</u><br>ε = 0.01</td><td>53.4</td><td>92.2</td><td>54.1</td><td>91.3</td></tr><tr><td><strong>Norm L<sub>∞</sub></strong><br>PGDi*, FGSM*, BIM*<br>ε = 0.03125</td><td>82.3</td><td>59.7</td><td>89.9</td><td>34.4</td></tr><tr><td>ε = 0.0625</td><td>92.0</td><td>29.6</td><td>96.4</td><td>9.0</td></tr><tr><td>ε = 0.25</td><td>95.9</td><td>8.8</td><td>96.7</td><td>3.5</td></tr><tr><td><strong>No norm</strong><br><u>STA</u><br>No ε</td><td>82.7</td><td>52.4</td><td>87.3</td><td>35.4</td></tr></tbody></table>

and L₂ norm, regardless of the algorithm or the perturbation magnitude, and in the case of L∞ norm with large perturbations. However, for the attacks with L∞ norm and small ε, although the proposed method's performance is comparable to that of NSS, we notice a slight degradation. To shed light on this, we remind that individual detectors aggregated are based on the classifier's logits; NSS, on the other hand, extracts natural scene statistics from the inputs. This more sophisticated technique makes NSS perform well when tested on attacks with similar ε and the same norm as the ones seen at training time. Similar conclusions can be drawn for the results on SVHN (cf. Table 5.2).

Table 5.3 shows the modularity of the proposed method when SOTA detection methods, NSS (a) and FS (b), are plugged in as a fifth detector. We test Ours+NSS on the attacks on which our aggregator was outperformed by the competitors. In all the cases, Ours+NSS outperforms “Ours” either in terms of AUROC and FPR. In most of the cases, Ours+NSS is also better than the individual competitor. In Appendix C.2.1 we provide further insights on this by showing that the same behavior is observed when we plug a SOTA unsupervised
method as fifth detector in our pool.

## Evaluation of the proposed aggregator in the non-simultaneous setting

In these experiments, we move from the simultaneous adversarial attack scenario to one where the different detectors are aggregated to detect one single attack at a time, as usually done in the literature. We report the complete results Table 5.4. Crucially, these experiments show that ensemble detectors can also improve the performance for specific attacks. In particular, we would like to draw attention to the fact that we outperform NSS in the vast majority of the cases. Moreover, we achieve a maximum gain of 82.8 percentage points in terms of AUROC↑% (cf. SA attack) and 97.6 percentage points in terms of FPR↓$_{95\%}$% (cf. FGSM with $\varepsilon = 0.5$ attack). On the other side, the competitor outperforms our proposed method only in a few cases, achieving a maximum gain of 5.9 percentage points in terms of AUROC↑% and 27.4 percentage points in terms of FPR↓$_{95\%}$% (cf. FGSM with $\varepsilon=0.03125$ attack in both the cases), and these gains are much lower than those obtained by the proposed method.

## 5.3 Final remarks

We introduced a new method to tackle the multi-armed adversarial attacks introduced in MEAD Chapter 4. We formalized the multi-armed attack detection problem as a minimax cross-entropy risk and derived a surrogate loss function. Based on this, we characterized our optimal soft-detector, which results in a mixture of experts, as the solution to a minimax problem. Our empirical results show that aggregating simple detectors using our method consistently improves detection performance. The achieved performance is comparable and, in a large set of cases, better than the best state-of-the-art (SOTA) method in the multi-armed attack scenarios. Our method has two key benefits: it is modular, allowing existing and future methods to be integrated, and it is general, able to recognize adversarial examples from various attack algorithms and loss functions. Additionally, our aggregator can potentially be extended to aggregate both supervised and unsupervised SOTA adversarial detection methods.

Limitations of the proposed method come from the fact it relies on a collection of detectors whose expertise is combined to obtain a more robust adversarial
<table><caption>**Table 5.4: The proposed method and NSS in the non-simultaneous setting.**
The column names ACE, KL, FR, and Gini denote the loss function used to craft the attacks. HOP, DeepFool, CW2, and STA attacks have already been considered individually in Table 5.2.</caption><thead><tr><th rowspan="3"></th><th colspan="4">CIFAR10</th></tr><tr><th colspan="2">Ours AUROC†% (FPR↓95%)</th><th colspan="2">NSS AUROC†% (FPR↓95%)</th></tr><tr><th>ACE</th><th>KL</th><th>FR</th><th>Gini</th></tr></thead><tbody><tr><td>PGD1</td><td></td><td></td><td></td><td></td></tr><tr><td>ε = 5</td><td>66.2 (83.6) - 49.9 (93.5)</td><td>64.2 (85.7) - 49.6 (93.0)</td><td>63.0 (87.1) - 49.9 (93.3)</td><td>80.7 (58.4) - 50.3 (93.2)</td></tr><tr><td>ε = 10</td><td>62.6 (87.5) - 56.9 (88.4)</td><td>62.3 (88.2) - 56.6 (88.3)</td><td>63.1 (86.5) - 57.0 (88.1)</td><td>86.9 (46.0) - 57.1 (88.8)</td></tr><tr><td>ε = 15</td><td>74.2 (81.4) - 63.1 (83.0)</td><td>75.2 (80.6) - 62.8 (83.1)</td><td>75.3 (79.4) - 63.2 (82.5)</td><td>90.0 (31.1) - 63.5 (84.0)</td></tr><tr><td>ε = 20</td><td>86.8 (65.3) - 68.5 (77.1)</td><td>87.5 (63.1) - 68.1 (77.3)</td><td>86.9 (63.3) - 68.7 (76.4)</td><td>91.7 (31.2) - 69.9 (77.6)</td></tr><tr><td>ε = 25</td><td>93.9 (38.4) - 73.1 (71.1)</td><td>94.3 (36.2) - 72.7 (71.8)</td><td>93.7 (41.1) - 73.4 (70.9)</td><td>92.3 (28.9) - 75.0 (71.4)</td></tr><tr><td>ε = 30</td><td>97.1 (12.3) - 77.1 (64.5)</td><td>97.2 (12.6) - 76.8 (65.1)</td><td>96.8 (15.9) - 77.4 (65.2)</td><td>92.6 (27.9) - 78.6 (67.3)</td></tr><tr><td>ε = 40</td><td>98.9 (1.0) - 83.5 (52.7)</td><td>99.0 (1.0) - 83.3 (53.5)</td><td>98.8 (1.0) - 83.6 (52.7)</td><td>92.7 (27.4) - 80.1 (64.9)</td></tr><tr><td>PGD2</td><td></td><td></td><td></td><td></td></tr><tr><td>ε = .125</td><td>67.9 (81.1) - 49.5 (93.8)</td><td>65.4 (84.3) - 49.1 (93.5)</td><td>63.9 (86.6) - 49.6 (93.5)</td><td>80.6 (58.4) - 49.5 (94.3)</td></tr><tr><td>ε = .25</td><td>62.3 (87.5) - 55.9 (89.1)</td><td>62.1 (88.0) - 55.6 (89.2)</td><td>62.6 (87.6) - 55.8 (89.4)</td><td>86.7 (46.5) - 55.9 (89.8)</td></tr><tr><td>ε = .3125</td><td>66.5 (86.1) - 59.4 (86.5)</td><td>67.0 (85.9) - 59.0 (86.6)</td><td>67.8 (84.8) - 59.3 (86.6)</td><td>88.4 (42.2) - 59.3 (87.7)</td></tr><tr><td>ε = .5</td><td>86.4 (67.1) - 68.3 (77.4)</td><td>87.2 (64.5) - 68.0 (77.4)</td><td>86.7 (64.0) - 68.4 (77.2)</td><td>91.4 (31.4) - 69.0 (78.7)</td></tr><tr><td>ε = 1</td><td>98.9 (0.9) - 84.4 (50.6)</td><td>98.9 (0.9) - 84.3 (50.5)</td><td>98.8 (0.9) - 84.7 (50.7)</td><td>92.5 (27.2) - 79.3 (66.8)</td></tr><tr><td>ε = 1.5</td><td>99.2 (0.9) - 92.8 (28.7)</td><td>99.3 (0.9) - 92.7 (28.9)</td><td>99.3 (0.7) - 93.0 (27.3)</td><td>92.5 (27.2) - 79.5 (66.5)</td></tr><tr><td>ε = 2</td><td>99.3 (0.8) - 96.8 (13.9)</td><td>99.3 (0.8) - 96.9 (13.1)</td><td>99.3 (0.9) - 95.9 (17.2)</td><td>92.5 (27.2) - 79.5 (66.5)</td></tr><tr><td>PGD3</td><td></td><td></td><td></td><td></td></tr><tr><td>ε = .03125</td><td>99.1 (0.9) - 92.3 (31.0)</td><td>99.1 (0.9) - 92.1 (31.9)</td><td>99.0 (0.9) - 92.2 (30.7)</td><td>94.8 (21.5) - 89.0 (44.0)</td></tr><tr><td>ε = .0625</td><td>99.3 (0.8) - 99.1 (3.3)</td><td>99.3 (0.8) - 99.1 (3.3)</td><td>99.3 (0.8) - 99.1 (3.6)</td><td>97.4 (8.0) - 98.1 (8.1)</td></tr><tr><td>ε = .125</td><td>99.3 (0.7) - 99.7 (0.6)</td><td>99.3 (0.9) - 99.7 (0.6)</td><td>99.3 (0.8) - 99.6 (0.6)</td><td>97.3 (7.3) - 99.6 (0.6)</td></tr><tr><td>ε = .25</td><td>99.3 (0.7) - 99.7 (0.6)</td><td>99.3 (0.9) - 99.7 (0.6)</td><td>99.3 (0.8) - 99.7 (0.6)</td><td>97.1 (7.3) - 99.6 (0.6)</td></tr><tr><td>ε = .3125</td><td>99.3 (0.9) - 99.7 (0.6)</td><td>99.3 (0.8) - 99.7 (0.6)</td><td>99.3 (0.8) - 99.7 (0.6)</td><td>97.1 (7.4) - 99.7 (0.6)</td></tr><tr><td>ε = .5</td><td>99.3 (0.8) - 99.7 (0.6)</td><td>99.3 (0.8) - 99.7 (0.6)</td><td>99.3 (0.8) - 99.7 (0.6)</td><td>97.1 (7.3) - 99.6 (0.6)</td></tr><tr><td>FGSM</td><td></td><td></td><td></td><td></td></tr><tr><td>ε = .03125</td><td>89.2 (47.5) - 94.1 (26.7)</td><td>91.3 (40.6) - 94.0 (27.0)</td><td>92.6 (34.1) - 96.8 (15.0)</td><td>90.7 (42.7) - 96.6 (15.3)</td></tr><tr><td>ε = .0625</td><td>96.4 (18.5) - 99.4 (1.3)</td><td>96.2 (18.7) - 99.4 (1.4)</td><td>97.6 (10.3) - 99.6 (0.6)</td><td>97.4 (11.9) - 99.6 (0.6)</td></tr><tr><td>ε = .125</td><td>99.3 (3.4) - 99.7 (0.6)</td><td>99.1 (4.3) - 99.7 (0.6)</td><td>99.3 (2.5) - 99.5 (0.6)</td><td>99.3 (2.4) - 99.5 (0.6)</td></tr><tr><td>ε = .25</td><td>99.8 (0.6) - 99.7 (0.6)</td><td>99.7 (0.8) - 99.7 (0.6)</td><td>99.6 (1.1) - 97.9 (0.6)</td><td>99.6 (1.1) - 97.7 (0.6)</td></tr><tr><td>ε = .3125</td><td>99.7 (0.9) - 99.7 (0.6)</td><td>99.7 (0.9) - 99.7 (0.6)</td><td>99.5 (1.5) - 95.8 (0.6)</td><td>99.5 (1.5) - 95.6 (0.6)</td></tr><tr><td>ε = .5</td><td>99.0 (4.9) - 99.7 (0.6)</td><td>99.2 (2.7) - 99.7 (0.6)</td><td>99.2 (2.4) - 84.9 (100.0)</td><td>99.2 (2.4) - 84.8 (100.0)</td></tr><tr><td>BIM</td><td></td><td></td><td></td><td></td></tr><tr><td>ε = .03125</td><td>98.3 (4.6) - 90.3 (37.7)</td><td>98.3 (4.4) - 90.2 (38.1)</td><td>97.8 (7.2) - 90.5 (37.0)</td><td>92.2 (32.6) - 88.2 (45.1)</td></tr><tr><td>ε = .0625</td><td>99.4 (0.8) - 98.2 (7.5)</td><td>99.4 (0.9) - 98.2 (7.5)</td><td>99.4 (0.8) - 98.3 (7.3)</td><td>96.6 (13.1) - 97.3 (12.9)</td></tr><tr><td>ε = .125</td><td>99.3 (0.9) - 99.6 (0.7)</td><td>99.3 (0.9) - 99.7 (0.7)</td><td>99.3 (0.8) - 99.6 (0.7)</td><td>97.8 (6.9) - 99.3 (1.9)</td></tr><tr><td>ε = .25</td><td>99.3 (0.8) - 99.7 (0.6)</td><td>99.3 (0.9) - 99.7 (0.6)</td><td>99.3 (0.8) - 99.7 (0.6)</td><td>97.4 (7.2) - 99.6 (0.6)</td></tr><tr><td>ε = .3125</td><td>99.3 (0.9) - 99.7 (0.6)</td><td>99.3 (0.8) - 99.7 (0.6)</td><td>99.3 (0.9) - 99.7 (0.6)</td><td>97.1 (7.4) - 99.7 (0.6)</td></tr><tr><td>ε = .5</td><td>99.3 (0.8) - 99.7 (0.6)</td><td>99.3 (0.8) - 99.7 (0.6)</td><td>99.3 (0.8) - 99.7 (0.6)</td><td>96.3 (7.3) - 99.7 (0.6)</td></tr><tr><td>SA</td><td></td><td></td><td></td><td></td></tr><tr><td>ε = .125</td><td>91.2 (39.6) - 9.4 (99.9)</td><td>91.2 (39.6) - 9.4 (99.9)</td><td>91.2 (39.6) - 9.4 (99.9)</td><td>91.2 (39.6) - 9.4 (99.9)</td></tr><tr><td>CWi</td><td></td><td></td><td></td><td></td></tr><tr><td>ε = .3125</td><td>80.7 (60.8) - 64.6 (89.8)</td><td>80.7 (60.8) - 64.6 (89.8)</td><td>80.7 (60.8) - 64.6 (89.8)</td><td>80.7 (60.8) - 64.6 (89.8)</td></tr></tbody></table>
detection. Such models could be potentially poisoned by a malicious actor, drastically reducing the aggregator's reliability. We think this could have a potentially severe societal impact if the proposed method happened to be deployed with no additional checks on the quality of the available detectors.
Conclusion of the Thesis

In this thesis, we have investigated how to formulate security problems as binary
hypothesis testing.

We first addressed the problem of *misclassification detection*. Given an input
sample and a pre-trained classifier, we want to understand whether the input
sample comes from the distribution of the correctly classified samples or the
incorrectly classified ones. Based on the soft-probabilities associated with the
example, the detector approximates the probability of the classifier outputting
the wrong class. The prediction will be rejected if the computed score exceeds
a certain threshold. Through simulations on image and text datasets, we have
demonstrated the superiority of the proposed detector over the SOTA techniques.
Furthermore, we distinguished two scenarios: the Totally Black-Box (TBB) sce-
nario, in which the detector has access only to the final soft-probabilities, and
the Partially Black-Box (PBB) scenario, in which the detector has access to the
logits and can perform input pre-processing.

We have then moved on to the problem of *multi-armed adversarial attack detection*. In this case, the goal of the detector is to check whether the input sample is natural or has been adversarially perturbed according to *some* strategy. We refer to this setting as ‘multi-armed’ as in the classical (‘single-armed’) detection setting the methods are generally validated by assuming a single attack strategy at a time. We formalize the problem as a minimax cross-entropy risk. Based on this formulation, we derive a surrogate loss function and use it to characterize our optimal soft-detector leading to our aggregator of detectors’ decisions. Experimentally, we have shown: (i) the classical framework led to an overopti-
mistic assessment of the detectors' performance; (ii) the proposed aggregator of
detectors' decisions represents a valid (and the first) solution to the multi-armed
setting.

The line of research we have pursued in this work offers many opportunities
for future work. For instance, as to the misclassification detection topic, we are
thoroughly studying how our framework adapts to the case of image segmen-
tation. In this context, the detector is asked to detect errors in the predicted
mask outputs from the pre-trained model. The analysis can be conducted either
by pixel or region. Interestingly, the results so far indicate that the proposed
detector can recognize when a region should be segmented but is not present in
the predicted mask. In particular, we are dealing with a critical safety system
like the one of medical diagnosis by performing simulations on datasets such as
Automated Cardiac Diagnosis Challenge [BLZ+18], and Brain Tumor Segmenta-
tion Challenge [MJB+15], to cite a few. Moreover, a straightforward extension of
the proposed method would be for regression tasks. Currently, the detector we
presented relies on the soft probability outputs from the classifier. Therefore, it
can not be used as it is also with regression models. A possibility in this sense
would be to quantize the output space and treat each of the bins into which the
space is divided as a possible class. Anyway, one issue in this sense would be how
to assign to each of the bins a probability.

A more ambitious goal would be to apply our detector aggregator to topics
beyond simultaneous adversarial attack detection. As long as the detector outputs
can be interpreted as a probability distribution across two categories, any existing
or future supervised or unsupervised method can be combined using our proposed
approach, making the aggregator a new ensemble technique. An example of this
extension is intrusion detection, where an improved detection framework is highly
desired, particularly with the use of ensemble learners [TL21].
Bibliography

[ACFH20] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: A query-efficient black-box adversarial attack via random search. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, *Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII*, volume 12368 of *Lecture Notes in Computer Science*, pages 484-501. Springer, 2020.

[ACW18] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In *International conference on machine learning*, pages 274-283. PMLR, 2018.

[AF21] Sahar Abdelnabi and Mario Fritz. What's in the box: Deflecting adversarial attacks by randomly deploying adversarially-disjoint models. In Trent Jaeger and Zhiyun Qian, editors, *MTD@CCS 2021: Proceedings of the 8th ACM Workshop on Moving Target Defense, Virtual Event, Republic of Korea, 15 November 2021*, pages 3-12. ACM, 2021.

[AHFD22] Ahmed Aldahdooh, Wassim Hamidouche, Sid Ahmed Fezza, and Olivier Deforges. Adversarial example detection for dnn models: A review and experimental comparison. *Artificial Intelligence Review*, 2022.

[Asm03] *Steady-State Properties of GI/G/1*, pages 266-301. Springer New York, New York, NY, 2003.
[BGMS17] Michael Backes, Manuel Gomez-Rodriguez, Praveen Manoharan, and Bartlomiej Surma. Reconciling privacy and utility in continuous-time diffusion networks. In *30th IEEE Computer Security Foundations Symposium, CSF 2017, Santa Barbara, CA, USA, August 21-25, 2017*, pages 292-304. IEEE Computer Society, 2017.

[BHP$^{+}$21] Oliver Bryniarski, Nabeel Hingun, Pedro Pachuca, Vincent Wang, and Nicholas Carlini. Evading adversarial example detection defenses with orthogonal projected gradient descent. *CoRR*, abs/2106.15023, 2021.

[BLZ$^{+}$18] Olivier Bernard, Alain Lalande, Clément Zotti, Frederic Cervenansky, Xin Yang, Pheng-Ann Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Ángel González Ballester, Gerard Sorroma, Sandy Napel, Steffen E. Petersen, Georgios Tziritas, Elias Grinias, Mahendra Khened, Alex Varghese, Ganapathy Krishnamurthi, Marc-Michel Rohé, Xavier Pennec, Maxime Sermesant, Fabian Isensee, Paul Jaeger, Klaus H. Maier-Hein, Peter M. Full, Ivo Wolf, Sandy Engelhardt, Christian F. Baumgartner, Lisa M. Koch, Jelmer M. Wolterink, Ivana Isgum, Yeonggul Jang, Yoonmi Hong, Jay Patravali, Shubham Jain, Olivier Humbert, and Pierre-Marc Jodoin. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: Is the problem solved? *IEEE Trans. Medical Imaging*, 37(11):2514-2525, 2018.

[BRY98] Andrew R. Barron, Jorma Rissanen, and Bin Yu. The minimum description length principle in coding and modeling. *IEEE Trans. Inf. Theory*, 44(6):2743-2760, 1998.

[CBB19] Francesco Crecchi, Davide Bacciu, and Battista Biggio. Detecting black-box adversarial examples through nonlinear dimensionality reduction. In *27th European Symposium on Artificial Neural Networks, ESANN 2019, Bruges, Belgium, April 24-26, 2019*, 2019.

[CH20] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In
*Proceedings of the 37th International Conference on Machine Learning*, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of *Proceedings of Machine Learning Research*, pages 2206-2216. PMLR, 2020.

[CJW20] Jianbo Chen, Michael I. Jordan, and Martin J. Wainwright. Hop-skipjumpattack: A query-efficient decision-based attack. In *2020 IEEE Symposium on Security and Privacy, SP 2020, San Francisco, CA, USA, May 18-21, 2020*, pages 1277-1294. IEEE, 2020.

[CW17a] Nicholas Carlini and David A. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Bhavani Thuraisingham, Battista Biggio, David Mandell Freeman, Brad Miller, and Arunesh Sinha, editors, *Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AISec@CCS 2017, Dallas, TX, USA, November 3, 2017*, pages 3-14. ACM, 2017.

[CW17b] Nicholas Carlini and David A. Wagner. Magnet and "efficient defenses against adversarial attacks" are not robust to adversarial examples. CoRR, abs/1711.08478, 2017.

[CW17c] Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In *2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017*, pages 39-57. IEEE Computer Society, 2017.

[DCLT19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)*, pages 4171-4186, 2019.

[DG06a] Jesse Davis and Mark Goadrich. The relationship between precision-recall and ROC curves. In *Machine Learning, Proceedings of the Twenty-Third International Conference (ICML 2006), Pittsburgh,*
Pennsylvania, USA, June 25-29, 2006, volume 148, pages 233-240, 2006.

[DG06b] Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In *Proceedings of the 23rd international conference on Machine learning*, pages 233-240, 2006.

[DT18] Terrance DeVries and Graham W. Taylor. Learning confidence for out-of-distribution detection in neural networks. *CoRR*, abs/1802.04865, 2018.

[ETT$^{+}$19] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, *Proceedings of the 36th International Conference on Machine Learning*, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of *Proceedings of Machine Learning Research*, pages 1802-1811. PMLR, 2019.

[FCSG17] Reuben Feinman, Ryan R. Curtin, Saurabh Shintre, and Andrew B. Gardner. Detecting adversarial samples from artifacts. *CoRR*, abs/1703.00410, 2017.

[GE17] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Ben-gio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, *Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA*, pages 4878-4887, 2017.

[GE19] Yonatan Geifman and Ran El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In *Proceedings of the 36th International Conference on Machine Learning*, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97, pages 2151-2159, 2019.
[GGP21] Federica Granese, Daniele Gorla, and Catuscia Palamidessi. Enhanced models for privacy and utility in continuous-time diffusion networks. *Int. J. Inf. Sec.*, 20(5):763–782, 2021.

[GJG$^{+22}$] Daniele Gorla, Louis Jalouzot, Federica Granese, Catuscia Palamidessi, and Pablo Piantanida. On the (im)possibility of estimating various notions of differential privacy. *CoRR*, abs/2208.14414, 2022.

[GKS21] Aditya Gangrade, Anil Kag, and Venkatesh Saligrama. Selective classification via one-sided prediction. In Arindam Banerjee and Kenji Fukumizu, editors, *The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event*, volume 130 of *Proceedings of Machine Learning Research*, pages 2179–2187. PMLR, 2021.

[GPR$^{+22}$] Federica Granese, Marine Picot, Marco Romanelli, Francisco Messina, and Pablo Piantanida. MEAD: A multi-armed approach for evaluation of adversarial examples detectors. *CoRR*, abs/2206.15415, 2022.

[GPSW17] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In *Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017*, volume 70, pages 1321–1330, 2017.

[GRG$^{+21}$] Federica Granese, Marco Romanelli, Daniele Gorla, Catuscia Palamidessi, and Pablo Piantanida. DOCTOR: A simple method for detecting misclassification errors. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, *Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual*, pages 5669–5681, 2021.
[GRGP23] Federica Granese, Marco Romanelli, Siddharth Garg, and Pablo Piantanida. A minimax approach against multi-armed adversarial attacks detection, 2023.

[GSS15] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun, editors, *3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings*, 2015.

[HAB19] Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In *IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019*, pages 41-50. Computer Vision Foundation / IEEE, 2019.

[HCSS21] Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ML safety. *CoRR*, abs/2109.13916, 2021.

[HLW16] Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. *CoRR*, abs/1608.06993, 2016.

[HMD19] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In *International Conference on Learning Representations*, 2019.

[HPS16] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, *Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain*, pages 3315-3323, 2016.

[HSJK20] Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized ODIN: detecting out-of-distribution image without learning
from out-of-distribution data. In *2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020*, Seattle, WA, USA, June 13-19, 2020, pages 10948-10957, 2020.

[HVD15] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In *NIPS Deep Learning and Representation Learning Workshop*, 2015.

[HZRS16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In *CVPR*, pages 770-778, 2016.

[IST+19] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, *Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada*, pages 125-136, 2019.

[JWX17] Qixiang Zhang, Jiayu Wu and Guoxi Xu. Tiny imagenet challenge. Technical report, 2017.

[KE16] Volodymyr Kuleshov and Stefano Ermon. Reliable confidence estimation via online learning. *CoRR*, abs/1607.03594, 2016.

[KFHD20] Anouar Kherchouche, Sid Ahmed Fezza, Wassim Hamidouche, and Olivier Déforges. Detection of adversarial examples in deep neural networks with natural scene statistics. In *2020 International Joint Conference on Neural Networks, IJCNN 2020, Glasgow, United Kingdom, July 19-24, 2020*, pages 1-7. IEEE, 2020.

[KGB] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In *5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings*.