SlowGuess/NeurIPS_2023_OCR huggingface dataset:
On the Role of Randomization in Adversarily Robust Classification
Lucas Gnecco Heredia $^{1*}$  Muni Sreenivas Pydi $^{1}$  Laurent Meunier $^{2}$  Benjamin Negrevergne $^{1}$  Yann Chevalyre $^{1}$
$^{1}$  CNRS, LAMSADE, Université Paris Dauphine - PSL  $^{2}$  Payflows {lucas.gnecco-heredia,muni.pydi}@dauphine.psl.eu {benjamin.negrevergne,yann.chevaleyre}@lamsade.dauphine.fr laurent@payflows.io
Deep neural networks are known to be vulnerable to small adversarial perturbations in test data. To defend against adversarial attacks, probabilistic classifiers have been proposed as an alternative to deterministic ones. However, literature has conflicting findings on the effectiveness of probabilistic classifiers in comparison to deterministic ones. In this paper, we clarify the role of randomization in building adversarially robust classifiers. Given a base hypothesis set of deterministic classifiers, we show the conditions under which a randomized ensemble outperforms the hypothesis set in adversarial risk, extending previous results. Additionally, we show that for any probabilistic binary classifier (including randomized ensembles), there exists a deterministic classifier that outperforms it. Finally, we give an explicit description of the deterministic hypothesis set that contains such a deterministic classifier for many types of commonly used probabilistic classifiers, i.e. randomized ensembles and parametric/input noise injection.
Modern machine learning algorithms such as neural networks are highly sensitive to small, imperceptible adversarial perturbations of inputs [3, 43]. In the last decade, there has been a back-and-forth in research progress between developing increasingly potent attacks [5, 10, 19, 25] and the creation of practical defense mechanisms through empirical design [8, 30, 32]. In particular, probabilistic classifiers (also called stochastic classifiers or randomized classifiers) have been widely used to build strong defenses which can be roughly categorized into two groups: noise injection techniques [14, 28, 34, 37, 47, 48] and randomized ensembles [31, 38]. The first group includes methods that add noise at inference, usually to the input [22, 48], an intermediate layer activation [22, 47, 51] or the weights of a parametric model [22] like a neural network. Most of this work is experimental, with some theoretical papers that try to justify the use of such methods [37]. The second group, inspired by game theory, create a mixed strategy over base classifiers as a finite mixture [31, 38]. The argument behind this kind of models is that it becomes harder for an attacker to fool multiple models at the same time [12, 36].
Intuitively, the greater flexibility of probabilistic classifiers implies that they should outperform their deterministic counterparts in adversarially robust classification. For instance, one of the earliest works using randomization to improve robustness to adversarial attacks [48] claims that "randomization at inference time makes the network much more robust to adversarial images". Dhillon et al. [14] propose a pruning method that "is stochastic and has more freedom to deceive the adversary".
Similarly, Panousis et al. [34] claim that for their method, "the stochastic nature of the proposed activation significantly contributes to the robustness of the model".
Lending support to the intuition that probabilistic classifiers are better than their deterministic counterparts, Pinot et al. [38] prove the non-existence of a Nash equilibrium in the zero-sum game between an adversary and a classifier when both use deterministic strategies. Moreover, they show that a randomized ensemble outperforms any deterministic classifier under a regularized adversary. Meunier et al. [31] shows that a mixed Nash equilibrium does exist in the adversarial game when the classifier is allowed to use randomized strategies, making a strong case for probabilistic classifiers.
In contrast to the above results advocating for probabilistic classifiers, Pydi and Jog [40] use an equivalence between adversarial risk minimization and a certain optimal transport problem between data generating distributions to show that in the case of binary classification there exists an approximate pure Nash equilibrium in the adversarial game, showing that there always exists a sequence of deterministic classifiers that are optimal for adversarial risk minimization. Trillos et al. [45] extend their results to multiclass classification using a multimarginal optimal transport formulation. More recently, Awasthi et al. [1, 2] prove the existence of a deterministic classifier that attains the optimal value for adversarial risk under mild assumptions. Trillos et al. [44] extends it to the multiclass setting.
How does one reconcile these apparently conflicting theoretical findings as a machine learning practitioner? First, it is important to note that the above theoretical results are valid when considering large hypothesis sets, like that of all possible measurable classifiers. In practice, one works with more limited hypotheses sets such as the space of linear classifiers or the space of classifiers that can be learned by an  $L$ -layered deep neural network, for which the above theoretical results do not directly apply. Second, the task of training a robust probabilistic classifier is non-trivial [12, 39]. Even the task of efficiently attacking a probabilistic classifier is not straightforward [6, 12, 13, 36]. Hence, it is important to understand the precise role of randomization in adversarial robustness and the conditions under which probabilistic classifiers offer a discernible advantage over deterministic classifiers. In this work, we address this issue by making the following contributions.
- From deterministic to probabilistic classifiers: Given a base hypothesis set (BHS)  $\mathcal{H}_b$  of deterministic classifiers, we prove the necessary and sufficient conditions for a probabilistic classifier built from  $\mathcal{H}_b$  to strictly outperform the best classifier in  $\mathcal{H}_b$  in adversarial risk. A quantity that plays a crucial role in our analysis is the expected matching penny gap wherein there exist classifiers that are vulnerable to adversarial perturbation at a point, but not all at the same time. These results are contained in Section 3.
- From probabilistic to deterministic classifiers: In the binary classification setting, given a probabilistic classifier  $\mathbf{h}$ , we show that there always exists a deterministic classifier  $h$  belonging to a "threshold" hypothesis set (THS) that is built from  $\mathbf{h}$  in a straightforward way. We then apply our results to common families of probabilistic classifiers, leading to two important conclusions: 1) For every randomized ensemble classifier [13, 38], there exists a deterministic weighted ensemble [17, 52] with better adversarial risk. 2) For every input noise injection classifier [22, 37, 39], there exists a deterministic classifier that is a slight generalization of randomized smoothing classifier [8, 26] with better adversarial risk. These results are contained in Section 4.
- Families of classifiers for which randomization does not help: Given a BHS  $\mathcal{H}_b$ , we show the conditions under which randomizing over classifiers in  $\mathcal{H}_b$  does not help to improve adversarial risk. Specifically, if the decision regions of  $\mathcal{H}_b$  are closed under union and intersection, our results imply that randomized ensembles built using  $\mathcal{H}_b$  offer no advantage over the deterministic classifiers in  $\mathcal{H}_b$ . These results are contained in Section 5.
Notation: We use  $\mathbb{1}\{C\}$  to denote an indicator function which takes a value 1 if the proposition  $C$  is true, and 0 otherwise. For an input space  $\mathcal{X} \subseteq \mathbb{R}^d$ , we use  $\sigma(\mathcal{X})$  to denote some  $\sigma$ -algebra over  $\mathcal{X}$  that makes  $(\mathcal{X}, \sigma(\mathcal{X}))$  a measurable space. If not specified,  $\sigma(\mathcal{X})$  will be the Borel  $\sigma$ -algebra, which is generated by the open sets of  $\mathcal{X}$ . We use  $\mathcal{P}(\mathcal{X})$  to denote the set of probability measures over the measure space  $(\mathcal{X}, \sigma(\mathcal{X}))$ . For a positive integer  $K$ , we use  $[K]$  to denote the range  $\{1, \ldots, K\}$ . For a vector  $u \in R^K$ , we denote  $u_i$  the  $i$ th component of  $u$ .
2.1 Adversarily Robust Classification
We consider a classification setting with input space  $\mathcal{X} \subseteq \mathbb{R}^d$  and a finite label space  $\mathcal{V}$ . The space  $\mathcal{X}$  is equipped with some norm  $\|\cdot\|$ , which is commonly set to be the  $\ell_2$  or  $\ell_{\infty}$  norms. Let  $\rho \in \mathcal{P}(\mathcal{X} \times \mathcal{V})$  denote the true data distribution, which can be factored as  $\rho(x,y) = \nu(y)\rho_y(x)$ , where  $\nu \in \mathcal{P}(\mathcal{V})$  denotes the marginal probability distribution of  $\rho$  over  $\mathcal{V}$  and  $\rho_y \in \mathcal{P}(\mathcal{X})$  denotes the conditional probability distribution of the data over  $\mathcal{X}$  given class  $y$ .
A deterministic classifier is a function  $h: \mathcal{X} \to \mathcal{Y}$  that maps each  $x \in \mathcal{X}$  to a fixed label in  $\mathcal{Y}$ . The 0-1 loss of  $h$  on a point  $(x, y) \in \mathcal{X} \times \mathcal{Y}$  is given by,  $\ell^{0-1}((x, y), h) = \mathbb{1}\{h(x) \neq y\}$ .
A probabilistic classifier is a function  $\mathbf{h}:\mathcal{X}\to \mathcal{P}(\mathcal{Y})$  that maps each  $x\in \mathcal{X}$  to a probability distribution over  $\mathcal{V}$ . To label  $x\in \mathcal{X}$  with  $\mathbf{h}$ , one samples a random label from the distribution  $\mathbf{h}(x)\in \mathcal{P}(\mathcal{V})$ . In practice, if  $\mathcal{V}$  consists of  $K$  classes,  $\mathcal{P}(\mathcal{V})$  is identifiable with the  $K$ -simplex  $\Delta^K$  that consist of vectors  $u\in \mathbb{R}^K$  such that  $\sum_{i = 1}^{K}u_{i} = 1$ . Therefore, one can think of  $\mathbf{h}(x)$  as a probability vector for every  $x$ .
The 0-1 loss of  $\mathbf{h}$  on  $(x,y)\in \mathcal{X}\times \mathcal{Y}$  is given by,  $\ell^{0 - 1}((x,y),\mathbf{h}) = \mathbb{E}_{\hat{y}\sim \mathbf{h}(x)}[\mathbb{1}\{\hat{y}\neq y\}] = 1 - \mathbf{h}(x)_y$  Note that  $\ell^{0 - 1}((x,y),\mathbf{h})\in [0,1]$  is a generalization of the classical 0-1 loss for deterministic classifiers, which can only take values in  $\{0,1\}$
Given  $x \in \mathcal{X}$ , we consider a data perturbing adversary of budget  $\epsilon \geq 0$  that can transport  $x$  to  $x' \in B_{\epsilon}(x) = \{x' \in \mathcal{X} \mid d(x, x') \leq \epsilon\}$ , where  $B_{\epsilon}(x)$  is the closed ball of radius  $\epsilon$  around  $x$ . The adversarial risk of a probabilistic classifier  $\mathbf{h}$  is defined as follows.
$$
\mathcal {R} _ {\epsilon} (\mathbf {h}) = \underset {y \sim \nu} {\mathbb {E}} \left[ \mathcal {R} _ {\epsilon} ^ {y} (\mathbf {h}) \right] = \underset {y \sim \nu} {\mathbb {E}} \underset {x \sim p _ {y}} {\mathbb {E}} \left[ \sup  _ {x ^ {\prime} \in B _ {\epsilon} (x)} \ell^ {0 - 1} \left(\left(x ^ {\prime}, y\right), \mathbf {h}\right) \right] ^ {1}. \tag {1}
$$
The adversarial risk of a deterministic classifier  $h$  is defined analogously, replacing  $\mathbf{h}$  by  $h$  in (1).
2.2 Threat model
We consider a white-box threat model, in which the adversary has complete access to the classifier proposed by the defender. In theory, we are assuming that the adversary is able to attain (or approximate to any precision) the inner supremum in (1). In practice, if the classifier is differentiable, the adversary will have full access to the gradients (and any derivative of any order), and it will have full knowledge of the classifier, including its architecture and any preprocessing used. The adversary can also query the classifier as many times as needed. The only limitation for the adversary regards the sources of randomness. In particular, when the proposed classifier is probabilistic and an inference pass is done by sampling a label from  $\mathbf{h}(x)$ , the adversary does not know and cannot manipulate the randomness behind this sampling.
It is worth noting that other works have considered similar inference processes, in which the label of a point is obtained by a sampling procedure [29]. The underlying question is similar to ours: does randomness improve adversarial robustness?. They propose a threat model in which an attacker can perform  $N$  independent attempts with the goal of producing at least one misclassification, which becomes easier if the predicted probability for the correct class is less than 1. In this paper, however, we are interested in the expected accuracy of a classifier from a theoretical point of view, which in practice translates to the probability of error, or the long term average performance or the classifier.
Our formulation of adversarial risk in (1) corresponds to the "constant-in-the-ball" risk in [20] or the "corrupted-instance" risk in [15]. Under this formulation, as pointed out in [40], an adversarial risk of 0 is only possible if the supports of the class-conditioned distributions  $\rho_{y}$  are non-overlapping and separated by at least  $2\epsilon$ . This is not the case with other formulations of risk introduced in [15, 20]. We focus on the former in this work.
<sup>1</sup>The measurability of the 0-1 loss under attack (inner part of (1)) is non-trivial and depends on various factors like the type of ball considered for the supremum (closed or open) [44], and the underlying  $\sigma$ -algebra [1, 40]. In Section 2.3 and Appendix A we will clarify the assumptions that ensure the well-definedness of the adversarial risk in our setting.
2.3 Probabilistic Classifiers Built from a Base Hypothesis Set
In this paper, we study probabilistic classifiers that are constructed from a base hypothesis set (BHS) of (possibly infinitely many) deterministic classifiers, denoted by  $\mathcal{H}_b$ . We use the name mixtures [38] for these type of classifiers. In the following, we show that many of the probabilistic classifiers that are commonly used in practice fall under this framework.
Let  $\mathcal{H}_b$  be a BHS of deterministic classifiers from  $\mathcal{X}$  to  $\mathcal{Y}$ , which we assume is identifiable with a Borel subset of some  $\mathbb{R}^p$ . Let  $\mu \in \mathcal{P}(\mathcal{H}_b)$  be a probability measure over the measure space  $(\mathcal{H}_b, \sigma(\mathcal{H}_b))$ , where  $\sigma(\mathcal{H}_b)$  denotes the Borel  $\sigma$ -algebra on  $\mathcal{H}_b$ . One can construct a probabilistic classifier  $\mathbf{h}_{\mu}: \mathcal{X} \to \mathcal{P}(\mathcal{Y})$ , built from  $\mathcal{H}_b$ , that maps  $x \in \mathcal{X}$  to a probability distribution  $\mu_x \in \mathcal{P}(\mathcal{Y})$ , where  $\mu_x(y) = \mathbb{P}_{h \sim \mu}(h(x) = y)$ . We now instantiate  $\mathcal{H}_b$  and  $\mu \in \mathcal{P}(\mathcal{H}_b)$  for two main types of probabilistic classifiers that are commonly used in practice: randomized ensembles and noise injection classifiers.
For a randomized ensemble classifier (REC),  $\mu \in \mathcal{P}_M(\mathcal{H}_b) \subset \mathcal{P}(\mathcal{H}_b)$  where  $\mathcal{P}_M(\mathcal{H}_b)$  is the set of all discrete measures on  $\mathcal{H}_b$  supported on a finite set of  $M$  deterministic classifiers,  $\{h_1, \ldots, h_M\}$ . In this case,  $\mathbf{h}_{\mu}(x)$  takes the value  $h_m(x)$  with probability  $p_m = \mu(h_m)$  for  $m \in [M]$ , where  $\sum_{m \in [M]} p_m = 1$ . RECs were introduced in [38] and play the role of mixed strategies in the adversarial robustness game. They are a simple randomization scheme when a finite number of classifiers are at hand, and both training and attacking them represent a challenge [12, 13].
For a weight-noise injection classifier (WNC),  $\mathcal{H}_b = \{h_w : w \in \mathcal{W}\}$  where  $h_w$  is a deterministic classifier with parameter  $w$ . In this case,  $\mu$  is taken to be a probability distribution over  $\mathcal{W}$  with the understanding that each  $w \in \mathcal{W}$  is associated with a unique  $h_w \in \mathcal{H}_b$ . For example,  $\mathcal{H}_b$  can be the set of all neural network classifiers with weights in the set  $\mathcal{W} \subseteq \mathbb{R}^p$ . Any probability distribution  $\mu$  on the space of weights  $\mathcal{W}$  results in a probabilistic classifier  $\mathbf{h}_{\mu}$ . Alternatively,  $\mathcal{H}_b$  can be generated by injecting noise  $z$  sampled from a distribution  $\mu$  to the weights  $w_0$  of a fixed classifier  $h_{w_0}$ . In this case, the probabilistic classifier  $\mathbf{h}_{\mu}$  maps  $x \in \mathcal{X}$  to a probability distribution  $\mu_x \in \mathcal{P}(\mathcal{Y})$ , where  $\mu_x(y) = \mathbb{P}_{z \sim \mu}(h_{w_0 + z}(x) = y)$ . Weight noise injection has been explicitly used in [22], but there are many other approaches that implicitly define a probability distribution over the parameters of a model [14, 35] and sample one or more at inference.
For an input-noise injection classifier (INC),  $\mathcal{H}_b = \{h_\eta : \eta \in \mathcal{X}\}$  where  $h_\eta(x) = h_0(x + \eta)$  for a fixed deterministic classifier  $h_0$ . In this case,  $\mu$  is taken to be a probability distribution over  $\mathcal{X}$  (which is unrelated to the data generating distribution), and  $\mathbf{h}_{\mu}$  maps  $x \in \mathcal{X}$  to a probability distribution  $\mu_x \in \mathcal{P}(\mathcal{Y})$ , where  $\mu_x(y) = \mathbb{P}_{\eta \sim \mu}(h(x + \eta) = y)$ . Injecting noise to the input has been used for decades as a regularization method [4, 21]. INCs are studied in [22, 37, 39] as a defense against adversarial attacks, and are closely related to randomized smoothing classifiers [8, 26].
Remark 1 (Measurability). To ensure the well-definedness of the adversarial risk, we need to ensure that the 0-1 loss under attack (inner part of (1)) is measurable. The 0-1 loss for a fixed class  $y \in \mathcal{Y}$  can now be seen as a function from the product space  $\mathcal{X} \times \mathcal{H}_b$  to  $\{0,1\}$ :
$$
f (x, h) = \ell^ {0 - 1} ((x, y), h) = \mathbb {1} \{h (x) \neq y \} \tag {2}
$$
For a distribution  $\mu \in \mathcal{P}(\mathcal{H}_b)$  and the associated probabilistic classifier  $\mathbf{h}_{\mu}$ , we can rewrite the 0-1 loss of  $\mathbf{h}_{\mu}$  at  $x$  as
$$
\ell^ {0 - 1} ((\cdot , y), \mathbf {h} _ {\mu}) = \underset {h \sim \mu} {\mathbb {E}} [ \mathbb {1} \{h (\cdot) \neq y \} ] = \int_ {\mathcal {H} _ {b}} f (\cdot , h) d \mu (h) \tag {3}
$$
Note that if  $f$  is Borel measurable in  $\mathcal{X} \times \mathcal{H}_b$ , then by Fubini-Tonelli's Theorem, the 0-1 loss as a function of  $x$  with an integral over  $\mathcal{H}_b$  shown in (3) is Borel measurable. By [16, Appendix A, Theorem 27], the 0-1 loss under attack is then universally measurable and therefore the adversarial risk, which requires the integral over  $\mathcal{X}$  of the loss under attack, is well-defined over this universal  $\sigma$ -algebra [44, Definition 2.7]. Recent work by Trillos et al. [44] have also shown the existence of Borel measurable solutions for the closed ball formulation of adversarial risk.
In this work, we assume that the function  $f$  in (2) is Borel measurable in  $\mathcal{X} \times \mathcal{H}_b$  for every  $y$ , so that (1) is well-defined. This assumption is already satisfied in the settings that are of interest for our work. As an example, it holds when the classifiers at hand are neural networks with continuous activation functions. More details on Appendix A. For a deeper study on the measurability and well-definedness of the adversarial risk in different threat models and settings, see [1, 16, 40, 44].
3 From Deterministic to Probabilistic Classifiers
Let  $\mathcal{H}_b$  be a class of deterministic classifiers. In this section, we compare the robustness of probabilistic classifiers built upon  $\mathcal{H}_b$  with the robustness of the class  $\mathcal{H}_b$  itself. Note that if we consider the trivial mixtures  $\mu = \delta_h$  for  $h\in \mathcal{H}_b$ , we obtain the original classifiers in  $\mathcal{H}_b$ , so it is always true that  $\inf_{\mu \in \mathcal{P}(\mathcal{H}_b)}\mathcal{R}_{\epsilon}(\mathbf{h}_{\mu})\leq \inf_{h\in \mathcal{H}_b}\mathcal{R}_{\epsilon}(h)$ . We are interested in the situations in which this gap is strict, meaning that mixtures strictly outperform the base classifiers. With this in mind, we introduce the notion of Matching Penny Gap to quantify the improvement of probabilistic classifiers over deterministic ones.
Theoretical Results on Robustness of Probabilistic Classifiers. We begin by presenting a theorem which shows that the adversarial risk of a probabilistic classifier is at most the average of the adversarial risk of the deterministic classifiers constituting it. The proof of this result will be a first step towards understanding the conditions that favor the mixture classifier.
Theorem 3.1. For a probabilistic classifier  $\mathbf{h}_{\mu}:\mathcal{X}\to \mathcal{P}(\mathcal{Y})$  constructed from a BHS  $\mathcal{H}_b$  using any  $\mu \in \mathcal{P}(\mathcal{H}_b)$ , we have  $\mathcal{R}_{\epsilon}(\mathbf{h}_{\mu})\leq \mathbb{E}_{h\sim \mu}\left[\mathcal{R}_{\epsilon}(h)\right]$ .
A natural follow-up question to is to ask what conditions guarantee a strictly better performance of a probabilistic classifier. We know that this gap can be strict, as can be seen in toy examples like the one shown in Figure 1b where  $\mathcal{H}_b$  is the set of all linear classifiers, and there exist two distinct classifiers  $f_{1}, f_{2}$  both attaining the minimum adversarial risk among all classifiers in  $\mathcal{H}_b$ . Any non-degenerate mixture of  $f_{1}, f_{2}$  attains a strictly better adversarial risk, demonstrating a strict advantage for probabilistic classifiers.
From the proof of Theorem 3.1 it is clear that a strict gap in performance between probabilistic and deterministic classifiers is only possible when the following strict inequality holds for a non-vanishing probability mass over  $(x,y)$ .
$$
\sup  _ {x ^ {\prime} \in B _ {e} (x)} \mathbb {E} _ {h \sim \mu} [ \mathbb {1} \left\{h \left(x ^ {\prime}\right) \neq y \right\} ] <   \mathbb {E} _ {h \sim \mu} \left[ \sup  _ {x ^ {\prime} \in B _ {e} (x)} \mathbb {1} \left\{h \left(x ^ {\prime}\right) \neq y \right\} \right]. \tag {4}
$$
The above condition holds at  $(x,y)$  if there exists a subset of vulnerable classifiers  $\mathcal{H}_{vb} \subseteq \mathcal{H}_b$  with  $\mu(\mathcal{H}_{vb}) > 0$  any of which can be forced to individually misclassify the point  $(x,y)$  by an adversary using (possibly different) perturbations  $x_h' \in B_\epsilon(x)$  for  $h \in \mathcal{H}_{vb}$ , but cannot be forced to misclassify all at the same time using the same perturbation  $x' \in B_\epsilon(x)$ . Such a configuration is reminiscent of the game of matching pennies [18, 46] (see Appendix B). If  $\mathcal{H}_b$  is in a matching penny configuration at  $(x,y)$ , a mixed strategy for classification (i.e. a mixture of  $\mathcal{H}_b$ ) achieves a decisive advantage over any pure strategy (i.e. any deterministic base classifier) because the adversary can only force a misclassification on a subset of all vulnerable classifiers. Such a configuration was first noted in [12] in the context of improved adversarial attacks on RECs, and also in [13] for computing the adversarial risk of RECs of size  $M = 2$ . Figure 1 illustrates such a condition on an example REC of size  $M = 2$  over a single point (Figure 1a), and over a simple discrete distribution (Figure 1b). We formalize this intuition with the definition below.
Definition 1 (Matching penny gap). The matching penny gap of a data point  $(x,y)\in (\mathcal{X}\times \mathcal{Y})$  with respect to a probabilistic classifier  $\mathbf{h}_{\mu}$  constructed from  $\mathcal{H}_b$  using  $\mu \in \mathcal{P}(\mathcal{H}_b)$  is defined as,
$$
\pi_ {\mathbf {h} _ {\mu}} (x, y) = \mu (\mathcal {H} _ {v b} (x, y)) - \mu^ {\max } (x, y), \tag {5}
$$