SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Multiclass Setting. The toy example in [45, Section 5.2] shows that randomization might be necessary in the multi-class setting, as it is no longer true that there is always a deterministic optimal classifier, even when considering very general families of deterministic classifiers like all measurable functions. A natural road for future work is to further characterize the situations in which probabilistic classifiers strictly outperform deterministic ones in the multi-class setting.
Training algorithms for diverse ensembles and RECs. There are numerous works based on the idea of training diverse classifiers that are not simultaneously vulnerable to create robust ensembles [11, 24, 33, 41, 49, 50]. Most of these approaches try to induce orthogonality in decision boundaries so that gradient-based attacks do not transfer between models. This intuition is validated by Corollary 3.1, since such diversification would tend to increase the matching penny gap. It should be noted that ensembles and RECs have different inference procedures, and attacking them represents different optimization problems. One avenue of research would be to make the link between the matching penny gap and the diversity more formal. In addition, designing training algorithms explicitly optimizing the matching penny gap would be valuable, particularly because existing training algorithms for RECs [31, 38] have been shown to be inadequate [12].
This work was funded by the French National Research Agency (DELCO ANR-19-CE23-0016).
[51] Tianyuan Yu, Yongxin Yang, Da Li, Timothy Hospedales, and Tao Xiang. Simple and effective stochastic neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3252-3260, 2021.
[52] Dinghuai Zhang, Hongyang Zhang, Aaron Courville, Yoshua Bengio, Pradeep Ravikumar, and Arun Sai Suggala. Building robust ensembles via margin boosting. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 26669-26692. PMLR, 17-23 Jul 2022.
Appendix A Measurability of the  $\ell^{0-1}$  under attack
In this section, we will go over the conditions that ensure the well-definedness of the adversarial risk defined in (1). We will show that the assumptions we make are verified in the settings that are of interest to our work. We highlight that the main purpose of this paper is not to deeply study the existence of measurable solutions to the adversarial problem, as was done in previous work [1, 40, 44]. In this work, we allow ourselves to add hypothesis on the input space  $\mathcal{X}$  and the families of classifiers  $\mathcal{H}_b$  that we consider, as long as they are valid in practice.
A.1 Discussion
Recall that in our setting we have the following. The input space  $\mathcal{X}$  is some subset (almost always bounded or even compact) of  $\mathbb{R}^d$ ,  $\mathcal{Y} = [K]$  and  $\mathcal{P}(\mathcal{Y}) = \Delta^K$ . On the other hand, for all the applications of interest to our work, we consider base hypothesis sets  $\mathcal{H}_b$  that are either finite or discrete (for randomized ensembles), or identifiable with subsets of some  $\mathbb{R}^p$  (for weight and input noise injection). For all these sets, we assume the usual topology and the Borel  $\sigma$ -algebra that is generated by the open sets. We further assume that  $\mathcal{X}$  and  $\mathcal{H}_b$  are Borel spaces, which is not restrictive. It allows, for example, any open or closed set of  $\mathbb{R}^n$ , which already encompasses all practical applications of interest. This hypothesis makes it simpler to work on the product measure space  $\mathcal{X} \times \mathcal{H}_b$ .
In the adversarial attacks literature, the question of measurability is always with respect to  $\mathcal{X}$  [2, 40]. For a deterministic classifier  $h: \mathcal{X} \to \mathcal{Y} = [K]$  to be measurable, the sets  $h^{-1}(k)$  must be measurable. In other words, each classifier  $h$  creates a  $K$ -partition of  $\mathcal{X}$  into measurable subsets  $h^{-1}(1), \ldots, h^{-1}(K)$ . This immediately translates into the  $\ell^{0-1}$ , as a function of  $x$ , being measurable, because  $\ell^{0-1}((\cdot, y), h)^{-1}(1) = \{x \in \mathcal{X} \mid h(x) \neq y\} = \mathcal{X} \setminus h^{-1}(y)$ . In other words, assuming classifiers are measurable functions translates into measurability of the 0-1 loss, so the arguments in previous work [1, 16, 40] can be used to justify that the adversarial risk is well-defined.
We introduce a new component, the BHS  $\mathcal{H}_b$ , and we now compute integrals over it. For this reason, we now consider more general functions of the form  $f:\mathcal{X}\times \mathcal{H}_b\to \{0,1\}$ ,  $f(x,h) = \mathbb{1}\{h(x)\neq y\}$  and study their measurability. If one fixes  $h$ , the function  $f^{h}(x) = \mathbb{1}\{h(x)\neq y\}$  becomes the same 0-1 loss considered earlier, so assuming every  $h$  is a measurable function translates into the fact that  $f$  is measurable w.r.t.  $x$  for every fixed  $h$ . The new part we have to deal with is the measurability of  $f$  w.r.t  $h$  for every fixed  $x$ .
As a first example, let us consider the case of linear classifiers. In this case, a classifier  $h$  can be represented as a  $K \times d$  matrix  $A$  with the classification rule  $h(x) = \operatorname{argmax}_k (Ax)_k$ , meaning  $h$  classifies  $x$  using the row with higher score. In this case, we can directly see the measurability of  $f$  in each component  $(f^h$  and  $f^x)$  by looking at the pre-images of 1, which are the sets that induce an error (recall  $y$  is fixed). For  $f^h$ , the pre-image is the set of  $x \in \mathcal{X}$  such that  $h(x) \neq y$ . This can be seen as the complement of the set  $X_y$  in which  $h$  predicts  $y$ .
$$
(f ^ {h}) ^ {- 1} (1) = \{x \in \mathcal {X} \mid h (x) \neq y \} = \{x \in \mathcal {X} \mid h (x) = y \} ^ {C} = X _ {y} ^ {C}
$$
The set  $X_{y}$  is described by  $K - 1$  linear equations using the rows  $a_{j}$  of the matrix  $A$  that defines the linear classifier  $h$ , as follows
$$
X _ {y} = \{x \in \mathcal {X} \mid a _ {y} ^ {T} x \geq a _ {j} ^ {T} x, \forall j \neq y \}
$$
This set is the finite intersection of hyperplanes, so it is Borel measurable, and therefore the function  $f^h$  is measurable on  $\mathcal{X}$  for any  $h$  linear classifier.
For the measurability of  $f^x$ , we have to consider the set  $H_y$  of classifiers that produce a prediction of  $y$  for a fixed  $x$ . Let us recall that each classifier is represented by a matrix  $A \in \mathbb{R}^{K \times d}$ , then
$$
H _ {y} = \{h \in \mathcal {H} _ {b} \mid h (x) = y \} = \{A \in \mathbb {R} ^ {K \times d} \mid a _ {y} ^ {T} x \geq a _ {j} ^ {T} x, \forall j \neq y \}
$$
$H_{y}$  is a convex cone, because if  $A,B\in H_y$  , then for a positive  $\alpha$ $\alpha A\in H_y$  and  $A + B\in H_y$  . This set is therefore measurable, and so we can conclude that  $f^{x}$  is measurable for every  $x$
When considering neural networks with weights parametrized by  $w$ , we can think of them as nonlinear functions plus a last linear layer as the one described earlier. Then, if the non-linear function is continuous w.r.t both  $x$  and  $w$ , the function  $f$  we are considering is again measurable w.r.t to  $x$  and  $w$ .
In summary, all the situations that are of interest to our work verify that the function  $f$  is defined over a product space  $\mathbb{R}^d \times \mathbb{R}^p$ , and it has good properties when seen as functions of  $x$  or  $h$  separately (measurable, continuous or differentiable). These separately measurable functions [7, Definition 4.47] might not be jointly measurable, i.e. measurable on the product space, which is one condition that would grant us the well-definedness of (1). Nevertheless, we can make use of the stronger properties these functions have and assume they are Carathéodory functions [7, Definition 4.50]. This will imply the joint measurability of  $f$ . For completeness, we rewrite the results from [7]:
Definition 2 (Carathéodory function [7, Definition 4.50]). Let  $(S,\Sigma)$  be a measurable space, and let  $X$  and  $Y$  be topological spaces. Let  $\mathcal{B}_Y$  be the Borel  $\sigma$ -algebra on  $Y$ . A function  $f:S\times X\to Y$  is a Carathéodory function if:
- For each  $x \in X$ , the function  $f^x = f(\cdot, x): S \to Y$  is  $(\Sigma, \mathcal{B}_Y)$ -measurable.
- For each  $s \in S$ , the function  $f^s = f(s, \cdot) : X \to Y$  is continuous.
As we have seen, the functions  $f: \mathcal{X} \times \mathcal{H}_b \to \mathbb{R}$  we consider can be assumed to be Carthéodory functions, as they are in general differentiable or continuous in both components. Even going away from practice, in theoretical works it is usual to consider classifiers  $h: \mathcal{X} \to \Delta^K$  as Borel measurable functions from  $\mathcal{X}$ , and adding the assumption of continuity w.r.t  $h$  for every fixed  $x$  it not restrictive (for example neural networks with continuous activation functions [31]).
Lemma A.1 (Carathéodory functions are jointly measurable [7, Lemma 4.51]). Let  $(S,\Sigma)$  be a measurable space,  $X$  a separable metrizable space, and  $Y$  a metrizable space. Then every Carathéodory function  $f:S\times X\to Y$  is jointly measurable
A.2 Conclusion
If we assume the hypotheses of Lemma A.1, that is
1. We assume that for every component  $y$ , the function  $f: \mathcal{X} \times \mathcal{H}_b \to \{0, 1\}$  defined as  $f(x, h) = \mathbb{1}\{h(x) \neq y\}$  is a Carathéodory function, which translates in our case into:
- Each  $h \in \mathcal{H}_b, h$  is a measurable function on  $\mathcal{X}$  (very common assumption).
- For each  $x$ , the function  $f^x: \mathcal{H}_b \to \{0,1\}$ ,  $f^x(h) = f(x,h)$  is continuous (not unrealistic, satisfied by neural networks with continuous activation functions)
2. The input space  $\mathcal{X}$  is a measurable space (satisfied by hypothesis,  $\mathcal{X} \subseteq \mathbb{R}^d$  is some Borel set).
3. The base hypothesis set  $\mathcal{H}_b$  is a separable metrizable space (valid when  $\mathcal{H}_b$  is a Borel subset of  $\mathbb{R}^p$ ).
4. The output space  $\{0,1\}$  is a metrizable space (satisfied).
then, by Lemma A.1, the function  $f$  is jointly measurable.
By Fubini-Tonelli's Theorem, the function  $x \to \int_{\mathcal{H}_b} f(x,h) d\mu(h)$  is measurable on  $\mathcal{X}$ . Then, following the same arguments as in [1, 16], the loss under attack can be defined and is measurable over the universal  $\sigma$ -algebra  $\mathcal{U}(\mathcal{X})$  on  $\mathcal{X}$ . Then, considering  $\mathcal{U}(\mathcal{X})$ , the adversarial risk is well-defined as it is an integral over  $\mathcal{X}$  using the completions of the class conditioned measures  $\rho_y$  over  $\mathcal{U}(\mathcal{X})$  (see (1)).
Appendix B On the name Matching penny gap
In the original matching penny game between player 1 (attacker) and player 2 (defender), each player has a penny coin and has to secretly position its penny in either heads or tails. Then, both coins are simultaneously revealed and the outcome of the game is decided as follows: attacker wins if both pennies match. If they do not match, then defender wins. This situation can be represented using the following matrix (attacker wants to maximize), where heads and tails have been replaced by  $f_{1}$  and  $f_{2}$  for reasons that will be explained later:
Attacker
Defender	f1	f2
f1	1	0
f2	0	1
This game has no Pure Nash Equilibrium. It has, however, a mixed Nash equilibrium which consists of the strategies  $\left(\frac{1}{2},\frac{1}{2}\right)$  for both players [18]. In the context of the matching pennies game, this can be seen as the strategy in which players toss the coin to obtain either heads or tails uniformly at random (assuming the coins are fair) instead of choosing one of the sides. Using this strategy, each player can be sure that, on average over multiple realizations of the game, they will win half of the times each. The fact that it is a Nash Equilibrium means that, if one player deviated from this strategy, the other player could find a way to win more than half the times. To see this, imagine that the defender player plays heads with probability  $\frac{1}{2} +\delta$ . Then, if attacker knew this, he could play the strategy always heads. Then, from all the realizations of the game, with probability  $\frac{1}{2} +\delta$  the coins would be both on the same side (heads), meaning attacker would win  $\frac{1}{2} +\delta$  of the times, gaining  $\delta$  w.r.t the equilibrium strategy.
The parallel with our work can be simply explained as follows (see Figure 2): Take a point  $(x_0, y_0)$  and two classifiers  $f_1, f_2$  that correctly classify  $x_0$ . Suppose that both  $f_1$  and  $f_2$  are vulnerable at  $x_0$ , which is represented by the colored areas in Figure 2. However, add the key assumption that they cannot be attacked at the same time, which is represented by the fact that the colored areas do not intersect inside the  $\epsilon$ -ball around  $x_0$ . That is, even though an optimal attacker can fool each
Figure 2: Example of classifiers in a matching penny configuration around a point  $x_0$ .
classifier individually, there is no point in the allowed perturbation region  $B_{\epsilon}(x_0)$  in which both are simultaneously fooled.
Consider now that the defender is using a randomized ensemble that picks either  $f_{1}$  or  $f_{2}$  at random for inference. This was introduced in [38] with the name mixtures of classifiers, related to mixed strategies in the context of game theory. In such setting, the optimal attacker that faces such mixture is now in a matching pennies game situation. At each inference pass, the attacker must choose which classifier to attack in order to craft the adversarial example  $x'$ , and if the chosen classifier matches with the one the defender used, then the attacker wins. If they do not match, the prediction made by the defender will be correct and the attacker would have lost. This is exactly the game of matching pennies!
What is particularly worse for defender is that, in the traditional formulation of adversarial attacks, the attacker is given the chance to play second and therefore can adapt to the classifier. In other words, attacker can craft its perturbation for the particular classifier proposed by defender, while defender has to propose a classifier that should be robust to every conceivable attack. This means that if defender uses  $f_{1}$  or  $f_{2}$  individually, then attacker would be able to craft an attack that induces an error. In this situation, and inspired by the game of matching pennies, defender can turn these two strategies, that have 0 robust accuracy on their own, into a strategy that guarantees an expected accuracy of  $\frac{1}{2}$  by randomizing over  $f_{1}$  and  $f_{2}$ . Recall that attacker is optimal for any fixed classifier, but it has no control over randomness, which means that when faced against a
mixture of  $f_{1}$  and  $f_{2}$ , attacker can compute any perturbation  $\delta$  from them, but he has no knowledge of which classifier will be used for each independent inference pass. This can be interpreted as follows: for a given inference run, there is a chance of  $\frac{1}{2}$  that the perturbation proposed by attacker is indeed an adversarial attack and induces an error.
Notice that if we increase the number of choices (classifiers) in matching pennies' configuration to  $M > 2$ , the game becomes harder for the attacker. Indeed, for every possible choice of classifier at each inference pass, there is one out of  $M$  choices that lead to a successful attack (attacking the one that was sampled for the inference pass), and  $M - 1$  that lead to a correct classification. An extreme example of such benefit to the defender is shown in Example 1.
Appendix C Details on the examples and remarks
C.1 Example 1: Maximum mating penny gap
Note that for this example, each classifier  $w$  satisfies that  $w^T x = 0$ , and therefore  $\mathbb{1}\{w^T x < 1\} = 1$ , which means that all  $w$  predict the correct label for the clean input  $x$ . Now we want to see that every  $w$  is vulnerable at  $x$ .
Recall that we defined  $\mathcal{H}_b$  as the space of linear classifiers  $w$  such that  $\| w\| _2 = \frac{1}{\epsilon}$ . We can rewrite this norm as a dual norm  $\| w\| _2 = \sup_{\| z\| = 1}w^T z$ . Moreover, this supremum is attained by some  $z_{w}$  so for each  $w$  we get  $z_{w}$  of norm 1, such that  $w^{T}z_{w} = \frac{1}{\epsilon}$ .
Figure 3: Visualization of Example 1. Best viewed in color. Here  $w$  is one of the linear classifiers considered in  $\mathcal{H}_b$ . For the budget  $\epsilon$ , there is exactly one point  $\epsilon \cdot z_w$  of intersection between the  $\epsilon$ -ball around  $(x, 1)$  and the hyperplane defined by the equation  $w^T x = 1$ . All the points in the half-space  $w^T x \geq 1$  are classified as 0. This includes  $\epsilon \cdot z_w$ , and this is the only point in the  $\epsilon$ -ball that belongs to this half-space. That is why  $\epsilon \cdot z_w$  is the only adversarial attack that can fool  $w$ , and it cannot fool any other  $w' \neq w$ .
Now, for each  $w \in \mathcal{H}_b$ , consider the adversarial example  $\epsilon \cdot z_w$ . It is a valid adversarial example because it has norm  $\epsilon$ , and  $w^T(\epsilon \cdot z_w) = \epsilon \cdot \frac{1}{\epsilon} = 1$ , meaning that the classifier predicts  $\mathbb{1}\{w^T(\epsilon \cdot z_w) < 1\} = 0$ , the wrong class. The next step is to guarantee that this perturbation is unique for each  $w$ , which holds because we are using the Euclidean norm.
Having that  $\epsilon \cdot z_w$  fools  $w$  and only  $w$ , we can consider for simplicity  $\mu$  the uniform distribution over  $\mathcal{H}_b$ . Let us compute the sets  $\mathcal{H}_{vb}(x,1)$  and  $\mu^{\max}(x,1) = 0$  to be able to compute the matching penny gap for this example.
As we just saw, every  $w$  is itself vulnerable. This means that  $\mathcal{H}_{vb}(x,1) = \mathcal{H}_b$ , and therefore  $\mu(\mathcal{H}_{vb}(x,1)) = 1$ .
Given the unicity of  $z_w$ , we know that no two classifiers can be fooled by the same perturbation. Therefore the family of simultaneously vulnerable classifiers only contains singletons  $\{w\}$ . As  $\mu(\{w\}) = 0$  for every  $w$ , taking the supremum gives  $\mu^{\max}(x,1) = 0$ .
Finally, expected the matching penny gap for this example with only one point is
$$
\underset {x, y} {\mathbb {E}} \left[ \pi_ {\mathbf {h} _ {\mu}} (x, 1) \right] = \pi_ {\mathbf {h} _ {\mu}} (x, 1) = \mu (\mathcal {H} _ {v b} (x, 1)) - \mu^ {\max} (x, 1) = 1 - 0 = 1.
$$
In other words, the mixture in this example has the best adversarial risk possible, even though it is built from classifiers with the worst adversarial risk possible individually.
C.2 Example 2: Minimum mathing penny gap / Parallel sets
To simplify the example, consider the compact set  $A = \{0\} \subset \mathbb{R}$ . Then, the family of all parallel classifiers  $h_r = A \oplus B_r(0)$  are the classifiers of the form  $h_r(x) = \mathbb{1}\{x \in (-r,r)\}$  for  $r > 0$ .
Recall that a matching penny configuration arises when 1) both classifiers are vulnerable, but 2) not simultaneously. That is, each one of them must be vulnerable, but no allowed perturbation can induce an error on both at the same time. We will see that this cannot happen with this family of classifiers that are "parallel".
W.l.o.g, take any point  $x > 0$  and suppose it is of class is 0. Take any two classifiers  $h_{r_1}, h_{r_2}$  with  $r_1 < r_2$  and fix the attacker budget to  $\epsilon$ . Note that  $h_{r_1}$  is vulnerable at  $x$  if an only if  $x - \epsilon \leq r_1$ . That is, the attacker must be able to move  $x$  inside  $(-r, r)$  with its budget of  $\epsilon$ . This also holds for  $h_{r_2}$ .
Note that to satisfy the condition that both classifiers are vulnerable, we must ensure that  $x - \epsilon \leq r_1$  and  $x - \epsilon \leq r_2$ . However, any  $x$  that satisfies  $x - \epsilon \leq r_1$  immediately satisfies  $x - \epsilon \leq r_2$ , as  $r_1 < r_2$ , meaning that the same perturbation induces an error on both classifiers. In conclusion, they cannot be both individually vulnerable without being simultaneously vulnerable at  $x$ .
(a) Example of parallel classifiers in  $\mathbb{R}$
(b) Example of parallel classifiers in  $\mathbb{R}^2$
For the case of general parallel sets  $A^{r_1}, A^{r_2}$ , note that if  $r_1 < r_2$  then  $A^{r_1} \subset A^{r_2}$ . Assuming that the classifiers predict 1 if the point is inside the set  $A^r$ , then the condition both classifiers are vulnerable means that a point  $x$  of class 0 is at a distance of at most  $\epsilon$  from both  $A^{r_1}, A^{r_2}$ . However, for any given  $x$ , being  $\epsilon$ -close to  $A^{r_1}$  implies that  $x$  is  $\epsilon$ -close to  $A^{r_2}$ , so any adversarial example of  $A^{r_1}$  will be also adversarial for  $A^{r_2}$ . In conclusion, they can never be in a matching penny configuration.
C.3 Multiple optimal classifiers in matching penny configuration
In this section, we are going to explain in more detail Remark 2. First, we rewrite Equation (9) which states the condition under which a mixture outperforms the individual classifiers composing it:
$$
\underset {(x, y) \sim \rho} {\mathbb {E}} \left[ \pi_ {\mathbf {h} _ {\mu^ {\prime}}} (x, y) \right] > \underset {h \sim \mu^ {\prime}} {\mathbb {E}} \left[ \mathcal {R} _ {\epsilon} (h) \right] - \inf  _ {h \in \mathcal {H} _ {b}} \mathcal {R} _ {\epsilon} (h)
$$
For simplicity, assume  $\mathcal{H}_b$  consists of two different classifiers  $h_1, h_2$  with the same adversarial risk and with a strictly positive matching penny gap, i.e.  $\mathbb{E}_{(x,y) \sim \rho}[\pi_{\mathbf{h}_\mu}(x,y)] > 0$  for some
$\mu = (\mu(h_1), \mu(h_2))$ . Then the REC  $\mathbf{h}_{\mu}$  that consists of these two classifiers will outperform each one of them. This is because the LHS of equation (9) is positive, while the RHS is exactly 0 because both classifiers have the same risk, so their average risk is equal to the best risk among them. If we add the assumption that  $h_1$  and  $h_2$  were optimal deterministic classifiers from a larger family  $\mathcal{H}_b$ , we would have created a mixture that outperforms all classifiers in it.
If now we try to soften the assumptions on  $h_1, h_2$ , and say that they have very similar adversarial risks, i.e.  $\mathcal{R}_{\epsilon}(h_2) = \mathcal{R}_{\epsilon}(h_1) + \delta$  for some small  $\delta$ , then the RHS of (9) is no longer 0 but  $\frac{\delta}{2}$ . In this case, for condition (9) to hold we need the matching penny gap to be not only strictly positive, but greater than  $\frac{\delta}{2}$ .
The important intuition is that mixing classifiers that have very different adversarial risks is not ideal, as one will then need a much greater expected matching penny gap in order to actually outperform the best individual classifier. Another interesting intuition is that one can create a good mixture from very bad classifiers, as was seen in Example 1, where individual classifiers had adversarial risk of 1 (the worst possible), but mixing them resulted in an adversarial risk of 0 (the best possible). Even if Example 1 is merely a toy example, it highlights the intuition that it is possible for many non-robust classifiers to gain robustness as a mixture if they interact nicely between them and the dataset (have a high expected matching penny gap).
C.4 Example with linear classifiers: Mixtures can improve robustness (Figure 1b)
Figure 5: The mixture of  $f_{1}$  and  $f_{2}$  with uniform weights has lower adversarial risk than any linear classifier, motivating the use of mixtures as a robust alternative.
C.4.1 Detailed description of the example