Title: On the Role of Randomization in Adversarially Robust Classification
Venue: NeurIPS 2023
Accepted: yes
Keywords: adversarial attacks

robustness

adversarial

attacks

deep learning

randomization

randomized ensembles
Abstract: Deep neural networks are known to be vulnerable to small adversarial perturbations in test data. To defend against adversarial attacks, probabilistic classifiers have been proposed as an alternative to deterministic ones. However, literature has conflicting findings on the effectiveness of probabilistic classifiers in comparison to deterministic ones. In this paper, we clarify the role of randomization in building adversarially robust classifiers.
Given a base hypothesis set of deterministic classifiers, we show the conditions under which a randomized ensemble outperforms the hypothesis set in adversarial risk, extending previous results.
Additionally, we show that for any probabilistic binary classifier (including randomized ensembles), there exists a deterministic classifier that outperforms it. Finally, we give an explicit description of the deterministic hypothesis set that contains such a deterministic classifier for many types of commonly used probabilistic classifiers, *i.e.* randomized ensembles and parametric/input noise injection.

On the Role of Randomization in Adversarially Robust Classification

Abstract

Deep neural networks are known to be vulnerable to small adversarial perturbations in test data. To defend against adversarial attacks, probabilistic classifiers have been proposed as an alternative to deterministic ones. However, literature has conflicting findings on the effectiveness of probabilistic classifiers in comparison to deterministic ones. In this paper, we clarify the role of randomization in building adversarially robust classifiers. Given a base hypothesis set of deterministic classifiers, we show the conditions under which a randomized ensemble outperforms the hypothesis set in adversarial risk, extending previous results. Additionally, we show that for any probabilistic binary classifier (including randomized ensembles), there exists a deterministic classifier that outperforms it. Finally, we give an explicit description of the deterministic hypothesis set that contains such a deterministic classifier for many types of commonly used probabilistic classifiers, i.e. randomized ensembles and parametric/input noise injection.

1 Introduction

Modern machine learning algorithms such as neural networks are highly sensitive to small, imperceptible adversarial perturbations of inputs [3, 43]. In the last decade, there has been a back-and-forth in research progress between developing increasingly potent attacks [5, 10, 19, 25] and the creation of practical defense mechanisms through empirical design [8, 30, 32]. In particular, probabilistic classifiers (also called stochastic classifiers or randomized classifiers) have been widely used to build strong defenses which can be roughly categorized into two groups: noise injection techniques [14, 28, 34, 37, 47, 48] and randomized ensembles [31, 38]. The first group includes methods that add noise at inference, usually to the input [22, 48], an intermediate layer activation [22, 47, 51] or the weights of a parametric model [22] like a neural network. Most of this work is experimental, with some theoretical papers that try to justify the use of such methods [37]. The second group, inspired by game theory, create a mixed strategy over base classifiers as a finite mixture [31, 38]. The argument behind this kind of models is that it becomes harder for an attacker to fool multiple models at the same time [12, 36].

Intuitively, the greater flexibility of probabilistic classifiers implies that they should outperform their deterministic counterparts in adversarially robust classification. For instance, one of the earliest works using randomization to improve robustness to adversarial attacks [48] claims that 'randomization at inference time makes the network much more robust to adversarial images' . Dhillon et al. [14] propose a pruning method that 'is stochastic and has more freedom to deceive the adversary' .

Similarly, Panousis et al. [34] claim that for their method, 'the stochastic nature of the proposed activation significantly contributes to the robustness of the model' .

Lending support to the intuition that probabilistic classifiers are better than their deterministic counterparts, Pinot et al. [38] prove the non-existence of a Nash equilibrium in the zero-sum game between an adversary and a classifier when both use deterministic strategies. Moreover, they show that a randomized ensemble outperforms any deterministic classifier under a regularized adversary. Meunier et al. [31] shows that a mixed Nash equilibrium does exist in the adversarial game when the classifier is allowed to use randomized strategies, making a strong case for probabilistic classifiers.

In contrast to the above results advocating for probabilistic classifiers, Pydi and Jog [40] use an equivalence between adversarial risk minimization and a certain optimal transport problem between data generating distributions to show that in the case of binary classification there exists an approximate pure Nash equilibrium in the adversarial game, showing that there always exists a sequence of deterministic classifiers that are optimal for adversarial risk minimization. Trillos et al. [45] extend their results to multiclass classification using a multimarginal optimal transport formulation. More recently, Awasthi et al. [1, 2] prove the existence of a deterministic classifier that attains the optimal value for adversarial risk under mild assumptions. Trillos et al. [44] extends it to the multiclass setting.

How does one reconcile these apparently conflicting theoretical findings as a machine learning practitioner? First, it is important to note that the above theoretical results are valid when considering large hypothesis sets, like that of all possible measurable classifiers. In practice, one works with more limited hypotheses sets such as the space of linear classifiers or the space of classifiers that can be learned by an L -layered deep neural network, for which the above theoretical results do not directly apply. Second, the task of training a robust probabilistic classifier is non-trivial [12, 39]. Even the task of efficiently attacking a probabilistic classifier is not straightforward [6, 12, 13, 36]. Hence, it is important to understand the precise role of randomization in adversarial robustness and the conditions under which probabilistic classifiers offer a discernible advantage over deterministic classifiers. In this work, we address this issue by making the following contributions.

From deterministic to probabilistic classifiers: Given a base hypothesis set (BHS) H b of deterministic classifiers, we prove the necessary and sufficient conditions for a probabilistic classifier built from H b to strictly outperform the best classifier in H b in adversarial risk. A quantity that plays a crucial role in our analysis is the expected matching penny gap wherein there exist classifiers that are vulnerable to adversarial perturbation at a point, but not all at the same time. These results are contained in Section 3.

From probabilistic to deterministic classifiers: In the binary classification setting, given a probabilistic classifier h , we show that there always exists a deterministic classifier h belonging to a 'threshold' hypothesis set (THS) that is built from h in a straightforward way. We then apply our results to common families of probabilistic classifiers, leading to two important conclusions: 1) For every randomized ensemble classifier [13, 38], there exists a deterministic weighted ensemble [17, 52] with better adversarial risk. 2) For every input noise injection classifier [22, 37, 39], there exists a deterministic classifier that is a slight generalization of randomized smoothing classifier [8, 26] with better adversarial risk. These results are contained in Section 4.

Families of classifiers for which randomization does not help: Given a BHS H b , we show the conditions under which randomizing over classifiers in H b does not help to improve adversarial risk. Specifically, if the decision regions of H b are closed under union and intersection, our results imply that randomized ensembles built using H b offer no advantage over the deterministic classifiers in H b . These results are contained in Section 5.

Notation: We use 1 { C } to denote an indicator function which takes a value 1 if the proposition C is true, and 0 otherwise. For an input space X ⊆ R d , we use σ ( X ) to denote some σ -algebra over X that makes ( X , σ ( X )) a measurable space. If not specified, σ ( X ) will be the Borel σ -algebra, which is generated by the open sets of X . We use P ( X ) to denote the set of probability measures over the measure space ( X , σ ( X )) . For a positive integer K , we use [ K ] to denote the range { 1 , . . . , K } . For a vector u ∈ R K , we denote u i the i th component of u .

2 Preliminaries

2.1 Adversarially Robust Classification

We consider a classification setting with input space X ⊆ R d and a finite label space Y . The space X is equipped with some norm ∥·∥ , which is commonly set to be the ℓ 2 or ℓ ∞ norms. Let ρ ∈ P ( X ×Y ) denote the true data distribution, which can be factored as ρ ( x, y ) = ν ( y ) ρ y ( x ) , where ν ∈ P ( Y ) denotes the marginal probability distribution of ρ over Y and ρ y ∈ P ( X ) denotes the conditional probability distribution of the data over X given class y .

̸

A deterministic classifier is a function h : X → Y that maps each x ∈ X to a fixed label in Y . The 0 -1 loss of h on a point ( x, y ) ∈ X × Y is given by, ℓ 0 -1 (( x, y ) , h ) = 1 { h ( x ) = y } .

A probabilistic classifier is a function h : X → P ( Y ) that maps each x ∈ X to a probability distribution over Y . To label x ∈ X with h , one samples a random label from the distribution h ( x ) ∈ P ( Y ) . In practice, if Y consists of K classes, P ( Y ) is identifiable with the K -simplex ∆ K that consist of vectors u ∈ R K such that ∑ K i =1 u i = 1 . Therefore, one can think of h ( x ) as a probability vector for every x .

̸

The 0 -1 loss of h on ( x, y ) ∈ X ×Y is given by, ℓ 0 -1 (( x, y ) , h ) = E ˆ y ∼ h ( x ) [ 1 { ˆ y = y } ] = 1 -h ( x ) y . Note that ℓ 0 -1 (( x, y ) , h ) ∈ [0 , 1] is a generalization of the classical 0 -1 loss for deterministic classifiers, which can only take values in { 0 , 1 } .

Given x ∈ X , we consider a data perturbing adversary of budget ϵ ≥ 0 that can transport x to x ′ ∈ B ϵ ( x ) = { x ′ ∈ X | d ( x, x ′ ) ≤ ϵ } , where B ϵ ( x ) is the closed ball of radius ϵ around x . The adversarial risk of a probabilistic classifier h is defined as follows.

The adversarial risk of a deterministic classifier h is defined analogously, replacing h by h in (1).

2.2 Threat model

We consider a white-box threat model, in which the adversary has complete access to the classifier proposed by the defender. In theory, we are assuming that the adversary is able to attain (or approximate to any precision) the inner supremum in (1). In practice, if the classifier is differentiable, the adversary will have full access to the gradients (and any derivative of any order), and it will have full knowledge of the classifier, including its architecture and any preprocessing used. The adversary can also query the classifier as many times as needed. The only limitation for the adversary regards the sources of randomness. In particular, when the proposed classifier is probabilistic and an inference pass is done by sampling a label from h ( x ) , the adversary does not know and cannot manipulate the randomness behind this sampling.

It is worth noting that other works have considered similar inference processes, in which the label of a point is obtained by a sampling procedure [29]. The underlying question is similar to ours: does randomness improve adversarial robustness? . They propose a threat model in which an attacker can perform N independent attempts with the goal of producing at least one misclassification , which becomes easier if the predicted probability for the correct class is less than 1. In this paper, however, we are interested in the expected accuracy of a classifier from a theoretical point of view, which in practice translates to the probability of error, or the long term average performance or the classifier.

Our formulation of adversarial risk in (1) corresponds to the 'constant-in-the-ball' risk in [20] or the 'corrupted-instance' risk in [15]. Under this formulation, as pointed out in [40], an adversarial risk of 0 is only possible is the supports of the class-conditioned distributions ρ y are non-overlapping and separated by at least 2 ϵ . This is not the case with other formulations of risk introduced in [15, 20]. We focus on the former in this work.

2.3 Probabilistic Classifiers Built from a Base Hypothesis Set

In this paper, we study probabilistic classifiers that are constructed from a base hypothesis set (BHS) of (possibly infinitely many) deterministic classifiers, denoted by H b . We use the name mixtures [38] for these type of classifiers. In the following, we show that many of the probabilistic classifiers that are commonly used in practice fall under this framework.

Let H b be a BHS of deterministic classifiers from X to Y , which we assume is identifiable with a Borel subset of some R p . Let µ ∈ P ( H b ) be a probability measure over the measure space ( H b , σ ( H b )) , where σ ( H b ) denotes the Borel σ -algebra on H b . One can construct a probabilistic classifier h µ : X → P ( Y ) , built from H b , that maps x ∈ X to a probability distribution µ x ∈ P ( Y ) , where µ x ( y ) = P h ∼ µ ( h ( x ) = y ) . We now instantiate H b and µ ∈ P ( H b ) for two main types of probabilistic classifiers that are commonly used in practice: randomized ensembles and noise injection classifiers.

For a randomized ensemble classifier (REC) , µ ∈ P M ( H b ) ⊂ P ( H b ) where P M ( H b ) is the set of all discrete measures on H b supported on a finite set of M deterministic classifiers, { h 1 , . . . , h M } . In this case, h µ ( x ) takes the value h m ( x ) with probability p m = µ ( h m ) for m ∈ [ M ] , where ∑ m ∈ [ M ] p m = 1 . RECs were introduced in [38] and play the role of mixed strategies in the adversarial robustness game. They are a simple randomization scheme when a finite number of classifiers are at hand, and both training and attacking them represent a challenge [12, 13].

For a weight-noise injection classifier (WNC) , H b = { h w : w ∈ W} where h w is a deterministic classifier with parameter w . In this case, µ is taken to be a probability distribution over W with the understanding that each w ∈ W is associated with a unique h w ∈ H b . For example, H b can be the set of all neural network classifiers with weights in the set W ⊆ R p . Any probability distribution µ on the space of weights W results in a probabilistic classifier h µ . Alternatively, H b can be generated by injecting noise z sampled from a distribution µ to the weights w 0 of a fixed classifier h w 0 . In this case, the probabilistic classifier h µ maps x ∈ X to a probability distribution µ x ∈ P ( Y ) , where µ x ( y ) = P z ∼ µ ( h w 0 + z ( x ) = y ) . Weight noise injection has been explicitly used in [22], but there are many other approaches that implicitly define a probability distribution over the parameters of a model [14, 35] and sample one or more at inference.

For an input-noise injection classifier (INC) , H b = { h η : η ∈ X} where h η ( x ) = h 0 ( x + η ) for a fixed deterministic classifier h 0 . In this case, µ is taken to be a probability distribution over X (which is unrelated to the data generating distribution), and h µ maps x ∈ X to a probability distribution µ x ∈ P ( Y ) , where µ x ( y ) = P η ∼ µ ( h ( x + η ) = y ) . Injecting noise to the input has been used for decades as a regularization method [4, 21]. INCs are studied in [22, 37, 39] as a defense against adversarial attacks, and are closely related to randomized smoothing classifiers [8, 26].

Remark 1 (Measurability) . To ensure the well-definedess of the adversarial risk, we need to ensure that the 0 -1 loss under attack (inner part of (1)) is measurable. The 0 -1 loss for a fixed class y ∈ Y can now be seen as a function from the product space X × H b to { 0 , 1 } :

̸

For a distribution µ ∈ P ( H b ) and the associated probabilistic classifier h µ , we can rewrite the 0 -1 loss of h µ at x as

̸

Note that if f is Borel measurable in X × H b , then by Fubini-Tonelli's Theorem, the 0 -1 loss as a function of x with an integral over H b shown in (3) is Borel measurable. By [16, Appendix A, Theorem 27], the 0 -1 loss under attack is then universally measurable and therefore the adversarial risk, which requires the integral over X of the loss under attack, is well-defined over this universal σ -algebra [44, Definition 2.7]. Recent work by Trillos et al. [44] have also shown the existence of Borel measurable solutions for the closed ball formulation of adversarial risk.

In this work, we assume that the function f in (2) is Borel measurable in X × H b for every y , so that (1) is well-defined. This assumption is already satisfied in the settings that are of interest for our work. As an example, it holds when the classifiers at hand are neural networks with continuous activation functions. More details on Appendix A. For a deeper study on the measurability and well-definedness of the adversarial risk in different threat models and settings, see [1, 16, 40, 44].

3 From Deterministic to Probabilistic Classifiers

Let H b be a class of deterministic classifiers. In this section, we compare the robustness of probabilistic classifiers built upon H b with the robustness of the class H b itself. Note that if we consider the trivial mixtures µ = δ h for h ∈ H b , we obtain the original classifiers in H b , so it is always true that inf µ ∈P ( H b ) R ϵ ( h µ ) ≤ inf h ∈H b R ϵ ( h ) . We are interested in the situations in which this gap is strict, meaning that mixtures strictly outperform the base classifiers. With this in mind, we introduce the notion of Matching Penny Gap to quantify the improvement of probabilistic classifiers over deterministic ones.

Theoretical Results on Robustness of Probabilistic Classifiers. We begin by presenting a theorem which shows that the adversarial risk of a probabilistic classifier is at most the average of the adversarial risk of the deterministic classifiers constituting it. The proof of this result will be a first step towards understanding the conditions that favor the mixture classifier.

Theorem 3.1. For a probabilistic classifier h µ : X → P ( Y ) constructed from a BHS H b using any µ ∈ P ( H b ) , we have R ϵ ( h µ ) ≤ E h ∼ µ [ R ϵ ( h )] .

A natural follow-up question to is to ask what conditions guarantee a strictly better performance of a probabilistic classifier . We know that this gap can be strict, as can be seen in toy examples like the one shown in Figure 1b where H b is the set of all linear classifiers, and there exist two distinct classifiers f 1 , f 2 both attaining the minimum adversarial risk among all classifiers in H b . Any non-degenerate mixture of f 1 , f 2 attains a strictly better adversarial risk, demonstrating a strict advantage for probabilistic classifiers.

From the proof of Theorem 3.1 it is clear that a strict gap in performance between probabilistic and deterministic classifiers is only possible when the following strict inequality holds for a non-vanishing probability mass over ( x, y ) .

̸

̸

The above condition holds at ( x, y ) if there exists a subset of vulnerable classifiers H vb ⊆ H b with µ ( H vb ) > 0 any of which can be forced to individually misclassify the point ( x, y ) by an adversary using (possibly different) perturbations x ′ h ∈ B ϵ ( x ) for h ∈ H vb , but cannot be forced to misclassify all at the same time using the same perturbation x ′ ∈ B ϵ ( x ) . Such a configuration is reminiscent of the game of matching pennies [18, 46] (see Appendix B). If H b is in a matching penny configuration at ( x, y ) , a mixed strategy for classification (i.e. a mixture of H b ) achieves a decisive advantage over any pure strategy (i.e. any deterministic base classifier) because the adversary can only force a misclassification on a subset of all vulnerable classifiers. Such a configuration was first noted in [12] in the context of improved adversarial attacks on RECs, and also in [13] for computing the adversarial risk of RECs of size M = 2 . Figure 1 illustrates such a condition on an example REC of size M = 2 over a single point (Figure 1a), and over a simple discrete distribution (Figure 1b). We formalize this intuition with the definition below.

Definition 1 (Matching penny gap) . The matching penny gap of a data point ( x, y ) ∈ ( X × Y ) with respect to a probabilistic classifier h µ constructed from H b using µ ∈ P ( H b ) is defined as,

where H vb ( x, y ) ⊆ H b denotes the vulnerable subset and µ max ( x, y ) the maximal simultaneous vulnerability of base classifiers H b , defined below.

̸

̸

If π h µ ( x, y ) > 0 , we say that h µ is in matching penny configuration at ( x, y ) .

For example, in Figure 1a, π h µ ( x 0 , y 0 ) = 1 -max { µ ( f 1 ) , µ ( f 2 ) } = min { µ ( f 1 ) , µ ( f 2 ) } where H b = { f 1 , f 2 } . The subset H vb ( x, y ) contains all classifiers that can be individually fooled by an

optimal attacker. The collection of subsets H svb ( x, y ) contains all subsets H ′ of classifiers that can be simultaneously fooled . Then, µ max ( x, y ) is the maximum mass of classifiers that can be fooled simultaneously. Thus, π h µ ( x, y ) measures the gap between the mass of classifiers that are individually vulnerable and the maximum mass of classifiers that can be fooled with only one perturbation.

f1

f2

̸

̸

C2

C1

+

C4

fi(α) = 1

f(x)= 1

fi(x) = 0

f2(x)= 0

The following theorem strengthens Theorem 3.1 by showing that R ϵ ( h ) is a strictly convex function if and only if there is a non-zero expected matching penny gap.

Theorem 3.2. For a probabilistic classifier h µ : X → P ( Y ) constructed from a BHS H b using any µ ∈ P ( H b ) ,

Proof sketch. By interchanging expectations over h ∼ µ and ( x, y ) ∼ ρ and using the fact that sup x ′ ∈ B ϵ ( x ) 1 { h ( x ′ ) = y } = 1 if and only if h ∈ H vb ( x, y ) , we first show the following.

̸

Arguing from the definition of µ max ( x, y ) , we then show that sup x ′ ∈ B ϵ ( x ) E h ∼ µ [ 1 { h ( x ′ ) = y } ] = µ max ( x, y ) . Taking expectation with respect to ( x, y ) ∼ ρ , we get,

̸

Combining (8) and (7) yields (6).

The following corollary shows that a lower bound on the expected matching penny gap is both necessary and sufficient for a mixture to strictly outperform any deterministic classifier in H b .

Corollary 3.1. For µ ′ ∈ P ( H b ) , R ϵ ( h µ ′ ) < inf h ∈H b R ϵ ( h ) if and only if the following condition holds.

Additionally, inf µ ∈P ( H b ) R ϵ ( h µ ) < inf h ∈H b R ϵ ( h ) if and only if there exists µ ′ ∈ P ( H b ) for which (9) holds.

Remark 2 (Multiple optimal classifiers in matching penny configuration) . For finite H b , the assumption (9) of Corollary 3.1 holds whenever there exist two distinct optimal classifiers h ∗ 1 , h ∗ 2 ∈ H b with a positive expected matching penny gap. In such a case, the RHS of (9) is zero for any µ that is a mixture of h ∗ 1 , h ∗ 2 . This is indeed the case in Figure 1b. More generally, assumption (9) holds if there is a subset of classifiers in a matching penny configuration that are 'near optimal' on average. More details in Appendix C.3.

The following example depicts the scenario of Remark 2 in the extreme case in which there exist infinitely many distinct optimal classifiers in matching penny configuration, leading to the maximum possible advantage of using probabilistic classifiers.

Example 1 (Maximum expected matching penny gap of 1 ) . Consider a discrete data distribution ρ = δ ( x = 0 , y = 1) and a BHS composed of infinitely many linear classifiers, H b = { h : h ( x ) = 1 { w T x < 1 } , ∥ w ∥ 2 = 1 ϵ } where ∥ · ∥ 2 is the Euclidean norm. Observe that every classifier in H b is vulnerable at ( x, y ) = (0 , 1) and so R ϵ ( h ) = 1 for all h ∈ H b . However, no pair is simultaneously vulnerable. Hence, π h µ ( x, y ) = 1 for any distribution µ ∈ P ( H b ) that is continuous over the entire support H b . By Theorem 3.2 we get that R ϵ ( h µ ) = 0 for any such µ . Therefore, any such h µ outperforms every h ∈ H b , and we have 0 = inf µ ∈P ( H b ) R ϵ ( h µ ) < inf h ∈H b R ϵ ( h ) = 1 . More details on Appendix C.1.

Remark 3 (Probabilistic classifiers with zero matching penny gap are not useful) . Suppose the expected matching penny gap E ( x,y ) ∼ ρ [ π h µ ′ ( x, y )] is zero for some µ ′ ∈ P ( H b ) . Then the left hand side of (9) is zero, whereas the right-hand side E h ∼ µ ′ [ R ϵ ( h )] -inf h ∈H b R ϵ ( h ) is always non-negative. Hence, (9) does not hold and so R ϵ ( h µ ′ ) ≥ inf h ∈H b R ϵ ( h ) . Therefore, any such probabilistic classifier underperforms the best deterministic classifier in the base set H b .

The following example illustrates a scenario described in Remark 3, where we examine classifiers with decision boundaries that are parallel sets [23].

Example 2 (Minimum expected matching penny gap of 0 / Parallel decision boundaries) . Fix any binary classifier h : X → { 0 , 1 } with non-empty decision region A h = { x ∈ X : h ( x ) = 1 } ⊂ X ⊆ R d . Let H b be composed of all classifiers whose decision regions are r -parallel sets of A h defined as A r h = A h ⊕ B r (0) where ⊕ denotes the Minkowski sum, i.e., H b = { h : ∃ r ≥ 0 s.t. A r h = { x ∈ X : h ( x ) = 1 }} . Because of the parallel decision boundaries, whenever two classifiers in H b are vulnerable, they are simultaneously vulnerable, and never exhibit a matching penny configuration. Therefore, E ( x,y ) ∼ ρ [ π h µ ( x, y )] = 0 for any µ ∈ P ( H b ) . More details on Appendix C.2.

Application to RECs and Links with [13]. In the case of RECs where µ = ∑ m ∈ [ M ] p m δ h m i.e., h µ is a mixture of M deterministic classifiers in H b = { h m } m ∈ [ M ] , we can instantiate Theorem 3.2 as follows. As the family H svb ( x, y ) is finite, the supremum µ max ( x, y ) is always attained by some subset of simultaneously vulnerable classifiers H max svb ( x, y ) . We can then write:

Alternatively, we can use (8) to write R ϵ ( h µ ) = ∑ m ∈ [ M ] p m E ( x,y ) ∼ ρ 1 { h m ∈ H max svb ( x, y ) } . At each ( x, y ) , testing whether h m ∈ H max svb ( x, y ) reduces to solving a combinatorial optimization problem, as noted in [13]. Any ( x, y ) falls into one of finitely many configurations, depending on which subset of classifiers are vulnerable or simultaneously vulnerable at ( x, y ) . Dbouk and Shanbhag [13] use this to derive upper and lower bounds on R ϵ ( h µ ) . Specifically, [13, Proposition 1] is equivalent to (8) for the special case of RECs. Also, [13, Theorem 1] can be proved as an application of Theorem 3.2 to RECs with M = 2 . To establish the link, one must note that E ( x,y ) ∼ ρ [ π h µ ′ ( x, y )] = 1 2 ρ ( { ( x, y ) ∈ R } ) where R ⊆ X × Y indicates the set of all points where h 1 , h 2 are in matching penny configuration.

4 From Probabilistic to Deterministic Classifiers

In this section, we prove that for any probabilistic binary classifier h , there exists a deterministic classifier h in a 'threshold' hypothesis set H T ( h ) with at least the same adversarial risk. In

Section 4.1 we present the main theorem and in Section 4.2 we apply the theorem to various classes of probabilistic classifiers.

4.1 Reducing a Probabilistic Classifier to a Deterministic One Through Threshold Classifiers

In the case of binary classification, i.e. Y = { 0 , 1 } any distribution ν ∈ P ( Y ) is uniquely determined by a scalar α = ν ( y = 1) ∈ [0 , 1] . Hence, any probabilistic binary classifier is identified with a function h : X → [0 , 1] , where h ( x ) = ν ( y = 1 | x ) ∈ [0 , 1] . Accordingly, ℓ 0 -1 (( x, 0) , h ) = h ( x ) and ℓ 0 -1 (( x, 1) , h ) = 1 -h ( x ) .

Given a probabilistic binary classifier h : X → [0 , 1] and a threshold α ∈ [0 , 1] , the α -threshold classifier h α : X → { 0 , 1 } is defined as h α ( x ) = 1 { h ( x ) > α } , and the threshold hypothesis set (THS) of h is given by H T ( h ) = { h α } α ∈ [0 , 1] . In Theorem 4.1 we show that there exists h α ∗ ∈ H T ( h ) such that R ϵ ( h α ∗ ) ≤ R ϵ ( h ) . The following lemma plays a crucial role in proving Theorem 4.1.

Lemma 4.1. Let h : X → [0 , 1] be any measurable function. For any ≻∈ { >, ≥} , the following inequality holds, and it becomes an equality if h is continuous or takes finitely many values:

Note that Lemma 4.1 is a generalization of the layer-cake representation of h ( x ) given by,

Theorem 4.1. Let h : X → [0 , 1] be any probabilistic binary classifier. Let h α be the α -threshold classifier. Then the following equation holds:

Further, if h is either continuous or takes finitely many values, the first inequality in (12) becomes an equality.

Note that h takes finitely many values in the case of RECs, and h is continuous in the case of INCs and WNCs whenever the noise distribution admits a density. Hence, R ϵ ( h ) = ∫ 1 0 R ϵ ( h α ) dα in all these cases.

Remark 4. Theorem 4.1 says that in the binary case (K = 2), if one is able to consider complex enough hypotheses sets that contain the H T ( h ) , then randomization is not necessary because there is a deterministic classifier with equal or better adversarial risk.

It was very recently shown with a toy example [45, Section 5.2] that Theorem 4.1 does not hold in the multi-class case K > 2 . This example shows that even when the family of probabilistic classifiers considered is very general (all Borel measurable functions), simple data distributions can create a situation in which the optimal classifier is probabilistic, and there is no optimal deterministic classifier. In other words, there is a strict gap in adversarial risk between the best deterministic classifier and the best probabilistic one.

4.2 Applications: Connections to Weighted Ensembles and Randomized Smoothing

In this section, we apply Theorem 4.1 to probabilistic classifiers presented in Section 2.3.

RECs. When µ = ∑ m ∈ [ M ] p m δ h m and Y = { 0 , 1 } , the REC h µ can be written as h µ ( x ) = ∑ M m =1 p m h m ( x ) . Let us introduce the constant classifier h 0 ( x ) = 1 for all x , and p 0 = -α . Then h α ( x ) = 1 { ∑ K m =1 p m h m ( x ) > α } = 1 { ∑ M m =0 p m h m ( x ) > 0 } . This shows that h α is a weighted ensemble , such as those that the boosting algorithm can learn [17].

Further, a REC h built with M base binary classifiers can take at most p ≤ 2 M distinct values, corresponding to all possible partial sums of the weights p m . Let 0 = r 1 ≤ . . . ≤ r p = 1 be

the possible values. Then, for any α ∈ [ r i , r i +1 ) , h α = h r i . Applying Theorem 4.1 yields,

Equation (13) shows that the THS H T ( h ) = { h α } α ∈ [0 , 1] is composed of at most p ≤ 2 M distinct classifiers, each of which is in turn a weighted ensemble. In case of uniform weights i.e. p m = 1 M , there are only M distinct weighted ensembles in H T ( h ) .

INCs. Let h η : X → { 0 , 1 } be the deterministic binary classifiers created from a base classifier h , as defined in Section 2.3. The probabilistic classifier h µ is defined as h µ ( x ) = P η ∼ µ ( h ( x + η ) = 1) . Thus, the α -threshold classifier takes a similar form to the well-known classifier obtained by randomized smoothing .

Randomized smoothing was first introduced in [26] and refined in [8, 27, 42] as a way to create classifiers that are certifiably robust. Given a deterministic classifier h : X → Y and a noise distribution µ ∈ P ( X ) , the smoothed model will output a prediction according to the following rule:

In other words, Equation (14) generalizes the randomized smoothing classifier in the binary case by replacing the 1 2 threshold by α . Theorem 4.1 then states that for any INC, there exists a deterministic randomized smoothing classifier with threshold α that is at least as robust .

5 Families of Binary Classifiers for Which Randomization Does not Help

In Sections 3 and 4, we discussed two types of hypothesis sets for binary classification: 1) base set H b from which a probabilistic classifier is built using some µ ∈ P ( H b ) , and 2) threshold set H T ( h ) which is built from a base probabilistic classifier h . Recapitulating this two-step process, one may define the completion of a base set H b of binary classifiers w.r.t. a set of probability distributions M ⊆ P ( H b ) as, H b ( M ) = ∪ µ ∈M H T ( h µ ) . Observe that H b ⊆ H b ( P ( H b )) . In the following theorem, we show that if H b = H b ( M ) , then probabilistic binary classifiers built from H b using any µ ∈ M do not offer robustness gains compared to the deterministic classifiers in H b .

Theorem 5.1. If H b = H b ( M ) and δ h ∈ M for all h ∈ H b , then inf h ∈H b R ϵ ( h ) = inf µ ∈M R ϵ ( h µ ) .

Proof.

where the first inequality follows from Theorem 4.1 and the second by considering µ = δ h for any h ∈ H b . The desired conclusion follows by observing that the first and last terms in (16) are identical from the assumption H b = H b ( M ) .

A trivial example where the assumptions of Theorem 5.1 hold is when H b is the set of all measurable functions h : X → { 0 , 1 } . Such a H b is commonly used in the study of optimal adversarial risk [2, 40]. In the following corollary, we show that the assumptions of Theorem 5.1 also hold in the case of RECs built using H b that satisfy a 'closure' property.

Corollary 5.1. Let H b be any family of deterministic binary classifiers. Let M = P M ( H b ) ⊂ P ( H b ) be the subset of probability measures over H b defining RECs as in Section 2.3. Let A = { h -1 (1) : h ∈ H b } . If A is closed under union and intersection, then

Remark 5 (Implications on the optimal adversarial classifier) . Pydi and Jog [40, Theorem 8] show the existence of a binary deterministic classifier attaining the minimum adversarial risk over the space of all Lebesgue measurable classifiers in R d . Awasthi et al. [2, Theorem 1] prove a similar result over the space of universally measurable classifiers in R d . For both these families the assumptions of Theorem 5.1 hold for M = P ( H b ) , and so probabilistic classifiers offer no additional advantage over deterministic ones. Further, suppose H ∗ is a finite subset of classifiers achieving optimal adversarial risk. Then by Theorem 3.2, for any µ ∈ P ( H ∗ ) , the expected matching penny gap for h µ must be zero, otherwise we could strictly improve the adversarial risk by considering the mixture built using µ . In other words, any pair of optimally robust binary deterministic classifiers in the settings of [2, 40] can only be in a matching penny configuration on a null subset E ⊂ X × Y .

6 Conclusion and Future Work

In this paper, we have studied the robustness improvements brought by randomization. First, we studied the situation in which one expands a base family of classifiers by considering probability distributions over it. We showed that under some conditions on the data distribution and the configuration of the base classifiers, such a probabilistic expansion could offer gains in adversarial robustness (See Corollary 3.1), characterized by the matching penny gap . These results generalize previous work that focused on RECs [13]. Next, we showed that for any binary probabilistic classifier, there is always another deterministic extension with classifiers of comparable or better robustness. This result is linked with the existence results in [2, 40]. As a direct consequence of this result, we show that in the binary setting, deterministic weighted ensembles are at least as robust as randomized ensembles and randomized smoothing is at least as robust as noise injection.

There are many avenues for future research.

Improving the Matching Penny Gap. An interesting direction would be finding tighter bounds on the matching penny gap for particular and widely used probabilistic classifiers like RECs, INCs and WNCs. It would be interesting to establish the conditions under which each method can offer robustness gains, and to quantify those gains in terms of parameters such as the strength of the noise injected or the number of classifiers in the REC.

Studying the Threshold Hypothesis Sets. We have seen in Section 4.2 that different probabilistic classifiers induce different THS. In particular, we showed the THS corresponding to RECs and INCs. It would be useful to formally describe the THS induced by other popular families of probabilistic classifiers in the literature. It would also be useful to quantify the complexity gap between the initial H b and the THS to understand the risk-complexity trade-off we would have to incur.

Multiclass Setting. The toy example in [45, Section 5.2] shows that randomization might be necessary in the multi-class setting, as it is no longer true that there is always a deterministic optimal classifier, even when considering very general families of deterministic classifiers like all measurable functions. A natural road for future work is to further characterize the situations in which probabilistic classifiers strictly outperform deterministic ones in the multi-class setting.

Training algorithms for diverse ensembles and RECs. There are numerous works based on the idea of training diverse classifiers that are not simultaneously vulnerable to create robust ensembles [11, 24, 33, 41, 49, 50]. Most of these approaches try to induce orthogonality in decision boundaries so that gradient-based attacks do not transfer between models. This intuition is validated by Corollary 3.1, since such diversification would tend to increase the matching penny gap. It should be noted that ensembles and RECs have different inference procedures, and attacking them represents different optimization problems. One avenue of research would be to make the link between the matching penny gap and the diversity more formal. In addition, designing training algorithms explicitly optimizing the matching penny gap would be valuable, particularly because existing training algorithms for RECs [31, 38] have been shown to be inadequate [12].

References

Pranjal Awasthi, Natalie Frank, and Mehryar Mohri. On the existence of the adversarial bayes classifier. Advances in Neural Information Processing Systems , 34:2978-2990, 2021.

Pranjal Awasthi, Natalie S Frank, and Mehryar Mohri. On the existence of the adversarial bayes classifier (extended version). arXiv preprint arXiv:2112.01694 , 2021.

Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndi´ c, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases , pages 387-402. Springer, 2013.

Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation , 7(1):108-116, 1995.

Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM workshop on artificial intelligence and security , pages 3-14, 2017.

Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705 , 2019.

D Charalambos and Border Aliprantis. Infinite Dimensional Analysis: A Hitchhiker's Guide . Springer-Verlag Berlin and Heidelberg GmbH & Company KG, 2006.

Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning , pages 1310-1320. PMLR, 2019.

Yves Crama and Peter L Hammer. Boolean functions: Theory, algorithms, and applications . Cambridge University Press, 2011.

Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning , pages 2206-2216. PMLR, 2020.

Ali Dabouei, Sobhan Soleymani, Fariborz Taherkhani, Jeremy Dawson, and Nasser M Nasrabadi. Exploiting joint robustness to adversarial perturbations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1122-1131, 2020.

Hassan Dbouk and Naresh Shanbhag. Adversarial vulnerability of randomized ensembles. In International Conference on Machine Learning , pages 4890-4917. PMLR, 2022.

Hassan Dbouk and Naresh Shanbhag. On the robustness of randomized ensembles to adversarial perturbations. In International Conference on Machine Learning , pages 7303-7328. PMLR, 2023.

Guneet S. Dhillon, Kamyar Azizzadenesheli, Zachary C. Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Animashree Anandkumar. Stochastic activation pruning for robust adversarial defense. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/forum?id=H1uR4GZRZ .

Dimitrios Diochnos, Saeed Mahloujifar, and Mohammad Mahmoody. Adversarial risk and robustness: General definitions and implications for the uniform distribution. Advances in Neural Information Processing Systems , 31, 2018.

Natalie S Frank and Jonathan Niles-Weed. Existence and minimax theorems for adversarial surrogate risks in binary classification. arXiv preprint arXiv:2206.09098 , 2022.

Yoav Freund, Robert Schapire, and Naoki Abe. A short introduction to boosting. JournalJapanese Society For Artificial Intelligence , 14(771-780):1612, 1999.

Drew Fudenberg and Jean Tirole. Game theory . MIT press, 1991.

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1412.6572 .

Pascale Gourdeau, Varun Kanade, Marta Kwiatkowska, and James Worrell. On the hardness of robust classification. The Journal of Machine Learning Research , 22(1):12521-12549, 2021.

Yves Grandvalet, Stéphane Canu, and Stéphane Boucheron. Noise injection: Theoretical prospects. Neural Computation , 9(5):1093-1108, 1997.

Zhezhi He, Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 588-597, 2019.

Varun Jog. Reverse euclidean and gaussian isoperimetric inequalities for parallel sets with applications. IEEE Transactions on Information Theory , 67(10):6368-6383, 2021.

Sanjay Kariyappa and Moinuddin K. Qureshi. Improving adversarial robustness of ensembles with diversity training. CoRR , abs/1901.09981, 2019. URL http://arxiv.org/abs/1901. 09981 .

Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=BJm4T4Kgx .

Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy (SP) , pages 656-672. IEEE, 2019.

Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. Certified adversarial robustness with additive noise. Advances in neural information processing systems , 32, 2019.

Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 369-385, 2018.

Keane Lucas, Matthew Jagielski, Florian Tramèr, Lujo Bauer, and Nicholas Carlini. Randomness in ml defenses helps persistent attackers and hinders evaluators. arXiv preprint arXiv:2302.13464 , 2023.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/ forum?id=rJzIBfZAb .

Laurent Meunier, Meyer Scetbon, Rafael B Pinot, Jamal Atif, and Yann Chevaleyre. Mixed nash equilibria in the adversarial examples game. In International Conference on Machine Learning , pages 7677-7687. PMLR, 2021.

Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robustness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9078-9086, 2019.

Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning , pages 4970-4979. PMLR, 2019.

Konstantinos Panousis, Sotirios Chatzis, Antonios Alexos, and Sergios Theodoridis. Local competition and stochasticity for adversarial robustness in deep learning. In International Conference on Artificial Intelligence and Statistics , pages 3862-3870. PMLR, 2021.

Konstantinos P Panousis, Sotirios Chatzis, and Sergios Theodoridis. Stochastic local winnertakes-all networks enable profound adversarial robustness. arXiv preprint arXiv:2112.02671 , 2021.

Juan C Perdomo and Yaron Singer. Robust attacks against multiple classifiers. arXiv preprint arXiv:1906.02816 , 2019.

Rafael Pinot, Laurent Meunier, Alexandre Araujo, Hisashi Kashima, Florian Yger, Cédric GouyPailler, and Jamal Atif. Theoretical evidence for adversarial robustness through randomization. Advances in Neural Information Processing Systems , 32, 2019.

Rafael Pinot, Raphael Ettedgui, Geovani Rizk, Yann Chevaleyre, and Jamal Atif. Randomization matters how to defend against strong adversarial attacks. In International Conference on Machine Learning , pages 7717-7727. PMLR, 2020.

Rafael Pinot, Laurent Meunier, Florian Yger, Cédric Gouy-Pailler, Yann Chevaleyre, and Jamal Atif. On the robustness of randomized classifiers to adversarial examples. Machine Learning , 111(9):3425-3457, 2022.

Muni Sreenivas Pydi and Varun Jog. The many faces of adversarial risk: An expanded study. IEEE Transactions on Information Theory , pages 1-1, 2023. doi: 10.1109/TIT.2023.3303221.

Andrew Ross, Weiwei Pan, Leo Celi, and Finale Doshi-Velez. Ensembles of locally independent prediction models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 5527-5536, 2020.

Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Advances in Neural Information Processing Systems , 32, 2019.

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014. URL http://arxiv.org/abs/1312.6199 .

Nicolas Garcia Trillos, Matt Jacobs, and Jakwang Kim. On the existence of solutions to adversarial training in multiclass classification. arXiv preprint arXiv:2305.00075 , 2023.

Nicolás Garcıa Trillos, Matt Jacobs, and Jakwang Kim. The multimarginal optimal transport formulation of adversarial multiclass classification. Journal of Machine Learning Research , 24 (45):1-56, 2023.

John Von Neumann. On the theory of games of strategy. Contributions to the Theory of Games , 4:13-42, 1959.

Bao Wang, Zuoqiang Shi, and Stanley Osher. Resnets ensemble via the feynman-kac formalism to improve natural and robust accuracies. Advances in Neural Information Processing Systems , 32, 2019.

Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. arXiv preprint arXiv:1711.01991 , 2017.

Huanrui Yang, Jingyang Zhang, Hongliang Dong, Nathan Inkawhich, Andrew Gardner, Andrew Touchet, Wesley Wilkes, Heath Berry, and Hai Li. Dverge: diversifying vulnerabilities for enhanced robust generation of ensembles. Advances in Neural Information Processing Systems , 33:5505-5515, 2020.

Zhuolin Yang, Linyi Li, Xiaojun Xu, Shiliang Zuo, Qian Chen, Pan Zhou, Benjamin Rubinstein, Ce Zhang, and Bo Li. Trs: Transferability reduced ensemble via promoting gradient diversity and model smoothness. Advances in Neural Information Processing Systems , 34:17642-17655, 2021.

Tianyuan Yu, Yongxin Yang, Da Li, Timothy Hospedales, and Tao Xiang. Simple and effective stochastic neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 3252-3260, 2021.

Dinghuai Zhang, Hongyang Zhang, Aaron Courville, Yoshua Bengio, Pradeep Ravikumar, and Arun Sai Suggala. Building robust ensembles via margin boosting. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 26669-26692. PMLR, 17-23 Jul 2022.

Supplementary Material

Appendix A Measurability of the ℓ 0 -1 under attack

In this section, we will go over the conditions that ensure the well-definedness of the adversarial risk defined in (1). We will show that the assumptions we make are verified in the settings that are of interest to our work. We highlight that the main purpose of this paper is not to deeply study the existence of measurable solutions to the adversarial problem, as was done in previous work [1, 40, 44]. In this work, we allow ourselves to add hypothesis on the input space X and the families of classifiers H b that we consider, as long as they are valid in practice.

A.1 Discussion

Recall that in our setting we have the following. The input space X is some subset (almost always bounded or even compact) of R d , Y = [ K ] and P ( Y ) = ∆ K . On the other hand, for all the applications of interest to our work, we consider base hypothesis sets H b that are either finite or discrete (for randomized ensembles), or identifiable with subsets of some R p (for weight and input noise injection). For all these sets, we assume the usual topology and the Borel σ -algebra that is generated by the open sets. We further assume that X and H b are Borel spaces , which is not restrictive. It allows, for example, any open or closed set of R n , which already encompasses all practical applications of interest. This hypothesis makes it simpler to work on the product measure space X × H b .

̸

In the adversarial attacks literature, the question of measurability is always with respect to X [2, 40]. For a deterministic classifier h : X → Y = [ K ] to be measurable, the sets h -1 ( k ) must be measurable. In other words, each classifier h creates a K -partition of X into measurable subsets h -1 (1) , . . . , h -1 ( K ) . This immediately translates into the ℓ 0 -1 , as a function of x , being measurable, because ℓ 0 -1 (( · , y ) , h ) -1 (1) = { x ∈ X | h ( x ) = y } = X \ h -1 ( y ) . In other words, assuming classifiers are measurable functions translates into measurability of the 0 -1 loss, so the arguments in previous work [1, 16, 40] can be used to justify that the adversarial risk is well-defined.

̸

We introduce a new component, the BHS H b , and we now compute integrals over it. For this reason, we now consider more general functions of the form f : X × H b →{ 0 , 1 } , f ( x, h ) = 1 { h ( x ) = y } and study their measurability. If one fixes h , the function f h ( x ) = 1 { h ( x ) = y } becomes the same 0 -1 loss considered earlier, so assuming every h is a measurable function translates into the fact that f is measurable w.r.t. x for every fixed h . The new part we have to deal with is the measurability of f w.r.t h for every fixed x .

̸

̸

As a first example, let us consider the case of linear classifiers. In this case, a classifier h can be represented as a K × d matrix A with the classification rule h ( x ) = argmax k ( Ax ) k , meaning h classifies x using the row with higher score. In this case, we can directly see the measurability of f in each component ( f h and f x ) by looking at the pre-images of 1 , which are the sets that induce an error (recall y is fixed). For f h , the pre-image is the set of x ∈ X such that h ( x ) = y . This can be seen as the complement of the set X y in which h predicts y .

̸

The set X y is described by K -1 linear equations using the rows a j of the matrix A that defines the linear classifier h , as follows

̸

This set is the finite intersection of hyperplanes, so it is Borel measurable, and therefore the function f h is measurable on X for any h linear classifier.

For the measurability of f x , we have to consider the set H y of classifiers that produce a prediction of y for a fixed x . Let us recall that each classifier is represented by a matrix A ∈ R K × d , then

̸

H y is a convex cone, because if A,B ∈ H y , then for a positive α , αA ∈ H y and A + B ∈ H y . This set is therefore measurable, and so we can conclude that f x is measurable for every x .

When considering neural networks with weights parametrized by w , we can think of them as nonlinear functions plus a last linear layer as the one described earlier. Then, if the non-linear function is continuous w.r.t both x and w , the function f we are considering is again measurable w.r.t to x and w .

In summary, all the situations that are of interest to our work verify that the function f is defined over a product space R d × R p , and it has good properties when seen as functions of x or h separately (measurable, continuous or differentiable). These separately measurable functions [7, Definition 4.47] might not be jointly measurable , i.e. measurable on the product space, which is one condition that would grant us the well-definedess of (1). Nevertheless, we can make use of the stronger properties these functions have and assume they are Carathéodory functions [7, Definition 4.50]. This will imply the joint measurability of f . For completeness, we rewrite the results from [7]:

Definition 2 (Carahéodory function [7, Definition 4.50]) . Let ( S, Σ) be a measurable space, and let X and Y be topological spaces. Let B Y be the Borel σ -algebra on Y . A function f : S × X → Y is a Carathéodory function if:

For each x ∈ X , the function f x = f ( · , x ) : S → Y is (Σ , B Y ) -measurable.

For each s ∈ S , the function f s = f ( s, · ) : X → Y is continuous.

As we have seen, the functions f : X × H b → R we consider can be assumed to be Carthéodory functions, as they are in general differentiable or continuous in both components. Even going away from practice, in theoretical works it is usual to consider classifiers h : X → ∆ K as Borel measurable functions from X , and adding the assumption of continuity w.r.t h for every fixed x it not restrictive (for example neural networks with continuous activation functions [31]).

Lemma A.1 (Carahéodory functions are jointly measurable [7, Lemma 4.51]) . Let ( S, Σ) be a measurable space, X a separable metrizable space, and Y a metrizable space. Then every Carathéodory function f : S × X → Y is jointly measurable

A.2 Conclusion

If we assume the hypotheses of Lemma A.1, that is

̸

We assume that for every component y , the function f : X × H b → { 0 , 1 } defined as f ( x, h ) = 1 { h ( x ) = y } is a Carathéodory function , which translates in our case into:

Each h ∈ H b , h is a measurable function on X (very common assumption).

For each x , the function f x : H b → { 0 , 1 } , f x ( h ) = f ( x, h ) is continuous (not unrealistic, satisfied by neural networks with continuous activation functions)

The input space X is a measurable space (satisfied by hypothesis, X ⊆ R d is some Borel set).

The base hypothesis set H b is a separable metrizable space (valid when H b is a Borel subset of R p ).

The output space { 0 , 1 } is a metrizable space (satisfied).

then, by Lemma A.1, the function f is jointly measurable.

By Fubini-Tonelli's Theorem, the function x → ∫ H b f ( x, h ) dµ ( h ) is measurable on X . Then, following the same arguments as in [1, 16], the loss under attack can be defined and is measurable over the universal σ -algebra U ( X ) on X . Then, considering U ( X ) , the adversarial risk is well-defined as it is an integral over X using the completions of the class conditioned measures ρ y over U ( X ) (see (1)).

Appendix B On the name Matching penny gap

In the original matching penny game between player 1 ( attacker ) and player 2 ( defender ), each player has a penny coin and has to secretly position its penny in either heads or tails . Then, both coins are simultaneously revealed and the outcome of the game is decided as follows: attacker wins if both pennies match. If they do not match, then defender wins. This situation can be represented using the following matrix ( attacker wants to maximize), where heads and tails have been replaced by f 1 and f 2 for reasons that will be explained later:

Attacker

f

1

f

2

f

1

1

0

f

2

0

1

This game has no Pure Nash Equilibrium. It has, however, a mixed Nash equilibrium which consist of the strategies ( 1 2 , 1 2 ) for both players [18]. In the context of the matching pennies game , this can be seen as the strategy in which players toss the coin to obtain either heads or tails uniformly at random (assuming the coins are fair) instead of choosing one of the sides. Using this strategy, each player can be sure that, on average over multiple realizations of the game, they will win half of the times each. The fact that it is a Nash Equilibrium means that, if one player deviated from this strategy, the other player could find a way to win more than half the times. To see this, imagine that the defender player plays heads with probability 1 2 + δ . Then, if attacker knew this, he could play the strategy always heads . Then, from all the realizations of the game, with probability 1 2 + δ the coins would be both on the same side ( heads ), meaning attacker would win 1 2 + δ of the times, gaining δ w.r.t the equilibrium strategy.

The parallel with our work can be simply explained as follows (see Figure 2): Take a point ( x 0 , y 0 ) and two classifiers f 1 , f 2 that correctly classify x 0 . Suppose that both f 1 and f 2 are vulnerable at x 0 , which is represented by the colored areas in Figure 2. However, add the key assumption that they cannot be attacked at the same time , which is represented by the fact that the colored areas do not intersect inside the ϵ -ball around x 0 . That is, even though an optimal attacker can fool each

classifier individually, there is no point in the allowed perturbation region B ϵ ( x 0 ) in which both are simultaneously fooled.

Consider now that the defender is using a randomized ensemble that picks either f 1 or f 2 at random for inference. This was introduced in [38] with the name mixtures of classifiers, related to mixed strategies in the context of game theory. In such setting, the optimal attacker that faces such mixture is now in a matching pennies game situation. At each inference pass, the attacker must choose which classifier to attack in order to craft the adversarial example x ′ , and if the chosen classifier matches with the one the defender used, then the attacker wins. If they do not match, the prediction made by the defender will be correct and the attacker would have lost. This is exactly the game of matching pennies !

What is particularly worse for defender is that, in the traditional formulation of adversarial attacks, the attacker is given the chance to play second and therefore can adapt to the classifier. In other words, attacker can craft its perturbation for the particular classifier proposed by defender , while defender has to propose a classifier that should be robust to every conceivable attack. This means that if defender uses f 1 or f 2 individually, then attacker would be able to craft an attack that induces an error. In this situation, and inspired by the game of matching pennies , defender can turn these two strategies, that have 0 robust accuracy on their own, into a strategy that guarantees an expected accuracy of 1 2 by randomizing over f 1 and f 2 . Recall that attacker is optimal for any fixed classifier, but it has no control over randomness , which means that when faced against a

mixture of f 1 and f 2 , attacker can compute any perturbation δ from them, but he has no knowledge of which classifier will be used for each independent inference pass . This can be interpreted as follows: for a given inference run, there is a chance of 1 2 that the perturbation proposed by attacker is indeed an adversarial attack and induces an error.

Notice that if we increase the number of choices (classifiers) in matching pennies' configuration to M > 2 , the game becomes harder for the attacker. Indeed, for every possible choice of classifier at each inference pass, there is one out of M choices that lead to a successful attack (attacking the one that was sampled for the inference pass), and M -1 that lead to a correct classification. An extreme example of such benefit to the defender is shown in Example 1.

Appendix C Details on the examples and remarks

C.1 Example 1: Maximum mathing penny gap

Note that for this example, each classifier w satisfies that w T x = 0 , and therefore 1 { w T x < 1 } = 1 , which means that all w predict the correct label for the clean input x . Now we want to see that every w is vulnerable at x .

Recall that we defined H b as the space of linear classifiers w such that ∥ w ∥ 2 = 1 ϵ . We can rewrite this norm as a dual norm ∥ w ∥ 2 = sup ∥ z ∥ =1 w T z . Moreover, this supremum is attained by some z w , so for each w we get z w of norm 1, such that w T z w = 1 ϵ .

(

x,

1)

ϵ

w

ϵ

·

z

w

̸

Now, for each w ∈ H b , consider the adversarial example ϵ · z w . It is a valid adversarial example because it has norm ϵ , and w T ( ϵ · z w ) = ϵ · 1 ϵ = 1 , meaning that the classifier predicts 1 { w T ( ϵ · z w ) < 1 } = 0 , the wrong class. The next step is to guarantee that this perturbation is unique for each w , which holds because we are using the Euclidean norm.

Having that ϵ · z w fools w and only w , we can consider for simplicity µ the uniform distribution over H b . Let us compute the sets H vb ( x, 1) and µ max ( x, 1) = 0 to be able to compute the matching penny gap for this example.

As we just saw, every w is itself vulnerable. This means that H vb ( x, 1) = H b , and therefore µ ( H vb ( x, 1)) = 1 .

Given the unicity of z w , we know that no two classifiers can be fooled by the same perturbation. Therefore the family of simultaneously vulnerable classifiers only contains singletons { w } . As µ ( { w } ) = 0 for every w , taking the supremum gives µ max ( x, 1) = 0 .

Finally, expected the matching penny gap for this example with only one point is

In other words, the mixture in this example has the best adversarial risk possible, even though it is built from classifiers with the worst adversarial risk possible individually.

C.2 Example 2: Minimum mathing penny gap / Parallel sets

To simplify the example, consider the compact set A = { 0 } ⊂ R . Then, the family of all parallel classifiers h r = A ⊕ B r (0) are the classifiers of the form h r ( x ) = 1 { x ∈ ( -r, r ) } for r > 0 .

Recall that a matching penny configuration arises when 1) both classifiers are vulnerable , but 2) not simultaneously . That is, each one of them must be vulnerable, but no allowed perturbation can induce an error on both at the same time. We will see that this cannot happen with this family of classifiers that are 'parallel'.

W.l.o.g, take any point x > 0 and suppose it is of class is 0. Take any two classifiers h r 1 , h r 2 with r 1 < r 2 and fix the attacker budget to ϵ . Note that h r 1 is vulnerable at x if an only if x -ϵ ≤ r 1 . That is, the attacker must be able to move x inside ( -r, r ) with its budget of ϵ . This also holds for h r 2 .

Note that to satisfy the condition that both classifiers are vulnerable , we must ensure that x -ϵ ≤ r 1 and x -ϵ ≤ r 2 . However, any x that satisfies x -ϵ ≤ r 1 immediately satisfies x -ϵ ≤ r 2 , as r 1 < r 2 , meaning that the same perturbation induces an error on both classifiers. In conclusion, they cannot be both individually vulnerable without being simultaneously vulnerable at x .

0

-

r

1

r

1

-

r

2

r

2

h

1

h

2

(a) Example of parallel classifiers in

R

A

r

1

A

r

2

(b) Example of parallel classifiers in

R

2

For the case of general parallel sets A r 1 , A r 2 , note that if r 1 < r 2 then A r 1 ⊂ A r 2 . Assuming that the classifiers predict 1 if the point is inside the set A r , then the condition both classifiers are vulnerable means that a point x of class 0 is at a distance of at most ϵ from both A r 1 , A r 2 . However, for any given x , being ϵ -close to A r 1 implies that x is ϵ -close to A r 2 , so any adversarial example of A r 1 will be also adversarial for A r 2 . In conclusion, they can never be in a matching penny configuration.

C.3 Multiple optimal classifiers in matching penny configuration

In this section, we are going to explain in more detail Remark 2. First, we rewrite Equation (9) which states the condition under which a mixture outperforms the individual classifiers composing it:

For simplicity, assume H b consist of two different classifiers h 1 , h 2 with the same adversarial risk and with a strictly positive matching penny gap , i.e. E ( x,y ) ∼ ρ [ π h µ ( x, y )] > 0 for some

µ = ( µ ( h 1 ) , µ ( h 2 )) . Then the REC h µ that consists of these two classifiers will outperform each one of them. This is because the LHS of equation (9) is positive, while the RHS is exactly 0 because both classifiers have the same risk, so their average risk is equal to the best risk among them. If we add the assumption that h 1 and h 2 were optimal deterministic classifiers from a larger family H b , we would have created a mixture that outperforms all classifiers in it.

If now we try to soften the assumptions on h 1 , h 2 , and say that they have very similar adversarial risks, i.e. R ϵ ( h 2 ) = R ϵ ( h 1 ) + δ for some small δ , then the RHS of (9) is no longer 0 but δ 2 . In this case, for condition (9) to hold we need the matching penny gap to be not only strictly positive, but greater than δ 2 .

The important intuition is that mixing classifiers that have very different adversarial risks is not ideal, as one will then need a much greater expected matching penny gap in order to actually outperform the best individual classifier. Another interesting intuition is that one can create a good mixture from very bad classifiers, as was seen in Example 1, where individual classifiers had adversarial risk of 1 (the worst possible), but mixing them resulted in an adversarial risk of 0 (the best possible). Even if Example 1 is merely a toy example, it highlights the intuition that it is possible for many non-robust classifiers to gain robustness as a mixture if they interact nicely between them and the dataset (have a high expected matching penny gap).

C.4 Example with linear classifiers: Mixtures can improve robustness (Figure 1b)

C2

C1

33

+

C4

fi(c)=1

f2(α)=1

fi(∞)= 0

f2(c)= 0

C.4.1 Detailed description of the example

Here is the exact description of the example that was introduced in Figure 1b. We will use colors to simplify the distinction between points. The points have the following coordinates: x 1 = (0 , 1) , x 2 = ( -2 . 7 , 1 . 1) , x 3 = (2 . 7 , 1 . 1) , x 4 = (0 , -2) . For this extended analysis, it will be easier to work with classes in {-1 , 1 } , so we have labels y 1 = 1 and y 2 = y 3 = y 4 = -1 . We set ϵ = 1 .

Let f = ( w,b ) be a generic linear classifier in R 2 , where w = ( w 1 , w 2 ) is the normal vector of the hyperplane and b is the bias term. The classification is done using the rule sign ( f ( x )) = sign ( w T x + b ) , as in standard binary classification with labels {-1 , 1 } .

In this setting, a point ( x, y ) ∈ R 2 ×{-1 , 1 } is correctly classified if y · f ( x ) > 0 . Moreover, for linear classifiers, the optimal adversarial perturbation is known and robustness at a point can be easily studied. Note that given a perturbation δ , we can check if it induces a misclassification by simply computing y · f ( x + δ ) . We can say f is robust at ( x, y ) if, for every perturbation of norm at most ϵ ,

we have that y · f ( x + δ ) > 0 . This can be developed to get

For f to be robust at ( x, y ) , Equation (17) must hold for all δ , in particular for the quantity maximizing the RHS. This happens for δ ∗ = -yϵ w ∥ w ∥ . Inserting this worst case perturbation, we obtain a simple condition for robustness in the linear case:

Without loss of generality, we can assume w has norm 1 so the last equation further simplifies. We can use these inequalities to show that no linear classifier can be robust at x 1 , x 2 and x 4 at the same time. Indeed, being robust at each point gives us the following inequalities:

Using the inequalities for x 1 and x 4 , we get the condition w 2 > 2 3 . Using the inequalities for x 1 and x 2 gives us that 2 . 7 w 1 -0 . 1 w 2 > 2 . Together with the bound on w 2 , we get the bound w 1 > 62 81 . These two bounds make it impossible for w to have norm 1, contradicting our hypothesis. An analogous reasoning shows that no linear classifier can be robust at x 1 , x 3 and x 4 at the same time.

Knowing that a linear classifier can't be robust at x 1 , x 2 and x 4 simultaneously, we can further ask what is the best adversarial risk one can get. One can easily check that it is possible to be robust on any of the pairs ( x 1 , x 2 ) , ( x 2 , x 4 ) or ( x 1 , x 2 ) . At each time, robustness on the two selected points comes at the expense of non-robustness on the third point. This fact discards the strategy of being robust at the pair ( x 2 , x 4 ) , because being non-robust at x 1 implies a risk of 1 2 , way higher than the risk of 1 6 one would have to pay for being non-robust at x 2 or x 4 . With this being said, the other two solutions are optimal in terms of adversarial risk for linear classifiers, with an adversarial risk of 1 3 . Nevertheless, for building a robust mixture, only one of them will be useful.

C.4.2 Matching pennies situation.

In order to show that the matching pennies situation can exist on x 4 , the simplest thing is to propose two linear classifiers for which it happens. We propose the following linear classifiers:

f 1 = ( w 1 , b 1 ) with w 1 = (0 . 825 , 0 . 565132728) and b 1 = 0 . 536876091 .

f 2 = ( w 2 , b 2 ) with w 2 = ( -0 . 825 , 0 . 565132728) and b 2 = 0 . 536876091 .

Using Equation (18) one can check that these classifiers match the situation that is illustrated in Figure 5, i.e. they are both robust at x 1 , they are robust at x 3 and x 2 respectively, they are non-robust at x 4 but they can't be attacked on the same region, as their intersections with the 1 -ball around x 4 are disjoint (gray circular sectors in Figure 5). Let us denote h the REC f 1 and f 2 with probabilities ( 1 2 , 1 2 ) .

C.4.3 Adversarial risk calculation for the mixture of f 1 and f 2 .

For x 2 , f 1 is not robust, meaning its adversarial 0 -1 loss on x 2 is 1. On the other hand, f 2 is robust on x 2 , as no perturbation of norm at most ϵ will make it predict the wrong class. For these reasons, the adversarial 0 -1 loss on x 2 for h is 1 2 . The same goes for x 3 , and therefore they each add 1 12 to the total adversarial risk (mass of each point times the 0 -1 loss).

At the point x 4 , given the matching pennies situation, the adversarial 0 -1 for h is 1 2 , for a total adversarial risk of 1 12 . As both f 1 and f 2 are robust on x 1 , the adversarial risk on x 1 is 0, and we can conclude that the total adversarial risk of h is 3 12 = 1 4 , which is less than the optimal adversarial risk when considering linear classifiers only, that is 1 3 .

C.5 On the toy example in Trillos et al. [45, Section 5.2]

The example shown in Trillos et al. [45, Section 5.2] involves three points of different classes. We are interested in Case 4-(i), in which each pair of points can be merged into one by the adversary, but the three cannot be merged because their three ϵ -balls have an empty intersection, as depicted in Figure 6. For simplicity, we assume each point has a probability ω i of 1 3 , which satisfies the condition ω 1 < ω 2 + ω 3 on Case 4 - (i) in [45].

x

1

x

2

x

3

Authors show that in this case, the optimal attack consist on distributing the mass of the original points into the two neighbors in such a way that, for each of the barycenters ¯ x ij , the mass coming from each of the original points x i and x j is the same. In our simplified example, this means that each original point sends 1 6 of its mass to each neighboring barycenter, as depicted in Figure 7a. When it comes to an optimal classifier, the one presented in Figure 7b is optimal and coincides with the one described in the original work of [45]. This classifier predicts the original class for each x i inside the part of the ball B ϵ ( x i ) that is not intersecting any of the other balls. For each intersection B ϵ ( x j ) ∩ B ϵ ( x k ) , the classifier predicts class j and k with probability 1 2 each, and the third class i with probability 0. This classifier achieves an adversarial risk of 1 2 , because at any point of the new adversarial distribution that is supported on { ¯ x 12 , ¯ x 13 , ¯ x 23 } , this classifier has probability 1 2 of predicting the correct class.

x

1

x

2

x

3

¯

x

12

1

6

1

6

¯

x

13

1

6

1

6

¯

x

23

1

6

1

6

(a) Optimal attack.

x

1

x

2

x

3

(b) Optimal classifier.

From the perspective of our work, it is interesting that the optimal probabilistic classifier shown in Figure 7b can be built as a uniform REC of 6 deterministic classifiers f ijk that predict i in B ϵ ( x i ) , j in B ϵ ( x j ) \ B ϵ ( x i ) and k in B ϵ ( x k ) \ ( B ϵ ( x i ) ∪ B ϵ ( x j )) (see Figure 8). Each f ijk has standard accuracy of 1, and adversarial accuracy of 1 3 (risk of 2 3 ) against an optimal attack, like the constant

classifiers, because it can only be robust at one point at most. They are optimal when restricting to deterministic classifiers. Additionally, the uniform mixture of these 6 classifiers coincides with the classifier presented in Figure 7b, which has standard accuracy of 1, and adversarial accuracy of 1 2 , meaning an increase in performance of 1 2 -1 3 = 1 6 with respect to the deterministic f ijk .

x

3

x

2

x

1

(a) Deterministic classifier f 123

x

1

x

3

x

2

(b) Deterministic classifier

f

231

One can compute the matching penny gap of this mixture of 6 classifiers in this dataset as follows: at each original point x i , there are only two classifiers that are robust ( f ijk and f ikj ), which means µ ( H vb ( x i , i )) = 4 6 = 2 3 . On the other hand, a single perturbation will not fool all the four vulnerable models. Starting from x i , the attacker might move this point towards x j or x k . Any of these perturbations will fool three of the six classifiers. For example, the attack x 1 → ¯ x 12 will fool the subset { f 213 , f 231 , f 321 } , while the other three classifiers will correctly predict the class 1 at ¯ x 12 . In conclusion, µ max ( x i , i ) = 3 6 = 1 2 . We conclude that at each point, the matching penny gap is exactly 2 3 -1 2 = 1 6 , for a total expected matching penny gap of 1 6 (recall every point had the same mass). The average risk is 2 3 , and the risk of the mixture can be found using Equation (6) from Theorem 3.2 as 2 3 -1 6 = 1 2 .

Appendix D Proofs

Theorem 3.1. For a probabilistic classifier h µ : X → P ( Y ) constructed from a BHS H b using any µ ∈ P ( H b ) , we have R ϵ ( h µ ) ≤ E h ∼ µ [ R ϵ ( h )] .

Proof. For any µ ∈ P ( H b ) , we have the following.

̸

Taking infimum with respect to µ on both sides of the above inequality, we get the following.

̸

̸

For any h ∈ H b , we may choose the Dirac measure µ h that assigns probability 1 to h in order to obtain inf µ ∈P ( H b ) E h ∼ µ [ R ϵ ( h )] ≤ E h ∼ µ h [ R ϵ ( h )] = R ϵ ( h ) . Taking infimum over h ∈ H b , we get the following.

The remaining assertion of the theorem follows by combining (19) with (20).

Theorem 3.2. For a probabilistic classifier h µ : X → P ( Y ) constructed from a BHS H b using any µ ∈ P ( H b ) ,

̸

Proof. Observe that for any h ∈ H b , sup x ′ ∈ B ϵ ( x ) 1 { h ( x ′ ) = y } = 1 if and only if h ∈ H vb ( x, y ) . Hence,

̸

Taking expectation on both sides with respect to ( x, y ) ∼ ρ , we get

̸

where the second equality follows from switching the order of the two preceding expectations. For any x ′ ∈ B ϵ ( x ) ,

̸

̸

where the above inequality holds because the µ measure of any subset of H b that is simultaneously vulnerable at some x ′ ∈ B ϵ ( x ) is at most µ max ( x, y ) . Taking supremum over all x ′ ∈ B ϵ ( x ) in the above inequality, we get the following.

̸

We will now show that the above inequality also holds in the other direction.

Let {H k } ∞ k =1 be a sequence of sets, H k ∈ H svb ( x, y ) , such that lim k →∞ µ ( H k ) = µ max ( x, y ) . For each H k , we have by definition of H svb ( x, y ) that there exists some x k ∈ B ϵ ( x ) such that all classifiers h ∈ H k are fooled by x k . In other words, the total mass of classifiers that are fooled by x k is greater or equal to µ ( H k ) . This gives

̸

̸

̸

This is true for every k , so taking k → ∞ we get that sup x ′ ∈ B ϵ ( x ) E h ∼ µ [ 1 { h ( x ′ ) = y } ] ≥ lim k →∞ µ ( H k ) = µ max ( x, y ) .

Taking expectation on both sides with respect to ( x, y ) ∼ ρ , we get

̸

Combining (22) and (21) yields (6).

̸

Corollary 3.1. For µ ′ ∈ P ( H b ) , R ϵ ( h µ ′ ) < inf h ∈H b R ϵ ( h ) if and only if the following condition holds.

Additionally, inf µ ∈P ( H b ) R ϵ ( h µ ) < inf h ∈H b R ϵ ( h ) if and only if there exists µ ′ ∈ P ( H b ) for which (9) holds.

Proof. Suppose (9) holds for some µ ′ ∈ P ( H b ) . Then,

where the first equality follows from Theorem 3.2 and the second inequality follows from the assumption in (9). Suppose (9) does not hold for µ ′ ∈ P ( H b ) . Then,

Suppose (9) does not hold for any µ ′ ∈ P ( H b ) , then taking infimum with respect to µ ′ ∈ P ( H b ) in the above inequality, we get inf µ ′ ∈P ( H b ) R ϵ ( h µ ′ ) ≥ inf h ∈H b R ϵ ( h ) .

Lemma 4.1. Let h : X → [0 , 1] be any measurable function. For any ≻∈ { >, ≥} , the following inequality holds, and it becomes an equality if h is continuous or takes finitely many values:

Proof. As both functions only take the values 0 and 1 , it suffices to show that if the RHS is equal to 1 , then so is the LHS. Suppose sup x ′ ∈ B ϵ ( x ) 1 { h ( x ′ ) ≻ α } = 1 . As the function 1 { h ( x ′ ) ≻ α } takes only a finite number of values, this implies that there exists some x ∗ ∈ B ϵ ( x ) such that 1 { h ( x ∗ ) ≻ α } = 1 . This means that h ( x ∗ ) ≻ α , and therefore sup x ′ ∈ B ϵ ( x ) h ( x ′ ) ≻ α . This makes the LHS equal to 1 .

If we further assume that h is continuous or takes a finite number of values, then if the LHS is equal to one,

Theorem 4.1. Let h : X → [0 , 1] be any probabilistic binary classifier. Let h α be the α -threshold classifier. Then the following equation holds:

Further, if h is either continuous or takes finitely many values, the first inequality in (12) becomes an equality.

Proof. We begin the proof by rewriting the adversarial risk of h in terms of the adversarial risk for each class. To alleviate notation, we denote ℓ 0 -1 ϵ (( x, y ) , h ) = sup x ′ ∈ B ϵ ( x ) ℓ 0 -1 (( x ′ , y ) , h ) . Recall from Equation (1) that

Let us now develop the terms R y ϵ ( h ) . For any ≻∈ { >, ≥} we obtain the following:

In the last equation, we were able to interchange the two integrals using Tonnelli's theorem. Indeed, the function ( x, α ) ↦→ 1 { ℓ 0 -1 ϵ (( x, y ) , h ) ≻ α } is Lebesgue measurable in X × R if ℓ 0 -1 ϵ (( · , 0) , h ) is Lebesgue measurable, which is the case [40, Theorem 1].

The next step is to use Lemma 4.1 to interchange the supremum operator and the indicator function. This will make the adversarial risk of the h α appear. For y = 0 and replacing the operator ≻ by > , we obtain the following:

Now we do a similar development for y = 1 and replacing the operator ≻ by ≥ to obtain:

In the last equation, the change of variable u = 1 -α allows us to complete the calculations and obtain the desired result. Putting everything together, we obtain the following result about the original probabilistic classifier h :

In particular, Equation (27) implies that R ϵ ( h ) ≥ min α R ϵ ( h α ) , meaning that for any h probabilistic binary classifier, there is always a deterministic α -threshold classifier with better or equal adversarial risk.

Corollary 5.1. Let H b be any family of deterministic binary classifiers. Let M = P M ( H b ) ⊂ P ( H b ) be the subset of probability measures over H b defining RECs as in Section 2.3. Let A = { h -1 (1) : h ∈ H b } . If A is closed under union and intersection, then

Proof. Theorem 4.1 applied to RECs (see Section 4.2) tells us that H b ( P M ( H b )) is the set of all weighted ensembles built from H b . It is clear that for any h ∈ H b , δ h ∈ P M ( H b ) , so H b ⊆ H b ( P M ( H b )) . Let us now show the other inclusion.

Take any µ ∈ P M ( H b ) and consider any weighted ensemble h α over H b of the form h α ( x ) = 1 { ∑ m ∈ [ M ] p m h m > α } with p m = µ ( h m ) . Define the function g α : { 0 , 1 } M → { 0 , 1 } as g α ( z 1 . . . z M ) = 1 { ∑ M m =1 p m z m > α } . Then, h α can be written as g α ( h 1 ( x ) . . . h M ( x )) . Because the p m are positive, the function g α is a monotone boolean function. As any monotone boolean function, g α can be written as a disjunctive normal form (DNF) without negations [9]. Thus, the set A ENS = { x ∈ X : h α ( x ) = 1 } is a union of intersections over the sets A 1 . . . A M where A k = h -1 m (1) . Because A is closed under union and intersection, A ENS ∈ A , which means that h α ∈ H b . Thus, H b ( P M ( H b )) ⊆ H b .

As all the hypothesis of Theorem 5.1 are met, we can conclude that inf h ∈H b R ϵ ( h ) = inf µ ∈P M ( H b ) R ϵ ( h µ ) .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates randomized classifiers and their robustness against adversarial examples. The authors' contributions can be found along three different axes. First, the authors give the necessary and sufficient conditions so that from a set of deterministic classifiers one can build a randomized classifier that outperforms the best deterministic classifier from the original set (w.r.t. adversarial risk). In sequence the authors present some results in the opposite direction in the binary classification case; e.g., for every randomized ensemble classifier there exists a deterministic weighted ensemble with better adversarial risk. Finally, the authors identify the conditions under which randomization can provably help on a deterministic base hypothesis set. \n\nAfter rebuttal: Upgraded my score from \"Weak Accept\" to \"Accept\".\n\nSTRENGTHS:\nSome of the ideas of the paper are building on ideas and findings of prior work by Dbouk and Shanbhag; namely, the observation that a randomized classifier can achieve a decisive advantage over any deterministic classifier, because the adversary can only force a misclassification on a subset of all vulnerable classifiers, where \"vulnerable\" means classifiers that can be fooled with perturbations of certain magnitude. This observation and the continuation of the work is interesting and the paper is well-written for the largest part, though, near the end I found it more dense than what I would have preferred it to be. I expect this paper to influence to some extent the work that will be done on the robustness of randomized classifiers in the near future. Overall an interesting paper with interesting findings, though, for completeness, I would like to see the authors add some information in the supplementary material even for simple claims; e.g., clearly write down the calculations for Figure 1b, or clarify other small remarks that they have here and there. I also like Section 6; the conclusion that the authors have written -- a clear summary of their contributions and a summary of ideas related to their work that can be explored as future work.\n\nWEAKNESSES:\nFigure 1 appears 2 pages after it is first mentioned. One page difference is ok, but two pages is strange.\n\nThe authors assume that the label y does not change within the perturbation budget $\\epsilon$ around $x$. While this is a reasonable assumption, and in fact it is the prevailing assumption in adversarial robustness, nevertheless, it is not the only assumption one can make. Clarifying this and citing any of the following two papers (or both?) so that someone can find more information about the topic, I believe can be important and potentially also motivate further avenues for randomized classification. The two papers are:\n\n[M1] Adversarial Risk and Robustness: General Definitions and Implications for the Uniform Distribution\nhttps://proceedings.neurips.cc/paper_files/paper/2018/file/3483e5ec0489e5c394b028ec4e81f3e1-Paper.pdf\n\n[M2] On the Hardness of Robust Classification\nhttps://www.jmlr.org/papers/volume22/20-285/20-285.pdf\n\nLine 102: Write down explicitly the adversarial risk of a deterministic classifier in the appendix, if it does not fit in the main part of the paper, so that there is no ambiguity.\n\nI think the \"Matching Penny\" notions are not clearly defined in the paper. It is apparent that some of the ideas have appeared in prior work but the authors spend nearly no time explaining the original ideas and the original context.  I believe this should change.  In particular, the authors cite two books in lines 168-170 without saying much more; the reader may have no idea about these books and this line of work.\n\nIt is quite unfortunate that Definition 1 splits between pages because the notation is a bit overloaded and it is clarified on the following page.\n\nQUESTIONS:\nQ1. In lines 144-146 it is stated \"Consequently, we get that there always exists a probabilistic classifier that is at least as good as the best deterministic classifier in the BHS $\\mathcal{H}_b$.\" Can you please explain why and how this follows from the previous sentence?\n\nQ2. In lines 210-211 we have \"Observe that every classifier in $H_b$ is vulnerable at $(x,y) = (0,1)$ and so $R_{\\epsilon}(h) = 1$ for all $h\\in H_b$\". I am not sure I follow; since $x = 0$, then $w^T x = 0$ and therefore 1{$w^Tx < 1$} should return 1, which is really the label $y$. Can you explain?\n\nLIMITATIONS:\nI think the paper is ok in this regard, especially in the light of the clear ideas for future work that one can pursue and are listed in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSeveral proposed adversarial defenses involve randomized classifiers. Prior theoretical work analyzes both the randomized and non-randomized cases. This paper answers: 1) When does randomization 'help' in adversarial classification? 2) When does it suffice to consider deterministic classifiers?\n\nThe authors start by describing three common paradigms ( randomized ensemble classifier (REC), weight-noise injection classifier (WNC), input-noise injection classifier (INC)) for constructing randomized classifiers. Subsequently, the authors give a condition describing when randomized classifier over a hypothesis set $\\mathcal H_b$ will outperform deterministic classifiers. Next, the authors show that one can always threshhold a binary randomized classifier to get a better performing deterministic classifier. Lastly, the authors show that if $\\cH_b$ is all measurable sets, then randomization over $\\cH_b$ does not improve the adversarial risk.\n\nSTRENGTHS:\n- The paper answers and important question in the field of adversarial learning: when does randomization help with adversarial classification?\n- Proofs seem simple and clear\n\nWEAKNESSES:\nThe exposition is not great. Here are some specific issues.\n0. Please state your threat model. Your adversary has access to the randomized classifier, but does not know which random function the learner uses every time a point is classified.  \n1. Equation 1:  $\\mathcal R_\\epsilon ^y$ is not defined before equation (1)\nNote that you also need to explain that $\\sup_{x'\\in B_\\e(x)} \\ell^{0-1}((x',y),h)$ is measureable. See appendix A of the following reference for this result:\nExistence and minimax theorems for adversarial surrogate risks in binary classification. N. S. Frank, J. Niles-Weed.\n2. Foot note 1: I understand how such a proof would go, but I would expect to see a proof with a maximizing sequence rather than a maximizer in the supplementary material\n3. I couldn't understand the argument in line 191-192\n4. lines 203-204: I couldn't understand the sentence. \"More generally, assumption...\"\n5. I had a hard time understanding the parallel decision boundaries discussion in example 2\n6. line 276: I think an REC built from 2 base classifiers takes on continuously many values. Can you better explain 276-278?\n7. The supplementary material is quite disorganized\n\nQUESTIONS:\n- line 108: The Borel sigma algebra is defined in terms of the open sets on a topological space. How are you defining the topology on the set $\\mathcal H_b$? It seems that in the REC, ENC, INC paradigms, there is a \"continuous\" map from $\\mathbb R^d$ to $\\mathcal H_b$. Perhaps you could use this map. \n- Can you explain the name \"penny matching gap\"?\n- In section 4.1, I would expect that one can always take the threshhold $\\alpha=1/2$. Why is this not the case?\n\nLIMITATIONS:\nSome of the results only hold in the binary case. The paper discusses this limitation", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an analysis of the role of randomization in developing adversarially robust classifiers. It explores the conditions under which randomized ensembles outperform deterministic classifiers regarding adversarial risk. The study also highlights the existence of deterministic classifiers that is at least the same adversarial risk as the probabilistic model, where the authors also connect the theory to weighted ensembles and randomized smoothing. Overall, the paper offers valuable insights into randomization for building robust classifiers in adversarial settings.\n\nSTRENGTHS:\n1. The paper is well-written, ensuring clarity and ease of understanding for readers.\n\n2. The paper offers a foundational understanding of the significance of randomization in building robust models. Especially, I find constructing a probabilistic model from a deterministic model and back particularly interesting. \n\n3. The paper includes an insightful analysis of existing methods (e.g., randomized smoothing ), showcasing their effectiveness.\n\nWEAKNESSES:\n1. While the theoretical analysis presented in this paper is intriguing, it leaves me questioning its practical implications in constructing robust classifiers. Specifically, I am concerned about how these theoretical findings translate into practical strategies for building classifiers that perform effectively in real-world scenarios. For example, Theorem 4, in its current form, only addresses the binary case, limiting its applicability. I think the authors can add more discussion on how to guide the future empirical robust model design. \n\n2. The authors show there exists \"a probabilistic classifier that is at least as robust as the best deterministic classifier.\" However, I think it is more important to guide future research to find such a better probabilistic classifier in a principled way.\n\n3. Minor: It is recommended to explain the term \"Borel σ-algebra.\" \n4. Minor: Moving the fig1 to near page 4 is better. \n\nAs I am not an expert in theoretical research, evaluating the novelty and contribution is hard for me.\n\nQUESTIONS:\n1. In adversarial literature, we might consider the attackers can submit malicious input multiple times, where one misclassification is considered a \"success attack.\" I believe the probabilistic model might be more likely to be broken. What happens to the current theorem if considering such an attacking scenario? I found a recent work [1] which is considering this issue. \n\n2. This paper considers a BHS composed of infinitely many linear classifiers. How does the theorem change if BHS is composed of non-linear classifiers? \n\n3. The paper mainly discussed robustness. How does the clean performance change of building a probabilistic model from deterministic models and back?\n\n[1] Lucas et al. Randomness in ML Defenses Helps Persistent Attackers and Hinders Evaluators. https://arxiv.org/pdf/2302.13464.pdf\n\nLIMITATIONS:\nThe authors are suggested to include a paragraph discussing the limitations of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
The paper addresses the theoretical problem of how randomization affects adversarial robustness. It establishes a condition that is both necessary and sufficient for enhancing the robustness of probabilistic models. Furthermore, it proves that in binary classification, a deterministic model can always achieve at least the same level of robustness. The paper also discusses the crucial condition for the effectiveness of randomization. Although the assumptions are strict, the theories in this work can serve as a foundation to further investigate adversarial robustness in probabilistic classifiers.

STRENGTHS:
1. The paper is well-written and well-organized.
2. The paper works on an interesting theoretical problem of the effect of using randomization on adversarial robustness. The theoretical result gives a promising hint to better understand the randomization methods.

WEAKNESSES:
1. The theories and proofs in this work heavily rely on binary classifiers with a misleading title. This restricts the practical application of these theories in real randomization scenarios.
2. The work presented lacks in applying and evaluating the theory, even in binary cases. It does not explore the potential of the proposed theory to improve randomization or identify scenarios where a deterministic classifier is possible. For instance, can we find or estimate better deterministic models corresponding to baseline probabilistic classifiers? How can we apply the theories to enhance adversarial robustness? Including quantitative evaluations would significantly improve the quality of the work.

QUESTIONS:
N.A.

LIMITATIONS:
See weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27