Title: Adversarial Examples Might be Avoidable: The Role of Data Concentration in Adversarial Robustness
Venue: NeurIPS 2023
Accepted: yes
Keywords: Adversarial Robustness

Geometry in Data

Low Dimensional Modeling
Abstract: The susceptibility of modern machine learning classifiers to adversarial examples has motivated theoretical results suggesting that these might be unavoidable. However, these results can be too general to be applicable to natural data distributions. Indeed, humans are quite robust for tasks involving vision. This apparent conflict motivates a deeper dive into the question: Are adversarial examples truly unavoidable? 
In this work, we theoretically demonstrate that a key property of the data distribution -- concentration on small-volume subsets of the input space -- determines whether a robust classifier exists. We further demonstrate that, for a data distribution concentrated on a union of low-dimensional linear subspaces, utilizing structure in data naturally leads to classifiers that enjoy data-dependent polyhedral robustness guarantees, improving upon methods for provable certification in certain regimes.

Adversarial Examples Might be Avoidable: The Role of Data Concentration in Adversarial Robustness

Abstract

The susceptibility of modern machine learning classifiers to adversarial examples has motivated theoretical results suggesting that these might be unavoidable. However, these results can be too general to be applicable to natural data distributions. Indeed, humans are quite robust for tasks involving vision. This apparent conflict motivates a deeper dive into the question: Are adversarial examples truly unavoidable? In this work, we theoretically demonstrate that a key property of the data distribution - concentration on small-volume subsets of the input space - determines whether a robust classifier exists. We further demonstrate that, for a data distribution concentrated on a union of low-dimensional linear subspaces, utilizing structure in data naturally leads to classifiers that enjoy data-dependent polyhedral robustness guarantees, improving upon methods for provable certification in certain regimes.

1 Introduction, Motivation and Contributions

Research in adversarial learning has shown that traditional neural network based classification models are prone to anomalous behaviour when their inputs are modified by tiny, human-imperceptible perturbations. Such perturbations, called adversarial examples, lead to a large degradation in the accuracy of classifiers [55]. This behavior is problematic when such classification models are deployed in security sensitive applications. Accordingly, researchers have and continue to come up with defenses against such adversarial attacks for neural networks.

Such defenses [49, 60, 42, 22] modify the training algorithm, alter the network weights, or employ preprocessing to obtain classifiers that have improved empirical performance against adversarially corrupted inputs. However, many of these defenses have been later broken by new adaptive attacks [1, 8]. This motivated recent impossibility results for adversarial defenses, which aim to show that all defenses admit adversarial examples. While initially such results were shown for specially parameterized data distributions [18], they were subsequently expanded to cover general data distributions on the unit sphere and the unit cube [48], as well as for distributions over more general manifolds [12].

On the other hand, we humans are an example of a classifier capable of very good (albeit imperfect [17]) robust accuracy against glyph[lscript] 2 -bounded attacks for natural image classification. Even more, a large body of recent work has constructed certified defenses [11, 63, 10, 29, 19, 54] which obtain non-trivial performance guarantees under adversarially perturbed inputs for common datasets like MNIST, CIFAR-10 and ImageNet. This apparent contention between impossibility results and the existence of robust classifiers for natural datasets indicates that the bigger picture is more nuanced, and motivates a closer look at the impossibility results for adversarial examples.

Our first contribution is to show that these results can be circumvented by data distributions whose mass concentrates on small regions of the input space. This naturally leads to the question of whether such a construction is necessary for adversarial robustness. We answer this question in the affirmative, formally proving that a successful defense exists only when the data distribution concentrates on an

exponentially small volume of the input space. At the same time, this suggests that exploiting the inherent structure in the data is critical for obtaining classifiers with broader robustness guarantees.

Surprisingly, almost 1 all certified defenses do not exploit any structural aspects of the data distribution like concentration or low-dimensionality. Motivated by our theoretical findings, we study the special case of data distributions concentrated near a union of low-dimensional linear subspaces, to create a certified defense for perturbations that go beyond traditional glyph[lscript] p -norm bounds. We find that simply exploiting the low-dimensional data structure leads to a natural classification algorithm for which we can derive norm-independent polyhedral certificates. We show that our method can certify accurate predictions under adversarial examples with an glyph[lscript] p norm larger than what can be certified by applying existing, off-the-shelf methods like randomized smoothing [11]. Thus, we demonstrate the importance of structure in data for both the theory and practice of certified adversarial robustness.

More precisely, we make the following main contributions in this work:

We formalize a notion of ( glyph[epsilon1], δ ) -concentration of a probability distribution q in Section 2, which states that q assigns at least 1 -δ mass to a subset of the ambient space having volume O (exp ( -nglyph[epsilon1] )) . We show that ( glyph[epsilon1], δ ) -concentration of q is a necessary condition for the existence of any classifier obtaining at most δ error over q , under perturbations of size glyph[epsilon1] .

We find that ( glyph[epsilon1], δ ) -concentration is too general to be a sufficient condition for the existence of a robust classifier, and we follow up with a stronger notion of concentration in Section 3 which is sufficient for the existence of robust classifiers. Following this stronger notion, we construct an example of a strongly-concentrated distribution, which circumvents existing impossibility results on the existence of robust classifiers.

We then consider a data distribution q concentrated on a union of low-dimensional linear subspaces in Section 4. We construct a classifier for q that is robust to perturbations following threat models more general than glyph[lscript] p . Our analysis results in polyhedral certified regions whose faces and extreme rays are described by selected points in the training data.

Weperform empirical evaluations on MNIST in Section 5, demonstrating that our certificates are complementary to existing off-the-shelf approaches like Randomized Smoothing (RS), in the sense that both methods have different strengths. In particular, we demonstrate a region of adversarial perturbations where our method is certifiably robust, but RS is not. We then combine our method with RS to obtain certificates that enjoy the best of both worlds.

2 Existence of Robust Classifier Implies Concentration

We will consider a classification problem over X × Y defined by the data distribution p such that X is bounded and Y = { 1 , 2 , . . . , K } . We let q k denote the conditional distribution p X | Y = k for class k ∈ Y . We will assume that the data is normalized, i.e., X = B glyph[lscript] 2 (0 , 1) , and the boundary of the domain is far from the data, i.e., for any x ∼ q k , an adversarial perturbation of glyph[lscript] 2 norm at most glyph[epsilon1] does not take x outside the domain X . 2

We define the robust risk of a classifier f : X → Y against an adversary making perturbations whose glyph[lscript] 2 norm is bounded by glyph[epsilon1] as 3

glyph[negationslash]

We can now define a robust classifier in our setting.

Definition 2.1 (Robust Classifier) . A classifier g is defined to be ( glyph[epsilon1], δ ) -robust if the robust risk against perturbations with glyph[lscript] 2 norm bounded by glyph[epsilon1] is at most δ , i.e., if R ( g, glyph[epsilon1] ) ≤ δ .

The goal of this section is to show that if our data distribution p admits an ( glyph[epsilon1], δ ) -robust classifier, then p has to be concentrated . Intuitively, this means that p assigns a 'large' measure to sets of 'small' volume. We define this formally now.

Definition 2.2 (Concentrated Distribution) . A probability distribution q over a domain X ⊆ R n is said to be ( C, glyph[epsilon1], δ ) -concentrated, if there exists a subset S ⊆ X such that q ( S ) ≥ 1 -δ but Vol( S ) ≤ C exp( -nglyph[epsilon1] ) . Here, Vol denotes the standard Lebesgue measure on R n , and q ( S ) denotes the measure of S under q .

With the above definitions in place, we are ready to state our first main result.

Theorem 2.1. If there exists an ( glyph[epsilon1], δ ) -robust classifier f for a data distribution p , then at least one of the class conditionals q 1 , q 2 , . . . , q K , say q ¯ k , must be ( ¯ C,glyph[epsilon1], δ ) -concentrated. Further, if the classes are balanced, then all the class conditionals are ( C max , glyph[epsilon1],Kδ ) -concentrated. Here, ¯ C = Vol { x : f ( x ) = ¯ k } , and C max = max k Vol { x : f ( x ) = k } are constants dependent on f .

The proof is a natural application of the Brunn-Minkowski theorem from high-dimensional geometry, essentially using the fact that an glyph[epsilon1] -shrinkage of a high-dimensional set has very small volume. We provide a brief sketch here, deferring the full proof to Appendix A.

Proof Sketch. Due to the existence of a robust classifier f , i.e. , R ( f, glyph[epsilon1] ) ≤ δ , the first observation is that there must be at least one class which is classified with robust accuracy at least 1 -δ . Say this class is k , and the set of all points which do not admit an glyph[epsilon1] -adversarial example for class k is S . Now, the second step is to show that S has the same measure (under q k ) as the glyph[epsilon1] -shrinkage (in the glyph[lscript] 2 norm) of the set of all points classified as class k . Finally, the third step involves using the Brunn-Minkowski theorem, to show that this glyph[epsilon1] -shrinkage has a volume O (exp( -nglyph[epsilon1] )) , thus completing the argument.

Discussion on Theorem 2.1. We pause here to understand some implications of this result.

Firstly, recall the apparently conflicting conclusions from Section 1 between impossibility results (suggesting that robust classifiers do not exist) and the existence of robust classifiers in practice (such as that of human vision for natural data distributions). Theorem 2.1 shows that whenever a robust classifier exists, the underlying data distribution has to be concentrated. In particular, this suggests that natural distributions corresponding to MNIST, CIFAR and ImageNet might be concentrated. This indicates a resolution to the conflict: concentrated distributions must somehow circumvent existing impossibility results. Indeed, this is precisely what we will show in Section 3.

Secondly, while our results are derived for the glyph[lscript] 2 norm, it is not very hard to extend this reasoning to general glyph[lscript] p norms. In other words, whenever a classifier robust to glyph[lscript] p -norm perturbations exists, the underlying data distribution must be concentrated.

Thirdly, Theorem 2.1 has a direct implication towards classifier design. Since we now know that natural image distributions are concentrated, one should design classifiers that are tuned for small-volume regions in the input space. This might be the deeper principle behind the recent success [66] of robust classifiers adapted to glyph[lscript] p -ball like regions in the input space.

Finally, the extent of concentration implied by Theorem 2.1 depends on the classifier f , via the parameters glyph[epsilon1], δ and ¯ C . On one hand, we get high concentration when glyph[epsilon1] is large, δ is small, and ¯ C is small. On the other hand, if the distribution p admits an ( glyph[epsilon1], δ ) -robust classifier such that ¯ C is large (e.g., a constant classifier), then we get low concentration via Theorem 2.1. This is not a limitation of our proof technique, but a consequence of the fact that some simple data distributions might be very lowly concentrated, but still admit very robust classifiers, e.g., for a distribution having 95% dogs and 5% cats, the constant classifier which always predicts 'dog' is quite robust.

We have thus seen that data concentration is a necessary condition for the existence of a robust classifier. A natural question is whether it is also sufficient. We address this question now.

3 Strong Concentration Implies Existence of Robust Classifier

Say our distribution p is such that all the class conditionals q 1 , q 2 , . . . , q k are ( C, glyph[epsilon1], δ ) -concentrated. Is this sufficient for the existence of a robust classifier? The answer is negative, as we have not precluded the case where all of the q k are concentrated over the same subset S of the ambient space. In other words, it might be possible that there exists a small-volume set S ⊆ X such that q k ( S ) is high for all k . This means that whenever a data point lies in S , it would be hard to distinguish

which class it came from. In this case, even an accurate classifier might not exist, let alone a robust classifier 4 . To get around such issues, we define a stronger notion of concentration, as follows.

glyph[negationslash]

Definition 3.1 (Strongly Concentrated Distributions) . A distribution p is said to be ( glyph[epsilon1], δ, γ ) -stronglyconcentrated if each class conditional distribution q k is concentrated over the set S k ⊆ X such that q k ( S k ) ≥ 1 -δ , and q k ( ⋃ k ′ = k S +2 glyph[epsilon1] k ′ ) ≤ γ , where S + glyph[epsilon1] denotes the glyph[epsilon1] -expansion of the set S in the glyph[lscript] 2 norm, i.e., S + glyph[epsilon1] = { x : ∃ ¯ x ∈ S such that ‖ x -¯ x ‖ 2 ≤ glyph[epsilon1] } .

In essence, Definition 3.1 states that each of the class conditionals are concentrated on subsets of the ambient space, which do not intersect too much with one another 5 . Hence, it is natural to expect that we would be able to construct a robust classifier by exploiting these subsets. Building upon this idea, we are able to show Theorem 3.1:

Theorem 3.1. If the data distribution p is ( glyph[epsilon1], δ, γ ) -strongly-concentrated, then there exists an ( glyph[epsilon1], δ + γ ) -robust classifier for p .

The basic observation behind this result is that if the conditional distributions q k had disjoint supports which were well-separated from each other, then one could obtain a robust classifier by predicting the class k on the entire glyph[epsilon1] -expansion of the set S k where the conditional q k concentrates, for all k . To go beyond this idealized case, we can exploit the strong concentration condition to carefully remove the intersections at the cost of at most γ in robust accuracy. We make these arguments more precise in the full proof, deferred to Appendix B, and we pause here to note some implications for existing results.

Implications for Existing Impossibility Results. To understand how Theorem 3.1 circumvents the previous impossibility results, consider the setting from [48] where the data domain is the sphere S n -1 = { x ∈ R n : ‖ x ‖ 2 = 1 } , and we have a binary classification setting with class conditionals q 1 and q 2 . The adversary is allowed to make bounded perturbations w.r.t. the geodesic distance. In this setting, it can be shown (see [48, Theorem 1]) that any classifier admits glyph[epsilon1] -adversarial examples for the minority class (say class 1 ), with probability at least

where α q 1 = sup x ∈ S n -1 q 1 ( x ) depends on the conditional distribution q 1 , and β is a normalizing constant that depends on the dimension n . Note that this result assumes little about the conditional q 1 . Now, by constructing a strongly-concentrated data distribution over the domain, we will show that the lower bound in (2) becomes vacuous.

Example 3.1. The data domain is the unit sphere S n -1 equipped with the geodesic distance d . The label domain is { 1 , 2 } . P is an arbitrary, but fixed, point lying on S n -1 . The conditional density of class 1 , i.e., q 1 is now defined as

where C = 0 . 1 is a normalizing constant. The conditional density of class 2 is defined to be uniform over the complement of the support of q 1 , i.e. q 2 = Unif( { x ∈ S n -1 : d ( x, P ) > 0 . 1 } ) . Finally, the classes are balanced, i.e., p Y (1) = p Y (2) = 1 / 2 .

The data distribution constructed in Example 3.1 makes Eq. (2) vacuous, as the supremum over the density q 1 is unbounded. Additionally,

P

the linear classifier defined by the half-space { x : 〈 x, P 〉 ≤ cos(0 . 1) } is robust (Appendix C provides a derivation of the robust risk, and further comments on generalizing this example). Example 3.1 is plotted for n = 3 dimensions in Fig. 1.

glyph[negationslash]

glyph[negationslash]

Compatibility of Theorem 3.1 with existing Negative Results Thus, we see that strongly concentrated distributions are able to circumvent existing impossibility results on the existence of robust classifiers. However, this does not invalidate any existing results. Firstly, measure-concentrationbased results [48, 18, 12] provide non-vacuous guarantees given a sufficiently flat (not concentrated) data distribution, and hence do not contradict our results. Secondly, our results are existential and do not provide, in general, an algorithm to construct a robust classifier given a strongly-concentrated distribution. Hence, we also do not contradict the existing stream of results on the computational hardness of finding robust classifiers [6, 56, 47]. Our positive results are complementary to all such negative results, demonstrating a general class of data distributions where robust classifiers do exist.

For the reminder of this paper, we will look at a specific member of the above class of strongly concentrated data distributions and show how we can practically construct robust classifiers.

4 Adversarially Robust Classification on Union of Linear Subspaces

The union of subspaces model has been shown to be very useful in classical computer vision for a wide variety of tasks, which include clustering faces under varying illumination, image segmentation, and video segmentation [57]. Its concise mathematical description often enables the construction and theoretical analysis of algorithms that also perform well in practice. In this section, we will study robust classification on data distributions concentrated on a union of low-dimensional linear subspaces. This data structure will allow us to obtain a non-trivial, practically relevant case where we can show a provable improvement over existing methods for constructing robust classifiers in certain settings. Before delving further, we now provide a simple example (which is illustrated in Fig. 2) demonstrating how distributions concentrated about linear subspaces are concentrated precisely in the sense of Definition 3.1, and therefore allow for the existence of adversarially robust classifiers.

Example 4.1. The data domain is the ball B glyph[lscript] ∞ (0 , 1) equipped with the glyph[lscript] 2 distance. The label domain is { 1 , 2 } . Subspace S 1 is given by S 1 = { x : x glyph[latticetop] e 1 = 0 } , and S 2 is given by S 2 = { x : x glyph[latticetop] e 2 = 0 } , where e 1 , e 2 are the standard unit vectors. The conditional densities are defined as

where α > 0 is a large constant. Finally, the classes are balanced, i.e., p Y (1) = p Y (2) = 1 / 2 . With these parameters, q 1 , q 2 are both (0 . 5 , α/n -1 , 0) -concentrated over their respective supports. Additionally, p is ( glyph[epsilon1], 0 , e -α / 2 + 2 glyph[epsilon1] ) -strongly-concentrated. A robust classifier f can be constructed following the proof of Theorem 3.1, and it obtains a robust accuracy R ( f, glyph[epsilon1] ) ≤ e -α / 2 + 2 glyph[epsilon1] . See Appendix D for more details.

f

(

x

) = 1

f

(

x

) = 2

We will now study a specific choice of p that generalizes Example 4.1 and will let us move beyond the above simple binary setting. Recall that we have a classification problem specified by a data distribution p over the data domain X × Y = B (0 , 1) × { 1 , 2 , . . . , K } . Firstly, the classes are balanced, i.e. , p Y ( k ) = 1 /K for all k ∈ Y . Secondly, the conditional density, i.e., q k = p X | Y = k , is concentrated on the set S + γ k ∩X , where S k is a low-dimensional linear subspace, and the superscript denotes an γ -expansion, for a small γ > 0 .

For the purpose of building our robust classifier, we will assume access to a training dataset of M clean data points ( s 1 , y 1 ) , ( s 2 , y 2 ) , . . . , ( s M , y M ) , such that, for all i , the point s i lies exactly on one of the K low-dimensional linear subspaces. We will use the notation S = [ s 1 , s 2 , . . . , s M ] for the training data matrix and y = ( y 1 , y 2 , . . . , y M ) for the training labels. We will assume that M is large enough that every x ∈ ∪ k S k can be represented as a linear combination of the columns of S .

Now, the robust classification problem we aim to tackle is to obtain a predictor g : X → Y which obtains a low robust risk, with respect to an additive adversary A that we now define. For any data-point x ∼ p , A will be constrained to make an additive perturbation v such that dist glyph[lscript] 2 ( x + v, ∪ i S i ) ≤ glyph[epsilon1] . In other words, the attacked point can have glyph[lscript] 2 distance at most glyph[epsilon1] from any of the linear

subspaces S 1 , . . . , S k . Note that A is more powerful than an glyph[lscript] 2 -bounded adversary as the norm of the perturbation ‖ v ‖ 2 might be large, as v might be parallel to a subspace.

Under such an adversary A , given a (possibly adversarially perturbed) input x , it makes sense to try to recover the corresponding point s lying on the union of subspaces, such that x = s + n , such that ‖ n ‖ 2 ≤ glyph[epsilon1] . One way to do this is to represent s as a linear combination of a small number of columns of S , i.e. , x = S c + n . This can be formulated as an optimization problem that minimizes the cardinality of c , given by ‖ c ‖ 0 , subject to an approximation error constraint. Since such a problem is hard because of the glyph[lscript] 0 pseudo-norm, we relax this to the problem

Under a suitable choice of λ , this problem can be equivalently written as

for which we can obtain the dual problem given by

Our main observation is to leverage the stability of the set of active constraints of this dual to obtain a robust classifier. One can note that each constraint of Eq. (5) corresponds to one training data point s i - when the i th constraint is active at optimality, s i is being used to reconstruct x . Intuitively, one should then somehow use the label y i while predicting the label for x . Indeed, we will show that predicting the majority label among the active y i leads to a robust classifier.

We will firstly obtain a geometric characterization of the problem in Eq. (5) by viewing it as the evaluation of a projection operator onto a certain convex set, illustrated in Fig. 3. Observe that for λ > 0 , the objective (5) is strongly concave in d and the problem has a unique solution, denoted by d ∗ λ ( x ) . It is not hard to show that this solution can be obtained by the projection operator

where K ◦ is the polar of the convex hull of ± S . Denoting T = [ S , -S ] , we can rewrite Problem (6) as d ∗ λ ( x ) = ( arg min d ‖ λx -d ‖ 2 sub. to T glyph[latticetop] d ≤ 1 ) . We now define the set of active constraints as

Geometry of the Dual (5) . It is illustrated in Fig. 3, where s 1 , s 2 are two chosen data-points. The blue shaded polytope is K ◦ . At λ = λ 1 , the point λ 1 x lies in the interior of K ◦ . Hence, A λ ( x ) is empty and supp( c ∗ ( x )) is also empty. As λ increases, a non-empty support is obtained for the first time at λ = 1 /γ K ◦ ( x ) . For all λ 2 x in the red shaded polyhedron, the projection d ∗ λ 2 ( x ) = Proj K ◦ ( λ 2 x ) lies on the face F . As λ increases further we reach the green polyhedron. Further increases in λ do not change the dual solution, which will always remain at the vertex d ∗ λ 3 ( x ) .

Geometrically, A λ ( x ) identifies the face of K ◦ which contains the projection of λx , if A λ ( x ) is non-empty (otherwise, λx lies inside the polyhedron K ◦ ). The support of the primal solution, c ∗ ( x ) , is a subset of A λ , i.e. supp( c ∗ ( x )) ⊆ A λ ( x ) . Note that whenever two

s

1

x

)

K

◦

λ

d

∗

λ

3

(

x

points, say x, x ′ , both lie in the same shaded polyhedron (red or green), their projections would lie on the same face of K ◦ . We now show this formally, in the main theorem of this section.

Theorem 4.1. The set of active constraints A λ defined in (7) is robust, i.e., A λ ( x ′ ) = A λ ( x ) for all λx ′ ∈ C ( x ) , where C ( x ) is the polyhedron defined as

with F ⊆ K ◦ being a facet of the polyhedron K ◦ that x projects to, defined as

and V being the cone generated by the constraints active at (i.e., normal to) F , defined as

The proof of Theorem 4.1 utilizes the geometry of the problem and properties of the projection operator, and is presented in Appendix E. We can now use this result to construct a robust classifier:

Lemma 4.2. Define the dual classifier as

where AGGREGATE is any deterministic mapping from a set of labels to Y , e.g., MAJORITY . Then, for all x ′ ∈ C ( x ) as defined in Theorem 4.1, g λ is certified to be robust, i.e., g λ ( x ′ ) = g λ ( x ) .

Implications. Having obtained a certifiably robust classifier g , we pause to understand some implications of the theory developed so far. We observe that the certified regions in Theorem 4.1 are not spherical, i.e. , the attacker can make additive perturbations having large glyph[lscript] 2 norm but still be unable to change the label predicted by g (see Fig. 4). This is in contrast to the glyph[lscript] 2 bounded certified regions that can be obtained by most existing work on certification schemes, and is a result of modelling data structure while constructing robust classifiers. Importantly, however, note that we do not assume that the attack is restricted to the subspace.

Connections to Classical Results. For glyph[epsilon1] = 0 , Eq. (3) is known as the primal form of the Basis Pursuit problem, and has been studied under a variety of conditions on S in the sparse representation and subspace clustering literature [20, 13, 50, 64, 30, 25]. Given an optimal solution c ∗ ( x ) of this basis pursuit problem, how can we accurately predict the label y ? One ideal situation could be that all columns in the support predict the same label, i.e. , y i is identical for all i ∈ supp ( c ∗ ( x )) . Indeed, this ideal case is well studied, and is ensured by necessary [25] and sufficient [50, 64, 30] conditions on the geometry of the subspaces S 1 , . . . , S K . Another situation could be that the majority of the columns in the support predict the correct label. In this case, we could predict Majority ( { y i : i ∈ supp( c ∗ ( x )) } ) to ensure accurate prediction. Theorem 4.1 allows us to obtain robustness guarantees which work for any such aggregation function which can determine a single label from the support. Hence, our results can guarantee robust prediction even when classical conditions are not satisfied. Lastly, note that our Theorem 4.1 shows that the entire active set remains unperturbed - In light of the recent results in [54], this could be relaxed for specific choices of maps acting on the estimated support.

5 Experiments

In this section, we will compare our certified defense derived in Section 4 to a popular defense technique called Randomized Smoothing (RS) [11], which can be used to obtain state-of-the-art certified robustness against glyph[lscript] 2 perturbations. RS transforms any given classifier f : X → Y to a certifiably robust classifier g RS σ : X → Y by taking a majority vote over inputs perturbed by Gaussian 6 noise N (0 , σ 2 I ) , i.e. ,

Then, at any point x , g RS σ can be shown to be certifiably robust to glyph[lscript] 2 perturbations of size at least r RS ( x ) = σ Φ -1 ( p ) where p = max k ∈Y Pr v ∼N (0 ,σ 2 I ) ( f ( x + v ) = k ) denotes the maximum probability of any class under Gaussian noise.

It is not immediately obvious how to compare the certificates provided by our method described above and that of RS, since the sets of the space they certify are different. The certified region

x

True

/lscript

2

Certi

fi

cate

+

Ours

Randomized

Smoothing

obtained by RS, C RS ( x ) = { ¯ x : ‖ x -¯ x ‖ 2 ≤ r ( x ) } , is a sphere (orange ball in Fig. 4). In contrast, our certificate C λ ( x ) from Theorem 4.1 is a polyhedron (resp., blue trapezoid), which, in general, is neither contained in C RS ( x ) , nor a superset of C RS ( x ) . Additionally, our certificate has no standard notion of size , unlike other work on elliptical certificates [14], making a size comparison non-trivial. To overcome these difficulties, we will evaluate two notions of attack size : in the first, we will compare the glyph[lscript] 2 norms of successful attacks projected onto our polyhedron, and in the second, we will compare the minimum glyph[lscript] 2 norm required for a successful attack. We will then combine our method with RS to get the best of both worlds, i.e. , the green shape in Fig. 4. In the following, we present both these evaluations on the MNIST [28] dataset, with each image normalized to unit glyph[lscript] 2 norm.

Comparison along Projection on C λ ( x ) . For the first case, we investigate the question: Are there perturbations for which our method is certifiably correct, but Randomized Smoothing fails? For an input point x , we can answer this question in the affirmative by obtaining an adversarial example ¯ x for g RS such that ¯ x lies inside our certified set C λ ( x ) . Then, this perturbation v = x -¯ x is certified by our method, but has glyph[lscript] 2 norm larger than r RS ( x ) (by definition of the RS certificate).

To obtain such adversarial examples, we first train a standard CNN classifier f for MNIST, and then use RS 7 to obtain the classifier g RS σ . Then, for any ( x, y ) , we perform projected gradient descent to obtain ¯ x = x T by performing the following steps T times, starting with x 0 ← x :

Unlike the standard PGD attack (step I), the additional projection (step II) is not straightforward, and requires us to solve a quadratic optimization problem, which can be found in Appendix F. We can now evaluate g RS σ on these perturbations to empirically estimate the robust accuracy over C λ , i.e. , glyph[negationslash]

The results are plotted in Fig. 5, as the dotted curves. We also plot the certified accuracies 7 for comparison, as the solid curves. We see that the accuracy certified by RS drops below random chance (0.1) around glyph[epsilon1] = 0 . 06 (solid red curve). Similar to other certified defenses, RS certifies only a subset of the true robust accuracy of a classifier in general. This true robust accuracy curve is pointwise upper-bounded by the empirical robust accuracy curve corresponding to any attack, obtained via the steps I , II described earlier (dotted red curve). We then see that even the upper-bound drops below random chance around glyph[epsilon1] = 0 . 4 , suggesting that this might be a large enough attack strength so that an adversary only constrained in glyph[lscript] 2 norm is able to fool a general classifier. However, we are evaluating attacks lying on our certified set and it is still possible to recover the true

0.00

0.25

0.50

0.75

1.00

/epsilon1

0.00

0.25

0.50

0.75

1.00

Projected Robust Accuracy

Ours Certified

g

RS

0

.

02

Empirical

g

RS

0

.

02

Certified

g

RS

0

.

04

Empirical

g

RS

0

.

04

Certified

class (blue solid curve), albeit by our specialized classifier g λ suited to the data structure. Additionally, this suggests that our certified set contains useful class-specific information - this is indeed true, and we present some qualitative examples of images in our certified set in Appendix F. To summarize, we have numerically demonstrated that exploiting data structure in classifier design leads to certified regions capturing class-relevant regions beyond glyph[lscript] p -balls .

Comparison along glyph[lscript] 2 balls. For the second case, we ask the question: Are there perturbations for which RS is certifiably correct, but our method is not? When an input point x has a large enough RS certificate r RS ( x ) ≥ r 0 , some part of the sphere B glyph[lscript] 2 ( x, r RS ( x )) might lie outside our polyhedral certificate C λ ( x ) (blue region in Fig. 4). In theory, the minimum r 0 required can be computed via an expensive optimization program that we specify in Appendix F. In practice, however, we use a black-box attack [9] to find such perturbations. We provide qualitative examples and additional experiments on CIFAR-10 in Appendix F.

Combining Our Method with RS. We now improve our certified regions using randomized smoothing. For this purpose, we treat our classifier g λ as the base classifier f in (12), to obtain Smooth σ (g λ ) (abbreviated as Ours RS σ ). We then plot the glyph[lscript] 2 certified accuracy 7 [11, Sec 3.2.2] in

0.000

0.025

0.050

0.075

0.100

0.25

0.50

0.75

1.00

Certified Accuracy

Ours RS

0

.

02

Ours RS

0

.

04

g

RS

0

.

02

g

RS

0

.

04

Certified Radius

Fig. 6, where note that, as opposed to Fig. 5, the attacks are not constrained to lie on our certificate anymore.

As a final remark, we note that our objective in Fig. 6 was simply to explore RS as a method for obtaining an glyph[lscript] 2 certificate for our method, and we did not tune our method or RS for performance. In particular, we believe that a wide array of tricks developed in the literature for improving RS performance [41, 45, 63] could be employed to improve the curves in Fig. 6. We now discuss our work in the context of existing literature in adversarial robustness, and sparse representation learning.

6 Discussion and Related Work

Our proof techniques utilize tools from high-dimensional probability, and have the same flavor as recent impossibility results for adversarial robustness [12, 48, 47]. Our geometric treatment of the dual optimization problem is similar to the literature on sparse-representation [13, 20] and subspace clustering [50, 51, 64, 25], which is concerned with the question of representing a point x by the linear combination of columns of a dictionary S using sparse coefficients c . As mentioned in Section 4, there exist geometric conditions on S such that all such candidate vectors c are subspace-preserving , i.e., for all the indices i in the support of c , it can be guaranteed that s i belongs to the correct subspace. On the other hand, the question of classification of a point x in a union of subspaces given by the columns of S , or subspace classification, has also been studied extensively in classical sparse-representation literature [62, 61, 7, 27, 65, 65, 65, 39, 23, 32]. The predominant approach is to solve an glyph[lscript] 1 minimization problem to obtain coefficients c so that x = S c + e , and then predict the subspace that minimizes the representation error. Various global conditions can be imposed on S to guarantee the success of such an approach [62], and its generalizations [15, 16]. Our work differs from these approaches in that we aim to obtain conditions on perturbations to x that ensure accurate classification, and as such we base our robust classification decision upon properties of solutions of the dual of the glyph[lscript] 1 problem.

Our results are complementary to [44, 5], who obtain a lower bound on the robust risk R ( f, glyph[epsilon1] ) , in a binary classification setting, in terms of the Wasserstein distance D between the class conditionals q 0 and q 1 , i.e. , R ( f, glyph[epsilon1] ) ≥ 1 -D ( q 0 , q 1 ) . Using this result, [44, 5] roughly state that the robust risk increases as the class conditionals get closer, i.e. , it becomes more likely to sample x ∼ q 0 , x ′ ∼ q 1 , such that ‖ x -x ′ ‖ ≤ glyph[epsilon1] . In comparison, for two classes, our Theorem 3.1 roughly states that for low robust risk, q 0 and q 1 should be concentrated on small-volume subsets separated from one another. Thus, the message is similar, while our Theorem 3.1 is derived in a more general multi-class setting. Note that [44, 5] do not explicitly require small-volume subsets, but this is implicit, as 1 -D ( q 0 , q 1 )

is large due to concentration of measure when q 0 , q 1 do not concentrate on very small subsets. We provide an analogue of the empirical results of [44, 5] in Appendix F.6.

Our notion of concentration is also related to concentration of a measure µ , as considered in [35, 36, 70], which denotes how much the µ -measure of any set blows up after expansion by glyph[epsilon1] . Under this definition, the uniform measure has a high degree of concentration in high dimensions, and this is called the concentration of measure phenomenon. In contrast, our Definition 2.2 of concentrated data distributions can be seen as a relative notion of concentration with respect to the uniform measure µ , in that we call a class conditional q concentrated when it assigns a large q -measure to a set of very small volume, ( i.e. , q ( S ) is high whereas µ ( S ) is very low). In essence, Definition 2.2 defines concentration relative to the uniform distribution, whereas [35, 36, 70] define concentration independently. Definition 2.2 is useful in the context of adversarial robustness as it allows us to separate the concentration of data distributions (which is unknown) from the concentration of the uniform measure in high dimensions (for which there is a good understanding). This allows us to derive results in the non-realizable setting, where errors are measured against a probabilistic ground truth label Y , which is strictly more general than the realizable setting which requires a deterministic ground truth classifier f ∗ . As a result of this realizable setting, the analysis in [21, 35, 36, 70] needs to assume a non-empty error region A for the learnt classifier g with respect to f ∗ , in order to reason about glyph[epsilon1] -expansions A + glyph[epsilon1] . The results in [35, 36] indicate that the robust risk grows quickly as the volume of A increases. However, humans seem to be a case where the natural accuracy is not perfect (e.g., we might be confused between a 6 and a poorly written 5 in MNIST), yet we seem to be very robust against small glyph[lscript] 2 perturbations. This points to a slack in the analysis in [35, 36], and our work fills this gap by considering glyph[epsilon1] expansions of a different family of sets.

Finally, our work is also related to recent empirical work obtaining robust classifiers by denoising a given input x of any adversarial corruptions, before passing it to a classifier [46, 38]. However, such approaches lack theoretical guarantees, and might be broken using specialized attacks [40]. Similarly, work on improving the robustness of deep network-based classifiers by adversarial training off the data-manifold can be seen as an empirical generalization of our attack model [24, 37, 69, 33]. More generally, it has been studied how adversarial examples relate to the underlying data-manifold [53, 26, 34]. Recent work also studies the robustness of classification using projections onto a single low-dimensional linear subspace [3, 2, 43]. The work in [3] studies an the attack model of bounded glyph[lscript] 2 , glyph[lscript] ∞ attacks, and they provide robustness certificates by obtaining guarantees on the distortion of a data-point x as it is projected onto a single linear subspace using a projection matrix Π . In contrast, our work can be seen as projecting a perturbed point onto a union of multiple low-dimensional subspaces. The resultant richer geometry allows us to obtain more general certificates.

7 Conclusion and Future Work

To conclude, we studied conditions under which a robust classifier exists for a given classification task. We showed that concentration of the data distribution on small subsets of the input space is necessary for any classifier to be robust to small adversarial perturbations, and that a stronger notion of concentration is sufficient. We then studied a special concentrated data distribution, that of data distributed near low-dimensional linear subspaces. For this special case of our results, we constructed a provably robust classifier, and then experimentally evaluated its benefits w.r.t. known techniques.

For the above special case, we assume access to a clean dataset S lying perfectly on the union of low-dimensional linear subspaces, while in reality one might only have access to noisy samples. In light of existing results on noisy subspace clustering [59], an immediate future direction is to adapt our guarantees to support noise in the training data. Similarly, while the assumption of lowdimensional subspace structure in S enables us to obtain novel unbounded robustness certificates, real world datasets might not satisfy this structure. We hope to mitigate this limitation by extending our formulation to handle data lying on a general image manifold in the future.

Acknowledgments and Disclosure of Funding

We would like to thank Amitabh Basu for helpful insights into the optimization formulation for the largest glyph[lscript] 2 ball contained in a polyhedron. This work was supported by DARPA grant HR00112020010,

NSF grants 1934979 and 2212457, and the NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning (NSF grant 2031985, Simons grant 814201).

References

Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning , pages 274-283. PMLR, 2018.

Pranjal Awasthi, Vaggos Chatziafratis, Xue Chen, and Aravindan Vijayaraghavan. Adversarially robust low dimensional representations. In Conference on Learning Theory , pages 237-325. PMLR, 2021.

Pranjal Awasthi, Himanshu Jain, Ankit Singh Rawat, and Aravindan Vijayaraghavan. Adversarial robustness via robust low rank representations. Advances in Neural Information Processing Systems , 33:11391-11403, 2020.

Amitabh Basu. Introduction to convexity. Optimization , 52:37, 2019.

Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. Lower bounds on adversarial robustness from optimal transport. Advances in Neural Information Processing Systems , 32, 2019.

Sébastien Bubeck, Yin Tat Lee, Eric Price, and Ilya Razenshteyn. Adversarial examples from computational constraints. In International Conference on Machine Learning , pages 831-840. PMLR, 2019.

Raffaele Cappelli, Dario Maio, and Davide Maltoni. Subspace classification for face recognition. In International Workshop on Biometric Authentication , pages 133-141. Springer, 2002.

Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705 , 2019.

Jianbo Chen, Michael I Jordan, and Martin J Wainwright. Hopskipjumpattack: A query-efficient decision-based attack. In 2020 ieee symposium on security and privacy (sp) , pages 1277-1294. IEEE, 2020.

Ping-yeh Chiang, Renkun Ni, Ahmed Abdelkader, Chen Zhu, Christoph Studer, and Tom Goldstein. Certified defenses for adversarial patches. In 8th International Conference on Learning Representations (ICLR 2020)(virtual) . International Conference on Learning Representations, 2020.

Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning , pages 1310-1320. PMLR, 2019.

Elvis Dohmatob. Generalized no free lunch theorem for adversarial robustness. In International Conference on Machine Learning , pages 1646-1654. PMLR, 2019.

David L Donoho and Michael Elad. Optimally sparse representation in general (nonorthogonal) dictionaries via l1 minimization. Proceedings of the National Academy of Sciences , 100(5):21972202, 2003.

Francisco Eiras, Motasem Alfarra, Philip Torr, M Pawan Kumar, Puneet K Dokania, Bernard Ghanem, and Adel Bibi. Ancer: Anisotropic certification via sample-wise volume maximization. Transactions of Machine Learning Research .

Ehsan Elhamifar and René Vidal. Robust classification using structured sparse representation. In CVPR 2011 , pages 1873-1879. IEEE, 2011.

Ehsan Elhamifar and René Vidal. Block-sparse recovery via convex optimization. IEEE Transactions on Signal Processing , 60(8):4094-4107, 2012.

Gamaleldin Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alexey Kurakin, Ian Goodfellow, and Jascha Sohl-Dickstein. Adversarial examples that fool both computer vision and time-limited humans. Advances in neural information processing systems , 31, 2018.

Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi. Adversarial vulnerability for any classifier. Advances in neural information processing systems , 31, 2018.

Marc Fischer, Maximilian Baader, and Martin Vechev. Certified defense to image transformations via randomized smoothing. Advances in Neural Information Processing Systems , 33:8404-8417, 2020.

Simon Foucart and Holger Rauhut. An invitation to compressive sensing. In A mathematical introduction to compressive sensing , pages 1-39. Springer, 2013.

Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S Schoenholz, Maithra Raghu, Martin Wattenberg, Ian Goodfellow, and G Brain. The relationship between high-dimensional geometry and adversarial examples. arXiv preprint arXiv:1801.02774 , 2018.

Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In International Conference on Learning Representations , 2018.

Kang-hua Hui, Chun-li Li, and Lei Zhang. Sparse neighbor representation for classification. Pattern Recognition Letters , 33(5):661-669, 2012.

Charles Jin and Martin Rinard. Manifold regularization for locally stable deep neural networks. arXiv preprint arXiv:2003.04286 , 2020.

Mustafa D Kaba, Chong You, Daniel P Robinson, Enrique Mallada, and Rene Vidal. A nullspace property for subspace-preserving recovery. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 5180-5188. PMLR, 18-24 Jul 2021.

Marc Khoury and Dylan Hadfield-Menell. On the geometry of adversarial examples. arXiv preprint arXiv:1811.00525 , 2018.

Jorma Laaksonen et al. Subspace classifiers in recognition of handwritten digits . Helsinki University of Technology, 1997.

Yann LeCun. The mnist database of handwritten digits. http://yann.lecun.com/exdb/mnist/ , 1998.

Alexander Levine and Soheil Feizi. (de) randomized smoothing for certifiable defense against patch attacks. Advances in Neural Information Processing Systems , 33:6465-6475, 2020.

Chun-Guang Li, Chong You, and René Vidal. On geometric analysis of affine sparse subspace clustering. IEEE Journal of Selected Topics in Signal Processing , 12(6):1520-1533, 2018.

Shengqiao Li. Concise formulas for the area and volume of a hyperspherical cap. Asian Journal of Mathematics & Statistics , 4(1):66-70, 2010.

Wei Li, Eric W Tramel, Saurabh Prasad, and James E Fowler. Nearest regularized subspace for hyperspectral classification. IEEE Transactions on Geoscience and Remote Sensing , 52(1):477489, 2013.

Wei-An Lin, Chun Pong Lau, Alexander Levine, Rama Chellappa, and Soheil Feizi. Dual manifold adversarial robustness: Defense against lp and non-lp adversarial attacks. Advances in Neural Information Processing Systems , 33:3487-3498, 2020.

Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In International Conference on Learning Representations , 2018.

Saeed Mahloujifar, Dimitrios I Diochnos, and Mohammad Mahmoody. The curse of concentration in robust learning: Evasion and poisoning attacks from concentration of measure. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 4536-4543, 2019.

Saeed Mahloujifar, Xiao Zhang, Mohammad Mahmoody, and David Evans. Empirically measuring concentration: Fundamental limits on intrinsic robustness. Advances in Neural Information Processing Systems , 32, 2019.

Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robustness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9078-9086, 2019.

Seyed-Mohsen Moosavi-Dezfooli, Ashish Shrivastava, and Oncel Tuzel. Divide, denoise, and defend against adversarial attacks. arXiv preprint arXiv:1802.06806 , 2018.

Imran Naseem, Roberto Togneri, and Mohammed Bennamoun. Linear regression for face recognition. IEEE transactions on pattern analysis and machine intelligence , 32(11):21062112, 2010.

Zhonghan Niu, Zhaoxi Chen, Linyi Li, Yubin Yang, Bo Li, and Jinfeng Yi. On the limitations of denoising strategies as adversarial defenses. arXiv preprint arXiv:2012.09384 , 2020.

Ambar Pal and Jeremias Sulam. Understanding noise-augmented training for randomized smoothing. Transactions on Machine Learning Research , 2023.

Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP) , pages 582-597. IEEE, 2016.

Samuel Pfrommer, Brendon G Anderson, and Somayeh Sojoudi. Projected randomized smoothing for certified adversarial robustness. 2022.

Muni Sreenivas Pydi and Varun Jog. Adversarial risk via optimal transport and optimal couplings. In International Conference on Machine Learning , pages 7814-7823. PMLR, 2020.

Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Advances in Neural Information Processing Systems , 32, 2019.

Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. In International Conference on Learning Representations , 2018.

Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adversarially robust generalization requires more data. Advances in neural information processing systems , 31, 2018.

Ali Shafahi, W Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adversarial examples inevitable? arXiv preprint arXiv:1809.02104 , 2018.

Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advances in Neural Information Processing Systems , 32, 2019.

Mahdi Soltanolkotabi and Emmanuel J Candes. A geometric analysis of subspace clustering with outliers. The Annals of Statistics , 40(4):2195-2238, 2012.

Mahdi Soltanolkotabi, Ehsan Elhamifar, and Emmanuel J Candes. Robust subspace clustering. The annals of Statistics , 42(2):669-699, 2014.

Daniel A Spielman and Shang-Hua Teng. Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time. Journal of the ACM (JACM) , 51(3):385-463, 2004.

David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6976-6987, 2019.

Jeremias Sulam, Ramchandran Muthukumar, and Raman Arora. Adversarial robustness of supervised sparse coding. Advances in neural information processing systems , 33:2110-2121, 2020.

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013.

Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.12152 , 2018.

René Vidal, Yi Ma, S Shankar Sastry, René Vidal, Yi Ma, and S Shankar Sastry. Generalized Principal Component Analysis . Springer, 2016.

Václav Vorácek and Matthias Hein. Provably adversarially robust nearest prototype classifiers. In International Conference on Machine Learning , pages 22361-22383. PMLR, 2022.

Yu-Xiang Wang and Huan Xu. Noisy sparse subspace clustering. In International Conference on Machine Learning , pages 89-97. PMLR, 2013.

Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations , 2019.

John Wright, Yi Ma, Julien Mairal, Guillermo Sapiro, Thomas S Huang, and Shuicheng Yan. Sparse representation for computer vision and pattern recognition. Proceedings of the IEEE , 98(6):1031-1044, 2010.

John Wright, Allen Y Yang, Arvind Ganesh, S Shankar Sastry, and Yi Ma. Robust face recognition via sparse representation. IEEE transactions on pattern analysis and machine intelligence , 31(2):210-227, 2008.

Greg Yang, Tony Duan, J Edward Hu, Hadi Salman, Ilya Razenshteyn, and Jerry Li. Randomized smoothing of all shapes and sizes. In International Conference on Machine Learning , pages 10693-10705. PMLR, 2020.

Chong You and René Vidal. Geometric conditions for subspace-sparse recovery. In International Conference on Machine Learning , pages 1585-1593. PMLR, 2015.

Guoxian Yu, Guoji Zhang, Zili Zhang, Zhiwen Yu, and Lin Deng. Semi-supervised classification based on subspace sparse representation. Knowledge and Information Systems , 43(1):81-101, 2015.

Bohang Zhang, Du Jiang, Di He, and Liwei Wang. Rethinking lipschitz neural networks for certified l-infinity robustness. arXiv preprint arXiv:2210.01787 , 2022.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Shangzhi Zhang, Chong You, René Vidal, and Chun-Guang Li. Learning a self-expressive network for subspace clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12393-12403, 2021.

Shufei Zhang, Kaizhu Huang, Jianke Zhu, and Yang Liu. Manifold adversarial training for supervised and semi-supervised learning. Neural Networks , 140:282-293, 2021.

Xiao Zhang and David Evans. Understanding intrinsic robustness using label uncertainty. arXiv preprint arXiv:2107.03250 , 2021.

A Proof of Theorem 2.1

We will make the technical assumption in Section 2 precise. Recall that all quantities are normalized so that X is an glyph[lscript] 2 ball of radius 1 , i.e., X = B glyph[lscript] 2 (0 , 1) . Recall that q k : X → R ≥ 0 denotes the conditional distribution for class k ∈ Y , having support Q k . We will assume that there is a sufficient gap between the supports and the boundary of the domain, i.e., Q k ⊆ B glyph[lscript] 2 (0 , c ) , and that the glyph[lscript] 2 -adversarial attack has strength glyph[epsilon1] ≤ c , for some constant c , say c = 0 . 1 .

Theorem 2.1. If there exists an ( glyph[epsilon1], δ ) -robust classifier f for a data distribution p , then at least one of the class conditionals q 1 , q 2 , . . . , q K , say q ¯ k , must be ( ¯ C,glyph[epsilon1], δ ) -concentrated. Further, if the classes are balanced, then all the class conditionals are ( C max , glyph[epsilon1],Kδ ) -concentrated. Here, ¯ C = Vol { x : f ( x ) = ¯ k } , and C max = max k Vol { x : f ( x ) = k } are constants dependent on f .

Proof. Note that the definition (1) of ( glyph[epsilon1], δ ) -robustness may be rewritten as glyph[negationslash]

where q ( S ) denotes the q -measure of the set S . We are given f such that R ( f, glyph[epsilon1] ) ≤ δ , and we want to find a set S ⊆ X over which p concentrates.

glyph[negationslash]

Since ∑ k p Y ( y = k ) = 1 , the LHS in (15) is a convex combination, and we have glyph[negationslash]

We will express q ˆ k ( U ) in terms of the classification regions of f .

Let A ⊆ X denote the region where the classifier predicts ˆ k , i.e. A = { x ∈ X : f ( x ) = ˆ k } . Define A -glyph[epsilon1] to be the set of all points in A at a glyph[lscript] 2 distance atleast glyph[epsilon1] from the boundary, i.e. A -glyph[epsilon1] = { x ∈ A : B ( x, glyph[epsilon1] ) ⊆ A } . Now consider any point x ∈ X . We have the following 4 mutually exclusive, and exhaustive cases:

glyph[negationslash]

x lies outside A , i.e., x glyph[negationslash]∈ A . In this case, x ∈ U trivially as f ( x ) = k .

x ∈ A , and the entire glyph[epsilon1] -ball around x lies inside A , i.e., B ( x, glyph[epsilon1] ) ⊆ A . In this case, every point in the glyph[epsilon1] -ball around x is classified into class ˆ k , and x has no glyph[epsilon1] -adversarial example for class ˆ k . Note that the set of all these points is A -glyph[epsilon1] by definition.

glyph[negationslash]

x ∈ A , and some portion of the glyph[epsilon1] -ball around x lies outside A , i.e., B ( x, glyph[epsilon1] ) glyph[negationslash]⊆ A , and, there is a point ¯ x ∈ B ( x, glyph[epsilon1] ) ∩ X such that f (¯ x ) = k . In this case, ¯ x is an adversarial example for x , and ¯ x ∈ U .

glyph[negationslash]

x ∈ A , and some portion of the glyph[epsilon1] -ball around x lies outside A , i.e., B ( x, glyph[epsilon1] ) glyph[negationslash]⊆ A , but there is no point ¯ x ∈ B ( x, glyph[epsilon1] ) ∩ X such that f (¯ x ) = k . This can happen only when for all ¯ x ∈ B ( x, glyph[epsilon1] ) \ A we have ¯ x glyph[negationslash]∈ X . In other words, any glyph[epsilon1] perturbation that takes x outside A , also takes it outside the domain X . This implies that B ( x, glyph[epsilon1] ) glyph[negationslash]⊆ X , which in turn means that the distance of x from the boundary of X is atmost glyph[epsilon1] .

We see that U comprises of the points covered in cases (1) and (3). Recall that we assumed that any point having distance less than 0 . 1 to the boundary of X does not lie in the support of q ¯ k . But all points covered in case (4) lie less than glyph[epsilon1] -close to the boundary, and glyph[epsilon1] ≤ 0 . 1 . This implies that q ¯ k glyph[negationslash]

assigns 0 measure to the points covered in case (4). Together, cases (1), (3) and (4) comprise the set ( X \ A ) ∪ ( A \ A -glyph[epsilon1] ) . Thus, we have shown

Next, we will show that the set A -glyph[epsilon1] is exponentially small, by appealing to the Brunn-Minkowski inequality, which states that for compact sets E,F ⊂ R n ,

where Vol is the n -dimensional volume in R n , and E + F denotes the minkowski sum of the sets E and F . Applying the inequality with E = A -glyph[epsilon1] and F = B glyph[lscript] 2 (0 , glyph[epsilon1] ) , we have

where ¯ C = Vol( A ) . From (17) and (18), we see that q ¯ k is ( ¯ C,glyph[epsilon1], δ ) -concentrated. Additionally, if the classes are balanced, we can use the fact that every term of (15) can be atmost Kδ , and apply the above reasoning for each class to conclude that each q k is ( C max , glyph[epsilon1],Kδ ) -concentrated, for some suitable choice of C max , e.g., C max = max k Vol { x ∈ X : f ( x ) = k } .

B Proof of Theorem 3.1

Theorem 3.1. If the data distribution p is ( glyph[epsilon1], δ, γ ) -strongly-concentrated, then there exists an ( glyph[epsilon1], δ + γ ) -robust classifier for p .

Proof. For each k ∈ { 1 , 2 , . . . , K } , let S k be the support of the conditional density q k . Recall that S + glyph[epsilon1] is defined to be the glyph[epsilon1] -expansion of the set S . Define C k to be the glyph[epsilon1] -expanded version of the concentrated region S k but removing the glyph[epsilon1] -expanded version of all other regions S k ′ , as glyph[negationslash]

We will use these regions to define our classifier f : X → { 1 , 2 , . . . , K } as

.

We will show that R ( f, glyph[epsilon1] ) ≤ δ + γ , which can be recalled to be

glyph[negationslash]

In the above expression, the q k mass is over the set of all points x ∈ X that admit an glyph[epsilon1] -adversarial example for the class k , as glyph[negationslash]

Define C -glyph[epsilon1] k to be the set of points in C k at a distance atleast glyph[epsilon1] from the boundary of C k as C -glyph[epsilon1] k = { x ∈ C k : B ( x, glyph[epsilon1] ) ⊆ C k } . For any point x ∈ U k , we can find ¯ x ∈ B ( x, glyph[epsilon1] ) from (19) such that

¯ x glyph[negationslash]∈ C k , showing that x glyph[negationslash]∈ C -glyph[epsilon1] k . Thus, U k is a subset of the complement of C -glyph[epsilon1] k , i.e., U k ⊆ X \ C -glyph[epsilon1] k , and we have

Now we will need to show a few properties of the glyph[epsilon1] -contraction. Firstly, for a set M = N ∩ O , we have M -glyph[epsilon1] = N -glyph[epsilon1] ∩ O -glyph[epsilon1] , which can be seen as

Secondly, for a set M = N c , where c denotes complement, we have M -glyph[epsilon1] = ( N + glyph[epsilon1] ) c . This can be seen as

Thirdly, for a set M = N \ O , we have M -glyph[epsilon1] = ( N ∩ O c ) -glyph[epsilon1] = N -glyph[epsilon1] ∩ ( O c ) -glyph[epsilon1] by the first property, and then N -glyph[epsilon1] ∩ ( O c ) -glyph[epsilon1] = N -glyph[epsilon1] ∩ ( O + glyph[epsilon1] ) c by the second property. This implies

Fourthly, for a set M = N ∪ O , we have M c = N c ∩ O c . Taking glyph[epsilon1] -contractions, and applying the first and second properties, we get M + glyph[epsilon1] = N + glyph[epsilon1] ∪ O + glyph[epsilon1] . Applying the above properties to C -glyph[epsilon1] k , we have glyph[negationslash]

glyph[negationslash]

Recall that q k ( X -glyph[epsilon1] ) = q ( X ) = 1 by the support assumption. Hence, we have glyph[negationslash]

Finally, as ∑ k p Y ( y = k ) = 1 , we have R ( f, glyph[epsilon1] ) ≤ δ + γ

C Proofs for Example 3.1

Let θ 0 ≤ π/ 2 . We will show that the following classifier f is robust in the setting of Example 3.1:

Let C 1 = { x ∈ S n -1 : x glyph[latticetop] P ≥ θ 0 } be the set of points classified into class 1 by f . Similarly, let C 2 = S n -1 \ C 1 be the set of points classified into class 2 . The robust risk of f (measured w.r.t. perturbations in the geodesic distance d ) can be expanded as

where d + glyph[epsilon1] ( S ) denotes the glyph[epsilon1] -expansion of the set S under the distance d . Let glyph[epsilon1] ≤ θ 0 (otherwise, the first term is 1 / 2 ). Recall that the class conditional q 1 is defined as

glyph[negationslash]

glyph[negationslash]

where c ψ is a normalizing constant which ensures that q 1 integrates to 1 , i.e. , c ψ = ∫ S n -1 q 1 . Then, q 2 was defined to be uniform over the complement of the support of q 1 , i.e. q 2 = Unif( { x ∈ S n -1 : d ( x, P ) > θ 0 } ) .

We can expand the first term of Eq. (20) as

Now for any x glyph[negationslash]∈ C 2 , we have d ( x, C 2 ) = θ 0 -d ( x, P ) . Hence, { x : 0 < d ( x, C 2 ) ≤ glyph[epsilon1] } = { x : 0 . 1 -glyph[epsilon1] ≤ d ( x, P ) < θ 0 } . Let θ ( x ) denote the angle that x makes with P . Then, the geodesic distance is d ( x, P ) = θ ( x ) . The earlier set is the same as the set of all points satisfying θ 0 -glyph[epsilon1] ≤ θ ( x ) < 1 . This is nothing but the glyph[epsilon1] -base of the hyper-spherical cap having angle θ 0 .

We continue,

With a change of variables, the above can be evaluated as

where µ n -2 is the n -2 -dimensional volume ( n -1 -dimensional surface area). Now, µ n -2 ( { x ∈ S n -1 : θ ( x ) = θ } ) is nothing but the volume of an slice of the hypersphere S n -1 . This slice is in itself a hypersphere in R n -1 , having radius sin θ . Thus, its volume is the same as the volume of the S n -2 , scaled by (sin θ ) n -2 . We continue,

For illustration, we can take ψ ( θ ) = 1 , giving q 1 ( d + glyph[epsilon1] ( C 1 )) = glyph[epsilon1]/C from the above integral, with c ψ = ∫ θ 0 0 ψ ( θ ) dθ = θ 0 = θ 0 .

We can now follow the same process as above, and expand the Eq. (20) as

where µ n -1 is again the n -1 -dimensional volume. We use the following formula [31] for the surface area of a hyperspherical cap (valid when α ≤ π/ 2 ):

where I ( · , · ) is the incomplete regularized beta function. Letting d θ 0 = m ( π/ 2) + m ( π/ 2) -m ( θ 0 ) , we continue,

The risk from class 1 dominates till glyph[epsilon1] reaches around π/ 2 , then the risk from class 2 shoots up. We plot the resultant risk in Fig. 7 (left).

Having presented the basic construction, we now comment on how Example 3.1 can be generalized. Firstly, we can vary the support of q 1 , to have it cover a different fraction of the sphere. For instance, we can set θ 0 = π/ 2 to get a modified q 1 as

0.1

π/

4

π/

2

3

π/

4

π

/epsilon1

0.0

0.2

0.4

0.6

0.8

1.0

Rd

(

f, /epsilon1

)

0.1

π/

4

π/

2

3

π/

4

π

/epsilon1

0.0

0.2

0.4

0.6

0.8

1.0

Rd

(

g, /epsilon1

)

π/

4

π/

2

3

π/

4

π

/epsilon1

0.0

0.2

0.4

0.6

0.8

1.0

Rd

(

g, /epsilon1

)

and let q ′ 2 be uniform over the complement of the support of q ′ 1 . Along with balanced classes ( i.e. , P ( Y = 1) = P ( Y = 2) = 1 / 2 ), this gives the data distribution p 2 . The robust classifier would now be given by the half-space { x : d ( x, P ) ≤ π/ 2 } . The risk over p 2 can be computed as earlier, and is plotted in Fig. 7 (middle).

We can take this example further, and make both q 1 and q 2 concentrated. This can be done, for instance, by setting q ′′ 1 = q 1 , and setting q ′′ 2 as follows:

where -P is the antipodal point of P . As both class conditionals are now concentrated, the halfspace separating their supports becomes quite robust. This can be seen in the Fig. 7 (right).

Lastly, ψ can be taken to be a rapidly decaying function for even greater concentration and lesser robust risk, e.g., ψ ( θ ) = exp( -θ ) .

D Proofs for Example 4.1

Example 4.1. The data domain is the ball B glyph[lscript] ∞ (0 , 1) equipped with the glyph[lscript] 2 distance. The label domain is { 1 , 2 } . Subspace S 1 is given by S 1 = { x : x glyph[latticetop] e 1 = 0 } , and S 2 is given by S 2 = { x : x glyph[latticetop] e 2 = 0 } , where e 1 , e 2 are the standard unit vectors. The conditional densities are defined as

where α > 0 is a large constant. Finally, the classes are balanced, i.e., p Y (1) = p Y (2) = 1 / 2 . With these parameters, q 1 , q 2 are both (0 . 5 , α/n -1 , 0) -concentrated over their respective supports. Additionally, p is ( glyph[epsilon1], 0 , e -α / 2 + 2 glyph[epsilon1] ) -strongly-concentrated. A robust classifier f can be constructed following the proof of Theorem 3.1, and it obtains a robust accuracy R ( f, glyph[epsilon1] ) ≤ e -α / 2 + 2 glyph[epsilon1] . See Appendix D for more details.

It would be helpful to have Fig. 8 in mind for what follows.

/epsilon1

/epsilon1

f

(

x

) = 1

f

(

x

) = 2

-α/2

S+e-α/2

52

e

We will demonstrate that the classifier f illustrated in Fig. 8 has low robust risk. We define f for any x ∈ B ∞ (0 , 1) , as follows (let γ = e -α ):

The reason for splitting the cases for predicting 1 into two subcases, is that we will only need to analyse the first subcase and the second will not contribute to the robust risk.

Defining U 1 to be the set of all points x which admit an adversarial example for the class 1 , i.e. U 1 = { x : ∃ ¯ x such that ‖ ¯ x -x ‖ 2 ≤ glyph[epsilon1], f (¯ x ) = 2 } . It is clear that

U 2 is defined analogously, and

Now, R ( f, glyph[epsilon1] ) = 0 . 5 q 1 ( U 1 ) + 0 . 5 q 2 ( U 2 ) . We look at the first term, and simplify an upper bound:

Now, recall that q 1 = Unif( { x : | x 1 | ≤ γ/ 2 } ) , hence the expression in (21) evaluates to

The situation for q 2 is symmetric, and hence q 2 ( U 2 ) = 0 . 5( γ +4 glyph[epsilon1] ) . Adding them together, we see that the robust risk is upper bounded by

For the concentration parameters, note that the support of q 1 , i.e. , S 1 = { x : | x 1 | ≤ γ/ 2 , ‖ x ‖ ∞ ≤ 1 } has volume γ 2 n -1 = 2 n -1 exp( -α ) = 0 . 5 2 n exp( -n ( α/n )) ≤ 0 . 5 exp( -n ( α/n -1)) . Hence, q 1 is (0 . 5 , α/n -1 , 0) concentrated over S 1 . Similarly, q 2 is (0 . 5 , α/n -1 , 0) -concentrated over S 2 = { x : | x 2 | ≤ γ/ 2 , ‖ x ‖ ∞ ≤ 1 } . Finally,

E Proof of Theorem 4.1

Theorem 4.1. The set of active constraints A λ defined in (7) is robust, i.e., A λ ( x ′ ) = A λ ( x ) for all λx ′ ∈ C ( x ) , where C ( x ) is the polyhedron defined as

with F ⊆ K ◦ being a facet of the polyhedron K ◦ that x projects to, defined as

and V being the cone generated by the constraints active at (i.e., normal to) F , defined as

Proof. Let λx ′ ∈ C ( x ) as defined in (8). There exist f ∈ F ( x ) , v ∈ V ( x ) such that λx ′ = f + v . We will show that the projection of λx ′ onto F is given by f . Recall that

Choose any z ∈ K ◦ . Recall that Proj K ◦ ( λx ′ ) = min z ∈ K ◦ ‖ z -λx ′ ‖ 2 . Consider the objective,

We will show that 〈 z -f, v 〉 is negative. Consider any t i ∈ A λ ( x ) , and observe that 〈 z, t i 〉 ≤ 1 , as z ∈ K ◦ . However, from the definition of F ( x ) recall that 〈 f, t i 〉 = 1 . This implies that 〈 z -f, t i 〉 ≤ 0 . Using the fact that v ∈ V ( x ) , we expand the inner product as

The minimizer above is obtained at z = f ∈ C ( x ) , and hence we have shown that for all λx ′ ∈ C ( x ) , Proj K ◦ ( λx ′ ) = f . Finally, d ∗ λ ( x ′ ) = Proj K ◦ ( λx ′ ) = f ∈ F ( x ) . The theorem statement then follows from the definition of F ( x ) .

F Further Experimental Details and Qualitative Examples

F.1 Comparison along Projection on C λ

In this section, we will provide more details on attacking a classifier with perturbations that lie on our certified set C λ .

Using (24) in (23) to obtain

Details of the Projection Step II in (13) Recall from Theorem 4.1 that C λ ( x ) is defined as the Minkowski sum of the face F λ ( x ) (9) and the cone V λ ( x ) (10). Given an iterate x t , we can compute its projection onto C λ ( x 0 ) by solving the following optimization problem

Now every x ∈ C λ ( x 0 ) can be written as x = d + v where d ∈ F ( x 0 ) , v ∈ V ( x 0 ) . d is such that t glyph[latticetop] d = 1 for all t i ∈ A λ ( x 0 ) , and t glyph[latticetop] d ≤ 1 otherwise. Then, v is such that v = ∑ t i ∈ A λ ( x 0 ) α i t i for some α i ≥ 0 . Recall that T = [ t 1 , t 2 , . . . , t 2 M ] denotes the matrix containing the training data-points as well as their negations. Let the matrix A 1 ∈ R d ×| A λ ( x t ) | contain all the columns of T in A λ ( x t ) and let A 2 ∈ R d × (2 M -| A λ ( x t ) | ) contain all the remaining columns. Then the objective of the optimization problem above can be written as

(26) is a linearly constrained quadratic program, and can be solved by standard optimization tools. In particular, we use the qp solver from the python cvxopt library.

Visualization of the Certified Set In Fig. 9, we visualize the set A λ ( x ) , i.e. , the set of active constraints at the optimal solution of the dual problem, for several x in the MNIST test set. It can be seen that A λ ( x ) contains images of the same digit as x in most cases. This is expected, as taking the majority label among A λ ( x ) gives an accuracy of around 97% (solid blue curve in Fig. 5). Note that all these images t ∈ A λ ( x ) are also contained in the certified set, C λ ( x ) , which can be seen as t = t + 0 , with t ∈ F λ ( x ) , and 0 ∈ V λ ( x ) .

x

A

λ

(

x

)

x

A

λ

(

x

)

2

2

2

78

7

7

7

9

t

5

3

8

8

8

8

7

7

7

4

8

8

8

O

Visualization of Attacks restricted to the Certified Set In Fig. 10, we visualize the attacks restricted to our certified set C λ computed by (13) for the classifier g RS 0 . 02 . We observe that we can identify the correct digit from most of the attacked images, but the randomized smoothing classifier is incorrect at higher glyph[epsilon1] . A small number of these attacked images are close to the actual class decision boundary, and the class is ambigous. This is expected, both due to the inherent class ambiguity present in some MNIST images, as well as the large glyph[epsilon1] we are attacking with. For all these images, the prediction of our classifier g λ is certified to be accurate. For comparison, the RS certificate is unable to certify anything beyond glyph[epsilon1] ≥ 0 . 06 .

x

T

/epsilon1

0

0

.

09

0

.

2

0

.

4

0

.

6

5

5

4

4

4

4

3

4

4

8

8

9

5

5

8

8

3

1

3

2

2

4

4

3

2

3

7

8

2

3

7

8

7

G

F.2 Comparison along glyph[lscript] 2 balls

Exact Certification for g λ In this section, we first state and prove Theorem 4.1 that allows us to exactly compute the glyph[lscript] 2 certified radius for our classifier g λ at any point x .

Lemma F.1. For all x ′ ∈ C ( x ) as defined in Theorem 4.1, we have g λ ( x ′ ) = g λ ( x ) . Additionally, for all ‖ v ‖ 2 ≤ r 0 ( x ) we have g λ ( x + v ) = g λ ( x ) , where

where ext( F ( x )) is the set of extreme points of the polyhedron F ( x ) .

Discussion Before presenting the proof, let us ponder over the result. We see that (27) involves solving an optimization problem having as many constraints as the number of extreme points of the polyhedron F ( x ) . In general, this can be very large in high dimensions, and hence computationally inefficient to compute. As a result, while Lemma F.1 provides an exact certificate in theory, it is hard to use in practice without further approximations.

Proof. Recall that given a set A λ ( x ) , the polyhedron C ( x ) is defined in (8) as

where F ( x ) is a polyhedron, and V ( x ) is a cone given as the conic hull of A λ ( x ) , i.e. , V ( x ) = cone ( A λ ( x )) . We want to find the size r ( x ) of the largest glyph[lscript] 2 ball centered at x , i.e. , B ( x, r ( x )) , that can be inscribed within C ( x ) . From convex geometry [4], we know that any polyhedron can be described as an intersection over halfspaces obtained from its polar, C ◦ as

where H -( u , 0) = { x : 〈 u , x 〉 ≤ 0 } . The distance of x from the boundary of C ( x ) is the same as the smallest distance of x from any halfspace in (29). However, this distance is simply

In other words, we can expressing C ( x ) as the intersection over halfspaces whose normals lie in C ◦ ( x ) , i.e. ,

All that is left is to obtain a description of C ◦ ( x ) . This can be done by first expressing C ( x ) in a standard form, as

where ext ( F ( x )) denotes the extreme points of the polyhedron F ( x ) , conv ( · ) denotes the convex hull and cone ( · ) denotes the conic hull. Now we can apply a theorem in convex geometry to obtain the polar ([4, Th. 2.79]) as

Combining (31) and (33), we obtain the lemma statement. Finally, we note that as Theorem 4.1 shows that A λ ( x ′ ) = A λ ( x ) for all λx ′ ∈ C ( x ) , and the classifier g λ ( x ) is purely a function of A λ ( x ) , we have that g λ ( x ′ ) = g λ ( x ) for all λx ′ ∈ C ( x ) .

Black Box Attacks on g λ Since our classifier g λ is not explicitly differentiable with respect to its input, we use the HopSkipJump [9] black-box attack to obtain adversarial perturbations that cause a misclassification. The obtained attacked images, and the pertubation magnitude are shown in Fig. 11.

0

.

74

1

.

0

0

.

17

0

.

75

0

.

47

0

.

22

0

.

75

0

.

99

0

.

6

0

.

48

8

8

0

0

9

8

0

0

2

8

F.3 Details of Randomized Smoothing Certificates

For obtaining the certified accuracy curves using Randomized Smoothing [11] (solid lines in Figs. 5 and 6), we follow the certification and prediction algorithms for RS in [11, Sec 3.2.2]: given a base classifier f and a test image x , we use n 0 = 100 samples from an isotropic gaussian distribution N (0 , σ ) to prediction the majority class g RS σ ( x ) . We then use n = 100 samples to estimate the prediction probability p A under N (0 , σ ) with confidence atleast 0 . 999 . If p A is at least 0 . 5 , we report the certified radius σ Φ -1 ( p A ) . Otherwise, if p A < 0 . 5 , we abstain and return a certified radius of 0 .

Evaluating the Dual Classifier g λ We normalize each image in the MNIST dataset to have unit glyph[lscript] 2 norm, and resize to 32 × 32 . We pick 10000 images s i at random from the training set to obtain our data matrix S ∈ R 1024 × 10000 . We now pick a test point x , and then solve the dual problem (5) with λ = 2 , to obtain the dual solution d ∗ λ ( x ) . This allows us to then obtain the set of active constraints A λ ( x ) in (7). Then, we use the majority rule as the aggregate function in (11) to obtain the classfication output from the dual classifier g λ ( x ) . We sample 100 and 500 images x uniformly at random from the MNIST test set for generating the curves in Figs. 5 and 6, respectively. All experiments are performed on a NVIDIA GeForce RTX 2080 Ti GPU with 12 GB memory.

F.4 Discussion of Computational Cost

The computational complexity of obtaining the certificate C ( x ) in Theorem 4.1 is dominated by solving the linear optimization problem (6) to obtain the set A λ , which has n + M variables. It is known that in practice, the cost of solving LPs is much lower than the worst case [52], and it takes us 11 . 7 seconds on average on a single CPU for each image x without parallel processing.

For comparison to other certified defenses (like Randomized Smoothing), we perform T = 20 steps of (13), and each Step II of (13) requires solving a quadratic optimization program, given in (26). This is a linearly constrained quadratic program. In practice it takes us 18 . 1 seconds on average on a single CPU for each x without parallel processing.

F.5 Experiments on CIFAR-10

We provide experiments for our method applied to CIFAR-10. To do this, we first embed each CIFAR-10 image into a feature space, designed such that glyph[lscript] 2 -bounded perturbations in the feature space, i.e., φ ( x + v ) , ‖ v ‖ 2 ≤ glyph[epsilon1] , correspond to semantic perturbations in the input space, e.g., distorting the image. Such a feature space is obtained following recent popular work in learning perceptual metrics in vision [67], where the task is precisely to learn a feature representation where glyph[lscript] 2 distances align with human perception. We now use our subspace model on φ ( X ) , and perform exactly the experiments in Section 5, and make similar observations: we obtain reasonable robust accuracy using our method (Fig. 12 blue lines), and this accuracy is maintained within our certified polyhedra at high glyph[epsilon1] , where existing defenses are not robust.

Though such experiments showing feature space certificates have appeared in the literature [58], the question of whether the perceptual representation is itself susceptible to small adversarial perturbations still remains for future exploration.

0.00

0.25

0.50

0.75

1.00

/epsilon1

0.00

0.25

0.50

Projected Robust Accuracy

CIFAR-10 (LPIPS Features)

Ours Certified

g

RS

0

.

02

Empirical

g

RS

0

.

02

Certified

g

RS

0

.

04

Empirical

g

RS

0

.

04

Certified

0.00

0.25

0.50

0.75

1.00

/epsilon1

0.00

0.25

0.50

0.75

Projected Robust Accuracy

CIFAR-10 (SENet Features)

Ours Certified

h

RS

0

.

02

Empirical

h

RS

0

.

02

Certified

h

RS

0

.

04

Empirical

h

RS

0

.

04

Certified

F.6 Evaluating Theorem 3.1 on Empirical Distributions

The empirical results in [44, 5] create an empirical distribution ˆ p by selecting two classes from CIFAR-10, which simply places probability mass 1 /N on each of the N samples in the dataset. [44, 5] evaluate their lower bounds on ˆ p , and not on the actual real world distribution p , which might be arbitrarily complex. Hence, it is unclear whether the trends observed hold for p . Nevertheless, the

same experiment can be translated to our setting for any real world dataset, our Theorem 3.1 would then show the existence of a robust classifier for ˆ p . We would need to solve a discrete optimization problem for finding the concentrated sets in Theorem 3.1. Even though valid only for ˆ p , this analysis is interesting, and is reported in Fig. 13.

0.10

0.15

0.20

0.25

/epsilon1

0.00

0.25

0.50

0.75

1

-

δ

Theorem 3.1 on CIFAR-10

G Discussion on Strong Concentration

Are Natural Image Distributions Strongly Concentrated? Consider the task of classifying images of dogs vs cats. Any small glyph[lscript] 2 perturbation (say of size glyph[epsilon1] = 0 . 1 ) to an image of a dog is not likely to change it to an image of a cat - the true label (or a human decision) does not change with small perturbations. Let S cat be the set of images that are cats with a high confidence, and similarly for S dog . What we just said was that any image in the glyph[epsilon1] -expansion of S cat is not likely to be a dog, and vice versa. This is precisely the condition of separation in the definition of strong concentration: q dog ( S + glyph[epsilon1] cat ) ≤ γ , and q cat ( S + glyph[epsilon1] dog ) ≤ γ , for a small γ .

Now, despite the above, the task of practically classifying an image into a dog or a cat is not trivial at all, as the sets S dog and S cat might be very complex, and hence it might be computationally hard to find a predictor (neural network, or any other classifier) for distinguishing S dog and S cat . Using modern deep learning, one is able to learn an approximation to these sets that leads to small standard error, but this approximation is still bad enough that the learned predictor is very susceptible to adversarial examples. This is to say that task of robustly classifying an image into dog or cat is even harder.

Tightness of Strong Concentration In Definition 3.1, we are trying to obtain a sufficient condition for the existence of a robust classifier. There are two parts to Definition 3.1: the concentration, and the separation conditions. The first is essential, as Theorem 2.1 asserts that whenever a robust classifier exists for a data distribution, the class conditionals are concentrated (i.e., it is necessary).

glyph[negationslash]

As we argued above, the separation condition is not very strong, and should be satisfied by image distributions like MNIST, CIFAR-10 and ImageNet. Let us walk through this condition to see if it can be further relaxed. As we note at the start of Section 3, if all the class conditionals were to be concentrated on subsets S k that have high intersection among each other, then even benign classification would be hard (i.e., benign risk will be high), let alone robust classification. So, even for the existence of a good benign classifier, it is essential for S k ∩ S k ′ to be close to empty, for all k = k ′ .

glyph[negationslash]

Now, even if S k ∩ S k ′ were almost empty, for a robust classifier, we care about the glyph[epsilon1] -expansions of these sets to not intersect. In other words, we do not want an glyph[epsilon1] -perturbed cat to look like a dog. Hence, for the existence of a good robust classifier, S k ∩ S + glyph[epsilon1] k ′ should be close to empty, for all k = k ′ . This can be generalized in measure terms, to require q k ( S + glyph[epsilon1] k ′ ) ≤ γ , for a small γ . Upto this, all these conditions are essential for obtaining a robust classifier.

Now, note that in Definition 3.1, the expansion is taken for 2 glyph[epsilon1] , instead of glyph[epsilon1] . This is where our proof technique for Theorem 3.1 incurs a slack of glyph[epsilon1] , and we believe a different approach for constructing the robust classifier might be able to reduce the expansion requirement to glyph[epsilon1] . This could be an interesting future avenue for relaxing Definition 3.1 and thereby strengthening Theorem 3.1.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of adversarial robustness in multi-class classification. The authors show that a necessary condition for the existence of an adversarially robust classifier is the condition that at least one of the class conditional ditributions is \"concentrated\" in the sense that most of the probability mass is concentrated on a set of very small volume. The condition of concentration however is not sufficient for the existence of a robust classifier, because one also needs the class conditional ditributions to be well-separated. \n\nThe authors then study adversarial robustness for the special case of data distributions lying on a low-dimensional linear subspace. For this case, they show that robustness at any data point can be certified in a polyhedron that depends on the subspace representation of the point. Using experiments on MNIST dataset, they show that the polyhedron certificate can be complementary to the spherical certificate one can get with randomized smoothing.\n\nSTRENGTHS:\n- Necessary condition: The theorem showing the necessity of concentration for robustness results from a simple, but nice application of the Brunn-Minkowski inequality. I think this condition is novel, and complements existing results showing concentration of measure is bad for robustness\n- Polyhedral certificate: The theorem showing the polyhedral certificate for robustness appears non-trivial and interesting. The certificate obtained is complementary to the spherical certificates of randomized smoothing. \n- Writing: The paper is well-written and clear for the most part.\n\nWEAKNESSES:\n- Definition 3.1: I think the definition of strongly concentrated distributions is a misnomer. The critical assumption in this definition is in line 125 which assumes minimal overlap between the approximate supports of the class conditional distributions. Hence, this definition has more to do with \"well-separatedness\" of class conditional distributions, and in my opinion, has very little relation to concentration. \n- Novelty of the sufficient condition: If concentration on small volume sets is not sufficient for robustness, what is? As I state in the previous point, it is some notion of separation between class conditional distributions. Several papers already study such conditions. For example, [this](http://proceedings.mlr.press/v119/pydi20a/pydi20a.pdf) and [this](https://proceedings.neurips.cc/paper/2019/file/02bf86214e264535e3412283e817deaa-Paper.pdf) paper show that if the class-conditional distributions are well-separated in an optimal-transport distance, then there exists a robust classifier. It is also shown in these papers that such a separation condition holds empirically on real-world datasets. \n- The low-dimensional linear subspace assumption: First, I think this is a very strong assumption that real world datasets probably don't satisfy. Second, the motivation for studying distributions of this form appears weak, as one can imagine many other types of distributions that satisfy the necessary and sufficient conditions of sections 2 and 3 but are not supported on linear subspaces. Case in point is Example 3.1 in this very paper. \n- The utility of polyhedral certificate: Intuitively, one can imagine many \"benevolent perturbations\" to input data that do not affect the accuracy of a classifier. Spherical certificates for robustness make sense because the guarantee holds for any possible direction of the adversarial perturbation. A polyhedral certificate is not a \"worst-case\" certificate in this sense.\n\nQUESTIONS:\n- For $\\sigma = 0.04$ (red and green curves) Figure 6 seems to show that the the paper's method gives a worse certificate than randomized smoothing. Is this a typo?\n- Is it possible to find the radius of the largest $\\ell_p$ ball centered at x that fits inside C(x)? This will give a data-dependent spherical certificate at each point, which will be interesting. \n\nMinor suggestions:\n- I think it is better to show the dependence on dimension for the constants $\\epsilon, \\delta$ in the main theorems of sections 2 and 3. \n- The connections of the main theorem of section 4 to subspace clustering literature can probably be presented much earlier. Currently, the paper only brings up this connection at the end of section 4.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper gives a theoretical insight into the existence of robust classifiers for avoiding adversarial examples, which may be a potential direction for bridging the gap between human perception and machine intelligence. Specifically, this paper reveals the sufficiency and necessity of data concentration for robust classification. The authors conclude that concentration on small-volume subsets of the input space is the key to avoiding adversarial examples for machine learning. In two experiments, the proposed theoretical results are shown to be effective in improving projected robust accuracy and certified accuracy.\n\nSTRENGTHS:\n- The research problem is quite interesting, and is significant for the development of machine robustness and safety.\n- This paper is well-organized. The analyses are rigorous and solid.\n\nWEAKNESSES:\n- The main theoretical results are based on $\\ell_2$-normed adversarial training, whether the conclusions are still applicable on $\\ell_\\inf$ norm is unknown.\n\n- The empirical results are insufficient. The experiments are only conducted on every small dataset, whether the theoretical conclusion still holds on larger natural datasets is unknown. \n\n- the conclusion related to subspace training is similar to some existing adversarial training methods, such as [1], [2], which shows that adversarial training on subspace can help alleviate overfitting and improve generalization performance. Is there any difference between these works?  \n[1] Li et al., Subspace adversarial training, CVPR 2022.  \n[2] Li et al., On the adversarial robustness of subspace learning, TIP 2020.\n\n- Another question is that whether the subspace preserve training could be applied to transferred adversarial attacks. Could you make several discussions to justify this?\n\nQUESTIONS:\nPlease refer to weaknesses.\n\nLIMITATIONS:\nLimitations are well-discussed. No potential negative societal impact is found.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that adversarial examples can be avoided allowing for robust classification as long as the data distribution obeys certain properties. In particular, they show that the the input data distribution needs to be \"well-concentrated\" and not overlap too much in order to guarantee the existence of a robust classifier with provable certificates. They support their theoretical guarantees with experimental results on MNIST.\n\nSTRENGTHS:\n1. The problem of adversarial robustness is extremely important and impactful. Further, in the face of hardness results and well known existence of adversarial examples, this is a difficult problem to tackle\n2. The authors formalize the notion of \"well-concentrated\" distributions very well. I think the idea of distribution-specific robustness is well motivated and the right way to approach this problem.\n3. I think the paper is well written and the mathematical arguments look non-trivial and sound.\n\nWEAKNESSES:\n1. Since this work is primarily a theoretical contribution, I consider this a minor point - I would like to see experimental results done on a few settings larger than MNIST. (say Cifar-10, TinyImagenet)\n2. My main concern with the paper is that Definition 3.1 is possibly too strong. It almost seems like the definition is equivalent to creating a robust classifier since it ensures that the different classes are well separated and well concentrated. Is there some way to interpolate between Definition 2.2 and 3.1 which is the minimum assumptions required to guarantee the existence of a robust classifier?\n\nQUESTIONS:\n1. In line 229, do you mean strongly concave? Unless you transform the problem to a minimization problem.\n2. The assumption that adversarial perturbations do not take $x$ outside the domain $\\mathcal{X}$ seems somewhat unreasonable. Is this really necessary? I agree that this is needed for $f(\\bar{x})$ to be well defined, but isn't it even easier to guarantee robustness if it is not defined?\n3. The notion of requiring the data distribution to concentrate on an exponentially small volume of the input space looks closely related to the hardness results from Wang et al. \"Attack of the Tails: Yes, You Really Can Backdoor Federated Learning\". Their results show that adversariabl robustness in FL is provably hard which is the opposite of this work. However, I feel they are somewhat related. Can you comment?\n4. In (1), can the definition be changed to $f(\\bar{x}) \\neq f(x)$? This way robustness and accuracy can be disentangled. It seems somewhat unintuitive that an inaccurate classifier cannot be robust. I would expect that the $0$ classifier is the most robust.\n5. I have perhaps missed this, but I couldn't find any discussion of the computational cost of certifying robustness using this approach. I would like to see the dependence on $d$.\n6. In Figure 5, the \"Ours Certified\" line is 1.0 for all $\\epsilon \\in [0, 1]$. However, the certified region should not be symmetric around $x$, so this doesn't make sense to me.\n\nLIMITATIONS:\nThe authors discuss some of the limitations of their work in Section 6. But I would like them to consider some of my questions above and include those as well if they are relevant.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first studies the existence problem of a robust classifier from a data concentration perspective. In particular, they prove that the (strong) concentration property of class-conditional data distributions characterizes a sufficient and necessary condition for determining whether a robust classifier exists. This seemingly contradicting result complements existing works on impossibility results of adversarially robust classification [17, 43, 11]. Built upon the assumption that the underlying distribution is concentrated on a union of low-dimensional linear subspaces, the paper proposes a new way to derive robustness certificates by leveraging techniques from sparse representation and subspace clustering. Experiments on MINST are provided as empirical evidence supporting the effectiveness of the proposed certification method.\n\nSTRENGTHS:\nThe theoretical results are mostly well-written, and their implications are discussed in detail. I did not check the proof, but I think the results are technically correct. In addition, I very much like how this paper structures its theoretical results: a main theorem followed by a proof sketch and a detailed discussion of its implication.\n\nThe first part of this paper shows a somewhat surprising result compared with existing works deriving impossibility results of robust classification [17, 43, 11]. Assuming a robust classifier exists (i.e., a human classifier), the paper shows that the data distribution has to be concentrated in a small-volume subset in the input space. A robust classifier is guaranteed to exist if the class conditionals are both concentrated and well-separated. Given that human is robust to small, adversarial perturbations exists, these results suggest natural data distributions are likely to be concentrated, and existing impossibility results may be due to unrealistic distributional assumptions.\n\nThe second part of this paper aims to study how to leverage the underlying data structure (i.e., the fact/assumption that data are concentrated on a union of low-dimensional subspaces) for adversarially robust classification. The presented method uses the classical literature of sparse representation, which seems novel to the field of robustness certification. Overall, the paper did a good job of discussing their theoretical results and explaining their broader implications.\n\nWEAKNESSES:\nRegarding the first part of the paper, I think the presented theoretical possibility results are correct, which complements existing works [17, 43, 11]. However, I want to point out several missing references [1-4]. These works studied the concentration of measure phenomenon and its connection to robust classification/learning. There are some differences, but I believe those existing works are very relevant, thus, should be cited and discussed properly. In particular, your definition of (strongly) concentrated distribution and the notion of concentration of measure introduced in [2] seems different. Also, a key assumption imposed in them [1-4] is that there exists a ground-truth classifier (i.e., a human classifier) and a machine learning classifier is imperfect (see the definition of intrinsic robustness introduced in [3]). In other words, they assume a constant (say 1%) clean error of any machine learning classifier always exists. However, this assumption seems not to be considered in this work. Could you discuss those differences with comparisons to your work?\n\n[1] The Relationship Between High-Dimensional Geometry and Adversarial Examples. Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Wattenberg, Ian Goodfellow. ArXiv: 1801.02774.\n\n[2] The Curse of Concentration in Robust Learning: Evasion and Poisoning Attacks from Concentration of Measure. Saeed Mahloujifar, Dimitrios I. Diochnos, Mohammad Mahmoody. AAAI 2019\n\n[3] Empirically Measuring Concentration: Fundamental Limits on Intrinsic Robustness. Saeed Mahloujifar, Xiao Zhang, Mohammad Mahmoody, David Evans. NeurIPS 2019\n\n[4] Understanding Intrinsic Robustness Using Label Uncertainty. Xiao Zhang, David Evans. ICLR 2022\n\nIn addition, I am a little confused about the degree of concentration implied by Theorem 2.1. Theorem 2.1 says that if an $(\\epsilon, \\delta)$-robust classifier exists, all the class conditionals are $(\\epsilon, K\\delta)$-concentrated if they are balanced. This result seems interesting, but from my perspective, a discussion of whether this result holds for most data distributions or only applies to certain distributions is missing. Simply judging from Definition 2.1, there is little clue on what data distribution is concentrated and, more importantly, their degree of concentration. The constants $c_1$ and $c_2$ are vaguely described, so could you explain their dependence on other parameters such as the density function q and $\\delta$? If the data distribution lies on a low-dimensional manifold, does it always imply the underlying distribution is concentrated?  I recommend the authors provide examples of concentrated distributions with a characterization of their degree of concentration. \n\nThe result of Theorem 3.1 is not very surprising. The assumption that the underlying class conditional distribution is strongly concentrated seems strong. To be honest, I am not convinced that this assumption will hold for natural image distributions. Otherwise, image classification would be rather a simple task. Therefore, it would be important for the authors to clarify this in the response. \n\nRegarding the second part, I have to say I am not familiar with the literature on sparse representation. I can understand the motivation and main message of leveraging the underlying data structure, which is interesting. Still, I am concerned about the assumption that data distributions are concentrated on a union of low-dimensional linear subspaces to be too strong to hold in practice. The applicability of your method seems to be heavily dependent on whether the distribution satisfies the imposed subspace assumption. MNIST may be more aligned with such an assumption, but Gaussian-like data distributions such as CIFAR-10 and ImageNet may not be. Therefore, I am not confident that the proposed method will work for other image benchmarks like CIFAR-10.\n\nQUESTIONS:\n1. Could you provide a detailed discussion with related works [1-4] and explain why data concentration implies different conclusions about the possibility/impossibility of adversarially robust classification?\n\n2. If we assume a human classifier is adversarially robust to image classification tasks, does Theorem 2.1 implies how concentrated the input distribution is? If so, what are the important factors affecting the degree of concentration?\n\n3. Do the constants $c_1$ and $c_2$ in Definition 2.2 depend on the underlying density $q$ and $\\delta$? If so, could you specify their dependencies?\n\n4. Could you clarify the generalizability of the proposed certified defense in Section 4 if the underlying data distribution deviates from the imposed assumption regarding linear subspaces?\n\nLIMITATIONS:\nThe paper discusses the limitations and implications of their results well in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}