SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ambar Pal ambar@jhu.edu
Jeremias Sulam jsulam1@jhu.edu
Réné Vidal vidalr@upenn.edu
The susceptibility of modern machine learning classifiers to adversarial examples has motivated theoretical results suggesting that these might be unavoidable. However, these results can be too general to be applicable to natural data distributions. Indeed, humans are quite robust for tasks involving vision. This apparent conflict motivates a deeper dive into the question: Are adversarial examples truly unavoidable? In this work, we theoretically demonstrate that a key property of the data distribution – concentration on small-volume subsets of the input space – determines whether a robust classifier exists. We further demonstrate that, for a data distribution concentrated on a union of low-dimensional linear subspaces, utilizing structure in data naturally leads to classifiers that enjoy data-dependent polyhedral robustness guarantees, improving upon methods for provable certification in certain regimes.
1 Introduction, Motivation and Contributions
Research in adversarial learning has shown that traditional neural network based classification models are prone to anomalous behaviour when their inputs are modified by tiny, human-imperceptible perturbations. Such perturbations, called adversarial examples, lead to a large degradation in the accuracy of classifiers [55]. This behavior is problematic when such classification models are deployed in security sensitive applications. Accordingly, researchers have and continue to come up with defenses against such adversarial attacks for neural networks.
Such defenses [49, 60, 42, 22] modify the training algorithm, alter the network weights, or employ preprocessing to obtain classifiers that have improved empirical performance against adversarially corrupted inputs. However, many of these defenses have been later broken by new adaptive attacks [1, 8]. This motivated recent impossibility results for adversarial defenses, which aim to show that all defenses admit adversarial examples. While initially such results were shown for specially parameterized data distributions [18], they were subsequently expanded to cover general data distributions on the unit sphere and the unit cube [48], as well as for distributions over more general manifolds [12].
On the other hand, we humans are an example of a classifier capable of very good (albeit imperfect [17]) robust accuracy against  $\ell_2$ -bounded attacks for natural image classification. Even more, a large body of recent work has constructed certified defenses [11, 63, 10, 29, 19, 54] which obtain non-trivial performance guarantees under adversarially perturbed inputs for common datasets like MNIST, CIFAR-10 and ImageNet. This apparent contention between impossibility results and the existence of robust classifiers for natural datasets indicates that the bigger picture is more nuanced, and motivates a closer look at the impossibility results for adversarial examples.
Our first contribution is to show that these results can be circumvented by data distributions whose mass concentrates on small regions of the input space. This naturally leads to the question of whether such a construction is necessary for adversarial robustness. We answer this question in the affirmative, formally proving that a successful defense exists only when the data distribution concentrates on an
exponentially small volume of the input space. At the same time, this suggests that exploiting the inherent structure in the data is critical for obtaining classifiers with broader robustness guarantees.
Surprisingly, almost all certified defenses do not exploit any structural aspects of the data distribution like concentration or low-dimensionality. Motivated by our theoretical findings, we study the special case of data distributions concentrated near a union of low-dimensional linear subspaces, to create a certified defense for perturbations that go beyond traditional  $\ell_p$ -norm bounds. We find that simply exploiting the low-dimensional data structure leads to a natural classification algorithm for which we can derive norm-independent polyhedral certificates. We show that our method can certify accurate predictions under adversarial examples with an  $\ell_p$  norm larger than what can be certified by applying existing, off-the-shelf methods like randomized smoothing [11]. Thus, we demonstrate the importance of structure in data for both the theory and practice of certified adversarial robustness.
More precisely, we make the following main contributions in this work:
1. We formalize a notion of  $(\epsilon, \delta)$ -concentration of a probability distribution  $q$  in Section 2, which states that  $q$  assigns at least  $1 - \delta$  mass to a subset of the ambient space having volume  $O(\exp(-n\epsilon))$ . We show that  $(\epsilon, \delta)$ -concentration of  $q$  is a necessary condition for the existence of any classifier obtaining at most  $\delta$  error over  $q$ , under perturbations of size  $\epsilon$ .
2. We find that  $(\epsilon, \delta)$ -concentration is too general to be a sufficient condition for the existence of a robust classifier, and we follow up with a stronger notion of concentration in Section 3 which is sufficient for the existence of robust classifiers. Following this stronger notion, we construct an example of a strongly-concentrated distribution, which circumvents existing impossibility results on the existence of robust classifiers.
3. We then consider a data distribution  $q$  concentrated on a union of low-dimensional linear subspaces in Section 4. We construct a classifier for  $q$  that is robust to perturbations following threat models more general than  $\ell_p$ . Our analysis results in polyhedral certified regions whose faces and extreme rays are described by selected points in the training data.
4. We perform empirical evaluations on MNIST in Section 5, demonstrating that our certificates are complementary to existing off-the-shelf approaches like Randomized Smoothing (RS), in the sense that both methods have different strengths. In particular, we demonstrate a region of adversarial perturbations where our method is certifiably robust, but RS is not. We then combine our method with RS to obtain certificates that enjoy the best of both worlds.
2 Existence of Robust Classifier Implies Concentration
We will consider a classification problem over  $\mathcal{X} \times \mathcal{Y}$  defined by the data distribution  $p$  such that  $\mathcal{X}$  is bounded and  $\mathcal{Y} = \{1, 2, \dots, K\}$ . We let  $q_k$  denote the conditional distribution  $p_{X|Y=k}$  for class  $k \in \mathcal{Y}$ . We will assume that the data is normalized, i.e.,  $\mathcal{X} = B_{\ell_2}(0,1)$ , and the boundary of the domain is far from the data, i.e., for any  $x \sim q_k$ , an adversarial perturbation of  $\ell_2$  norm at most  $\epsilon$  does not take  $x$  outside the domain  $\mathcal{X}$ .
We define the robust risk of a classifier  $f\colon \mathcal{X}\to \mathcal{Y}$  against an adversary making perturbations whose  $\ell_2$  norm is bounded by  $\epsilon$  as
$$
R (f, \epsilon) = \Pr_ {(x, y) \sim p} (\exists \bar {x} \in B _ {\ell_ {2}} (x, \epsilon) \text {s u c h t h a t} f (\bar {x}) \neq y). \tag {1}
$$
We can now define a robust classifier in our setting.
Definition 2.1 (Robust Classifier). A classifier  $g$  is defined to be  $(\epsilon, \delta)$ -robust if the robust risk against perturbations with  $\ell_2$  norm bounded by  $\epsilon$  is at most  $\delta$ , i.e., if  $R(g, \epsilon) \leq \delta$ .
The goal of this section is to show that if our data distribution  $p$  admits an  $(\epsilon, \delta)$ -robust classifier, then  $p$  has to be concentrated. Intuitively, this means that  $p$  assigns a "large" measure to sets of "small" volume. We define this formally now.
See Section 6 for more details.
2More details in Appendix A.
Note that for  $f(\bar{x})$  to be defined, it is implicit that  $\bar{x}\in \mathcal{X}$  in (1).
Definition 2.2 (Concentrated Distribution). A probability distribution  $q$  over a domain  $\mathcal{X} \subseteq \mathbb{R}^n$  is said to be  $(C, \epsilon, \delta)$ -concentrated, if there exists a subset  $S \subseteq \mathcal{X}$  such that  $q(S) \geq 1 - \delta$  but  $\operatorname{Vol}(S) \leq C \exp(-n\epsilon)$ . Here, Vol denotes the standard Lebesgue measure on  $\mathbb{R}^n$ , and  $q(S)$  denotes the measure of  $S$  under  $q$ .
With the above definitions in place, we are ready to state our first main result.
Theorem 2.1. If there exists an  $(\epsilon, \delta)$ -robust classifier  $f$  for a data distribution  $p$ , then at least one of the class conditionals  $q_{1}, q_{2}, \ldots, q_{K}$ , say  $q_{\bar{k}}$ , must be  $(\bar{C}, \epsilon, \delta)$ -concentrated. Further, if the classes are balanced, then all the class conditionals are  $(C_{\max}, \epsilon, K\delta)$ -concentrated. Here,  $\bar{C} = \mathrm{Vol}\{x: f(x) = \bar{k}\}$ , and  $C_{\max} = \max_k \mathrm{Vol}\{x: f(x) = k\}$  are constants dependent on  $f$ .
The proof is a natural application of the Brunn-Minkowski theorem from high-dimensional geometry, essentially using the fact that an  $\epsilon$ -shrinkage of a high-dimensional set has very small volume. We provide a brief sketch here, deferring the full proof to Appendix A.
Proof Sketch. Due to the existence of a robust classifier  $f$ , i.e.,  $R(f, \epsilon) \leq \delta$ , the first observation is that there must be at least one class which is classified with robust accuracy at least  $1 - \delta$ . Say this class is  $k$ , and the set of all points which do not admit an  $\epsilon$ -adversarial example for class  $k$  is  $S$ . Now, the second step is to show that  $S$  has the same measure (under  $q_k$ ) as the  $\epsilon$ -shrinkage (in the  $\ell_2$  norm) of the set of all points classified as class  $k$ . Finally, the third step involves using the Brunn-Minkowski theorem, to show that this  $\epsilon$ -shrinkage has a volume  $O(\exp(-n\epsilon))$ , thus completing the argument.
Discussion on Theorem 2.1. We pause here to understand some implications of this result.
- Firstly, recall the apparently conflicting conclusions from Section 1 between impossibility results (suggesting that robust classifiers do not exist) and the existence of robust classifiers in practice (such as that of human vision for natural data distributions). Theorem 2.1 shows that whenever a robust classifier exists, the underlying data distribution has to be concentrated. In particular, this suggests that natural distributions corresponding to MNIST, CIFAR and ImageNet might be concentrated. This indicates a resolution to the conflict: concentrated distributions must somehow circumvent existing impossibility results. Indeed, this is precisely what we will show in Section 3.
- Secondly, while our results are derived for the  $\ell_2$  norm, it is not very hard to extend this reasoning to general  $\ell_p$  norms. In other words, whenever a classifier robust to  $\ell_p$ -norm perturbations exists, the underlying data distribution must be concentrated.
- Thirdly, Theorem 2.1 has a direct implication towards classifier design. Since we now know that natural image distributions are concentrated, one should design classifiers that are tuned for small-volume regions in the input space. This might be the deeper principle behind the recent success [66] of robust classifiers adapted to  $\ell_p$ -ball like regions in the input space.
- Finally, the extent of concentration implied by Theorem 2.1 depends on the classifier  $f$ , via the parameters  $\epsilon, \delta$  and  $\bar{C}$ . On one hand, we get high concentration when  $\epsilon$  is large,  $\delta$  is small, and  $\bar{C}$  is small. On the other hand, if the distribution  $p$  admits an  $(\epsilon, \delta)$ -robust classifier such that  $\bar{C}$  is large (e.g., a constant classifier), then we get low concentration via Theorem 2.1. This is not a limitation of our proof technique, but a consequence of the fact that some simple data distributions might be very lowly concentrated, but still admit very robust classifiers, e.g., for a distribution having  $95\%$  dogs and  $5\%$  cats, the constant classifier which always predicts "dog" is quite robust.
We have thus seen that data concentration is a necessary condition for the existence of a robust classifier. A natural question is whether it is also sufficient. We address this question now.
3 Strong Concentration Implies Existence of Robust Classifier
Say our distribution  $p$  is such that all the class conditionals  $q_{1}, q_{2}, \ldots, q_{k}$  are  $(C, \epsilon, \delta)$ -concentrated. Is this sufficient for the existence of a robust classifier? The answer is negative, as we have not precluded the case where all of the  $q_{k}$  are concentrated over the same subset  $S$  of the ambient space. In other words, it might be possible that there exists a small-volume set  $S \subseteq \mathcal{X}$  such that  $q_{k}(S)$  is high for all  $k$ . This means that whenever a data point lies in  $S$ , it would be hard to distinguish
which class it came from. In this case, even an accurate classifier might not exist, let alone a robust classifier<sup>4</sup>. To get around such issues, we define a stronger notion of concentration, as follows.
Definition 3.1 (Strongly Concentrated Distributions). A distribution  $p$  is said to be  $(\epsilon, \delta, \gamma)$ -strongly-concentrated if each class conditional distribution  $q_k$  is concentrated over the set  $S_k \subseteq \mathcal{X}$  such that  $q_k(S_k) \geq 1 - \delta$ , and  $q_k\left(\bigcup_{k' \neq k} S_{k'}^{+2\epsilon}\right) \leq \gamma$ , where  $S^{+\epsilon}$  denotes the  $\epsilon$ -expansion of the set  $S$  in the  $\ell_2$  norm, i.e.,  $S^{+\epsilon} = \{x: \exists \bar{x} \in S \text{ such that } \| x - \bar{x} \|_2 \leq \epsilon\}$ .
In essence, Definition 3.1 states that each of the class conditionals are concentrated on subsets of the ambient space, which do not intersect too much with one another<sup>5</sup>. Hence, it is natural to expect that we would be able to construct a robust classifier by exploiting these subsets. Building upon this idea, we are able to show Theorem 3.1:
Theorem 3.1. If the data distribution  $p$  is  $(\epsilon, \delta, \gamma)$ -strongly-concentrated, then there exists an  $(\epsilon, \delta + \gamma)$ -robust classifier for  $p$ .
The basic observation behind this result is that if the conditional distributions  $q_{k}$  had disjoint supports which were well-separated from each other, then one could obtain a robust classifier by predicting the class  $k$  on the entire  $\epsilon$ -expansion of the set  $S_{k}$  where the conditional  $q_{k}$  concentrates, for all  $k$ . To go beyond this idealized case, we can exploit the strong concentration condition to carefully remove the intersections at the cost of at most  $\gamma$  in robust accuracy. We make these arguments more precise in the full proof, deferred to Appendix B, and we pause here to note some implications for existing results.
Implications for Existing Impossibility Results. To understand how Theorem 3.1 circumvents the previous impossibility results, consider the setting from [48] where the data domain is the sphere  $\mathbb{S}^{n - 1} = \{x\in \mathbb{R}^n\colon \| x\| _2 = 1\}$ , and we have a binary classification setting with class conditionals  $q_{1}$  and  $q_{2}$ . The adversary is allowed to make bounded perturbations w.r.t. the geodesic distance. In this setting, it can be shown (see [48, Theorem 1]) that any classifier admits  $\epsilon$ -adversarial examples for the minority class (say class 1), with probability at least
$$
1 - \alpha_ {q _ {1}} \beta \exp \left(- \frac {n - 1}{2} \epsilon^ {2}\right), \tag {2}
$$
where  $\alpha_{q_1} = \sup_{x\in \mathbb{S}^{n - 1}}q_1(x)$  depends on the conditional distribution  $q_{1}$ , and  $\beta$  is a normalizing constant that depends on the dimension  $n$ . Note that this result assumes little about the conditional  $q_{1}$ . Now, by constructing a strongly-concentrated data distribution over the domain, we will show that the lower bound in (2) becomes vacuous.
Example 3.1. The data domain is the unit sphere  $\mathbb{S}^{n - 1}$  equipped with the geodesic distance  $d$ . The label domain is  $\{1,2\}$ .  $P$  is an arbitrary, but fixed, point lying on  $\mathbb{S}^{n - 1}$ . The conditional density of class 1, i.e.,  $q_{1}$  is now defined as
Figure 1: A plot of  $q_{1}$ . Redder colors denote a larger density, and the gray plane denotes the robust classifier.
$$
q _ {1} (x) = \left\{ \begin{array}{l l} \frac {1}{C} \frac {1}{\sin^ {n - 2} d (x , P)}, & \text {i f} d (x, P) \leq 0. 1 \\ 0, & \text {o t h e r w i s e} \end{array} \right.,
$$
where  $C = 0.1$  is a normalizing constant. The conditional density of class 2 is defined to be uniform over the complement of the support of  $q_{1}$ , i.e.  $q_{2} = \mathrm{Unif}(\{x \in \mathbb{S}^{n - 1} \colon d(x, P) > 0.1\})$ . Finally, the classes are balanced, i.e.,  $p_{Y}(1) = p_{Y}(2) = 1 / 2$ .
The data distribution constructed in Example 3.1 makes Eq. (2) vacuous, as the supremum over the density  $q_{1}$  is unbounded. Additionally,
the linear classifier defined by the half-space  $\{x\colon \langle x,P\rangle \leq \cos (0.1)\}$  is robust (Appendix C provides a derivation of the robust risk, and further comments on generalizing this example). Example 3.1 is plotted for  $n = 3$  dimensions in Fig. 1.
4Recall that classifier not accurate at a point  $(x,y)$  , i.e.,  $f(x)\neq y$  , is by definition not robust at  $x$  , as a  $v = 0$  perturbation is already sufficient to ensure  $f(x + v)\neq y$
5 More detailed discussion in Appendix G.
Compatibility of Theorem 3.1 with existing Negative Results Thus, we see that strongly concentrated distributions are able to circumvent existing impossibility results on the existence of robust classifiers. However, this does not invalidate any existing results. Firstly, measure-concentration-based results [48, 18, 12] provide non-vacuous guarantees given a sufficiently flat (not concentrated) data distribution, and hence do not contradict our results. Secondly, our results are existential and do not provide, in general, an algorithm to construct a robust classifier given a strongly-concentrated distribution. Hence, we also do not contradict the existing stream of results on the computational hardness of finding robust classifiers [6, 56, 47]. Our positive results are complementary to all such negative results, demonstrating a general class of data distributions where robust classifiers do exist.
For the reminder of this paper, we will look at a specific member of the above class of strongly concentrated data distributions and show how we can practically construct robust classifiers.
4 Adversarily Robust Classification on Union of Linear Subspaces
The union of subspaces model has been shown to be very useful in classical computer vision for a wide variety of tasks, which include clustering faces under varying illumination, image segmentation, and video segmentation [57]. Its concise mathematical description often enables the construction and theoretical analysis of algorithms that also perform well in practice. In this section, we will study robust classification on data distributions concentrated on a union of low-dimensional linear subspaces. This data structure will allow us to obtain a non-trivial, practically relevant case where we can show a provable improvement over existing methods for constructing robust classifiers in certain settings. Before delving further, we now provide a simple example (which is illustrated in Fig. 2) demonstrating how distributions concentrated about linear subspaces are concentrated precisely in the sense of Definition 3.1, and therefore allow for the existence of adversarially robust classifiers.
Example 4.1. The data domain is the ball  $B_{\ell_{\infty}}(0,1)$  equipped with the  $\ell_2$  distance. The label domain is  $\{1,2\}$ . Subspace  $S_{1}$  is given by  $S_{1} = \{x: x^{\top}e_{1} = 0\}$ , and  $S_{2}$  is given by  $S_{2} = \{x: x^{\top}e_{2} = 0\}$ , where  $e_1, e_2$  are the standard unit vectors. The conditional densities are defined as
$$
q _ {1} = \operatorname {U n i f} (\{x: \| x \| _ {\infty} \leq 1, | x ^ {\top} e _ {1} | \leq e ^ {- \alpha} / 2 \}), a n d,
$$
$$
q _ {2} = \operatorname {U n i f} (\{x: \| x \| _ {\infty} \leq 1, | x ^ {\top} e _ {2} | \leq e ^ {- \alpha} / 2 \}),
$$
where  $\alpha > 0$  is a large constant. Finally, the classes are balanced, i.e.,  $p_{Y}(1) = p_{Y}(2) = 1/2$ . With these parameters,  $q_{1}, q_{2}$  are both  $(0.5, \alpha / n - 1, 0)$ -concentrated over their respective supports. Additionally,  $p$  is  $(\epsilon, 0, e^{-\alpha} / 2 + 2\epsilon)$ -strongly-concentrated. A robust classifier  $f$  can be constructed following the proof of Theorem 3.1, and it obtains a robust accuracy  $R(f, \epsilon) \leq e^{-\alpha} / 2 + 2\epsilon$ . See Appendix D for more details.
Figure 2: A plot of  $q_{1}$  (orange),  $q_{2}$  (violet) and the decision boundaries of  $f$  (dashed).
We will now study a specific choice of  $p$  that generalizes Example 4.1 and will let us move beyond the above simple binary setting. Recall that we have a classification problem specified by a data distribution  $p$  over the data domain  $\mathcal{X} \times \mathcal{Y} = B(0,1) \times \{1,2,\dots,K\}$ . Firstly, the classes are balanced, i.e.,  $p_{Y}(k) = 1 / K$  for all  $k \in \mathcal{V}$ . Secondly, the conditional density, i.e.,  $q_{k} = p_{X|Y = k}$ , is concentrated on the set  $S_{k}^{+\gamma} \cap \mathcal{X}$ , where  $S_{k}$  is a low-dimensional linear subspace, and the superscript denotes an  $\gamma$ -expansion, for a small  $\gamma > 0$ .
For the purpose of building our robust classifier, we will assume access to a training dataset of  $M$  clean data points  $(s_1,y_1),(s_2,y_2),\ldots ,(s_M,y_M)$ , such that, for all  $i$ , the point  $s_i$  lies exactly on one of the  $K$  low-dimensional linear subspaces. We will use the notation  $\mathbf{S} = [s_1,s_2,\dots ,s_M]$  for the training data matrix and  $\mathbf{y} = (y_{1},y_{2},\dots ,y_{M})$  for the training labels. We will assume that  $M$  is large enough that every  $x\in \cup_k S_k$  can be represented as a linear combination of the columns of  $\mathbf{S}$ .
Now, the robust classification problem we aim to tackle is to obtain a predictor  $g \colon \mathcal{X} \to \mathcal{Y}$  which obtains a low robust risk, with respect to an additive adversary  $\mathcal{A}$  that we now define. For any data-point  $x \sim p$ ,  $\mathcal{A}$  will be constrained to make an additive perturbation  $v$  such that  $\mathrm{dist}_{\ell_2}(x + v, \cup_i S_i) \leq \epsilon$ . In other words, the attacked point can have  $\ell_2$  distance at most  $\epsilon$  from any of the linear
subspaces  $S_{1},\ldots ,S_{k}$ . Note that  $\mathcal{A}$  is more powerful than an  $\ell_2$ -bounded adversary as the norm of the perturbation  $\| v\| _2$  might be large, as  $v$  might be parallel to a subspace.
Under such an adversary  $\mathcal{A}$ , given a (possibly adversarially perturbed) input  $x$ , it makes sense to try to recover the corresponding point  $s$  lying on the union of subspaces, such that  $x = s + n$ , such that  $\|n\|_2 \leq \epsilon$ . One way to do this is to represent  $s$  as a linear combination of a small number of columns of  $\mathbf{S}$ , i.e.,  $x = \mathbf{Sc} + n$ . This can be formulated as an optimization problem that minimizes the cardinality of  $c$ , given by  $\|c\|_0$ , subject to an approximation error constraint. Since such a problem is hard because of the  $\ell_0$  pseudo-norm, we relax this to the problem
$$
\min  _ {c} \| c \| _ {1} \text {s . t .} \| x - \mathbf {S} c \| _ {2} \leq \epsilon . \tag {3}
$$
Under a suitable choice of  $\lambda$ , this problem can be equivalently written as
$$
\min  _ {c, e} \| c \| _ {1} + \frac {\lambda}{2} \| e \| _ {2} ^ {2} \text {s . t .} x = \mathbf {S} c + e, \tag {4}
$$
for which we can obtain the dual problem given by
$$
\max  _ {d} \langle x, d \rangle - \frac {1}{2 \lambda} \| d \| _ {2} ^ {2} \text {s . t .} \| \mathbf {S} ^ {\top} d \| _ {\infty} \leq 1. \tag {5}
$$