SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our main observation is to leverage the stability of the set of active constraints of this dual to obtain a robust classifier. One can note that each constraint of Eq. (5) corresponds to one training data point  $s_i$  - when the  $i^{\text{th}}$  constraint is active at optimality,  $s_i$  is being used to reconstruct  $x$ . Intuitively, one should then somehow use the label  $y_i$  while predicting the label for  $x$ . Indeed, we will show that predicting the majority label among the active  $y_i$  leads to a robust classifier.
We will firstly obtain a geometric characterization of the problem in Eq. (5) by viewing it as the evaluation of a projection operator onto a certain convex set, illustrated in Fig. 3. Observe that for  $\lambda > 0$ , the objective (5) is strongly concave in  $d$  and the problem has a unique solution, denoted by  $d_{\lambda}^{*}(x)$ . It is not hard to show that this solution can be obtained by the projection operator
$$
d _ {\lambda} ^ {*} (x) = \left(\underset {d} {\arg \min } \| \lambda x - d \| _ {2} \text {s . t .} \| \mathbf {S} ^ {\top} d \| _ {\infty} \leq 1\right) = \operatorname {P r o j} _ {K ^ {\circ}} (\lambda x), \tag {6}
$$
where  $K^{\circ}$  is the polar of the convex hull of  $\pm \mathbf{S}$ . Denoting  $\mathbf{T} = [\mathbf{S}, -\mathbf{S}]$ , we can rewrite Problem (6) as  $d_{\lambda}^{*}(x) = \left(\arg \min_{d} \| \lambda x - d \|_{2} \text{ sub. to } \mathbf{T}^{\top} d \leq \mathbf{1}\right)$ . We now define the set of active constraints as
$$
A _ {\lambda} (x) = \left\{t _ {i}: \langle t _ {i}, d _ {\lambda} ^ {*} (x) \rangle = 1 \right\}. \tag {7}
$$
Geometry of the Dual (5). It is illustrated in Fig. 3, where  $s_1, s_2$  are two chosen data-points. The blue shaded polytope is  $K^{\circ}$ . At  $\lambda = \lambda_1$ , the point  $\lambda_1x$  lies in the interior of  $K^{\circ}$ . Hence,  $A_{\lambda}(x)$  is empty and  $\operatorname{supp}(c^{*}(x))$  is also empty. As  $\lambda$  increases, a non-empty support is obtained for the first time at  $\lambda = 1 / \gamma_{K^{\circ}}(x)$ . For all  $\lambda_2x$  in the red shaded polyhedron, the projection  $d_{\lambda_2}^*(x) = \operatorname{Proj}_{K^{\circ}}(\lambda_2x)$  lies on the face  $F$ . As  $\lambda$  increases further we reach the green polyhedron. Further increases in  $\lambda$  do not change the dual solution, which will always remain at the vertex  $d_{\lambda_3}^*(x)$ .
Geometrically,  $A_{\lambda}(x)$  identifies the face of  $K^{\circ}$  which contains the projection of  $\lambda x$  if  $A_{\lambda}(x)$  is non-empty (otherwise,  $\lambda x$  lies inside the polyhedron  $K^{\circ}$ ). The support of the primal solution,  $c^{*}(x)$ , is a subset of  $A_{\lambda}$ , i.e.  $\operatorname{supp}(c^{*}(x)) \subseteq A_{\lambda}(x)$ . Note that whenever two
points, say  $x$ ,  $x'$ , both lie in the same shaded polyhedron (red or green), their projections would lie on the same face of  $K^{\circ}$ . We now show this formally, in the main theorem of this section.
Figure 3: Geometry of the dual problem (5). See description on the left.
Theorem 4.1. The set of active constraints  $A_{\lambda}$  defined in (7) is robust, i.e.,  $A_{\lambda}(x') = A_{\lambda}(x)$  for all  $\lambda x' \in C(x)$ , where  $C(x)$  is the polyhedron defined as
$$
C (x) = F (x) + V (x), \tag {8}
$$
with  $F \subseteq K^{\circ}$  being a facet of the polyhedron  $K^{\circ}$  that  $x$  projects to, defined as
$$
F (x) = \left\{d \left| \begin{array}{l} t _ {i} ^ {\top} d = 1, \forall t _ {i} \in A _ {\lambda} (x) \\ t _ {i} ^ {\top} d <   1, \text {o t h e r w i s e} \end{array} \right. \right\}, \tag {9}
$$
and  $V$  being the cone generated by the constraints active at (i.e., normal to)  $F$ , defined as
$$
V (x) = \left\{\sum_ {t _ {i} \in A _ {\lambda} (x)} \alpha_ {i} t _ {i}: \alpha_ {i} \geq 0, \forall t _ {i} \in A _ {\lambda} (x) \right\}. \tag {10}
$$
The proof of Theorem 4.1 utilizes the geometry of the problem and properties of the projection operator, and is presented in Appendix E. We can now use this result to construct a robust classifier:
Lemma 4.2. Define the dual classifier as
$$
g _ {\lambda} (x) = \operatorname {A G G R E G A T E} \left(\left\{y _ {i}: t _ {i} \in A _ {\lambda} (x) \right\}\right), \tag {11}
$$
where AGGREGATE is any deterministic mapping from a set of labels to  $\mathcal{V}$ , e.g., MAJORITY. Then, for all  $x' \in C(x)$  as defined in Theorem 4.1,  $g_{\lambda}$  is certified to be robust, i.e.,  $g_{\lambda}(x') = g_{\lambda}(x)$ .
Implications. Having obtained a certifiably robust classifier  $g$ , we pause to understand some implications of the theory developed so far. We observe that the certified regions in Theorem 4.1 are not spherical, i.e., the attacker can make additive perturbations having large  $\ell_2$  norm but still be unable to change the label predicted by  $g$  (see Fig. 4). This is in contrast to the  $\ell_2$  bounded certified regions that can be obtained by most existing work on certification schemes, and is a result of modelling data structure while constructing robust classifiers. Importantly, however, note that we do not assume that the attack is restricted to the subspace.
Connections to Classical Results. For  $\epsilon = 0$ , Eq. (3) is known as the primal form of the Basis Pursuit problem, and has been studied under a variety of conditions on  $\mathbf{S}$  in the sparse representation and subspace clustering literature [20, 13, 50, 64, 30, 25]. Given an optimal solution  $c^*(x)$  of this basis pursuit problem, how can we accurately predict the label  $y$ ? One ideal situation could be that all columns in the support predict the same label, i.e.,  $y_i$  is identical for all  $i \in \operatorname{supp}(c^*(x))$ . Indeed, this ideal case is well studied, and is ensured by necessary [25] and sufficient [50, 64, 30] conditions on the geometry of the subspaces  $S_1, \ldots, S_K$ . Another situation could be that the majority of the columns in the support predict the correct label. In this case, we could predict Majority  $(\{y_i : i \in \operatorname{supp}(c^*(x))\})$  to ensure accurate prediction. Theorem 4.1 allows us to obtain robustness guarantees which work for any such aggregation function which can determine a single label from the support. Hence, our results can guarantee robust prediction even when classical conditions are not satisfied. Lastly, note that our Theorem 4.1 shows that the entire active set remains unperturbed – In light of the recent results in [54], this could be relaxed for specific choices of maps acting on the estimated support.
In this section, we will compare our certified defense derived in Section 4 to a popular defense technique called Randomized Smoothing (RS) [11], which can be used to obtain state-of-the-art certified robustness against  $\ell_2$  perturbations. RS transforms any given classifier  $f\colon \mathcal{X}\to \mathcal{Y}$  to a certifiably robust classifier  $g_{\sigma}^{\mathrm{RS}}\colon \mathcal{X}\to \mathcal{Y}$  by taking a majority vote over inputs perturbed by Gaussian noise  $\mathcal{N}(0,\sigma^2 I)$ , i.e.,
$$
g _ {\sigma} ^ {\mathrm {R S}} (x) = \operatorname {S m o o t h} _ {\sigma} (f) = \underset {k \in \mathcal {Y}} {\arg \max } \Pr_ {v \sim \mathcal {N} \left(0, \sigma^ {2} I\right)} (f (x + v) = k). \tag {12}
$$
Then, at any point  $x$ ,  $g_{\sigma}^{\mathrm{RS}}$  can be shown to be certifiably robust to  $\ell_2$  perturbations of size at least  $r^{\mathrm{RS}}(x) = \sigma \Phi^{-1}(p)$  where  $p = \max_{k \in \mathcal{Y}} \operatorname{Pr}_{v \sim \mathcal{N}(0, \sigma^2 I)}(f(x + v) = k)$  denotes the maximum probability of any class under Gaussian noise.
It is not immediately obvious how to compare the certificates provided by our method described above and that of RS, since the sets of the space they certify are different. The certified region
In reality, the choice of the noise distribution is central to determining the type of certificate one can obtain [41, 63], but Gaussian suffices for our purposes here.
Figure 4: Comparing polyhedral and spherical certificates. Details in text.
obtained by RS,  $C^{\mathrm{RS}}(x) = \{\bar{x} \colon \| x - \bar{x} \|_2 \leq r(x)\}$ , is a sphere (orange ball in Fig. 4). In contrast, our certificate  $C_{\lambda}(x)$  from Theorem 4.1 is a polyhedron (resp., blue trapezoid), which, in general, is neither contained in  $C^{\mathrm{RS}}(x)$ , nor a superset of  $C^{\mathrm{RS}}(x)$ . Additionally, our certificate has no standard notion of size, unlike other work on elliptical certificates [14], making a size comparison non-trivial. To overcome these difficulties, we will evaluate two notions of attack size: in the first, we will compare the  $\ell_2$  norms of successful attacks projected onto our polyhedron, and in the second, we will compare the minimum  $\ell_2$  norm required for a successful attack. We will then combine our method with RS to get the best of both worlds, i.e., the green shape in Fig. 4. In the following, we present both these evaluations on the MNIST [28] dataset, with each image normalized to unit  $\ell_2$  norm.
Comparison along Projection on  $C_{\lambda}(x)$ . For the first case, we investigate the question: Are there perturbations for which our method is certifiably correct, but Randomized Smoothing fails? For an input point  $x$ , we can answer this question in the affirmative by obtaining an adversarial example  $\bar{x}$  for  $g^{\mathrm{RS}}$  such that  $\bar{x}$  lies inside our certified set  $C_{\lambda}(x)$ . Then, this perturbation  $v = x - \bar{x}$  is certified by our method, but has  $\ell_2$  norm larger than  $r^{\mathrm{RS}}(x)$  (by definition of the RS certificate).
To obtain such adversarial examples, we first train a standard CNN classifier  $f$  for MNIST, and then use  $\mathbf{RS}^7$  to obtain the classifier  $g_{\sigma}^{\mathrm{RS}}$ . Then, for any  $(x,y)$ , we perform projected gradient descent to obtain  $\bar{x} = x^T$  by performing the following steps  $T$  times, starting with  $x^0 \gets x$ :
$$
I. x ^ {t} \leftarrow \operatorname {P r o j} _ {B _ {\ell_ {2}} (x, \epsilon)} \left(x ^ {t - 1} + \eta \nabla_ {x} \operatorname {L o s s} \left(g _ {\sigma} ^ {\mathrm {R S}} \left(x ^ {t}\right), y\right)\right) \quad I I. x ^ {t} \leftarrow \operatorname {P r o j} _ {C _ {\lambda} (x)} \left(x ^ {t}\right) \tag {13}
$$
Unlike the standard PGD attack (step I), the additional projection (step II) is not straightforward, and requires us to solve a quadratic optimization problem, which can be found in Appendix F. We can now evaluate  $g_{\sigma}^{\mathrm{RS}}$  on these perturbations to empirically estimate the robust accuracy over  $C_{\lambda}$ , i.e.,
$$
\operatorname {P r o j e c t i o n R o b u s t A c c} (\epsilon) = \Pr_ {x, y \sim p _ {\mathrm {M N I S T}}} \left(\exists \bar {x} \in B _ {\ell_ {2}} (x, \epsilon) \cap C _ {\lambda} (x) \text {s u c h t h a t} g ^ {\mathrm {R S}} (\bar {x}) \neq y\right).
$$
The results are plotted in Fig. 5, as the dotted curves. We also plot the certified accuracies for comparison, as the solid curves. We see that the accuracy certified by RS drops below random chance (0.1) around  $\epsilon = 0.06$  (solid red curve). Similar to other certified defenses, RS certifies only a subset of the true robust accuracy of a classifier in general. This true robust accuracy curve is pointwise upper-bounded by the empirical robust accuracy curve corresponding to any attack, obtained via the steps I, II described earlier (dotted red curve). We then see that even the upper-bound drops below random chance around  $\epsilon = 0.4$ , suggesting that this might be a large enough attack strength so that an adversary only constrained in  $\ell_2$  norm is able to fool a general classifier. However, we are evaluating attacks lying on our certified set and it is still possible to recover the true
Figure 5: Comparing RS with Our method for adversarial perturbations computed by repeating Steps I, II (13).
Further details (e.g., smoothing parameters, certified accuracy computation) are provided in Appendix F.
class (blue solid curve), albeit by our specialized classifier  $g_{\lambda}$  suited to the data structure. Additionally, this suggests that our certified set contains useful class-specific information – this is indeed true, and we present some qualitative examples of images in our certified set in Appendix F. To summarize, we have numerically demonstrated that exploiting data structure in classifier design leads to certified regions capturing class-relevant regions beyond  $\ell_p$ -balls.
Comparison along  $\ell_2$  balls. For the second case, we ask the question: Are there perturbations for which RS is certifiably correct, but our method is not? When an input point  $x$  has a large enough RS certificate  $r^{\mathrm{RS}}(x) \geq r_0$ , some part of the sphere  $B_{\ell_2}(x, r^{\mathrm{RS}}(x))$  might lie outside our polyhedral certificate  $C_\lambda(x)$  (blue region in Fig. 4). In theory, the minimum  $r_0$  required can be computed via an expensive optimization program that we specify in Appendix F. In practice, however, we use a black-box attack [9] to find such perturbations. We provide qualitative examples and additional experiments on CIFAR-10 in Appendix F.
Combining Our Method with RS. We now improve our certified regions using randomized smoothing. For this purpose, we treat our classifier  $g_{\lambda}$  as the base classifier  $f$  in (12), to obtain  $\mathrm{Smooth}_{\sigma}(\mathrm{g}_{\lambda})$  (abbreviated as  $\mathrm{Ours}_{\sigma}^{\mathrm{RS}}$ ). We then plot the  $\ell_2$  certified accuracy[7] [11, Sec 3.2.2] in
Figure 6: Comparing Certified Accuracy after combining our method with RS.
Fig. 6, where note that, as opposed to Fig. 5, the attacks are not constrained to lie on our certificate anymore.
As a final remark, we note that our objective in Fig. 6 was simply to explore RS as a method for obtaining an  $\ell_2$  certificate for our method, and we did not tune our method or RS for performance. In particular, we believe that a wide array of tricks developed in the literature for improving RS performance [41, 45, 63] could be employed to improve the curves in Fig. 6. We now discuss our work in the context of existing literature in adversarial robustness, and sparse representation learning.
6 Discussion and Related Work
Our proof techniques utilize tools from high-dimensional probability, and have the same flavor as recent impossibility results for adversarial robustness [12, 48, 47]. Our geometric treatment of the dual optimization problem is similar to the literature on sparse-representation [13, 20] and subspace clustering [50, 51, 64, 25], which is concerned with the question of representing a point  $x$  by the linear combination of columns of a dictionary  $\mathbf{S}$  using sparse coefficients  $c$ . As mentioned in Section 4, there exist geometric conditions on  $\mathbf{S}$  such that all such candidate vectors  $c$  are subspace-preserving, i.e., for all the indices  $i$  in the support of  $c$ , it can be guaranteed that  $s_i$  belongs to the correct subspace. On the other hand, the question of classification of a point  $x$  in a union of subspaces given by the columns of  $\mathbf{S}$ , or subspace classification, has also been studied extensively in classical sparse-representation literature [62, 61, 7, 27, 65, 65, 65, 39, 23, 32]. The predominant approach is to solve an  $\ell_1$  minimization problem to obtain coefficients  $c$  so that  $x = \mathbf{Sc} + e$ , and then predict the subspace that minimizes the representation error. Various global conditions can be imposed on  $\mathbf{S}$  to guarantee the success of such an approach [62], and its generalizations [15, 16]. Our work differs from these approaches in that we aim to obtain conditions on perturbations to  $x$  that ensure accurate classification, and as such we base our robust classification decision upon properties of solutions of the dual of the  $\ell_1$  problem.
Our results are complementary to [44, 5], who obtain a lower bound on the robust risk  $R(f,\epsilon)$ , in a binary classification setting, in terms of the Wasserstein distance  $D$  between the class conditionals  $q_{0}$  and  $q_{1}$ , i.e.,  $R(f,\epsilon)\geq 1 - D(q_0,q_1)$ . Using this result, [44, 5] roughly state that the robust risk increases as the class conditionals get closer, i.e., it becomes more likely to sample  $x\sim q_0,x'\sim q_1$  such that  $\| x - x^{\prime}\| \leq \epsilon$ . In comparison, for two classes, our Theorem 3.1 roughly states that for low robust risk,  $q_{0}$  and  $q_{1}$  should be concentrated on small-volume subsets separated from one another. Thus, the message is similar, while our Theorem 3.1 is derived in a more general multi-class setting. Note that [44, 5] do not explicitly require small-volume subsets, but this is implicit, as  $1 - D(q_0,q_1)$
is large due to concentration of measure when  $q_0, q_1$  do not concentrate on very small subsets. We provide an analogue of the empirical results of [44, 5] in Appendix F.6.
Our notion of concentration is also related to concentration of a measure  $\mu$ , as considered in [35, 36, 70], which denotes how much the  $\mu$ -measure of any set blows up after expansion by  $\epsilon$ . Under this definition, the uniform measure has a high degree of concentration in high dimensions, and this is called the concentration of measure phenomenon. In contrast, our Definition 2.2 of concentrated data distributions can be seen as a relative notion of concentration with respect to the uniform measure  $\mu$ , in that we call a class conditional  $q$  concentrated when it assigns a large  $q$ -measure to a set of very small volume, (i.e.,  $q(S)$  is high whereas  $\mu(S)$  is very low). In essence, Definition 2.2 defines concentration relative to the uniform distribution, whereas [35, 36, 70] define concentration independently. Definition 2.2 is useful in the context of adversarial robustness as it allows us to separate the concentration of data distributions (which is unknown) from the concentration of the uniform measure in high dimensions (for which there is a good understanding). This allows us to derive results in the non-realizable setting, where errors are measured against a probabilistic ground truth label  $Y$ , which is strictly more general than the realizable setting which requires a deterministic ground truth classifier  $f^{*}$ . As a result of this realizable setting, the analysis in [21, 35, 36, 70] needs to assume a non-empty error region  $A$  for the learnt classifier  $g$  with respect to  $f^{*}$ , in order to reason about  $\epsilon$ -expansions  $A^{+\epsilon}$ . The results in [35, 36] indicate that the robust risk grows quickly as the volume of  $A$  increases. However, humans seem to be a case where the natural accuracy is not perfect (e.g., we might be confused between a 6 and a poorly written 5 in MNIST), yet we seem to be very robust against small  $\ell_{2}$  perturbations. This points to a slack in the analysis in [35, 36], and our work fills this gap by considering  $\epsilon$  expansions of a different family of sets.
Finally, our work is also related to recent empirical work obtaining robust classifiers by denoising a given input  $x$  of any adversarial corruptions, before passing it to a classifier [46, 38]. However, such approaches lack theoretical guarantees, and might be broken using specialized attacks [40]. Similarly, work on improving the robustness of deep network-based classifiers by adversarial training off the data-manifold can be seen as an empirical generalization of our attack model [24, 37, 69, 33]. More generally, it has been studied how adversarial examples relate to the underlying data-manifold [53, 26, 34]. Recent work also studies the robustness of classification using projections onto a single low-dimensional linear subspace [3, 2, 43]. The work in [3] studies an attack model of bounded  $\ell_2$ ,  $\ell_{\infty}$  attacks, and they provide robustness certificates by obtaining guarantees on the distortion of a data-point  $x$  as it is projected onto a single linear subspace using a projection matrix  $\Pi$ . In contrast, our work can be seen as projecting a perturbed point onto a union of multiple low-dimensional subspaces. The resultant richer geometry allows us to obtain more general certificates.
7 Conclusion and Future Work
To conclude, we studied conditions under which a robust classifier exists for a given classification task. We showed that concentration of the data distribution on small subsets of the input space is necessary for any classifier to be robust to small adversarial perturbations, and that a stronger notion of concentration is sufficient. We then studied a special concentrated data distribution, that of data distributed near low-dimensional linear subspaces. For this special case of our results, we constructed a provably robust classifier, and then experimentally evaluated its benefits w.r.t. known techniques.
For the above special case, we assume access to a clean dataset  $\mathbf{S}$  lying perfectly on the union of low-dimensional linear subspaces, while in reality one might only have access to noisy samples. In light of existing results on noisy subspace clustering [59], an immediate future direction is to adapt our guarantees to support noise in the training data. Similarly, while the assumption of low-dimensional subspace structure in  $\mathbf{S}$  enables us to obtain novel unbounded robustness certificates, real world datasets might not satisfy this structure. We hope to mitigate this limitation by extending our formulation to handle data lying on a general image manifold in the future.
We would like to thank Amitabh Basu for helpful insights into the optimization formulation for the largest  $\ell_2$  ball contained in a polyhedron. This work was supported by DARPA grant HR00112020010,
NSF grants 1934979 and 2212457, and the NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning (NSF grant 2031985, Simons grant 814201).