The Geometry of Adversarial Subspaces
Authors: Dylan M. Paiton
David Schultheiss
Matthias Kuemmerer
Zac Cranko
Matthias Bethge
Venue: ICLR 2022 Submitted (2022)
Keywords: adversarial attack
decision boundary
riemannian geometry
differential geometry
interpretability
adversarial training

Abstract: Artificial neural networks (ANNs) are constructed using well-understood mathematical operations, and yet their high-dimensional, non-linear, and compositional nature has hindered our ability to provide an intuitive description of how and why they produce any particular output. A striking example of this lack of understanding is our inability to design networks that are robust to adversarial input perturbations, which are often imperceptible to a human observer but cause significant undesirable changes in the network’s response. The primary contribution of this work is to further our understanding of the decision boundary geometry of ANN classifiers by utilizing such adversarial perturbations. For this purpose, we define adversarial subspaces, which are spanned by orthogonal directions of minimal perturbation to the decision boundary from any given input sample. We find that the decision boundary lies close to input samples in a large subspace, where the distance to the boundary grows smoothly and sub-linearly as one increases the dimensionality of the subspace. We undertake analysis to characterize the geometry of the boundary, which is more curved within the adversarial subspace than within a random subspace of equal dimensionality. To date, the most widely used defense against test-time adversarial attacks is adversarial training, where one incorporates adversarial attacks into the training procedure. Using our analysis, we provide new insight into the consequences of adversarial training by quantifying the increase in boundary distance within adversarial subspaces, the redistribution of proximal class labels, and the decrease in boundary curvature.

Under review as a conference paper at ICLR 2022
THE GEOMETRY OF ADVERSARIAL SUBSPACES
A Anonymous authors
Paper under double-blind review
ABSTRACT
Artificial neural networks (ANNs) are constructed using well-understood mathematical operations, and yet their high-dimensional, non-linear, and compositional nature has hindered our ability to provide an intuitive de- scription of how and why they produce any particular output. A striking example of this lack of understanding is our inability to design networks that are robust to adversarial input perturbations, which are often imper- ceptible to a human observer but cause significant undesirable changes in the network's 's response. The primary contribution of this work is to further our understanding of the decision boundary geometry of ANN classifiers by utilizing such adversarial perturbations. For this purpose, we define adver- sarial subspaccs, which are spanned by orthogonal directions of minimal perturbation to the decision boundary from any given input sample. We find that the decision boundary lies close to input samples in a large subspace, where the distance to the boundary grows smoothly and sub-linearly as increases the dimensionality of the subspace. We undertake analysis characterize the geometry of the boundary, which is more curved within the adversarial subspace than within a random subspace of equal dimensionality. To date, the most widely used defense against test-time adversarial attacks is adversarial training, where one incorporates adversarial attacks into the training procedure. Using our analysis, we provide new insight into the consequences of adversarial training by quantilying the increase boundary distance within adversarial subspaces, the redistribution of proximal class labels, and the decrease in boundary curvature.
INTRODUCTION
Artificial neural networks (ANNs) have been highly performant on common machine learning tasks, but their application space is limited by their suscoptibility to adversarial attacks (Szegedy et al. 2014), which underscores general lack of understanding for how and why they make their decisions. Examples produced by adversarial attacks are a worst-case demonstration of an ANN's inability to gracefully cope with ddentity-preservee distortions of its inputs. To construct one, an adversary must perturb the input in a small but specific way such that the output of the network changes significantly. Adversarial perturbations are defined to have minimal length, resulting in inputs that are as close to the network's 's decision boundary as possible. Here, we present method that uses an untargeted adversarial attack objective to find the multi-dimensional subspace where an ANN's decision boundary is closest to any given input sample. The resulting adversarial subspace allows us to visualize and understand ANN decision boundaries in terms of their curvature, their shape, and their proximity the input.
shifts or
One existing technique to explain ANN classifications has relied on two- wwo-dimensional visualiza- tions of the decision surface, although it suffers from a number of shortcomings, most notably high-dimensional points resolving to the same projected location, distortions of distances, and a lack of invertibility from the two-dinensional space back to the high-dimensional decision space (Féraud & Clérot, 2002: Rauber et al. 2016 Rodrigues et al. 2019). A popular alternative strategy is to investigate the docision boundary in the high-dieensional
input space itself, early attempts of which (Golland] 2001 Bachrens et al. 2010) lead the way to the modern interest in adversarial attacks (Bigiio et al.) 2013) SZegedy et al.) 2014). For example, one measure the distance to the decision boundary for random perturbation

---

Under review as a conference paper at ICLR 2022
predicted class
exemplar from class
1.0
first principal curvature direction
boundary image
tangoni manifold
0.75
exemplar from class
graiF
docision boundary
0.5
ad Averaaiin direction 1
ad adversaria! direction? 2
0.25
exemplar
0.0
from class 5
0.0
0.25
0
0.75
1.0
An adversarial subspace. (Left) For any given input example, we define subspace where the boundary is closest to the input. The white arrows depict our sampling scheme, which uniformly samples directions within the adversarial subspace where measure the distance to the decision boundary. The perturbation images are rescaled for visualization, but their f2 length is indicated on the axes. Fach color corresponds to the class categories for deer (class 4), dogs (class 5), and horses (class 7). This visualization is for an adversarially trained ResNet-50. (Right) We then measure the curvature of the decision boundary, both in the high-dimensional input space and in the adversarial subspace.
Figure 1:
directions away from adversarial examples (Tabacof & Valle 2016 He et al., 2018 or for orthogonal directions which are proximal to an initial adversarial example Tramer et al. 2017 Alternatively, much work has demonstrated the utility of visualizing two-dimensional cross-sections of the decision space between adversarial examples and random orthogonal directions WWarde-Farlee & Goodfellow 2016 Yu et al., 2019a) or orthogonal test set exam- ples (Swirszez et al. 2019). Unlike all of the existing approaches to characterize decision boundarics in the input space, which rely on at most one adversarial dircction per input, we propose jointly defining subspace of adversarial directions. The resulting analysis leads to a general explanation ion of classifier decision boundaries, as well as adversarial susceptibility, in the subspace where the boundary is closest. the input example.
As a further demonstration of the utility of our approach, we investigate the most widely used technique for defending against adversarial attacks, so-called adversarial training. The defense method requires including adversarial perturbations in the training process by either augmenting the dataset with pre-computed examples (Ilyas et al. 2019) or by incorporating an attack model into the training procedure [Goodfellow et al. 2014 Madry et al. 2018). It has been recently demonstrated that adversarial training performs well on the CIFAR-10 machine learning dataset when enough resources are dedicated to hyprparraeeter luning (Gowal et al.] 2020), although the performance is still far from that on unperturber images. We provide a perspective this defense by observing that it increases the distance to the decision boundary in the entire adversarial subspace, even though the method utilizes only a single adversarial perturbation per input sample. We additionally find that adversarial training increases the number of alternative label regions in adversarial subspaces, and changes the relative distribution of nearby alternate classes. Hercin, will focus defining the method and summarizing the subspace with a continued application of understanding adversarial training.
METHODS
In the following we will outline the novel mothods employed in this study. Namely, the method for finding adversarial subspaces is first described, and then we provide overview of how we measure decision boundary curvature (with a rigorous discussion given in Appendix Al Our primary investigation was conducted using the CIFAR-10 dataset (Krizhevsky &
2

---

Under review as a conference paper at ICLR 2022
distance coefficient dista distance to decision bnd.
3.0
natural
robust
2.0
1.0
0.2
1001
100
1977
0.0
1.5
9
17 25
33
41
49
255
dimension of adversarial space
Figure 2: Distances to the decision boundary. The distance to the decision boundary for random perturbations within an adversarial subspace grows sub-linearly with increased dimensionality. (Upper left) The dots indicate the adversarial perturbation length for that dimension (i.e. along the perturbation axis), averaged across images. The box plots show the distance to the decision boundary for unifrrmly-sampled directions within the n-dimensional ad adversarial subspace, where is increasing along the horizontal axis. The box indicates the interquartile range, the whiskers indicate the 10% and 90% data percentiles, and the solid horizontal lines indicate the mean. (Lower left) Same as the above, except now the vertical axis indicates the ratio of the largest distance to the decision boundary divided by the length of the given subspace dimension,. That this is small indicates that we rarely sampled boundary point more than 50% farther away than the largest adversarial dimension. (Right) Distribution of boundary distances as one increases subspace dimensionality, ie the probability (vertical axis) of sampled direction having a specific distance (horizontal axis). Observe that the mean shifts to the right, and the variance increases proportionately, which suggests that the distance smoothly increases from dimension to the next.
Hinton 2009) and compared two identical ResNet-50 ANNs (He et al. 2016) trained with and without adversarial examples. The networks were modified and retrained using C activation functions for the curvature analysis (ELU with a = 1). Further dataset and model details can be found in Appendix B In Appendix C we provide additional results on the CIFAR-10 datasct, including a comparison with the WideResNet-700 architecture with Swish activations from Gowal al. (2020). In Appendix D we provide results on the MNIST dataset, which demonstrates largely consistent trends across datasets. In Appendix D we include control lests to establish the consistency our result. for different random seeds. All measured distances are with respect to the f--nrr..
2.1 FINDING ADVERSARIAL SUBSPACES
We aim to find the set of orthonormal vectors that span the subspace where input sample has minimal distance to the decision boundary. In order to achieve this, we follow greedy optimization approach to span an adversarial subspace up to desired number of dimensions. More precisely, consider some neural network, f. that receives an n-pixel imput image,
R" and produces output vector of probabilities associated with I label categories, i E R!. For a given input, we wish to find an ordered set of orthogonal vectors, iee. Thus for each vector, we want
A = (0,, 62,-.., .., 5m]] m & n, such that (ōk, 5m) 0, Vk to solve an optimization objective that minimizes 1|6, llp- where [|,],, = (E, (6P)) is the 4p norm. We expect the omiizizer explicitly and jointly consider the boundedness of the input
3

---

Under review as a conference paper at ICLR 2022
domain, force the perturbations to be orthogonal, and produce perturbations that result in new (incorrect) classifier outputs. Such an objective intractable to solve for deep neural networks, but is commonly approximated by the untargeted adversarial attack objective which modifies the conditional to be subject to max fz(æ + 6) fi(r + 6) > O, where is j#i
the correct class index. The length of these vectors is nearly synonymous with distances to the decision boundary, where the actual decision boundary would be an infinitcsimal stop towards from 8. We must further relax the objective to differentiable approximation for finding a gradient based solution. While there are many such approximations, we opted for a modification of that which was used by (Segedy al. (2014) to find singular adversarial directions, which is to minimize the inverse of the cross entropy loss (the training objective of the model), weighed with distance penalization term. Specifically, l def Toz(2+6) K. || p.
with the cross entropy loss defined as LCE(z) (z) def log expfe(#) Using this loss term, we exp E, f)(ر)*
can now define our objective function as implemented:
minimize
LCE (x 8m) + 110m11p
subject to
max f (x (x++ (mm) - fe( + 5m) >
(1)
+ 8m E [0. 11ad
(6x. 6m) 0, Ak < m.
Intuitively, this maximizes the model loss with respect to the true class label while keeping the perturbation small as possible. The first constraint in Equation D is to ensure that the image is adversaria the second keeps the perturbed images within the allowable pixel bounds, and the last one enforces orthogonality of all found adversarial subspace dimensions. We found the optimal value of constant k in equation (1] with a binary search performed during run time. The optimization problem was solved using the greedy interior-point optimizer implemented by Ιpopt (Wächter & Biegler 2006).
Once the adversarial vectors are found, we conducted experiments measure the distance to the decision boundary in the space spanned by the vectors (i.e. within the adversarial subspace). The experiment was repeated on 100 test. images that were chosen to have equal label representation (10 images per label) and to be correctly classified by both networks. For each test image we found 50 orthogonal adversarial vectors, which allowed us to define a sequentially growing list of adversarial subspaces with dimensionality increasing from to 50. For a given image and subspace we sampled 100 directions via a lincar combination of normally distributed variables, which has been previously proven to provide even sampling Muller 1959). Finally, for each random direction we performed a binary search to find the decision boundary.
2.2 MEASURING DECISION BOUNDARY CURVATURE
As before, let fú, f3 R + R be the functions corresponding to i and j-th logits of a neural network, f. Let F.j det fa Jz- Then the (ddeissooo boundary is the set
By def (IE RRRR I 0} = fx RD [f((z) [z()))
(2)
The set Bij is defined independently of the sign of Fiy (that is, Bij = By)), although doing so will produce change of sign in caleulations. Therefore will continue to follow the above convention by identifying, at a point x, with the correct label associated to x, and
'Many works use "untargeted" adversarial attacks to refer to attacks that are actually targeted to the mestnd-most-likely output class (for example, Carlini & Wagner 2017 Kurakin et al. 2017 He al. 2018 Chen et all 2018). While these may be practically equivalent for single perturbation, they are not when linding multiple orthogonal directions. Therefooe, we will use more general definition of untargeted attack, following (Tabacof & Valle 2016 Finlay et al. 2019)..
"Since not targeting any specific class, the first constraint avoids completely uniform output distribution (i.e. class wins)
4

---

Under review as a conference paper at ICLR 2022
0.22
0.28
predicsed
0.56
6.11
0.13
0.35
clans
atrplane
cer
0.0
0.0
5.0
6.3
0.9
adv
Jvf di-
dir
ady
0.0
6:06
0.15
6/3
0.58
2.45
0.0
5.12
0.25
6.0
0.12
0.23
0.0
2.85
0.35
0.64
1.62
1.76
1.00
frog
0.22
6.21
046
2.51
07
ship
Figure 3: Boundary visualizations. Each subplot visualizes a two dimensional cross- section of the decision space for naturally trained network. Each cross-section is spanned by the vectors corresponding to the largest distance cocfficient found for a given input image (visualized as black arrows) and the according adversarial perturbation of the largest dimension of the subspace from which the distance coefficient was retrieved. The red dots indicate the position of the described adversarial perturbation. The number in the top right corner depicts the distance coefficient of the cross-section (as reported in Figure bottom left panel). Areas of the cross-section that lie outside the image bounds are shaded.
with the incorrect label. This convention (which agrees with Moosavi-Dezzooli et al. 2018 ensures that positive (resp. negative) curvature of Bi, near x corresponds to Bij bowing inward towards (resp. away from) æ (see Figure for an example of positive curvature). If the curvature along Bi, is zero near x then Bıj is affine. The complete details how our calculations were derived, and numerical considerations therein, are relegated to Appendix A however we give a brief intuitive account below.
In our curvature analysis use method known principal curvature decomposition to understand the degree of lincarity of the decision boundary. Since the decomposition is performed level-sct of the decision space, which in general is of codimension 1, it yiclds n - scalars and vectors, called the principal curvatures and principle curvature directions, respectively. The curvature directions point in the directions of maximal and minimal surface curvature, while the principal curvatures themselves indicate the signed magnitude of the curvature. Similar to principal component analysis, which organizes the space as an hoghogl basis of directions of maximal variance, principal curvature analysis organizes the (tangential) space of the decision boundary into orthogonal directions of maximum curvature. The principle curvatures and associated directions are the eigenvectors and eigenvalues of linear operator s R + R called the shape operator.
In order to study the curvature an adversarial (or random) subspace we introduce the subspace pullback of s
(PT)'s s def PsPT,
(3)
where P E mpmx an orthogonal projection, so that the image of PT spans the adversarial subspace of R in question. This definition (PT)*s essentially restricts the shape operator to a linear subspace of the langential space and is motivated by the principles of differential geometry, details of which can found Remark [ and the surrounding material in Appendix A] The pullback shape operator allows to compare the amount of curvature in the adversarial subspace to that of the entire response space.
RESULTS
By definition, the distance to the decision boundary increases for each additional dimension of an adversarial subspace. The circles in the top left panel of Figure 2 indicate these distances, averaged across the test images, which are found by greedily minimizing Equation [ The distance to the decision boundary grows sub-linearly of the subspace, with the fastest growth occurring in the first couple of dimensions. The leftmost box plots of the top left panel in figure 2 as well the Appendix Table confirms
increases the dimensionality
5

---

Under review as a conference paper at ICLR 2022
Test boundary
Adversarial boundary
A.
B.
0.5
0.5
Mean curvature
Training type
natural robust
0.0
0.0
40.5
40.5
Adversarial boundary
Random subspaces
Adversarial subspaces
10
10
C.
D.
E.
5-
5
Curvature
-50
-100
-5
-5
-10
-10
1500
3000
2
6
10
2
10
Principal curvature directions
Figure 4: Decision boundary curvature. All plots include data for 50 correctly classified test images with evenly distributed labels and the first 10 adversarial subspace dimensions. (A,B) The mean curvatures per boundary point are smaller when the boundary is separating two test images ("test boundary") than when it is separating a test image and an adversarial image ("adversarial boundary"). (C) Scatter plot of the curvature profiles for all images and directions with the adversarial boundary condition. The red inlay shows the full dimensionalityy (i.e. same horizontal axis as the parent, plot), but zoomed in (i.e. different vertical axis limits) to emphasize that while both models have positive and negative curvature, the adversarially trained boundary is more flat (i.e. linear). The data includes all adversarial subspace boundary points per test image. (D,E) Curvature of adversarial point boundaries when restricted to random adversarial subspaces. Curvature in the adversarial subspace is more extreme than when in an equal sized random subspace. The points indicate the mean across test images and the error lines indicate the standard deviation.
that the mean initial perturbation distances are comparable to more traditional adversarial attacks (Madry et al. 2018) and decision boundary distance estimates (Ding al.) 2020). While this gives an idea of the bouud of distances to the decision boundary, it does not provide information about the behavior of the boundary elsowhere in the subspace. To better understand this, we uniformly sampled 100 random directions within a given subspace and performed binary search find the decision boundary along those directions, We only report distances for decision boundary points that are within the lint allowable pixel box, which was never fewer than 40% of the subspace samples (see Appendix C Figure 7 The box plots in the top left panel of Figure 2 indicate the distribution of decision boundary distances across the sampled directions and test images. There is no consistent lz-length of adversarial attack that makes perturbation visible to a human observer, although in general the perturbations were visible after around dimension 20 for the robust network and beyond dimension 50 for the natural network (see Appendix C Figure 2 for examples). The bottom left panel of Figure plots a distance coefficient that is defined as the maximum distance to the decision boundary within an adversarial subspace divided by the longest vector defining the subspace (which is always associated with the highest dimension number). For example, a value of 2 would indicate that the largest sampled distance twice as far away as the largest found adversarial perturbation. The ad adversarial trained network consistently produces smaller coefficients, indicating that the boundary size less variable between the subspace axes. Finally, complete description of the boundary distances is conveyed as histograms in the rightmost panels, with dimensionality increasing from left to right, then top to bottom. Together, this provides further evidence that the decision boundary grows

---

Under review as a conference paper at ICLR 2022
consistently, without large variations in distance between the found adversarial subspace axes.
In Figure 3 we visualize two-dimensional cross-sections of the decision surface. The horizontal axis of each cross-section corresponds to an adversarial subspace perturbation, the black arrow indicates the farthest decision boundary point, and the vertical axis is found by orthogonalizing the direction of the farthest decision boundary point with respect to the adversarial perturbation using one step of the Gram-Schmidt process. The cross-sections are ordered according to the distance coefficient, which is displayed in the upper-right corner. We emphasize that because these are cross-sections, they avoid the shortcomings of typical low- dinenssiiin decision space projections noted in the introduction. Among the visualizations one can see examples of when the boundary increases smoothly from one axis to the next, and also the more rare (as indicated by Figure 2 cases when the boundary protrudes quite far from the origin when between the axes. From this analysis we conclude that adversarial examples are not isolated or due to sharp protrusions into otherwise homogeneously labeled region. Rather, there exists a smooth increase in distance to the dccision boundary from the extreme case of initially found adversarial example to semantically differentiated images.
In Figure we measure boundary curvature for several experimental conditions that vary the type of train- ing, the location of the boundary, and the subspace defining vectors. Panels A and B report the mean curvature (i.e. the mean of the eigenvalues of the shape operator) per point on the decision boundary. Both model types have larger boundary curvature for boundary points separating clean images from adversarial images (adversarial boundary; Panel B) when compared to boundary points separating un- perturbed test set examples (test boundary; Panel A). We additionally found that adversarial training tends to lincarize the decision boundary, both between correctly labeled test images and near adversarial ex- amples. Panel C is a scatter plot of the principal curvatures (i.e. eigenvalues of the shape operator) in the high-dimensional input space for all of the adversarial boundary points. We consistently find both positive and negative curvature at each bound-
Table 1: Entropy of unique proximal adversarial labels per class, averaged across a balanced sampling of cor- rectly labeled test images.
Label Natural
Robust
airplane
1.70
1.81
car
1.01
1.26
bird
1.72
1.59
cat
1.56
1.97
decr
1.41
1.98
dog
1.51
1.82
frog
1.12
1.51
horse
1.63
1.78
ship
1.74
1.39
truck 1.79
1.88
ary point and for both training types. Using the pullback defined in Equation B we can mcasure the subspace principal curvatures for random (Panel D) and adversarial (Pancl E) subspaces. The curvature within adversarial subspaces is notably larger than in random subspaces. The difference in curvature values between naturally and adversarially trained networks is less pronounced when limited to random subspaces suggesting that much of the linearization occurring during adversarial training is along directions aligned with adversarial examples. The found boundary points can be the closest to the clean input even when there is positive curvature, long the radius of curvature is greater than the distance to the input. However, for naturally trained networks, adversarial boundary points tend to have more negative mean curvature, both in the full input space (Panel B) and in the adversarial subspace (Panel E).
In addition to the earlier comparisons between natural and adversarial network training,
we measured the diversity of adversarial classes near test samples. The leftmost subplot of Figure gives the number of unique adversarial classes found in the first 50 dimensions of the adversarial subspaces, averaged across correctly labeled test images. The other plots show the distribution of adversarial classes present in the subspaces with respect to the origin label. These plots suggest that adversarial training results in increased diversity of alternate classes near any given test sample, which we further quantify by measuring the discrete entropy of each distribution in Table Thus, in addition to (or more probably as consequence of) increasing the distance to the boundary, adversarial training further modifies the density of alternative class labels near test images. While this could be considered
7

---

Under review as a conference paper at ICLR 2022
airplane
bird
cat
deer
Unique ad sdversarial classes
natural robust
4
Class density
0.7
dog
frog
horse
ship
truck
2
0.0
0
8
9
Original class
Adversarial class
Figure 5: Adversarial Class Composition. (Left) Within the adversarial subspaces of randomly chosen lest, images we quantified how many unique adversarial classes exist. In most cases adversarial training increases the diversity of alternate classes close to any test image. (Right) Each plot shows the distribution of nearby alternate classes for each origin class with natural and adversarial training. Adversarial training results in more even distribution of nearby classes for all categories except bird and ship.
desired effect, as it indicates that the robust network utilizes more of the high-dimensional space, it also leads to behavior that is less matched to human behavior (something that is often sought along with adversarial robustness). Indeed, while some categories, like "car" and "truck" are intuitively adjacent, the distribution of classes generally failed to match any clear intuitions. For example, observe the lack of symmetry between cars and trucks, or that adversarial training switches the "frog" category's most frequent alternate class from "bird" to "cat".
RELATED WORK
Related adversarial subspace studies: The work of Tramer et al. (017 is most similar to adversarial docomposition, in that they reported the dimensionality of subspace in the proximity of a given adversarial perturbation, although they do not comment on the class, number, proximity of nearby orthogonal adversarial examples. Work from Ma et al.) (2018) measured the local intrinsic dimensionality (LID) of adversarial subspaces, which indirectly estimates adversarial subspace dimensionality by measuring the distance from an adversarial example to batch of nearest umperturbed dataset examples. This idea has been extended to algorithms for detecting adversarial examples (Ma et al. (2088) Mao et al.) 2020)) although /Athalye et al. (2018) generated adversarial examples with indistinguishabi LID scores from natural examples. In terms of understanding ANN classifiers, such an approach leaves open the questions of how proximal the decision boundary is from input samples in greater than one dimension or the actual dimensionality adversarial subspaces. Our approach is the first to examine the decision boundary where it is nearest to test samples by defining adversarial subspaces using multiple orthogonal adversarial directions. Our emphasis the subspace where the decision boundary is closest to test images is also unique when compared to work that characterizes the decision boundary in random directions, which will be considerably farther away from the origin (e.g. Fawzi al. 2016 Tabacof & Valle 2016 Liu et al. 2017 He et al.) (2018). Finally, most of these studies perform a clipping operation after defining or finding the desired directions, which results in a non-orthogonal set (see Appendix
F for more detailed discussion). We overcome this problem by including the valid pixel range as inequality constraints in the optimization process. Our work complementary to previous studies that investigate the relationship between adversarial robust and inpul dimensionality (Amsaleg al., 2017 Slmon-Gabrit et al., 2019), since the comparative trends observed between robust and natural networks are relatively unchanged with small (MNIST) or large (CIFAR) dimensional inputs.
Neuron response curvature: The use of response curvature to understand nonlinear functions has a long history in neuroscience (for a review, see Gollisch & Herz 2012 as well
models of neural computation (e.g. Zetzsche & Krieger 2001 Rust et al.] 2005 Golden
(et al.) 2016 Cohen et al.) 2020). The principal curvature decomposition was previously employed in this context by Golden et. al. (2019), who used it. to understand invariance
8

---

Under review as a conference paper at ICLR 2022
and selectivity of individual ANN neurons, and by Paiton et al. (020)) who first found relationship between individual neuron response curvature and adversarial robustness. Our work is distinct from the aforementioned studies in that investigate the curvature of the decision boundary, as opposed to that of individual neurons. However, the relationship between decision boundary curvature and adversarial robustness has been previously studied by Moosavi-Dezfool and colleagues, who provided a number of compelling results identifying, "an increasing [adversarial] vulnerability with respect to the curvature of the decision boundary" MMosaviiDezfooli et al., 2018). As we discussed in the previous section, our analysis provides support for hypotheses that go beyond the corrclation between decision boundary curvature and network robustness, in that we observe different boundary curvature in adversarial subspaces than random subspaces or the full input space. We addit ionally note that while previous work has utilized Riemannian curvalure analysis to understand neural networks (viz. Fawzi et al. 2016 Poole et al. 2016 Moosavi- Dezfooli et al. 2018 2019 Golden et al., 2019), each study measures different quantities without complete details of the derived formulas. We demonstrate in Appendix G that these formulas do not find the correct level-sct curvature of simple manifolds. This together with our rigorous derivation and discussion in Appendix A constitutes, to the best of our knowledge, the first principled and analytically supported measurement of decision boundary curvature of trained nonlinear neural networks.
CONCLUSION
There is an ongoing debate on the cause, prevalence, and uniformity of adversarial examples.
Several studies have supported a hypothesis that most adversarial exampler inhabit, relatively
dense regions of the input space (Goodfellow et al. 2014 Tabacof & Valle 2016 Moosavi-
Dezfooli et al. 2017a He et al. 2018). However, other works have suggested that adversarial examples live in isolated pockets or thin "spikes" of incorrect classification regions (Szegedy et al. 2014 Yu et al.) 2019b). Here we provide quantitative evaluation supporting the hypothesis that decision boundarics are close to input samples in large subspace. Additional work has suggested that it is the lincarity of deep neural networks that leads to adversarial examples (Goodfellow al. 2014), and that adversarial training leads to less linear decision boundary (Madry et al.,
2018). However, more recent. work has proposed the alternatio live:
that increasing decision boundary linearity causes increased robustness (Li et al. 2019b
Moosavi-Dezfooo et al. 2019 Terjék 2020 Sarkar & Iyengar (020). Our curvature analysis
confirms that adversarial training linearizes the boundary, which also congruent with hypotheses about the robustness benefits of smoothing the decision surface (Zhang et al. 2019 Wu et al. 2020). We further compare the boundary curvature in adversarial subspaces and random subspaces to show increase in curvature in adversarial subspaces for both training methods. Since we do not report the causal relationship between adversarial directions and boundary curvature, our results have no impact on hypotheses for adversarial susceptibility due to overfitting (Tanay & Griffin 2016) or the perceptual quality of adversarial features [yas et al. 2019 although we see this as an interesting future direction of study.
Adversarial subspaces allow for a quantitative description of the nonlincar geometry and class composition of the network dccision boundary in the subspace spanned by its most susceptible perturbation directions. They also assist in explaining the prevalence of adversarial examples as well as the effects of adversarial training. Understanding this space better can improve neural network performance with semi-supervised relabeling (Benalo et al. 2018) 2017
architecture design (Rauber al., [2018), and adversarial robustness (Tramer Moosavi-Dezfooll et al.] 2019). Importantly, we also provide quantitative and falsinable interpretability approach for understanding ANN decisions (Ribeiro et al., 2016: Vlassopoulos ct al. 2020). which is a necessary prerequisite for the technology to be applied to many industries (Doshi-Vele & Kim 2017 Leavitt & Morcos (020)). We have identified an
interesting avenue for future work in performing large-scale analysis of the adversarial subspaces of various neural network architectures, as well as estimating decision boundary geometry of non-smooth (e.g. ReLU or stochastic, ei al. 2019a Dapello et 2020) neural networks (which requires adding a costly Hessian approximation step (Moosavi Dezfooli
et. al. 2019
9

---

Under review as a conference paper at ICLR 2022
AUTHOR CONTRIBUTIONS
ACKNOWLEDGMENTS
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Laurent Amsaleg, James Bailey Dominique Barbe, Sarah Erfani, Michael E Houle, Vinh Nguyen, and Miloš Radovanović. The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality. In 2017 TEEE Workshop on Information Forensics and Security (WIPS), pp. 6. IEEE, 2017.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In Jemnifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 274 283. PMLR, 10 15 Jul mlr wttpss///80//taaeeeee31 html
2018. URL
David Bachrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research, JMLR, 11:1803-1831, 2010.
Barbara Caroline Benato, Alexandru Cristian Telea, and Alexandre Xavier Falcão. Semi- supervised learning with interactive label propagation guided by feature space projections. In 2018 3ist Conference on Graphics, Patterns and Images, SIBGRAPI, pp. 392-399. IEEE, 2018.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE symposium on securily and privacy, pp. 39 57. IEEE, 2017.
Pin-Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsich. Ead: clastic- net attacks to deep neural networks via adversarial examples. In Thirty-second AAAI conference on artificial intelligence, 2018.
Djork-Aré Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXio:1511.0790 2015.
Uri Cohen, SueYeon Chung, Daniel D Lee, and Haim Sompolinsky, Separability and geometry of object manifolds in deep neural networks. Nature
Watureeco communications, 11(1):1-13, 2020.
Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David D Cox, and James DiCarlo. Simulating a primary visual cortex at the front of cnns improves robustness to image perturbations. BioRzie, 2020.
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct input space margin maximization through adversarial training. In International Conference on Learning Represenlations, ICLR, 2020.
Finale Roshi-Velor and Been Kim. Towards a rigorous science of interpretable machine learning. Xiviv preprint ar Xrv 700.00000, 2017.
Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Robustness of classifiers: from adversarial to random noise. Tn D. Lee, M. Sugiyama, U. Tuxbuur, 1. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, NearlPS, volume 29. Curran Associates, Inc., 2016. URL https: pegrroceedings.neurip. Pwper.papprrrrr
Raphael Féraud and Fabrice Clérot. A methodology to explain neural network classification. Neural networks, 15(2):237-22) 2002.
Chris Finlay, Aram- Alexandee Pooladian, and Adam Oberman. The logbarrier adversarial attack: making effective use of decision boundary information. In Proceedings of the TEFF/CV International Conference on Computer Vision, pp. 4862-48 -4674870, 2019.
11

---

Under review as a conference paper at ICLR 2022
J R. Golden, K P Vilankar, M C K Wu, and D J Field. Conjectures regarding the nonlinear geometry of visual neurons. Vision research, 120:74-92 92, 2016.
James R Golden, Kedarnath P Vilankar, and David J Field. Selective and invariant features of neural response surfaces measured with principal curvature. bioRriv, 2019. URL https://doi http:::://oo..r///...oo 1 12. 22.26........... 888933
Polina Golland. Discriminative direction for kernel classifiers. In Advances in neural information processing systems, NewrIPS, volume 14, pp. 745-752, 2001.
Tim Gollisch and Andreas VM Herz. The iso-respouse method: measuring neuronal stimulus
integration with closed-loop experiments. Fronfiers in neural circuits, 6:104, 2012.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXin:1412.6577 2014.
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.033 0.03333, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on compuler vision and pallern recognition, CVPR, pp- 770- 778, 2016.
Warren He, Bo Li, and Dawn Song. Decision boundary analysis of adversarial examples. In International Conference on Learning Representations, TCLR, 2018.
Andrew Nyas, Shibani Santurkar, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples not bugs, they are features. In Advances in neural information processing systems, NewrIPS, volume 32, 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL https://www.cs. woronto edu/ -kriz/cifar .hmll
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In International Conference on Learning Representations, ICLR, 2017.
Matthew Leavitt and Ari Morcos. Towards falsifiable interpretability research. arXiv preprint arXiv:2010. 12016, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324,, 1998.
John M. Lee. Introduction to Smooth Manifolds. Number 218 in Graduate Texts in Mathematics. Springer, edition, 2013. ISBN 978-1-4419-9981-- 978-1-4419-9962-5
John M. Lee. Introduction Riemannian Manifolds. Springer International Publishing, edition, 2018. ISBN 978-3-319-91754-.
Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. Certified adversarial robustness with additive noise. Advances in Neurol Information Processing Systems, 32:9464 9474, 2019a.
Qiyang Li, Saminul Haque, Cem Anil, James Lucas, Roger B Grosse, and Jörn-Henrik Jacobsen. Preventing gradient attenuation in lipschitz constrained convolutional networks. Advances in neural information processing systems, NewrIPS, 32:15390-154P0 15402, 2019b.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black- bax attacks. In International Conference on Learning Representations, ICLR, 2017.
12

---

Under review as a conference paper at ICLR 2022
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In International Conference on Learning Represenlations, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant. to adversarial attacks. In International Conference on Learning Represenlations, TCLR, 2018.
Jan R. Magmus and Heinz Neudecker. Matrix Differential Calculus with Applications in Stalislics and Bconometrics. John Wiley, 1999. ISBN 778-0-471-986322 978-0-171-98633-11
Xiaofeng Mao, Yucfeng Chen, Yuhong Li, Yuan Hc, and Hui Xue. Learning to characterize adversarial subspacos. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2438 2442, 2020. doi: N0.1109/ICASSP-00777
2020.9052933.
Seyed-Mohsen Moosavi-Dezfooli. Geometry of adversarial robustness of deep networks:
methods and applications. PhD thesis, EPFL, 2019. URL https://ännooscience.
ch/record/271933
Scyed-Mohsen Moosavi-Dezfool: Alhusscin Fawzi, Omar Fawzi, and Pascal Frossard. Univer- sal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pallern recomnilion, CVPR, pp. 1765 1773, 2017a.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, Pascal Frossard, and Stefano Soatto. Analysis of universal adversarial perturbations. ArXiv e-prints, pp. arXiv 1705, 2017b.
Seyed-Mohsen Moosavi-Dezfooli Alhussein Fawzi, Omar Fawzi, Pascal Frossard, and Stefano Soatto. Robustness of classifiers to universal perturbations: A geometric perspective. In International Conference on Learning Representations, ICLR, 2018.
Seyed- Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robustness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 9078-9086, 2019.
Mervin E. Muller. A note method for generating points uniformly n-dimensional spheres. Commun. ACM, 2(4):19 20, April 1959, ISSN 0001-0782, doi: 10.1145/377939 377946. URL https:://oi..rgg// http://260...rg/0011155///////9/
Dylan M Paiton, Charles G Frye, Sheng Lundquist, Joel Bowen, Ryan Zarcone, and Bruno A Olshausen. Selectivity and robustness of sparse coding networks. Journal of Vision, 20(12):10-10, 2020.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29:3360-3366 -366000336.20 2016.
Paulo E Rauber, Samuel G Fadel, Alexandre X Falcao, and Alexandru C Telea. Visualizing the hidden activity of artificial neural networks. IEEE transactions on visualization and computer graphics, 11110110, 2016.
Paulo E Rauber, Alexandre X Falcao, and Alexandru C Telea Projections visual aids for classification system design. Information Visualization, 17(4):282-30) -05, 2018
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp.
2016.
Francisco Rodrigues, Mateus Espadoto, Roberto Hirata, and Alexandru C Telea Constructing and visualizing high-quality classifier decision boundary maps. Information, 10(9):280,
2019.
13

---

Under review as a conference paper at ICLR 2022
Nicole C' Rust, Odelia Schwartz, J Anthony Movshon, and Eero P Simoncelli. Spatiotemporal elements of macaque v1 receptive fields. Neuron, 46(6):945 956, 2005.
Anindya Sarkar and Raghu Iyengar. Enforcing linearity in dnn succours robustness and adversarial image generation. In Igor Farkaš, Paolo Masulli, and Stefan Wermter (eds.), Artificial Neural Networks and Machine Learning ICANN 2020, pp. 52-64, Cham, 2020. Springer International Publishing. ISBN 978-3-030-61609-0.
Carl-Johann Simon-Gabriel, Yann Ollivier, Leon Bottou, Bernhard Schölkopf, and David Lopez-Paz. First-order adversarial vulncrability of neural networks and input dimension. In International Conference on Machine Learning, pp. 5809 5817. PMLR, 2019.
Grzegorz Swirszez, Brendan O'Donoghue, and Pushmeet Kohli. Visualizations of decision regions in the presence of adversarial examples. In ICLR Debugging Machine Learning Models Workshop, 2019.
Christian Szegedy, Wojeiech Zaremba, Nya Sutskever, Joan Bruna, Dumitru Erhan, Tan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR, 2014. URL https: bctps://openreviiee net/ forum?id=kklr MTHMRQJG.
Pedro Tabacof and Eduardo Valle. Exploring the space of adversarial images. In 2016 International Joint Conference on Neural Networks, IJCNN, pp. 426-433. IEEE, 2016.
Thomas Tanay and Lewis Griffin. A boundary tilting persepective on the phenomenon of adversarial examples. arXiv preprint XrXi 07690, 2016.
Dávid Terjék. Adversarial lipschitz regularization. In International Conference on Learning Representations, ICLR, 2020.
Florian Tramer, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint (rXxc:1704.08455 2017.
Georgios Vlassopoulos, Tim van Erven, Henry Brighton, and Vlado Menkovski. Explaining predictions by approximating the local decision boundary. Xie preprint arXic:2006. 07985, 2020.
Andreas Wächter and Lorenz T Biegler. On the implementation of an interior- poinn filter line-search algorithm for large-scale nonlinear programming. Mathematical programmming, 106(1):25-57, 2006.
David Warde-Farley and lan J Goodfellow. Adversarial perturbations of deep neural net- works. In Tamir Hazan, George Papandreou, and Daniel Tarlow (eds.), Perturbations, Optimization, and Statistics, pp. 311. MIT Press, 2016.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 33, 2020.
Fuxun Yu, Zhuwei Qin, Chenchen Liu, Liang Zhao, Yanzhi Wang, and Xiang Chen. In- terpreting and evaluating neural network robustness. In Proceedings of the Twenty- Eighth International Joint. Conference on Artificial Intelligence, IJCAI-19, pp. 4199-4205. International Joint Conferences Artificial Intelligence Organization, 7 2019a. doi: 10. 10.24963/Fo 24993/ijcai.222 201g//.019//8.. URL https:: //doi. org/ 24963/ijcai. 2019/583.
Tao Yu, Shengyuan Hu, Chuan Guo, Wei-Lun Chao, and Kilian Q Weinberger. A new defense against adversarial images: Turning a weakness into strength. In Advances in Neural Information Processing Systems, NeurIPS, volume 32, 2019b.
Christoph Zetzsche and Erhardt Barth. Image surface predicates and the neural encoding of two-dimensional signal variations. In Bernice E. Rogowitz and Jan P. Allebach (eds.), Human Vision and Electronic Imaging: Models, Methods, and Applications, volume 1249, pp. 209-216. International Society for Optics and Photonics, SPIE-Int Soc Optical Eng, oct 1990. doi: 10.1117/12.19667. URL 10. 19657 //dx., doi. org/10
1117/12. 19667
14

---

Under review as a conference paper at ICLR 2022
Christoph Zetzsche and Gerhard Krieger. Nonlinear mechanisms and higher-order statistics in biological vision and electronic image processing: review and perspectives. Journal of Electronic Imaging. 10(1):56-100) 2001.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jor- dan. Theoretically principled trade-off wetw robustness and accuracy. In International Conference on Machine Learning, pp. 7472 7482. PMLR, 2019.
15

---

Under review as a conference paper at ICLR 2022
A Boundary curvature analysis
16
B Dataset and models
21
C Additional CIEAR-10 results
D Additional MNIST results
24
E Seed consistency
27
Comparisons to related work: Orthogonality and clipping
Comparisons to related work: Decision boundary curvature
30
A BOUNDARY CURVATURE ANALYSIS
The following definitions are common. Our notation follows Lee 2013 geometry, and Magnus Neudecker (1999) for linear algebra.
2018 for differential
A.1 INTRODUCTION
In order to derive our main curvalure expressions for graph submanifold, we first introduce some notation and recall some concepts.
Tangent Spaces If M is n-dimensional smooth manifold then the tangent space at p E M is a vector space T of derivations at p, which is isometric to R In particular, T the set of directional derivative operators acting on the smooth real functions f M R.
The disjoint union of all of the tangent spaces is the tangent bundle TM der Hp) x T,M p M}. If we equip M with smooth, bilinear, positive definite, sym- metric form g TM x TM + R then (M,g) is said be a Riemannian manifold and g the Riemannian metric tensor.
When it unambiguous, denote by I(E) the smooth mappings M E (for some topological space E). Most often we will refer to x(M) def T(TM), and represent elements of ((M) with the letters X, Y,Y.Z.
Differentials If M and N are manifolds and F M + N is differentiable, then the differential of F at p M the linear mapping denoted dFC, T,M T,N, where the action on tangent vectors is defined by
dF,(v)) o)
That is, the rate of change of the function f in the direction v e T,M the point p. As is common practice, we often omit the p subscripts for clarity.
The linear operator df linear functional when N in the above definition R, and at each E M there exists a vector grad fp E Tp So that
YyEMYYET,,M dfp(w) = g(grad fp. v),
and grad f x(M) is the associated vector field.
Connections A connection is operator V X(M) x(M) - x(M), linear in its first argument, and satisfying a product rule its second. Surprisingly, a connection uniquely defines connection in each tensor bundle (which overload with V) (Lee 2018
Including the (0, Oe-tens bundle, that is, the smooth real functions M.
16

---

Under review as a conference paper at ICLR 2022
Prop. 4.15). Importantly for us, we have xxf=xx for all X E {(M) (for any connection Riemannian manifold always posesses a Sevi-Cevita connection Lee 2018 $5), denoted by V, which is the unique connection satisfying
Vx.Y.Zex(M) Vxg(Y, - (VxX, Vx,2..
To f E (M)
we associale the Hessian operator V which is symmetric (0, 2)-
field
Lee
2018 Ex. 4.22) with v ((X,)) def Vx df() for X,Y X(M). We have (via
Lec
2018
Prop. 4.21)
Vx.Yex(M): Vx = Vx(Vyf) Vivxry/= - (Vx))
(4)
Using the compatibility of V with the metric we arrive at the following expression for the Hessian, which will be useful later
VXX(X,)) B X(Y)) - (VxV)s
Xglgrad (,Y) S VxY)
Vxg(grad f.Y)
g(Vx x grad f, Y) + (grad f, VxY) gggadd VxY)
o (Vx x grad f,Y).
(5)
A.1.1 SCAMANIPOLDS
If (M, g), (M, are two Riemanian manifolds with M C M, then an injection a: M + M is called inclusion mapping. If the inclusion is smooth isometry, that is
VUTETM g(u, "(du(u),
cquivalently, g -- then M is called Riemanian submanifold of M.
Using the inclusion map differential, we embed TM in TM, If M, M are of dimensions m n. then T.M occupies an m dimensional subspace of T.M for each each p E M, and defined in
M. The normal
bundle formed from the orthogonal compliment of T T, similar way to the tangent bundle, denoted NM and having dimension of m. For vectors E TM we have the decomposition T vd with TT TM and x NM.
Some care must be taken when taking covariant derivatives in TM with the ambient connection, formally
VXYETM: : x del
d.(x)d(()) dY(),
da(X)
where the overlines denote arbitrary smooth extensions of dc(X) and dc(Y) to open neigh- bourhoods of M in M.
Second Fundamental Form Tet M C M be an embedded sumbanifold. The second fundamental form is the (bilincar) mapping II x(M) x x(M) + T(NM) with IIX, der (VxY)+ If we equip M with a normal field N e I(NM), then the scalar second fundamental form is
Ix
II(,IIX))))
For the remainder we assume N is fixed. The shape operator X(M) + X(M) is the endomorphism obtained from 1y that satisfifes
Vx,YeX(M) (x
(6)
that is, by raising an index.
"That is, the action of vector field through the connection agrees with the ordinary directional derivative.
17

---

Under review as a conference paper at ICLR 2022
Curvature Let
be the shape operator associated to normal field on an embedded
submanifold M of dimension n. Then at p E M, the (real) eigenvalues of s are called principal curvatures and the eigenvectors are called principal directions p. The Gaussian curvature the product of the principal curvatures, det(s), and the mean curvature is their average, n tr(s)-
Assume (M,9) and (M,g) g)) are Riemannian manifolds with an inclusion mapping 1 M + M and M has the submanifold structure. Suppose M posesses a shape operator (or any other linear endomorphism) š TM TM. The submanifold structure then gives us an operator :: TM + TM, the pullback of by 4, via
VXETM: 188 de- s
Remark 1 (Euclidean subspace). We consider the special case of the Riemannian manifold RT, Suppose P R" + II is orthogonal (n I m) and let s _Rn + R be a linear operator. Then p. R + Rm is an inclusion map and the subspace pullback of s to R is
(P')*= 8 PsP
We refer to the curvature quantities and directions of (PT)'s 8 with the suspace prefix. That is, subspace principal curvature, subspace Gaussian curvature, and
A.2 HYPERSURFACES
Let (M,g) be an embedded submanifold of (M.g)- Suppose F: PUUG M U IR is a local defining function for M, that is, = X.Y E X(M). From the connection product rule (Lee 2018 p. 89)
for open U C M. Choose arbitrary
Vx Ierad grad F) Igrad Fl
Vx grad F + (x Igrad Fl) grad F.
Whence
8(Vx grad F,Y) Igrad Fl
(x Igradd F
(VxN,
g(grad F,Y)
8(Vx gradFF Igrad Fl
(7)
The zero the underbrace follows because grad F E NM and E TM.
The Weingarten
equation (Lee) 2018 Thm. 8.13(c)) yields sX = V giving us
g(V grad F,Y)
B (FF(X,))
(I, g(xX = (V.N,
(8)
Igrad F
Igrad F
A.2.1
THE CURVATURE OF A GRAPH MANIFOLD
The notation and conventions of this section closely follow Lee 2018 (8). Let M be the graph of smooth function f: R" + R, that is, M = ff
RT Since f is smooth M is indeed manifold (Lee) 2013 Ex. 1.30, p. 20). We regard (M, .) Riemannian submanifold of (RR+11 (s. 7), with the inclusion map :: M + Bu+1
The function F R"*1 - IR is global defining function for M with F(x, () def f(x) - L. Denote by Vʃ the vector partial derivatives f, and by Hy and HF, the matrices of second order partial derivatives of and f, all with respect usual coordinate frames, (x"), on Ra and n-+1 Then
VXYYTM V²F(X,Y) = (Hr X,Y).
At p M the differential dip has the Jacobian matrix
(Id ()) x" (p)))
18

---

Under review as a conference paper at ICLR 2022
We commit the mild sin of identifying dip with its Jacobian and X,Y TM with their parameterizations under the coordinate frame, so that we may use the notation of matrix multiplication.
In dropping the p-subscripts, the left inverse, (de)= is has the parameterization
(dd)) (Ld+Vf -! (db)
Thus
ld
Hr :
9)
Id Hy,
(de) HF(dc) = V
V
whence
(d.)- Hr((d)) (Id (I++V/Vf") - (da) Hy((d)
(Id+V/VfT) -1 Hy
(9)
We can now compute the shape operator of M associated to the unit normal field grad F/lgrad Fl. Pick arbitrary X,Y T.M. From
there is
-IN(X,Y) grad F
g(V x grad F, dc(Y))
g(d od))
(10)
and, using the bilinearity of of g we have
g(de
Hp deX, Y)
g(sX, I
grad F
This, together with yields the shape operator:
Vld+vsVF*) Hy
(11)
2 +1
Whence the Gaussian and mean curvature are the scalar fields
det (Td V Hy)
and tr((Id + VI VVV H,)
(12)
(1 +
n 1 +1)
respectively.
A.2.2 THE CURVATURE OF A LEVEL-SET MANIFOLD
We now consider the case where M is the level-set of a smooth function f IR" + so that for some E R have M fx E R" f(x) = c). The following theorem can be used to ensure the level-sets of f do indeed posses a manifold structure, which is a simplification of the constant rank level-set. theorem (Lee) 2013 Thm. 5.12) for our setting.
Theorem 1. Let f I=" + R be smooth function having consland rank Then the level-sels of I are all smooth manifolds of codimension r.
We use the same construction Lee
2013
Ex. 1.32, $5) to calculate the Gaussian and x R) n M.
mean curvature at a point (zo, vo) E
From the implicit function theorem [Lee neighbourhoods Un x S == xRR with (mo, 3o) E U,x Vo and a smooth function g Up Vo
2013
Thm. C.40) there open connected
so that (U, x V.M is the graph of g. That is, (Un x Vo) = {(x,g(z)) x x Up S Ra-
Consequentially the gradient of has the structure
Vreu. : (()) = = Dif(a,((x)) + D2f ( =0
(z. (t)
Vg(x)) -
(13)
19

---

Under review as a conference paper at ICLR 2022
Let 0 def 0/0z/ and og def 02/00/ dzl. Then for i,j [n 1] and x E Uo
0,g(z) - (())
(14)
and
02 = (0,
8 Onjj f + 0 mm f0 (0.f)
0 [ + 10gg
2
1
8n nj + 8º f0 (0:9)
0 0 f0,g
fnf
nj f0.g + 0 f0,00,g + of 8 10gg
f
1
8 [ 03g) 8h 1b,00 S4
(15)
J.
where every function on the right hand side is evaluated at either the point (z, gx))) (depending on the function) If we let V def (0/0z)... ..... 0/0z=~1)7, and H be, similarly, the (n 1)x (n -1) matrix of cross-partial derivatives, there are the more compact expressions for and
Vg
vs.
JnJ
and
1
Hg 0n
(Vg + Vgg) diag(VOn.) Jann f. (VgVg") + +Hy]
where the sum Vg + VgJ is to be interpreted as the matrix of pairwise sums of the def 0g/00e Ox + 0g/ 0w'. Then, as in Section A.2.1 F(x) der ".......--")) = local defining function for M on (U, x N. The local graph parameterization provided by g is Φ Uo + M with x(x) (x, and its differential has the parameterization
elements of Vg: (Vg Vg
dø = (Id Vg)
Finally, mutatis mutandis, the shape operator is given by 11 and the Gaussian and mcan curvature be computed in 12
Remark 2 (Numerical considerations). In order compute the level-set curvature we needed to apply the implicit function thcorem. However, it may be difficult numerically to stabilize the inverse operation in (13 To remody this problem, one may apply change of basis before computing the quantities (14b and (15). Let B E Baxn be orthogonal change of
basis operator. Then we apply all our operations to BTM del (BTx x M).
Differentiating f def fo BT yields the revised quantities
VEERm: : Vf(x)= = B Vf(B²)) and Hy lT.TB
Thiis easy do Do since (za,g(za)) (zr, yo)-
20

---

Under review as a conference paper at ICLR 2022
Table Accuracies of CIFAR models at different perturbation lengths
Dimension
2
3
6
7
8
10
95.25%
95. 95.25%
/ 95. 25.25%
/ 95.25%
/ 95.25%
95.25%
95.25%
95.25%
/ 95.25%
95.25%
90.83%
90.83%
90.83%
90.83%
90.83%
90.83%
90.83%
90.83%
90.83%
90.83%
0.1
51.43%
62.86%
70.48%
/ 76.2%
79.06%
80.01%
81.92%
81.92%
/ 82.87%
83.82%
86.29%
90.83%
90.83%
90.83%
90.83%
90.83%
90.83%
90.83%
90.83%
90.83%
0.25
1.91%
4.76%
5.72%
6.67%
7.62%
8.57%
9.52%
9.52%
9.52%
10.48%
79.93%
89.01%
89.92%
90.83%
90.83%
90.83%
90.83%
90.83%
90.83%
90.83%
0.5
0.0% /
0.0% /
0.0% /
0.0% /
0.0%
0.0%
0.0% /
0.0% /
0.0% /
0.0%
65.4%
84.47%
86.29%
87.2%
87.2%
87.2%
87.2%
87.2%
87.2%
87.2%
0.0% /
0.0% /
0.0% /
0.0%
0.0%
0.0% /
0.0% /
0.0% /
0.0%
0.0%
36.33%
54.5%
59.04%
61.76%
65.4%
69.03%
69.94%
70.85%
72.66%
73.57%
0.0% /
0.0%
0.0%
0.0%
0.0%
0.0% /
0.0% /
0.0%
0.0%
0.0%
2.72%
5.45%
10.9%
12.72%
13.62%
14.53%
15.44%
15.44%
16.35%
17.26%
B DATASET AND MODELS
For the adversarial subspace analysis used two identical ResNet-50 architectures (He
ct al.] 2016) and two identical WideResNet-TT architectures (Gowal et al. 20200- For each architecture, one model was trained the unmodified CIFAR-TO dataset (Krizhessky & Hinton 2009). For the second model we additionally incorporated adversarial examples in The training process, widely known adversarial training. A training perturbation size of 0.5 was used. To perform the curvature analysis, we used twice-differentiable ELU activation functions (Clevert et al. 2015) for both models, which had little impact on the clean or adversarial accuracies.
In the analysis we include 100 CIFAR-10 test images, class- balanced, all unseen during training, and correctly classified by both models. The optimization algorithm stopped after finding 50 orthogonal adversarial vectors for all 100 test images. For the curvature analysis we used a subset of 50 class-balanced images and 10 adversarial directions per image.
Additionally, we only considered one seed per CIFAR-10 model because we found that the analysis the MNIST trained models had little variation across seeds (see Appendix D The accuracy both models at different perturbation lengths and dimensious is shown in Table 2 Here, the first line of each corresponds to the accuracy of the naturally trained model, whereas the second line is referring to the adversarially trained model.
We repeated many of the experiments on the MNIST dataset (LeCun et al. 1998 and found largely consistent results (see Appendix D
C ADDITIONAL CIFAR-10 RESULTS
Here we show additional analysis to complement our findings on the CIFAR-10 dataset.
Figure shows five adversarially perturbed images and their corresponding original image found for the two ResNet-50 models. We can see that that the perturbation length is increasing with dimension number. Interestingly, while the adversarial categories may not be semantically adjacent to the clean classes, the adversarially trained model seems to require more semantically meaningful perturbations, e.g. the images of boats the top row, misclassified cars, are perturbed to have elements looking like wheels. We can also see that there is no clear, consistent lz-length of an adversarial attack that makes an image ambiguous for a human observer. For example, the robust model example in the first (ship) row of the adversarial 11 column pair has a perturbation length of 1.12 and could be mistaken car first glance. In contrast, the robust model example in the fourth (horse) row of the adversarial column pair is still clearly identifiable as a horse with a larger perturbation length of 1.2. This illustrates that the direction of the perturbation as important the
21

---

Under review as a conference paper at ICLR 2022
Adv
Adv 11
Adv 21
Adv 31
Adv 41
Original
Natural
Robust
Natural
Robust
Natural
Robust
Natural
Robust
Natural
Robust
Airplaue
(0.78)
(1.97)
deg((1660)
(1.69)
(1.6)
(0.12)
degt (1.20)
(3.88)
deg (2.08)
12.16)
bind (0.25)
(1-46166)
(0:55)
deer (2.45)
hini (0-43) sirplane (2 (2.64)
(0-45)
deer (2 (2.96)
(Q-AR)
hird (2.900
Figure 6: CIFAR-10 example adversarials. The panels show adversarial examples for
given input image. Column pairs are sorted by ad adversarial dimension. Within a pair, each column includes examples for the natural and robust models, respectively. The labels
below the images indicate the class assigned by the model. The numbers in brackets are the fz-norms the respective images.
perturbation magnitude itself, ct al. 2019). although assessing the "perceptual distance" is an open area of research.
previously reported by (Goodfellow et al. 2014 lyas
As was described earlier, uni- formly randomly sampled direc- tions within adversarial subspaces to measure the distance to the de- cision boundary. Figure 7 shows how many of the samples pointed to decision boundaries that were outside of the allowed input pixel bounds.
out of hounda ratio
0.2
natural
robust
Figure N shows that perturbation lengths are increasing sublincarly and monotonically with increasing number of adversarial dimensions. For the naturally trained model there also a much larger increase of adversarial dimension with in- creasing distance to distance to de- cision boundary. This figure is dif- ferent from e.g. Figure 2 in that it displays the lengths of the adver- sarial subspace axes, not the lengths of the random perturbations within these subspaces. Note that in the right panel of this figure both models plateau at 50 dimensions because stopped the oplimizer at this point.
17 25
33
41
49
dimension of adversarial sapee
Figure 7: Ratio of out of bounds samples. We uniformly sample the distance to the decision bound- ary with increasing dimensionnllity per input image and adversarial subspace dimension. We report. the frac- tion of directions pointing to out-of bounds boundaries, averaged across 100 images.
We also provide additional, and more detailed information on the distributions of distances to the decision boundary of randomly sampled vectors in adversarial subspaces in Figure D Each subplot shows a subspace of increasing dimensionality, ascending left-to-right, then top-to-bottom. We observe that the variance of the distances to the decision boundary of randomly sampled vectors smoothly increasing with dimensionality.
22

---

Under review as a conference paper at ICLR 2022
dimension of adversarial space
natural
50
adversarial vector length
3
robust
40
2
30
20
10
11
16
21
26
36
distance to decision boundary
Figure 8: Perturbation lengths and dimensionality. (left) The subfigure shows the distributtio of the n-th found adversarial perturbations across images. (right. We tested at distance increments per image. The lines represent the means over all test images and the color shaded areas show the 10th and 90th percentiles.
0.001 how many adversarial dimensions were found that distance
A
B
0.
0.25
-01 D 01
C
mm
mn
m
011
0.25
255
077
Figure 9: Distributions of distances to decision boundary. (A,B) Distributions for natural model. (C,D) Distributions for adversarially trained model. (A-C) We show the complete histograms corresponding provide the mean-normalized histograms to corresponding to the histograms in A and C to emphasize the smooth increase variance of distances with increasing dimensionality.
Figure 2 dimensions. (C,D) Additionally, we
23

---

Under review as a conference paper at ICLR 2022
distance coefficient distance to decision bnd.
0.4
ResNet-50
WideResNNet-70-16
0.2
0.2
1.5
5
10
15
20
dimension of adversarial space
Figure 10: Distances to the decision boundary. Compare with Figure
ADDITIONAL MODEL COMPARISON
To show that findings hold for different models, present results for two different architectures: a ResNet-50 (He et al. 2016)) and Ved-7-esNet-70-166 of which the latter is the state-of-the-arl adversarially trained network for CIFAR-10 images (Gowal et al. 2020 Both architectures were Irained with and without adversarial perturbations, resulting in models overall. Figures I0 through [4 repeat analysis compare natural and adversarial trained models, but instead compare models of different architectures and constant training procedure.
Figures 10 and 14b show comparison of the two naturally trained models. These figures indicate that there is little difference between the two model architectures regarding distance to the decision boundary and perturbation length over dimensions. Figures 12 13 and 14a show a comparison the two adversarially trained models. Here, the figures show that perturbation lengths well distances to the decision boundary are larger for the WideRes.Net-77 16 compared to the ResNet-50. This is expected, as the WideResNet has higher adversarial accuracy and thus larger perturbations are needed to fool it. At the level of these aggregate statistics, the hyperparameter differences in adversarial training (Figure 12 have more of an effect than the model architecture differences themselves (Figure 10 altough all differences are much less pronounced than the differences between adversarial and naturally trained networks (Figure? 2
The box plots depicting the distances to the decision boundary Figures 10 and 12 show that, for both architectures, the adversarial subspace is large, continuous, and increases sublinearly in volume as the dimensionality is increased. The observation that this finding agnostic to model architecture further supported by the similarity in distributions of samples mcasuring the distance to decision boundary that can be seen on the right hand side of the same figures. Thus, the observed trends and conclusions drawn from the analysis provided in this study hold across archilectures.
D ADDITIONAL MNIST RESULTS
We conducted the same experiments described in Section on networks trained with the MNIST dataset. As a test set we used 500 images (50 per class), unseen in training and correctly labeled by all investigated models.
24

---

Under review as a conference paper at ICLR 2022
dimension of adversarial space
ResNet-50
20
adversarial vector length
0.3
WideReaNet-70-16
15
0.2
10
0.1
5
10
15
20
0.25
0.5
nth adversarial perturbation
distance to decision boundary
Figure 11: Perturbation lengths and dimensionality. Compare with Figure
distance coefficient distance to decision bnd.
ResNet-50
3.0
WideResNNet-7-11
2.0
1.0
0.1
1.4
1.2
1
0.8
5
10
15
20
2.5
dimension adversarial space
Figure 12: Distances to the decision boundary. Compare with Figure 2
In Figure 18 we can see that perturbation lengths are much larger for MNIST than for CIFAR-10 10, although the relative increase in distances for the robust network is still present. The optimization was in general more difficult with MNIST, likely due the fact that the images all live very close to the allowable input bounds. Specifically, the algorithm found less than 30 adversarial dimensions for more than 95% of the test images (Figure 22 to be compared with finding >50 dimensions for 100% of the CIFAR-10 test images.
That is harder to fool the MNIST model can also be seen in Table the accuracies of both models are much higher for the same perturbation lengths than the accuracies of the CIFAR-10 models. For the curvature analysis, on the other hand, we found that the overall values larger on CIFAR-10. Additionally, the adversarially trained model had more decision boundary curvature within adversarial subspaces on MÑIST than on CIFAR-10.
We noticed that most of the MNIST adversarials lic in thin regions between the decision boundary and the valid pixel box boundary (compare Figure 17 against Figure 3) Figure 15 shows that this leads to large rates of out of bounds samples when measuring the distance to the decision boundary. For example, 90% of the time the decision boundary is outside the valid pixel range in the 6 dimensional adversarial subspace of the naturally trained model.
25

---

Under review as a conference paper at ICLR 2022
dimension of adversarial space
20
adversarial vector length
2.5
15
1.5
10
ResNet-50
0.5
WideResNet-7700
10
20
0
2
nth adversarial perturbation
distance to decision boundary
Figure 13: Perturbation lengths and dimensionality. Compare with Figure
0.25
0.5
out of bounds ratio
0.4
of bounds
0.15
0.3
0.2
0.1
0.1
ResNet-50
0.05
ResNet-50
Wide Reset--666
WideReaNeet770-16
0
20
10
15
dimension of adversarial sapce
dimension of adversarial sapce
(a)
(b)
Figure 14: Ratio of out of bounds samples. Compare with Figure
7
Table 3: Accuracies of MNIST models (seed 0) at different perturbation lengths
Dimension 6
2
10
0
98.59%
98.59%
/ 98.59%
/ 98.59%
98.59%
98.59%
98.59%
98.59%
98.59% 98.56%
98.59% 98.56%
98.56%
98.56%
98.56%
98.56%
98.56%
98.56%
98.56%
98.56%
0.5
90.51%
97.41%
98.0%
98.39%
98 38.39%
98.39%
/ 98.37%
98. 08.35%
98.29%
/ 98.23%
97.77%
98.56%
98.56%
98.56%
98.56%
98.56%
98.56%
98.56%
98.56%
98.56%
60.53%
90.31%
/ 95.83%
/ 97.00 07.01%
4%
97. 97.58%
/ 97.9 07.94%
97 07.86%
97 97.69%
/ 97.51% /
94.03%
98.36%
98.56%
98.56%
98. 98.56%
98.56%
98.56%
98.56%
98.56%
98.56%
1.77% /
18.34%
47.91%
/ 71.58%
83.54 54%
88.51%
91.63%
/ 93.0% /
93.8%
93.55% /
68.99%
89.3%
95.01%
96.59%
97.57%
97.69%
97.73%
97.72%
98.56%
98.56%
0.0%
0.0%
0.2%
4.14%
15.05%
26.41%
/ 32.43%
38.61%
37.46%
32.74%
0.59%
9.46%
27.99%
46.52%
65.77%
76.32%
82.59%
83.94%
85.12%
80.16%
0.0%
0.0%
0.0%
0.39%
1.98%
7.86%
10.45%
18.94%
16.48%
16.55%
0.0%
0.0%
0.2%
7.69%
24.69%
37.51%
47.63%
48.86%
54.54%
40.74%
26

---

Under review as a conference paper at ICLR 2022
out of bounds ratio
0.75
0.25
natural
robust
3
dimension of a'versaraaal sapce
Figure 15: Ratio of out of bounds samples. Compare to Figure 7
distance to decision bnd.
5
natural
distance coefficient
1.4
A
4
robust
0.2
1.2
3
0
2
1
0.8
0.2
0
2
5
3
6
dimension adversarial space
Figure 16: MNIST distances to the Decision Boundary. We repeated the distance
analysis from Figure 2 on the MNIST dataset Only test images with at least 8 adversarial dimensions were included in this Figure
E SEED CONSISTENCY
We trained the natural and robust MNIST models with five different seeds and tracted adversarial subspaces for each model to examine the influence of training seeds on our results. Figure 22 shows that there slight variation number of adversarial subspace directions found by the optimizer. However, the overall trend of bar heights is signif-
mean distance
0.02
0
25
50
75
100
similar across seeds and there icant outliers.
n samples
Figure 21: Sampling size adversarial
subspaces. The line shows the mean running difference in distance to the decision boundary means across 100 samples sample size from to 100. The shaded area depicts the standard deviation from the mean line.
To further validate our results, repeated the experiment for Figure 16 (measuring the length of the adversarial vector one increases dimensionality on the MNIST trained networks) across five different seeds per training type. The results are shown in Figure 23 The variation across seeds for single image is smaller than the variation across images for single seed, which can be seen by comparing Figure 23 Figure [6] In general, we also found that the results and trends were consistent. ween MNIST and CIFAR-10 networks (with differences noted in Appendix D). Thus, we considered one seed per CIFAR-10 model as sufficiently representative.
increases the
We also investigated the influence of random number generation relevant for sampling within adversarial subspaces. As an empirical test, we varied the number of samples and measured the change in mean distance to the decision boundary one increases the number of samples. This was repeated for 100 different 50-dimensional subspaces (i.e. subspaces for 100 different test images) with the naturally trained CIFAR network. Figure 21 shows that the absolute change decreases steadily to an average distance of 0.004. Thus, the mean variation from 100 samples is much smaller than the general distance scales we measuring throughout
27

---

Under review as a conference paper at ICLR 2022
predicted
1.50
ciaee
0.0
0.0
5.0
6.3
0.9
adv
dir
dir
0.0
1,03
2.39
6/3
1.40
2.97
0.0
2.67
6:15
6.0
1.48
2.56
0.0
244
4.11
1.87
1.45
1.84
0.2
244
1.25
2322
2.00
Figure 17: MNIST boundary visualizations. This figure is the MNIST version of Figure
8
Note that for MNIST, the decision boundary lies outside the valid pixel range more often
han for CIFAR.
natural
8
adversarial vector length
6
robust
adversarial dimension
4
2
2
3
5
6
7
8
2.5
5
adversarial dimension
distance decision boundary
Figure 18: MNIST perturbation lengths and dimensions. Compare with Figure
the paper. We also measured the variation distances across random seeds for the sampling process. To do this, we used 10 different seeds to generate 100 samples in 50-dimensional subspace corresponding to one test image. For each seed, we compute a mean distance, averaged across 100 samples, giving us 10 values. The mean distance across seeds was 0.0933, and the standard deviation was 0.0007. The standard deviation is less than 1% of the mean and thus sufficiently small to consider estimate with 100 samples reprosentative.
F COMPARISONS TO RELATED WORK: ORTHOCONALITY AND CLIPPING
One obstacle in finding proper adversarial subspaces spanned by orthogonal vectors is assuring that perturbations do not cause pixel values to be of the valid input boundaries [0, 1]. We noticed that many studies solve this problem by element wise pixel clipping, Here, we show that pixel clipping is problematic when talking about the dimensionality of subspaces. Element wisc pixel clipping is non-lincar operation performed on vector and can therefore change its direction the input space and its relative angle other, fixed vectors.
To show that clipping has a non-negligible effect, investigated relative angles of basis vectors that span (sub-) spaces for 100 MNIST test images relevant for two studies.
First, in their work, Tramèr et al. (2017) propose the GAAS method. GAAS estimates the dimensionality of adversarial subspaces by defining an orthogonal set vectors aligned with an attack vector to input image. However, before passing the orthogonal basis vectors to the model, a clipping operation is performed. Figure 24 (left) shows that this clipping operation causes the input vectors to be non-orthogonal orthogonality would constitute angle of 90" or T. In fact, for the 100 sample images we tested, there is not a single vector pair that is completely orthogonal. The author's conclusions regarding the dimensionality
28

---

Under review as a conference paper at ICLR 2022
Adv
Adv 2
Adv
Adv 4
Adv Robust
Original
Natural
Robust
Natural
Robust
Natural
Robust
Natural
Robust
Natural
2-
S
S
5 3:1.96)
5
5
S
5
S
5(1.56)
712.50)
3
0
3
3
3
3
3
3
3
(22322
11.500
(2.38)
12.000
22 12.55
7....99)
8
(00.88)
(3.33)
5.0..7)
281.481
2 (3.44)
8:1.900
7 7.6444)
2.22.55)
(4..0))
Figure 19: MNIST example ad adversarials. Compare to Figure G
Test boundary
Adversarial boundary
0.005
A.
0.005
B.
Mear curvatire
Training type Natural Robust
0.000
0.000
-0.005
0.005
Adversarial boundary, full dimensionality
Random subspaces
Adversarial subspaces
C.
0.04
D.
0.04
Curvature
0.02
0.02
0
0.02
0.00
0.00
0.00
00.02
0.02
0.02
0.04
0.04
Principal curvature directions
Figure 20: MNIST decision boundary curvature. Compare with Figure
of adversarial subspaces and transferable subspaces are therefore grounded on ill-defined subspaces, spanned by non-orthogonal vectors.
In another study, He et al. (2018) estimate the distance to the decision boundary for the complete imput space MNIST. For that, basis of 784 vectors, spanning the whole imput space, is randomly generated. Again, the inputs are clipped before they are passed to the model leading to the same problem as described carlier. And as before, there are no orthogonal vector pairs, and angels are accumulated at much smaller values than Fas can be seen Figure 24 (right). We therefore note that it not guaranteed that the input space is evenly covered and that edges of the 784-dimensional hypercube (in the case of MNIST) are arguably overrepresented.
29

---

Under review as a conference paper at ICLR 2022
Natural models
Robust models
seed
11 samples
2
2
200
5
5
10
20
30
10
20
30
found dimensions
found dimensions
Figure 22: Optimization consistency on MNIST. The bars show number of samples for which at least dimensions were found during optimization. Different colors indicale distinct seeds for model training.
natural
adversarial voctor length
5
robust
4
2
adversarial dimension
Figure 23: Distance consistency on MNIST. Box plots show the length of the adversarial vector increases dimensionality. The data is computed for single image, where the variation comes from five different seeds used to initialize the networks before training. Note that the variation is much smaller than the variation across images for single seed, as can be seen in Figure
G COMPARISONS TO RELATED WORK: DECISION BOUNDARY CURVATURE
Table 4: Decision boundary curvature formulas present in the literature for a decision function Fiy R -> IR. The generating matriz column refers to the matrix used to yield eigenvalues for the principle curvatures, and is roughly comparable to the shape operator.
Reference
Derivation
Generating matrix
Dimension
P PHy, TV P PH P
Id V., V VF,l" Not given
Poole al. 2016
None
n. X n n X n n X n
Moosavi-Deztonn et al.
2018
None
Moosavi-Deztooli et al.
2019
None
Crossenetropy Hessian PHe, TV
Moosavi-Dezfool
2019
p. 90
None
Id V
VFT
n X
This work
Appendix A
using
15
x
This section provides a detailed comparison our analysis against, previously published works on the principal curvature analysis applied to decision boundaries. We include table for a quick summary.
The use of differential geometry to understand the non-linear response properties of model neurons has a long history in computational neuroscience, with early contributions made by Zetzsche & Barth 1990). More recently, several works have specifically used Reimannian geometry to measure the principal curvature of response manifolds (Fawzi et al. [0166] Poole
et. al. 2016 Moosavi-Dezzoolii et al. 2018
2019 Golden et al. 2019). As discuss in
30

---

Under review as a conference paper at ICLR 2022
Tramèr et al. method
He et al. method
1e6
n1 samples
400
4
200
1.18
1.31
1.44
1.57
1.18
1.31
1.44
1.57
angle (rad)
angle (rad)
Clipping causes non-orthogonality. The histograms show angles between vectors that define "subspaces" in the GAAS method proposed by Tramer al. (2017) (left) and distance to decision boundary estimations by He et al. (2018).
Figure 24:
Appendix A certain smoothhess assumptions are required for such an analysis. Our present study, as well Poole et al. (2016) and Golden et al. (2019), avoid this problem by using wice-diiffernttabbe activation lunctions. In the case of Golden et al. (2019): ) this was done without retraining the network, while Poole et al. (0016) used nets with random weights. We instead trained the twice- differentiable networks from scratch. Moosavi-Dezfooll et al. used finite-difeerence approximation to avoid the problem (described in Moosavi-Dezfoon
ct al. 2019 oq 1 and surrounding discussion). Another high-level difference, discussed in detail Appendix A is the choice of measuring the response graph curvature (which is dimensional for n dimensional inputs) or the level-set curvature (which is - dimensional for dimensional inputs) of given function. This difference was discussed in (Golden let al. 2019 at the end of Section 4 and in Figure 4), although their technical derivations are limited only the graph formulation. The other studies did not explicitly discuss this difference. Importantly, though, the decision boundary is the level-set of the logit difference function, as defined in Equation
Several works from Moosavi-Dezfood and colleagues explore decision boundary curvature and its relationship to adversarial robustness (Fawzi al. 2016 Moosavi-Dezfooll et al.) 2017b 2018 2019 Moosavi- Deziool 2019). Within these works they provide inspired perspective on explaining adversarial robustness in terms the curvature decision boundaries. Almost all of the work is with respect to universal adversarial perturbations, which share common subspace (Moosavi-Dezfooll et al. 2017a), although Fawzi et al. (2016) alternatively look at random subspaces. Their universal adversarial perturbation subspaces differ from our adversarial subspaces in that theirs are defined by orthogonal directions that transfer well across inputs instead of those that point the nearest decision boundary for given input. Despite these differences, the stated quantity of interest, that is, the principle curvatures at point on the docision boundary is the same their work as well as ours.
However, each of these papers have slightly different analytic formula for estimating the curvature of the boundary within given subspace (Fawzi et al. 2016 Moosavi Dezfooli et al. 2017b), in the high-dimensional input space (Moosavi-Deztood et al. 2018 2019).
We found the most complete description be given in Moosavi-Deztoolf (2019). which
summarizes majority of the earlier work and applies the decision boundary curvature analysis to both subspaces and the higl-dimensional input space. Therein, Moosavi-Deztooll 22019 Section. 7.4) state the following formula, without derivation, for the shape operator in the notation of Section 2.2
$ (F,( PHF, P,
where Hr., is the Hessian of the boundary function, (x)( del f((x)) fz( and P der Id - ( is described projection operator. This formulation is similar to that used by (Poole et al., 20166), with the only difference being that they use normalized gradients in the projection operator.
31

---

Under review as a conference paper at ICLR 2022
Table 5: Calculations from alternative attempts find incorrect principal curvature for
level-set on sphere of varying dimensionality and radius. The quantity reported is the mean of the absolute difference between each computed principal curvature and the inverse radius.
dimensionnllly-rrddus Moosavi et al. Poole et al.
Ours
02 00.1
7.6o-01
5.0c-+00
0.0c++00
02 01.0
2,4e++01
5.0e-01
0.0e+-00
10.0
3.1e+04
5.0e-02
1.3e-17
05 00.1
7.2e-01
2.0e++00
2.2e-15
01.0
7.2e++01
2.0e-01
2.2e-16
05 10.0
7.9e+044
2.0e-02
3.8e-17
10 00.1
6.4e-01
1.0e++00
2.3e-15
10 01.0
1.5c+022
1.0c-01
2.2c-16
10 10.0
1.5e+005
1.0c-02
4.4c-17
Table Calculations from alternative attempts find incorrect Gaussian curvature (i.e. the product of the principal curvatures) for a Hyperboloid graph.
Analysis type Absolute error
Moosavi et al.
6.8e+00
Poole et al. 1.4e-03
Golden et al. 4.1e-10 10
Ours
4.1c-10
Our formulation differs from the above in several ways, the most important of which that they claim to transform the Hessian by projecting it onto the tangent space (as written above), while we apply the implicit function theorem parameterize the decision boundary as the graph of function (see Appendix Section A.2.2 Equation 13p. For computing the curvature of the decision boundary, exclusively projecting the Hessian is not going to result in the correct calculations, since for the decision boundary (as defined in Section 2.2 Equation 2 you need a different normal than for the graph of the response function. More specifically, the normal used for computing the shape operator of the graph surface is different from that for computing the shape operator the decision boundary. The latter has to be exclusively in the pixel space, while the former always has component in the direction of the function response. Projecting the Hessian can restrict the computed curvatures of directions, but it cannot change the computed curvatures in said subspacc. Another important difference is their projection operator is computed directly from the gradient, and thus has a dimensionality equal to the input dimeesioonllity To calculate the curvature of decision boundary, one needs a level-set formulation, which should have one fewer dimensions than the input space.
a cortain subspace
The consequence of this difference can be made clear when looking at the curvature of an n-dimensional sphere, defined by the quadratic function:
F(x) F(
where is the input point vector and H is the sphere's Hessian (i.e. an identity matrix in Raxn). Given this function, the principal curvatures of the level-sct at some radius, are all known be (up a sign change) However, we show in table 3 that the formulations provided by Moosavi-Dexloolli (2019) and Poole et al. (2016) produced curvatures that are often several orders of magnitude off from the correct values the case of high-dimensional inputs. Due to the dimensionality issue above, computed the average error across each principal curvature, regardless the number of curvatures reported, as they should all equal
We additionally tested a graph formulation (defined in Section A.2.1 using a one-shected hyperboloid. This formulation lives in the full space, there principal curvature per imput dimension. However, we show similar errors in Table 0 Note, that the formulation from Golden al. 2019) matched our own, although they did not provide equation
32

---

Under review as a conference paper at ICLR 2022
for mapping this to the level-set curvature. Our own implementations high precision on these and other experiments with known functions, all of which can be replicated from our provided code. Finally, while the subspace projection methods proposed by (Fawzi et al., 2016 Moosavi-Dezzooli et al. 2017b) potentially provide metrics related to curvature in the subspaces they defined, chose to instead derive our pullback method described in Equation 3 from primary sources, as explained in Appendix A
correct with
33


Reviews:
[{"review_id": "e7pK-C9-18K", "reviewer": "Reviewer_Pjpz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n1. The study of an interesting problem.\n\n2. Some of the findings are not new but do have some insights.\n\n\nWeak points:\n1. The paper is poorly presented. It is hard to see the implications of the findings to DNN robustness. The ideas are poorly motivated. \n\n2. There are so many different ways of studying the adversarial subspaces, why do the authors choose to study geometry in certain ways? How does this work help deep learning differently from prior works?\n\n3. The findings in this work may not be completely new, some of them are caused by the smoothing effect of adversarial training.\n\n4. The dimensionality of inputs/representation space has been thoroughly studied in existing works [1,2]. How the dimensionality investigation in this work is different? Is there a difference in studying the input space vs the representation space? It seems to me some of the studies make more sense in the representation space.\n\n5. The geometric property of adversarial subspaces has been studied in [3], one well-known adversarial subspace characterization work.  What is the difference between this paper to [3], are the subspaces studied in [3] the same as here? In [3], they define adversarial subspaces to be the space around adversarial examples, which is quite intuitive. The adversarial subspaces here are defined to be where misclassification occurs (more general, not tied to the worse-case adversarial examples). Does this mean misclassificaiton=adversarial? How about incorrectly classified natural examples, i.e., natural generalization error?\n\n6. Many understanding and adversarial defense work also provide important understandings about adversarial vulnerability, adversarial robustness and properties of adversarially-trained models. For example, the loss landscape understanding [4,5], decision tiltiling understanding [7], boundary smoothing [8] all look quite relevant. Without a thorough discussion of the existing understandings, it is hard to assess the novelty of this work and how the findings add to the adversarial robustness. \n\n7. It is unclear how the understandings developed in this paper can help future research. Can it be used to design or train more robust DNNs, evaluate their robustness, or maybe just a visualization method to help better understand the model? \n\n[1] First-order Adversarial Vulnerability of Neural Networks and Input Dimension, ICML 2019.\n\n[2] The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality, WIFS, 2017.\n\n[3] Characterizing adversarial subspaces using local intrinsic dimensionality, Ma et al. ICLR 2018.\n\n[4] Interpreting and Evaluating Neural Network Robustness, IJCAI, 19.\n\n[5] Adversarial Weight Perturbation Helps Robust Generalization, NeurIPS, 2020.\n\n[7] A Boundary Tilting Persepective on the Phenomenon of Adversarial Examples.\n\n[8] Theoretically Principled Trade-off between Robustness and Accuracy. ICML 2019.\n", "summary_of_the_paper": "This paper studies the geometry of adversarial subspaces around input samples, with a focus on their spatial properties and relation to the decision boundary. Orthogonal adversarial directions are found by untargeted adversarial perturbation and are used to define the adversarial subspaces. The principal curvature decomposition is applied to find the principle curvature (the degree of linearity) of the decision boundary in adversarial supaces. Based on the results obtained from these experiments, the authors draw several conclusions including 1) input samples lie close to decision boundary in a large subspace; 2) the distance to the boundary grows with the increase of the subspace dimensionality; 3) boundary in more curved in adversarial subspaces and adversarial training tends to even the boundary to different classes.", "summary_of_the_review": "This paper raises more questions than answers. It is hard to say the findings are reliable, accurate, or novel. The observations for adversarially trained models are more like a side effect of boundary smoothing, which is what adv training is designed for. Min-max optimization gives us a better explanation [8]. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nv-DsrjGNW1", "reviewer": "Reviewer_xfHm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the paper are:\n- fully taking into account the boundedness of the input domain when finding adversarial subspaces\n- principled and (almost) fully rigorous way of defining and measuring the curvature of the decision boundary\n- empirical investigation of the geometry of the decision boundary, leading to interesting (although not shocking) results\n\nI have two doubts about the paper, which might require more discussion in the paper (or I didn't understand something):\n1. Riemannian geometry machinery requires a smooth (i.e., infinitely differentiable) manifold. We know that the activation functions used in typical neural network models are not smooth. Hence, the decision boundaries will not be smooth either (but will be locally smooth). How does that affect the results? Is the local curvature representable of the large-scale geometry of the differentiable manifold if the manifold is not smooth? A zig-zagging line is locally linear almost everywhere, but can curve in space as much as it wants. If we sample its local curvature randomly, we will \"see it\" as a straight line.\n2. There is a lot of space in 50 dimensions. Is sampling e.g. 100 random vectors in a 50-dimensional adversarial subspace going to give us a representative sample? Have the authors tried to repeat the analyses with different random seeds and see if the empirical results are stable?\n\nThe most important doubt is 1. I think it invalidates the results of the curvature analysis, viz the example of the zig-zag line. Since this analysis is a major selling point of the paper, I cannot recommend accepting it at this stage.\n\nSome other comments:\n1. I think the paper \"Visualizations of Decision Regions in the Presence of Adversarial Examples\" (Swirszcz, O'Donoghue and Kohli) should also be referenced as it is related work.\n2. The first sentence in the caption for Figure 1 is hard to understand.\n3. In the last sentence of the caption for Figure 2, what is \"even subsampling\"?\n4. What is meant by \"well approximated\" in the discussion of the modification to the optimisation problem described by Eqn. (1)? Can the Authors provide some proof that this approximation works?\n5. I think the folk lore is that in order to prevent the perturbations from being visible to human viewer, it's better to cap their L_inf norm, not L_2. Something to consider.\n\n==== EDIT AFTER AUTHOR REPLY ====\n\nThe Authors have completely clarified Q2 (representative samples) to my satisfaction. Re Q1 (smoothness), they explained that they use a differentiable (though not smooth) ELU activation function, which alleviates my prior concerns, but raises the question of the applicability of their results to the more common case of ANNs using non-differentiable activation functions like RELU. However, this may be seen as an open question to be explored in future work. (It would also be nice if the Authors reviewed prior work for curvature analysis and commented on whether it also avoided the problem of using non-differentiable network architectures).", "summary_of_the_paper": "The paper investigates the geometry of the decision boundary surrounding test examples which are subjected to successful adversarial attacks. It uses Riemannian geometry to calculate the curvature of the boundary in a principled way. It also finds \"adversarial subspaces\" in the input space, which are spanned by orthogonal vectors pointing from the test example to the closest adversarial examples. Improving on prior work, the Authors ensure that the spanning vectors remain orthogonal after clipping them to ensure that the perturbed examples remain in the domain of the classifier (i.e., the [0, 1]^N hypercube). The empirical results are obtained by randomly sampling the adversarial subspaces. Using this method, the authors investigate the impact of adversarial training on the decision boundary: its distance from test examples and its curvature.", "summary_of_the_review": "I think that the paper is well-written, continues an interesting line of research and improves on it. The Authors clarified the technical questions. While I'm not sure how applicable the results are to common ANN architectures (because of the smoothness requirements), I recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NW-w4BJviGt", "reviewer": "Reviewer_kBnb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper studies the curvature properties of class boundaries of\nneural network architectures. A modified version of the optimization\nstrategy for adversarial perturbation vector generation (Szegedy 2014)\nis used to obtain an adversarial subspace. A distribution of the\ndistance of decision boundary (equi-logit points) to the test samples\nis computed in the adversarial subspace. The distribution shows that\nthe distances grow slowly as larger adversarial subspaces are\nconsidered. This explains the success of adversarial attacks. The\nspace operator is next used to obtain the tangent subspace. Basis of\nthe tangent subspace is ordered based on curvature. The subspace pull\nback is defined to measure the susceptibility of the image to an\nattack. It is observed that the subspace pull back distribution is\nsupported by most of the directions.\n\nIt is also observed that use of adversarial training increases the\nentropy of the class distribution close to a boundary. It also\nmodifies the pull back factor. This is a viable explanation of the\nsuccess of adversarial training strategies.\n\nThe paper uses a number of known analysis techniques to an elegant\nexplanation of adversarial robustness. Empirical studies support the\nclaims.", "summary_of_the_paper": "The paper studies the curvature properties of class boundaries of\nneural network architectures. A modified version of the optimization\nstrategy for adversarial perturbation vector generation (Szegedy 2014)\nis used to obtain an adversarial subspace.\nThe paper uses a number of known analysis techniques to an elegant\nexplanation of adversarial robustness. Empirical studies support the\nclaims.\n\n", "summary_of_the_review": "I'm leaning towards accepting this paper because it conducts an extensive analysis with existing approaches towards elegant explanations of adversarial robustness,", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LiP09SqMX3-", "reviewer": "Reviewer_Dw9Y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "<Updated after the author response: Thank you for the detailed response, clarifying comments, and additional experiments. I have increased my score to 6.> \n\n1. Using adversarial robustness to explain how and why deep networks perform well is an interesting idea, but it wasn't clear to me if this method can be used to explain the performance differences between a large array of standard deep networks. For example, if ResNet152 works better than AlexNet, can we use approaches introduced in this paper to explain their performances, or is the approach here limited to the comparison between adversarially trained vs. vanilla networks only? Empirical demonstration of the proposed approaches on standard deep networks would be helpful, as well as their limits and generalities. \n\n2. There are several prior works which deal with the geometry and curvature of deep network representations and boundary surfaces [1,2]. How does this work differ from them? \n\n\n3. One of the well-known inductive biases for adversarial robustness is to introduce noise or random perturbations [3,4]. Can you still apply the methods introduced here in the presence of additive noise or representation stochasticity? \n\n4. The empirical findings here (on the increase in boundary distance within adversarial subspaces, and the decrease in boundary curvature in an adversarially trained model) are perhaps not too surprising. The results from the comparison between adversarially trained model vs. standard model are reassuring, but the paper can be made stronger by comparing several adversarially trained models (or several adversarial attack methods), and demonstrate that the reported trends hold against multiple adversarially robust models. \n\nReferences \n\n[1] Lahiri et al: https://arxiv.org/pdf/1606.05340.pdf\n\n[2] Cohen et al: https://www.nature.com/articles/s41467-020-14578-5\n\n[3] Li et al https://arxiv.org/abs/1809.03113\n\n[4] Dapello et al https://proceedings.neurips.cc/paper/2020/hash/98b17f068d5d9b7668e19fb8ae470841-Abstract.html", "summary_of_the_paper": "This paper proposes methods to understand the geometry of decision boundaries of ANN classifiers by using adversarial perturbations as a tool. They introduce two methods: (1) a method for finding adversarial subspaces (where input sample has minimal distance to the decision boundary); (2)  a method for measuring the curvature of a decision boundary. These methods allow for new findings such as (1) distance to the boundary grows with increasing dimensionality of adversarial subspace; and (2) decision boundary is more curved in adversarial subspace than random subspace; and (3) insights into why adversarial training works well.", "summary_of_the_review": "Investigating the robustness of deep networks using the geometry of decision boundaries and adversarial subspaces is a well motivated and interesting idea. My main reservation about this paper is that the proposed interpretability metrics are demonstrated in a sanity-check style manner on only a small set of models. In my view, the utility of such interpretability metrics can be really demonstrated when they are applied against several models of similar types (e.g., an array of standard deep network models, and an array of adversarially robust models), and when it is shown that the empirical claims with the proposed metrics hold for a large array of standard models/training methods. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work investigates the decision boundary of neural networks by quantifying in various ways the shape and curvature of the error set local to correctly classified inputs, dubbed the "adversarial subspace". First, a method is introduced which seeks to find the largest set of orthogonal directions starting at in input x which will all intersect the error set local to an image. This is motivated as a certain geometric measure of the error set, large sprawling error sets may have many orthogonal directions which intersect it local to the given input while small narrow error sets may have relatively few. Using this geometric measure, the authors compare the shape of adversarial subspaces of various image models both with and without adversarial training. After the rebuttal period, reviewers all felt that the work was borderline, with no one strongly advocating for the work. As noted by some reviewers, while some experiments may be interesting it was unclear what new insights the work contributes. For example, the authors argue that the change in the geometry of the error set explains why adversarial training works. It is unclear how this is an explanation more than it is simply an observation that the error set geometry has changed. An analogy would be trying to explain why Resnet-50 performs better than AlexNet by showing that it has higher test accuracy---this only shows that it is better, but doesn't explain why.

During the discussion period the AC raised additional concerns regarding a sanity check that the author's main algorithm should pass. In particular, consider an error set x_1 >= K(x_2^2 + ... + x_n^2) + C, parameterized by constants K and C > 0. For all choices of K and C and starting point x = (0, ..., 0), the authors main algorithm will always return 1 as the dimensionality of this error set. It will find the vector (1, 0, ..., 0) and then terminate. However, this is problematic because we can choose K and C to make this error set either very narrow (e.g. K=C=100) or very wide (K = .000001, C = .00001)---the proposed algorithm will be unable to distinguish between this two extremes. Given this, it seems that greedily selecting the set of orthogonal directions starting at x can be very suboptimal if the intent is to find a maximum sized set of orthogonal directions. 

To conclude, the work would be substantially improved if it addresses two major weaknesses. First, there needs to be a clearer motivation for studying this notion of geometry of the error set, what new insights can the authors provide other than adversarial training changes the shape of the error set? Second, the method doesn't seem to be principled given it is unable to distinguish between the two extreme cases discussed above.
Decision: Reject