Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the curvature properties of class boundaries of
neural network architectures. A modified version of the optimization
strategy for adversarial perturbation vector generation (Szegedy 2014)
is used to obtain an adversarial subspace.
The paper uses a number of known analysis techniques to an elegant
explanation of adversarial robustness. Empirical studies support the
claims.


main_review: The paper studies the curvature properties of class boundaries of
neural network architectures. A modified version of the optimization
strategy for adversarial perturbation vector generation (Szegedy 2014)
is used to obtain an adversarial subspace. A distribution of the
distance of decision boundary (equi-logit points) to the test samples
is computed in the adversarial subspace. The distribution shows that
the distances grow slowly as larger adversarial subspaces are
considered. This explains the success of adversarial attacks. The
space operator is next used to obtain the tangent subspace. Basis of
the tangent subspace is ordered based on curvature. The subspace pull
back is defined to measure the susceptibility of the image to an
attack. It is observed that the subspace pull back distribution is
supported by most of the directions.

It is also observed that use of adversarial training increases the
entropy of the class distribution close to a boundary. It also
modifies the pull back factor. This is a viable explanation of the
success of adversarial training strategies.

The paper uses a number of known analysis techniques to an elegant
explanation of adversarial robustness. Empirical studies support the
claims.
summary_of_the_review: I'm leaning towards accepting this paper because it conducts an extensive analysis with existing approaches towards elegant explanations of adversarial robustness,
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes methods to understand the geometry of decision boundaries of ANN classifiers by using adversarial perturbations as a tool. They introduce two methods: (1) a method for finding adversarial subspaces (where input sample has minimal distance to the decision boundary); (2)  a method for measuring the curvature of a decision boundary. These methods allow for new findings such as (1) distance to the boundary grows with increasing dimensionality of adversarial subspace; and (2) decision boundary is more curved in adversarial subspace than random subspace; and (3) insights into why adversarial training works well.
main_review: <Updated after the author response: Thank you for the detailed response, clarifying comments, and additional experiments. I have increased my score to 6.> 

1. Using adversarial robustness to explain how and why deep networks perform well is an interesting idea, but it wasn't clear to me if this method can be used to explain the performance differences between a large array of standard deep networks. For example, if ResNet152 works better than AlexNet, can we use approaches introduced in this paper to explain their performances, or is the approach here limited to the comparison between adversarially trained vs. vanilla networks only? Empirical demonstration of the proposed approaches on standard deep networks would be helpful, as well as their limits and generalities. 

2. There are several prior works which deal with the geometry and curvature of deep network representations and boundary surfaces [1,2]. How does this work differ from them? 


3. One of the well-known inductive biases for adversarial robustness is to introduce noise or random perturbations [3,4]. Can you still apply the methods introduced here in the presence of additive noise or representation stochasticity? 

4. The empirical findings here (on the increase in boundary distance within adversarial subspaces, and the decrease in boundary curvature in an adversarially trained model) are perhaps not too surprising. The results from the comparison between adversarially trained model vs. standard model are reassuring, but the paper can be made stronger by comparing several adversarially trained models (or several adversarial attack methods), and demonstrate that the reported trends hold against multiple adversarially robust models. 

References 

[1] Lahiri et al: https://arxiv.org/pdf/1606.05340.pdf

[2] Cohen et al: https://www.nature.com/articles/s41467-020-14578-5

[3] Li et al https://arxiv.org/abs/1809.03113

[4] Dapello et al https://proceedings.neurips.cc/paper/2020/hash/98b17f068d5d9b7668e19fb8ae470841-Abstract.html
summary_of_the_review: Investigating the robustness of deep networks using the geometry of decision boundaries and adversarial subspaces is a well motivated and interesting idea. My main reservation about this paper is that the proposed interpretability metrics are demonstrated in a sanity-check style manner on only a small set of models. In my view, the utility of such interpretability metrics can be really demonstrated when they are applied against several models of similar types (e.g., an array of standard deep network models, and an array of adversarially robust models), and when it is shown that the empirical claims with the proposed metrics hold for a large array of standard models/training methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the geometry of adversarial subspaces around input samples, with a focus on their spatial properties and relation to the decision boundary. Orthogonal adversarial directions are found by untargeted adversarial perturbation and are used to define the adversarial subspaces. The principal curvature decomposition is applied to find the principle curvature (the degree of linearity) of the decision boundary in adversarial supaces. Based on the results obtained from these experiments, the authors draw several conclusions including 1) input samples lie close to decision boundary in a large subspace; 2) the distance to the boundary grows with the increase of the subspace dimensionality; 3) boundary in more curved in adversarial subspaces and adversarial training tends to even the boundary to different classes.
main_review: Strong points:
1. The study of an interesting problem.

2. Some of the findings are not new but do have some insights.


Weak points:
1. The paper is poorly presented. It is hard to see the implications of the findings to DNN robustness. The ideas are poorly motivated. 

2. There are so many different ways of studying the adversarial subspaces, why do the authors choose to study geometry in certain ways? How does this work help deep learning differently from prior works?

3. The findings in this work may not be completely new, some of them are caused by the smoothing effect of adversarial training.

4. The dimensionality of inputs/representation space has been thoroughly studied in existing works [1,2]. How the dimensionality investigation in this work is different? Is there a difference in studying the input space vs the representation space? It seems to me some of the studies make more sense in the representation space.

5. The geometric property of adversarial subspaces has been studied in [3], one well-known adversarial subspace characterization work.  What is the difference between this paper to [3], are the subspaces studied in [3] the same as here? In [3], they define adversarial subspaces to be the space around adversarial examples, which is quite intuitive. The adversarial subspaces here are defined to be where misclassification occurs (more general, not tied to the worse-case adversarial examples). Does this mean misclassificaiton=adversarial? How about incorrectly classified natural examples, i.e., natural generalization error?

6. Many understanding and adversarial defense work also provide important understandings about adversarial vulnerability, adversarial robustness and properties of adversarially-trained models. For example, the loss landscape understanding [4,5], decision tiltiling understanding [7], boundary smoothing [8] all look quite relevant. Without a thorough discussion of the existing understandings, it is hard to assess the novelty of this work and how the findings add to the adversarial robustness. 

7. It is unclear how the understandings developed in this paper can help future research. Can it be used to design or train more robust DNNs, evaluate their robustness, or maybe just a visualization method to help better understand the model? 

[1] First-order Adversarial Vulnerability of Neural Networks and Input Dimension, ICML 2019.

[2] The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality, WIFS, 2017.

[3] Characterizing adversarial subspaces using local intrinsic dimensionality, Ma et al. ICLR 2018.

[4] Interpreting and Evaluating Neural Network Robustness, IJCAI, 19.

[5] Adversarial Weight Perturbation Helps Robust Generalization, NeurIPS, 2020.

[7] A Boundary Tilting Persepective on the Phenomenon of Adversarial Examples.

[8] Theoretically Principled Trade-off between Robustness and Accuracy. ICML 2019.

summary_of_the_review: This paper raises more questions than answers. It is hard to say the findings are reliable, accurate, or novel. The observations for adversarially trained models are more like a side effect of boundary smoothing, which is what adv training is designed for. Min-max optimization gives us a better explanation [8].
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the geometry of the decision boundary surrounding test examples which are subjected to successful adversarial attacks. It uses Riemannian geometry to calculate the curvature of the boundary in a principled way. It also finds "adversarial subspaces" in the input space, which are spanned by orthogonal vectors pointing from the test example to the closest adversarial examples. Improving on prior work, the Authors ensure that the spanning vectors remain orthogonal after clipping them to ensure that the perturbed examples remain in the domain of the classifier (i.e., the [0, 1]^N hypercube). The empirical results are obtained by randomly sampling the adversarial subspaces. Using this method, the authors investigate the impact of adversarial training on the decision boundary: its distance from test examples and its curvature.
main_review: The strengths of the paper are:
- fully taking into account the boundedness of the input domain when finding adversarial subspaces
- principled and (almost) fully rigorous way of defining and measuring the curvature of the decision boundary
- empirical investigation of the geometry of the decision boundary, leading to interesting (although not shocking) results

I have two doubts about the paper, which might require more discussion in the paper (or I didn't understand something):
1. Riemannian geometry machinery requires a smooth (i.e., infinitely differentiable) manifold. We know that the activation functions used in typical neural network models are not smooth. Hence, the decision boundaries will not be smooth either (but will be locally smooth). How does that affect the results? Is the local curvature representable of the large-scale geometry of the differentiable manifold if the manifold is not smooth? A zig-zagging line is locally linear almost everywhere, but can curve in space as much as it wants. If we sample its local curvature randomly, we will "see it" as a straight line.
2. There is a lot of space in 50 dimensions. Is sampling e.g. 100 random vectors in a 50-dimensional adversarial subspace going to give us a representative sample? Have the authors tried to repeat the analyses with different random seeds and see if the empirical results are stable?

The most important doubt is 1. I think it invalidates the results of the curvature analysis, viz the example of the zig-zag line. Since this analysis is a major selling point of the paper, I cannot recommend accepting it at this stage.

Some other comments:
1. I think the paper "Visualizations of Decision Regions in the Presence of Adversarial Examples" (Swirszcz, O'Donoghue and Kohli) should also be referenced as it is related work.
2. The first sentence in the caption for Figure 1 is hard to understand.
3. In the last sentence of the caption for Figure 2, what is "even subsampling"?
4. What is meant by "well approximated" in the discussion of the modification to the optimisation problem described by Eqn. (1)? Can the Authors provide some proof that this approximation works?
5. I think the folk lore is that in order to prevent the perturbations from being visible to human viewer, it's better to cap their L_inf norm, not L_2. Something to consider.

==== EDIT AFTER AUTHOR REPLY ====

The Authors have completely clarified Q2 (representative samples) to my satisfaction. Re Q1 (smoothness), they explained that they use a differentiable (though not smooth) ELU activation function, which alleviates my prior concerns, but raises the question of the applicability of their results to the more common case of ANNs using non-differentiable activation functions like RELU. However, this may be seen as an open question to be explored in future work. (It would also be nice if the Authors reviewed prior work for curvature analysis and commented on whether it also avoided the problem of using non-differentiable network architectures).
summary_of_the_review: I think that the paper is well-written, continues an interesting line of research and improves on it. The Authors clarified the technical questions. While I'm not sure how applicable the results are to common ANN architectures (because of the smoothness requirements), I recommend acceptance.
review_title: null

Meta-review: The work investigates the decision boundary of neural networks by quantifying in various ways the shape and curvature of the error set local to correctly classified inputs, dubbed the "adversarial subspace". First, a method is introduced which seeks to find the largest set of orthogonal directions starting at in input x which will all intersect the error set local to an image. This is motivated as a certain geometric measure of the error set, large sprawling error sets may have many orthogonal directions which intersect it local to the given input while small narrow error sets may have relatively few. Using this geometric measure, the authors compare the shape of adversarial subspaces of various image models both with and without adversarial training. After the rebuttal period, reviewers all felt that the work was borderline, with no one strongly advocating for the work. As noted by some reviewers, while some experiments may be interesting it was unclear what new insights the work contributes. For example, the authors argue that the change in the geometry of the error set explains why adversarial training works. It is unclear how this is an explanation more than it is simply an observation that the error set geometry has changed. An analogy would be trying to explain why Resnet-50 performs better than AlexNet by showing that it has higher test accuracy---this only shows that it is better, but doesn't explain why.

During the discussion period the AC raised additional concerns regarding a sanity check that the author's main algorithm should pass. In particular, consider an error set x_1 >= K(x_2^2 + ... + x_n^2) + C, parameterized by constants K and C > 0. For all choices of K and C and starting point x = (0, ..., 0), the authors main algorithm will always return 1 as the dimensionality of this error set. It will find the vector (1, 0, ..., 0) and then terminate. However, this is problematic because we can choose K and C to make this error set either very narrow (e.g. K=C=100) or very wide (K = .000001, C = .00001)---the proposed algorithm will be unable to distinguish between this two extremes. Given this, it seems that greedily selecting the set of orthogonal directions starting at x can be very suboptimal if the intent is to find a maximum sized set of orthogonal directions. 

To conclude, the work would be substantially improved if it addresses two major weaknesses. First, there needs to be a clearer motivation for studying this notion of geometry of the error set, what new insights can the authors provide other than adversarial training changes the shape of the error set? Second, the method doesn't seem to be principled given it is unable to distinguish between the two extreme cases discussed above.
Decision: Reject