Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper argues that adversarial robustness might be at odds with robustness with natural distribution shifts. To that end, the authors first illustrate this intuition on a synthetic setting consisting of a linear regression model trained on Gaussian data with controlled spurious correlations. In this setup, they show numerically that the adversarially trained models tend to rely more on the spurious correlations than their standard couunterparts, and have worse robustness agains certain distributions shifts. Afterwards, the paper presents several evaluations of an adversarially trained model suite on multiple distribution shift benchmarks, including real-world ones and synthetic ones. The experiments on these model seem to agree with the numerical simulations presented in the prior section, i.e, adverarially trained models rely more on certain spurious correlations.
strengths_and_weaknesses: ## Strengths

1. **Strong and clear message**: I really appreciate the core message of the paper which tries to show there is an odd and counterintuitive relation between $\ell_p$ adversarial robustness and reliance on spurious correlations. This is a valuable insight for the community, as it reminds us of the bizarre dynamics that $\ell_p$ adversarial training has on the feature extraction power of neural networks.
2. **Good balance between synthetic experiments and practical evaluations**: Overall, I find the numerical simulations on the synthetic data presented on Fig. 2 and Fig. 3 quite interesting, and they nicely complement the main observations provided on ObjectNet, RIVAL10, ImageNet-9, and Waterbirds. 

## Weaknesses

1. **Novelty**: My main concern is with regards to the added value that these observations provide to the community given what is already known from prior literature. In particular, as described in the introduction, both [38] and [55] had already observed a stronger reliance on background spurious features for adversarially trained models, which means that the main new insight from this work is the observation that the same increase in spurious correlation also happens in ObjectNet, ImageNet-9 and Waterbirds.
2. **Some hand-wavy arguments**: In relation to the previous point, I understand that the main objective of this work is therefore to provide some insights on why adversarially trained models are less robust to distribution shifts. In this regard, although I find the numerical simulations in Sec. 3 quite interesting, I do not think they can answer the question of why the same effect happens on real data. All in all, the spurious correlations this work alludes to are mostly semantic, e.g., backgrounds, or object positionings, and they are hard to map to the geometric concepts in Sec. 3. This makes some of the arguments regarding the "strength of different spurious features", or the reasons why we do not see the same effects on ImageNet-C or for $\ell_\infty$ models, quite hand-wavy as they are not well-defined, nor rooted on any quantifiable evidence.
3. **Clarity (minor)**: Some passages in this manuscript feel a bit rushed and are hard to read, e.g., the presentation of the experiments leading to Fig. 4. The writing, in generall, could be more polished, and the paper be better formatted. In this sense, I believe that making the presentation of Fig. 5, Fig. 7 and Fig. 8 consistent with be nice.
questions: I have my concerns accepting the arguments in Sec. 4.4. In particular, the authors claim "[...] this experiment shows that exclusively linking spurious correlations to adversarial vulnerability is a fallacy". However, I do not think this experiment shows that. At most, it vaguely discards that certain color spurious correlations are the root cause of adversarial vulnerabilities, but there could exist thousand other types of spurious correlations that could explain the existence of adversarial examples. Bearing this in mind, I would appreciate a further discussion from the authors on this particular point in which they argued why their work demonstrates that adversarial examples cannot be due to any form of spurious correlation.

I would also appreciate if the authors could further comment on the novel aspects of their work on top of the observations in [38] and [55].
limitations: I see no clear negative societal impact stemming directly from this work. Furthemore, the authors provide a short societal impact statement in the appendix.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper offers an analysis of the explicit tradeoffs between adversarial robustness and natural distributional robustness. The analysis well explains the phenomenon that adversarial training sometimes increases model sensitivity to spurious features in experiments. The reason is that adversarial training may actually increase model reliance on spurious features.
strengths_and_weaknesses: **Pros:**
- This paper is the first to theoretically show that adversarial training may increase model sensitivity to spurious features, though some researchers implicitly supposed that adversarial training would always decrease model reliance on spurious features.
- A corollary of the analysis is that adversarial training may harm model robustness to the distribution shifts that break the spurious correlations. This result is significant and thought-provoking.

**Cons:**
- Section 3 only gives one analytical result showing a simplified version of the adversarial loss. This result is good, which indeed implies that the optimal $\theta$ may be non-zero on the set of spurious features. However, the theoretical analysis fails to show how much adversarial training can increase model reliance on spurious features. The paper then relies on verbal reasoning to reach the remaining theoretical results, without further formulating clearer theorems or corollaries. Thus, the so-called "explicit" tradeoffs between adversarial and distributional robustness are not so obvious. 
questions: - Adversarial training has been extended to improve model robustness beyond $L_p$ balls, such as [1][2]. It is then natural to wonder whether a method like [1] can increase model reliance on spurious features and thus decrease distributional robustness. After all, the methods like [1][2] are somewhat designed to improve distribution robustness.

[1] Laidlaw, Cassidy, Sahil Singla, and Soheil Feizi. "Perceptual Adversarial Robustness: Defense Against Unseen Threat Models." International Conference on Learning Representations. 2020.  
[2] Lin, Wei-An, et al. "Dual manifold adversarial robustness: Defense against lp and non-lp adversarial attacks." Advances in Neural Information Processing Systems 33 (2020): 3487-3498.
limitations: See above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors claim that the adversarial training encourages to model to reply more on spurious features, which hurts the robustness under distributional shift. To support their claim, they proposed a simplified setting: a linear model with some standard core and spurious features. They found the adversarial training increased the model weights on spurious features. Finally, a few synthetic/real datasets are studied to verify the hypothesis empirically.

strengths_and_weaknesses: Overall, the paper made a counter-intuitive claim that adversarial robustness hurts distributional robustness; and this is due to the shifts in weights from the core feature set to the spurious feature set. The theoretical analysis, although seems a bit over-simplified, correlates well with almost all the empirical results. The experimentations are well designed and convincing, and I have no difficulty to follow.

questions: First, although the dichotomy of “core” features and “spurious” features are mentioned many times in previous works, it would be great to have some formal definitions in theory and some examples. In image recognition, is it true that “foreground” means “core features” and “background” means “spurious features”?

Secondly, I find the definition of “distribution shift” in this paper a bit restrictive in that it only changes the “spurious features” (L210 - L213). But in reality, both feature sets will be affected.

Thirdly, the results under the Linf constraint seem to be inconsistent: sometimes it behaves just like L2 constraint, otherwise, it behaves as the theory implies. For example, in Figure 5 the OOD accuracy drops as quickly as L2 constraint; while Figure 6 shows the RCS doesn’t change.

Finally, if we think the other way round: suppose we train the classifier with a distributionally robust optimizer. Then according to this paper, the model should only use the core features, thus even more susceptible to adversarial examples than the original setting. But I don’t think it will be observed in real experiments. 

limitations: Please refer to the section above.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors investigate the connection between adversarial robustness and reliance on spurious features.  They demonstrate that adversarial training with respect to Lp-based threat models increases reliance on spurious features.  They provide theoretical analysis of a simple linear regression setting and provide experimental results on image datasets to demonstrate that adversarially robust models have higher reliance on spurious features, leading to lower natural distributional robustness.
strengths_and_weaknesses: Strengths:
- connects 2 directions of research (natural distributional robustness and adversarial robustness) which to the best of my knowledge has not been studied in depth in prior work
- experiments are in depth

Weaknesses:
- While the authors claim that adversarial robustness increases reliance on spurious features and frame this as a surprising result, I'm actually not surprised by this.  I think it really depends on what spurious features are being considered.  Adversarial training increases robustness against certain spurious features specifically small high frequency changes (ie. single pixel change), but for other existing spurious features which are considered in this paper, I do not see why adversarial training would suddenly prioritize using core features over the many other spurious features (ie background and color correlations) which may be easier to learn.  Because of this, I'm not very convinced about the significance of this work.
- Given that generally we are interested in classification problems in adversarial ML, I would have been more excited to see theoretical results for a binary classification problem rather than regression problem, but this is a minor point
- mathematical typos in the proof sketch in the main text and proof in the appendix make proof hard to follow (see questions).
questions: - One point of confusion I had was that the authors state that Theorem 1 implies that adversarial training directs the model towards using spurious features (lines 154-156).  How exactly Theorem 1 implies this is unclear to me, could the authors clarify this point?
- In equation 5, what is P?
- In the toy experiments in section 3, wouldn't the dependence on core vs spurious features for an adversarially robust model depend on the number of features that a considered core vs spurious?  In the toy example, there are 2 core features vs 3 spurious features.  I think that in the setting where there are more core compared to spurious features, then we would not see the jump in spurious feature reliance as we do in the toy examples.
- Following the proof, shouldn't equation (3) be $\text{max}_{||\delta||\le \epsilon} (Y - \langle X + \delta, \theta \rangle)^2 = (|Y - \langle X, \theta \rangle| + \epsilon ||\theta||_q)^2$?
- In the proof of Theorem 1, in proving equation 3, it states that "with a suitable choice of $\theta$" the equality (b) holds, but the statement in equation 3 should hold for any $\theta$ when it is used when rewriting equation 1, since loss is evaluated for any given $\theta$, which may not be the specific value of $\theta$ such that exact equality holds.  Should this be "with a suitable choice of $\delta$ instead?  I think this also goes for the equality of the triangle inequality portion too, since we are choosing the max over $\delta$, we would choose $\delta$ to achieve the upper bound rather than putting restrictions on $\theta$.
- Also in the sketch proof of Theorem 1 in main text, in the derivation after line 159, I believe that you are missing a $||\theta||_q$: ie) in the 2nd line the last term should be $2 \epsilon ||\theta||_q \mathbb{E}[|Y - \langle X, \theta \rangle|]$, but the $||\theta||_q$ is missing.  This error is also present in the proof in the appendix and carries down for many steps of the proof until the very last step in the proof of equation (2).
- in after 576 of the Appendix, in the equality marked by (a), the last term is missing $c_1$.  This also carries down to the next step in the proof of equation (2).
limitations: n/a
review_title: null

Meta-review: metareview: All reviewers recommend accepting the paper, so I will follow their suggestion - congratulations!

However, I should add that I do not find the arguments in the paper fully convincing. Earlier work has shown that adversarially trained models behave like "standard" ImageNet models (i.e., models trained without a robustness-enhancing method) on ObjectNet, which is one of the datasets in this paper - see https://arxiv.org/abs/2007.00644 . In these results, there is no trade-off between adversarial robustness and robustness on ObjectNet (on the other hand, adversarial robustness also doesn't help on ObjectNet). I encourage the authors to engage with the prior work on out-of-distribution generalization more deeply because this will help the reader understand how the new results here relate to other findings in out-of-distribution generalization.
confidence: Certain
Decision: Accept