Explicit Tradeoffs between Adversarial and Natural Distributional Robustness
Authors: Mazda Moayeri
Kiarash Banihashem
Soheil Feizi
Venue: NeurIPS 2022 Accept (2022)
Keywords: robustness
adversarial
distributional
spurious correlations

Abstract: Several existing works study either adversarial or natural distributional robustness of deep neural networks separately. In practice, however, models need to enjoy both types of robustness to ensure reliability. In this work, we bridge this gap and show that in fact, {\it explicit tradeoffs} exist between adversarial and natural distributional robustness. We first consider a simple linear regression setting on Gaussian data with disjoint sets of \emph{core} and \emph{spurious} features. In this setting, through theoretical and empirical analysis, we show that (i) adversarial training with $\ell_1$ and $\ell_2$ norms increases the model reliance on spurious features; (ii) For $\ell_\infty$ adversarial training, spurious reliance only occurs when the scale of the spurious features is larger than that of the core features; (iii) 
adversarial training can have {\it an unintended consequence} in reducing distributional robustness, specifically when spurious correlations are changed in the new test domain. Next, we present extensive empirical evidence, using a test suite of twenty adversarially trained models evaluated on five benchmark datasets (ObjectNet, RIVAL10, Salient ImageNet-1M, ImageNet-9, Waterbirds), that adversarially trained classifiers rely on backgrounds more than their standardly trained counterparts, validating our theoretical results. We also show that spurious correlations in training data (when preserved in the test domain) can {\it improve} adversarial robustness, revealing that previous claims that adversarial vulnerability is rooted in spurious correlations are incomplete.

Explicit Tradeoffs between Adversarial and Natural Distributional Robustness
Mazda Moayeri mmoayeri @umd. edu
Kiarash Banihashem ki ciarash@una edu
Soheil Feizi sfeiziocs, umd. edu
Department of Computer Science University of Maryland
Abstract
Several existing works study either adversarial or natural distributional robustness of deep neural networks separately. In practice, however, models need enjoy both types of robustness to ensure reliability. In this work, we bridge this gap and show that in fact, explicit tradeoffs exist between adversarial and natural distributional robustness. We first consider a simple linear regression setting on Gaussian data with disjoint sets of core and spurious features. In this setting, through theoretical and empirical analysis, we show that (i) adversarial training with l and &2 norms increases the model reliance spurious features; (ii) For foo adversarial training, spurious reliance only occurs when the scale of the spurious features is larger than that of the core features; (iii) adversarial training can have an unintended consequence in reducing distributional robustness, specifically when spurious correlations are changed in the new test domain. Next, we present extensive empirical evidence, using a test suite of twenty adversarially trained models evaluated on five benchmark datasets (ObjectNet, RIVALIO. Salient ImageNet- IM, ImageNet-9, Waterbirds), that ad adversarially trained classifiers rely on backgrounds more than their standardly trained counterparts, validating our theoretical results. We also show that spurious correlations in training data (when preserved in the test domain) can improve adversarial robustness, revealing that previous claims that adversarial vulnerability is rooted in spurious correlations are incomplete.
1 Introduction
Despite continuously improving upon state of the art accuracy various benchmarks, deep image classifiers remain brittle distribution shifts, suffering massive performance drops when evaluated on non-i.i.d. data. For example, the accuracy of object detectors trained on ImageNet (12] reduces by 40-45% on ObjectNet [5], where images are taken within houscholds at various viewpoints and rotations. The reliance of deep models spurious features, which correlate with class labels but irrelevant to the true labeling function BC), one cause of poor model robustness, as performance degrades when spurious correlations are broken. Indeed, model reliance spurious features like texture [17] and background [69 [ is well documented. Of greater concern, deep models in safety critical applications such detection of pneumonia
722 and COVID-19 19 [I]] have been observed to rely on hospital specific spurious markers, causing poor generalization to new hospitals.
pose another troubling distribution shift, where imperceptible input perturbations can cause model accuracy to drop to zero. Many works have been proposed to improve the adversarial robustness of deep models 3b 74
Adversarial examples 663 [8]
52
59
37
10
66 (55) including the widely
popular adversarial training, where inputs are augmented via adversarial attack during training
40
While spurious correlation robustness has also attracted lots of attention
2
30
39
550
the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Feature Sensitivity
Spurious Feature Sensitivity
Distributional Robustness
RCS (Salient ImageNett
Backoround Gap (IN-3)
ObjectNet: ImagoNet AC
RFS (RIVAL1O)
Ge
0.43
0.00
0.30
0.20
0.00
No AT
1. AT (E 4/255)
AT 3)
Figure 1: Snapshot of empirical evidence using R/VALJO, Salient ImageNet- ImageNer-9, Wa- terbirds, and ObjectNer benchmarks. Results averaged over ResNet1 and ResNet50. Adversarial training, especially under & norm, reduces (increases) sensitivity to core (spurious) features. The increased reliance spurious features leads to worse distributional robustness.
two problems are most often considered independently, despite both being essential to the safe and reliable deployment of deep models the wild.
In the few works that do consider adversarial and spurious correlation robustness in tandem, the prevailing argument that the origin of adversarial vulnerability is that the model focuses on spurious correlations that can be manipulated [77 73. 20 However, recently. noise-basee analyses on
RIVALIO 4 and Salient ImageNet- IM datasets suggest adversarial training may actually increase model sensitivity to spurious features; a result that is both counter-intuitive and in direct contrast existing ideology.
To better understand this observation, we first appeal to a simple linear regression problem on Gaussian data with disentangled and spurious features. In this setting, theoretically show
Adversarial training under &g and & norms increase model reliance on spurious features, as using spurious features forces an attacker to spread its budget over additional features.
For (x adversarial training, increased spurious feature reliance only occurs when the scale of the spurious feature larger than that of the core features. That is, spurious features are used when perturbations that corrupt core features
small to disrupt spurious correlations.
Due increased spurious feature reliance, there is explicit tradeoff wadeoffbettween between adversarial and distributional robustness. Specifically, we show that adversarial training decreases model robustness distribution shifts in the test domain where spurious correlations are broken.
To validate our theory, we evaluate twenty models adversarially trained using & and for projected gra- dient descent [51]]. Specifically, we inspect performance multiple spurious robustness benchmarks over synthetic (ImageNet-9 [69] Waterbirds (50) and real (RIVAL10 E22,, Salient ImageNet- IM 61, ObjectNet (5) datasets. Figure summarizes our experiments, where find that adversarially trained models consistently show greater sensitivity to spurious features compared to standardly trained baselines, with the effect more dramatic for & adversarial training than Ex Finally, we show that the presence of spurious correlations in training data (when preserved in test domain) can improve adversarial robustness, with stronger spurious correlations leading greater accuracy under attack.
Our work combines two prevalent but often separately considered notions of robustness, yielding surprising Aecoretically-derivee and empirically-sspporteee results. We hope our contributions grant insight to both adversarial and distributional robustness communities, and emphasize the need for holistic evaluations of model robustness.
2 Review of Literature
Adversarial Robustness. Since adversarial examples were first observed in deep models
63
18
the
phenomenon has been extensively studied. New attacks [8] 35
4K [5] and defenses [36
14
3N
High HCS for foo AT models is due to reduced seale of contextual bias in Salient ImageNet since the data
diversity weakens background correlations. In E and E, adversarial training. models rely on spurious features regardless of their scale. See details in Section

---

are introduced frequently, in game of cat and mouse where the attacker generally has the upper hand
3.
Certified defenses seek to break this cycle by offering provable robustness guarantees
Arguably the most popular defense is adversarial training [40] where images are augmented with adversarial perturbations during training, amounting to a min max optimization.
59
37
52
T0
Natural Distributional Robustness. In contrast to synthetic adversarial perturbations, many works seek characterize the robustness of deep models to naturally occurring distribution shifts, for instance due to common corruptions (noise, blur, etc) [2]] or changes in rendition [2] [- [3]
compiles ten benchmarks of realistic distribution shifts over diverse applications (medical, economic,
etc). Many algorithms have been proposed to improve out distributtion robustness [39 50 2 34],
though in comprehensive evaluations, their gains over empirical risk minimization are marginal. as they often only hold for certain distribution shifts [7] 21. 7ZT identifies diversity and correlation shifts as two key dimensions to OOD robustness; our work focuses on the latter.
Spurious Correlations. Solely optimizing for accuracy leads deep models to rely on any patterns pre- dictive of class in the training domain. This includes spurious features, which are irrelevant to the true labeling function. Natural image datasets are riddled with spurious features 33 60 reliance becomes problematic under distribution shifts that break their correlation to class labels: sidewalk segmentation struggles in the absence of cars [57] familiar objects cannot be recognized in unfamiliar poses [ or uncommon settings [X 5. etc. A natural and ubiquitous spurious correlation in vision is image backgrounds, observed in numerous prior works to be leveraged by models for classification
Spurious feature
69 42
and object detection 48 Spurious correlations also relate to algorithmic
biases
1 177
with implications for fairness 19 f
29
reflecting the importance of this issue.
Accordingly, many works seek improve spurious correlation robustness. Families of approaches include optimizing for worst group accuracy (50
learning invariant latent spaces 47 2. appealing to meta-learning 46 or causality 45 Our work does not focus on mitigation methods, but instead sheds insight on how optimizing for a different notion of robustness (i.e. adver- sarial) affects spurious feature reliance, and consequently, natural distributional robustness. Generally, models trained under ERM are believed to have a propensity to use spurious features, especially when they are easy to learn, due to bias of learners (algorithmic and human) to absorb simple features first (566 and take shortcuts [T However, recent work suggests that core features are still Icarned under ERM even when spurious features are favored, and simple finetuning on data without the spurious feature can efficiently reduce spurious feature reliance without full model retraining 30
24
49
43
Unintended Outcomes of Adversarial Training. Adversarial training achieves improved accuracy under attack, but comes at the cost of standard accuracy, with multiple works provably demonstrating Notably, 277 inspires our theoretical analysis, though we focus on the effect of adversarial training on out-of -atto--distribution (OOD) robustness to spurious correlation shifts, rather than its effect on standard accuracy. A more positive outcome is that adversarial training leads to perceptually aligned gradients [54], with applications to model debugging [62) 57], and further, transfer learning on adversarially robust features (despite being less accurate on the original task) yields better accuracy on downstream tasks compared to features learned from standard training [51]
this tradeoff 74
14 [3
To our knowledge, robustness to adversarial and natural distribution shifts have not been studied in tandem. However, spurious correlations are at times mentioned with adversarial robustness, usually in claims that the origin of adversarial vulnerability is in model's focus on (imperceptible) Our results create tension with the contrapositive of their argument, as we show that mitigating adversarial vulnerability (via adversarial training) results in increased spurious feature reliance. We do this analytically in a simple linear regression setting (Section B), and empirically on multiple benchmarks, with an emphasis on natural spurious features (i.e.. backgrounds) in our experiments (Section B. Further, we even demonstrate a case where the presence of spurious feature leads to improved adversarial robustness (Section 4.3) ). We note that the spurious features we observe to be positively associated with adversarial robustness may be distinct from those that prior works claim contribute to adversarial vulnerability. However, our result of adversarial training leading to increased spurious feature reliance (of any kind) is novel and contrary to common understanding. Given the critical nature of adversarial and spurious correlation robustness for model security, reliability, and fairness, the significance of our result in revealing potential misconceptions on the interplay of these two crucial modes of robustness should not be understated.
spurious features [77 [3 26
42 and [1l recently observed decreased core sensitivity on a handful of E2 adversarially trained models, in spirit with our findings, but with no explanation. We offer the first rigorous analysis
3

---

of this counterintuitive phenomenon, evaluating 5 to 10 times as many models in 5 times as many settings. More importantly, we theoretically prove that adversarial training increases spurious feature reliance, contributing novel fundamental insight as to how optimizing for adversarial robustness can lead to reduced robustness to natural distribution shits, uncovering important effects like the norm of adversarial training and the scale of spurious features at play.
3 Theoretical Analysis on Linear models
We begin by analysing the effects of adversarial training on simple linear regression model. Consider the model Y = [X,,00s, + W where X E RT is the input variable, popt E Rm is the optimal parameter, (.. represents the inner product and W E R noise variable. We assume that the input variables follow a multivariate Gaussian distribution ((,, Σ) where I E Rmxm is the covariance matrix and further assume that W is sampled from the Gaussian distribution N N(0, ow)). We assume that the set of features [m]] consists of two groups, the core features C' and the spurious features S. Without loss of generality, we assume that = (I,...,)) and S = {p + I..... mj. We assume that the optimal parameter 8eM has non-zero entries the set C' only. This implies that the output depends on the imput only through the core features and conditioned on the core features, is independent of the spurious ones. More formally, represent the core and spurious subsets of the input, respectively.
assume that Y - Xs X where Xs and Xc
For loss functions, we define the standard loss function as I(Ø) = E [(v - (x, and the adversarial loss function as
(p) = E
max (Y (X (X 8,0)) |1 Se
where p is the attack norm and represents the norm budget.
We first show an equivalent form of ① that is more amenable to analysis. Theorem 1. Assume that Y (X, Gam) + W where W (O, oih) is independent of X and
gro E mM is a fixed parameter. Assume further that X follows the distribution [0, E) and define o as (0- BoriyT E10 (0pl) + oz. The loss function ① is equivalent
(0) = og + (ci0s e- 11011,)2
(2)
where c = < and Il-ll, is the dual norm of 11-11pp i.e, =. Furthermore,
the above formulation is convex in
The above result is similar to Proposition 3.2 in 2 which provides characterization results for the l2 norm. The key distinction of our results is providing a simple convex formulation of the robust minimization problem, allowing the results to be easily generalized for an arbitrary lp norm. The theorem shows that the optimal value 0 minimizing the adversarial loss (f) not fop and in general, may be non-zero on the set of spurious features S. This means that adversarial training directs the model towards using the spurious correlations in order to increase robustness.
The proof of the theorem is provided in the Appendix. The main structure of the proof is similar to that of We first show that the inner maximization problem of |I can be rewritten as
max (Y (X + 8,0))² (Ix (X, 0) +e +01++))))))))
(3)
This allows us to rewrite M as
Lp.e- = E (Ix
+e (10111101)))
(X, E - 09)}] + ||01 1 + 2 |101| E ||l - (X,
E [(X,6 0 g + w) + 2 1101 +2 ||0||, و E [[ 0 gore) wl].
where for (a) we have used the fact that Y = (X, 0, + +W Using the fact that X and W are Gaussian, show that this is equal to of + +² |0| 2 c1 ||0| og which can further simplify to

---

2 with some algebraic manipulation. Next, using standard techniques for analysing composition of convex functions, we show that og and (c100 + ||0||g )² are both convex in 9, implying that p. (0) convex in as well.
Using the convex formulation in Theorem L we evaluate the linear model for different values of parameters to better understand the reliance of the model on spurious features. In our experiments, we consider a simple model with 5 features, the first two of which are core features. We let be a parameter controlling the correlation degree between the core and spurious features, with larger values corresponding to higher correlation. For the distribution of the data, define the matrix Q and Q matrices as
11
Qis
=
0
0
Qis =
(4)
VEi Ri,y'
7 77
0
0
1
Note that Q is obtained by normalizing the rows of Q. Each row of the Q matrix corresponds to an input feature. We let E take the value QQ". This is equivalent to sampling X from the distribution QN I). Throughout our experiments, we set dw 0.1.
For an arbitrary vector 0, we define its Norm Fraction over Spurious features NFS(0) as
NFS Eies 82
(5)
Σ,0g
Intuitively, NFS measures the degree to which a model relies on spurious features. For our first experiment, we perform adversarial training with varying parameter to obtain predictor 0 and evaluate its NFS. We consider different choices of the Ep norm as well as the correlation parameter The results can be seen in Figure 2
g-0.00
1-0225
5-0.00
1=0,00
g-0.25
3-0.50
gp0.00 20
g-4 25
o=0
0.6
0.6
0.6
NES
NES
0.2
0.2
0.0 1
0.0
0.0
0
2
0.0
0.5
1.0
used in adversarial training
used in adversarial training
used adversarial training
(a) & norm
(b) Eg norm
(c) foo norm
Figure 2: Reliance of the adversarially trained model on spurious features as measured by NFS (see Equation 5 for different choices of Ep norm and different values of adversarial budget and spurious correlation parameter 17.
As seen in Figure for the é and & norms, increasing the adversary's 's budget causes the model to rely more on the spurious features. To understand why this happens, is helpful to consider a game theoretic perspective: If the model only looks at the core features, then the adversary will only need to perturb these features. Thus, even though the spurious features are normally less suitable for prediction, they now have the advantage of being less perturbed. Assuming is large enough, the model will be better off looking at these features in forming its prediction. Of course, if the model only looks at the spurious features, core features will become even more informative as they would be unperturbed as well. In the game's 's equilibrium, the model would use both the core and spurious features, relying more on the spurious features with increased values of
Interestingly, this reasoning does not always apply for the lox. norm. Indeed, if the model were to only look at the core features, the adversary may still perturb the spurious features with no extra cost. This is because the fo norm only measures the maximum perturbation in each feature and as long as the perturbations on the spurious features are not larger than the perturbations on the core features,
5

---

the norm would not change. The results of Figure 2c support this as the adversary does not rely on the spurious features for its predictions.
Importantly, however, in some cases the adversary may require a larger budget to perturb the spu- rious features compared core features. Indeed, if we were scale feature by multiplication with a large number, then the adversary would require a larger budget to perturb that feature as the budget for that feature has effectively decreased. We can therefore expect that for the foo norm, scaling up the spurious features would cause the model to rely on them for prediction. Figure 3 shows that this is indeed the case. The figure shows the norm fraction over spurious (NFS) after scaling the spurious features for different values of c. The scaling done by mul- tiplying the rows of Q corresponding to the spurious features by scaling parameter. As seen in Figure 3 larger values of the scaling parameter, well as larger values of c, cause the model to become more reliant on the spurious features.
0.8
in ndversarial taining
0.8
0,94
0.6
0.4
1.15 1.08
Next, we measure the effect of using spurious features on the model's distributional robustness. To do this, we train two adversarial models. The first model, which we denote by "core", uses only the core features while the second model, denoted by "total", uses all of the features. We then simulate a distribution shift that breaks the spurious correlations by adding random Gaussian noise to the spurious rows of matrix Q defined in { Specifically, for each entry of Q in a spurious row, we add a noise sampled from (00 o2) where aQ is a noise parameter. We use a fixed value of rj = 0.25 and vary the parameter e, the norm p used in adversarial training as well as the variance of the Gaussian noise. The results are shown in Figure
0.2
0.0
1.0 2.5 4.0 5.5 7.0 8.5 10.0
Scale of the sparious feautres
Figure 3: Norm fraction over spuri- ous features (NFS) with an &oo con- strained adversary function of the scale of the spurious features and perturbation budget e.
.00 laa
13, und
sotal
esl 30, core
.80 cor
e=1.00 com
35, c00
1:100 cor
es
Loss after distriuution shift shiff
Loos ofter dlsmchuttoon shift
Loos allerr dlatciouii
1.0
1.5
0.8
1.6
1.0
0.6
1.4
0.5
0.4
0.2
1.2
0.3
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Neist
Noise punimeter
Noise pormeter
(a) la norm
(b) 82 norm
(c) foo norm
Figure 4: Effect of reliance on spurious features on distributional robustness. Each figures compares two models, one using only the core features, and another using all of the features (denoted by "core" and "total" respectively). For the E and Eg norms, the scale of the spurious features is 1 while for the Eo norm, the scale is
We observe that the clean (i.e. in distribution; oq 0) loss of the "core" model may be higher or lower than that of the "total" model, but in both cases, the total model is consistently more vulnerable distributional shifts resulted from breaking spurious correlations.
Empirical Evidence
We now demonstrate increased spurious feature reliance in adversarially trained models over multiple benchmarks. We evaluate models on two backbones (ResNet18, ResNet50) adversarially trained on ImageNet 2 using two norms (f2, lox) under five attack budgets (denoted e) per norm, resulting in a 2 x 2 x 20 model test suite, as well as standardly trained baselines. See appendix for details.
4.1 AT hurts Natural Distributional Robustness only when Spurious Correlations are broken
We first show reduced distributional robustness of adversarial models occurs specifically in cases where natural spurious correlations are broken. We appeal to the ImageNet-C [22] and ObjectNet

---

ResNet18
ResNet50
I. AT No AT
35
AT
Out of Distribution Accuraty
25
Out of Diszribution Accuraty
30-
No AT
20
25-
20-
Imagetet.cco Objecttett
Objectfiet
15
ImageNettCC
10-
10-
10
50
10
40
70
Distribution Accuracy
Distribution Accuracy
Figure 5: OOD accuracy vs standard ImageNet accuracy for adversarially trained ResNets. ImageNet- accuracy is closely tied to standard accuracy, but for ObjectNet, where spurious correlations are broken, performance drop is more severe than a linear relation with standard accuracy would entail.
S OOD benchmarks. ImageNet-C augments ImageNet samples with common corruptions like noise or blurring, distorting both core and spurious features equally. Crucially, these corruptions do not break spurious correlations. On the other hand, ObjectNet formed by having workers capture images of common household objects (including samples from 113 classes of ImageNet) in their homes. Thus, only spurious features are affected. Namely, ObjectNet introduces distribution shifts in background, rotation, and viewpoint. We plot accuracies on these benchmarks in figure S
Recall that adversarially trained models have lower standard accuracy, which can confound analysis, so we compare the drop in OOD accuracy to the drop in ImageNet accuracy across our model suite. Observe that the ratio of ImageNet-C accuracy ImageNet accuracy is roughly constant across models. However, the ratio of ObjectNet accuracy to ImageNet accuracy is lower for adversarially trained models. Therefore, even after controlling for reduced standard accuracy, the distributional robustness of adversarially trained models is worse than that of standard models. Importantly, this effect does hold for distribution shifts that maintain spurious correlations, indicating that the reduced distributional robustness is due to increased spurious feature reliance.
4.2 Reduced Core Sensitivity, and Difference in the Effect of le & lo Adversarial Training
RIVALL0
Salient ImageNet-
Lo Setta
l.te 0225
Le Saza
fed Goa
Le Sunta
A7
Relative Sensivvity (RCS)
0.6
Relative Foreground Sensitvity
0.6
0.4
0.3
Reshet552
ResNe:50s
0.3
Rashet50s
ResNet18s
ResNet18s AT Bashet
2, Bestari1.
bute3
brread
bitw
for
Adversarial Training Attack Budget
Adversarial Training Attack Budget
Figure 6: Noise-based evaluation of model sensitivity to foreground (RFS on RIVAL10) core (RCS on Salient ImageNet- regions. Lower values entail greater sensitivity to spurious regions.
now directly quantify sensitivity to core features via RIVAL10 and Salient ImageNet- 1M datasets The premise of this analysis is that model sensitivity to an input region can be quantified introduced the noise-based metric relative foreground sensitivity (RFS), which is the gap between accuracy drops due to background
42 61
by the drop accuracy due to corrupting that region 6C

---

and foreground noise, normalized so to allow for comparisons across models with varying general noise robustness. RIVALIO object segmentations allow for RFS computation. Analogously, relative core sensitivity (RCS) is computed using Salient ImageNet- 1 I''s soft segmentations of core input regions. A key distinction between the two metrics that RCS is computed directly on pretrained models performing the original 1000-way ImageNet classification task, while RFS first requires models to be finetuned on the much coarser 10-way classification task of RIVALIO. Also, Salient ImageNet- IM includes all ImageNet images, while RIVALIO only consists of 20 ImageNet classes.
Figure 6 shows a decrease in RFS and RCS as the attack budget e seen during & adversarial training rises. Thus. adversarial training reduces core feature sensitivity relative to spurious feature sensitivity. Notably, this effect does not hold for models udversarially trained with attacks under the fx norm for RCS, though it does for RFS. Alluding to our theoretical result, we conjecture that in Salient ImageNet- IM, the scales the spurious features are much smaller than in RIVAL11, due to the diversity of images and finer grain of classes. That is, smaller perturbation is needed to alter spurious feature so that correlates with an incorrect class when there are 1000 classes than when there are only 10 classes with generally disparate backgrounds.
4.3 Adversarial Training Increases Background Reliance in Synthetic Datasets
Now, we take a closer look at the reliance of adversarially trained models on the contextual spurious feature of backgrounds via the synthetic datasets ImageNet-9 [69 and Waterbirds [50]. Both
datasets use segmentations to superimpose objects over varying backgrounds, detailed below.
Background Gap
Background Gap
Accuracy on IN-9 IN-9 Subset
122
Accuracy on It-99 Subset
E'
008
75
80
112
600-
70
75
0'00
1222
65
ppt
70
6'T
60
333
336
122.
Mixed Same
Mixed Same
60
50
Mixed Rand
Mixed Rand
No AT
5 25
4 ---1-
hE=3
6--5
====0.5
1.6-10
===
1. 1.E-4.0
1,5=0.5
bE=1
4.6 4E-3-
4==5
1.E= =.5
===
1.
1.
1.= 8.0
No
ResNet1B Adversarial Training Norm and Budget
ResNet50 Adversarial Training Norm and Budget
Figure 7: Background Gap (difference accuracies on ImageNet-9 subsets MIXED-SAB -AME and
MIXED-RAND), The drop in accuracy due to background cross-class swapping (MIXED-RANC -RAN) causes larger drops in accuracy for robust models, especially l adversarially trained models.
ImageNet-9 (IN-9) organizes a subset of ImageNet into nine superclasses. Multiple validation sets ex- ist for IN-9, where backgrounds foregrounds are altered; use LLXEE--AAME and MIXED-RAND. In both sets, original backgrounds are swapped out for new ones. Crucially, in MIXED-S AME, the new backgrounds taken from other instances within the same class, while MIXED- RAND uses random backgrounds. The metric, Background Gap, is the difference in model accuracy on MIXED-SAME and MIXED-RAND (i.e. drop due breaking spurious background correlation).
Figure shows that the background gap for our test suite twenty robust ResNets and two standardly trained baselines. Nine out of the ten Eg adversarially trained models have larger background gaps than the standard baselines, while the same true for five out the ten for adversarially trained models. When considering relative gaps (i.e. percent of the accuracy increase in gap becomes even more dramatic, with the & adversarially trained ResNet50 for e = having 18.9% relative drop, compared the 9.7% relative drop for the standardly trained ResNet50.
MIXED- -AME). the
Waterbirds combines foregrounds from Caltech Birds 65 and backgrounds from SUN Places 68
The task is binary classification of land birds and water birds. The majority group (95% of training

---

Waterbirds
Waterbirds
Validation Accuracy
85
EVV
TS1
65
9
166
152
142
Validation Accuracy
14
335
16.
292
145
204
090
85
1400
30
1S.
80
033
80
75
Majority
Majority
70
Minority
Minority
No AT
6 0.5
1E=3
-== 0.5
1-
===20
1_E 4.0
E.
No AT
b
4.8 f,5=0.5
4--3-
I. 5.
t==1.0
t-
E.
ResNet18 Adversarial Training
Budget
ResNet50 Adversarial Training and Budget
Figure 8: Accuracies on Waterbirds subsets where spurious correlation is intact (majority group: e.g.
land birds on land backgrounds) and where it is broken (minority group: e.g. land birds on water backgrounds). The drop in performance due to breaking the background correlation grows larger for robust models, especially & adversarially trained models.
samples) consists of land birds over land backgrounds and water birds over water backgrounds. The minority group breaks this spurious correlation, placing land birds over water backgrounds, and vice versa. The test set is evenly split between these groups. We train only final linear layer atop the frozen feature extractors (so that models remain adversarially robust) for each of our models on the Waterbirds training set for ten epochs, saving the model with highest validation accuracy.
Figure shows majority and minority group accuracies, and the gap between them, for our test suite of models. Again, we see increased gaps for robust models, with 100% of & and 60% of & models respectively having larger gaps than the standardly trained baseline on the corresponding backbone.
In both benchmarks, breaking the background spurious correlation causes a more significant drop in performance for udversarially trained models than standardly trained models, indicating that adversarial training led to increased reliance on backgrounds. The observed affect is stronger for ls adversarially trained models than loo ones. Further, the gaps grow near monotonically with
4.4 Reverse Effect: Presence of Spurious Correlations Can Improve Adversarial Robustness
Finally, we show evidence that is directly odds with the claim that spurious features lead to adversarial vulnerability. We train &s on CIFARIO [ with a spurious feature injected. Namely, images have all values in one color channel slightly increased. The majority group consists of red-shifted images from classes 0 4-4 and green shifted images from classes
Accuracy under Attack
0.8
p= 39:1
p= 19:1
0.6
p=
0.4
- 9, while the minority group has reverse color-shifts. The parameter is the ratio between majority and minority group size, controlling
0.2
strength the spurious correlation (higher means stronger spurious correlation; spurious feature has no predictive power). We then evaluate the accuracy of the trained model under adversarial attack L..d. test set (spurious feature retained. no distribution shift).
means the
PGD z. Attack Budget (c/255)
Figure 9: Injecting spurious feature improves adversarial robustness.
Figure 9 visualizes the results. Not surprisingly, the clean accu- racy is higher for models trained on data with higher p. as the spurious feature is more predictive for higher p. In fact, the added predictive influence the spurious feature leads to better accuracy under attack, with the gap between highest and lowest / values growing up to four fold compared to the base- line gap in clean accuracy. Thus, using spurious feature can improve adversarial robustness. Despite being on contrived example, this experiment shows that, while some spurious correlations may cause adversarial vulnerability, others do the opposite: the picture is more nuanced than previously assumed.

---

5 Acknowledgements
This project was supported in part by NSF CAREER AWARD 1942230, HROO1 I 19S0026 (GARD), ONR YIP award N00014-22-1-22711 Army Grant No. 2911NF212007,, the NSF award CCF2212458, an AWS Machine Learning Research Award.
References
[1] M. A. Alcom, Q. Li, Z. Gong, Wang, L. Mai, W.-S. Ku, and A. M. Nguyen. Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4840 4849, 2019.
[2] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization, 2020.
[3] A. Athalye, N. Carlini, and D. A. Wagner. Obfuscated gradients give a false sense of security: Cireumventing defenses adversarial examples. In ICML, 2018.
[4] B. Aubin, A. Slowik, M. Arjovsky, L. Bottou, and D. Lopez-Paz. Linear unit-tests for invariance discovery. ArXiv, abs/2102.10867, 2021.
[5] A. Barbu, D. Mayo, Alverio, W. Luo, C. Wang, D. Gutfreund, J. B. Tenenbaum, and B. Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurlPS, 2019.
[6] Beery, G. V. Horn, and P. Perona. Recognition in terra incognita. CoRR, abs/ 8078840777777 04975, 2018.
[7] J. Buolamwini and T. Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In FAT, 2018.
[8] N. Carlini and D. A. Wagner. Towards evaluating the robustness of neural networks. 2017 IEEE Symposium on Security and Privacy (SP), pages 39-57, 2017.
(9] A. Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5 2: 2:153-163, 2017.
[10] M. Cohen, Rosenfeld, and J. Z. Kolter. Certified adversarial robustness via randomized smoothing. In ICML, 2019.
[11] A. J. DeGrave, J. D. Janizek, and S.-I. Lee. Ai for radiographic covid- 19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7):610-615 2021.
[12] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[13] J. Djolonga. J. Yung, M. Tschannen, R. Romijnders, L. Beyer, A. Kolesnikov, J. Puigcerver, M. Minderer, A. D' Amour, D. Moldovan, S. Gelly, N. Houlsby, X. Zhai, and M. Lucic. On robustness and transferability of convolutional neural networks. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16453-16463, 2021.
[14] E. Dobriban, H. Hassani, D. Hong, and A. Robey, Provable tradeoff's in adversarially robust classification, ArXiv, abs/2006.05
2020.
[15] L. Engstrom, B. Tran, D. Tsipras, L. Schmidt, and A. Madry. Exploring the landscape of spatial robustness. In ICML, 2019.
[16] R. Geirhos, J. Jacobsen, C. Michaelis, R. S. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. CoRR, abs/2004.07780, 2020.
[17] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. Imagenet- trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness, 2018.
10

---

[18] I. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
[19] T. B. Hashimoto, M. Srivastava, H. Namkoong, and P. Liang. Fairness without demographies in repeated loss minimization. In ICML. 2018.
[20] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.
[21] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. L. Zhu, S. Parajuli, M. Guo, D. X. Song, J. Steinhardt, and Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8320-8329, 2021.
[22] D. Hendrycks and T. G. Dietterich. Benchmarking neural network robustness common corruptions and perturbations. ArXiv, abs/1903. 12261, 2019.
[23] A. M. Hosseini, A. M. Abouei, and M. H. Rohban. The interplay between distribution parameters and the accuracy-robusten tradeoff classification. arXiv preprint arXiv:2107.00247, 2021.
[24] W. Hu, G. Niu, I. Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? In ICML, 2018
[25] G. Huang, Z. Liu, and K. Q. Weinberger. Densely connected convolutional networks. 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 2261-2269, 2017.
[26] A. Ilyas, Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry. Adversarial examples are not bugs, they are features. In NeurlPS, 2019.
[27] A. Javanmard, M. Soltanolkotabi, and H. Hassani. Precise tradeoff's in adversarial training for linear regression. In Conference on Learning Theory, pages 2034-2078. PMLR, 2020.
[28] P. Kattakinda and S. Feizi. Focus: Familiar objects in common and uncommon settings. ArXiv, abs/2110.03880 2021.
[29] F. Khani and P. Liang. Removing spurious features can hurt accuracy and affect groups disproportionately. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021.
[30] Kirichenko, Izmailov, and A. G. Wilson. Last layer re-training is sufficient for robustness to spurious correlations, 2022.
[31] W. Koh, Sagawa, H. Marklund, M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Ya- sunaga, R. L. Phillips, S. Beery, J. Leskovec, A. Kundaje, E. Pierson. S. Levine, C. Finn, and Liang. Wilds: benchmark of in-the-wild distribution shifts. In ICML, 2021.
[32] A. Kolesnikov and C. H. Lampert. Improving annotation. ArXiv, 005508538, 2016.
object localization by micro-
[33] A. Krizhevsky, V. Nair, and G. Hinton. Cifar-10 (canadian institute for advanced research).
[34] D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, Binas, R. Priol, and A. Courville. Out-of-distribttion generalization via risk extrapolation (rex). In ICML, 2021.
[35] C. Laidlaw and S. Feizi. Functional adversarial attacks. In NeurlPS, 2019.
[36] C. Laidlaw, S. Singla, and S. Feizi. Perceptual adversarial robustness: Defense against unseen threat models. ArXiv, abs/2006. abs/2006.12655, 2021.
[37] A. Levine and Feizi. Improved, deterministic smoothing for certified robustness. ICMI., 2021.

---

[38] W.-A. Lin, C. P. Lau, A. Levine, R. Chellappa, and S. Feizi. Dual manifold adversarial robustness: Defense against ip and non-lp adversarial attacks. ArXiv, has/2009.024707 2020.
[39] E. Z. Liu, B. Haghgoo, A. S. Chen, A. Raghunathan, P. W. Koh, S. Sagawa, P. Liang, and C. Finn. Just train twice: Improving group robustness without training group information. In ICML, 2021.
[40] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018.
[41] M. Moayeri and S. Feizi. Sample efficient detection and classification of adversarial attacks via self-supervised embeddings. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7657-7666, 2021.
[42] M. Moayeri, P. Pope, Y. Balaji, and S. Feizi. A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes, 2022.
[43] S.-M. Moosavi-Dezfoolii, A. Fawzi, and P. Frossard. Deepfool: A simple and accurate method to fool deep neural networks. 2016 IEEE Conference Computer Vision and Paltern Recognition (CVPR), pages 2574-2582, 2016.
[44] N. Papernot, P. Mcdaniel, X. Wu, S. Jha, and A. Swami. Distillation as a defense adversarial perturbations against deep neural networks. 2016 IEEE Symposium Security and Privacy (SP), pages 582-597, 2016.
[45] Peters, P. Buhlmann, and N. Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 78, 2015.
[46] M. Ren, W. Zeng. B. Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. In ICML, 2018.
[47] A. Robey, G. J. Pappas, and H. Hassani. Model-based domain generalization. In NeurlPS, 2021.
[48] A. Rosenfeld, R. S. Zemel, J. K. Tsotsos. The elephant in the room. ArXiv, abs/a 033055 2018.
[49] S. Sagawa, W. Koh, T. B. Hashimoto, and Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ArXiv, 1.0073,, 2019.
[50] Sagawa*, P. W. Koh*, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks. In International Conference on Learning Representutions, 2020.
[51] H. Salman, A. Ilyas, L. Engstrom, A. Kapoor, and A. Madry. Do adversarially robust imagenet models transfer better? In ArXiv preprint arXiv:2007.08488) 2020.
[52] H. Salman, Yang, Li, Zhang, H. Zhang, I. P. Razenshteyn, and S. Bubeck. Provably robust deep learning via adversarially trained smoothed classifiers. ArXiv, abs/1906.04584, 2019.
[53] M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4510-4520, 2018
[54] S. Santurkar, A. Ilyas, D. Tsipras, L. Engstrom, B. Tran, and A. Madry. Image synthesis with a single (robust) classifier. In NewrIPS, 2019.
[55] A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. P. Dickerson, C. Studer, L. S. Davis, G. Taylor, and Goldstein. Adversarial training for free! In NeurIPS, 2019.
[56] H. Shah, K. Tamuly, A. Raghunathan, Jain, and Netrapalli. The pitfalls of simplicity bias in neural networks. ArXiv, abs/2006.07710, 2020.
12

---

[57] R. Shetty, B. Schiele, and M. Fritz. Not using the car to see the sidewalk - quantifying and controlling the effects of context in classification and segmentation. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8210-8218, 2019.
[58] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.15555 2015.
[59] S. Singla and S. Feizi. Second-order provable defenses against adversarial attacks. ArXiv, abs/2006.00731, 2020.
[60] S. Singla and S. Feizi. Salient imagenet: How to discover spurious features in deep learning?, 2021.
[61] S. Singla, M. Moayeri, and S. Feizi. Core risk minimization using salient imagenet, 2022.
[62] S. Singla, B. Nushi, S. Shah, E. Kamar, and E. Horvitz. Understanding failures of deep networks via robust feature extraction. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12848- 12857, 2021.
[63] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intrigu- ing properties of neural networks. In International Conference on Learning Representations, 2014.
[64] H. Wang. S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506- 10518, 2019.
[65] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Saltech-UCSI Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
[66] E. Wong, L. Rice, and J. Z. Kolter. Fast is better than free: Revisiting adversarial training. CoRR, 100/9000,,0994,, 2020.
[67] E. Wong, S. Santurkar, and A. Madry. Leveraging sparse linear layers for debuggable deep networks. ArXiv, abs/2 105.4105,,4857, 2021.
[68] Xiao, Hays, Ehinger, Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages -3922, 2010.
[69] K. Y. Xiao, L. Engstrom, A. Ilyas, and A. Madry, Noise signal: The role of image backgrounds object recognition. In International Conference on Leurning Representations, 2021.
[70] S. Xie, R. B. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5987-5997 2017.
[71] N. Ye, K. Li, H. Bai, R. Yu, L. Hong. F. Zhou, Z. Li, and J. Zhu. Ood-bench: Quantifying and understanding two dimensions of duitt--diittibutionn generalization. 2021.
[72] J. R. Zech, M. A. Badgeley, M. Liu, A. B. Costa, J. J. Titano, and E. K. Oermann. Variable generalization performance of deep learning model to detect pneumonia in chest radiographs: cross-sectional study. PLoS Medicine, 15, 2018.
[73] C. H. Zhang, K. Zhang, and Y. Li. A causal view on robustness of neural networks. ArXiv, abs/2005.0 000000 2020.
[74] H. R. Zhang, Y. Yu, J. Jiao, F. Xing, E. Ghaoui, and M. I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML., 2019.
[75] Zhang, Menon, A. Veit, S. Bhojanapalli, Kumar, and S. Sra. Coping with label shift via distributionally robust optimisation. ArXiv, abs/2010. 2021.
13

---

[76] X. Zhang, X. Zhou, M. Lin, and J. Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6848 2018.
[77] Y. Zhang, M. Gong, T. Liu, G. Niu, X. Tian, B. Han, B. Scholkopf, and K. Zhung. Causaladv: Adversarial robustness through the lens of causality. In ICLR, 2022.
14


Reviews:
[{"review_id": "piwVwJHVZXV", "reviewer": "Reviewer_R1sj", "summary": "This paper argues that adversarial robustness might be at odds with robustness with natural distribution shifts. To that end, the authors first illustrate this intuition on a synthetic setting consisting of a linear regression model trained on Gaussian data with controlled spurious correlations. In this setup, they show numerically that the adversarially trained models tend to rely more on the spurious correlations than their standard couunterparts, and have worse robustness agains certain distributions shifts. Afterwards, the paper presents several evaluations of an adversarially trained model suite on multiple distribution shift benchmarks, including real-world ones and synthetic ones. The experiments on these model seem to agree with the numerical simulations presented in the prior section, i.e, adverarially trained models rely more on certain spurious correlations.", "questions": "I have my concerns accepting the arguments in Sec. 4.4. In particular, the authors claim \"[...] this experiment shows that exclusively linking spurious correlations to adversarial vulnerability is a fallacy\". However, I do not think this experiment shows that. At most, it vaguely discards that certain color spurious correlations are the root cause of adversarial vulnerabilities, but there could exist thousand other types of spurious correlations that could explain the existence of adversarial examples. Bearing this in mind, I would appreciate a further discussion from the authors on this particular point in which they argued why their work demonstrates that adversarial examples cannot be due to any form of spurious correlation.\n\nI would also appreciate if the authors could further comment on the novel aspects of their work on top of the observations in [38] and [55].", "limitations": "I see no clear negative societal impact stemming directly from this work. Furthemore, the authors provide a short societal impact statement in the appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. **Strong and clear message**: I really appreciate the core message of the paper which tries to show there is an odd and counterintuitive relation between $\\ell_p$ adversarial robustness and reliance on spurious correlations. This is a valuable insight for the community, as it reminds us of the bizarre dynamics that $\\ell_p$ adversarial training has on the feature extraction power of neural networks.\n2. **Good balance between synthetic experiments and practical evaluations**: Overall, I find the numerical simulations on the synthetic data presented on Fig. 2 and Fig. 3 quite interesting, and they nicely complement the main observations provided on ObjectNet, RIVAL10, ImageNet-9, and Waterbirds. \n\n## Weaknesses\n\n1. **Novelty**: My main concern is with regards to the added value that these observations provide to the community given what is already known from prior literature. In particular, as described in the introduction, both [38] and [55] had already observed a stronger reliance on background spurious features for adversarially trained models, which means that the main new insight from this work is the observation that the same increase in spurious correlation also happens in ObjectNet, ImageNet-9 and Waterbirds.\n2. **Some hand-wavy arguments**: In relation to the previous point, I understand that the main objective of this work is therefore to provide some insights on why adversarially trained models are less robust to distribution shifts. In this regard, although I find the numerical simulations in Sec. 3 quite interesting, I do not think they can answer the question of why the same effect happens on real data. All in all, the spurious correlations this work alludes to are mostly semantic, e.g., backgrounds, or object positionings, and they are hard to map to the geometric concepts in Sec. 3. This makes some of the arguments regarding the \"strength of different spurious features\", or the reasons why we do not see the same effects on ImageNet-C or for $\\ell_\\infty$ models, quite hand-wavy as they are not well-defined, nor rooted on any quantifiable evidence.\n3. **Clarity (minor)**: Some passages in this manuscript feel a bit rushed and are hard to read, e.g., the presentation of the experiments leading to Fig. 4. The writing, in generall, could be more polished, and the paper be better formatted. In this sense, I believe that making the presentation of Fig. 5, Fig. 7 and Fig. 8 consistent with be nice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fIRcIJKNOD", "reviewer": "Reviewer_U2bQ", "summary": "This paper offers an analysis of the explicit tradeoffs between adversarial robustness and natural distributional robustness. The analysis well explains the phenomenon that adversarial training sometimes increases model sensitivity to spurious features in experiments. The reason is that adversarial training may actually increase model reliance on spurious features.", "questions": "- Adversarial training has been extended to improve model robustness beyond $L_p$ balls, such as [1][2]. It is then natural to wonder whether a method like [1] can increase model reliance on spurious features and thus decrease distributional robustness. After all, the methods like [1][2] are somewhat designed to improve distribution robustness.\n\n[1] Laidlaw, Cassidy, Sahil Singla, and Soheil Feizi. \"Perceptual Adversarial Robustness: Defense Against Unseen Threat Models.\" International Conference on Learning Representations. 2020.  \n[2] Lin, Wei-An, et al. \"Dual manifold adversarial robustness: Defense against lp and non-lp adversarial attacks.\" Advances in Neural Information Processing Systems 33 (2020): 3487-3498.", "limitations": "See above.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pros:**\n- This paper is the first to theoretically show that adversarial training may increase model sensitivity to spurious features, though some researchers implicitly supposed that adversarial training would always decrease model reliance on spurious features.\n- A corollary of the analysis is that adversarial training may harm model robustness to the distribution shifts that break the spurious correlations. This result is significant and thought-provoking.\n\n**Cons:**\n- Section 3 only gives one analytical result showing a simplified version of the adversarial loss. This result is good, which indeed implies that the optimal $\\theta$ may be non-zero on the set of spurious features. However, the theoretical analysis fails to show how much adversarial training can increase model reliance on spurious features. The paper then relies on verbal reasoning to reach the remaining theoretical results, without further formulating clearer theorems or corollaries. Thus, the so-called \"explicit\" tradeoffs between adversarial and distributional robustness are not so obvious. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ITi4OnlAQDX", "reviewer": "Reviewer_mc4G", "summary": "The authors claim that the adversarial training encourages to model to reply more on spurious features, which hurts the robustness under distributional shift. To support their claim, they proposed a simplified setting: a linear model with some standard core and spurious features. They found the adversarial training increased the model weights on spurious features. Finally, a few synthetic/real datasets are studied to verify the hypothesis empirically.\n", "questions": "First, although the dichotomy of “core” features and “spurious” features are mentioned many times in previous works, it would be great to have some formal definitions in theory and some examples. In image recognition, is it true that “foreground” means “core features” and “background” means “spurious features”?\n\nSecondly, I find the definition of “distribution shift” in this paper a bit restrictive in that it only changes the “spurious features” (L210 - L213). But in reality, both feature sets will be affected.\n\nThirdly, the results under the Linf constraint seem to be inconsistent: sometimes it behaves just like L2 constraint, otherwise, it behaves as the theory implies. For example, in Figure 5 the OOD accuracy drops as quickly as L2 constraint; while Figure 6 shows the RCS doesn’t change.\n\nFinally, if we think the other way round: suppose we train the classifier with a distributionally robust optimizer. Then according to this paper, the model should only use the core features, thus even more susceptible to adversarial examples than the original setting. But I don’t think it will be observed in real experiments. \n", "limitations": "Please refer to the section above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper made a counter-intuitive claim that adversarial robustness hurts distributional robustness; and this is due to the shifts in weights from the core feature set to the spurious feature set. The theoretical analysis, although seems a bit over-simplified, correlates well with almost all the empirical results. The experimentations are well designed and convincing, and I have no difficulty to follow.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4Hxn181EOL6", "reviewer": "Reviewer_gNwp", "summary": "The authors investigate the connection between adversarial robustness and reliance on spurious features.  They demonstrate that adversarial training with respect to Lp-based threat models increases reliance on spurious features.  They provide theoretical analysis of a simple linear regression setting and provide experimental results on image datasets to demonstrate that adversarially robust models have higher reliance on spurious features, leading to lower natural distributional robustness.", "questions": "- One point of confusion I had was that the authors state that Theorem 1 implies that adversarial training directs the model towards using spurious features (lines 154-156).  How exactly Theorem 1 implies this is unclear to me, could the authors clarify this point?\n- In equation 5, what is P?\n- In the toy experiments in section 3, wouldn't the dependence on core vs spurious features for an adversarially robust model depend on the number of features that a considered core vs spurious?  In the toy example, there are 2 core features vs 3 spurious features.  I think that in the setting where there are more core compared to spurious features, then we would not see the jump in spurious feature reliance as we do in the toy examples.\n- Following the proof, shouldn't equation (3) be $\\text{max}_{||\\delta||\\le \\epsilon} (Y - \\langle X + \\delta, \\theta \\rangle)^2 = (|Y - \\langle X, \\theta \\rangle| + \\epsilon ||\\theta||_q)^2$?\n- In the proof of Theorem 1, in proving equation 3, it states that \"with a suitable choice of $\\theta$\" the equality (b) holds, but the statement in equation 3 should hold for any $\\theta$ when it is used when rewriting equation 1, since loss is evaluated for any given $\\theta$, which may not be the specific value of $\\theta$ such that exact equality holds.  Should this be \"with a suitable choice of $\\delta$ instead?  I think this also goes for the equality of the triangle inequality portion too, since we are choosing the max over $\\delta$, we would choose $\\delta$ to achieve the upper bound rather than putting restrictions on $\\theta$.\n- Also in the sketch proof of Theorem 1 in main text, in the derivation after line 159, I believe that you are missing a $||\\theta||_q$: ie) in the 2nd line the last term should be $2 \\epsilon ||\\theta||_q \\mathbb{E}[|Y - \\langle X, \\theta \\rangle|]$, but the $||\\theta||_q$ is missing.  This error is also present in the proof in the appendix and carries down for many steps of the proof until the very last step in the proof of equation (2).\n- in after 576 of the Appendix, in the equality marked by (a), the last term is missing $c_1$.  This also carries down to the next step in the proof of equation (2).", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- connects 2 directions of research (natural distributional robustness and adversarial robustness) which to the best of my knowledge has not been studied in depth in prior work\n- experiments are in depth\n\nWeaknesses:\n- While the authors claim that adversarial robustness increases reliance on spurious features and frame this as a surprising result, I'm actually not surprised by this.  I think it really depends on what spurious features are being considered.  Adversarial training increases robustness against certain spurious features specifically small high frequency changes (ie. single pixel change), but for other existing spurious features which are considered in this paper, I do not see why adversarial training would suddenly prioritize using core features over the many other spurious features (ie background and color correlations) which may be easier to learn.  Because of this, I'm not very convinced about the significance of this work.\n- Given that generally we are interested in classification problems in adversarial ML, I would have been more excited to see theoretical results for a binary classification problem rather than regression problem, but this is a minor point\n- mathematical typos in the proof sketch in the main text and proof in the appendix make proof hard to follow (see questions).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept