On Feature Learning in the Presence of Spurious Correlations
Authors: Pavel Izmailov
Polina Kirichenko
Nate Gruver
Andrew Gordon Wilson
Venue: NeurIPS 2022 Accept (2022)
Keywords: spurious correlations
robustness
representation learning

Abstract: Deep classifiers are known to rely on spurious features — patterns which are correlated with the target on the training data but not inherently relevant to the learning problem, such as the image backgrounds when classifying the foregrounds. In this paper we evaluate the amount of information about the core (non-spurious) features that can be decoded from the representations learned by standard empirical risk minimization (ERM) and specialized group robustness training. Following recent work on Deep Feature Reweighting (DFR), we evaluate the feature representations by re-training the last layer of the model on a held-out set where the spurious correlation is broken. On multiple vision and NLP problems, we show that the features learned by simple ERM are highly competitive with the features learned by specialized group robustness methods targeted at reducing the effect of spurious correlations. Moreover, we show that the quality of learned feature representations is greatly affected by the design decisions beyond the training method, such as the model architecture and pre-training strategy. On the other hand, we find that strong regularization is not necessary for learning high-quality feature representations.
Finally, using insights from our analysis, we significantly improve upon the best results reported in the literature on the popular Waterbirds, CelebA hair color prediction and WILDS-FMOW problems, achieving 97\%, 92\% and 50\% worst-group accuracies, respectively.

On Feature Learning in the Presence of Spurious Correlations
Pavel Izmailoy* Polina Kirichenko" New York University
Nate Gruver"
Andrew Gordon Wilson
Abstract
Deep classifiers known to rely spurious features pptterns which are corre- lated with the target on the training data but not inherently relevant to the learning problem, such as the image backgrounds when classifying the foregrounds. In this paper we evaluate the amount of information about the core (non-spurious) features that can be decoded from the representations learned by standard empirical risk minimization (ERM) and specialized group robustness training. Following recent work on Deep Feature Reweighting (DFR), we evaluate the feature representations by re-training the last layer of the model on a held-out set where the spurious correlation is broken. On multiple vision and NLP problems, we show that the features learned by simple ERM are highly competitive with the features learned by specialized group robustness methods targeted reducing the effect of spurious correlations. Moreover, we show that the quality of learned feature representations is greatly affected by the design decisions beyond the training method, such as the model architecture and pre-training strategy. On the other hand, we find that strong regularization is not necessary for learning high-quality feature representations. Finally, using insights from our analysis, we significantly improve upon the best results reported in the literature on the popular Waterbirds, CelebA hair color pre- diction and WILDS-FMOW problems, achieving 97%, 92% and 50% worst-group accuracies, respectively.
Introduction
In classification problems, feature is spurious if it is predictive of the label without being causally related to it. Models that exploit the predictive power of spurious features can achieve strong average performance training and in-distribution test data but often perform poorly sub-groups of the data where the spurious correlation does not hold [18]. For example, neural networks trained on ImageNet known rely on backgrounds [93] or texture [19], which are often correlated with labels without being causally significant. Similarly in natural language processing, models often rely on specific words and syntactic heuristics when predicting the sentiment of a sentence or the relationship between a pair of sentences [56, 22]. In extreme cases, neural networks completely ignore task-relevant core features and only use spurious features in their predictions [97. 79], achieving zero accuracy on the subgroups of the data where the spurious correlation does not hold.
In recent work, Kirichenko al. [40] showed that, surprisingly, standard Empirical Risk Mini- mization (ERM) learns a high-quality representation of the core features on datasets with spurious correlations, even when the model primarily relies on spurious features make predictions. More- over, they showed that is often possible to recover state-of-the-art performance on benchmark problems by simply retraining the last layer of the model on small held-out dataset where the spurious correlation does not hold. This procedure is called Deep Feature Reweighting (DFR).
Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this paper, we provide an in-depth study of the factors that affect the quality of learned repre- sentations the presence of spurious correlations: how accurately can we decode the core features from the learned representations. Following Kirichenko et al. [40], we break the problem of training a robust classifier into two tasks: extracting feature representations and training a linear classifier on these features. In order to study the feature learning in isolation, we use the DFR procedure to learn an optimal linear classifier on the feature representations, and evaluate the features learned with different training methods, neural network architectures, and hyper-parametent
First, on a range of problems with spurious correlations we show that while specialized group robust- ness methods such as group distributionally robust optimization (group DRO) [76] can significantly outperform the standard ERM training, the quality of the features learned by ERM is highly competi- tive: by applying the DFR procedure to the features learned by ERM and group DRO we achieve similar performance. Furthermore, we show that the performance improvements of group DRO are largely explained by the better weighting of the learned features in the last classification layer, and not by learning a better representation of the core features. This observation has high practical significance, the problem of training the last layer of the model is much simpler both conceptually and computationally than training the full model to avoid spurious correlations [40].
Next, focusing on the ERM training, we explore the effect of model class, pretraining strategy and regularization on feature learning. We find linear dependence between the in-distrbbution accuracy of the model and the worst group accuracy after applying DFR, meaning that on natural datasets good generalization typically implies good feature learning, even in the presence of spurious features. Further, we show that the pre-training strategy has very significant effect on the quality of the learned features, while strong regularization does not significantly improve the feature representations on most benchmarks.
Finally, by finetuning a state-of-the-art ConvNext model [52], we significantly outperform the best reported results on the popular Waterbirds [76], CelebA hair color and WILDS FMOW [43] spurious correlation benchmarks, using only simple ERM training followed by DFR.
Our code is available at github. wwt. wwt.////waallooaaaa/////aaooooaa _eature_eeaariinnngggggooeee
feature
2 Related Work
Numerous works describe how neural networks can rely on spurious correlations in real world problems. In vision, neural networks can learn to rely on an image's background [93, 76, 61], secondary objects [45, 72, 83, 80, 2, 60], object textures [19] and other semantically irrelevant features [9, 49]. Spurious correlations are especially problematic in high-risk domains such as medical imaging, where it was shown that neural networks can use hospital-specific metal tokens [97] or cues of disease treatment [65] rather than symptoms to perform automated diagnosis on chest X-ray images. Spurious features are also extremely prevalent in NLP, where models can achieve good performance on benchmarks without properly solving them, c.g. by using simple syntactic heuristics such lexical overlap between the two sentences in order to classify the relationship between them [64, 22, 37, 56]. For a comprehensive survey of the area, see Geirhos et al. [19].
Because of the high practical significance of spurious correlations, many group robustness methods have been proposed. These methods aim to reduce the reliance of deep learning models on spurious correlations and improve worst group performance. Group DRO [76] is the state-of-the-art group robustness method, which minimizes the worst-group loss instead of the average loss. Other works focus on automatically identifying the minority group examples [50, 62, 13, 100], learning several diverse classifiers that use different features [48, 66, 85] or using partially available group labels [84, 63]. Group subsampling was shown to be strong baseline for some benchmarks [34, 78).
In this work, we focus on feature learning in the presence of spurious correlations. Hermann and Lampinen [31] perform a conceptually similar study, but focusing on synthetic datasets. Similar to their work, we explore how well the different features of the data can be decoded from the features learned by deep neural networks, but on large-scale natural datasets. Hermann et al. [30] explore the feature learning in the context of texture bias [19], finding that data augmentation has a profound effect on the texture bias while architectures and training objectives have a relatively small effect. Ghosal et al. [20] show that Vision Transformer models pretrained on ImageNet22k [44] significantly outperform standard CNN models several spurious correlation benchmarks.

---

Lovering et al. [54] explore the factors which affect the extractability of features after pre-training and fine-tuning of NLP models. Kaushik et al. [36] construct counterfactually augmented sentiment analysis and naural language inference datasets (CAD) and show that combining CAD with the original data reduces the reliance on spurious correlations on the corresponding benchmarks. Kaushik et al. [38] explain the efficacy of CAD and show that while adding noise to causal features degrades in-distribution and -uitt--distribution performance, adding noise to non-causal features improves robustess. Eisenstein [17] and Veitch et al. [86] formally define and study spurious features in NLP from the perspective of causality.
Kirichenko et al. [40] show that models trained with standard ERM training often learn nigh-qualig representations of the core features, and propose the DFR procedure (see Section 3) which we use extensively in this paper. Related observations have also been reported in other works in the context of spurious correlations [58], domain generalization [73] and long-tail classification [35]. While we build on the observations of Kirichenko et al. [40], our work provides profound new insights and greatly expands on the scope of their work. In particular, we investigate the feature representations learned by methods beyond standard ERM, and the role of model architecture, pre- training, regularization and data augmentation on learning semantic structure. We also extend our analysis beyond the standard spurious correlation benchmarks studied by Kirichenko et al. [40], by considering the challenging real world satellite imaging and chest X-ray datasets.
In an independent and concurrent work, Shi et al. [81] also propose an evaluation framework for out- of-distribution generalization based on last layer retraining, inspired by the observations of Kirichenko et al. [40] and Kang et al. [35]. They focus on the comparison of supervised, self-supervised and unsupervised training methods, providing complementary observations to our work.
3 Backkround
Preliminaries. We consider classification tasks with inputs x E X and classes y E V. We assume that the data distribution consists of groups G which are not equally represented in the training data. The distribution of groups can change between the training and test distributions, with majority groups becoming less common minority groups becoming more common. Because of the imbalance in training data, models trained with ERM often have a gap between average and worst group performance on test. Throughout this paper. we will be studying worst group accuracy (WGA), ic. the lowest test accuracy across all the groups G. For most problems considered in this paper, we assume that each data point has attribute E S which is spuriously correlated with the label y. and the groups are defined by a combination of the label and spurious attribute: G E y x S. In test might find that is longer correlated with y, and thus a model that has learned to rely on the spurious feature during training will perform poorly at test time. Models that rely on the spurious features will typically achieve poor worst group accuracy, while models that rely on core features will have more uniform accuracies across the groups In Appendix A, we describe the groups, spurious and core features in the datasets that we use in this paper.
distribution
In order to perform controlled experiments, we assume that we have access to the spurious attributes (or group labels) for training validation data, which we use for training of group robustness basclines and feature quality evaluation. However, we emphasize that our results on the features learned by ERM hold generally, even when spurious attributes unknown, as ERM does not use the information about the spurious features: we only use the spurious attributes to perform analysis.
Deep feature reweighting. Suppose we are given a model m X + C, where X' is the input space and is the set of classes. Kirichenko et al. [40] assume that the model m consists of a feature extractor (typically, sequence of convolutional or transformer layers) followed by a classification head (typically, a single linear layer): m h e, where : X - F is a feature extractor and h: F + C is a classification head. They discard the classification head, and use the feature extractor to compute the set of embeddings D. (e)xz;),)))) of all the datapoints in the reweighting dataset D. the reweighting dataset used to retrain the last layer of the model, and contains group- balanced data where the spurious correlation does not hold. Finally, they train a logistic regression classifier : F + C on the dataset D... For stability, logistic regression models are trained 10 times on different random group-balanced subsets of the reweighting dataset D, and the weights of the learned logistic regression models are averaged (see Appendix of Kirichenko et al. [40] for full details on the DFR procedure). Then, the final model used on new test data is given by mi I (o e.
3

---

Thoughout this paper, we use group- balanced held-out dataset (subset of the validation dataset where each group has the same number of datapoints) as the reweighting dataset D; Kirichenko et al. [40] denote this variation of the method as DFR
4 Experimental Setup and Evaluation Procedure
In this section, we describe the datasets, models and evaluation procedure that we use throughout the paper.
Datasets. In order to cover broad range of practical scenarios, consider four image classifica- tion and two text classification problems.
Waterbirds [76] is binary image classification problem, where the class corresponds to the type of the bird (landbird or waterbird), and the background is spuriously correlated with the class. Namely, most landbirds are shown on land, and most waterbirds are shown over water.
CelebA hair color [51] is binary image classification problem, where the goal is to predict whether a person shown in the image is blond; the gender of the person serves as a spurious feature, as 94% of the images with the "blond" label depiet females.
WILDS-FMOW [12, 43, 77] is a satellite image classification problem, where the classes correspond to one of 62 land use or building types, and the spurious attribute corresponds to the region (Africa, Americas, Asia, Europe. Occania or Other; the "Other" region is not used in the evaluation). We note that for the FMOW datasets the groups Gs are defined by the value of the spurious attribute, and not the combination of the spurious attribute and the class label Sy.s. as described in Section 3. Moreover, on FMOW there is also domain shift: the images for test and validation data (used for last layer retraining) are collected in 2016 and 2017, while the training data collected before 2016. For more details, please see Appendix A.
CXR-14 14 [89] is dataset with chest X-ray images for which we focus binary classifica- tion problem of pneumothorax prediction. Oakden-Rayner et al. [65] showed that there is hidden stratification in the dataset such that most images from the positive class contain a chest drain, which is a non-causal feature related to treatment of the disease. While for all other benchmarks we report WGA on test data, for this dataset, following prior work [65, 48, 70], we report worst group AUC because of the heavy class imbalance.
MultiNLI [91, 76] is a text classification problem, where the task is to classify the relationship between given pair of sentences as contradiction, entailment or neither of them. In this dataset, the presence of negation words (c.g. "never") in the second sentence is spuriously correlated with the "contradiction" class.
CivilComments [8, 43] is text classification problem, where the goal to classify whether a given comment is toxic. We follow Idrissi et al. [34] and use the coarse version of the dataset both for training and evaluation, where the spurious attribute is s comment mentions at least one of the following categories: male, female. LGBT, black, white, Christian, Muslim, other religion; otherwise, the spurious label The presence of the eight categories above is spuriously correlated with the comment being classified as toxic.
the
The Waterbirds, CelebA, CivilComments and MultiNLI datasets are commonly used to benchmark the performance of group robustness methods [see e.g. 34, 50, 63). The FMOW and CXR-14 14 datasets present challenging real-world problems with spurious correlations. In these datasets, the inputs do not resemble natural images from datasets such as ImageNet [75], so models have to learn the relevant features from data to achieve good performance, and cannot simply rely on feature transfer. We provide detailed descriptions of the data and show example datapoints in Appendix A, Figures and 7.
We take the minimum out of the two scores on test: AUC for classifying negative class against positive examples with chest drain and AUC for negative class against positive without chest drain.

---

Models. Following prior work [76, 34] we use ResNet-50 [26] model pretrained on ImageNetlk [75] on Waterbirds, CelebA and FMOW. For the NLP problems, we use BERT model [14] pre- trained on Book Corpus and English Wikipedia data. On CXR-14, following prior work [e.g. 70, 65, 48] we use DenseNet-12 121 model [32] pretrained on ImageNetIk. In Section 6, we provide an extensive study of the effect of architecture and pretraining on the image classification problems, and in Appendix E we perform a similar study on the MultiNLI text classification problem.
Evaluation strategy. We use DFR to evaluate the quality of the learned feature representations, as described in Section we measure how well the core features can be decoded from the learned representations with last layer retraining. In some of the experiments, we also train a classifier to predict the spurious attribute s instead of the class label y. Using this classifier, we can evaluate the decodability of the spurious feature from the learned feature representation. We refer to this procedure as s-DFR and the corresponding worst group accuracy (in predicting the spurious attribute s) as DFR s-WGA. Additionally, we evaluate the worst group accuracy and mean" accuracy of the base model without applying DFR, which we refer to as base WGA and base accuracy respectively.
5 ERM vs Group Robustness Training
Multiple methods have been proposed for training classifiers which are more robust to spurious correlations, with significant improvements in worst group accuracy compared to standard training. In this section, we use DFR to investigate whether the improvements of group robustness methods are caused by better feature representations or by better weighting of the learned features.
Methods. We consider 4 methods for learning the features. ERM or Empirical Risk Minimization
is the standard training on the original training data, without any techniques targeted at improving worst group performance. RWG reweights the loss on each of the groups according to the size of the group and RWY reweights the loss on each class according to the size of the class [34]. Group DRO [76] is a state-of-the-art method which uses the group information on the training data to minimize the worst group loss instead of the average loss. Group DRO is often considered as an oracle method or upper-bound on the worst group performance under spurious correlations [50, 13].
On the CXR dataset the group labels are not available the train data, we cannot apply RWG or group DRO; on this dataset we compare ERM to RWY. On several datasets, the performance of RWY and RWG methods deteriorates during training. For these datasets, additionally report the results for the checkpoint obtained with early stopping (RWY-ES and RWG-ES). For group DRO, we report the performance with early stopping on all datasets except for CXR (GDRO-ES). In all cases, early stopping is performed based on the worst-group accuracy on the validation set.
selection. We train ERM models, RWG and RWY with the same hyper- parameters shared between all the image datasets (apart from batch size which is set to 32 on Waterbirds, and 100 on the other datasets) and between the natural language datasets. We do not tune of these methods for worst group accuracy. For group DRO, we run a grid search
the tlyee--paraneter over the values of the generalization adjustment C, weight decay and learning rate hyper-parameters, and select the best combination according to the worst-group accuracy on validation data with early stopping. For details, please Appendix B.
5.1 Results
We compare feature learning methods all datasets in Figure 1. As expected, the worst group accuracy of group robustness methods is significantly better than the ERM worst group accuracy on most datasets. For example, on Waterbirds, ERM only gets 68.8% WGA, while group DRO with early stopping gets 90.6%. However, after applying DFR the performance of ERM and group DRO is very close, with a slight advantage for ERM (91.1% 1% for ERM and 90% for group DRO), and similar observations hold on all datasets.
The results for RWG and RWY are analogous. Namely, when combined with carly stopping, these methods outperform ERM on base model performance. Once we apply DFR, however, the gap in
2 Following Sagawa et al. [76] and Kirichenko et al. [40], we evaluate the mean accuracy according to the group distribution in the training set. This way, mean accuracy represents the in di-iistributin generalization of the model.
5

---

Acourac
8B
39
ES
72
72
72
79 79
Waterbirds
CelebA
FMOW
CXR
MultiNLI
Civil Comments
ERM
RWY
I
RWY-E5
RWG
RWG-E5
GDRO
GORO-ES
DFR Improvement
DFR Redaction
Figure 1: ERM vs group robustness methods. Performance ERM on group robustness methods on vision and NLP benchmark problems. ES stands for early stopping. For each method on each dataset report the base model worst group accuracy (shown with the number inside each bar), and the worst group accuracy after applying DFR (shown above each bar). On CXR-14, we report worst group AUC. While on many datasets the base performance of ERM much worse compared to group robustness methods, the performance of the different methods is similar after we apply DFR, suggesting that the strong performance group robustness methods largely caused by better weighting of the learned features rather than better feature representations.
performance between the methods becomes very small. fact, on all the datasets and for all the methods the improvement in worst group accuracy from using any of the considered group robustness methods compared to ERM does not exceed 1-2% after applying DFR.
These results suggest that the improvements over ERM in base model performance for methods such as group DRO and RWG are largely the result of better weighting of the learned features rather than learning better representations of the core features. Indeed, if the core feature was better represented by group robustness methods, we would expect to see significant improvement over ERM after applying DFR.
This observation significant both practically and scientifically. Practically, the problem of training the last layer is simpler, more data efficient and less compute intensive than retraining the full model [40]. Our results suggest that for many problems, practitioners can primarily focus on retraining the last layer, as training the feature extractor model with group robustness methods does not provide significant improvements. Scientifically, robustness spurious correlations is often implicitly or explicitly associated with the quality of learned feature representations [e.g. 3, 4, 74, 100, 48, 99]. Our results suggest that the quality of feature representations not significantly affected by group DRO, refining our understanding of group robustness training and representation learning in the presence of spurious correlations.
Effect of early stopping. Early stopping is crucial to achieving strong base model performance with RWY, RWG and group DRO on many of the datasets. In Figure 1 report the results both with and with- out early stopping on datasets where the validation worst group accuracy significantly degrades over the course of training. Generally, early stopping does not appear to significantly improve the quality of the learned feature representations even in these prob- lems: after applying DFR, methods with and without early stopping achieve similar performance. This observation suggests that late in training, neural net- works may start to assign higher weight to the spu- rious features, but the information about the core features is still preserved in the learned representa- tions. For ERM and Group DRO, we explore the DER WGA performance iteration in Appendix D, also finding that the length training has a relatively small effect on the final DFR WGA.
Waterbirds
CelebA
MultiNLI
D.5
OFR WGA
0,9
0.s5
0.6
0.7
0.70
09
Base WGA
Base WGA
Base WGA
Last Checkpaint
Early Stopping
Group DRO. Worst group ac- curacy before and after applying DFR for range of group DRO runs. DFR does not improve the best runs, indicating that group DRO already learns nearly optimal last layer.
Figure 2:
function of the training
Group DRO analysis. In Figure we report the worst group accuracy of multiple group DRO runs before and after applying DFR. For each run, evaluate the best checkpoint according to validation

---

accuracy (i.e. the checkpoint selected by early stopping) and the last checkpoint saved after fixed number of epochs. We observe that while the last checkpoints perform poorly in terms of base WGA in most runs, DFR can significantly improve their performance, removing the need for early stopping. Interestingly, we find that the best performing group DRO models cannot be improved by DFR on each of the datasets. This result suggests that the weighting of the features learned by group DRO is already close to optimal, again indicating that the success of group DRO can largely be attributed to learning better weighting for the features in the last linear layer, rather than learning better features.
Group robustness methods (e.g. group DRO) perform well because they improve the last linear layer, not the underlying feature representations.
Effect of the Base Model
Most of the prior work spurious correlation considers fixed model class for each problem: for example, on Waterbirds and CelebA datasets almost all the papers use a ResNet-50 base model pre-trained on ImageNet [e.g. 76, 50, 48, 40, 84, 63, 100]. Recently, Ghosal al. [20] showed that vision transformer models [15] may provide better robustness to spurious correlations if pretrained on a large dataset. Here, we perform a systematic large-scale evaluation of the effect of base model choice on the quality of learned feature representations.
We repeat the experiments on the effect of base model, pretraining, and training on the target dataset presented in this section on the MultiNLI text classification task in Appendix E, with similar observations.
6.1 Effect of the architecture and pretraining strategy
In Figure 3, we plot the base model mean and worst group accuracy and DFR worst group accuracy for wide range of models and pretraining strategies on each of the four image classification datasets. We provide model descriptions and training prper-parameters in Appendix C. We train a total of 78 models on Waterbirds, 78 on CclebA, 40 on FMOW and 40 on CXR.
Accuracy on the line. Miller et al. [59] showed that for many distribution shifts in practice the out- of-distribution performance is highly correlated with the in-distribution generalization performance. In Figure 3 (top row), for each dataset we show scatter plots of base model mean accuracy vs base model worst group accuracy, analogously to Miller et al. [59]. For FMOW (which was also considered by Miller et al. [59]) we observe a linear correlation between the base model mean and worst group accuracies. On CXR, the dependence is also roughly linear. However, both on CelebA and on Waterbirds, the correlation does not appear entirely linear. In particular, on Waterbirds there is large number of models that have similar worst group accuracy z 20%, for which there appears to be little correlation between the base model WGA and mean accuracy. On CelebA, the same phenomenon occurs for the best performing models, with WGA between 40% and 50%.
DFR accuracy is on the line. Next, in the bottom panels of Figure 3, for each of the datasets we report the base model mean accuracy vs DFR WGA. On all datasets other than CXR3, we observe a high linear correlation between the metrics, including the Waterbirds and CelebA. In particular, the models with the best mean (in fin-distribution) accuracy also achieve the best DFR WGA. Note that this is not the case for the base model WGA on CelebA, where the best base model WGA is 51% achieved by a ResNet- 101 model pretrained on ImageNetIk; this model only achieves mean accuracy of 95.45% compared to 96.2% accuracy for the ConvNext XLarge model. The results in Figure confirm that models that achieve better accuracy the training data distribution generally learn better representations of the core features, and provide better worst group accuracies with DFR.
Are VITs more robust than CNNs? Ghosal et al. [20] noted that vision transformers pre-trained on ImageNet22k achieved better worst group accuracies on benchmarks with spurious correlations than popular CNN models. In particular, with a VIT-B/16 model, they achieve 89.3% worst group accuracy on Waterbirds. With the ConvNext Large model pretrained on ImageNe122k with ImageNet
*On CXR, we found that the base model is often hard to improve with DFR, suggesting that it already learns to weight the features correctly. We discuss the possible reasons and the nature of spurious features on this dataset in Appendix A.2.
7

---

Waterbirds
CelebA
FMOW
CXR
0.74
Blope=8.3
3.ope=3.9
slope=11
slope=0.7
Base WGA
r²-0.9
0.4
2 0.9
0.7
0.0
0.70
0.4
0.68
0.2
0.66
0.9
1.0
0.90
0.96
0.3
0.4
0.5
0.30
0.35
Base Acc
Base Acc
Base Acc
Base Acc
Waterbirds
CelebA
FMOW
CXR
Slope=3.9
Siope-4.6
slope-0.0 0.8
0.80
slope-Da 0.4
DFR WGA
0.9
-2=0.8
0.75
0.6
0.40
0.70
0.6
0.35
0.65
0.30
0.00
0.9
0.90
0.95
0.2
0.3
0.4
0.0
0.9
Base Acc
Base Acc
Base Acc
Base Acc
Architecture:
5e
O
[wide] resnet
conmenx
dersenet
A
beit
aleenet
resneat
Protraining
Sup IN224
INIA
Sup IN12
Contrantion INT8
MAE IM1k -+ Sup IN
Sup IN224
Deit INLA
MAE INI
Rendom
Figure 3: Effect of the model architecture and pretraining. Base model in-distribution accuracy plotted against base worst-grous accuracy (top row) and DFR worst group accuracy (bottom row). For each panel, we additionally estimate the slope and the r² score for the linear fit to the data. On all datasets other than CXR, the DFR WGA linearly correlated with base model in-distribution performance. For the base model WGA, the correlation with in-distribution performance is not entirely linear. Generally, models with better in-distribution performance provide better worst-group performance and better core feature representations.
finetuning, we achieve 88.9% worst group accuracy on the Waterbirds dataset. Notably, ConvNext is CNN model and not vision transformer. Generally, we observe that the models that provide the best in-distribution performance also provide better WGA. In our experiments, we did not observe qualitative differences between the results for vision transformers and CNN models.
ERM features are sufficient for SOTA performance. The DFR WGA results for the Waterbirds,
CelebA and FMOW datasets significantly improve upon the previous best reported results, to the best of our knowledge. In particular, the ConvNext Large model pretrained on ImageNet22k with ImageNetIk finetuning achieves 97.2% DER WGA on Waterbirds and 92.2% on CclebA; FMOW, we only considered smaller ConvNext versions due to computational constraints, still achieving 50.6% DFR WGA with ConvNext Small pretrained on ImageNet22k; the current best results the WILDS leaderboard for this dataset are 47.6% followed by 35.5% [43]. We note that our DFR evaluation uses the validation set to train the last layer of the model, similarly to e.g. Nam et al. [63], and unlike most standard group robustness methods which only use the validation set to tune the parameters. However, the results presented in this section prove that standard ERM with strong pretrained model can achieve outstanding results on the robustness benchmarks, significantly improving upon specialized group robustness methods using weaker model.
Strong in-distribution generalization correlates with improved robustness to spurious correla- tions (measured by the DFR worst group accuracy), and this trend holds regardless of the underlying architecture (CNN or ViT).
6.2 Does training on the target data improve features?
Above, we have shown that the choice of the base model architecture and pretraining has a large effect on the quality of the learned feature representations, as measured by DFR WGA. It is then natural ask how much of the feature learning happens during training on the target data, and how much is simply transferred from the pretraining task. To answer this question, for the models trained
8

---

Waterbirds
CelebA
FMOW
CXR
Trained DFR WGA
1.0
0,9
0.6
0.7
0.00
0.4
0.
0.8
0.4
0.6
Init DFR WGA
Init DFR WGA
Init DFR WGA
Init DFR WGA
Waterbirds
CelebA
FMOW
Trained DFR s-WGA
1.00
0.05
0.9
0.90
0.56
0.8
0.7
0.8
0,9
0.7
0.0
0.2 03 04 U.S
Init DFR 5-WGA
Init DFR s-WGA
Init DFR s-WGA
Architecture:
5esnet50
(uiide
dersenet
A
beit
alexnet
Pretroining:
Sup IN22X - INLK
Sup IN1
Contrastive IN18
MAE IN1K - Sup INIK
Sup IN22K
Dait INIA
MAE INIK
Random Init
Figure 4: Effect of training target data. DFR WGA (top panels) and DFR s-WGA (bottom panels) before and after training. On Waterbirds, the results do not significantly improve from training on the target data. On the other datasets, the performance improves consistently after training on the target data. Interestingly, on CelebA the DFR s- WGGA decreases during training for many models, meaning that the spurious gender feature becomes less predictable from the learned representations.
in the previous section, we run the DFR evaluation on the initial weights of the models, without training the feature extractor the target data. We report the DFR WGA and DFR s-WGA results in Figure 4. We repeat this experiment on the MultiNLI dataset in Appendix Table
Surprisingly, find that Waterbirds, for most models the improvement from training the target (Waterbirds) data is very small, if any. For example, for the leageNettk-prretriine ResNet-50 model that was not trained on Waterbirds data at all we get 88.2% worst group accuracy by simply training the last layer on the validation data with DFR. If we finetune the feature extractor on the achieve 92.9% DER WGA. For reference, the saaiee-f-the--ttt group DRO method achieves 91% WGA on Waterbirds with this architecture.
Waterbirds training data,
Furthermore, with the ConvNext Large model pretrained on ImageNer22k, get 94% DFR WGA without training the feature extractor on Waterbirds data, exceeding the best results previously reported in the literature, to the best of our knowledge. This strong performance is not particularly surprising, as ImageNet22k has many of the Waterbirds bird types as classes. The performance is almost unchanged by training the target data: DFR WGA after training is 94.3%. From these results, we can conclude that Waterbirds performance should not be used as a primary metric for feature learning performance, especially if large-scale pretraining is used! Indeed, it is possible to achieve outstanding performance, exceeding the previously reported state-of- oftthe--ttt, without training the features the target data.
On the other datasets, the feature learning more pronounced: the DFR WGA improves after training for all the models considered. However, on CelebA it is still possible to achieve 88.3% WGA without training on CelebA data, with ConvNext Xaarge pretrained on ImageNet22k, while the best result that we were able to achieve with feature extractors trained on CelebA is 92.2%.
Is the spurious feature representation improved during training? We additionally explore the quality of representation of the spurious feature via DFR s-GAA We show the results in Figure (bottom row). Interestingly, on CelebA the spurious gender feature becomes less predictable during training for many of the models. On FMOW and Waterbirds there is no consistent trend, and the spurious feature does not become significantly more or less predictable during training.
'This model is used initialization by most works that report experiments the Waterbirds dataset.

---

Strong feature extractors trained on large-scale datasets sufficient for outstanding per- formance on benchmark spurious correlation datasets, without any finetuning on the target data. Finetuning can improve the quality of the learned representations but often only to relatively small degree (e.g. Waterbirds).
6.3 Effect of pretraining strategy
Finally, using a fixed ResNet-50 model architecture evaluate the effect of pretraining strategy. On each of the four datasets, we train a ResNet-50 model ini- tialized with (1) random initialization, (2) supervised pretraining, (3) DINO pretraining [10], (4) SimCLR pretraining [11] and (5) Barlow Twins pretraining [96] on ImageNet We report the results in Figure On Waterbirds and FMOW datasets, the randomly initialized model does not provide competitive per- formance. On CelebA, still underperforms the pre- trained models, but the gap is much smaller. Among all pretraining methods, supervised is preferable, but the contrastive methods are highly competitive.
29
m 74
DFR VGA
As 4) 45
Waterbirds
CelebA
FMOW
CXR
Rand
Dino
Barlow Twins
Supervised
SimCLR
Figure 5: Effect of pretraining, Supervised pretraining provides the best performance, but contrastive methods are competitive. Random initialization performs poorly.
In Appendix Table on MultiNLI with BERT models, also show that pretraining is crucial for strong performance, while the specific choice of pretraining data has a smaller effect. In Appendix D we provide further ablations exploring the effect of weight decay and data augmentation on the learned feature representations. We find that while appropriate regularization can be helpful, models trained with no weight decay and no data augmentation can achieve highly competitive DFR WGA, especially on the standard Waterbirds and CelebA benchmarks.
ImageNet pretraining improves the representation of core features on many image datasets with spurious correlations.
Discussion
The worst group performance model is affected by two factors: the quality of the representation of the core features produced by the feature extractor and the weight assigned to the core features in the last classification layer. In contrast to prior work, we consider the quality of the feature extractor in isolation, focusing on realistic datasets and large-scale models. We find that many of the popular group robustness methods improve the worst group performance primarily by learning a better last layer and not by learning better feature representation. Similarly, regularization techniques such as early stopping and strong weight decay can improve the worst group accuracy by learning a better last layer, but do not lead to a consistent improvement in terms of the quality of the learned feature representations. On the other hand, the base model architecture and pre-traine pre-training strategy have a major effect on the quality of the feature representations.
Our observations suggest an important open question: it possible to significantly improve upon standard ERM in terms of the quality of the learned representations for given base model? In the future work, we will evaluate methods such as Rich Feature Construction [99]. gradient starvation [68], ensembling [47] and other techniques for increasing feature diversity [c.g. 48]. We hope that our work will also inspire new group robustness methods targeted specifically at improving the quality of the core feature representations.
This research is supported by NSF CAREER IIS -144494 NSF I-DISRE 193471, NIH 301DA048764-01AA) NSF IIS- 1910266, NSF 1922658 NRT-HDR: FUTURE Foun- dations, Translation, and Responsibility for Data Science, NSF Award 1922658, Meta Core Data Science, Google AI Research, BigHat Biosciences, Capital One, and an Amazon Research Award. This work was supported in part through the NYU IT High Performance Computing resources, services, and staff expertise.
Acknowledgements.
10

---

References
[1] Agarwal, A., Beygelzimer. A.. Dudik, M., Langford, J., and Wallach, H. (2018). A reductions approach fair classification. In International Conference on Machine Learning, pages 60-69. PMLR. (Cited on page 28)
[2] Alcom, M. A., Li, Q. Gong, Z., Wang. C., Mai, L.,, Ku, W.-S., and Nguyen, A. (2019), Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4845-4854, (Cited on page 2)
[3] Arjovsky, M., Bottou, L., Gulrajani, I,, and Lopez-Paz, D. (2019). Invariant risk minimization. arXiv preprint arXiv:1907.0289) (Cited page 6)
[4] Bahng, H., Chun, S., Yun, S., Choo, J., and Oh, S. J. (2020). Learning de-biased representations with biased representations. In International Conference on Machine Learning, pages 528 539. PMLR. (Cited on page 6)
[5] Balestriero, R., Bottou, L., and LeCun, (2022). The effects of regularization and data augmentation are class dependent. arXiv e-prints, pages arXiv-22004 (Cited on page 26)
[6] Bao, H., Dong. L., and Wei, F. (2021). Beit: Bert pre-training of image transformers. arXiv preprint OrXiv:22166.1 08254. (Cited on page 22)
[7] Biewald, L. (2020). Experiment tracking with weights and biases. Software available from wandb.com. (Cited on page 18)
[8] Borkan, D., Dixon, I.,, Sorensen, J., Thain, N., and Vasserman, L. (2019). Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference, pages 491-500. (Cited on page 4, 19)
[9] Brendel, and Bethge, M. (2019). Approximating cans with wwfloooll-eeatreeeeeeeeee-----ee models works surpris- ingly well on imagenet. arXiv preprint arXiv: 1904 (Cited on page 2)
[10] Caron, M., Touvron, H., Misra, 1. Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021). Emerging properties in self-supervised vision transformers. In Proceedings of the IREF/CVF International Conference Computer Vision, pages 9650-9660. (Cited on page 10, 22)
[11] Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). A simple framework for contrastive learning of visual representations, In International conference on machine learning, pages 1597-1607. PMLR. (Cited on page 10, 22)
[12] Christie, G., Fendley, N., Wilson, J., and Mukherjee, (2018). Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition. (Cited on page 4, 18)
[13] Creager, E., Jacobsen, J.-H., and Zemel, R. (2021), Environment inference for invariant Icarning. In International Conference on Machine Learning, pages
PMLR. (Cited on page 2, 5)
[14] Devlin, J., Chang, M.-W., Lee, K., and Toutamova, K. (2018). Bert: Pre-training of deep hidirectional transformers for language understanding. arXiv preprint 1810:0810004880) (Cited page 5)
[15] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M.. Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image worth 16x16 words: Transformers for image recognition at scale. arXiv preprint a 1999., (Cited on page 7, 22)
[16] Dwork. C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. (2012). Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214-226. (Cited on page 28)
[17] Eisenstein, J. (2022). Informativeness and invariance: Two perspectives on spurious correlations in natural language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4326-4331. (Cited on page 3)
[18] Geirhos, R., Jacobsen, J.-H. Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. (2020). Shortcut learning deep neural networks. Nature Machine Intelligence, 2(1 2(1666656677, (Cited on page 1)
[19] Geirhos, R., Rubisch, P. Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W. (2018). Imagenet- trained cnns are biased towards texture: increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: arXiv:1811.12233 (Cited page 2)

---

[20] Ghosal, S. S., Ming. Y., and Li, Y. (2022). Are vision transformers robust to spurious correlations? arXiv e-prints, pages arXiv-2203. (Cited on page 2.
[21] Goyal, P., Duval, Q. Reizenstein, J., Leavitt, M., Xu, M., Lefaudeux, B., Singh, M., Reis, V., Caron, M., Bojanowski, P. Joulin, A., and Misra, I. (2021). Vissl. (Cited on page 18)
[22] Gururangan, S., Swayamdipta, s., Levy, O., Schwartz, R., Bowman, S. R., and Smith, N. A. (2018). Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.00 1803.0234.. (Cited on page 1,
2)
[23] Hardt, M.. Price. E., and Srebro, N. (2016). Equality of opportunity in supervised learning. Advances in neural information processing systems, 29. (Cited on page 28)
[24] Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., del Río, F., Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., and Oliphant, T. E. (2020). Array programming with NumPy. Nature, 585(7825):357 -62... (Cited on page 18)
[25] Hc, K., Chen, X., Xic, S., Li, Y., Dollár, P., and Girshick, R. (2021a). Masked autoencoders are scalable vision learners. arXiv preprint wwx.v:2//1.0000 1606777. (Cited on page 22)
[26] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778. (Cited on page 5, 22)
[27] He, P., Gao, J., and Chen. W. (2021b). Debertav3: Improving deberta using electra-style pre-training with gradient-disee diiiiannddde embedding sharing. arXiv preprint ar.5v:2 (Cited on page 27)
[28] He, P., Liu, X., Gao, and Chen, W. (2020). Deberta: Decoding-enhanced hert with disentangled attention. arXiv preprint arXiv:2006.0366 236544 (Cited on page 27)
[29] Hendrycks, D., Mu, N., Cubuk, D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. (2019). Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint 1rXiv:191..02781. (Cited on page 22, 26)
[30] Hermann, K., Chen, T., and Korbliihh, S. (2020). The origins and prevalence of texture bias in convolutional neural networks. Advances in Neural Information Processing Systems, 33:19000-19015. (Cited on page 2)
[31] Hermann, K. and Lampinen, A. (2020). What shapes feature representations? exploring datasets, architec- tures, and training. Advances Neural Information Processing Systems, 33:9995-10006. (Cited page 2)
[32] Huang, G., Liu, Z.,, Van Der Maaten, L.,, and Weinberger, K. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708. (Cited on page 5, 22)
[33] Hunter, J. D. (2007). Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 933:90-955 (Cited on page 18)
[34] Idrissi, B. Y., Arjovsky, M.. Pezeshki, M.. and Lopez-Paz, D. (2021). Simple data balancing achieves competitive vorst-group-accuracy arXiv preprint arXiv:21 14503. (Cited on page 2, 4, 5, 19, 24)
[35] Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., and Kalantidis, Y. (2019). Decoupling representation and classifier for long-tailed recognition. arXiv preprint 1tXiv:1910.09//1 (Cited on page 3)
[36] Kaushik, D.,. Hovy, E., and Lipton, Z. (2020). Learning the difference that makes difference with counteractualll--aggeeeee data, International Conference on Learning Representations. (Cited on page 3)
[37] Kaushik, D. and L Lipton, Z. C. (2018). How much reading does reading comprehension require? a critical investigation of popular benchmarks. arXiv preprint 1 008.04996. (Cited on page 2)
[38] Kaushik, D., Setlur, A., Hovy, E., and Lipton, Z. C. (2021). Explaining the efficacy of counterfactually augmented data. International Conference Learning Representations (ICLR). (Cited on page 3)
[39] Khani, F. Raghunathan, A., and Liang. P. (2019). Maximum weighted loss discrepancy. arXiv preprint wwXiv1196.035/26 (Cited page 28)
12

---

[40] Kirichenko, P., Izmailov, P., and Wilson. A. G. (2022). Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204. 02937. (Cited page 1, 2, 4, 7, 24, 25)
[41] Kleinberg. J., Mullainathan, S., and Raghavan, M. (2016). Inherent trade-off's in the fair determination of risk scores. arXiv preprint arXiv:1609. 000000 (Cited on page 28)
[42] Kluyver, T., Ragan- Kelley, B., Pérez, F., Granger, B., Bussonnier, M., Frederic, J., Kelley, K., Hamrick, J. Grout, J., Corlay, S., Ivanov, P., Avila, D., Abdalla, S., and Willing, C. (2016). Jupyter notebooks a publishing format for reproducible computational workflows. In Loizides, F. and Schmidt, B., editors, Positioning and Power in Academic Publishing: Players, Agents and Agendas, pages 87 90. IOS Press. (Cited page 18)
[43] Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M. Zhang, M. Balsubramani, A., Hu, W., Yasunaga, M., Phillips, I.., Gao, I., al. (2021). Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664, PMLR. (Cited on page 2, 4, 8, 18, 19)
[44] Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. (2020). Big transfer (bit): General visual representation learning. In European conference on computer vision, pages 491-507. Springer. (Cited on page 2)
[45] Kolesnikov, A. and Lampert, C. H. (2016). Improving weakly-supervised object localization by micro- annotation. arXiv preprint Xix: 055305538. (Cited on page 2)
[46] Krizhevsky, A. (2014). One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:104..5997, (Cited on page 22)
[47] Lakshminarayanan, B.. Pritzel, A., and Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30. (Cited on page 10)
[48] Lee, Y., Yao, H., and Finn, C. (2022). Diversify and disambiguate: Learning from underspecified data. arXiv preprint M2Xix:2222.0311 (Cited on page 2, 4, 5, 6, 7, 10, 19, 28)
[49] Li, Y., Li, Y., and Vasconcelos, N. (2018). Resound: Towards action recognition without representation bias. In Proceedingy of the Earopean Conference on Computer Vision (ECCV), pages 513-528. (Cited on page2 2)
[50] Liu, F. 7., Haghgoo, B., Chen, A. S., Raghunathan, A., Koh, P. W., Sagawa, S., Liang, P., and Finn, C. (2021). Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pages 0781-6792. PMLR. (Cited on page 2, 4, 5,
[51] Liu, Z., Luo, P., Wang. X., and Tang, X. (2015). Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 77300-7388 (Cited on page 4, 18)
[52] Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022). A convnet for the 2020s. aixiv preprint ar Xiix:2201.035555 (Cited on page 2, 22)
[53] Loshchilov, and Hutter, (2017). arXiv:1711.05101. (Cited on page 24, 28)
Decoupled weight decay regularization. arXiv preprint
[54] Lovering. C., Jha, R., Linzen, T., and Pavlick, E. (2020). Predicting inductive biases of pre-trained models. In International Conference Learning Reprexentutions. (Cited on page 3)
[55] Marcel, S. and Rodriguez, Y. (2010). Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM international conference Multimedia, pages 1485-1488. (Cited on page 18)
[56] McCoy, R. T., Pavlick, E., and Linzen, T. (2019). Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprini arXiv:1902.0100 01000. (Cited on page 1,
[57] McKinney, W. (2010). Data structures for statistical computing in python. In van der Walt, S. and Millman, J., editors, Proceedings of the 9th Python Science Conference, pages 51 56. (Cited on page 18)
[58] Menon, A. K., Rawat, A. S.,, and Kumar, S. (2020). Overparameterisation and worst-case generalisation: friend or foe? In International Conference on Learning Representations. (Cited on page 3)
[59] Miller, J. P., Taori, R., Raghunathan, A., Sagawa, S., Koh, W., Shankar, V., Liang, P., Carmon, Y.. and Schmidt, L. (2021). Accuracy on the line: on the strong correlation between out cittof-ddstibutiin and in disdribbuutinn generalization. In International Conference on Machine Learning, pages 7721-7733 PMLR. (Cited on page 7)
13

---

[60] Mo, S., Kang, H. Sohn, K., Li, C.-L., and Shin, J. (2021). Object-aware contrastive learning for debiased scene representation. Advances in Neural Information Processing Systems, 34. (Cited page 2)
[61] Moayeri, M.. Pope, P., Balaji. Y., and Feizi, S. (2022). A comprehensive study of image classification model sensitivity to foregrounds. backgrounds, and visual attributes. arXiv preprint 22X11:200.11066. (Cited on page 2)
[62] Nam, J., Cha, H., Ahn, S., Lec, J., and Shin, J. (2020). Learning from failure: De- classifier from biased classifier. Advances in Neural Information Processing Systems, 33:20673 206644 (Cited on page 2)
[63] Nam. J., Kim, J., Lee, J., and Shin, J. (2022). Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. In International Conference on Learning Representations. (Cited on page 2, 4, 7, 8)
[64] Niven, and Kao. H.-Y. (2019). Probing neural network comprehension of natural language arguments. arXiv preprint arx: 1907.0735. (Cited on page 2)
[65] Oakden Dakden-Rayner, L., Dunnmon, J., Cameiro, G., and Ré, (2020). Hidden stratification causes clinically meaningful ful failures in machine learning for medical imaging. In Proceedings of the ACM conference on health, inference, and learning, pages 151 151-159. (Cited on page 2, 4, 5, 18, 19)
[66] Pagliardini, M., Jaggi, M., Fleuret, F., and Karimireddy, S. (2022). Agree to disagree: Diversity through disagreement for better transferability. arXiv preprint 22X.v22002.04414 (Cited on page 2)
[67] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang. E.,, Vetioo, Z., Lin, Z.,, Desmaison, A., Antiga, L.,, and Lerer, A. (2017). Automatic diffcrentiation in pytorch. (Cited on page 18)
[68] Pezeshki, M., Kaba, O., Bengio, Y., Courville, A. C., Precup. D., and Lajoie, G. (2021). Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34. (Cited on page 10)
[69] Pleiss, G., Raghavan, M., Wu, F,, Kleinberg, J., and Weinberger. K. Q. (2017). On fairness and calibration. Advances in neural information processing systems, 30. (Cited on page 28)
[70] Rajpurkar, P., Irvin, J., Zhu, K., Yang. B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya. K.. et al. (2017). Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep (Cited on page 4, 5, 19)
learning. arXiv preprint urXiv:1711.0522.
[71] Robbins, H. and Monro, S. (1951). A stochastic approximation method. The annals of mathematical statistics, pages 400-400 407-407.660 (Cited on page 24)
[72] Rosenfeld, A., Zemel. R., and Tsotsos. J. K. (2018). The elephant in the room. arXiv: 03305. (Cited on page 2)
arXiv preprint
[73] Rosenfeld, E., Ravikumar, P., and Risteski, A. (2022). Domain-adjusse regression or: Erm may already lear features sufficient for ofist---ditiiitttin generalization. arXiv preprint 20ti:220020085566 (Cited on page
[74] Ruan, Y., Dubois, Y., and Maddison, C. J. (2021). Optimal representations for covariate shift. arXiv preprint arXiv:2201.000 (Cited on page 6)
[75] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, 7.,, Karpathy, A., Khosla, A., Bersstein, M., et al. (2015). Imagenct large scale visual recognition challenge. International journal of computer vision, 1 115(3):211-25
(Cited on page 4, 5)
[76] Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. (2019). Distrihutionally robust neural net- works for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.0977 (Cited on page 2, 4, 5, 7, 18, 23, 24, 26, 27)
[77] Sagawa, S., Koh, P. W., Lee, T.. Gao, I., Xie, S. M., Shen, K., Kumar, A., Hu, W., Yasunaga, M.. Marklund, H., et al. (2021). Extending the wilds benchmark for unsupervised adaptation. arXiv preprint arXiv:2112.0500 02500000 (Cited on page 4, 18, 19)
[78] Sagawa, S., Raghunathan, A., Koh, P. W., and Liang, P. (2020). An investigation of why overparameteriza- tion exacerbates spurious correlations. In International Conference on Machine Learning, pages 8346-8356. PMLR. (Cited on page 2)
[79] Shah, H., Tamuly, K., Raghunathan, A., Jain, P., and Netrapalli, P. (2020). The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573 9585, (Cited page 1)
14

---

[80] Shetty, R., Schiele. B., and Fritz. M. (2019). Not using the car to see the sidewalk-qu quantitying and controlling the effects of context in classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8218-8226 (Cited on page 2)
[81] Shi, Y., Daunhawer, I., Vogt, J. E.,, Torr, H., and Sanyal, A. (2022). How rohust are pre-trained models distribution shift? arXiv preprint (tl::20060088777 (Cited on page
[82] Simonyan, K. and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1556. (Cited on page 22)
[83] Singla, S. and Feizi, S. (2021). Salient imagenet: How to discover spurious features in deep learning? arXiv preprint orXiv:21 2 10.04301. (Cited on page 2)
[84] Sohoni, N., Sanjabi, M., Ballas, N., Grover, A., Nie, S., Firooz, H., and Ré, C. (2021). Barack: Partially supervised group robustness with guarantees, arXiv preprint arXiv:2201.0000 (Cited on page 2, 7)
[85] Teney, D., Abbasnejad, E., Lucey, S., and Hengel, A. v. d. (2021). Evading the simplicity hias: Training a diverse set of models discovers solutions with superior ood generalization. arXiv preprint 2 105.056/2. (Cited on page 2, 28)
[86] Veitch, V., D' D'Amour, A., Yadlowsky, S., and Eisenstein, J. (2021). Counterfactual invariance to spurious correlations: Why and how to pass stress tests, arXiv preprint arXiv:2106. 00545. (Cited on page 3)
[87] Virtanen, P., Gommers, R., Oliphant, E.., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weekesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C. J., Polat, I., Feng. Y.. Moore, E. W., VanderPlas, J.. Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E. A., Harris, C. R., Archibald, A. M.. Ribeiro, A. H., Pedregosa, F., van Mulbregt. P., and SciPy 1.0 Contributors (2020). SciPy 1.0: Fundamental Algorithms for Scientific Computing Python. Nature Methods, 17:261-272 (Cited on page 18)
[88] Wah, C.,, Branson, S., Welinder, P., Perona, P., and Belongie, S. (2011). The caltech-uesd birds-200-2011 dataset. (Cited on page 18)
[89] Wang, X.,
Peng, Y., Lu, L., Lu, Z.. Bagheri, M., and Summers, R. M. (2017). Chestx-ray8: Hospital- scale chest x-ray database and benchmarks on weakly- classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097 2106. (Cited on page 4, 18)
[90] Wightman, R. (2019). pytorch-image wwwgammodeleeee (Cited on page 18)
Pytorch image models.
httts::///gttuub.oomrrugeg
[91] Williams, A.. Nangia, N., and Bowman, S. R. (2017). broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint 1704.0770..0544 (Cited on page 4, 29)
[92] Wolf, T., Debut, L., Sanh, V., Chaumond, J.. Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019), Huggingface's transformers: State-of-the-art natural language processing. arXiv (Cited on page 18)
preprint
[93] Xiao, K., Engstrrom, I.., Ilyas, A., and Madry, A. (2020), Noise or signal: The role of image backgrounds in object recognition. orXiv preprint arXiv:2006.0900 09994. (Cited on page 1,
[94] Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. (2017). Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1492 1500. (Cited on page 22)
[95] Zagonuyko, S. and Komodakis, N. (2016). Wide residual networks. arXiv preprint arXiv: 1rXiv: 1605.07146. (Cited on page 22)
[96] Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. (2021). Barlow twins: Self-supervised learning via redundancy reduction. International Conference on Machine Learning, pages (Cited on page 10, 22)
PMLR.
197] Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., and Oermann, E. K. (2018). Variable generalization performance of deep learning model to detect pneumonia in chest radiographs: a cross- sectional study. PLoS medicine, 15(1 15(11)=e1026683 (Cited on page 1,
[98] Zhang, H. Cisse, M., Dauphin, Y. N., and Lopez-Paz Paz, D. (2017). mixup: Beyond empirical risk minimiza- tion. arXiv preprint XrXiv:1710.09412. (Cited on page 22, 26)
15

---

[99] Zhang, J., Lopez-Paz, D., and Bottou, L. (2022). Rich feature construction for the optimization- generalization dilemma. arXiv preprint arXiv:2203 15516. (Cited page 6. 10)
[100] Zhang, M., Sohoni, N. S., Zhang, H. R., Finn, C.,, and Ré, C. (2021). Correct-n-contrast: A contrastive approach for improving robustness spurious correlations. In NeurlPS 2021 Workshop Distribution Shifts: Connecting Methods and Applications. (Cited on page 6, 7)
[101] Zhong, Z., Zheng, L.,, Kang, G., Li, S., and Yang. Y. (2020). Random erasing data augmentation. In Proceedings the AAAI conference on artificial intelligence, volume 34, pages 13001-13 13008.. (Cited on page 22, 26)
[102] Zhou, B., Lapedriza, A., Khosla, A.. Oliva, A., and Torralba, A. (2017). Places: A million image database for scene recognition. IEEE Irunsactions on paltern analysis and machine intelligence, 40(6):1452- 1464. (Cited on page 18)
16

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Limitations are described in the
appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Broader impact is descussed in the appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] No theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] No theoretical results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] We report errorbars the main experimental results in the appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We include an estimate of the total compute used in the appendix.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] We mention the license of all assets in the appendix.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] Not including new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] Not including new assets
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] Not including new assets
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A] No research on human subjects or crowdsourcing.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] No research on human subjects or crowdsoureing.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] No research human subjects crowd- sourcing.
17


Reviews:
[{"review_id": "yjAiE9JhXZG", "reviewer": "Reviewer_DoXQ", "summary": "This is primarily an empirical paper about learning in the presence of spurious correlations. They run a lot of tests that show training a model with empirical risk minimization (ERM) followed by deep feature reweighting (DFR) (which is retraining last layer on some on a held-out set that doesn't have spurious correlations) yields results that are not too different from group robustness training (Group DRO)", "questions": "My questions are all integrated into \"Strengths and Weaknesses\"", "limitations": "Not required here.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Thanks to the authors for the hard work on this paper. I like it overall, and think it is a valuable contribution.\n\nStrengths\n------------\n- A large amount of insightful experiments\n- Clear writing\n- Sensible comparisons and conclusions\n\nWeaknesses\n----------------\n- The conclusion of this paper hinges on the idea that: \"If two different methods get roughly the same held-out performance, then they must be learning the same kind of thing.\" I don't think that's necessarily true, and your experiments don't really prove it. The conclusion in lines 207-211 could be made stronger with other types of analyses. To have more certainty it's that \"better weighting of the learned features rather than learning better representations of the core features\" you could, for example, actually extract the representations learned from ERM+DFT and GDRO and see if they look similar. This experiment could be tricky, but if you make sure to have identical weight initialization, and run the experiment ~10 times, you could show that the feature spaces learned are really similar or not.\n- Why is early stopping important for RWY, RWG and GDRO? Are there some simple experiments you could do to elucidate this?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "r2HHULhF8mp", "reviewer": "Reviewer_CCN4", "summary": "This paper considers deep learning in the common case where the training data contains spurious correlations. The main takeaway is that empirical risk minimization (alone) is sufficient to obtain state-of-the-art performance; specialized group robustness methods do not appear to provide a significant benefit. This is demonstrated on six datasets spanning both vision and text problems. The effect of the architecture, pretraining strategy, and regularization is also considered.", "questions": "For all datasets considered, I assume the labeled spurious attributes form an *incomplete* labeling (there always exists some latent spurious attributes). How do latent spurious features impact the conclusions of the work?", "limitations": "I would have liked to see the scope of the paper defined a bit more clearly. In addition, while both vision and text datasets are used, the majority of the experiments are only on the image datasets.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Spurious correlations are a concern when fitting neural networks, so this paper tackles an important problem. Overall, I found the presentation to be quite good and the experiments fairly convincing. My main issue with the work in its current form is that the scope (and therefore potential impact) of the work is more limited than the title and introduction suggest: the spurious correlations studied are *labeled* properties of the inputs, rather than latent spurious features. In most cases, one does not have access to labeled attributes (or even class labels!) when fitting a neural network, and therefore this work has a narrower scope than expected.\n\nThis being the case, I think it is notable that specialized group robustness methods appear to perform no better than ERM when it comes to learning the in the presence of spurious correlations. In addition, the empirical observations regarding regularization and other effects of the base model are interesting, although many of them rely on DFR, which AFAIK is not peer reviewed.\n\nThe DFR procedure is somewhat similar to what's done in contrastive learning, e.g. \"Supervised Contrastive Learning,\" except there the second stage is performed on the original dataset. This often results in improved performance thanks to the contrastive objective. I'm curious how supervised contrastive learning would impact the results, both using the original dataset or the \"reweighted\" one.\n\nI would have liked to see the analysis of pretraining to include additional experiments with text / BERT.\n\nA smaller concern is that this paper leans heavily on Deep Feature Reweighting (DFR), which appears in a recent arxiv preprint (Kirichenko et al. [22]). Unfortunately, reading that pre-print is necessary to understand this work; reading the short description in S3 was not sufficient to follow along. It would be better to make this paper self-contained, especially given how simple the DFR idea is (e.g., define the reweighting dataset).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DrxN8TLFe_Y", "reviewer": "Reviewer_amrA", "summary": "This paper studies the reliance of deep learning models on spurious correlations. In particular, the authors look at the quality of feature representations learned by models trained via ERM versus models trained using group robustness methods. They evaluate these feature representations by utilizing the Deep Feature Retraining (DFR) procedure: retraining the last layer of the model on a held-out set which would likely not contain spurious correlations present in the training set. This procedure helps reveal how much information about causal factors is present in the learned representations. The authors further explore how these feature representations are influenced by the model architecture, pre-training task and strategy, regularization (via weight decay, data augmentation), training length, and whether or not model has been trained on the target data. They find that the quality of these representations often depends heavily on choice of data augmentation, model architecture, and the pre-training strategy involved, while regularization and training time may not be as helpful in improving the quality of said representations. The authors share results on CelebA, Waterbirds, WILDS-FMOW, CXR, MultiNLI, and CivilComments datasets.\n\nOverall, the paper presents interesting results and insights. I have some minor comments which I hope the authors will address during the response period.", "questions": "- Line 364: What changes in each of these three runs?\n- Regarding results on CXR: You suggest that the lack of improvement is because the model already learns the features correctly and DFR is unable to improve much upon it. Could it also be the case that this is one of the cases of failure modes for DFR?\n- Regarding effect of regularization on FMOW: Is it plausible that the trends are this way because models trained on FMOW are already regularized quite significantly (given the amount of noise in the images), and increasing regularization (through weight decay or MixUp) is weakening the signal further?", "limitations": "As this is an analysis paper, it is hard to understand the limitations and potential negative social impact, but I would urge the authors to think about potential negative impacts arising from misinterpretation of their analysis.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n- This work looks at an interesting and well motivated problem.\n- The experimental setup is well designed and results offer insights that would be useful to future researchers.\n- The paper is well written and organized.\n\n### Weaknesses:\n- The experiments on NLP datasets are based on a BERT model. While I understand that the goal here is not to create a state of the art model but to analyze representations learned by a model, significantly better models (DeBERTa, ERNIE, T5, etc) are out there that the authors could have used. \n- There are several works in NLP that have looked at the problem of spurious correlations ([1,2,3,4] are just a few examples), addressing them and understanding when models weigh causal features vs non causal features. The paper currently does not position itself well in that literature.\n\n### Additional comments:\n- Section 3, Preliminaries (Lines 99-102): This appears to be incorrect in light of [1]. In fact, as most Machine Learning tasks are anticausal, models will rely on spurious correlations regardless. As [1] show in their anticausal setup, a model will rely on spurious factors most of the time (unless the spurious features observe higher noise compared to the causal features).\n\n[1] Kaushik, D., Setlur, A., Hovy, E. H., & Lipton, Z. C. Explaining the Efficacy of Counterfactually Augmented Data. ICLR 2021.\n[2] Eisenstein, J. (2022). Uninformative Input Features and Counterfactual Invariance: Two Perspectives on Spurious Correlations in Natural Language. arXiv preprint arXiv:2204.04487.\n[3] Veitch, V., D'Amour, A., Yadlowsky, S., & Eisenstein, J. Counterfactual invariance to spurious correlations in text classification. NeurIPS 2021.\n[4] Kaushik, D., Hovy, E., & Lipton, Z. Learning The Difference That Makes A Difference With Counterfactually-Augmented Data. ICLR 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept