Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations
Authors: Polina Kirichenko
Pavel Izmailov
Andrew Gordon Wilson
Venue: ICLR 2023 notable top 25% (2023)
Keywords: spurious correlations
robustness

Abstract: Neural network classifiers can largely rely on simple spurious features, such as image backgrounds, to make predictions. However, even in these cases, we show that they still often learn core features associated with the desired attributes of the data, contrary to recent findings. Inspired by this insight, we demonstrate that simple last layer retraining can match or outperform state-of-the-art approaches on spurious correlation benchmarks, but with profoundly lower complexity and computational expenses. Moreover, we show that last layer retraining on large ImageNet-trained models can also significantly reduce reliance on background and texture information, improving robustness to covariate shift, after only minutes of training on a single GPU.

Published as a conference paper at ICLR 2023
LAST LAYER RE-TRAINING IS SUFFICIENT FOR ROBUSTNESS TO SPURIOUS CORRELATIONS
Polina Kirichenko New York University
Pavel Izmailov* New York University
Andrew Gordon Wilson New York University
ABSTRACT
Neural network classifiers can largely rely on simple spurious features, such as backgrounds, to make predictions. However, even in these cases, we show that they still often learn core features associated with the desired attributes of the data, contrary to recent findings. Inspired by this insight, we demonstrate that simple last layer retraining can match or outperform state-of-the-art approaches on spurious correlation benchmarks, but with profoundly lower complexity and computational expenses. Moreover, we show that last layer retraining on large ImageNet-trained models can also significantly reduce reliance on background and texture information, improving robustness to covariate shift, after only minutes of training single GPU.
INTRODUCTION
Realistic datasets in deep learning are riddled with spurious correlations - ppaterns that are predictive of the target in the train data, but that are irrelevant to the true labeling function. For example, most of the images labeled as butterfly on ImageNet also show flowers (Singla & Feizi 202T), and most of the images labeled as tench show a fisherman holding the tench (Brendel & Bethge 2019). Deep neural networks rely on these spurious features, and consequently degrade in performance when tested on datapoints where the spurious correlations break, for example, on images with unusual background contexts (Geirhos et al. 22200; RRoeefeld et al.) 2018; Beery et al. (018)). In an especially alarming example, CNNs trained to recognize pneumonia were shown to rely on hospital-specific metal tokens in the chest X-ray scans, instead of features relevant pneumonia (Zech et al. 2018).
In this paper, we investigate what features are fact learned on datasets with spurious correlations. We find that even when neural networks appear to heavily rely on spurious features and perform poorly on minority groups where the spurious correlation is broken, they still learn the core features sufficiently well. These core features, associated with the semantic structure of the problem, are learned even in cases when the spurious features are much simpler than the core features (see Section 4.2) and in some cases even when minority group examples present in the training data! While both the relevant and spurious features are learned, the spurious features can be highly weighted in the final classification layer of the model, leading poor predictions on the minority groups.
Inspired by these observations, we propose Deep Feature Reweighting (DFR), a simple and effective method for improving worst-group accuracy neural networks in the presence of spurious features. We illustrate DFR in Figure In DFR, we simply retrain the last layer of a classification model trained with standard Empirical Risk Minimization (ERM), using small set of reweighting data where the spurious correlation does not hold. DFR achieves offte-oftthe-a art performance on popular spurious correlation benchmarks by simply reweighting the features of trained ERM classifier, with no need to re-train the feature extractor. Moreover, we show that DFR can be used to reduce reliance on background and texture information and improve robustness to certain types of covariate shift in large- scale models trained on ImageNet, by simply retraining the last layer of these models. We note that the reason DFR can be so successful is because standard neural networks are in fact learning core features, even if they do not primarily rely on these features to make predictions, contrary to recent findings (Hermann & Lampinen 2020 [Shah et al. 2020). Since DFR only requires retraining a last layer, amounting to logistic regression, is extremely simple, easy to tune and computationally inexpensive relative to the alternatives, yet can provide state-of-the-art performance. Indeed, DFR can reduce tex- ture bias and improve robustness of large ImageNet trained models, in only minutes on a single GPU. Our code is available at gightu...b..... www/pol inakir aetaaaiiiihheee www.rriccceehheeee /eep feature reweighting
Equul contribution.

---

Published as a conference paper at ICLR 2023
Training Data
Train ERM
BG-Based Prediction
Reweighting Data
FG-Based Prediction
Feature Extractor
Spurious:
DFR
Core: FG
Retrain linear layer
BG Features
FG Features
Large weights
Small weights
Deep feature reweighting (DFR). An illustration of the DFR method on the Waterbirds dataset, where the background (BG) is spuriously correlated with the foreground (FG). Standard ERM classifiers learn both features relevant the background and the foreground. and weight them in way that the model performs poorly on images with confusing backgrounds. With DFR, we simply reweight these features by retraining the last linear layer on a small dataset where the backgrounds are not spuriously correlated with the foreground. The resulting DFR model primarily relies on the foreground, and performs much better on images with confusing backgrounds.
Figure 1:
2 PROBLEM SETTING
We consider classification problems, where assume that the data consists of several groups G. which are often defined by combination of label and spurious attribute. Each group has its own data distribution pa (ar,, y), and the training data distribution p(z, = Σ, osp, (x, y). where a, is the proportion of group Gj in the data. For example, in the Waterbirds dataset (Sagawa al., 2019), the task classify whether an image shows a landbird or waterbird. The groups correspond to images of waterbirds on water background (G1), waterbirds on land background (G2), landbirds on water background (9g) and landbirds on land background (94). See Figure for visual description of the Waterbirds data. We will consider the scenario when the groups are not equally represented in the data: for example, on Waterbirds the sizes of the groups are 3498, 184, 56 and 1057, respectively. The larger groups G1, G4 are referred to as majority groups and the smaller G2, G3 are referred to as minority groups. As a result of this heavy imbalance, the background becomes spurious feature, i.e. it is a feature that is correlated with the target the train data, but it is not predictive of the target on the minority groups. Throughout the paper we will discuss multiple examples of spurious correlations in both natural and synthetic datasets. In this paper, we study the effect of spurious correlations on the features learned by standard neural networks, and based on our findings propose simple way of reducing the reliance on spurious features assuming access to small set of data where the groups are equally represented.
mixture of the group distributions
RELATED WORK
Feature learning in the presence of spurious correlations. The poor performance of neural networks datasets with spurious correlations inspired research in understanding when and how the spurious features are learned. Geirhos et al. (2020) provide a detailed survey of the results in this area. Several works explore the behavior of maximum-manne
classifiers, SGD training dynamics and inductive biases of neural network models in the presence of spurious features (Nagarajan et al.) 2020 Pezeshki et al., 202T Rahaman et al.) 2019). Shah et al. (2020) show empirically that in certain scenarios neural networks can suffer from extreme simplicity bias and rely on simple spurious features, while ignoring the core features; in Section[4.2) we revisit these problems and provide further discussion. Hermann & Lampinen (2020) and Jacobsen et al. (2018) 8) also show synthetic and natural examples, where neural networks ignore relevant features, and Sciieeaa et al. (202T) explore which types of shortcuts more likely to be learned. Kolesnikov & Lampert (2066) on the other hand show that on realistic datasets core and spurious features can often be distinguished from the latent representations learned by neural network in the context of object localization.
Group robustness. The methods achieving the best worst-group performance typically build on the distributionally robust optimization (DRO) framework, where the worst-case loss minimized instead of the average loss (Ben- Tal et al. 2013 Hu et al. 2018, (Sagawa et al.) 2019 Oren et al.) 2019 Zhang et al.) 2020). Notably, Group DRO (Sagawa et al., 2019), which optimizes a soft version of the worst-group loss holds state-of-the-arr results multiple benchmarks with spurious correlations.
2

---

Published as a conference paper at ICLR 2023
Several methods have been proposed for the scenario where group labels are not known, and need to be inferred from the data; these methods typically train a pair of networks, where the first model is used to identify the challenging minority examples and define weighted loss for the second model (Liu et al., 2021; Nam et al. 2020, Yaghoobzadeh et al. 2019 Utama et al., 2020 Creager et al.) [2021 Dagaev al. 2021] Zhang al,] 2002). Other works proposed semi-supervised methods for the scenario where the group labels are provided for a small fraction of the train datapoints (Sohoni ct al. 202T; Nam et al. 2022), Idrissi et al. (202T) recently showed that with careful tuning simple approaches such as data subsampling and reweighting can provide competitive performance. We note that all of the methods described above use a validation set with a high representation of minority groups to tune the hyper-parameters and optimize worst-group performance.
In closely related work, Menon et al. (2020) considered classifier retraining and threshold correction on a subset of the training data for correcting the subgroup bias. In our work, we focus on classifier retraining and show that re-using train data for last layer retraining done in MMnon et al. (2020) is suboptimal, while retraining the last layer on held-out data achieves significantly better performance across various spurious correlations benchmarks (see Section 6). Moreover, we make multiple contributions beyond the scope of Menon et al. (2020), for example: we analyze feature learning in the extreme simplicity bias scenarios (Section 4) show strong performance on natural language processing datasets with spurious correlations (Section 6)) and demonstrate how last layer retraining can be used correct background and texture bias in large-scale models on ImageNet (Section 7
In an independent and concurrent work, Rosenfeld et al. (2022) show that ERM learns high quality representations for domain generalization, and by training the last layer on the target domain it is possible to achieve strong results. Kang et al. (2019) propose to use classifier re- re-training, among other methods, in the context of long-tail classification. The observations in our work are complimentary, as we focus on spurious correlation robustness instead of domain generalization and long-tail classifica- tion. Moreover, there are important algorithmic differences between DFR, and the methods of Kang et al.] (2019) and Rosenfeld et al. (2022). In particular, Kang et al.] (2019) retrain the last layer on the training data with Jass-balanced data sampling, while we use held-out data and group-balances subsampling; they also do not use regularization. [Rosenfeld et al. (2022) only present last layer retraining results as motivation, and do not use their proposed DARE method. We present a detailed discussion of these differences in Appendix [A] and empirical comparisons in Appendix G Due to the page limit, we provide further discussion of prior work in spurious correlations, transler learning and other related areas in Appendix A
4 UNDERSTANDING REPRESENTATION LEARNING WITH SPURIOUS CORRELATIONS
In this section we investigate the solutions learned by standard ERM classifiers on datasets with spurious correlations. We show that while these classifiers underperform on the minority groups, they still learn the core features that can be used to make correct predictions the minority groups.
4.1 FEATURE LEARNING ON WATERBIRDS DATA
We first consider the Waterbirds dataset (Sagawa et al. 2019) (see Section 2 which is generated synthetically by combining images of birds from the CUB dataset (Wah
Train Data
Test Data (Worst Acc)
(Spurious Corr.)
Original
FG-Only
cet al. 2011) and backgrounds from the Places dataset (Zhou et al. 2017) (see (Sggawa al. 2019 for detailed description of the data generation process). For the exper- iments in this section, we generate several variations of the Waterbirds data following the procedure analogous to Sagawa et al. (2019). The Original dataset is analogous to the standard Waterbirds data, but we vary the degree of spurious correlation between the background and the target: 50% (Balanced dataset), 95% (as in Sagawa et al. (2019) and 100% (no minority group examples in the train dataset). The FG-Only dataset contains images of the birds on uniform grey background instead of the Places background, removing the spurious feature. We show examples of datapoints from each variation of the dataset in Appendix Figure 4 In Appendix B.I we provide the full details for the
Balanced (50%)
91.9%
94.7%
Original (95%)
73.8%
93.7%
Original ( (100%)
38.4%
94%
FG-Only (-)
75.2%
95.5%
Table 1: ERM classifiers trained on Wa- terbirds with Original and FG- FG-Only im- ages achieve similar FG-Only accuracy.
3

---

Published as a conference paper at ICLR 2023
worss group
99
99
99
192.99
41
06
07
77
acc. 1%
0
0
0
spur, corT. strength
100%
99%
95%
100%
99%
95%
100%
99%
95%
MNIST MNIST
MNIST
Fashion
MNIST CIFAR
Original Aoc.
Core-Only Acc.
Decoded Acc.
Optimal Acc.
Figure 2: Feature learning and simplicity bias. ResNet-20 ERM classifiers trained on Dominoes data with varying levels of spurious correlation between and spurious features. We show worst-group test accuracy for: Original data, data with only core features present (Core-Only), and accuracy of decoding the core feature from the latent representations of the Original data with logistic regression. We additionally report optimal accuracy: accuracy of a model trained and evaluated on the Core-Only data. Even in cases when the model achieves 0% accuracy the Original data, the core features can still be decoded from latent representations.
experiment in this section, and additionally consider the reverse Waterbirds problem, where instead of predicting the bird type, the task is to predict the background type, with similar results.
We train a ResNet-50 model with ERM on each of the variations of the data and report the results in Table Following prior work (e.g., Sagawa et al., 2019: [iu et al., 2022 Idrissi et al. 202T). we initialize the model with weights pretrained on ImageNet. For the models trained on the Original data, there a large difference between the mean and worst group accuracy on the Original test data: the model heavily relies on the background information in its predictions, so the performance on minority groups is poor. The model trained without minority groups is especially affected, only achieving 38.4% worst group accuracy. However, surprisingly. on the Original data can make much better predictions on the FG-Only test data: we remove the spurious feature (background) from the inputs at test time, the models make predictions based on the core feature (bird), and achieve worst-group accuracy close to 94%, which is only slightly lower than the accuracy of model trained directly the FG-Only data and comparable to the accuracy of a model that was trained on balanced train d
find that all the models trained
In summary, we conclude that while the models pre-trained on ImageNet and trained on the Original data make use of background information make predictions, they still learn the features relevant to classifying the birds almost as well as the models trained on the data without spurious correlations. We will see that we can retrain the last layer of the Original-trained models and dramatically improve the worst- grosu performance on the Original data, by emphasizing the features relevant to the bird. While we use pre- trained models in this section, following the common practice on Waterbirds, we show similar results other datasets without pre-training (see Section 4.2 Appendix C.I
4.2 SIMPLICITY BIAS
Shah et al. (2020) showed that neural networks can suffer from extreme simplicity bias, a tendency to completely rely on the simple features, while ignoring similarly predictive (or even more predictive) complex features.
this section, we explore whether the neural networks can still learn the core features in the extreme simplicity bias scenarios, where the spurious features correlated with the target. Following Shah et al. (2020) and Pagliardini et al. (002)., we consider Dominoes binary classification datasets, where the top half of the image shows MNIST digits (LeCun et al. 1998) from classes (0. 1). and the bottom half shows MNIST images from classes (7. 9) AMNIST-MNIST FaSHion-MNIST (Xiao al. 2017) images from classes (coat, dress) (MNIST- Fashion) or CIFAR- (Krizhevsky et al. 2209) images from classes (car, truck) (MNIST-CIFAR) In all Dominoes datasets, the top half of the image (MNIST 0- images) presents linearly separable feature; the bottom half of the image presents a harder to learn feature. See Appendix B.2 for more details about the experimental set-up and datasets, and Appendix Figure for image examples.
simple and highly
'On the FG-Only data the groups only differ by the bird type, as remove the hackground. The difference between mean and worst-group accuracy because the target classes are not balanced the training data. In Appendix [B.4, we demonstrate logit additivity: the class logits Waterbirds are well approximated as the sum of the logits for the corresponding hackground image and the logits for the foreground image.
4

---

Published as a conference paper at ICLR 2023
We use the simple feature (top half of the images) spurious feature and the complex feature (bottom half) as the core feature; we generate datasets with 100%. 99% and 95% correlations between the spurious feature and the target in train data, while the core feature is perfectly aligned with the target. We then define 4 groups Gj based on the values of the spurious and core features, where the minority groups correspond to images with top and bottom halves that do not match. The groups on validation and test are balanced.
We train ResNet-20 model on each variation of the dataset. In Figure 2 we report the worst group performance for each of the datasets and each spurious correlation strength. In addition to the worst-gnn worst-group accuracy on the Original test data, we report the Core- Care-Only worst- worst-group accuracy, where we evaluate the model on datapoints with the spurious top half of the image replaced with a black image. Similarly to [Shah al. (220)) and Pagliardiii spurious features are perfectly correlated with the target on Dominoes datasets, the model relies just on the simple spurious feature to make predictions and achieves 0% worst-group accuracy. However, with 99% and 95% spurious correlation levels on train, we observe that models learned the core features well, as indicated both by their performance on the Original test data and especially increased performance on Core-Only Only test data where spurious features are absent. For reference, on each dataset we also report the Optimal accuracy, which is the accuracy of a model trained and evaluated on the Core-Only data. The optimal accuracy provides an upper bound on the accuracy that we can expect on each of the datasets.
al. (2022), observe that when the
Decoding feature representations. The performance on the Original and even Core-Only data might not be fully representative of whether or not the network learned high-quality representation of the core features. Indeed, even if we remove the MNIST digit from the top half of the image, the network can still primarily rely on the (empty) top half in its predictions: an empty image may be more likely to come from class 1, which typically has fewer white pixels than class O. To see how much information about the core feature contained in the latent representation, we evaluate the decoded accuracy: for each problem we train logistic regression classifier on the features extracted by the final convolutional layer of the network. We use group-balanced validation set to train the logistic regression model, and then report the worst-group accuracy on test set. In Figure 2 we observe that for MNIST-MNIS' and 100%, reweighting the features leads to high worst group test accuracy. Moreover, all Dominoes datasets for 99% and 95% spurious correlations level the core features can be decoded with high accuracy and almost match the optimal accuracy. This decoding serves as basis of our DFR method, which we describe in detail in Section 5 In Appendix [B.2 we report an additional baseline and verify that the model indeed learns non-trivial representation of the core feature.
even when the spurious correlation is
Relation to prior work. With the same Dominoes datasets that we consider in this section, Shah et al. (2020) showed that neural networks tend to rely entirely on the simple features. However, they only considered the 100% spurious correlation strength and accuracy on the Original test data. Our results do not contradict their findings but provide new insights: even in these most challenging cases, the networks still represent information about the complex core feature. Moreover, this information can be decoded achieve high accuracy on the mixed group examples. Hermann & Lampinen (2020) considered different set of synthetic datasets, showing that in some cases neural networks fail to represent information about some of the predictive features. In particular, they also considered decoding the information about these features from the latent representations and different spurious correlation strengths. Our results add to their observations and show that while it is possible to construct examples where predictive features are suppressed, in many challenging practical scenarios, neural networks learn a high quality representation of the core features relevant to the problem even if they rely on the spurious features.
ColorMNIST. In Appendix B.3 we additionally show results on ColorMNIST dataset with varied spurious correlations strength in train data: decoding core features from the trained representations on this problem also achieves results close to optimal accuracy and demonstrates that the core features are learned even in the cases when the model initially achieved 0% worst-group accuracy.
In summary, we find that, surprisingly. if (1) the strength of the spurious correlation is lower than 100% or (2) the difference in complexity between the core and spurious features not as stark as on MNIST-CIFAR, the core feature can be decoded from the learned embeddings with high accuracy.
5

---

Published as a conference paper at ICLR 2023
5 DEEP FEATURE REWEIGHTING
In Section 4 we have seen that neural networks trained with standard ERM learn multiple features relevant to the predictive task, such as features of both the background and the object represented in the image. Inspired by these observations, we propose Deep Feature Reweighting (DFR), a simple and practical method for improving robustness to spurious correlations and distribution shift.
Let us assume that we have access to a dataset D = {I,, Vi which can exhibit spurious correlations. Furthermore, we assume that we have access to (typically much smaller) dataset D, where the groups are represented equally. D can be subset of the train dataset D, separate set of datapoints. We will refer to DD reweighting dataset. We start by training neural network on all of the available data 'D with standard ERM without any group or class reweighting. For this stage we do not need any information beyond the training data and labels. Here, we assume that the network consists of a feature extractor (such as a sequence of convolutional or transformer layers), followed by fully-connected classification layer mapping the features to class logits. In the second stage of the procedure, we simply discard the classification head and train new classification head from scratch on the available balanced data D. We use the new classification head to make predictions on the test data. We illustrate DFR in Figure Notation: we will use notation DFR where D is the dataset used to train the base feature extractor model and D - to train the last linear layer.
6 FEATURE REWEIGHTING IMPROVES ROBUSTNESS
In this section, we evaluate DFR on benchmark problems with spurious correlations.
Data. See Section 2 for description of the Waterbirds data. On CelebA hair color prediction(Liu ctt al. 2015), the groups are non-blond females (G)) ) blond females (G2), non-blond males (Ga) and blond males (Gu) with proportions 44%, 14%, 41%. and 1% of the data, respectively; the group G4 is the minority group, and the gender serves as a spurious feature. On MulriNLI, the goal is to classify contradiction, entailment or
the relation between a pair of sentences (premise and hypothesis) neutral; the presence of negation words ("no", "never", is correlated with the contradiction class, and serves spurious feature. Finally, we consider the CivilComments (Borkan et al.. 2019) dataset implemented in the WILDS benchmark (Koh et al.) 2021 Sagawa et al.) 2021). where the goal is to classify comments as toxic or neutral. Each comment labeled with 8 attributes sy (male, female, LGBT, black, white, Christian, Muslim, other religion) based on whether or not the corresponding characteristic is mentioned in the comment. For evaluation, we follow the standard WILDS protocol and report the worst accuracy across the 16 overlapping groups corresponding to pairs (y. s;) for each of the attributes 8,. See Figures 6 7 for a visual description of all four datasets.
Baselines. We consider 6 baseline methods that work under different assumptions on the information available at the training time. Empirical Risk Minimization (FRM) represents conventional training without any procedures for improving worst-group accuracies. Just Train Twice (JTT) (Liu et al. 202T) is a method that detects the minority group examples on train data, only using group labels on the validation set to tune syper-parameters. Correct-n- Contrast (CnC) (Zhang al. 2022) detects the minority group examples similarly to JTT, and uses a contrastive objective to learn representations robust spurious correlations. Group DRO (Sagawa et al.) 2019) is state-of-the-art method, which uses group information on train and adaptively upweights worst-group examples during training. SUBG is ERM applied to a random subset of the data where the groups are equally represented, which was recently shown to be a surprisingly strong baseline (Idrissi et al. 202I). Finally, Spread Spurious Attribute (SSA) (Nam al., 2022) attempts to fully exploit the group- validation data with semi- supervised approach that propagates the group labels to the the training data. We discuss the assumptions on the data for each of these baselines in Appendix C.2
DFR. We evaluate DFR where we use group- group-balanced Subae of the validation data available for each of the problems as the reweighting dataset D. We use the standard training dataset (with group imbalance) D to train the feature extractor. In order to make use of more of the available data, we train logistic regression 10 times using different random balanced subsets of the data, and average the weights of the learned models. We report full details and several ablation studies in Appendix C
We keep all of the data from the smallest group. and subsample the data from the other groups to the same size. On CivilComments the groups overlap, for each datapoint we use the smallest group that contains it when constructing the graupebalanced reweighting dataset.

---

Published as a conference paper at ICLR 2023
Group Info
Waterbirds
CelebA
MultiLI
CivilComments Mean(S)
Method
Train / Val
Worst(S)
Meun(
(orst(S))
Meun((S)
Worst(S)
Mean(S)
Worst(S:)
JTT
xIV
86.7
93.3
81.1
88.0
72.6
78.6
69.3
91.1
CnC
XIV
88.5
90.9
88 820.9
80.920.5
68.9
81.720.5
SUBG
VIV
89.111.1
85.612.
68.920.8
SSA
X144
89.0+0.6
92. 220.
89.8 19.8-1.
92.8+a.1
76.6+0.7
79.9
69.9
88.2+2.
Group DRO
VIV
91.4
93.5
88,9
92.9
77.7
81.4
69.9
88,9
Base (ERM)
X
74.9+2. 4
98. 1
16. 16.9+2.8
95.320
65.920.3
82 840.
55.6.0.6
=2.1=0.1
DHR
XIVV
92.9
94.240.4
88.3
91.310.3
74.7g0-
82.1
70.1 10.1......
87.2g00
Table 2: Spurious correlation benchmark results. Worst-group and mean test accuracy of DFR variations and baselines on benchmark datasets. For mean accuracy, we follow Sagawa et al. (2019) and weight the group accuracies according to their prevalence in the training data. The Group Info column shows whether group labels are available to the methods on train and validation datasets. DFR Val uses the validation data to train the model parameters (last layer) in addition to hyper- parameter tuming. indicated with vV: SSA also uses the validation set to train the model. For DFR we report the ttd over independent runs. DFR is competitive with state-of-the-ar..
Following prior work (c.g. Liu et al. 2021). we use ResNet-50 model (He al. 2016) pretrained on ImageNet for Waterbirds and CelebA and BERT model (Devlin et al. 2018) pre-trained on Book Corpus and English Wikipedia data for MultiNLI and CivilComments.
Syper--arameters.
Tr the size of the reweighting set D is small relative to the number of features produced by the feature extractor. For this reason, we use errreuularization to allow the model to learn sparse the strength of the regularization term. We tune all the perrappareeteess on the validation data provided with each of the datasets. We note that the prior work methods including Group DRO, JTT. CnC, SUBG, SSA and others extensively tune hyper-pas parameters on the validation data. For DFR Val we split the validation in half, and use one half to tune the eegularization strength parameter; then, retrain the logistic regression with the optimal regularization on the full validation set.
For DFR Val
solutions and drop irrelevant features. For DFR Val Ir we only tune single hyper-parameter
Results. We report the results for DFR Val and the baselines in Table DFR is competitive with the state-of-the-art Group DRO across the board, achieving the best results among all methods on Waterbirds and CivilComments. Moreover, DFR achieves similar performance to SSA, a method designed to make optimal use of the group information on the validation data (DFR Val is slightly better on Waterbirds and CivilComments while SSA is better on CelebA and MultiNLD,, Both methods use the same exact setting and group information to train and tune the model. We explore other variations of DFR in Appendix
To sum up, D Val matches the performance of the best available methods, while only using the group labels on small validation set. This sfate-of-the-art performance is achieved by simply reweighting the features learned by standard ERM, with no need for advanced regularization techniques.
7 NATURAL SPURIOUS CORRELATIONS ON IMAGENET
In the previous section we focused on benchmark problems constructed to highlight the effect of spurious features. Computer vision classifiers are known to learn undesirable patterns and rely on spurious features in real-worlh problems (see e.g. Roseefelld et al., 2018; (Singla & Feizi 1021 Geirhos et al. 2020). In this section we explore two prominent shortcomings of ImageNet classifiers: background reliance (Xiao et al. 2020) and texture bias (Geirhos et al.) 2018).
7.1 BACKGROUND RELIANCE
Prior work has demonstrated that computer vision models such as ImageNet classifiers can rely on image background to make their predictions (Zhang et al., 2007; Ribeiro et al.] 2016, [Shetty al. 2019 Xiao et al. 2020 Singla & Feizi 2021). Here, we show that it is possible to reduce the background reliance of ImageNet-traime
models by simply retraining their last layer with DFR.
Xiao et al. (2020) proposed several datasets in the Backgrounds Challenge to study the effect of the backgrounds on predictions of ImageNet models. The datasets in the Backgrounds Challenge based the ImageNet-9 dataset, subset of ImageNet structured into 9 coarse coarse-grain classes (see Xiao et al. (2020) for details). ImageNet-9 contains 45k training images and 4050 validation images. We consider three datasets from the Backgrounds Challenge: (1) Original contains the
7

---

Published as a conference paper at ICLR 2023
ImageNet-9
Mixed-Rand
FG-Only
Paintings-BG
ImageNet-R
Validaiion Accurccy
0.98
0.925
0.90
0.900
0.925
0.50
0.94
0.875
0.85
14 Sk 10A
20A
454
16 5A 10A
20A
464
14 SA 10A 20k 45k
14 54
10k 204 45k
14 5k
10A
20A 45A
Mised-Rand Data
# Mand
Mised-Rand Data
Moxdd-Rand Data
Mixcd-Rand Data
DFR
DFROGIMW
Baseline
Figure 3: Background reliance. Accuracy of DFR and DFR on different ImageNet-9 validation splits with an ImageNet-trained ResNet-50 feature extractor. DFR reduces background reliance with minimal drop in performance the Original data.
original images; (2) Mixed-Rand contains images with random combinations of backgrounds and foregrounds (objects); (3) FG-Only contains images showing just the object with a black background. We additionally consider Paintings- BG using paintings from Kaggle's pa painm mainteree inttee-by-uuubee rs dataset ( (https: www kaggle........ com/
/) as background for the ImageNet-9 validation data. Finally, we consider the ImageNet-R dataset (Hendrycks et al.) ( restricted the ImageNet-9 classes. See Appendix D for details and Figure 11 for example images.
We use an ImageNet-trained ResNet-50 as feature extractor and train DFR with different reweighting datasets. As a Baseline, we train DFR on the Original 45A: training datapoints (we cannot use the ImageNet-trained ResNet-50 last layer, as ImageNet-9 has a different set classes). We train DFRMR and DFROG+MR on Mixed- Rand training data and combination of Mixed- finedd-Rand and Original training data respectively. In Figure 3 we report the predictive accuracy of these methods on different validation datasets function of the number of Mixed-Rand data observed. We select the observed subset of the data randomly; for OG+MR in each case we use the same amount of Mixed-Ran and Original training datapoints. We repeat this experiment with a VIT-B-16 model (Dosovitskiy et al. 2020) pretrained on ImageNet-21 k and report the results in the Appendix Figure
First, we notice that the baseline model provides significantly better performance on FG-Only (92%) than on the Mixed-Rand (86%) validation set, suggesting that the feature extractor learned the features needed to classify the foreground. as well as the background features. With access to Mixed- Rand data, we can reweight the foreground and background features with DFR and significantly improve the performance on mixed-rand, FG-Only and Paintings-BC datasets. At the same time, OFRR is able to mostly maintain the performance on the Original ImageNet-9 data; the small drop in performance is because the background is relevant to predicting the class on this validation set. Finally, on ImageNet-R, DFR provides a small improvement when we use all of 45k datapoints; the covariate shift in ImageNet-R is not primarily background-based does not provide a big improvement. We provide additional results in See Appendix
reducing background reliance
7.2 TEXTURE-VS-SIE
BLAS
Geirhos et al. (2018) showed that images with conflicting texture and shape, Nettraied CNNs tend to make predictions based on the texture, while humans usually predict based on the shape of the object. The authors designed the GST dataset with conflicting cues, and proposed the term texture bias to refer to the fraction of datapoints on which a model (or human) makes predictions based on texture; conversely, shape bias is the fraction of the datapoints on which prediction made based on the shape of the object. Geirhos et al.) (2018) 8) showed that it is possible to increase the shape bias of CNNs by training on Stylized ImageNet (SIN), a dataset obtained from ImageNet by removing the texture information via style transfer (see Appendix Figure] I1 for example images). Using SIN in combination with ImageNet (SIN+IN), they also obtained improved robustness to corruptions. Finally, they proposed the Shape-RN-SS model. ResNet-50 (RN-50) trained on SIN+IN and finetuned on IN, which outperforms the ImageNet- trained ResNet-50 on in-distribution data and out-of-distribution robustness.
Here, explore whether it is possible to change the shape bias of ImageNet-traine models by simply retraining the last layer with DFR. Intuitively, we expect that the standard ImageNet-trand trained model already learns both the shape and texture features. Indeed, Hermann et al. (2020) showed that
8

---

Published as a conference paper at ICLR 2023
Top-1 Acc (%)
Training Data
Shape bias (%)
Method
ImageNet
ImageNet-R
ImageNet-C
IN
21.4
76.0
23.8
39.8
RN-50
SIN
81.4
60.3
26.9
38.1
IN+SIN
34.7
74.6
27.6
45.7
Shape-RN RN-50
IN+SIN
20.5
76.8
25.6
42.3
SIN
34.0
65.1
24.6
36.7
DFR
IN+SIN
30.6
74.5
27.2
40.7
Table Shape bias. Shape bias and accuracy on ImageNet validation set variations for ResNet-50 trained on different datasets and DFR with an ImageNet-trained ResNet-50 as a feature extractor. For each metric, we show the best result in bold. By retraining just the last layer with DFR, we can significantly increase the shape bias compared to the base model (21.4% + 34% for DFR(SIN)) and improve performance on ImageNet-R/C.
shape information is partially decodable from the features of ImageNet models on the GST dataset. Here, instead of targeting GST, we apply DFR to the large-scale SIN dataset, and explore both the shape bias and the predictive performance of the resulting models. In Table 3 report the shape bias, as well as predictive accuracy ResNet-50 models trained on ImageNet (IN), SIN, IN+SIN and the Shape-RN RN-50 model, and the DFR models trained on SIN and IN+SIN. The DFR models use an IN-trained ResNet-50 model as a feature extractor. See Appendix for details.
First, we observe that while the SIN-trained RN-50 achieves shape bias of 81.4%, reported by Geirhos et al.) (2018), the models trained on combinations of IN and SIN are still biased towards texture. Curiously, the Shape-RN-50 model proposed by Geirhos et al. (2018) has almost identical shape bias to standard IN-trained RN-50! At the same time, Shape- RN-50 outperforms IN-trained RN-50 on all the datasets that we consider, and Geirhos al. (0188) showed that Chape-RN-50 significantly outperforms IN-trained RN-50 in transfer learning to a segmentation problem, suggesting that it learned better shape-based features. The fact that the shape bias of this model lower than that of an IN-trained RN-50, suggests that the shape bias is largely affected by the last linear layer of the model: even if the model extracted high-quality features capturing the shape information, the last layer can still assign a higher weight to the texture information.
Next, DFR can significantly increase the shape bias of an IN-trained model. DER achieves a comparable shape bias to that of a model trained from scratch on a combination of IN and SIN datasets. Finally, DFR improves the performance on both ImageNet-R and ImageNet-C (Hendrycks & Dietterich 2019) datasets compared to the base RN-50 model. In the Appendix Table we show similar results for a VIT-B-16 model pretrained on ImageNet-21 k and finetuned on ImageNet; there, DNSIN can also improve the shape bias and performance ImageNet-C. but does not help ImageNet-R. To sum up, by reweighting the features learned by an ImageNet-trained model, we can significantly increase its shape bias and improve robustness certain corruptions. However, to achieve the highest possible shape bias, is still preferable to re-train the model from scratch, as RN-50(SIN) achieves a much higher shape bias compared all other methods.
DISCUSSION
We have shown that neural networks simultaneously learn multiple different features, including relevant semantic structure, even in the presence spurious correlations. By retraining the last layer of the network with DFR, we can significantly reduce the impact of spurious features and improve worst-group-Pee coost-group-perfommaane performance of the models. In particular, DFR achieves state-of-the-a performance on spurious correlation benchmarks, and can reduce the reliance of ImageNet trained models on background and texture information. DFR is extremely simple, cheap and effective: it only has hyper-pare parameter, and we can run it on ImageNet- our results will be useful to practitioners and will inspire further research in feature learning in the presence of spurious correlations. We provide further discussion in Appendix L
data in a matter of minutes. We hope that

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We would like thank Micah Goldblum, Wanqian Yang, Mare Finzi, Wesley Maddox, Robin Schirrmeister, Yogesh Balaji, Timur Garipov and Vadim Bereznyuk for helpful comments. This research is supported by NSF CAREER IIS- 2144492, NSF I-DISRE 193471, NIH ROIDA048764- 0IAI, NSF 198-1910266, NSF 1922658 NRT-HDR: FUTURE Foundations, Translation, and Re- sponsibility for Data Science, NSF Award 1922658, Meta Core Data Science, Google Al Research, BigHat Biosciences, Capital One, and an Amazon Research Award. This work was supported in part through the NYU IT High Performance Computing resources, services, and staff expertise.
REFERENCES
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of 2021.
large scale pre-training, arXiv preprint
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A reductions approach to fair classification. International Conference on Machine Learning, pp- 60 60-69. PMLR, 2018.
Michael A Alcom, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
4845 4854, 2019.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz, Invariant risk minimization. arXiv preprint arXiv: 1907.02297 2019.
Benjamin Aubin, Agnieszka Stowik, Martin Arjovsky, Leon Bottou, and David Lopez-Paz. Linear unit-tests for invariance discovery. arXiv preprint arXiv:2102.10867 2021.
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Scong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning, pp. 528 539. PMLR, 2020.
Sara Beery, Grant Van Horn, and Pictro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456-473, 2018.
Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2):341-357, 2013.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. Advances in neural information processing systems, 24, 2011.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.0720 072558. 2021.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended hias with real data for text classification. In Companion proceedings of the 2019 world wide web conference, pp. 491-500, 2019.
Wieland Brendel and Matthias Bethge. Approximating cans with bag-of-local-featuree models works surprisingly well on imagenet. arXiv preprint (00ivv:994400000
2019.
Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. Advances in neural information processing systems, 32, 2019.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distrihutionnaaaar margin loss. Advances in neural information processing systems, 32, 2019.
Kaidi Cao, Yining Chen, Junwei Lu, Nikos Arechiga, Adrien Gaidon, and Tengyu Ma. Heteroskedas- tic and imbalanced deep learning with adaptive regularization. arXiv preprint 27Xiv 2020.
10

---

Published as a conference paper at ICLR 2023
Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. International Conference on Machine Learning, pp. 2189-2200. PMLR, 2021.
Nikolay Dagaev, Brett Roads, Xiaoliang Luo. Daniel N Barry, Kaustubh R Patil, and Bradley C Love. A too-good-to-bettru prior to reduce shortcut reliance. arXiv preprint arXiv:2102.0644 2021.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprini urXiv:/810.0450 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010.11929, 2020.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations theoretical computer science conference, pp. 214-226, 2012.
Jacob Eisenstein. Informativeness and invariance: Two perspectives spurious correlations in natural language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4326-4331, 2022.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 1180-1189, PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky, Domain-adversane training of neural networks. The journal of machine learning research, (7(1):209-2003) 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet- trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: arXiv:1811.12255 2018.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1):65665550 673,66733622 2020.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. In Advances in Neural Information Processing Systems 34, 2021.
Jacob Gildenblat and contributors. Pytorch library for cam methods. https: http//ggttttttttte com/
jacobgil/pytorch grac- 2021.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:200.001434, 2020.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre
Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585 September 2020. doi: 10.1038/<41586- -2000222292220222--9999 2649-2.
URL //tp:i 0

---

Published as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377,, 2021.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint
2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler 7hu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of dittrf-dittribution generalization. ICCV, 2021.
Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems, 33:9995-10006, 2020.
Katherine Hermann, Ting Chen, and Simon Korbllith. The origins and prevalence of texture bias in convolutional neural networks. Advances in Neural Information Processing Systems, 33: 19000-19015, 2020.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pp. 2029-2037. PMLR, 2018.
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learning? arXiv preprint 160ivv6608.0000 2016.
J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9()): 90-95, 2007. doi: 10.1 101100CC.E22000.555
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-accuracy arXiv preprint arXiv:2110.14500 2021.
Md Amirul Islam, Matthew Kowal, Patrick Esser, Sen Jia, Bjorn Ommer, Konstantinos G Derpanis, and Neil Bruce. Shape or texture: Understanding discrininative features in cnns. arXiv preprint arXiv:2101.11604, 2021.
förn-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance causes adversarial vulnerability. arXiv preprint 1eIiiv:811..000 2018.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng. and Yannis Kalantidis, Decoupling representation and classifier for long-tailed recognition. arXiv preprint 19100:9110 2019.
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. Learning the difference that makes difference with countefaccualll--aggmennt data. International Conference on Learning Representations, 2020. http://///pppeeeeeeeeeeeeee/rrrs
Divyansh Kaushik, Amrith Setlur, Eduard Hovy, and Zachary C Lipton. Explaining the efficacy of counterfactually augmented data. International Conference on Learning Representations (ICLR), 2021.
Fereshte Khani, Aditi Raghunathan, and Percy Liang. Maximum weighted loss discrepancy. arXiv preprint arXiv:I906.0333 2019.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not leam: Training deep neural networks with biased data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9012-9020, 2019.
12

---

Published as a conference paper at ICLR 2023
Jon Kleinberg. Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determi- nation of risk scores. arXiv preprint (rXiv:16090058000 2016.
Thomas Kluyver, Benjamin Kellany Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, and Carol Willing. Jupyter notebooks
publishing format for reproducible Loizides and B. Schmidt (eds.), ), Positioning and Power in Academic
computational workflows. Publishing: Players, Agents and Agendas, pp. 87 - 90. IOS Press, 2016.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning. pp. 5637-5664. PMLR, 2021.
Alexander Kolesnikov and Christoph H Lampert. Improving weakly-supervised object localization by micro-annotation arXiv preprint arXiv: 1605.05300 2016.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision, pp. 491-507. Springer, 2020.
Simon Kornblith, Jonathon Shiens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2661-2671, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In International Conference on Machine Learning, pp. 5815-5822 5826. PMLR, 2021.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distributiin. arXiv preprint arXiv:2202.10054 2022.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):227--2324, 1998.
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from underspeci- fied data, arXiv preprint 2rXiv:2202.03418, 2022.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624-639, 2018a.
Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9572- 2019.
Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: tion bias. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 513-528, 2018b.
Towards action recognition without representa-
Evan 7. Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Perey Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 2017.
arXiv preprint
13

---

Published as a conference paper at ICLR 2023
Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick. Predicting inductive biases of pre-trained models. In International Conference on Learning Representations. 2020.
Wesley Maddox, Shuai Tang, Pablo Moreno, Andrew Gordon Wilson, and Andreas Damianou. Fast adaptation with linearized neural networks. International Conference on Artificial Intelligence and Statistics, pp. 2737-2745. PMLR, 2021.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pp. 181-196, 2018.
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM international conference on Multimediaa, pp. 1485-1488, 2010.
R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint 19Xiv:1902.01007, 2019.
Wes McKinney. Data structures for statistical computing in python. In Stéfan van der Walt and Jarrod Millman (eds.), Proceedings of the 9th Python in Science Conference, pp. 51 56, 2010.
Aditya Krishna Menon, Ankit Singh Rawat, and Sanjiv Kumar. Overparamettrrsationn and worst-case generalisation: friend foe? In International Conference on Learning Representations, 2020.
Mazda Moayeri, Phillip Pope, Yogesh Balaji, and Sohcil Feizi. A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes. arXiv preprint arXiv:2201. 2022.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. International Conference on Machine Learning, pp. 10-18. PMLR, 2013.
Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of out-of-distribution generalization. arXiv preprint arXiv:2010.15777 2020.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems, 33:20673-20682. 2020.
Junhyun Nam, Jachyung Kim, Jaeho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. In International Conference Learning Representations, 2022.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512-523, 2020.
Yonatan Oren, Shiori Sagawa, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust language modeling. arXiv preprint 1909 2019.
Matteo Pagliardini, Martin Jaggi, François Fleuret, and Sai Praneeth Karimireddy. Agree to disagree: Diversity through disagreement for better transferability. arXiv preprint 2:XX::2002.04414 2022.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 12(10):13551133 2009.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Detito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-288 28300, 2011.
14

---

Published as a conference paper at ICLR 2023
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 1865):947-1112 2016.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guil- laume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and calibration. Advances in neural information processing systems, 30, 2017.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pp. 5301-5310. PMLR, 2019.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pp. 4334 4343. PMLR, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016
Amir Rosenfeld, Richard Zemel, and John K Tsotsos The elephant in the room. arXiv preprint (rXiv:8808.03300 2018.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint arXiv:2202.06855 2022.
Yangjun Ruan, Yann Dubois, and Chris Maddison. Optimal representations for covariate shift. arXiv preprint arXiv: arXiv:2201.00057 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang, Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: /91 /08731, 2019.
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, et al. Extending the wilds benchmark for unsupervised adaptation. arXiv preprint arXiv:2112.050900 2021.
Luca Scimeca, Scong Joon Oh, Sanghyuk Chun, Michael Poli, and Sangdoo Yun. Which shortcut cues will dnns choose? a study from the parameter-space perspective. arXiv preprint arXiv:2110.03095, 2021.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585, 2020.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Con features off-the-shelf: astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 806-813, 2014.
Rakshith Shetty, Bernt Schiele, and Mario Fritz. Not using the car to see the sidewalk Bedewalk-quantffiigg and controlling the effects context in classification and segmentation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2019.
Sahil Singla and Soheil Feizi. Salient imagenet: How discover spurious features in deep learning?
arXiv preprint arXiv:2110.0433 2021.
Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, and Eric Horvitz. Understanding failures of deep networks via robust feature extraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12853-12862, 2021.
15

---

Published as a conference paper at ICLR 2023
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: aine-graieed robustness in coarse- grained classification problems. Advances in Neural Information Processing Systems, 01s11933--13355 2020.
coarse-grained
Nimit Sohoni, Maziar Sanjabi, Nicolas Ballas, Aditya Grover, Shaoliang Nie, Hamed Firooz, and Christopher Ré. Barack: Partially supervised group robustness with guarantees. arXiv preprint arXiv:2201.0007. 2021.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unrcasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference computer vision, pp. 843 03--552,200 2017.
Enzo Tartaglione, Carlo Alberto Barbano, and Marco Grangetto. End: Entangling and disentangling deep representations for bias correction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13508 13517, 2021.
Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton van den Hengel. Evading the simplicity bias: Training a diverse set of models discovers solutions with superior ood generalization. arXiv preprint arXiv:2105.05612, 2021a.
Damien Teney, Ehsan Abbasnejad, and Anton van den Hengel. Unshuffling data for improved generalization in visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1417-1427, 2021b.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Towards debiasing nlu models from unknown biases. arXiv preprint arXiv:2009.122 12303, 2020.
Victor Veitch, Alexander D' Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious correlations: Why and how to pass stress tests. arXiv preprint arXiv:2106.00544 2021.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Îlhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: 10.1038//411922-019990
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongic. The caltech-uesd birds-200-2011 dataset. 2011.
Haohan Wang, Zexue Hc, Zachary C Lipton, and Eric P Xing. Learning robust representations by projecting superficial statistics out. arXiv preprint arXiv:1903.062555 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, et al. Huggingface's transformers: State-of-the ootte-of-the-rt natural language processing. arXiv preprint arXiv:1910.037711 2019.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint wrX1v:2006.09994, 2020.
Yilun Xu, Hao He, Tianxiao Shen, and Tommi S. Jaakkola. Controlling directions orthogonal to classifier. In International Conference on Learning Representations, 2022.
Yadollah Yaghoobzadeh, Soroush Mehri, Remi Tachet, Timothy J Hazen, and Alessandro Sor- doni. Increasing robustness to spurious correlations using forgettable examples. arXiv preprint 191iv: 1.13911008/1 2019.
16

---

Published as a conference paper at ICLR 2023
John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: scetioncctional study. PLoS medicine, 15(11):e1002683, 2018.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019.
Jianguo Zhang, Marcin Marszatek, Svetlana Lazebnik, and Cordelia Schmid. Local features and kernels for classification of texture and object categories: A comprehensive study, International journal of computer vision, (3(2):213-235 2007.
Jingzhao Zhang, Aditya Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. arXiv preprint arXiv:2010.122388 2020.
Michael Zhang, Nimit S Sohoni, Hongyang Zhang, Chelsea Finn, and Christopher Ré. Correct-n- contrast: A contrastive approach for improving robustness to spurious correlations. arXiv preprint arXiv:2203.01517, 2022.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligenee 40(6))1452-144 2017.
Wei Zhu, Haitian Zheng, Haofu Liao, Weijian Li, and Jiebo Luo. Learning bias-invariant repre- sentation by cross-sample mutual information minimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15002-15012 2021.
17

---

Published as a conference paper at ICLR 2023
APPENDIX OUTLINE
This appendix is organized as follows. In Section A we provide additional related work discussion. In Section B we present details on the experiments on feature learning in the presence of spurious correlations. In Section we provide details on the results for the spurious correlation benchmark datasets and perform ablation studies. We provide details on the experiments on the reliance of ImageNet-trained models the background in Section (aooonDDaa and the texture-bias in Section [E] In Section F we provide further discussion of the results presented in the paper and future work. In Section G compare DFR to methods proposed in Kang et al. (2019) and other last layer retraining
variations.
Code references. In addition to the experiment-specific packages that we discuss throughout the appendix, used the following libraries and tools in this work: NumPy (Harris et al. 2020). SciPy (Virtanen et al. al.]| 22200), PyTorch (Paszke et al. (017]. Jupyter notebooks (Kluyver et al. 2016
Matplotlib (Hunter 2007), Pandas (McKinney) 20TO), transformers (Wolf al. 2019).
A ADDITIONAL RELATED WORK
Differences with Kang et al. 20019) Our work and Kang et al. 2019) consider different settings: Kang et al. (2019) considers long-tail classification with class imbalance, while we consider spurious correlations and shortcut learning. In spurious correlation robustness, there is often no class imbalance, and the methods Kang al.] (019) cannot be directly applied. Our conceptual results, such as the ability control the reliance background or texture features in trained models are also orthogonal to the observations of Kang et al. (2019).
Algorithmically, DFR is related to the methods of Kang et (2019), but there are still important differences. In the Learning Weight Scaling (LWS) method, the authors rescale the logits of the classifier with scalar weights f;: the weight of the i-th row of the weight matrix the last layer updated to w; f.w; The parameters f; are trained by minimizing the loss on the training set with class-balanced sampling. particular, LWS not full last layer retraining, as only one parameter per class is learned.
Kang et al. (019) also proposed Classifier Re-training (cRT) approach which retrains the last layer on the traiming set with class-balanced sampling, where are equally likely to sample datapoints from each class. cRT is closer to DFR than LWS, as it retrains all parameters in the last layer.
The algorithmic differences of these methods with DFR are as follows:
In DFR, we use a group-balanced subset of the data instead of distinction is important, in the group robustness setting the classes often balanced, so LWS and cRT are not immediately applicable to the spurious correlation setting.
sampling. This
In DFR, we subsample the reweighting dataset to be group balanced instead of using class- or group-balanced sampling. Specifically, we produce dataset where the number of examples in each group is the same, and only use these datapoints. This detail hugely important, as boup-balancee sampling does not produce classifiers robust to spurious correlations (e.g. see RWG and SUBG methods in Idrissi et al. (02T).
In DFR we use held-out data for retraining the last layer, and the training data. This is the important distinction between DaR Val and DFR LWS and cRT both retrain the classifier on the training data.
There are also technical differences in how train the last layer in DFR: we average the weights of several independent runs of last layer retraining on different group-balances balanced subsets of the data, and use strong E regularization.
To sum up. both cRT and LWS are not immediately applicable to spurious correlation robustness. Moreover, even we adapt cRT to the spurious correlation setting, there are still major differences with DFR: subsampling
graunebalanced sampling, held-out data vs training data, and regularization. We present comparisons to LWS, cRT and related last layer retraining variations on Waterbirds and CelebA in Appendix G where show that both DFR versions outperform methods proposed in Kang et al. (2019) and other ablations. In Appendix C we also show that regularization and multiple re-runs of last layer retraining are important for strong performance with DFR.
18

---

Published as a conference paper at ICLR 2023
Differences with Rosenfeld et al. 2022). While our works make similar high-level observations, they are actually complementary to each other. In particular, our works don't 't share any datasets, experiments problem settings. Rosenfeld et al. (2022) focus on domain generalization, where the goal is to train a model that generalizes to unseen domains. In this setting, we know the domain labels on the train, and we have no data from the test domains. In spurious correlations, the goal is to train a model that does not rely on spurious features. We typically have examples from all the test groups in train, but the groups are highly imbalanced. As in domain generalization we do not have access to target domain data, Rosenfeld et al. (2022) refer to last layer retraining on the target domain as "cheating" (see e-g. the caption of Figure 1 in their paper). Consequently, they propose a different method, DARE, which is very different from DFR. DARE estimates means and covariance matrices for each domain to whiten out the features. On test, they apply approximate whitening to new domain, and they don't use test domain data to retrain the last layer. On the other hand, DFR uses an ERM-trained feature extractor and simply retrains the last layer on a group-balancee reweighting dataset.
To sum up, the observations from our work and the concurrent work by Rosenteld et al. (2022) are complimentary: we both show that ERM learns the core features well, but the settings, methods and experiments are different. In particular, it is not trivial to apply DARE to spurious correlations or DFR to domain generalization.
Other group robustness methods. Another group of papers proposes regularization techniques to learn diverse solutions on the train data, focusing on different groups of features (Teney al. 2021a] Lee et al. [022] Pagliardini et al. 2022, Pezeshki et al. 2021). Xu et al. (2022) show how to train orthogonal classifiers, i.e. classifiers invariant to given spurious features the data. Other papers proposed methods based on meta-learning the weights for a weighted loss (Ren et al.) 2018) and
group-agnostic adaptive regularization (Cao al.. 2019 2020). (Shhoni et al. (2020) use clustering of internal features to provide approximate labels for group distributionally robust optimization. number of prior works address de-biasing classifiers by leveraging prior knowledge on the bias
type (Li & Vasconcelos 2019 Kim et al. 2019: Tartaglione et al., [021 Teney et al. 202 lb) Zhu
ct al. 2021 Wang ct al. 2019 Cadene et al. 2019 Li ct al. 2018b).
Spurious correlations in natural image datasets. Multiple works demonstrated that natural image datasets contain spurious correlations that hurt neural network models (Kolesnikov & Lampert 2016
Xiao et al., 2020, Shetty et al. 2019 Alccon et al., 2019 Singla & Feizi| 202 (iiglla et al. 1021
Moayeri et al. 2022). Notably, Geirhos et al. (2018) demonstrated that ImageNet-trand -trinndd CNNs are biased towards texture rather than shape of the objects. Follow-up work explored this texture bias and showed that despite being texture-biased, CNNs still often represent information about the shape in their feature representations (Hermann al. 2020 [slam al., 202T). In Section 7 we show that it is possible to reduce the reliance of ImageNet-trainee models on background context and texture information by retraining just the last layer of the model.
Transfer learning. Transfer learning (Pan & Yang 2009; Sharif Razavian et al. 2014) is an extremely popular framework in modern machine learning. Multiple works demonstrate its effec- tiveness (e.g. Girshick, 2015) Huh et al.] [016;; He et al. 2017: Sun et al.) 2017 Mahajan et al.)
2018 Kolesnikov et al. 2020 Maddox et al. 2021), ) and study when and why transfer learning can
be effective (Zhai et al. 2019 Neyshabur et al., 20200 Abnar et al., [021 Kornblith et al.) 2019)
Kumar et al., 2022). Bommasani et al. (2021) provide a comprehensive discussion of modern transfer learning with large-scale models. While algorithmically our proposed DFR method is related to transfer learning, it has a different motivation and works on different problems. We discuss the
relation between DFR and transfer learning in detail in Section 5
Related methods have been developed in several areas of machine learning, such as ML Fairness
(Dwork et al. [012) Hardt et al.) 2016: Kleinberg al. 2016; Pleiss et al., 2017 Agarwal et al.
2018:
Khani et al. 2201)), 9,,,),,,)))) NLP (McCoy et al. 2019 I Lovering et al.] [0200 Kaushik et al. 2020
2021
Eisenstein 2022] Veitch et 2021 domain adaptation (Ganin & Lempitsky [2055] Ganin
et al.
2016) and domain generalization (Blanchard et al. 2011 Muandet et al. 2013 Li et al.
2018a
Gulrajani & Lopez-Paz 20200 Ruan et al.| including works on Invariant Risk Minimization
and causality (Peters et al. 2016 Arjovsky et al. 2019 Krueger et al. 2021
Aubin et al. 2021
19

---

Published as a conference paper at ICLR 2023
Original
FG-Only
BG-Only
O
/
1
7
9
MNIST-MNIST
MNIST-FashionMNISTT
MNIST-CIFAR
Figure 4: Data examples. Variations of the waterbirds (Top) and Dominoes (Bottom) datasets generated for the experiment Section
B DETAILS: UNDERSTANDING REPRESENTATION LEARNING WITH SPURIOUS CORRELATIONS
Here provide details the experiments in Section
B. FEATURE LEARNING WATERBIRDS
Inverse problem. In Table 4 we present the results the inverted Waterbirds problem, where the goal is to predict the background type while the bird type serves as a spurious feature. We note that prior work did not consider this reverse Waterbirds problem. The results for the inverted problem are analogous to the results for the standard Waterbirds ( models trained on the Original data learn the background features sufficiently well predict the background type with high accuracy when the spurious foreground feature is not present, but perform poorly when presented with conflicting background and foreground features. While is often suggested than neural networks are biased to learn the background (Xiao et al. 2020), we see that in fact the network relies on the spurious foreground feature (bird) when trained to predict the background.
Data. We show examples of Original, FG-Only and BG-Only Waterbirds images in Figure 4 To generate the data, we follow the instructions at git www.cooo//ooaeccc httm//kkhaaggeee/rrruuuuu DRO# waterbirds] but in addition to the Original data we save the backgrounds and foregrounds separately. Consequently, the FG-Only data contains the same exact birds images as the Original data, and the BG-Only data contains the same exact places images the Original data. For the 100% spurious correlation strength we simply discard all the minority groups data from the Original (95% spurious correlation) training dataset. For the Balanced data, start with the Original data with 95% spurious correlation and replaced the background in the smallest possible number of images (chosen randomly) achieve a 50% spurious correlation strength.
Strper-parameters For the experiments in this section we use ResNet- 50 model pretrained on ImageNet, imported from the torchvision package: torchvision. moneels. resnet50 (pretrained=Trun) We train the models for 50 epochs with SGD with a constant learning rate of 10 mo- mentum decay of 0.9, batch size 32 and weight decay of 1 We use random crops [RandomResizedCrgg (224, scale- (0. 7, 1.0), ratio (0.75, 4./3.), interpolation-2)) and horizontal flips in barcchvssion.trane forms as data augmentation.
(Marcel & Rodriguez, 2010).
()) implemented
B.2 DOMINOES DATASETS
Data. We show data examples from Dominoes datasets (MNIST-MNIST, MNIIS--FsshonMNIIT
and MNIST-CIFAR) in Figure 4
The top half of each image shows MNIST digits from classes
20

---

Published as a conference paper at ICLR 2023
Spurious Corr. (%)
Test Data (Worst/Mean, %)
Train Data
Original
BG-Only
Balanced
50
93.2/95.6
93.6/96.0
Original
95
77.4/91.2
93.1/95.7
Original
100
36.1/77.5
92.7/94.8
Place-Only
91.8/94.2
92.4/95.2
Table 4: Feature learning on Inverted Waterhirds. ERM classifiers trained on Inverted Waterbirds with Original and BG-Only images. Here the target is associated with the background type, and the foreground (bird type) serves as the spurious feature. All the models trained on the Original data including the model trained without any minority group examples (Spurious corr. 100%) underperform on the worst-group accuracy on the Original data, but perform well on the BG-Only data, almost matching the performance of the BG-Only trained model.
{0, 1). and the bottom half shows: MNIST images from classes {7, 9] for MNIST-MNIST, Fashion- MNIST images from classes [coat, dress} for MNIISTFFsshioMMNISST. and CIFAR-I0 images from classes (car, truck) for MNIST-CIFAR. The label corresponds the more complex bottom part of the image, but the top and bottom parts are correlated (we consider 95%, 99% and 100% levels of spurious correlation strength in experiments). 20% of the training data was reserved for validation or reweighling dataset (see Section 5) where each group is equally represented.
Syper-parammters. We used a randomly initialized ResNet-20 architecture for this set of ex- periments. We trained the network for 500 epochs with SGD with batch size 32, weight decay 10-3, initial learning rate value 10-2 and cosine annealing learning rate schedule. For the logistic regression model, we first extract the embeddings from the penultimate layer of the network, then use the logistic regression implementation (sklearn 1 linear model from the sci scikit kit-lea r package (Pedregosa et al.] 2011). We use & regularization and tune the inverse regularization strength parameter C in the range {0.1, 10, 100, 1000). For more details Deep Feature Reweighting implementation and tuning, see section
on)
Transfer from simple to complex features. In addition to the results presented in Figure 2 we measure the decoded accuracy using the model trained just on the spurious simple features: the top half of the image showing an MNIST digit. After training, we retrain the last layer of the model using a validation split of the corresponding Dominoes dataset which has both top and bottom parts. In this case, we measure the transfer learning performance with the features learned on the binary MNIST classification problem applied the more complex bottom half of the image. We obtain the following transfer accuracy results: 92.5% on MNIST MNIST-MNIST, 92.1 % on MNIST-Fashion and 61.4% on MNIST-CIFAR. correlation levels 99% and 95% is better than the transfer accuracy. For the 100% spurious correlation, transfer achieves comparable results on MNIST-Fashion and MNIST-CIFAR, but on MNIST-MNIST the decoded accuracy with model trained on the data with the core feature is significantly higher (99% vs 92%).. These results confirm that for the spurious correlation strength below 100%, the model learns high quality representation of the core features, which cannot be explained by transfer learning from the spurious feature.
all datasets, the decoded accuracy reported in Figure for the spurious
B.3 COLORMNIST
Data. We follow Zhang al. 2022 ) to generate ColorMNIST dataset: there are classes cor- responding to pairs of digits (0, (2. 3), (4, 5), (6, 7), (8, 9), and 5 fixed colors such that in train data each class y has a color which is associated with with correlation Peorrs and the remaining examples are colored with the remaining colors uniformly at random. Validation split is randomly sampled 10% fraction of the original MNIST data, and validation and test data exmaples are colored with 5 colors uniformly at random.
prrpet-paraestrs. We train small LeNet-like Convolutional Neural Network model on this dataset varying Pcorr- The model is trained for 5 epochs, with batch size 32, learning rate 10 10-3, and weight decay 5 x 10-4 following Zhang et al. (2022). CNN consists of 2 convolutional layers with and 16 output filters, followed by a max pooling layer, then another convolutional layer with 32 output filters and fully-connected layer.
21

---

Published as a conference paper at ICLR 2023
FG-Only Logits
BG-Only Logits
BG-Only Logits
FG + BG Logits
10
5
0
5
Original Logits
Original Logits
FG Only Logits
Original Logits
Figure Logit additivity. Distribution of logits for the negative class on the test data for model trained Waterbirds dataset. We show scatter plots of the logits for the Original, FG-Only and BG-Only images. The logits on Original images are well aligned with sums of logits on the corresponding FG and BG images (rightmost pancl), suggesting that the foreground and background features are processed close to independently in the network.
Peorr
corr.
0.8
0.9
0.95
0.995
1.0
ERM
94.511.0
85.1
66.0
40.8 13.6
0.0
0.0
DFR Val
93.8+u.9
92.64u.7
91.640.s
80.4+1.1
77.3
Table 5: Results on ColorMNIST. Worst-group accuracy of a small CNN model trained on 5-class ColorMNIST dataset, varying the spurious correlation strength Poorr between classes and colors. DFR achieves strong performance even in the challenging settings with strong correlation between class labels and colors. We report mean std over independent runs of the method.
We report worst-group accuracy for ERM and DFRY, in Table The no corr. results for ERM correspond to the model trained on the dataset without correlation between colors and class labels. Notably, DFR recovers strong worst-group performance even in the cases when base model has 0% gorst-group accuracy (Peory = 1.0 and Poar = 0.995). For Peory < 0.95 DFR's worst-group accuracy is closely matching the optimal accuracy of the model trained on data without spurious correlations.
B.4 LOGIT ADDITIVITY
To better understand why the models trained on the Original Waterbirds data perform well on FG- Only images, in Figure we inspect the logits of trained model. We show scatter plots of logits for the negative class (logits for the positive class behave analogously) on the Original, FG-Only and BG-Only test data. Both logits FG-Only and BG-Only data correlate with the logits on the Original images, and FG-Only show a higher correlation. The FG-Only and BG-Only logits are not correlated with each other, as in test data the groups balanced and the foreground and background are independent.
We find that the sum of the logits for the BG-Only and the logits for the FG-Only images provides good approximation of the logits on the corresponding Original image (combining the foreground and the background). We term this phenomenon logit additivity: Waterbirds, logits for the different predictive features (both core and spurious) are computed close to independently and added together in the last classification layer.
22

---

Published as a conference paper at ICLR 2023
Waterbirds dataset
S1
Sa
Si
94
Image Examples
Description
landbird on land
lundbird water
waterbird
waterbird on water
Class Label
0 184 (4%)
Train data
3498 (73%)
56 (1%)
1057 (22%)
# Val data
466
133
Target: hird type;
Spurious feature: hackground type:
Minority: G2. Go
CelebA hair color dataset
G2
G3
C4
Image Examples
Description
Non-blond woman
Non-blond mam
Blond woman 1 22880 (14%) 2874
Blond man
Class Label
0 66874 (41%) 8276
1 1387 (1 (%) 182
# Train data
71629 (44%)
# Val data
Target: hair color;
Spurious feature: gender;
Minority: G4
Figure 6: Waterbirds and CelebA data. Dataset descriptions and example images from each group on Waterbirds and CelebA datasets.
C DETAILS: SPURIOUS CORRELATION BENCHMARKS
Here we provide details and additional ablations on the experiments in Section
Data. We use the standard Waterbirds, CelebA and MultiNLI datasets, following e.g. (Sgawa al. 2019 Liu et al.) Idrissi al. 2021). We use CivilComments dataset as implemented in the WILDS benchmark (Koh et al. 2021). See Figures 6, for group descriptions and data examples.
Base model hyper-parameters. For the experiments on Waterbirds and CelebA we use a ResNet-50 model pretrained on ImageNet, imported from the torchvision package: torchvision. .ooesss.meel...o.oo resnet50 (prettaaertta
We use random crops [RandomResizedCrog (224, scale= 7 1.0), ratio= 75, 4./3.).
interpolation=2) and horizontal flips SaandomHorizontalFFee data augmentation. We train all models with SGD with momentum decay 0.9 and a constant learning rate. On Waterbirds, we train the models for 100 epochs with weight decay 10-3, learning rate 10-3 and batch size 32. On CelebA, we train the models for 50 epochs with weight decay 10-4 learning rate 10-3 and batch size 128. We do use early stopping. On MultiNLI and Civil- Comments, we use the BERT for classification model from the transformers package: www.FForSequeeelasssiiic.c
()) implemented
in coreevissoo..taasstee
www.rrrrrerrrrrrereeeeeereeeeeee wwwtrainedd..... (" bert uncased" We train the BERT models with the AdamW (Loshchilov & Hutter 2017) with linear learning rate annealing with initial learning rate 10 batch size 16 and weight decay 10-1 for 5 epochs.
num 1 aasellssum__lassse
DFR details. For DFR, we first extract and save the embeddings (inputs the classification layer of the base model) of the training, validation and testing data using the base model. We then preprocess the embeddings to have zero mean and unit standard deviation using the stan- dard scaler sklearn. preprocess ing StandardScaler from the seikit-less pack- age. In each case, we compute the preprocessing statistics on the reweighting data used to train the last layer. To retrain the last layer, (sklearn. linear......... _inear_modell. model. iegggsstcceggeesiinn We use regularization. For DFR Val
the logistic regression implementation from the scikit-learn package. only tune the inverse regularization strength parame-
23

---

Published as a conference paper at ICLR 2023
MulsiNLI
Class
# Train data
Val
Text examples
Description
"if residents are unhoppy, they can put wheels their homes and go someplace else, she suid. [SEP] residents are stuck here but they can't go onywhere else."
contradiction
G1
57498 (28%)
22814
no negations
"within this conflict of values clash about art. [SEP] there clash obout ort.'
contradiction
22
11158 (5%)
4634
has negations
"there something like amusement in the old man't 's voice. [SEP]
entailment no negations
G3
(32%)
umusement."
"in 1988, total for the
G1
was obout $36. [SEP] the postal service cost citicens almust nothing the 80', y.
entuilment
1521 (1%)
613
has negations
"yeah even cooking over open, fire little more fun 't't ISEP!
neutral negations
9s
66630 (32%)
26655
like the flavur of the fond."
Go
"thut's 's not too bud [SEP] nothing" Target: contradiction eatailment neutral;
neutral
1992 (1%)
797
negations
Spurious feature: has negation words.
Minority: Gr. So.
CiviCComments
Text examples
Class label
Description
Train data
Val
data
tm quite surprised
you.
non-loxic no identities
Infrared camos penetrate tinfoll. "I think have minnderstood what sstrww men' are. But I'm glad that your gravy good."
148186 (55%)
25159
non-toxic
identities
90337 (33%)
14966
"Hohalahu puiling his in Snopes. Pathetic."
toxic identities
12731 (5%)
2111
"That sounds like something a white person would say.""
toxic
17784
2944
identities
Target: Toxic not toxic comment:
Spurious feature: mentions protected categories.
Figure MultiNLI and CivilComments data. Dataset descriptions and data examples from different groups on MultiNLI and CivilComments. We underline the words corresponding to the spurious feature. CivilComments contains 16 overlapping groups (corresponding to toxic non- toxic comments and mentions of one of the protected identities: male, female. LGBT, black, white, Christian, Muslim, other religion. We only show examples with mentions of the male and white identities.
24

---

Published as a conference paper at ICLR 2023
ImageNet Pretrain
Dataset Fine-tune
Waterbirds
CelebA
Method
Worst(
Mean(%)
Wors(( (6)
Mean (%)
Base Model
V
74.9
98, 10.1
26.9
95.320
DFR DFR Vil
/
90.240.8
97. 97.0=0.3
85.440.4 80.7
/
/
92.940.2
94.220.4
88.3
89.640.4
Base Model
/
6.943.0
88, 88.0
39.8+2.0
95. 75.74
DFR
X
/
4-.4
69.8a7.0
83.4
87.540.3
DFR
53.9
62.6.2.2
85.0
87.640.3
DFR
X 88.7400.4
89.7.0.1
73.1
80.9
Table 6: Effect of ImageNet pretraining and dataset fine-tuning. Results for DFR the Waterbirds and CelebA datasets when using an ImageNet-trained model as a feature extractor, training the feature extractor from random initialization initializing the feature extractor with ImageNet- trained weights and fine-tuning on the target data. On Waterbirds, we can achieve surprisingly strong worst group accuracy of 88.7% without finetuning on the target data. On CelebA, we can achieve reasonable worst group accuracy of 85% without pretraining. However, on both datasets, both ImageNet pretraining and dataset finetuning are needed to achieve optimal performance. All methods in Table 2 use ImageNet-trained models as initialization and finetune on the target dataset.
Finetuned
WB
CelebA
Last Layer
93. +0.2
88.340.5
Last Layers
93.1+0.5
87.2+0.9
Last Block
90.7
83.0
All Layers
35.640.1
77.8
Table 7: Retraining multiple layers. We retrain the last linear layer, last two layers (linear, last batchnorm and conv), last residual block, and all layers from scratch group-balance validation data. Retraining the last layer is sufficient for optimal performance, and moreover retraining more layers can hurt the results.
ter C: we consider the values in range {1., 0.7, 0.3, 0.07, 0.03, 0.01 and select the value that
leads to the best worst-group performance on the available validation data (as described in Section we use half of the validation to train the logistic regression model and the other half to tune the parameters at the tuning stage). For DFR we additionally tune the class weights: we set the weight for one of the classes to 1 and consider the weights for the other class in range {1,2, 2,3, 10, 100, 300, 1000]; we then switch the classes and repeat the procedure. For the final evaluation, we use the best values of the hyper-parameters obtained during the tuning phase and train a logistic regression model on all the available reweighting data. We train the logistic regression model on the reweighting data 10 times with random subsets of the data (we take all of the data from the smallest group, and subsample the other groups randomly to have the same number datapoints) and average the weights of the learned models. We report the model with averaged weights.
6, for D. Val
C.1 ABLATION STUDIES
Is ImageNet pretraining necessary for image benchmarks? DFR relies on the ability of the feature extractor to learn diverse features, which may suggest that ImageNet pretraining is crucial. In Appendix Table 6, we report the results on the same problems, but training the feature extractor from scratch. We find that on Waterbirds, ImageNet pretraining indeed has dramatic effect on the performance of DFR as well as the base feature extractor model. However, on CelebA DFR shows strong performance regardless of pretraining. The difference between Waterbirds and CelebA is that Waterbirds contains only 4.8k training points, making it difficult to learn a meaningful feature extractor from scratch. Furthermore, on both datasets, finetuning the feature extractor on the target data is crucial: just using the features extracted by an ImageNet-traine trainee model leads to poor results. We note that all the baselines considered Table use ImageNet pretraining.
Retraining multiple layers. In Table we perform another ablation by retraining multiple layers from seratch on the group-balanced validation dataset on Waterbirds and CelebA. We find that retraining just the last layer provides optimal performance, retraining the last two layers (the last
25

---

Published as a conference paper at ICLR 2023
Base Model Hypers
Waterbirds
Method
Ir
wd
batch size aug
Worst(%)
Mean (%)
10-3
10-3
32
Base Model
74.9
98. 10.1
DFR
94.2 92.9
10 3
10
32
X
Base Model
73.4
97.7
DFR
90.9
Base Model
24.1
94.4
10
10-3
32
DFR Val
88.0
88.6
Base Model
66.4
97.1
10
10-2
32
X
DFR Val
90.1
90.5
Base Model
71.5
98.1
3 10
10-a
32
DFR
91.9
93.5
Base Model
76.5
98.0
3 10 3
10-3
X
DFR
89.5
94.5
Base Model DFR
73.5
98.0
10
10
64
V
93.1
95.0
10
10-3
64
X
Base Model DFR,,
69.5
97.4
89.0
93.6
Base Model Hypers
CelebA
Method
Ir
wd
batch size aug
Worst( ())
Mean %)
Base Model
46.9
95.3g0
10-3
10-4
128
DFRY
88.341.1
91.320.3
Base Model DFR 86.2+1.2
44.3
95 95.220.1
10
10
128
90.8
Base Model
46.74..0
95.3
10
10
128
X
DFR Val
86.9
91.6g0.2
X Base Model
40.64
95.1
10-3
10-3
128
DFRY. Val
85.641.4
91.8=0.5
Table 8: Effect of base model hyper-parameters We report the results DFR for range of base model cyyer-parametee on Waterbirds and CelebA as well as the performance of the corresponding base models. While the quality of the base model has an effect on DFR, the results are fairly robust. For subset of configurations we report the mean and standard deviation over 3 independent runs of base model and DFR.
Method
Group Info Train / Val
Waterbirds
CelebA
MultiNLI
CivilComments
Wors((S)
Mean(S))
Worst(3))
Mean(Si)
Worsi(S))
Mcan((5)
Worst(*3)
Mean(Si)
Base (ERM)
x/ x
74.9
98.120.1
46.9
95.3go
65.94n.
82.81a.1
55.620s
g2.1
DER DER
VI
90,2#0.
97.0
80.7
90.6
71.5
82,5+a2
58.0.1.a
92.0ao.1
XI
92.9.a.2
94.2=0.4
88.3+1.1
91.3 91.3+0.s
74.7
82.1
70.1
87.2e0.3
Base Model trained without minority groups
Base (ERM)
3L.+
96.0=o.2
21.7+a.2
95.2+o.1
06.0+1.
82.5+0.
8.444.9
73.8=0.a
DERTAM
89.9
94.320.
89.0
91.6204
73.0gn.
82.2.0.
66.52o7
85.920.
Table 9: DFR variations. Worst-group and mean test accuracy of DFR variations on the bench- mark datasets problems. DFR Val achieves the best performance across the board. Interestingly, DFR outperforms DFR on all dataset and even outperforms DaR Val ments, report mta over 5 independent runs of the method.
on CelebA. For all experi-
convolutional layer and the fully-connected classifier layer) or the last residual block is competitive (but worse), while retraining the full network is lot worse.
26

---

Published as a conference paper at ICLR 2023
Number of retrains
3
5
10
20
Waterbirds 91.21+1.82
92.88 45
91.73=1.25
93. 23.
92.89+0.1
CelebA 85.09
88.64 11.00
87.80 1.17
88.02
88.37 12.02
Table 10: Ablation on the number of retrains in DFR Worst group accuracy of DFR varying the number of logistic regression retrains on Waterbirds and CelebA: we train the logistic regression model on the validation data several times with random subsets of the data (we take all of the data from the smallest group, and subsample the other groups randomly to have the same number of datapoints) and average the weights of the learned models. Averaging more than linear model leads to improved performance.
What if we just use ImageNet features? As a baseline, we apply DFR to features extracted by a model pretrained on ImageNet with no fine-tuning on CelehA and Waterbirds data. We report the results in Table (DFR and DFR CelebA fine-tuning is needed to get reasonable performance. The results in Table 6 suggest that both ImageNet pretraining and fine-tuning on the target data are needed to train the best feature extractor for DFR.
Val lines). While on Waterbirds the performance is fairly good, on
Robustness to base model hyper- prramteerseetess In Table a range of configurations of the baseline model hyper- parameters. While the quality of the base model clearly has effect on DFR performance, we achieve competitive results for all the hyper- parameter configurations that we consider, even when the base model performs poorly. For example, on Waterbirds with data augmentation, learning rate 10-3 and weight decay 10-2 the base model achieves worst group accuracy of 24.1%, but by retraining the last layer of this model with DER It still achieve 88% worst group accuracy.
report the results of DFR Val for
Val we
Ablation on the number of linear model retrains. We study the effect of the number of logistic regression retrains on different balanced subsets of the validation set in Table on Waterbirds and CelebA. We retrain the last linear layer multiple times and average the parameters of the resulting models, as described in Appendix C In general, is better to train more than model and average their weights. This effect is especially prominent on CelebA, where single model gets 85% worst group accuracy, while the average of 3 models gets 88.6%. As we increase the number of linear model retrains. the improvements saturate.
Ablation on the é regularization. We emphasize that it is beneficial to use é regularization in spurious correlations benchmarks where the number of last layer features is much higher than the reweighting dataset size (e.g. in Waterbirds last layer in DFR Val 50 is 2048). Without e, regularization, DFRY achieves 87.72 0.42% worst group accuracy on Waterbirds and 86.03 + 0.42% on CelebA, as opposed to 92.9 + 0.2% and 88.3 + 1.1% we choose & regularization strength through vross-validation as described in Appendix
approximately 500 examples for retraining the Ir while the dimensionality of the penultimate layer representations in ResNet-
Full model fine-tune fine-tuning on validation. As additional baseline, finetune the full model (as opposed to just the last layer) on the group- bolunbal validation set for 10 epochs with SGD starting from the Re checkpoint pre- mreitrained ImageNet and without training the corresponding train splits of Waterbirds and CelebA. We achieve 89.3 + 1.3% worst group accuracy on Waterbirds and 84.4 + 0.5% on CelebA. While these results are good relative to ERM the standard training set, they are still significantly worse than DFR (see Table 2
Val
C.2 PRIOR WORK ASSUMPTIONS
In Section 6 we compare DFR to ERM, Group DRO
Sagawa et al. 2019). JTT (Liu et al., 2021 ).
CnC (Zhang et al. 2022), SUBG (Idrissi al. 2021 and SSA (Nam al. 2022). These methods differ in assumptions about the amount of group information available.
Group DRO and SUBG assume that both train and validation data have group labels, and the hyper- parameters are tuned using worst-group validation accuracy. DFRD and DFR NY match the setting
27

---

Val and Published as a conference paper at ICLR 2023
of these methods and use the same data and group information; in particular, these methods only use the validation set with group labels to tune the hyper-panne parameters.
number of prior works (e.g. Liu et al. 2021 Creager et al. 2021 Zhang et al.
2022) sidestep
the assumption of knowing the group labels on train data, but still rely on tuning parrae-ar using worst-group accuracy on validation, and thus, having group labels on validation data. In fact, [drissi et al. 202T)) showed that ERM is a strong baseline when tuned with worst-group accuracy on validation.
Lee et al. 2022 explore the setting where they do not necessarily require group labels validation data, but their method implicitly relies on the presence of sufficiently many minority examples in the validation set such that different prediction heads would disagree on those examples choose the most reliable classifier.
Recent works Nam et al. (2022) and Sohoni et al. (2021) explore the setting where the group information is available
small subset of the data (e.g. the validation set), and the goal is to use the available data optimally, both to train the model and to tune the syper-parameters. These methods
use semi-supervised learning to extrapolate the available group labels to the rest of the training data. We consider this same setting with DFR).. Val where we use the validation data to retrain the last layer of the model.
C.3 DFR VARIATIONS
Here, we discuss two additional variations of DFR In DFR,; we use a random group-balanced subset of the train data as D. In DF (NM stands for "No Minority") we use a random group- balanced subset of the train data D, but remove the minority groups (G2, G3 Waterbirds and G4 on CelebA) from the data D used to train the feature extractor.
We compare different DFR versions in Table 9 All three DFR variations obtain results compet- itive with otate-of-the-art Group DRO results on the Waterbirds data. On CelebA, DFR DE match Group DRO, while DFR performs slightly worse, but still on par with JTT. In particular, DFR matches the state-of-the-ar Group DRO by using the same data to train and tune the model. Notably, DFR achieves these results with the features extracted by the network trained without seeing any examples from the minority groups! Even without minority groups, ERM models extract the core features sufficiently well to achieve state-of-the-arr results on the image classification benchmarks.
DFR also significantly improves performance compared to the base model across the board, but underperforms compared DFR it is crucial to retrain the last layer on new data that was not used to train the feature extractor.
On the NLP datasets, DFRY. Val outperforms the other variations, but in all cases all DFR variations significantly improve performance compared to the base model. Notably, on CivilComments the no minority version group of the dataset only retains the toxic comments which mention the protected identities and non-toxic comments which do not mention these identities. As result, the base model only achieves 8.4% worst group performance! DFR is still able to recover 66.5% worst group accuracy using the features extracted by this model.
C.4 WHY IS DF TR TR-NM BETTER THAN DFR Tx ?
Let us consider the second stage of DFR where we fix the feature encoder f((), and train logistic regression model L on the dataset f(D), where by f(D) we denote the dataset with labels from the reweighting dataset D and features from D extracted by f. We then evaluate the logistic regression model on the features extracted from the test data, (Drea).
Let us use M to denote a minority group in the reweighting dataset D. and Mese to denote the same minority group in the test data. For DFR NM and DFRY. Vol the model f trained without observing any data from M or MTes Assuming the datapoints in A and Tess are iid samples from the same distirbution, the distribution features in f(M) and (es will also be identical.
On the other hand, with DFR the minority group datapoints M are used to train the feature extractor f. In this case, can no longer assume that the distribution of features f(M) will be the
28

---

Published as a conference paper at ICLR 2023
features
Test features
features
Test features
D features
Test features
25
50
25
25
25
(a) DFR Ir
(b) DFR Val
(c) DFR T--M
Figure 8: DFR Variations. Visualization of the features extracted from the reweighting dataset D and the test data for different variations of DFR on the Waterbirds data. We show projections of the 2048-dimensional features the top-2 principal components extracted from D. With DFR Val and DFR the distribution the features for the minority groups G2 and G3 does not change between significant distribution shift.
the reweighting and test data, while with DFR;;
same as the distribution of TeM Consequently, DFR the logistic regression model will be evaluated under distribution shift, which makes the problem much more challenging and leads to inferior performance of DFR on the Waterbirds data. We verify this intuition Figure 8 where we visualize the feature embeddings for the reweighting dataset D and the test data. We see that as we predicted, the distribution of the minority group features coincides between D and test data for DFR and DFRY, while DFR shows significant distribution shift.
What D should be used in practice? In T the best performance is achieved by DFR and D which retrain the last layer data that not used in training the feature extractor. In practice, we recommend collecting a group-balanced validation set, which can be used both to tune the hyper-parameters and re-train the last layer of the model,
do in DFR
C.5 RELATION TO TRANSFER LEARNING
Algorithmically, DFR is special case of transfer learning, where D serves the source data, and D the target data (Sharif Razavian et al. 2014). However, the motivation DFR different from that of standard transfer learning: in DFR we trying to correct the behavior of pretrained model, and reduce the effect of spurious features, while in transfer learning the goal is to learn general features that generalize well to diverse downstream tasks. For example, we of ImageNet-trrined models on the background or texture and improve their robustness to covariate shift (see Section 7. while in standard transfer learning we would typically use pretrained model as initialization or a feature extractor to learn a good solution on a new dataset. In the context of spurious correlations, one would not expect DFR work as well as it does: the only reason DFR is successful is because, contrary to conventional wisdom, neural network classifiers are in fact learning substantial information about core features, even when they seem to rely on spurious features to make predictions. Moreover, in Appendix Table 6 we show that transfer learning with features learned ImageNet does not work nearly as well as DFR on the spurious correlation benchmarks.
DFR reduce the reliance
29

---

Published as a conference paper at ICLR 2023
ImageNet
Mixed Rand
FG Only
Paintings BG
ImageNet R
Validation Acturacy
0.92
0.94
0.98
0.88
0.55
0.92
0.86
0.88
0.50
0.94
0.90
0.84
5k 10k
20A
45k
1k 5A
10k
20k
16 SA 10k 20A 45k
10k 204 45k
5k
10k 20A 15k
Mixed-Rand Data
# Mixed Rand
Mixed-Rand
DFRMA
DFR
Baseline
Figure 9: ImageNet background reliance (VIT-B-16). Performance of DFR trained on MixedRand data and MixedRand + Original data different ImageNet-9 validation splits. All methods an VIT-B-16 feature extractor trained on ImageNet21k Ik and finetuned on ImageNet. DFR can reduce background reliance with a minimal drop in performance on the Original data. See Figure for analogous results with ResNet-50 feature extractor.
D DETAILS: IMAGENET BACKGROUND RELIANCE
Here provide details the experiments background reliance in Section 7.1 and additional evaluations verifying the main results.
Data. We use the ImageNet-9 dataset (Xiao et al. 2020). To test whether our models can gen- eralize unusual backgrounds,
additionally generate Paintings-BG. data shown in Figure For the Paintings-BCG data we use the Original images and segmentation masks for ImageNet-9 data provided by Xiao et al. (020)). and combine them with random paintings from Kaggle's sainter-by-nunbee
as backgrounds. For the ImageNet-R dataset (Hendrycks et al.) 2021), only use the images that fall into one of the ImageNet-9 categories, and evaluate the accuracy with respect to these categories. We show examples of images from different dataset variations in Figure
dataset available at kaggle. www/c//ainter-by-nneeeeee
Base model hyper-parameters We use ResNet-50 model pretrained on Im- ageNet and a VIT-B-16 model pretrained on ImageNet-21k and finetuned on ImageNet. The ResNet-50 model is imported from the torchvision pack- age: torchvision moweel......mmm... resnet50 (pretrraine==Truee 16 model is imported from the Iuvemmeeooo//yyyorrreeeerrrreee age available at hetthu.www/llemmmmsss////oooeeee ViT (ivB__16_mmaggeeetr
The VIT-B- pack- we use the command: pretrained=Truee load the model. To extract the embed- dings, we remove the last linear classification layer from each of the models. We preprocess the data using the torchvision. trans paclage Compose Resize CenterCrop (crop_ee __iz_)__ie____________ ToTensor (), Normalize ( [[ 485, 0.456,
pessize_sizee
0. 406] [2. 0. 224, 0. 0.225])]); where (res i ze ze_size, crop_s size) are equal to (256, 224) for the Re and (384, 384) for the VIT-B-16. We do not apply any data augmentation.
DFR details. We Train DFR on random subsets of the Mixed-Rand train data of different sizes (DRRMR) or combinations of the Mixed-Rand and Original data (DFROG+MR) For DFR OG+MR we use the same number Original and Mixed-Rami datapoints in all experiments. As the full ImageNet-9 contains 50k datapoints, we train the logistic regression on GPU with simple implementation in PyTorch (Paszke et al. 2017). We then preprocess the embeddings to have zero mean and unit standard deviation using the standard scaler sk learn. preprocessi ng. Standa rdScal l e from the scikit-leazn package. In each case, we compute the preprocessing statistics on the reweighting data used to train the last layer. For the experiments in this section use lg regularization (as the number of datapoints is large relative to the number of observations, we do not have to use f1). We set the regularization coefficient À to be 100 and train the logistic regression model with the loss Ex.EED L(y, wr + b) + |/1 where L(-.)) is the cross-entropy loss. We use full-batch SGD with learning rate 1 and no momentum to train the model for 1000 epochs. We did not tune the parameter SGD parae-easss
30

---

Published as a conference paper at ICLR 2023
Baseline
DFR MR
Baseline
DFR MR
Baseline
DFR MR
Figure 10: GradCAM visualizations of the features used by the baseline model and DFR on ImageNet-9.
Results for VIT-B-16. We report the results for the VIT-B-16 B-- base model Figure The results are generally analogous to the results for ResNet-50: it is possible to significantly reduce the reliance of the model on the background by retraining the last layer with DFR. For the VIT, removing the background dependence hurts the performance
the Original data slightly, but greatly improves the performance on the images with unusual backgrounds (Mixed-Rane FG-Only, Paintings-BG), Removing the background dependence does not improve the performance ImageNet-R.
BG modification. For each test image we generate images with 7 modified backgrounds: MixRand (random Only-BG-T ImageNet-9 background, [1]), Paintings-BG, and constant black, white, red, green and blue backgrounds. Then, for each model from Section we compute the percentage the datapoints, which changing the background with fixed foreground does not change the predictions compared to the original image. We get 87.5% for the baseline model, 92.4% for DFRMR and 93.1 1% for OG+MR suggesting that the DFRMR MR and DFR OG+MR models are significantly more robust to modifying the background.
Prediction based on BG. Next, for each model we evaluate the percentage of MixRand datapoints on which the model makes predictions that match the background class and not the foreground class. We get 14.8% for the baseline model, 11.2% for DFR and 11.7% for DFROG+ME The baseline model predicts the background class significantly more frequently than the DFR models.
GradCAM. Finally, to gain visual intuition into the features used by DRR and the baseline models, in Figure IO we make GradCAM feature visualizations on three images from ImageNet- We use the pytorch-grad-can package (Gildenblat contributors (02T) available at htt..o////acooooippotoocccc
While the baseline model uses the background context, the DFR features are more compact and focus on the target object, again suggesting that DFR reduces the reliance on the background information.
31

---

Published as a conference paper at ICLR 2023
munical Jnsf summett
Insert
06 Lurrrivee
92 Pr
02, Wheeled vehidee
00_dog dog
AutoZase
Car Maxem
05, neert
whe ad vehlic
36, 9ch
(a) IN-9 Original Xiao et al. 2020) (b) IN-9 FG-Only (iiao al. 2020 (c) IN-9 Mixed-Ree Rand
(iao et al. 2020
(d) Stylized ImageNet (Geirhos et al.
2018)
porille
cheetah
(c) ImageNet-R (Hendrycks et al.
2021
02 wheeled vehicle
seprepile
30_40g
50 mept
repole
bird
primate
03 rptiee
01 sird
(f) ImageNet-9, Paintings-BCC (ours)
Figure 11: ImageNet variations. Examples of datapoints from the ImageNet variations used in the experiments.
32

---

Published as a conference paper at ICLR 2023
E DETAILS: IMAGENET TEXTURE BIAS DETAILS
Here we provide details on the experiments on texture bias in Section 7
Data. We generate the stylized ImageNet (SIN) data following the instructions at (Geirhos et al. 2018). For evaluation, we use ImageNet-C (Hendryeks & Dietterich/ 2019) and ImageNet-R datasets (Hendrycks al. 202T). For ImageNet-C we report the average performance across all 19 corruption types and 5 corrpption intensities. We show examples of stylized ImageNet images in Figure [1]
Base model hyper-parameters We use the same ResNet-50 and VIT-B-16 models as described in Appendix [
DFR details. We train DFR using the embeddings of the original ImageNet (IN), styl- ized ImageNet (SIN) and their combination (IN+SIN) as the reweighting dataset. We pre- process the base model embeddings by manually subtracting the mean and dividing by standard deviation computed on the reweighting data used to train DFR; we did not use sklearn. preprocess
StandardScaler due to the large size of the datasets (1.2M datapoints for IN and SIN; 2.4M datapoints for IN+SIN). We train the logistic regression for the last layer on single GPU with simple implementation in PyTorch. We use SGD with learning rate 1, no momentum, no regularization and batch size of 10s to train the model for 100 epochs. We tuned the batch size (in the range { (10*, 104, 100}) and picked the batch size that leads to the best performance on the ImageNet validation set (10*). We did not tune the other hyper-parameters.
Results for VIT-B-16. We report the results for the VIT-B-16 base model in Table For this model, baselines trained from scratch on IN+SIN and SIN are not available so we only report the results for the standard model trained on ImageNet2 Ik and finetuned on ImageNet: for DFR we report the results using IN+SIN as the reweighting dataset. Despite the large-scale pretraining on ImageNet21k, we find that we can still improve the shape bias (36% - 39.9%) as well as robustness to ImageNet-C corruptions (49.7% + 52% Top-1 accuracy). On the original ImageNet and ImageNet-R the performance of DFR is similar to that of the baseline model.
Detailed texture bias evaluation. To provide further insight into the texture bias of the models trained with DFR, Figure 12 we report the fraction of shape and texture decisions for different classes following Geirhos et al. (2018). We produce the figure using the moshlnshuman codebase available at ggithub.com/beeee wwt.///oowgaaa/moooeeoooeooo wow.aaasmmmddel----aaaaa (Giirhos et al., 2021). We report the results for both DFR models and models trained from scratch IN, SIN and IN+SIN as well as the SeapeResNet-50 model and humans (results from Geirhos et al. (2018)). When trained on the same data, models trained from scratch achieve higher shape bias than DFR models, but DFR can still significantly improve the shape bias compared to the base model trained on IN.
Detailed ImageNet-C results. In Figure 13 report the Top-1 accuracy for DFR models and models trained from scratch on IN, SIN and IN+SIN on individual ImageNet-C datasets. We use the ResNet-50 base model. The model trained from scratch on IN+SIN provides the best robustness across the board, but DFR trained on IN+SIN also provides an improvement over the baseline RN50(IN) model on many corruptions.
33

---

Published as a conference paper at ICLR 2023
Fraction of 'shape' decisions
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Shape categories
0.2
0.7 0.8 0.9
Fraction of 'texture' decisions
Figure 12: Shape-texture bias report. Detailed report of the shape-texture bias generated using the model-vs -hummnn codebase (https: http//gttub....
-uuaaa (Geirhos et al. 2021). The plot shows the fraction of the decisions made based on shape and texture information respectively examples with conflicting cues (Geirhos al. 2018). The brown diamonds (o) show human predictions and the circles (o) show the performance of ResNet-50 models trained different datasets: ImageNet (IN, yellow), Stylized ImageNet (SIN, blue), ImageNet + Styl- ized ImageNet (IN+SIN, red), ImageNet Stylized ImageNet Finetuned on ImageNet (IN+SIN-II, pink); these models are provided the model --vs-human codebase. For each dataset (except for IN+SIN~IN) we report the results for DFR using an ImageNet-trane trained ResNet-50 model as a feature extractor with squares of the corresponding colors. Reweighting the features with DFR we can significatly increase the shape bias: DFR trained on SIN (blue squares) virtually matches the shape bias of the model trained from scratch on IN+SIN (red circles). However, the model trained just on SIN (blue circles) from scratch still provides significantly higher shape bias, that we cannot match with DFR.
pretrained model
Top-1 Acc (%) /Top-5 Acc (%)
Training Data
Shape bias (%)
Method
ImageNet
ImageNet-R
ImageNet-C
VIT-B-16 IN21k IN
36
79.2/95.0 79.7/94.5
29.1/42.0 29.0/41.4
49.7/69.3
DFR
IN+SIN
39.9
52.0/71.0
Table 11: Texture-vs-shape bias results for VIT-B-16. Shape bias, top-1 and top-5 accuracy on ImageNet validation set variations for VIT-B-16 pretrained on ImageNet2 and finetuned on ImageNet and DFR using this model as a feature extractor and reweighting the features on combined ImageNet and Stylized ImageNet datasets. By retraining just the last layer with DFR, the shape bias compared to the feature extractor model and improve robustness to covariate shift on ImageNet-C.
increase
34

---

Published as a conference paper at ICLR 2023
brightness
defocus blur
fog
gaussian
glass blur
06
0.6
0.70
05
0.05
0.4
C4
0.2
022
jpey compression
pixelate
shot noise
spatter
200m blur
0.0
0.-
03
05-
04
04
63
03
0.2
0.3
0.1
02
contrast
elastic transform
frost
gaussian noise
impulse nolse
0.6
0.4
03-
0.3
03
0.2
02
0.2
0.3
motion blur
saturate
spackle
0.70
RNSO(IN)
6.3
RNS0(SIN)
0.5
0.00
RNSO SIN)
0.4
0.55
MASOONN-SIN-NM
DIT(IN-SII
DFR(SIN)
0.40
0.2
DER(IN)
ImageNet-C results. Results of ResNet-50 models trained different ImageNet variations (shown in circles) and DFR using an ImageNet-trained ResNet-50 model as feature extractor ImageNet-C and ImageNet-R datasets. Each panel corresponds to different corruption, and the horizontal axis represents the corruption intensity. Retraining the last layer on ImageNet- Stylized (DFR(SIN), red squares) improves robustness ImageNet-C corruptions compared the base model (RN50(IN), blue circles), but does not match the robustness of a model trained from scratch on SIN IN+SIN.
Figure 13;
35

---

Published as a conference paper at ICLR 2023
F FURTHER DISCUSSION
Spurious correlations and representation learning. Prior work has often associated poor robustness to spurious correlations with the quality of representations learned by the model (Arjovsky) et al., 2019 Bahng et al.. 2020 Ruan et al. [021] ) and suggested that the entire model needs to be carefully trained to avoid relying on spurious features (e.g. Sagawa et al., 2019 [drissi et al. 2021
Liu et al. | Zhang et al. 2022 Sohoni et al. 202T Pezeshki et al. 20.21 Lee al.) 2022). Our
work presents a different view: representations learned with standard ERM even without seeing any minority group examples are sufficient to achieve trate-of-the-eee performance on popular spurious correlation benchmarks. The issue of spurious correlations is not in the features extracted by the models (though the representations learned by ERM can be improved (e.g.,
(e.g.,HHrrmann [Hermann & Lampinen
2020; Pezeshki et al., 202I)), but in the weights assigned to these features. Thus we can simply re- ve-weight these features for substantially improved robustness.
Practical advantages of DFR. DFR is extremely simple, cheap and effective. In particular, DFR has only one tunable
- the strength of regularization of the logistic regression. Furthermore, DFR is highly robust to the choice of base model, as we demonstrate in Appendix Table 8] and does not require early stopping or other highly problem-specifis tuning such as in Idrissi et al. (202T) and other prior works. Morcover, as DFR only requires re-training the last linear layer of the model, it is also extremely fast and easy to run. For example, we can train DFR on the -2.adpttaoonnt Stylized ImageNet dataset in Section 7n single GPU in matter of minutes, after extracting the embeddings on all of the datapoints, which only needs to be done once, On the other hand, existing methods such Group DRO (Sagawa et al. 2019) require training the model from scratch multiple times to select the best papper-prramterss which may be impractical for large-scale problems.
On the need for reweighting data in DFR. Virtually all methods in the spurious correlation literature, even the ones that do not explicitly use group information to train the model, use group- labeled data to tune the hyper-parameters (we discuss the assumptions of different methods in prior work in Appendix C.2). If a practitioner has access to group-labeled data, we believe that they should leverage this data to find a better model instead of just tuning the hyper-parameters. Finally, note that DFR can be easily combined with methods that automatically estimate group labels (e.g. Liu et al. 2021 Creager al., 202T (Sohoni et al. 2021): we can retrain the last layer using these estimated labels instead of ground truth.
Future work. There are many exciting directions for future research. DFR largely reduces the issue of spurious correlations to a linear problem: how do we train an optimal linear classifier on given features to avoid spurious correlations?
particular, we can try to avoid the need for a balanced reweighting dataset by carefully studying this linear problem and only using the features that are robustly predictive across all of the training data. We can also consider other types of supervision, such saliency maps (Singla & Feizi 202T) segmentation masks to tell the last layer of the model what to pay attention to in the data. Finally, we can leverage better representation learning methods
(Pezeshki et al. (202), ), including self-supervised learning methods (e.g. Chen al. 2020 2021), to further improve the performance of DFR.
He et al.)
G COMPARISON TO KANG ET AL. (2019)
We compare D and DFRF: to LWS and cRT methods proposed in Kang al. (019). and perform other related ablations in Table [2] We discuss the LWS and cRT methods in detail as
well as their algorithmic differences with DFR in Section and Appendix A We adapt the original implementation for LWS and cRTU]
In addition to LWS and cRT, we evaluate last layer re-training on the training and validation data with bolaned data sampling. These methods serve as intermediate variations between DFR and cRT, as they use balanced sampling instead of subsampling compared to DFR, but use group information instead of class information compared to cRT. For DFR, we report the performance of DFR and DFR Val
We train LWS, cRT and last layer re-training variations for 10 epochs with SGD, using cosine learning rate schedule decaying from 0.2 to 0; DFR implementation details can be found in Appendix C
36

---

Published as a conference paper at ICLR 2023
Class or group
Waterbirds WGA
CelebA
Method
Data
Balancing
balancing
WGA
Balanced
LWS
Kang et al.
2019)
Class
Train
40.03
35.55
sampling
Balanced
cRT
Kang et al.
2019)
Class
Train
sampling
74.48
52.8845.96
Balanced sampling 76.4811.24
Re-training last layer
Group
Train
56.11 16.77
Balanced 89.21+1.04
Re-training last layer
Group
Validation
sampling
67.6642.14
DFR IT
Ir
Group
Train
Subsampling
90.2
80.7
DFRT Val
Group
Validation
Subsampling
92.9
88.3
Table 12:
Comparison to LWS and cRT from Kang et al. (2019) and ablations. We compare Val It and DFR to LWS and cRT proposed in Kang et al. (019) for long-tail classification, as well as variations of last layer retraining on Waterbirds and CelebA dataset. The methods differ in (1) the data (train or validation) used for retraining last layer or logits scales in LWS, (2) whether class or group labels are used for balancing the dataset, and (3) the type of balancing which is either subsampling the dataset or using a balanced dataloader which first selects the class group label uniformly at random and then samples an example from that class or group. Notice the significant improvement that gain in terms of WGA as we change the data from train to validation, and as change the balancing from balanced sampling to subsampling. DFR Val performs best compared to other methods.
DFR
Since the original LWS and cRT methods were proposed to address the class imbalance problem, they perform poorly in terms of the worst group accuracy in the spurious correlation setting. LWS performs especially poorly, as it only retrains a single scaling parameter per class, which is not sufficient remove the reliance on spurious features. Re-training the last layer with balanced data sampling with respect to the group labels does improve performance compared to these original methods from Kang et al. (2019) as well as ERM, but underperforms compared both DFR versions. This ablation highlights the importance of subsampling compared to balanced sampling (see also Idrissi et al. 2021
DFRY.l achieves the best performance across the board. To sum up, for optimal performance, it is important to use held-out data (as opposed to re-using the train data) and to perform data subsampling according to group labels (as opposed to using group-balanced data sampling).
'The difference is less pronounced on Waterbirds with retraining on the validation set because the validation set is relatively group-halanced. In CelebA the group distribution the same in validation and training sets, so suhsampling performs much better than balanced sampling.
37


Reviews:
[{"review_id": "hi2nlgV-uSz", "reviewer": "Reviewer_v7pb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a surprisingly simple approach for mitigating spurious correlation via linear probing on a balanced dataset. The paper conducts comprehensive empirical studies on worst group robustness benchmarks and ImageNet variants. ", "summary_of_the_review": "The paper provides a simple method for mitigating spurious correlation. Although no theoretical analysis is provided, the paper conducts comprehensive empirical studies and is well written.  ", "strength_and_weaknesses": "Strengths \n- The proposed idea of retraining the linear head is simple and natural\n- The paper conducts extensive well-designed studies and the method demonstrates very competitive performance compared to recent works on worst group robustness. \n\nWeaknesses\n- The phenomenon that pre-trained networks still learn causal features even under severe spurious correlation in the training set is surprising. No theoretical explanations are provided to explain the phenomenon. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GHH6jRIm-Dy", "reviewer": "Reviewer_8GtD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the concern that neural networks trained for classification exhibit a bias towards learning spurious correlations that can negatively impact generalisation. In particular, this paper empirically demonstrates that despite learning spurious features and assigning them larger weights, neural networks can also learn more complex features simultaneously. Based on this observation, the authors propose a method, named Deep Feature Reweighting (DFR), to retrain the classification layer of a deep neural network with a small, controlled, balanced data set to reweight the importance of spurious and complex features.", "summary_of_the_review": "My overall impression of this paper is positive, since I believe the results provide strong evidence of the capability of neural networks to learn complex representations, even when the signal of spurious features is highly correlated with the labels. While this has been shown before and future work should shed more lights on the conditions when this is the case, among other open questions, I believe that this paper advances our understanding of the representations learnt by neural networks.", "strength_and_weaknesses": "### Strengths\n\nThis paper is conceptually simple, therefore easy to follow, and it is also mostly well-written. As an important strength, I highlight that the paper provides convincing empirical evidence that neural networks trained with empirical risk minimisation (ERM) are able to learn feature extractors of complex features, even in the presence of strong correlation of the labels with spurious features and the weights of the output layer mostly depend on the latter. In particular, the evidence come from two results: one, that such models obtain good (but far from optimal) performance on test sets without spurious correlations; two, that reweighting the output layer by training on a balanced data set achieves near optimal results. I think this strong evidence sharpens our understanding on the representation learning of neural networks trained with ERM and may inspire interesting avenues for future improvements on the generalisation of deep learning methods.\n\n### Weaknesses\n\nIn view of what I consider the main strength of the paper, it is also fair to highlight that this result is not completely novel, since it has been already suggested by Rosenfeld et al. (2022), a paper that the authors cite and briefly summarise. I belief that the overlap is substantial enough to deserve a more detailed discussion of the similarities and differences between the two works.\n\nBesides this comment on the novelty, in my opinion the paper places much importance on the proposed method to retrain the last layer of a pre-trained model by using a relatively small but controlled and balance data set. While the method proves to be effective at reducing the reliance on spurious features at the classification layer, the wide applicability of the method (and other related methods) is questionable, in that it requires both prior identification of the spurious features, as well as access to a balanced data set for re-training. I would argue that in general the main problem with spurious correlations is that they may be very hard to spot or identify a priori.\n\nRelated to the previous concern, in Section 4.2, the authors propose to use the \"decoded accuracy\" (accuracy after re-training the classification on a group-balanced validation set) as a way to analyse the reliance on \"high-quality representations\" (as opposed to the accuracy on the core-only set). However, how can the authors guarantee that the balanced validation set does not contain other spurious correlations (as in the example of the zero-vs-ones provided in the paragraph)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Diz6iO9Qjb", "reviewer": "Reviewer_AQfF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of spurious correlations. Authors propose to improve robustness against spurious correlations through a simple approach (Deep Feature Reweighting): specifically re-train the last layer of a neural network using a small set of reweighting\ndata where the spurious correlation does not hold. Further, they also show that despite relying on spurious correlations, the model still learns the core/invariant features. Empirically their proposed method achieves state-of-the-art performance.", "summary_of_the_review": "The paper is well-organized, well-motivated, and supported by empirical evaluations. Hence, I recommend acceptance.", "strength_and_weaknesses": "Strength:\n\n1. The paper is well-written and easy to follow. \n2. The proposed approach is simple to implement yet effective in combating spurious correlations.\n3. Experimental analysis in Section 4 is interesting and provides new findings.\n4. Empirically achieves SOTA performance.\n\nWeaknesses:\n1. Requires access to group-labeled samples. Although Table 5 shows it is a common assumption in the literature.\n\nQ1. It is difficult to understand why using group-labeled samples from training data performs worse than using group-labeled validation data. Are the number of samples in different groups used for reweighting kept similar for both DFR_Tr and DFR_val ?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AkOMCqoYhph", "reviewer": "Reviewer_xHWG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work shows that neural network models trained on datasets affected by the presence of simple spurious correlations also learn “core” features (i.e., features that are truly discriminative and non-spuriously correlated with the training labels). Given this observation the authors propose an algorithm named “Deep Features Reweighing” (DFR) that consists of retaining only the last layer using a “clean” validation set (i.e., a set that is not entirely affected by the spurious correlation present in the training set). \n\nResults are shown on Waterbirds, CelebA, MultiNLI and CivilComments datasets and performance are compared to different states of the art algorithms (Group DRO, JTT, CnC, SUBG, SSA). The results presented show that DFR outperforms many state of the art techniques in terms of worst-group and mean accuracy, and it is on-par with Group DRO. Unlike Group DRO, however, DFR does not require the training set to contain group-information (while this information is required for the validation set for both algorithms).", "summary_of_the_review": "The work is very interesting and it is worth sharing the findings with the community. While the novelty is limited (see [Kang2019]) the analysis are interesting. The paper needs, however, to either highlight the differences with [Kang2019] or change the focus to be on the application of a known technique (LWS) to this problem (and on the analysis already provided) rather than on presenting a new algorithm. I would be happy to upgrade my rating if the authors could provide an explanation about why DFR is different than LWS (including comparison) or change slightly the focus as just mentioned.\n", "strength_and_weaknesses": "Strength\n- The paper is well written and tackles an important problem: the effect of spurious correlation on ML models.\n- The analysis is interesting.\n- The algorithm is simple yet effective.\n\nWeaknesses\n- The biggest weakness in my opinion is the similarity with the idea presented in “DECOUPLING REPRESENTATION AND CLASSIFIER FOR LONG-TAILED RECOGNITION” [Kang2019], concretely the algorithm that Kang et al. refer to as Learnable Weight Scaling (LWS). The authors of the current paper did cite this work and state that the two works focus on different settings. While this might be true (as LWS focuses on long tail while DFR focuses on spurious correlation) in practice the algorithms seem identical. If I am correct, then the novelty of the current paper is mostly in the observations and analysis rather than in proposing a “new” algorithm. This should be stated and in such case there would be no need to introduce a new “name” for an existing algorithm. If the algorithms are similar, but not identical, then the difference with LWS should be described in more detail and the empirical comparison should appear in Table 2.\n- Ablation studies are missing and they could make the paper stronger and the finding more insightful, especially given the similarity with [Kang2019] additional ablation studies would increase the contribution. See below for some suggestions.\n- Some statements might need to be rephrased to represent the technique and the results more candidly. Se below for some suggestions.\n\nDetails\n1. “In order to make use of more of the available data, we train logistic regression 10 times using different random balanced subsets of the data, and average the weights of the learned models”. If averaging the model leads to important gain and it is what it is mostly reported in the experiments than it should described in Figure 1 and in the intro. This is actually a missing ablation study to understand the contribution of the weighting vs the simple rescaling. How much of the gain is related to the averaging of multiple models and how much is the use of a balanced unseen dataset? \n2. The authors say “we initialize the model with weights pretrained on ImageNet”. This naturally makes one wonder if the presence of core features is “just” due to this pre-training or if a model will learn core features even without ImageNet pre-training. It seems that this pre-training affects experiments in section 4.1 but possibly not 4.2? If this is the case I’d invite the author to state this important difference in the main paper (rather than differ this detail to the appendix). I’d be curious to see the results of 4.1 without pre-training, if confirmed I think the message would be stronger.\n3. The conclusions from 4.1 are too general given the pre-training step “we conclude that while the models trained on the Original data make use of background information to make predictions, they still learn the features relevant to classifying the birds almost as well as the models trained on the data without spurious correlations”. Without proving the point above (what happens when one trains only on the dataset affected by spurious correlation - without pre-training) this should be re-stated as “we conclude that while models that are pre-trained on ImageNet and fine-tuned on the Original data make use of background information to make predictions, they still learn the features relevant to classifying the birds almost as well as the models trained on the data without spurious correlations”\n4. Related to the above point. When using a model pre-trained on imagenet, what happens if one only fine-tunes directly on the validation set (skipping the training on the larger but biased set)?\n5. “For all DFR variations, the size of the reweighting set Dˆ is small relative to the number of features”  could be stated more precisely: what does small mean in this context?\n6. I suggest to de-emphisize if possible  DFR^train_train. It is a good experiment to show the need for the use of a new set of data. However, It is a bit distracting when read in the current order. I’d consider introducing it after the main result, as an ablation study on the use of the new vs old data. Currently it is presented at the beginning and it creates the expectation for this to work comparably to D^Val_Train.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%