Towards Last-layer Retraining for Group Robustness with Fewer Annotations
Authors: Tyler LaBonte
Vidya Muthukumar
Abhishek Kumar
Venue: NeurIPS 2023 poster (2023)
Keywords: spurious correlations
group robustness
last-layer retraining
distribution shift
TL;DR: We examine the capability of last-layer retraining to improve robustness to spurious correlations without group annotations.

Abstract: Empirical risk minimization (ERM) of neural networks is prone to over-reliance on spurious correlations and poor generalization on minority groups. The recent deep feature reweighting (DFR) technique achieves state-of-the-art group robustness via simple last-layer retraining, but it requires held-out group and class annotations to construct a group-balanced reweighting dataset. In this work, we examine this impractical requirement and find that last-layer retraining can be surprisingly effective with no group annotations (other than for model selection) and only a handful of class annotations. We first show that last-layer retraining can greatly improve worst-group accuracy even when the reweighting dataset has only a small proportion of worst-group data. This implies a "free lunch" where holding out a subset of training data to retrain the last layer can substantially outperform ERM on the entire dataset with no additional data, annotations, or computation for training. To further improve group robustness, we introduce a lightweight method called selective last-layer finetuning (SELF), which constructs the reweighting dataset using misclassifications or disagreements. Our experiments present the first evidence that model disagreement upsamples worst-group data, enabling SELF to nearly match DFR on four well-established benchmarks across vision and language tasks with no group annotations and less than 3% of the held-out class annotations.

Towards Last-layer Retraining for Group Robustness with Fewer Annotations
Tyler LaBonte' Vidya Muthukumar Abhishek Kumar
H. Milton Stewart School of Industrial and Systems Engineering. Georgia Institute of Technology
School of Electrical and Computer Engineering, Georgia Institute Technology
Google DeepMind
(tlabonte, vmuthukumarr) edu abhishk@google.com
Abstract
Empirical risk minimization (ERM) of neural networks is prone to over-reliance spurious correlations and poor generalization minority groups. The recent deep feature reweighting (DFR) technique [33] achieves state-of-the-art group robustness via simple last-layer retraining, but requires held-out group and class annotations to construct a group-balanced reweighting dataset. In this work, we examine this impractical requirement and find that last-layer retraining can be surprisingly effective with no group annotations (other than for model selection) and only handful of class annotations. We first show that last-layer retraining can greatly improve worst-group accuracy even when the reweighting dataset has only a small proportion of worst-group data. This implies "free lunch" where holding out subset of training data retrain the last layer can substantially outperform ERM further improve
on the entire dataset with no additional data or annotations. group robustness, we introduce lightweight method called selective last-layer fine- tuning (SELF), which constructs the reweighting dataset using misclassifications or disagreements. Our empirical and theoretical results present the first evidence that model disagreement upsamples worst-group data, enabling SELF to nearly match DFR on four well-established benchmarks across vision and language tasks with no group annotations and less than 3% of the held-out class annotations. Our code is available at https: : thub. com/ wtw.aaateaeeeeeeeeeeeeeeee llatteeeeelaaaeeeeeeeeeeeeeet -ayer-la-r-r--- weayeer-retrannngg
1 Introduction
Classification tasks in machine learning often suffer from spurious correlations: patterns which are predictive of the target class in the training dataset but irrelevant the true classification function. These spurious correlations, often in conjunction with the target class, create minority groups which underrepresented in the training dataset. For example, in the task of classifying cows and camels, the training dataset may be biased so that a desert background spuriously correlated with the camel class, creating minority group of camels grass backgrounds [5]. Beyond this simple scenario, spurious correlations have been observed in high-consequence applications including medicine [79], justice [9], and facial recognition [42].
Neural networks trained via the standard procedure of empirical risk minimization (ERM) [68], which minimizes the average training loss, tend to overfit spurious correlations and generalize poorly on minority groups [18]. Even worse, it possible for ERM models to rely exclusively on the spurious feature and incur minority group performance that is no better than random guessing [60]. Therefore, maximizing the model's group robustness, quantified by its worst accuracy on any group, desirable objective in the presence of spurious correlations [58].
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In contrast to more generic distribution shift settings (e.g., domain generalization [35]), the presence of spurious correlations enables the improvement of group robustness merely by addressing model bias (without collecting additional minority group data). The recently proposed deep feature reweighting (DFR) [33, 28] technique efficiently corrects model bias by retraining the last layer of the neural network, simple procedure which achieves state-of-the-ar group robustness. The key hypothesis underlying DFR is that ERM models which overfit to spurious correlations still learn core features that correlate with the ground-truth label on all groups, but they perform poorly because they overweight the spurious features in the last layer. Ostensibly, retraining the last layer on a group- reweighting dataset would then upweight the core features and improve porst-group accuracy.
DFR compares favorably to existing methods such as group distributionally robust optimization (DRO) [58], which requires group annotations for the entire training dataset. However, DFR still necessitates a smaller reweighting dataset with group and class annotations to achieve maximal performance [33]. This requirement limits its practical application, as the groups are often unknown ahead of time or difficult annotate (e.g,, due to financial, privacy, or fairness concerns).
Our contributions
In this paper, we present comprehensive examination of the performance of last-layer retraining in the absence of group and class annotations on four well-established benchmarks for group robustness across vision and language tasks.' We first investigate the necessity of the reweighting dataset being balanced across groups, and we show that last-layer retraining can substantially improve worst-group accuracy when the reweighting dataset has only small proportion of worst-group data. Based on this observation, we propose class-balanced last-layer retraining as a simple but strong baseline for group robustness without group annotations. We show that, average over the four datasets, this method achieves 94% of DFR worst- grosp accuracy compared to 76% without class balancing.
The strong performance of class-balanced last-layer retraining reveals "free lunch" with practical ramifications. While bllaseel ERM was recently proposed by [27] as a competitive baseline for worst-group accuracy, we show that instead of using the entire training dataset for ERM, dependence on spurious correlations can be reduced by randomly splitting the training data in two, then performing ERM training on the first split and class-balanced last-layer retraining on the second. Our experiments indicate that this technique can improve worst-group accuracy by up to 17% over class-balanced ERM on the original dataset using additional data annotations (even for model selection) surprising and unexplained result given that the two splits have equally drastic group imbalance.
While retraining the last layer on a balsnce held-out dataset can be effective, it is inferior to DFR when group imbalance large. To close this gap. we propose selective last-layer finetuning (SELF), which selects a small, more proup-balanced reweighting dataset and finetunes the last layer instead of retraining. We implement SELF using points that are either misclassified or disagree in their predictions relative to regularized model. Disagreement SELF does not require class annotations for the entire held-out set (only for the disagreements). We show that disagreement SELF between the ERM and early-stopped models performs the best in general, surpassing class-balanced last-layer retraining by up to 12% worst-group accuracy with less than 3% of the held-out class annotations.
Overall, our work shows benefits of last-layer retraining well beyond group- balanced held-out dataset. Our results call for further investigation of the DFR hypothesis: while group balance the most important factor in DFR performance, we show that a significant gain is solely due to class balancing, and the performance discrepancy between misclassification SELF and disagreement SELF suggests that worst-group accuracy may be affected by characteristics of the reweighting dataset other than group balance. Our main results are summarized and compared to previous methods in Table 1.
2 Related work
This work subsumes and improves our two previous workshop papers: [38] broadly covers Section 4, while [37] represents preliminary investigation into Section 5, which we have substantially updated with new methodology, evaluation, and theoretical analysis for this version.
Our methods in Section 4 do not require group annotations at all, while our SELF method in Section 5 assumes access to small validation set with group annotations for model selection following previous work [58, 41, 48, 33, 28J. We show in Appendix B that SELF is robust even with 1% of these annotations.

---

Table 1: Comparison other group robustness methods. We discuss our DFR implementation in Section 3, and our proposed methods of class- baas-ebl (CB) last-layer retraining and early-stop (ES) disagreement SELF in Sections 4 and 5, respectively. Class-balanced ERM is trained on the combined training and held-out datasets. DFR uses group annotations on the held-out dataset, while Group DRO-ES requires them for the training dataset. ES disagreement SELF uses class annotations on the training dataset (for ERM), but requests few 20 labels from the held-out dataset. All methods except ERM and CB last-layer retraining use a small set of group annotations for model selection. We list the mean and standard deviation over three independent runs.
Annotations
Worst-group test accuracy
Method
Group
Class
Waterbirds
CelebA
CivilComments
MultiLII
ERM
81.9
67.245.6
61.420.7
69.2+1.
JTT [41, 27]
85. 65.6
75.647.7
67.5 67.5
RWY-ES (27,28)
X
V
74.5+0.0
76.841.7
78.9+1.0
68.0+0.4
CnC
X
88.5+0.3
88.8
CB last-layer retraining
V
92.6+0.8
73.7+
80.4
64.7+1.1
ES disagreement SELF
X
93.040.3
83.910.9
79.1
70.7
DFR (our impl.)
V
92.440.9
87. 87.041.1
81,841.6
70.8
DFR [33, 28]
91.1
89.4+0.9
78.8
72. 6
Group DRO-ES [58, 28]
90.7
90.611.6
80.4
73.5
Spurious correlations. The performance of empirical risk minimization (ERM) in the presence of spurious correlations has been extensively studied [18]. In vision, ERM models are widely known to rely on spurious attributes like background [58, 75], texture [17]. and secondary objects [56, 61, 62] to perform classification. In language, ERM models often utilize syntactic or statistical heuristics as a substitute for semantic understanding [20, 50, 46). This behavior can lead to bias against demographic minorities [25, 6, 67, 23, 8] or failure high-coneequence
high-consequence applications [42, 9, 79J.
Robustness and group annotations. If group annotations are available in the training dataset, group distributionally robust optimization (DRO) [58] can improve robustness by minimizing the worst-group loss, while other techniques learn invariant or diverse features [1, 19, 80, 76J. Methods which use only partial group annotations include deep feature reweighting (DFR) [33, 28], which retrains the last layer performs DRO with group pseudo-labels. Recently, more lightweight methods that only adjust the model predictions using group annotated held-out set have also been proposed [70]. However, since the groups are often unknown ahead of time or difficult to annotate in practice, there has been significant interest in methods which do not utilize group annotations except for model selection. The bulk of these techniques utilize auxiliary models to pseudo-label the minority group or spurious feature [63, 47, 77, 11. 31, 66, 32, 64, 81]; notably. just train twice (JTT) [41] upweights samples misclassified by an carly-stopped model. Other techniques reweight or subsample the classes [27] or train with robust losses and regularization [54, 78].
sroup-balanced held-out set, and spread spurious attribute [48], which
Unlike DFR, our methods utilize no held-out group annotations, and unlike JTT, we do not have to "train twice", only finetune the last layer. We also show that SELF performs best using disagreements between the ERM and carly-stopped models instead of misclassifications as in JTT. Moreover, while JTT assumes the early-stopped model has low worst-group accuracy which improves during training, SELF performs well even when the early-stopped model has high worst-group accuracy which decreases during training - substantially improving performance on datasets such as CivilComments. The concurrent work of Qiu et al. [55] proposed a similar method to our Section 4.2; while they use a tunable loss upweight misclassified samples, our method shows that similar results can be achieved with no hyperparameter tuning (and therefore no group annotations for the validation set). Finally, while our results partially corroborate the findings of Idrissi et al. [27] that class balancing during ERM is effective for group robustness, we observe that class balanced last-layer retraining renders ERM class balancing optional. We compare our results with previous methods in Table 1.
Generalization via disagreement. Disagreement based active learning for improving in- distribution generalization has been well-studied since before the deep learning era [10, 4, 21]. More recent research has utilized disagreements between SGD runs to predict in-distribution gen-
3

---

Table 2: Dataset composition. We study four establissseed benchmarks for group robustness across vision and language tasks. The class probabilities change dramatically when conditioned on the spurious feature. Note that Waterbirds is the only dataset that has a distribution shift and MultiNLI the only dataset which is
a priori. Probabilities may not sum to due to rounding.
Group
Training distribution p
Data quantity
Dataset
Class y
Spurious s
p(y)
p(g)
p(y]s)
Train
Val
Test
landbird landbird
land
.730
.984
3498
467
2225
.768
water
.038
.148
184
466
2225
Waterbirds
waterbird
land
.012
.016
56
133
642
.232
waterbird water
.220
.852
1057
133
6422
non-blond
female male
.400
.758
71629
8535
9767
.851
non-blond
.111
.980
66874
8276
7535
CelebA
blond female blond male
.141
.212
22880
2874
2480
.149
.009
.020
1387
182
180
neutral neutral toxic toxic
no identity identity no identity identity
.551
.921
148186
25159
74780
.877
.36
.836
90337
14966
43778
CivilComments
.047
.079
12731
2111
6455
.133
.066
.664
17784
2944
8769
contradiction contradiction
no negation negation no negation negation no negation negation .333
.299
330
57498
22814
34597
.33
.054
.761
11158
4634
6655
entailment
.327
.352
67376
26949
40496
MultiNLI
.34
entailment neither
.007
.104
1521
613
886
.323
.348
66630
26655
39930
neither
.010
.136
1992
797
1148
eralization [30], as well as between model classes (e.g., CNNs and Transformers) to predict out-of- distribution generalization [3]. Two works that are concurrent to ours, diversity by disagreement [52] and diversify and disambiguate [40], are methods for generalization under distribution shift which maximize the disagreement between multiple predictors to learn diverse ensemble. Compared to our work, these methods are optimized for training datasets which exhibit a complete correlation (i.e., contain minority group data) and they underperform in the less extreme spurious correlation setting we study.
Preliminaries
Setting. We consider classification tasks with input domain X' and target classes V. We assume S is a set of spurious features such that each sample & E X' is associated with exactly one feature 8 E S. In conjunction with the target class, the spurious features partition the dataset into groups While the groups may be heavily imbalanced in the training distribution, we desire model which is invariant to the spurious feature and thus has roughly uniform performance over Therefore, evaluate worst group accuracy (WGA), Le., the minimum accuracy among all groups [58].
y S.
We will often refer datasets and models group-balanced ballanced or expectation, the dataset composed of an equal number of samples from groups in 9 or classes in Vi respectively. This balance can be achieved by training on a subset with equal data from each group/class, or sampling from the data so that each minibatch is balanced in expectation [27]. To make the latter more concrete, for group balancing where ý is the training distribution; class balancing is the same with y instead of S. We use the minibatch sampling approach for both dass-balanced ERM and dlas-bblancee last-layer retraining, and we provide a comparison with the subset method in Appendix A.
meaning that in
first sample
Unif(S), then sample x (((1s)
Deep feature reweighting. The recently proposed deep feature reweighting (DFR) [33, 28] method achieves state-of-the-ee WGA by performing ERM on the training dataset, then retraining the last layer of the neural network on a ggrop-baalatte balanced held-out dataset, called the reweighting dataset. In the original implementation, half the validation set is used to construct the reweighting dataset: all data from the smallest group included and the other groups are randomly downsampled that size.

---

Then, the feature embeddings (i.e. the outputs of the penultimate layer) of the reweighting dataset are pre-computed and used to train logistic regression model with explicit & --regularization. The results are averaged over 10 randomly subsampled reweighting datasets, and hyperparameter search performed over & regularization strength on the other half of the validation set.
To emphasize practicality and efficiency, our implementation of DFR has some differences from the original. (i) Instead of logistic regression, we train the last layer on the reweighting dataset via minibatch optimization using SGD and AdamW [43] for the vision and language tasks, respectively. This fits well into standard training pipelines and avoids pre-computing the feature embeddings and writing them to disk, which can be slow and memory-intensine hyperparameters, we use a fixed-value lg regularization instead of searching over ε regularization strength. We observed similar performance for & and Eg regularization, which we believe is because l e-regularieed gradients do not induce sparsity [39]. (ii) We sample uniformly at random from the groups in the held-out dataset (to get group balanced minibatches) instead of averaging over group-balanced subsets of the data [27]. We compare the implementations in detail in Appendix A.
(ii) To reduce the number of
Datasets and models. We study four datasets which are well-established as benchmarks for group robustness across vision and language tasks, detailed Table 2 and summarized below.
Waterbirds [71, 69, 58] is image classification dataset where the task is to predict whether a bird is a landbird or a waterbird. The spurious feature is the image background: more landbirds are present on land backgrounds than waterbirds, and vice versa."
CelebA [42. 58] is an image classification dataset where the task is to predict whether person is blond or not. The spurious feature is gender, with 16x more blond women than blond men in the training set.
CivilComments [7. 35] is a text classification dataset where the task is to predict whether a comment is toxic or not. The spurious feature is the presence of one of the following categories: male, female, LGBT, black, white, Christian, Muslim, or other religion." More toxic comments contain one of these categories than non-toxic comments, and vice versa.
MultiNLI [73, 58] is a text classification dataset where the task is predict whether pair of sentences is a contradiction, entailment, or neither. The spurious feature a negation in the second sentence more contradictions have this property than entailments or neutral pairs.
Importantly, Waterbirds is the only dataset that has distribution shift its validation and test datasets are group-balanced conditioned on the classes, although still daas-immallanced. 4 As result, methods which use the validation set for training can improve performance without explicit group balancing.
We utilize ResNet-50 [24] pretrained on ImageNet- 1K [57] for Waterbirds and CelebA, and BERT [14] model pretrained on Book Corpus [83] and English Wikipedia for CivilComments and MultiNLI Following previous work, we use half the validation set for feature reweighting [33, 28) and half for model selection with group annotations [58, 41, 33, 48, 28). We run each experiment on three random seeds and do not utilize explicit early stopping except as part of SELF (see Section 5). See Appendix D for further details on the training procedure.
4 Class-balanced last-layer retraining
In this section, we investigate the necessity of group- balanced reweighting dataset for DFR and show that class dlss-bblanced last-layer retraining is a simple but strong baseline for group robustness without group annotations. To enable a fair comparison with our implementation of DFR (see Section 3), class-balanced last-layer retraining follows the same training procedure, except the reweighting dataset is constructed by sampling uniformly over the classes y instead of the groups G.
We note that the Waterbirds dataset is known to contain incorrect labels [66]. We report results on the original, un corrected version for a fair comparison with previous work.
"This version CivilComments has four groups, used in this work and by [58, 28, 33]. There is another
version where the identity categories are not collapsed into one spurious feature; this version is used by [41, 81], so we do not report their CivilCommente accuracies in Table 1. Both versions use the WILDS split [35].
"The Waterbirds validation and test datasets still contain more landbirds than waterbirds, but each class has equal quantities of land and water backgrounds.
5

---

100
(osr CAN
80
Wetertirds
Wetertinds
CavbA
DavbA
CiviCommems
Ch Commemsess
MUNNLI
MUNNLI
Inrriggruup
wand--grou
(a) Worst-group test accuracy
(b) Relative % of max WGA increase ERM
How much worst-group data does last-layer retraining really need? We perform an ablation on the percentage of worst-group data used for bass-balanced last-layer retraining, while keeping the total data constant. The results show that last-layer retraining can substantially improve worst-group accuracy even when the reweighting dataset has only a small proportion of worst group data, and that class balancing can be a major factor in its performance. The underperformance on Waterbirds below 20% is because there is too little worst-group data to observe consistent model behavior (less than ten samples). The stars * denote the baseline percentage of worst-group data in the training dataset. We plot the mean over three independent runs. See Tables 12 and for details.
Figure 1:
Table 3: Last-layer retraining the held-out dataset. While unbalanced (UB) last-layer retraining decreases performance, Blassballanced (CB) last-layer retraining nearly matches DFR on Waterbirds and CivilComments. However, still trails DFR CelebA and MultiNLI; with the SELF method described in Section 5. CB ERM is trained on the combined training and held-out datasets using Mlass-balance minibatches. We list the mean and standard deviation three independent runs.
improve these results
Group annotations
Worst-group test accuracy
Method
Waterbirds
CelebA CivilComments
MultiNLI
Class-balanced ERM UB last-layer retraining
X
81.92a.4
67.2
61.4
69.2
X
88.02o.s
41.9+1.4
57.624.2
64.621.0
CB last-layer retraining
X
92.640.8
73.7+2.8
80.4+0.8
61.7+1.1
DFR (our impl.) DFR [33, 28]
V
92.4
87 87.0+1.1
81.841.6
70.8
/
91.1
89.4
78.8
72.610.3
4.1 An ablation on the proportion of worst-group data
In this section, perform ablation the percentage of worst-group data in the reweighting dataset and show that class-balanced last-layer retraining can substantially improve WGA even when the reweighting dataset has only a small proportion of worst group data. For ablation, choose the worst groups based on the performance of ERM, and if two groups have similarly poor performance, we vary both. The worst groups for Waterbirds are landbirds on water backgrounds and waterbirds on land backgrounds; for CelebA blond men; for CivilComments toxic comments containing identity categories; and for MultiLLI entailments and neutral pairs containing negations.
For these experiments, we begin with the DFR reweighting dataset, i.e., a random group-balancee balanced subset of the held-out dataset, where the size of each group is the minimum of the worst-group size and half the size of any other group. We define this dataset to include 100% of the worst-grouu data. We then reduce the percentage of worst-group data, while correspondingly increasing the percentage of data from the same class without the spurious feature. For example, the DFR reweighting dataset on CelebA has 92 points from each group, reducing the worst group to 25% results in 92 non-blond females, 92 non-blond males, 161 blond females, and 23 blond males. The total data is kept constant.
'This is necessary because we keep the total data constant. By reducing the quantity of worst-group data to will most double the size of any other group.

---

80
90
Clensbnderodd (CB)) ERM
Cree-coenned (CRy FRM
07
CB ERM and GB lestopper reineringg
80
CB ERM and CB boat-aye
80
80
78
Worl Morsiggraa test
7D
70
88
50
Waterbrda
CaMbA
CiviCommnns
MUBINII
50
Weterbidds
CalebA
CniCCemmnti
MuINU
(a) Training dataset only
(b) Combined training and held-out datasets
Figure A "free lunch" in group robustness. compare class-balanced (CB) ERM on the entire dataset to splitting the dataset and performing CB ERM the first (95%) split and CB last-layer retraining on the second (5%) split. This technique improves worst-group accuracy on Waterbirds, CelebA, and CivilComments by up to 17% while using additional data annotations for training beyond ERM. We believe underperforms on MultiNLI because there not enough data in the first split, i.e., ERM performance can be improved by collecting more data. We plot the mean and standard deviation over three independent runs. See Table for detailed results.
The results of this ablation are displayed in Figure 1. While a smooth increase in WGA with increasing worst-group data is expected, the extent of the early increase is surprising: over the four datasets, an average of 67% of the increase in WGA over glass-unbalanceed ERM obtained by the first 25% of worst-group data. In particular, CelebA and CivilComments the most class-imbllanees datasets we study experience significant improvement even at low percentages of worst-group data. This phenomenon suggests that, while group balancing is still important for best results, class balancing is a factor the performance of DFR on these two datasets.
Furthermore, class- balance last-layer retraining can improve worst-group accuracy even when the held-out dataset has similar group imbalance the training dataset (i.e., at the stars * in Figure I). Based on this observation, we propose that glass-balanced last-layer retraining on the entire held-out dataset can simple but strong baseline for group robustness without group annotations. Table 3 details the results; on average over the four datasets, Mlass-balanced last-layer retraining achieves 94% of DFR performance, compared to 76% without class balancing. However, it still trails DFR by significant amount on CelebA and MultiNLI the most group-imbalanced datasets we study. We improve these results with our selective last-layer finetuning (SELF) method described Section 5.
Moreover, our experiments in Figure (deferred to Appendix A) indicate that class-balanced last layer retraining has similar performance regardless of whether it initialized with lass-unballaced or class-balen balanced ERM features. Contrasting with Idrissi et al. [27]. our results suggest that class balancing the ERM stage optional. This result has practical relevance for expensive models pre- pre-trained without class balancing (e.g., large language models), as the benefits of class balancing on downstream tasks can be reaped by simply retraining the last layer instead of training a new model.
4.2 A "free lunch" in group robustness
Motivated by the promising results of Section 4.1, where we performed class-balanced last-layer retraining a fixed held-out set, we now ask how can best utilize a realistic training dataset? In particular, practical applications often work with predetermined data, annotation, and compute budget. Within this budget, and with no explicit held-out dataset, would one achieve better group robustness by using the entire dataset for ERM or by holding out subset for last-layer retraining?
We investigate this question into two, then performing Mlass-balanced ERM training on the first split (95% of the data) and class-bales balancd last-layer retraining on the second split (5% of the data). Figure 2a illustrates the results of our experiments on the training dataset and Figure 2b on the combined training and held-out datasets. Since the quantity of data is higher in the case of combining the training and held-out datasets, we expect all numbers to be higher in Figure 2b compared to Figure 2a (especially on
four benchmark datasets by randomly splitting the initial dataset

---

Table 4: Comparison selective last-layer finetuning methods. SELF nearly matches DFR and improves WGA over class-balanced (CB) last-layer retraining by up to 12%. Early-stop (ES) dis- agreement SELF performs the best overall, and disagreement methods perform especially well on CivilComments. Dropout and ES disagreement request few as class annotations from the held-out dataset. All SELF methods use small set of group annotations for model selection. We list the mean and standard deviation over three independent runs.
Method
Held out annotations
Worst-group test accuracy
Group
Class
Waterbirds
CelebA
CivilComments
MultiNLI
Class-balanced ERM CB last-layer retraining
/
81.943.4
67.2
61.4.0.7
6..2
X
/
92.640.8
73.7
80.4.0.a
64.7+1.1
Random SELF Misclassification SELF
X
91.1+1.9
80.2+6.4
80.541.6
65.0444.0
X
V
92.640.a
83.0+6.1
62.7+4.8
72.2+2.2
ES misclassification SELF
X
V
92.2+b.7
80.443.9
65.8+7.4
73.3
Dropout disagreement SELF
X
92.3+0.5
85.7
69.9
68.713.4
ES disagreement SELF
X
93.0+0.3
83.920.9
79.1
70.742.5
DER (our impl.) DFR [33, 28]
V
92.410.9
87.0+1.1
81.811.
70.8
/
91.110.8
89.4
78.8
72.610.3
Waterbirds, which has a more group-balances validation set). We perform no hyperparameter tuning for these experiments, and therefore we do not utilize group annotations even for model selection.
Figure 2 indicates that splitting the dataset and performing last-layer retraining substantially improves worst-group accuracy Waterbirds, CelebA, and CivilComments. It decreases performance on MultiNLI, which is the only dataset where adding held-out data from the same distribution signifi- cantly increases ERM worst-group accuracy compared DFR. Specifically, class-balanced ERM achieves 67.4 + 2.4% WGA on the training dataset and 69 2 = 1.6% on the combined training and held-out datasets, while our DFR implementation achieves 70.8 + 0.8%. Therefore. we hypothesize that last-layer retraining on the second split can improve group robustness only if there is enough data for FRM to perform near-optimally on the first split, i.e,, the performance of ERM on the first. split is limited by dataset bias rather than sample variance.
Based on this hypothesis, holding out 5% of data, perform last-layer retraining on the held-out dataset instead of ERM on the initial dataset. We call this technique a "free lunch" because improves worst-group accuracy with no additional data or annotations beyond ERM (including for model selection). In particular, we utilize less data for ERM training and less compute due the efficient nature of last-layer retraining. Therefore, we believe this method is especially relevant to practitioners, and it can be easily implemented with little change to data processing or model training workflows.
answer to the posed question is: if ERM performance is stable when
critical remaining question is why last-layer retraining improves group robustness; since the training and held-out datasets have equally drastic group imbalance, it is counterintuitive that reducing the quantity of data used for ERM and performing last-layer retraining would increase worst-group accuracy. In some cases, as seen on MultiNLI in Figure 2, training an ERM model with less data can be detrimental - but on the other three datasets, last-layer retraining substantially improves over ERM. We leave future empirical and theoretical work better understand this phenomenon.
5 Selective last-layer finetuning
While blass-balanced last-layer retraining can improve worst- group accuracy without group annota- tions, its performance still inferior to DFR, particularly on the highly group-imbalanced Celeb and MultiNLI datasets. In these cases, training the entire held-out set can be detrimental; instead, we show that constructing the reweighting dataset by detecting and upsampling worst-group data more effective. We propose selective last-layer finetuning (SELF), which uses auxiliary model to select a small, more group-balanced reweighting dataset, then finetunes the ERM last layer instead of retraining entirely to avoid overfitting this smaller dataset.

---

SELF does not use group annotations for training and can be implemented even when class annotations are unavailable in the held-out dataset, making applicable settings not captured by current techniques (e.g., adaptation to an unlabeled target domain [45]). If class annotations are not available, we select points which are disagreed upon by ERM and regularized model, then request their labels. If class annotations are available, we can still use disagreement, or alternatively, we select points which are misclassified by the ERM or early-stopped model. Our experiments indicate that disagreement SELF between the ERM and early-stopped models performs the best overall, increasing the gorst-group accuracy of class-balanced last-layer retraining to near-DFR levels while requesting less than 3% of the held-out class annotations. Our results are detailed in Table 4.
We formalize SELF as follows. Let f.g X + RD be models with logit outputs, X S X' be held-out dataset, c TR + IR be cost function, and n be an integer. SELF constructs the reweighting set D by greedily selecting the n points in X whose outputs incur the highest cost, i.e.
argmax Ed((r))g())). SEX, 8|=n #E
Then, SELF requests class annotations and performs class-balanced finetuning (i.e., starting from the ERM weights) of the last layer of the ERM model on D, using the same implementation last-layer retraining (see Section 4). We study the following four variants of SELF:
Misclassification f is ERM model and g is the labeling function, i.e., for datapoint x with label y, the output g(x) is the one-hot encoded vector for y. The cost function e is cross entropy loss.
Early-stop misclassification." I is an early-stopped ERM model and g is the labeling function. The cost function c is cross entropy loss.
Dropout disagreement: f is an ERM model and g is the same model where inference is run with nodes randomly dropped out in the last layer [65]. The cost function cis KL divergence with softmax.
Early-stop disagreement: I is an ERM model and g is an early-stopped ERM model. The cost function is KI. divergence with softmax.
We also experiment with a pure random baseline, referred Random SELF in Table 4, where D consists of n random points from X. While Random SELF suffers high variance as expected, perhaps surprisingly its performance still competitive on average. The reweighting dataset size n is a hyperparameter, and for disagreement SELF it corresponds to the quantity of class annotations requested. Please see Appendices B and D for hyperparameter details and ablation studies.
5.1 Analysis of SELF performance
To quantify the extent of SELF's upsampling, we plot the percentage of the reweighting dataset consisting of worst-group data in Figure In particular, we present the best of our knowledge the first evidence that model disagreement effectively upsamples worst-group data, an observation which may be of independent interest. With that said, Figure 3 does not formally establish that better group balance has a strong correlation with WGA rather, it suggests an additional subtlety in the DFR hypothesis, since group balance alone does not fully explain the performance of last-layer retraining. In addition to the balance of the reweighting dataset, it is likely that characteristics of the specific data selected also contribute to SELF results. Additionally, the competitive performance of pure random SELF in Table 4 further questions the importance of group balancing in DFR.
The importance of which data selected could explain why disagreement SELF often outperforms misclassification SELF despite having access to less information. While misclassification selects the most difficult or noisiest points, disagreement selects the most uncertain points. For example, dropout models approximate a theoretically justified uncertainty metric [16] which is likely to be higher on worst-group data, and early-stoppe majority group. Training on the most uncertain data is a key tenet of active learning [13, 59, 12], which rationalizes the performance of disagreement SELF and provides motivation for further investigation of why last-layer retraining improves group robustness.
models tend to fit simple patterns first [2, 41] including the
Using carly-stop misclassification to upsample worst-group data is the premise of JTT [41].

---

Beseline
% of worst-grouu data
60
Balanced
Micdass focaton
ES minc
disagmeeent
40
ES dissg werment
20
Waterbirds
CelebA
CMilComments
MuhiNNI
Figure SELF upsamples the worst group. We plot the percentage of the reweighting dataset consisting of worst-group data for each SELF method. "Baseline" represents the percentage of worst-group data in the held-out dataset, while "Balanced" is the percentage of worst-group data necessary to achieve group balance. This is to the best of our knowledge the first empirical evidence that model disagreement is an effective method for upsampling worst-group data. The worst groups in each dataset are listed in Section 4.1. We plot the mean for the best dataset size n over three independent runs. See Table 15 for detailed results.
A remaining question is why the performance of disagreement SELF is strong CivilComments compared to misclasiification-aass methods such as JTT [41] and ES misclassification SELF. We show in Figure 6 (deferred to Appendix B) that, contrary to the assumptions made by JTT and other early-stop misclassification methods, the worst-group accuracy decreases with training on CivilComments. This highlights potential advantage of disagreement methods over misclassification methods, as disagreement is justified regardless of whether the regularized or ERM model has greater dependence on spurious features. Moreover, while misclassification methods do indeed upsample the minority group in Figure 3, we show in Table 10 (deferred to Appendix B) that the held-out training accuracy of misclassification SELF tends be very low down 0% on MultiNLI evidence that misclassifications are too difficult to learn without modifying the features.
Theoretical proof-of-concept. The observations of Section 5.1 raise the question of whether disagreement SELF can ever be shown to provably upsample minority group points. We provide simple proof-of-concept for SELF by considering the last layer to be linear model (which could be under- or overparameterized) with core. spurious and junk features. We show that the disagreement between the regularized model and the ERM model (measured by total variation distance between the predicted distributions) is provably higher on minority examples than majority examples, regardless of model dependence on the spurious feature. particular, this shows provable benefits of disagreement SELF even in situations where the early-stopped model has higher worst-group accuracy than the convergent model, which may not be captured by related methods the literature [41]. Our detailed setup, assumptions, and main theoretical result (Theorem 1) are stated in detail in Appendix C.
6 Conclusion
In this paper, we presented a comprehensive examination of the performance of last-layer retraining in the absence of group and class annotations. We showed that class-balanced last-layer retraining is a simple but strong baseline for group robustness, and that holding out a subset of the training data to retrain the last layer substantially improve worst-group accuracy. We then proposed selective last-layer finetuning (SELF), whose early-stop disagreement version improves performance near-DFR levels with no group annotations and less than 3% of the held-out class annotations.
Our work has generated several open questions which could prove fruitful for further research. First, why does last-layer retraining on held-out split of the training dataset improve group robustness (see Section 4.2)? Second, to what extent does the precise data selected matter for reweighting, e.g., which of disagreement SELF and misclassification SELF would perform better when hyperparameters are set to equalize group balance? Third, what the optimal strategy in general for obtaining the regularized model in disagreement SELF (dropout, early-stopping, different technique) and why?
10

---

Acknowledgments. We thank Google Cloud for the gift of compute credits, Jacob Abernethy and Eva Dyer for the additional compute assistance, and anonymous reviewers for their helpful feedback. We also thank Harikrishna Narasimhan for reading an earlier version of the manuscript and providing thoughtful comments and Seokhyeon Jeong for catching an error in an earlier version of the code. T.L. acknowledges support from the Dol NDSEG Fellowship. V.M. acknowledges support from the NSF (through CAREER award CCF-2239151 and award IIS-2212182), an Adobe Data Science Research Award, Amazon Research Award, and Google Research Collabs Award.

---

References
[1] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint 1907.02893, 2019. Cited page 3.
[2] Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In International Conference on Machine Learning (ICML), 2017. Cited on page 9.
[3] Christina Baek, Yiding Jiang, Aditi Raghunathan, and Zico Kolter. Agreement-on-the-leeee Predicting the performance of neural networks under distribution shift. In Conference on Neural Information Processing Systems (NeurIPS), 2022. Cited on page
[4] Maria-Florins Balcan, Alina Beygelzimer, and John Langford. Agnostic active learning. In International Conference on Machine Learning (ICML), 2006. Cited on page 3.
[5] Sara Beery, Grant van Horn, and Pictro Perona. Recognition in terra incognita. In European Conference on Computer Vision (ECCV), 2018. Cited on page
[6] Su Lin Blodgett, Lisa Green, and Brendan O' O'Connor. Demographic dialectal variation in social media: A case study of African-American English. In Empirical Methods in Natural Language Processing (EMNLP). 2016. Cited on page 3.
[7] Daniel Borkan, Lucas Dixon, Jeffrey Sorenson, Nithium Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In World Wide Web (WWW), 2019. Cited on page 5.
[8] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability, and Transparency in Machine Learning (FATML), 2018. Cited on page 3.
[9] Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. In Conference on Fairness, Accountability, and Transparency in Machine Learning (FATML), 2016. Cited on pages and 3.
[10] David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine Learning, 15())201-222, 1994. Cited on page 3.
[11] Elliot Creager, Jömn-Henrie Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference Machine Learning (ICML), 2021. Cited page 3.
[12] Aron Culotta and Andrew McCallum. Reducing labeling effort for structured prediction tasks. In Conference on Artificial Intelligence (AAAI), 2005. Cited on page 9.
[13] Ido Dagan and Sean P. Engelson. Committee-based sampling for training probabilistic classifiers. In International Conference on Machine Learning (ICML), 1995. Cited on page 9.
[14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; pre-training of deep bidirectional transformers for language understanding. In North American Association for Computational Linguistics (NAACL), 2019. Cited on pages 5 and 25.
[15) Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. In Conference Neural Information Processing Systems (NeurIPS), 2020. Cited on page 23.
[16) Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference Machine Learning (ICMI., 2016. Cited on page 9.
[17] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imaaeeet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICLR), 2019. Cited on page 3.
12

---

[18] Robert Geirhos, Jern-Herrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2:665-673 2020. Cited on pages and 3.
[19] Karan Goel, Albert Gu, Yixuan Li, and Christopher Ré. Model patching: Closing the sub- group performance gap with data augmentation. In International Conference on Learning Representations (ICLR), 2021. Cited on page 3.
[20] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In North American Association for Computational Linguistics (NAACL), 2018. Cited on page 3.
[21] Steve Hanneke. Theory of Disagreement-Basee Based Active Learning. Foundations and Trends in Machine Learning, 2014. Cited on page 3.
[22] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Couraaeea,, Eric Wieser, Julian Taylor, Sebastian Berg, Nathanicl J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fer- nández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchane Kevin Sheppard, Tyler Reddy, Warren Weekesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, (85(())357-332) 2020. Cited page 25.
[23] Tatsunori B. Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang, Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning (ICML), 2018. Cited on page 3.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. Cited on pages 5 and 25.
[25] Dirk Hovy and Anders Søgaard. Tagging performance correlates with author age. In Association for Computational Linguistics (ACL), 2015. Cited on page 3.
[26] John D. Hunter. Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 933)900-95, 2007. Cited on page 25.
[27] Badr Youbi Idrissi, Martín Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive Reasoning (CLeaR), 2022. Cited on pages 2, 3, 4, 5, 7, and 25.
In Conference on Causal Learning and
(28) Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew Gordon Wilson. On feature learn ing in the presence of spurious correlations. In Conference on Neural Information Processing Systems (NeurIPS), 2022. Cited on pages 2, 3, 4, 5, 6, 8, 18, 25, and 28.
[29] Stanistaw Jastrzębski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks. In International Conference on Learning Representations (ICLR),
2020. Cited on page 23.
[30] Yiding Jiang, Vaishnavh Nagarajan, Christina Back, and J. Zico Kolter. Assessing generalization of SGD via disagreement. In International Conference on Learning Representations (ICLR),
2022. Cited on page 4.
[31] Eungyeup Kim, Jihyeon Lee, and Jaegul Choo. BiaSwap: removing dataset bias with bias- tailored swapping augmentation. In International Conference on Computer Vision (ICCV),
2021. Cited on page 3.
[32] Nayeong Kim, Sehyun Hwang, Sungsoo Ahn, Jaesik Park, and Suha Kwak. Learning debiased classifier with biased committee. Conference on Neural Information Processing Systems (NeurlPS), 2022. Cited on p.ge
[33] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. In International Conference on Learning Representations (ICLR), 2023. Cited on pages 1, 2, 3, 4, 5, 6. 8, 18, 25, and 28.
13

---

[34] Frederic Koehler, Lijia Zhou, Danica J. Sutherland, and Nathan Srebro. Uniform convergence of interpolators: Gaussian width, norm bounds and benign overfitting. In Conference on Neural Information Processing Systems (NeurIPS), 2021. Cited on page 23.
[35] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021. Cited on pages and 5.
[36] Tyler LaBonte. Milkshake: Quick and extendable experimentation with classification models. http://github www.coo//oo/oo mwmlaaoote/mikkshaaett 2023. Cited on page 25.
[37] Tyler LaBonte, Vidya Muthukumar, and Abhishek Kumar. Dropout disagreement: A recipe for group robustness with fewer annotations. In Conference on Neural Information Processing Systems (NeurlPS) Workshop on Distribution Shifts, 2022. Cited on page 2.
[38] Tyler LaBonte, Vidya Muthukumar, and Abhishek Kumar. Saving a split for last-layer retraining can improve group robustness without group annotations. In International Conference on Machine Learning (ICML) Workshop on Spurious Correlations, Invariance, and Stability, 2023.
Cited on page 2.
[39] John Langford, Lihong Li, and Tong Zhang. Sparse online learning via truncated gradient. Journal of Machine Learning Research (JMLR), 10(1):777-801, 2009. Cited on page 5.
[40] Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from underspecified data. In International Conference on Learning Representations (ICLR), 2023. Cited on pages 4 and 28.
[41] Evan Zheran Liu, Behzad Haghgoo, Annie S. Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning (ICMI., 2021. Cited on pages 2, 3, 5, 9, 10, 21, and 28.
[42] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision (ICCV), 2015. Cited on pages 3, and 5.
[43] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Conference on Learning Representations (ICLR), 2019. Cited on pages 5 and 25.
[44] ViscoV maintainers and contributors. Torch Vision: PyTorch's computer vision library. GitHub, 2016. Cited on page 25.
[45] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In Conference on Learning Theory (COLT), 2009. Cited on page 9.
[46] Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Association for Computational Linguistics (ACL), 2019. Cited on page 3.
[47] Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: Training debiased classifier from biased classifier. In Conference on Neural Information Processing Systems (NeurIPS), 2020. Cited on page 3.
[48] Junhyun Nam, Jachyung Kim, Jacho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. In International Conference on Learning Representations (ICLR), 2022. Cited on pages 3, 5, and 28.
[49] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv: 141..6412..644 2014, Cited on page 23.
[50] Timothy Niven and Hung- Yu Kao. Probing neural network comprehension of natural language arguments. In Association for Computational Linguistics (ACL), 2019. Cited on page 3.
14

---

[51] Luke Oakden-Raye Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden strat- ification causes clinically meaningful failures in machine learning for medical imaging. In Conference on Neural Information Processing Systems (NeurIPS) Workshop on Machine Learn- ing for Health, 2019. Cited on page
[52] Matteo Pagliardini. Martin Jaggi, François Fleuret, and Sai Praneeth Karimireddy. Agree to disagree: Diversity through disagreement for better transferability. In International Conference on Learning Representations (ICLR), 2023. Cited on pages 4 and 28.
[53] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary VeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In Conference on Neural Information Processing Systems (NeurIPS) Workshop on Automatic Difserentiation, 2017. Cited page 25.
[54] Mohammad Pezeshki, Sékou- Oumoar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. In Conference on Neural Information Processing Systems (NeurIPS), 2021. Cited on page 3.
[55] Shikai Qiu, Andres Potapczynski, Pavel Izmailov, and Andrew Gordon Wilson. Simple and fast group robustness by automatic feature reweighting. In International Conference Machine Learning (ICML), 2023. Cited on page 3.
[56] Amir Rosenfeld, Richard Zemel, and John K. Tsotsos. The elephant in the room. arXiv preprint 1808. 003800 2018. Cited on page 3.
[57] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision
(IJCV), 115(1):211 252, 2015. Cited on pages 5 and 25.
[58] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations (ICLR), 2020. Cited on pages 1, 2, 3, 4, 5, 25, and 28.
[59] Tobias Scheffer, Christian Decomain, and Stefan Wrobel. Active hidden markov models for information extraction. In International Conference Advances in Intelligent Data (CAIDA), 2001. Cited on page 9.
[60] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Pranceth Netrapalli. The pitfalls simplicity bias in neural networks. In Conference on Neural Information Processing Systems (NeurIPS), 2020. Cited on page 1.
[61] Rakshith Shetty, Bernt Schiele, and Mario Fritz. Not using the car to see the sidewalk: Quanti- fying and controlling the effects of context in classification and segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. Cited on page 3.
[62] Sahil Singla and Soheil Feizi. Salient ImageNet: how to discover spurious features in deep learning? In International Conference on Learning Representations (ICLR), 2022. Cited on page 3.
[63] Nimit S. Sohoni, Jared A. Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. In Conference on Neural Information Processing Systems (NeurIPS), 2020. Cited on page 3.
[64] Nimit S. Sohoni, Maziar Sanjabi, Nicolas Ballas, Aditya Grover, Shaoliang Nie, Hamed Firooz, and Christopher Ré. BARACK: partially supervised group robustness with guarantees. In International Conference on Machine Learning (ICML) Workshop on Spurious Correlations, Invariance, and Stability, 2022. Cited page 3.
[65] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-195) 2014. Cited on page 9.
15

---

[66] Saeid Asgari Taghanaki, Aliasghar Khani, Fereshte Khani, Ali Gholami, Linh Trana, Ali Mahdavi-Amiri, and Ghassan Hamarnh. MaskTune: mitigating spurious correlations by forcing to explore. In Conference on Neural Information Processing Systems (NeurIPS), 2022. Cited on pages 3 and
[67] Rachael Tatman. Gender and dialect bias YouTube's automatic captions. Association for Compulational Linguistics (ACL) Workshop Ethics in Natural Language Processing, 2017. Cited on page 3.
[68] Vladimir Vapnik. Statistical Learning Theory. Wiley, 1998. Cited on page 1.
[69] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD birds-200-2011 dataset. Technical report, California Institute of Technology, 2011 1. Cited on page 5.
[70] Jiaheng Wei, Harikrishna Narasimhan, Ehsan Amid, Wen-Sheng Chu, Yang Liu, and Ab- hishek Kumar. Distributionally robust post-hoc classifiers under prior shifts. In The Eleventh Interaational Conference on Learning Representations, 2022. Cited on page 3.
[71] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-UCSD birds 200. Technical report, California Institute of Technology, 2010. Cited on page 5.
[72] William Falcon and the PyTorch Lightning maintainers and contributors. PyTorch Lightning. GitHub, 2019. Cited page 25.
[73] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In North American Association for Computational Linguistics (NAACL), 2018. Cited on page 5.
[74] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the- art natural language processing. In Conference on Empirical Methods in Natural Language Processing (EMNLP) System Demonstrations, 2020. Cited on page 25.
[75] Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. In International Conference on Learning Representations (ICLR), 2021. Cited on page 3.
[76] Yilun Xu, Hao He, Tianxiao Shen, and Tommi Jaakkola. Controlling directions orthogonal to a classifier. In International Conference on Learning Representations (ICLR), 2022. Cited on page 3.
[77] Yadollah Yaghoobzadeh, Soroush Mehri, Remi Tachet, T.J. Hazen, and Alessandro Sordoni. Increasing robustness spurious correlations using forgettable examples. In European Associa- tion for Computational Linguistics (EACL), 2021. Cited on page 3.
[78] Yao-Yuan Yang, Chi-Ning Chou, and Kamalika Chaudhuri. Understanding rare spurious correlations in neural networks. arXiv preprint 129, 2022. Cited on page 3.
[79] John R. Zech, Marcus A. Badgeley, Manway Liu, Anthony Costa, Joseph J. Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: cross-sectional study. PLoS Medicine, 15:e1002683, 2018 Cited on pages and 3.
[80] Jianyu Zhang, David Lopez-Paz, and Léon Bottou. Rich feature construction for the ieaiiization-generlliaaaa
dilemma. In International Conference on Machine Learning (ICML), 2022. Cited on page 3.
[81] Michael Zhang, Nimit S. Sohoni, Hongyang R. Zhang, Chelsea Finn, and Christopher Ré. Correct-n-contrast: contrastive approach for improving robustness to spurious correlations. In International Conference on Machine Learning (ICML), 2022. Cited on pages and 5.
16

---

[82] Lijia Zhou, Danica J. Sutherland, and Nathan Srebro. On uniform convergence and low-norm interpolation learning. In Conference on Neural Information Processing Systems (NeurIPS), 2020. Cited on page 23.
[83] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In International Conference Computer Vision (ICCV),
2015. Cited on pages and 25.
17

---

Appendix Outline
The appendix organized as follows. Appendix A we include additional experiments for Sections 3 and 4. In Appendix B we include additional experiments for Section 5. In Appendix C we include the statement and proof of our theoretical proof-of-coneep (Theorem 1). In Appendix we describe in detail our training procedure and hyperparameters. Appendix E include tables to accompany the figures in the main paper. Finally, Appendix F discusses broader impacts, limitations, and compute.
A Additional experiments for Sections 3 and 4
In this section we include the detailed results of additional experiments for Sections 3 and 4.
Table 5: Empirical risk minimization. First, we compare class-unbalanced (CU) and class-balanced (CB) ERM on the training dataset vs. the combined training and held-out datasets (i.e., the training dataset plus half the validation dataset). We list the mean and standard deviation over three indepen- dent runs. Note that the Waterbirds validation dataset has different distribution than the training dataset (and, in particular, is group-balance)), and that MultiNLI is class-balancee a priori.
Held-out dataset included
Worst-group test accuracy
Method
Waterbirds
CelebA
CivilComments
MultiNLI
CU ERM
x
72.4
44.1
63.8
67.412.4
CB ERM
x
72.643.2
66.3
60.2
67.442.
CU ERM
V
81.6
44.5g3.4
59.1
69.121.3
CB ERM
/
81.943.4
67. 67.2#s.6
61.4+0.7
69. 69.2+1.
Table Balancing methodology comparison. Next, we compare last-layer retraining on the held- out set with different balancing methodologies, each initialized with class-balanced ERM features. In "sampling", we sample from the held-out dataset non-uniform rate so that each minibatch is class- or group-bal balunbd in expectation, while in "subset", we train on random class- or group-bur balp-ccalcddd subset the held-out dataset. Specifically, for group-balanced sampling we first sample then sample x ~ p(-(s) where P is the training distribution; class-balance y instead of S. For subset balancing, we keep all data from the smallest group/class and downsample the others uniformly at random to that size. We list the mean and standard deviation over three independent runs.
Unif(S), sampling is the same with
Worst-group test accuracy
Last-layer retraining method
Waterbirds
CelebA
CivilComments
MulGiLI
Class-unbalanced
88.0
41.941.4
57.644.2
64.641.0
Class-balanced sampling Classebalanced subset
92.6+0.8
73. 73.7+28
80. 40.4
64.7+1.1
92.1
74.642.0
80.2+1.4
64.5
Group-balanced sampling Group-balanced subset
92.4
87.0+11
81.8
70.8
91.621.9
88. 18.1
79.2
68.3
DFR [33, 28]
91.1
89.4
78.8
72.6
18

---

Table 7: Necessity the held-out dataset. Next, we investigate whether holding out a subset the training dataset for class-balanced (CB) last-layer retraining is essential, or if retraining on the entire (previously seen) dataset is also effective. For retraining on the training dataset, we use the same class-balanced last-layer retraining procedure as the held-out dataset, but we train for 20 epochs for the vision tasks and 2 epochs for the language tasks. For retraining on the held-out dataset, we report the best over four splits (i.e., the same numbers as Figure 2). Our results suggest that holding out data necessary to achieve maximal worst-group accuracy, though last-layer retraining on the training dataset interestingly prevents the performance decrease MultiNLL. We list the mean and standard deviation over three independent runs.
Worst-group test accuracy
Method
Waterbirds
CelebA
CivilComments
MultiNLI
CB ERM
72.613.2
66.313.2
60.2 20.2
67.442.4
CB last-layer retraining on training dataset
71.0
66,941.4
61.920.8
67.0+1.5
CB last-layer retraining held-out dataset
77.4.0.3
73.0+2.3
77.9
63.0
Table 8: Average accuracy performance. We detail the average test accuracy of our methods the 4 benchmark datasets. Both of our methods have similar average accuracy to DFR, which experiences slight accuracy/robusttess tradeoff compared to ERM (as is typical in the robustness literature). We list the mean and standard deviation over three independent seeds.
Average test accuracy
Method
Group Anns
Waterbirds
CelebA CivilComments
MultiNII
HRM
X
90.2
94.4
92.0
81.810.2
CB last-layer retraining
X
94.8
93.6
87.1 10.0
82.0.o.2
ES disagreement SELF
X
94.0+1.7
91.7+0.4
87.7+0.6
81 81.2+0.7
DFR (our impl.)
94.940.3
92.640.5
87.5
81.740.2
Retraining with UB ERM features Retraining with CB ERM features
Worst-grod poost-rruppppoo last aocuracy
85
80
65
60
Waterbirds
CelabA
CiliComments
MURINLI
Figure 4: Class-balanced last-layer retraining renders class-balanced ERM optional. We com- pare balanced (CB) last-layer retraining on the held-out dataset initialized with unbalanced (UB) or CB ERM features. Our results show that CB ERM unnecessary as long as the last layer retrained with class balancing. We plot the mean and standard deviation over three independent runs.
19

---

B Additional experiments for Section 5
In this section we include the detailed results of additional experiments for Section 5.
80
Wirkk sour
76
layor ning
DFR
Macisootication
miedwoed oodion
86
75
diaag semmnt
Random
000
003
Rewsightng datasol
Reweighting
Waterbirds
(b) CelebA
770
Wers group
70
Roweighing
Reweightng
(c) CivilComments
(d) MultiNLI
Figure 5: SELF reweighting dataset size ablation. We compare the class-balanced (ERM) baseline CB last-layer retraining, deep feature reweighting, and the four variants of SELF which describe in Section 5. The æ axis the size of the reweighting dataset, i.e., the number of points selected from the held-out dataset, and the axis is the worst-group accuracy. The held-out dataset has fixed size of 600 for Waterbirds, 9934 for CelebA, 22590 for CivilComments, and 41231 for MultiNLL. A general trend is that SELF methods tend to improve the size n of the reweighting dataset increases, except on MultiNLI We plot the mean over three independent runs and leave out error bars for readability.
Table 9; Total variation distance comparison. We compare the usage of KL divergence in our early-stop disagreement SELF method to total variation distance (TVD) as used in our Theorem 1. Our method is robust to the choice of distance function (though TVD is worse CelebA) and therefore our usage of TVD in Theorem is empirically justified. We list the mean and standard deviation over three independent seeds.
Worst-group test accuracy
Method
Waterbirds
CelebA
CivilComments
MultiNLI
KL divergence
93.0+0.3
83.940.9 72.9+3.4
79.1 78.6
70.742.5
Total variation distance
92.2+0.6
66.342.3
20

---

100
G Noturaty (80
80
Actunacy
Andbidtard
Inddislooug
85
non-Serrlnnele
walggboeffand
Wilorbi waterr
80
bionsfemale
70
40
60
100
15
Epoch
Epoch
(a) Waterbirds
(b) CelebA
ti maggidnn
05
90
wntrlmenino regation
wnbelmeninngatton
Group Acuss (10
86
reutraliio
50
neutralidertyy
80
taxomoo denity
78
78
Epoch
Epoch
(c) CivilComments
MultiNLI
Figure 6: Group validation accuracies over training. We plot the validation accuracies by group for the four benchmark datasets over the course of ERM training. Contrary to the assumptions made by JTT [41] and other carly-stop misclassification methods, the worst-group accuracy decreases with training on CivilComments. This highlights potential advantage of disagreement methods over misclassification methods: disagreement works regardless of whether the carly-stopped convergent model has a greater dependence on the spurious feature, as long as there is large relative difference. We plot the mean three independent and leave out error bars for readability.
Table 10: SELF training accuracies. We detail the training accuracies on the SELF held-out dataset using the best configuration of each method with respect to worst- group validation accuracy. We find that the misclassification techniques have much lower training accuracy than disagreement or random as low as 0% on MultiNLII meaning that the misclassified points cannot be fil without changing the features. We list the mean and standard deviation over three independent seeds.
Held-out training accuracy
Method
Waterbirds
CelebA
CivilComments
MultiNLI
Random SELF
96.3+00
90.619.2
91.147.88
86.7
Misclassification SELF
94.6
25.3
0.4
0.2
ES Misclassification SELF
91.3
28.5
1.1
0.0
Dropout Disagreement SELF
95.5
98.7
50.916.3
68.3
ES Disagreement SELF
97.0+2.0
79.9
58.143.3
52.612.5
21

---

Woriigmaa beed ACTUUTNCT
80
Wontegrup 2
mpl.
Ciste-balonoed ERM
Class balanoed ERM
10
50 100
10
20
50
100
% of group annotstions incuded
of group armotations Induddd
(a) Waterbirds
(b) CelebA
Woriingas bast acturity
70
88
82
EB disegnemmett SELF
ES designmmmnt BELF
(our impl)
DFR,
group 20
10
60
100
10
50
100
group
(c) CivilComments
MultiNLI
Figure 7: Label efficiency comparison. We compare the performance of early-stop disagreement SELF to DFR while varying the amount of group annotations used for validation and retraining, respectively. The results show that SELF is robust to hyperparameter tuning and massively reduce the annotation requirement: e.g., at 1% of data, Waterbirds has only examples and CelebA has only 99 examples. Moreover, SELF retains more performance than DFR when the number of group annotations is very low. We plot the mean and standard deviation over three independent seeds.
22

---

C Theoretical proof-of-concept for disagreement SELF
In this section, we describe and state our main theoretical proof-of-concept of disagreement SELF (Theorem 1). We begin with description of our setup and assumptions.
Setup. Let X' S IR" be the data domain, be the sample size, and d 2 v, n be the number of features. For x E X' and weights a, B > 0 we assumee that the learned predictor of both the ERM and regularized models takes the form
d-2
f(x) = ao (x) + Bdsp(() (x) +
Σ k=
de fann((r) (I)
and assume that the label y is generated by = sgn(()ooc(x:)) (iee,, zero label noise). Note that this specific split of features into core, spurious, and junk categories is an extension of common simplified setting in the literature on overparameterized linear models [82, 34], where only core and junk features are considered and only in-distribution accuracy is analyzed. We also consider the domain X' to be partitioned into minority group Xmin and majority group Xuaj such that y0.p(ar) < 0 for all x E Xii and sp (z) > for all E Xmaj- (By definition, yoocure (x) for any 2.)
Assumptions on the feature learning process. We make some extra simplifying assumptions on the feature learning process that are motivated by empirical observations in the literature. First, we assume that the learned features Ocure Osp, Pjunk are identical for the regularized and ERM models, and are only weighted differently in the last layer. This assumption is well-justified for dropout disagreement SELF since we only use dropout for inference in the last layer (with frozen features), and for early-stop disagreement SELF since feature learning has empirically been observed to occur during the initial phase of training [29, 15]. More concretely, we consider the ERM model fam to have weights Germ and 3am. and the regularized model fryg to have weights Oemn and Breg. Second, we assume that weights on the core and spurious features normalized such that Oermm + Berm Oreg + Breg. essentially positing that the proportion of junk feature strength is the same in both ERM and regularized models. This is also justified by the empirical observation that regularization methods such as early stopping make minimal difference to in-distribution accuracy [49].
Main result. The following result compares a minority group test example and majority group test example with features that are equal in magnitude, and shows that the disagreement is higher for minority group points regardless of whether the ERM or regularized model weights the spurious feature higher. We will use the total variation distance (TVD) as our cost function for disagreement SELF. We focus on TVD instead of KL divergence to simplify the proof; we show in Table 9 (detailed in Appendix B) that the empirical results are competitive using TVD (though worse on CelebA). We show that the difference of the TVD terms is proportional 18erm Breg := |aerm areg illustrating that the models disagree more drastically the greater the discrepancy between their dependence on the spurious feature (or, equivalently, the core feature).
Theorem 1 (Disagreement SELF). Consider two examples Tmin E Xnin and Timaj E Xmaj which have the same labels and whose features have the same magnitude, i.e., Deore (Fmin) = ( (Fmaj), ( ) mma)) and Dd=2 Ojunk (Fiin) Ed=2 Giink (Fmuj)- )- Let Pmin and Qiin be distribu- tions over (-1,1} } which lake value I with probubility (b(ferm(aiin + 11/ and (b(SSreg (egg((imm))) ) - respectively, and likewise for maj and Quai" Then, we have
TVD( Qmin) - TVD(P Qmai) = b min eooe (Imai)) ddp (2mpp)mma)) (3mam))) Bregl > 0. (3)
These are the estimated distributions use classify the inputs.
23

---

Proof. For any I E X' define P(x) (b(fam(z)) + 1)/2 and Q(z) = (b(fre((a)) + +1/2.. By the
assumptions on the features,
TVD()()))((())
P())) Q(x))
(4)
2 femm(1) (2) - Sreg (x))
(5)
2 (Keem (rrog) ) Ocore (I) - (Bemm - Srog ( (z)
(6)
Bern Bregl deooe (x) - Qsp(2))-
(7)
Let c = Mocor (Fmug)) ) and d = 1pp (Tmug) II By definition of the minority and majority groups, Ichcore (Imin) Qup (Tmmni) - Moone (Imaj) - tup (Immj)) ) = c + d lc - d| min(e, d) >
Together with b O, Equations and show the result.
24

---

D Training details
We utilize a ResNet-50 [24] pretrained on ImageNet- [57] for the vision tasks and BERT [14] model pretrained on Book Corpus [83] and English Wikipedia for the language tasks. These pretrained models are used as the initialization for ERM on the four datasets we study. We use standard ImageNet normalization with standard flip and crop data augmentation for the vision tasks and BERT tokenization for the language tasks [28]. Our implementation uses the following packages: NumPy [22]. PyTorch [53], Lightning [72], TorchVision [44], Matplotlib [26], Trans ormers [74), and Mi Wilkhhake [36].
For ERM and last-layer retraining (Section 4), we do not vary any hyperparameters: their fixed values listed in Table 1.. Recall that our SELF method (Section 5) splits the validation dataset in half, using the first half as a held-out dataset for selecting reweighting points and the second half for model selection [33, 28). We vary the reweighting dataset size n (the number of points selected from the held-out dataset) in the range (20, 100, 500). For the vision datasets, we search over learning rates (10 10 10-2) and train for 250 steps. For the language datasets, we search over learning rates 10-5, 10 10-4) and train for 500 steps. For the two early-stopping methods, we search over checkpoints saved 10%, 20%, and 50% of model training, while for dropout disagreement, we search over dropout probabilities of 0.5, 0.7, and 0.9 applied only at the last layer. See Appendix B for additional ablation studies.
Table 11: ERM and last-layer retraining hyperparameters. We standard hyperparameters following previous work [58, 27, 33, 28). For last-layer retraining, we keep all hyperparameters the same except the number of epochs CelebA, which increase to 100.
Dataset
Optimizer
Initial LR
L.R schedule
Batch size
Weight decay
Epochs
Waterhirds
SGD
3 x 10
Cosine
32
1 x 10
100
CelebA
SGD
3 10 10-3
Cosine
100
1 x 10
20
CivilComments
AdamW [43]
1 10
Linear
1 x 10
MultiNLI
AdamW [43]
1 10-5
Linear
16
1 1x 10
10
25

---

E
Additional tables
In this section we provide results in tabular form accompany figures in the main paper.
Table 12:
Table for Figure la. We perform an ablation on the percentage of worst-grow group data used for class-balanced last-layer retraining, while keeping the total data constant. Note that the 100% numbers may not equal our other group-halanced last-layer retraining numbers due to our methodology involving downsampling and starting from class-unbalanced ERM. We list the mean and standard deviation over three independent runs.
Percent of worst-group data included
Worst-group test accuracy
Waterbirds
CelebA
CivilComments
MultiNLI
2.5
53.5
61.1
75.3
67 1..
5.0
53.5+6.0
62.8+3.
75.4as.u
67.4+1.9
12.5
63.116.3
72.012.8
75.5
67.6 67.6
25.0
73.6
78.2+1.7
75.544.9
67.8
37.5
80.2
81.514.2
75.614..
67.8
50.0
83.2
83.712.8
75.845.0
67.912.0
62.5
86.2+1.3
85.743.1
75,8
67.9+2.0
75.0
86.5
87.042.5
46.1
68.0+1.9
87.5
88.8
88.2+1.3
76.0
68.3
100.0
90.4
88.7
76.1
68.3
Table 13: Table for Figure 1b. We compare the worst-group accuracy in Table 12 the maximum WGA increase over a class-unbalanced ERM (see Table 5). Note that Waterbirds performs poorly at low percentages because the small size of the held-out dataset leads included as a result of our downsampling methodology. We list the mean over three independent runs.
worst-group data being
Percent of worst-group
Percent of max WGA increase over ERM
data included
Waterbirds
CelebA
CivilComments
MultiNLI
2.5
-005.0
38.1
93.5
- -1.1
5.0
105.0
41.9
94.3
0.0
12.5
51.7
62.6
95.1
22.2
25.0
6.7
76.5
95.1
44.4
37.5
43.3
83.9
95.9
44.4
50.0
60.0
88.8
97.6
55.6
62.5
76.7
93.3
97.6
55.6
75.0
78.3
96.2
100.0
66.7
87.5
91.1
98.9
99.2
100.0
100.0
100.0
100.0
100.0
100.0
Table 14: Table for Figure 2. We compare class-balanced (CB) ERM on the entire dataset to splitting the dataset and performing CB ERM on the first (95%) split and CB last-layer retraining on the second (5%) split. The results with and without the held-out dataset correspond Figure 2a and 2b respectively. We list the mean and standard deviation over three independent runs.
Held-out dataset included
Worst-group test accuracy
Method
Waterbirds
CelebA
CivilComments
MultiNLI
CB ERM
X
72.623.2
66.343.2
60. 60.2+2.7
67. 67.4+2.
CB last-layer retraining
X
72.4
74.142.
80.7
64.4+1.1
CB ERM
V
81.923.4
67. 67.245.6
61.440.7
69.2+1.6
CB last-layer retraining
V
83.1 12.7
71.343.2
80.8
65.642.4
26

---

Table 15: Table for Figure 3. We list the percentage the reweighting dataset consisting worst- group data for each SELF method. "Baseline" represents the percentage of worst-group data in the held-out dataset, while "Balanced" is the percentage of worst-group data necessary to achieve group balance (recall that there may be multiple worst groups; they described in Section 4.1). We also include the worst-group accuracy (WGA) of each SELF method, to get sense of whether the amount of worst-group data correlates with performance (in loose qualitative sense). We list the mean over three independent runs and round to the nearest whole number.
WGA/Percent gorst-group data
Method
Waterbirds
CelebA
CivilComments
MultiNII
Baseline
50
7
2
Misclassification
90/45
85/25
64/51
65/
4
ES misclassification
91 45
83/
68/37
68
20
Dropout disagreement
91/55
82/13
79/13
66/
ES disagrecment
93/67
84/ 2
79/24
71/ 4
Balanced
50
25
25
33
Table 16: Table for Figure We compare different combinations of ERM and last-layer retraining with without class balancing. Notably, worst-group accuracy whether the ERM incorporates maas-bbllnncing or not. In Figure 4, we only plot the results which class- balanced last layer retraining (i.e., the last two of the table). We use the entire held-out set for last-layer retraining, and we list the mean and standard deviation over three independent runs.
last-layer retraining enables nearly the same
ERM class balancing
Last-layer retraining class balancing
Worst-group test accuracy
Waterbirds
CelebA
CivilComments
MultiNII
87.5 87.5
45.442.3
54.72a.
64.5
88.0 88.0+0.8
41.9414
57.6.4.2
64.611.0
92.3
71.142.4
78.4
64.7
92.640.8
73.742.8
80.4.0.8
64.7
27

---

Broader impacts, limitations, and compute
Broader impacts. We hope our work contributes to the safe and equitable application of machine learning and motivates further research in ML fairness. A potential negative outcome may arise if practitioners assume their models are bias-free after applying our techniques; while we show reduced dependence on spurious correlations, no method can completely alleviate the issue, and other modes of bias may exist. We encourage practitioners to conduct rigorous examinations of model fairness.
Limitations. Our methods take advantage of the specificity of the spurious correlation setting, and therefore would likely underperform datasets which exhibit more extreme complete correlation (i.e., contain zero minority group data) [52, 40J. Furthermore, following previous work in this setting [58, 41, 48. 33, 28], SELF utilizes a small validation dataset with group annotations for model selection. While we show in Appendix B that SELF is robust to using as few as 1% of these annotations, and our last-layer retraining method (Section 4) does not require them, completely removing this assumption is an important direction for future research.
Compute. Our experiments were conducted on Nvidia Tesla V100 and A5000 GPUs. We used about $25000 in compute credits in the course of this research. We believe this paper could be reproduced for under $5000 in credits, and majority of this compute would go towards the ablation studies. With that said, our last-layer retraining methods only train the linear classifier, making them cheap and efficient run even on older hardware.
28


Reviews:
[{"review_id": "5VhUT8A7yp", "reviewer": "Reviewer_KCny", "summary": "This work tackles an important problem of preventing the reliance of neural networks on spurious correlations. It builds on top of the work [1] primarily by using last layer re-training on class balanced held out dataset without the need for group annotations. They also additionally propose a simple but effective method SELF wherein the samples in disagreement with regularized models in the prediction with class annotations can be used for re-training the last layer. This method is particularly useful when group imbalance is large. \nResults are shown on Waterbirds, CelebA, MultiNLI and CivilComments datasets and performance is comparable to DFR [1].", "questions": "Covered in weaknesses.", "limitations": "A lot of the questions posed by the paper are still not addressed or analyzed . For e.g., It would be great to see more intuitive and theoretical reasoning on fundamental observations like: why last layer re-training and retraining on held out small class balanced data is improving the group robustness. Though experimental evidence has been provided, further analysis on the theoretical reasoning behind this would make a significant contribution in this subject.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper is well written and does very good analysis with ablation studies. \n\n(2) Makes an important observation that it is ok if re-weighting dataset has small proportion of worst group data. This proves that the class balancing is more important than the group balancing. \n\n(3) The paper addresses some of the practical challenges (in terms of computation and training overhead) by eliminating need to annotated group labels which is expensive, need to train twice, need to train the original ERM on class-balanced dataset. \n\n(4)The provided solutions are simple, and will not introduce any implementation or annotation overhead for already trained large scale models. \nThe paper addresses the limitation of accuracy gap when the group imbalance is high and proposes a solution for the same with SELF technique.", "weaknesses": "(1)The observation of class-balancing importance and re-evaluation of group balancing’s role in spurious correlation is a valuable contribution. However, solution in itself lacks novelty as it is not proposing a new algorithm per se. It seems very similar to [1] \"Last layer re-training is sufficient for robustness to spurious correlations\"  with lesser constraints in terms of re-training dataset. \n\n(2) This particular line under section 5  “In addition to the balance of the reweighting, dataset, it is likely that characteristics of the specific data selected also contribute to SELF results. “ is unclear.  Can you provide some qualitative examples from Celeb A where drop-out SELF disagreement does better than misclassification. More insights on what aspects of data makes the dropout SELF better would be good. Similarly, more qualitative example on the “uncertain data points” selected by the SELF-disagreement would provide more valuable insights. A lot of the observations are not supported by reasoning. \n\n(3) Since the key claim is the advantage of not needing group labels annotations - ablation studies on what is the overhead caused for obtaining group annotations (probably annotation time) will help quantify the effectiveness of the proposed method.\nSimilarly ablation studies on the training compute time / mem consumotion between [1] and your method can help quantify the gains if any in training ERM on 95% split and using a MLP instead of Logistic regression.\n\n\nReferences:\n[1] \"Last layer re-training is sufficient for robustness to spurious correlations\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We graciously thank Reviewer KCny for their insightful comments and thoughtful suggestions. Below, we provide responses to each of the reviewer’s points.\n\n(Weakness 1) We would like to clarify that we do not claim algorithmic novelty for the results in Section 4, only for our disagreement SELF algorithm in Section 5. We believe that our contributions in Section 4 instead fall in the category of foundational empirical insight; our experiments reveal previously unknown phenomena which are surprising in the context of previous literature and have important ramifications for future algorithm design. In particular, our Section 4 results call into question **why** DFR improves worst-group accuracy, as we show that perfect group balance is not needed to achieve good performance, and that last-layer retraining on the training distribution can be surprisingly effective. We hope these insights will motivate future theoretical work to understand the underlying foundations of spurious correlations, as well as motivate future empirical work that will not only push the boundaries of SOTA worst-group accuracy, but also provide simpler and more interpretable algorithms.\n\nWith that said, our disagreement SELF algorithm (Section 5) is entirely novel and differs from previous work [2, 3] in several important ways (besides not requiring group annotations). First, SELF finetunes the last layer instead of retraining as in [2]. Second, we show that tuning the $\\ell_1$ regularization as in [2] is unnecessary and good performance can be achieved by fixed $\\ell_2$ regularization, removing a hyperparameter. Third, we show that the best performance is achieved when using disagreements instead of misclassifications as in [3]. Fourth, while [3] assumes that the early-stopped model has low worst-group accuracy (causing misclassifications), SELF works even when the early-stopped model has higher worst-group accuracy than the convergent model (this is the case on CivilComments).\n\n(Weakness 2) We have included qualitative examples for misclassification SELF, dropout disagreement SELF, and early-stop disagreement SELF as Rebuttal Figure 2. With that said, we caution against drawing conclusions from a handful of qualitative examples. Therefore, we appeal to mathematical theory to justify why disagreement works well, and we recently proved a relevant theorem described in the (Limitations) section below. Likewise, we believe our observations on the distinction between misclassification and disagreement points are well-supported by previous study in the theory literature, including the references we cite on Lines 310-313.\n\n(Weakness 3) While our methods remove the need to annotate thousands of examples, we believe our most significant contribution is in enabling new capabilities (e.g., in settings with strict privacy or fairness constraints, disallowing or reducing the very presence of annotations) rather than accelerating existing annotation pipelines. For example, racial and gender identity are among the most important and socially relevant spurious features, but these attributes are sensitive and often cannot be collected in the first place (making annotation time irrelevant). One could imagine a scenario where a limited number of users may voluntarily self-identify, but too few to run DRO or even DFR; in that case, annotation time is negligible and the labeled samples may be used for model selection in our SELF algorithm. The reviewer may also be interested to see the results of an ablation on the validation set size, which we included as Rebuttal Figure 1.\n\nWe did not perform a detailed ablation study on the time/memory consumption of SGD vs logistic regression since the compute required is negligible on modern hardware regardless of implementation (roughly 2-4MiB of RAM and 10-20 minutes of training for one SELF instance on one Nvidia V100 GPU). With that said, the previous logistic regression version must pre-compute and save to disk the feature embeddings for the entire dataset, which can take up several GB and can be slow depending on disk I/O speed (e.g., if using an HDD rather than an SSD).\n\n(Limitations) We agree wholeheartedly with the reviewer that theoretical justification for the empirical phenomena identified in this paper is interesting and important. To this end, we have been considering multiple theoretical questions since the submission. (1) As the reviewer suggested, an intriguing question is why class-balanced last-layer retraining improves worst-group accuracy without group annotations. We have considered several possible avenues here, including whether last-layer retraining has a sparsity-inducing implicit regularization effect and whether it uses the held-out data to learn a non-degenerate classifier after the training data experiences neural collapse [1]. With that said, we think this question is interesting and difficult enough to be a separate paper of its own. (2) We are also interested in theoretical justification for the disagreement-based upsampling of minority groups phenomenon observed in Figure 3. On this front, we recently proved a result which shows that model disagreement provably upsamples minority group points. In particular, for an overparameterized linear regression setting with frozen features and four groups, we show that the KL divergence between the regularized and convergent models is always higher for minority group points than majority group points.\n\n[1] Papyan et al. “Prevalence of Neural Collapse during the terminal phase of deep learning training.” PNAS 117 (40) 24652-24663.\n\n[2] Kirichenko et al. “Last Layer Retraining is Sufficient for Robustness to Spurious Correlations.” ICLR 2023.\n\n[3] Liu et al. “Just Train Twice: Improving Group Robustness without Training Group Information.” ICML 2021."}, {"review_id": "oISRsb6tC2", "reviewer": "Reviewer_Ep6f", "summary": "The paper provides a detailed analysis of the DFR procedure [1]. The authors specifically consider the case when the group annotations are limited or not available during training, and show that it is still possible to largely remover the DFR performance. Specifically, the authors retrain the last layer on data with class balancing, and on data where examples are selected according to some rule (e.g. misclassified examples). The authors also provide many detailed ablations and understanding experiments that support their findings.\n", "questions": "**Q1. Early stopping:** What is the early stopping criterion for the models in Section 5?\n\n**Q2. Free lunch:** For the free-lunch results in Section 4.2 do you need regularization or early-stopping, optimized on validation WGA? Or is it quite robust to hyper-parameters?\n\n**Q3. Hyper-parameters:** Could you please list all hyper-parameters that a practitioner needs to tune for the SELF method?\n\nFinally, the paper [2] appears to be quite relevant, as it also attempts to automatically construct a reweighting dataset for DFR. However, [2] is a concurrent work, as it was published only after the submission of this paper. So there are no issues with the novelty of the paper.\n\n\n**References**\n\n[1] [_Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations_](https://openreview.net/forum?id=Zb6c8A-Fghk); P. Kirichenko, P. Izmailov, A. G. Wilson; ICLR 2023\n\n[2] [_Simple and Fast Group Robustness by Automatic Feature Reweighting_](https://arxiv.org/abs/2306.11074); S. Qiu, A. Potapczynski, P. Izmailov, A. G. Wilson; ICML 2023\n", "limitations": "Limitations are adequately discussed by the authors.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**S1.** The authors provide new interesting insights into the DFR procedure. For example, the results in Figure 1 on the number of minority data needed are quite surprising: even with a small increase in the proportion of minority data it is possible to significantly improve worst-group performance. Also the results in Table 3 on the effectiveness of last-layer retraining without group rebalancing are thought-provoking. Finally, the results in Figure 3, which show that better group balancing does not always lead to better WGA.\n\n**S2.** The authors propose new practical methods (SELF), which build on DFR, but relax the requirement for group-balanced data. The results are discussed in Section 5. The authors also perform detailed ablations on the design decisions for the method, such as what cost function $c$ to use for selecting a subset of the data for last layer retraining.\n\n**S3.** SELF can work even when the reweighting data doesn’t have class labels, and request the labels on a small subset.\n\nOverall, I believe this paper provides new insights that expand our understanding of DFR and more generally training models robust to spurious features. The proposed methodology is also promising.\n", "weaknesses": "I believe there is one important missing experiment, and several technical details that should be explained or corrected. I explain these in detail below.\n\n**W1.** Experiment: performance vs total number of group labeled examples\n\nAs every other group robustness method, SELF requires group-labeled data to perform hyper-parameter tuning. My understanding is that in all of the experiments, half of the validation data is used for hyper-parameter tuning. In other words, it is not true that the method can work without any group annotations (which is also true of JTT, CnC, and other methods). Moreover, SELF seems to involve more hyper-parameters than DFR (which only tunes regularization strength): length of last layer retraining, size of the reweighting dataset, learning rate, etc.\n\nConsequently, I believe it is important to evaluate performance as a function of the total number of the group-labeled datapoints used by the method. In case of DFR, these datapoints would be used for last-layer retraining, and in case of  SELF, they will only be used for hyper-parameter tuning. I think it may be reasonable to create a group-balanced validation set of size $n$, and plot WGA vs the size $n$. For example of a similar experiment, see Fig. 5 in [2].\n\nI believe, this experiment is quite important for evaluating the methodological contribution of the paper.\n\n**W2.** Civilcomments\n\nI blieve, the results presented in Table 1 for CivilComments mix two different versions of the dataset. Specifically, JTT and CnC use the [Wilds version](https://wilds.stanford.edu/datasets/#civilcomments) with 16 overlapping groups, while DFR, RWY-ES, Group-DRO and the methods described in the paper use a version with 4 groups, where all the spurious attributes are combined together. As a result, there is a big discrepancy in performance. I would recommend either (1) reporting the results for all methods on the WILDS version of the dataset, or (2) removing the JTT and CnC entries and clarifying the difference in the versions of the data.\n\n\n**W3.** Waterbirds\n\nOn waterbirds, the validation data is group balanced up to class balancing. As a result, by group-balancing, the authors are able to achieve DFR performance with class-balanced last-layer retraining (CB in Table 1). However, in this case the method is virtually identical to DFR.\n\nI think it would be more meaningful to do the experiment using a base reweighting dataset that has the same group distribution as the training data. This is the case for all other datasets.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We graciously thank Reviewer Ep6f for their in-depth analysis, insightful comments, and attention to detail. Below, we provide responses to each of the reviewer’s points.\n\n(W1) Thank you for this great suggestion. We have included the results of the requested ablation as Rebuttal Figure 1; the results show that ES disagreement SELF is robust to tuning hyperparameters with as little as 1% of the validation data (though variance increases as expected). Our SELF method improves greatly upon class-balanced ERM even with as few as 6 validation examples for Waterbirds and 99 examples for CelebA, massively reducing the group annotation requirement. The CivilComments and MultiNLI ablations will be done by next week and available upon request by the reviewers.\n\nWe would also like to clarify that length of last-layer retraining (i.e., the number of optimizer steps) is not a hyperparameter in our method, as the number of optimizer steps is kept constant as the reweighting dataset size changes. Please see (Q3) below for additional discussion on the hyperparameters that SELF tunes.\n\n(W2) Thank you for this good catch regarding the CivilComments dataset. We have removed the CivilComments JTT and CnC entries from Table 1 and detailed the difference between the two versions of the dataset in a footnote. We would like to clarify that every method in Table 1 uses the WILDS version [1] of the dataset, but the 4-group version we study collapses all the spurious attributes (“female”, “LGBT”, etc.) into one attribute. The original (non-WILDS) CivilComments dataset [2] is not used.\n\n(W3) We thank the reviewer for the suggestion. We actually included this experiment in the paper in Figure 2(a) with the corresponding table version in Appendix C Table 7, and we remark on the reviewer’s point on Lines 229-231. By performing class-balanced last-layer retraining on the training distribution, we improve WGA by 4.8% over class-balanced ERM, which is actually larger than the 3.3% increase gained by using the validation distribution.\n\n(Q1) The early-stop criterion for SELF is the percentage of total training completed. Specifically, we use models stopped at 10%, 20%, and 50% of total training (corresponding to, e.g., 2, 4, and 10 epochs for CelebA). This procedure is detailed on lines 291-292 of the paper.\n\n(Q2) The free-lunch results in Section 4.2 only tune one hyperparameter (split ratio) and do not utilize any early stopping or regularization, except for a weight decay value of 1e-4 (not tuned). As discussed on Line 242 and Appendix D, we found that a 95/5 split worked the best in general, and we therefore recommend practitioners to tune no hyperparameters at all. Please see the (Concurrent work) section below for additional discussion about the regularization and hyperparameters used in Section 4.2.\n\n(Q3) There are three SELF hyperparameters: (1) reweighting dataset size, (2) learning rate, and (3) early-stop threshold (early-stop SELF only) or dropout probability (dropout SELF only). The number of optimizer steps is kept constant as the reweighting dataset size changes. This procedure is detailed on lines 286-293 of the paper.\n\n(Concurrent work) We thank the reviewer for bringing up the reference [3], which we had also noticed after the initial submission of our work. We believe their results corroborate our findings in Section 4.2. A key difference between our methods is that we only tune one hyperparameter (split ratio) in Section 4, while [3] includes two hyperparameters ($\\gamma$ and $\\lambda$ in their Section 3.1) which tune mistake upweighting and $\\ell_2$ regularization towards the original last-layer weights. Our ablation in Appendix D shows that our Section 4 results are robust to split ratio and we recommend on Line 242 that practitioners **tune no hyperparameters at all**; therefore, our results are more practical because we do not require group annotations in the validation set. We remark that our Section 4.1 and Section 5 are entirely novel and no corresponding results appear in [3]. We have added a citation and discussion of [3] to Section 2.\n\n[1] Koh et al. “WILDS: A Benchmark of in-the-Wild Distribution Shifts.” ICML 2021.\n\n[2] Borkan et al. “Nuanced metrics for measuring unintended bias with real data for text classification.” WWW 2019.\n\n[3] Qiu et al. “Simple and Fast Group Robustness by Automatic Feature Reweighting.” ICML 2023."}, {"review_id": "CD8tcmn3qI", "reviewer": "Reviewer_x5jy", "summary": "This paper proposes to collect a dataset from misclassifications or disagreements to fine-tune a classifier for improving sub-group accuracy in presence of spurious correlations.\n", "questions": "Please refer to the Weaknesses section.", "limitations": "Please refer to the Weaknesses section.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The topic that this paper attempts to address is important. The paper is well-written as well. There are some interesting findings in the paper as well such as finetuning using a dataset collected from disagreements could potentially improve group-accuracy.", "weaknesses": "The level of novelty presented in the paper is insufficient. My primary concern stems from the method's reliance on group labels during model selection, also known as the validation set. This requirement renders these techniques nearly impractical in real-world scenarios, as defining groups is a non-trivial task even in datasets with just a hundred samples, and this becomes much more challenging for current datasets comprising billions of data points.\n\nThe Waterbirds dataset utilized in this study has been identified to contain certain bugs, which were subsequently addressed in the MaskTune paper [1]. Consequently, the current values provided may be misleading. It is worth noting that the MaskTune paper, contrary to the assumption, does not incorporate any group labels during training or model selection (validation). It would be valuable for the authors to present the worst group accuracies achieved by SELF without the utilization of any training or validation group labels. Additionally, I encourage the authors to compare these results with the methods outlined in Tables 1 and 2 of [1].\n\n[1] Asgari, Saeid, et al. \"Masktune: Mitigating spurious correlations by forcing to explore.\" Advances in Neural Information Processing Systems 35 (2022)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We warmly thank Reviewer x5jy for their comments and suggestions. Below, we provide responses to each of the reviewer’s points.\n\n(Setting) We thank the reviewer for the reference [1], and we have added a citation and discussion of [1] in Section 2. With that said, we explicitly focus on the setting where group annotations are available for model selection; this setting has been standard in the literature since JTT [2] and works in this area have contributed significant practical insights to the literature despite the annotation requirement [2, 4, 5, 6, 8]. Our main goal in this paper is **not** to beat the results in [1, 7], but rather to investigate the surprising performance of last-layer retraining in a more restrictive setting (our Section 4) and propose a SOTA algorithm for the setting of [2, 4, 5, 6, 8] based on a novel disagreement-based approach (our Section 5).\n\n(Model Selection) We have included the results of an ablation on the validation set size as Rebuttal Figure 1; the results show that ES disagreement SELF is robust to tuning hyperparameters with as little as 1% of the validation data (though variance increases as expected). Our SELF method improves greatly upon class-balanced ERM even with as few as 6 validation examples for Waterbirds and 99 examples for CelebA, massively reducing the group annotation requirement. The CivilComments and MultiNLI ablations will be done by next week and available upon request by the reviewers.\n\n(Section 4) We noticed the reviewer did not have any comments on our results in Section 4, which provided the separate insight that last-layer retraining can improve group robustness even if the reweighting dataset is drawn from the training distribution (i.e., is group-imbalanced). We would be very interested to hear your perspective on this set of results. Given the reviewer’s concern about model selection, we would like to emphasize that our results in Section 4 tune only the split ratio, which we show to be robust in Appendix D, and we recommend on Line 242 that practitioners **tune no hyperparameters at all** (therefore requiring no group annotations in the validation set). This is in contrast to concurrent work [5] whose results corroborate our Section 4.2, but they tune two required hyperparameters ($\\gamma$ and $\\lambda$ in their Section 3.1).\n\n(Waterbirds) Thank you for bringing the Waterbirds bugs to our attention: we have updated the paper to include a reference to this phenomenon. With that said, we believe testing on the original Waterbirds dataset is still important and useful for comparison to previous SOTA methods [2, 3, 4]. Furthermore, we test on 3 additional rigorous benchmark datasets across both vision and language domains and achieve SOTA on 2/3 of these among methods in the setting where group annotations are used for model selection but not for training.\n\n[1] Taghanaki et al. “MaskTune: Mitigating Spurious Correlations by Forcing to Explore.” NeurIPS 2022.\n\n[2] Liu et al. “Just Train Twice: Improving Group Robustness without Training Group Information.” ICML 2021.\n\n[3] Kirichenko et al. “Last Layer Retraining is Sufficient for Robustness to Spurious Correlations.” ICLR 2023.\n\n[4] Zhang et al. “Correct-N-Contrast: a Contrastive Approach for Improving Robustness to Spurious Correlations.” ICML 2022.\n\n[5] Qiu et al. “Simple and Fast Group Robustness by Automatic Feature Reweighting.” ICML 2023.\n\n[6] Sohoni et al. “BARACK: Partially Supervised Group Robustness With Guarantees”. ICML 2022 SCIS Workshop.\n\n[7] Lee et al. “Diversify and disambiguate: Learning from underspecified data.” ICLR 2023.\n\n[8] Nam et al. “Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimation.” ICLR 2022."}, {"review_id": "amd6zryY8N", "reviewer": "Reviewer_HP8n", "summary": "The paper tackles the aspect of last-layer retraining for better group robustness. The authors show that last-layer retraining can greatly improve worst-group accuracy with little worst-group data. Motivated by this, selective last-layer finetuning (SELF) is proposed to improve group robustness.", "questions": "Is worst-group accuracy a valid metric for evaluating robustness? A common phenomenon in long-tailed learning is that increasing tail-class accuracy often leads to decrease in head class accuracy. I recommend also adding overall accuracy as a metric for comparison. Also, WGA cannot reflect the uniformness of the overall performance.", "limitations": "The limitations are included in the appendix.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper focuses on the interesting aspect of improving robustness via last-layer retraining under spurious correlation. The problem is specific and practical.\n2. The experiments are well done. Also, the comparisons are consistent through out the paper.\n3. The findings and proposed methods are well presented. The finding of \"last-layer retraining can substantially improve WGA even when the reweighting dataset has only a small proportion of worst group data\" is interesting.\n", "weaknesses": "1. Novelty: \n\n(1) Last-layer retraining/finetuning is a common techqniue used in imbalanced/long-tailed learning and various properties were found, see [1][2]. This put doubt on the novelty of the claimed finding: \"holding out a subset of training data to retrain the last layer can substantially outperform ERM on the entire dataset with no additional data, annotations, or computation for training\". I wonder what are the differences if we treat tail classes in [1][2] as the worst group.\n\n(2) The proposed method: Disagreement-based methods were proposed and they were only adopted to the setting with spurious correlation. This adaptation is fine, but the performance in Table 4 cannot convince me since SELF cannot beat DFR in 2 out of 4 tasks.\n\n2. Writing: \n\n(1) Figure 1(a) misses legends. For Figure 1(b), please note that y-axis means relative increase to avoid confusion. \n\n(2) The contribution part is too long.\n\n[1] Decoupling Representation and Classifier for Long-tailed Recognition. ICLR 2020\n[2] BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition. CVPR 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We warmly thank Reviewer HP8n for their detailed comments, suggestions, and references. Below, we provide responses to each of the reviewer’s points.\n\n(Novelty 1) Thank you for the comments and the references [1, 2]; we have added citations and discussion to Section 2. We remark that the reviewer’s concern is discussed at length in the DFR paper [3] which introduced last-layer retraining to the spurious correlations setting. In their Appendix A, they argue that these methods are not directly applicable to spurious correlations robustness, and many of their points apply to our work. For example, a crucial difference is that [1] retrains on data from the training set, whereas we retrain on held-out data only. Nevertheless, we provide a worst-group accuracy comparison between the methods of [1] and our proposed methods on Waterbirds and CelebA below.\n\n| Method                   | Group Annotations | Retraining Data | Waterbirds WGA | CelebA WGA    |\n|--------------------------|-------------------|-----------------|----------------|---------------|\n| LWS [1]                  | No                | Train           | 40.0 +/- 8.1   | 35.6 +/- 14.4 |\n| cRT [1]                  | No                | Train           | 74.5 +/- 1.5   | 52.9 +/- 6.0  |\n| CB last-layer retraining (ours) | No                | Held-out        | 92.6 +/- 0.8   | 73.7 +/- 2.8  |\n| ES disagreement SELF (ours)    | No                | Held-out        | 93.0 +/- 0.3   | 83.9 +/- 0.9  |\n| DFR (our impl.)          | Yes               | Held-out        | 92.4 +/- 0.9   | 87.0 +/- 1.1  |\n\nRegarding the reviewer’s suggestion of treating tail classes as the worst group, we would like to clarify that the “worst group” is not a fixed subset of the data like tail classes are. Instead, the “worst group” is dynamic, and corresponds to the group attaining the minimum accuracy over all groups. Therefore, we cannot treat tail classes as the worst group, since we cannot explicitly control which group is the worst group. Furthermore, a “group” in our work is an element of the Cartesian product of the classes and the values of the spurious feature, so depending on the spurious feature, it may be the case that the tail classes are not all in the same group.\n\n(Novelty 2) We would like to clarify that the goal of our disagreement SELF method is **not** to beat DFR (i.e., achieve SOTA performance in the presence of group annotations). This is an unrealistic expectation, as we train with no group annotations on the reweighting dataset -- much less information than DFR, which requires group annotations for the entire reweighting dataset. Therefore, we consider DFR as an oracle method, and our goal is to get as close as possible to the oracle level. In this respect, the most important comparison to make is to other methods that also do not use group annotations, which we do in Table 1. The table shows that the performance of our methods are indeed SOTA on 3 out of 4 of the benchmarks among methods not using group annotations). Considering the exception (CelebA), we remark that the large gap to CnC on the CelebA dataset also exists in concurrent work [4], suggesting that it may be an inherent limitation of last-layer retraining and not a shortcoming of our method (since CnC modifies the features, while last-layer retraining does not). Our methods outperform all other methods with the exception of CnC on the CelebA dataset.\n\n(Writing 1) We thank the reviewer for the catch, and we have updated the figures according to the reviewer’s suggestions.\n\n(Writing 2) We thank the reviewer for the suggestion, and we have reduced the length of the contributions section by shortening the paragraphs.\n\n(Question) Thank you for the question and suggestion. We have included average accuracy numbers for ERM, class-balanced last-layer retraining, ES disagreement SELF, and DFR in Rebuttal Table 1. Our methods achieve comparable average accuracy to DFR, which is a favorable result as DFR is considered one of the SOTA methods for group robustness. We hope the average accuracy results assuage the reviewer’s concerns about how WGA alone does not reflect the uniformity of the group-accuracy distribution.\n\n[1] Kang et al. “Decoupling representation and classifier for long-tailed recognition.” ICLR 2020.\n\n[2] Zhou et al. “BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition.” CVPR 2020.\n\n[3] Kirichenko et al. “Last Layer Re-training is Sufficient for Robustness to Spurious Correlations.” ICLR 2023.\n\n[4] Qiu et al. “Simple and Fast Group Robustness by Automatic Feature Reweighting.” ICML 2023."}]

Author rebuttal:
Please see the attached PDF file for additional figures and tables.

Meta-review: This paper received diverging reviews. Reviewer x5jy expressed concern regarding novelty whereas the rest 3 reviewers acknowledge the contribution of this work. Upon reading through the paper and all the discussions, the AC feels that this work does contain enough novelty/contributions to be accepted despite not being groundbreaking, given the relaxed requirements on the annotations on held-out dataset. In light of this consideration, the AC recommends acceptance.
Decision: Accept (poster)