SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(d) MultiNLI
Figure 5: SELF reweighting dataset size ablation. We compare the class-balanced (ERM) baseline to CB last-layer retraining, deep feature reweighting, and the four variants of SELF which we describe in Section 5. The  $x$  axis is the size of the reweighting dataset, i.e., the number of points selected from the held-out dataset, and the  $y$  axis is the worst-group accuracy. The held-out dataset has a fixed size of 600 for Waterbirds, 9934 for CelebA, 22590 for CivilComments, and 41231 for MultiNLI. A general trend is that SELF methods tend to improve as the size  $n$  of the reweighting dataset increases, except on MultiNLI. We plot the mean over three independent runs and leave out error bars for readability.
Table 9: Total variation distance comparison. We compare the usage of KL divergence in our early-stop disagreement SELF method to total variation distance (TVD) as used in our Theorem 1. Our method is robust to the choice of distance function (though TVD is worse on CelebA) and therefore our usage of TVD in Theorem 1 is empirically justified. We list the mean and standard deviation over three independent seeds.
Method	Worst-group test accuracy
Waterbirds	CelebA	CivilComments	MultiNLI
KL divergence	93.0±0.3	83.9±0.9	79.1±2.1	70.7±2.5
Total variation distance	92.2±0.6	72.9±3.4	78.6±2.8	66.3±2.3
(b) CelebA
(d) MultiNLI
Figure 6: Group validation accuracies over training. We plot the validation accuracies by group for the four benchmark datasets over the course of ERM training. Contrary to the assumptions made by JTT [41] and other early-stop misclassification methods, the worst-group accuracy decreases with training on CivilComments. This highlights a potential advantage of disagreement methods over misclassification methods: disagreement works regardless of whether the early-stopped or convergent model has a greater dependence on the spurious feature, as long as there is a large relative difference. We plot the mean over three independent runs and leave out error bars for readability.
Table 10: SELF training accuracies. We detail the training accuracies on the SELF held-out dataset using the best configuration of each method with respect to worst-group validation accuracy. We find that the misclassification techniques have much lower training accuracy than disagreement or random - as low as  $0\%$  on MultiNLI - meaning that the misclassified points cannot be fit without changing the features. We list the mean and standard deviation over three independent seeds.
Method	Held-out training accuracy
Waterbirds	CelebA	CivilComments	MultiNLI
Random SELF	96.3±0.2	90.6±9.2	91.1±7.8	86.7±11.6
Misclassification SELF	94.6±1.6	25.3±21.9	0.4±0.2	0.2±0.3
ES Misclassification SELF	91.3±7.5	28.5±24.7	1.1±2.0	0.0±0.0
Dropout Disagreement SELF	95.5±1.4	98.7±1.5	50.9±6.3	68.3±24.7
ES Disagreement SELF	97.0±2.0	79.9±9.2	58.1±3.3	52.6±2.5
(a) Waterbirds
(c) CivilComments
(d) MultiNLI
Figure 7: Label efficiency comparison. We compare the performance of early-stop disagreement SELF to DFR while varying the amount of group annotations used for validation and retraining, respectively. The results show that SELF is robust to hyperparameter tuning and can massively reduce the annotation requirement: e.g., at  $1\%$  of data, Waterbirds has only 6 examples and CelebA has only 99 examples. Moreover, SELF retains more performance than DFR when the number of group annotations is very low. We plot the mean and standard deviation over three independent seeds.
C Theoretical proof-of-concept for disagreement SELF
In this section, we describe and state our main theoretical proof-of-concept of disagreement SELF (Theorem 1). We begin with a description of our setup and assumptions.
Setup. Let  $\mathcal{X} \subseteq \mathbb{R}^v$  be the data domain,  $n$  be the sample size, and  $d \geq v, n$  be the number of features. For  $x \in \mathcal{X}$  and weights  $\alpha, \beta > 0$  we assume that the learned predictor of both the ERM and regularized models takes the form
$$
\hat {f} (x) = \alpha \phi_ {\text {c o r e}} (x) + \beta \phi_ {\mathrm {s p}} (x) + \sum_ {k = 1} ^ {d - 2} \phi_ {\mathrm {j u n k}} ^ {k} (x) \tag {2}
$$
and assume that the label  $y$  is generated by  $y = \operatorname{sgn}(\phi_{\mathrm{core}}(x))$  (i.e., zero label noise). Note that this specific split of features into core, spurious, and junk categories is an extension of a common simplified setting in the literature on overparameterized linear models [82, 34], where only core and junk features are considered and only in-distribution accuracy is analyzed. We also consider the domain  $\mathcal{X}$  to be partitioned into a minority group  $\mathcal{X}_{\mathrm{min}}$  and majority group  $\mathcal{X}_{\mathrm{maj}}$  such that  $y\phi_{\mathrm{sp}}(x) < 0$  for all  $x \in \mathcal{X}_{\mathrm{min}}$  and  $y\phi_{\mathrm{sp}}(x) > 0$  for all  $x \in \mathcal{X}_{\mathrm{maj}}$ . (By definition,  $y\phi_{\mathrm{core}}(x) > 0$  for any  $x$ .)
Assumptions on the feature learning process. We make some extra simplifying assumptions on the feature learning process that are motivated by empirical observations in the literature. First, we assume that the learned features  $\phi_{\mathrm{core}}, \phi_{\mathrm{sp}}, \phi_{\mathrm{junk}}$  are identical for the regularized and ERM models, and are only weighted differently in the last layer. This assumption is well-justified for dropout disagreement SELF since we only use dropout for inference in the last layer (with frozen features), and for early-stop disagreement SELF since feature learning has empirically been observed to occur during the initial phase of training [29, 15]. More concretely, we consider the ERM model  $\hat{f}_{\mathrm{erm}}$  to have weights  $\alpha_{\mathrm{erm}}$  and  $\beta_{\mathrm{erm}}$ , and the regularized model  $\hat{f}_{\mathrm{reg}}$  to have weights  $\alpha_{\mathrm{erm}}$  and  $\beta_{\mathrm{reg}}$ . Second, we assume that weights on the core and spurious features are normalized such that  $\alpha_{\mathrm{erm}} + \beta_{\mathrm{erm}} = \alpha_{\mathrm{reg}} + \beta_{\mathrm{reg}}$ , essentially positing that the proportion of junk feature strength is the same in both ERM and regularized models. This is also justified by the empirical observation that regularization methods such as early stopping make minimal difference to in-distribution accuracy [49].
Main result. The following result compares a minority group test example and a majority group test example with features that are equal in magnitude, and shows that the disagreement is higher for minority group points regardless of whether the ERM or regularized model weights the spurious feature higher. We will use the total variation distance (TVD) as our cost function for disagreement SELF. We focus on TVD instead of KL divergence to simplify the proof; we show in Table 9 (detailed in Appendix B) that the empirical results are competitive using TVD (though worse on CelebA). We show that the difference of the TVD terms is proportional to  $|\beta_{\mathrm{erm}} - \beta_{\mathrm{reg}}| \coloneqq |\alpha_{\mathrm{erm}} - \alpha_{\mathrm{reg}}|$ , illustrating that the models disagree more drastically the greater the discrepancy between their dependence on the spurious feature (or, equivalently, the core feature).
Theorem 1 (Disagreement SELF). Consider two examples  $x_{\min} \in \mathcal{X}_{\min}$  and  $x_{\max} \in \mathcal{X}_{\max}$  which have the same labels and whose features have the same magnitude, i.e.,  $\phi_{\mathrm{core}}(x_{\mathrm{min}}) = \phi_{\mathrm{core}}(x_{\mathrm{maj}})$ ,  $|\phi_{\mathrm{sp}}(x_{\mathrm{min}})| = |\phi_{\mathrm{sp}}(x_{\mathrm{maj}})|$ , and  $\sum_{k=1}^{d-2} \phi_{\mathrm{junk}}^k(x_{\mathrm{min}}) = \sum_{k=1}^{d-2} \phi_{\mathrm{junk}}^k(x_{\mathrm{maj}})$ . Let  $P_{\min}$  and  $Q_{\min}$  be distributions over  $\{-1,1\}$  which take value 1 with probability  $(b(\hat{f}_{\mathrm{erm}}(x_{\min})) + 1)/2$  and  $(b(\hat{f}_{\mathrm{reg}}(x_{\min})) + 1)/2$  respectively, and likewise for  $P_{\mathrm{maj}}$  and  $Q_{\mathrm{maj}}$ . Then, we have
$$
\operatorname {T V D} \left(P _ {\min }, Q _ {\min }\right) - \operatorname {T V D} \left(P _ {\max }, Q _ {\max }\right) = b \min  \left(\left| \phi_ {\text {c o r e}} \left(x _ {\max }\right) \right|, \left| \phi_ {\mathrm {s p}} \left(x _ {\max }\right) \right|\right) \left| \beta_ {\mathrm {e r m}} - \beta_ {\mathrm {r e g}} \right| > 0. \tag {3}
$$
These are the estimated distributions we use to classify the inputs.
Proof. For any  $x \in \mathcal{X}$  define  $P(x) = (b(\hat{f}_{\mathrm{erm}}(x)) + 1) / 2$  and  $Q(x) = (b(\hat{f}_{\mathrm{reg}}(x)) + 1) / 2$ . By the assumptions on the features,
$$
\begin{array}{l} \operatorname {T V D} (P (x), Q (x)) = | P (x) - Q (x) | (4) \\ = \frac {b}{2} \left| \hat {f} _ {\text {e r m}} (x) - \hat {f} _ {\text {r e g}} (x) \right| (5) \\ = \frac {b}{2} | (\alpha_ {\text {e r m}} - \alpha_ {\text {r e g}}) \phi_ {\text {c o r e}} (x) - (\beta_ {\text {e r m}} - \beta_ {\text {r e g}}) \phi_ {\text {s p}} (x) | (6) \\ = \frac {b}{2} \left| \beta_ {\text {e r m}} - \beta_ {\text {r e g}} \right| \left| \phi_ {\text {c o r e}} (x) - \phi_ {\text {s p}} (x) \right|. (7) \\ \end{array}
$$
Let  $c = |\phi_{\mathrm{core}}(x_{\mathrm{maj}})|$  and  $d = |\phi_{\mathrm{sp}}(x_{\mathrm{maj}})|$ . By definition of the minority and majority groups,
$$
\left| \phi_ {\text {c o r e}} \left(x _ {\min }\right) - \phi_ {\mathrm {s p}} \left(x _ {\min }\right) \right| - \left| \phi_ {\text {c o r e}} \left(x _ {\max }\right) - \phi_ {\mathrm {s p}} \left(x _ {\max }\right) \right| = c + d - | c - d | = 2 \min  (c, d) > 0. \tag {8}
$$
Together with  $b \geq 0$ , Equations 7 and 8 show the result.
We utilize a ResNet-50 [24] pretrained on ImageNet-1K [57] for the vision tasks and a BERT [14] model pretrained on Book Corpus [83] and English Wikipedia for the language tasks. These pretrained models are used as the initialization for ERM on the four datasets we study. We use standard ImageNet normalization with standard flip and crop data augmentation for the vision tasks and BERT tokenization for the language tasks [28]. Our implementation uses the following packages: NumPy [22], PyTorch [53], Lightning [72], TorchVision [44], Matplotlib [26], Transformers [74], and Milkshake [36].
For ERM and last-layer retraining (Section 4), we do not vary any hyperparameters; their fixed values are listed in Table 11. Recall that our SELF method (Section 5) splits the validation dataset in half, using the first half as a held-out dataset for selecting weighting points and the second half for model selection [33, 28]. We vary the weighting dataset size  $n$  (the number of points selected from the held-out dataset) in the range (20, 100, 500). For the vision datasets, we search over learning rates  $(10^{-4}, 10^{-3}, 10^{-2})$  and train for 250 steps. For the language datasets, we search over learning rates  $(10^{-6}, 10^{-5}, 10^{-4})$  and train for 500 steps. For the two early-stopping methods, we search over checkpoints saved at  $10\%$ ,  $20\%$ , and  $50\%$  of model training, while for dropout disagreement, we search over dropout probabilities of 0.5, 0.7, and 0.9 applied only at the last layer. See Appendix B for additional ablation studies.
Table 11: ERM and last-layer retraining hyperparameters. We use standard hyperparameters following previous work [58, 27, 33, 28]. For last-layer retraining, we keep all hyperparameters the same except the number of epochs on CelebA, which we increase to 100.
Dataset	Optimizer	Initial LR	LR schedule	Batch size	Weight decay	Epochs
Waterbirds	SGD	3 × 10-3	Cosine	32	1 × 10-4	100
CelebA	SGD	3 × 10-3	Cosine	100	1 × 10-4	20
CivilComments	AdamW [43]	1 × 10-5	Linear	16	1 × 10-4	10
MultiNLI	AdamW [43]	1 × 10-5	Linear	16	1 × 10-4	10
E Additional tables
In this section we provide results in a tabular form to accompany figures in the main paper.
Table 12: Table for Figure 1a. We perform an ablation on the percentage of worst-group data used for class-balanced last-layer retraining, while keeping the total data constant. Note that the  $100\%$  numbers may not equal our other group-balanced last-layer retraining numbers due to our methodology involving downsampling and starting from a class-unbalanced ERM. We list the mean and standard deviation over three independent runs.
Percent of worst-group data included	Worst-group test accuracy
Waterbirds	CelebA	CivilComments	MultiNLI
2.5	53.5±4.4	61.1±2.9	75.3±4.9	67.3±1.8
5.0	53.5±6.0	62.8±3.9	75.4±5.0	67.4±1.9
12.5	63.1±6.3	72.0±2.8	75.5±5.1	67.6±1.9
25.0	73.6±4.2	78.2±1.7	75.5±4.9	67.8±2.1
37.5	80.2±2.9	81.5±4.2	75.6±4.8	67.8±2.1
50.0	83.2±3.7	83.7±2.8	75.8±5.0	67.9±2.0
62.5	86.2±1.3	85.7±3.1	75.8±4.8	67.9±2.0
75.0	86.5±2.1	87.0±2.5	76.1±4.7	68.0±1.9
87.5	88.8±0.6	88.2±1.3	76.0±4.7	68.3±1.8
100.0	90.4±0.9	88.7±0.7	76.1±4.6	68.3±1.9
Table 13: Table for Figure 1b. We compare the worst-group accuracy in Table 12 to the maximum WGA increase over a class-unbalanced ERM (see Table 5). Note that Waterbirds performs poorly at low percentages because the small size of the held-out dataset leads to no worst-group data being included as a result of our downsampling methodology. We list the mean over three independent runs.
Percent of worst-group data included	Percent of max WGA increase over ERM
Waterbirds	CelebA	CivilComments	MultiNLI
2.5	-105.0	38.1	93.5	-11.1
5.0	-105.0	41.9	94.3	0.0
12.5	-51.7	62.6	95.1	22.2
25.0	6.7	76.5	95.1	44.4
37.5	43.3	83.9	95.9	44.4
50.0	60.0	88.8	97.6	55.6
62.5	76.7	93.3	97.6	55.6
75.0	78.3	96.2	100.0	66.7
87.5	91.1	98.9	99.2	100.0
100.0	100.0	100.0	100.0	100.0
Table 14: Table for Figure 2. We compare class-balanced (CB) ERM on the entire dataset to splitting the dataset and performing CB ERM on the first (95%) split and CB last-layer retraining on the second (5%) split. The results with and without the held-out dataset correspond to Figure 2a and 2b respectively. We list the mean and standard deviation over three independent runs.
Method	Held-out dataset included	Worst-group test accuracy
Waterbirds	CelebA	CivilComments	MultiNLI
CB ERM	X	72.6±3.2	66.3±3.2	60.2±2.7	67.4±2.4
CB last-layer retraining	X	72.4±3.5	74.1±2.1	80.7±1.1	64.4±1.1
CB ERM	✓	81.9±3.4	67.2±5.6	61.4±0.7	69.2±1.6
CB last-layer retraining	✓	83.1±2.7	71.3±3.2	80.8±0.2	65.6±2.4
Table 15: Table for Figure 3. We list the percentage of the reweighting dataset consisting of worst-group data for each SELF method. "Baseline" represents the percentage of worst-group data in the held-out dataset, while "Balanced" is the percentage of worst-group data necessary to achieve group balance (recall that there may be multiple worst groups; they are described in Section 4.1). We also include the worst-group accuracy (WGA) of each SELF method, to get a sense of whether the amount of worst-group data correlates with performance (in a loose qualitative sense). We list the mean over three independent runs and round to the nearest whole number.
Method	WGA/Percent of worst-group data
Waterbirds	CelebA	CivilComments	MultiNLI
Baseline	50	1	7	2
Misclassification	90/45	85/25	64/51	65/4
ES misclassification	91/45	83/7	68/37	68/20
Dropout disagreement	91/55	82/13	79/13	66/3
ES disagreement	93/67	84/2	79/24	71/4
Balanced	50	25	25	33
Table 16: Table for Figure 4. We compare different combinations of ERM and last-layer retraining with or without class balancing. Notably, class-balanced last-layer retraining enables nearly the same worst-group accuracy whether the ERM incorporates class-balancing or not. In Figure 4, we only plot the results which use class-balanced last layer retraining (i.e., the last two rows of the table). We use the entire held-out set for last-layer retraining, and we list the mean and standard deviation over three independent runs.
ERM class balancing	Last-layer retraining class balancing	Worst-group test accuracy
Waterbirds	CelebA	CivilComments	MultiNLI
X	X	87.5±0.7	45.4±2.3	54.7±5.8	64.5±0.8
✓	X	88.0±0.8	41.9±1.4	57.6±4.2	64.6±1.0
X	✓	92.3±0.4	71.1±2.4	78.4±2.5	64.7±1.1
✓	✓	92.6±0.8	73.7±2.8	80.4±0.8	64.7±1.1
F Broader impacts, limitations, and compute
Broader impacts. We hope our work contributes to the safe and equitable application of machine learning and motivates further research in ML fairness. A potential negative outcome may arise if practitioners assume their models are bias-free after applying our techniques; while we show reduced dependence on spurious correlations, no method can completely alleviate the issue, and other modes of bias may exist. We encourage practitioners to conduct rigorous examinations of model fairness.
Limitations. Our methods take advantage of the specificity of the spurious correlation setting, and therefore would likely underperform on datasets which exhibit a more extreme complete correlation (i.e., contain zero minority group data) [52, 40]. Furthermore, following previous work in this setting [58, 41, 48, 33, 28], SELF utilizes a small validation dataset with group annotations for model selection. While we show in Appendix B that SELF is robust to using as few as  $1\%$  of these annotations, and our last-layer retraining method (Section 4) does not require them, completely removing this assumption is an important direction for future research.
Compute. Our experiments were conducted on Nvidia Tesla V100 and A5000 GPUs. We used about  $25000 in compute credits in the course of this research. We believe this paper could be reproduced for under$ 5000 in credits, and a majority of this compute would go towards the ablation studies. With that said, our last-layer retraining methods only train the linear classifier, making them cheap and efficient to run even on older hardware.