Bias Amplification Improves Worst-Group Accuracy without Group Information
Authors: Gaotang Li
Jiarui Liu
Wei Hu
Venue: Submitted to ICLR 2023 (2023)
Keywords: spurious correlation
worst-group accuracy
group robustness

Abstract: Neural networks produced by standard training are known to suffer from poor accuracy on rare subgroups despite achieving high accuracy on average, due to the correlations between certain spurious features and labels. Previous approaches based on worst-group loss minimization (\textit{e.g.} Group-DRO) are effective in improving worse-group accuracy but require expensive group annotations for all the training samples. In this paper, we focus on the more challenging and realistic setting where group annotations are only available on a small validation set or are not available at all. We propose \bam, a novel two-stage training algorithm: in the first stage, the model is trained using a \emph{bias amplification} scheme via introducing a learnable \emph{auxiliary variable} for each training sample together with the adoption of squared loss; in the second stage, we upweight the samples that the bias-amplified model misclassifies, and then continue training the same model on the reweighted dataset. Empirically, \bam leads to consistent improvement over its counterparts in worst-group accuracy, resulting in state-of-the-art performance in spurious correlation benchmarks in computer vision and natural language processing. Moreover, we find a simple stopping criterion that completely removes the need for group annotations, with little or no loss in worst-group accuracy. 

Under review as a conference paper at ICLR 2023
BIAS AMPLIFICATION IMPROVES WORST-GROUP ACCURACY WITHOUT GROUP INFORMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Neural networks produced by standard training are known to suffer from poor ac- curacy on rare subgroups despite achieving high accuracy on average, due to the correlations between certain spurious features and labels. Previous approaches based on worst-group loss minimization (e.g. Group-DRO) are effective in im- proving worse-group accuracy but require expensive group annotations for all the training samples. In this paper, we focus on the more challenging and realistic setting where group annotations are only available on a small validation set or are not available at all. We propose BAM, a novel two-stage training algorithm: in the first stage, the model is trained using a bias amplification scheme via intro- ducing learnable auxiliary variable for each training sample together with the adoption of squared loss; in the second stage, we upweight the samples that the bias aaplliili model misclassifies, and then continue training the same model on the reweighted dataset. Empirically, BAM leads to consistent improvement over its counterparts in worst-group accuracy, resulting in state-of-the-art performance in spurious correlation benchmarks in computer vision and natural language pro- cessing. Moreover, we find simple stopping criterion that completely removes the need for group annotations, with little or no loss in worst-group accuracy.
INTRODUCTION
The presence of spurious correlations in the data distribution, also referred to as "shortcuts" (Geirhos et al. 2020), is known to cause machine learning models to generate unintended decision rules that rely on spurious features. For example, image classifiers can largely use background instead of the intended combination of object features to make predictions (Beery al. 2018). Similar phe- nomenon is also prevalent in natural language processing (Gururangan et al.) 2018) and reinforce- ment learning (Lehman et al. 2020). In this paper, we focus the group robustness formulation of such problems (Sagawa et al. 2019), where we assume the existence of spurious attributes in the training data and define groups be the combination of class labels and spurious attributes. The objective is to achieve high worst-group accuracy on test data, which would indicate that the model not exploiting the spurious attributes.
Under this setup, one type of method uses distributionally robust optimization framework to di- rectly minimize the worst-group training loss (Sagawa al. 2019). While these methods are effec- tive in improving worst-group accuracy, they require knowing the group annotations for all training examples, which is expensive and oftentimes unrealistic. In order to resolve this issue, a line of re- cent work focused on designing methods that do not require group annotations for the training data, but need them for a small set of validation data (Liu et al. 2021 Nam et al.) 2020 2022 Zhang et al. 2022). A common feature shared by these methods is that they all consist of training two models: the first model is trained using plain empirical risk minimization (ERM) and is intended to be "biased" toward certain groups; then, certain results from the first model are utilized to train a debiased second model to achieve better worst- group performance. For instance, a representative method is JTT (Liu al., 2021). which, after training the first model using ERM for few epochs, trains the second model while upweighting the training examples incorrectly classified by the first model.
The core question that motivates this paper is: Since the first model is intended to be biased, can we amplify its bias in order to improve the final group robustness? Intuitively, a bias-amplified first model can provide better information guide the second model to be debiased, which can

---

Under review as a conference paper at ICLR 2023
Water
Land
Water
Land
Ampltiiation
Rebalanced Training
(9 (9,
R
Landbrd
Landtind
Landtard
Figure 1: Using Grad-CAM (Selvaraju et al. 2017 to visualize the effect of bias amplification and rebalanced training stages, where the classifier heavily relies on the background information to make predictions after bias amplification but focuses the useful feature (bird) itself after the rehalanced training stage.
potentially lead improving group robustness. To this end, we propose two-stage algorithm, BAM (Bias AMplification), for improving worst- worst-group accuracy without any group annotations for training data:
Stage 1: Bias amplification. We train a bias-amplified model by introducing trainable auxiliary variable for each training example, as well as using squared loss instead of cross- entropy loss.
Stage 2: Rebalanced training. We upweight the training examples that are misclassified in Stage 1, and continue training the same model instead of retraining a new model
Evaluated various benchmark datasets for spurious correlations, including Waterbirds (Wah al. Sagawa et al., 2019), CelebA (Liu et al.] 2015) Sagawa et al. (2019), MultiNLI (Williams et al, (Borkan al. 2019 Koh et al. 2021), find that BAM achieves state-of-the-art worst-group accuracy compared to existing methods that only use group annotations on a validation for hyperparameter tuning. We also conduct de- tailed ablation study and observe that every component in BAM (auxiliary variables, squared loss, continued training) is crucial its improved performance.
2011
2018
Sagawa al. 2019), and CivilComments-Wila
Furthermore, we explore the possibility of compleiely removing the need for group annotations. We find that low-class accuracy difference (which does not require any group annotations evaluate) is strongly correlated with high worst-group accuracy. Using minimum class accuracy difference as the stopping criterion, BAM outperforms the previous state-of-the-art annotation-free method, GEORGE (Sohoni al., 22200), by considerable margin and closes the performance gap be- tween GEORGE and fully- supervised Group-DRC by an average of 88% on the image classification datasets.
RELATED WORKS
variety of recent work discussed different realms robustness, for instance, class imbalance (He & Garcial 2009 Huang et al.) 2016 Khan et al.] 2017 Johnson & Khoshgoftaar 2019 Thabtah jet a 2020), and robustness in distribution shift, where the target data distribution is different from the source data distribution (Clark al. 2019 Zhang al. 2020 Marklund et al.) 2020 Lee al. 2022 Yao et al.] 2022). In this paper, we mainly focus on improving group robustness. Categorized by the amount of information have for training and validation, discuss three directions below:
Improving Group Robustness with Training Group Annotations. Multiple works have used training group annotations improve worst-group accuracy (Byrd & Lipton, 2019 Khani et al.) 2019 Goel et al. 2020 Cao et af.. (2000; SSgawa et al.) 2020). Other works inc uude minimizing the grost- training loss using distributionally robust optimization (Group-DRO) (Sagawa al. simple training data balancing (SUBG) (Idrissi et al. 2022). ) and retraining the last layer of the
2019
In Figure we use Grad-CAM visualization illustrate that bias-amplified model from Stage focuses more on the image background while the final model after Stage 2 facuses on the object target.
2

---

Under review as a conference paper at ICLR 2023
model on the group-balanee dataset (DFR) (Kirichenko et al. 2022). These methods achieve state- of-the-art performance on all benchmark datasets. However, the acquisition of spurious attributes of the entire training set is extremely expensive and unrealistic in real-world datasets.
Improving Group Robustness with Validation Group Annotations Only, Acknowledging the cost of obtaining group annotations, many recent works focus on the setting where training group annotations are not available (Duchi & Namkoong, 2019; Oren et al. 2019 Levy et al. 2020 Pezeshki et al. 2021). TTghanaki et al. (2021) proposes a transformation network remove the spurious correlated features from image datasets and then choose classifier architectures according the downstream task. Shu et al.)
(2019) utilizes a small set of unbiased meta-data to reweight data samples. CVaR DRO (Duchi et al.
2019) introduces a variant of distributionally robust optimization that dynamically reweights data samples that have the highest losses. In particular, the most popular recent methods of achieving high worst-group accuracies involve training two models. CNC (Zhang al., 2022) first trains an ERM model help infer pseudo group labels by clustering output fea- tures and then adopts a standard contrastive learning approach to improve robustness. SSA (Nam) ct al. 2022) uses a subset of the validation samples with group annotations for training to obtain pseudo spurious attributes, and then trains robust model by minimizing worst-group loss, the same as Group-DRO. Similarly, DFRY. Val (Kirichenko et al. 2022) uses validation data with group annota- tions for training and tuning hyperparameters, though just requires retraining the last layer of the model.
Approaches that are related to our method normally use the first model to identify minority samples and then train separate model based on the results predicted by the first model (Yaghoobzadeh et al., 2019 Utama et al. (2020). LfF (Nam et al., (2020) train two models concurrently, where one model is intentionally biased, and the other one is debiased by reweighting the gradient of the loss according to a relative difficulty score. JTT (Liu et al. (021)) first trains an ERM model to identify minority groups in the training set (similar to EIIL CCeageer et al.] 202T)), and then trains a second ERM model with these selected samples to be upweighted. However, the above two-model approaches all focus on the robust training of the second model and fail consider the potential of accumulating biased knowledge from the first model. Kim et al. 2022) introduces a multi-model approach that proposes to identify hard-to-leat classifiers of a committee, and simultaneously train main classifier through knowledge distillation. In addition, despite the reduced reliance, all these approaches still require a small amount of group- annotated samples.
samples and obtain their weights based on consensus of member
Improving Group Robustness without any Group Annotations. Relatively little work has been done under the condition that no group information is provided for both training and validation. Idrissi et al. 2202)); : Liu et al. (202T) observe a significant drop (10% 25%) in worst-group test accuracy if using the highest average validation accuracy as the stopping criterion without any group annotations. One the recent works, GEORGE (Sohoni et al., 2020). tries to separate unlabeled classes in deep model feature spaces and then use the generated pseudo labels to train the model via the distributionally robust optimization objective. Additionally, [Soo et al. (202)) clusters the pseudo-attriuuts based on the embedding feature of naively trained model, and then define trainable factor to reweight different clusters based on their sizes and target losses. However, there a considerable performance gap between the unsupervised and the supervised methods.
3 PRELIMINARIES
We adopt the group robustness formulation for spurious correlation (Sagawa et al. 2019). Consider a classification problem where each sample consists of an input x E X, a label y E y. and a spurious attribute A. For example, in CelebA, X' contains images of human faces and want to classify hair color into the labels y = [blonde, not blonde). Hair color may be highly correlated with gender A {male, female] which is spurious feature that can also predict the label. We say that each example belongs to group g = (y, a) E G y x A.
Let f X + Y' be a classifier learned from a training dataset D (x R)) We hope that does not overly rely on the spurious feature a to predict the label. To this end, we evaluate the model through its worst-group error:
Errwg (f) == max E E [1[(x) # vll. gEg
3

---

Under review as a conference paper at ICLR 2023
We focus on the setting where group annotations available in the training dataset. We con- sider two cases under this setting: (1) group annotations are available in a validation set solely for the purpose of hyperparameter tuning, and (2) no group annotations available at all. We will distinguish between these cases when comparing them with existing methods.
4
OUR APPROACH: BAM
Algorithm BAM
Input: Training dataset D, number of epochs T in Stage 1, auxiliary coefficient X, and upweight
factor j..
Stage Bias Amplification
1. Optimize R, (0, B) (T) for T epochs and save the model parameters Opinc-
misclassified by frias (.) fimm (-)- Stage 2: Rebalanced Training
2. Construct the error set E 2
3. Continue training the model starting from Asim to optimize R2(
4. Apply stopping criterion:
If group annotations are available for validation, stop the highest worst-group validation accuracy;
If no group annotations are available, stop at the lowest validation class difference
We now present BAM, a two-stage approach to improving worst- corst-group accuracy without any group annotations at training time. In Stage we train bias-am hias-amplified model and select examples that this model makes mistakes on. Then, Stage 2, we continue to train the same model while upweighting the samples selected from Stage 1.
4.1 STAGE 1: BIAS AMPLIFICATION
The key intuition behind previous two-stage approaches (e.g. JTT) is that standard training via ERM tends first fit easy-to-leam groups with spurious correlations, but not the other hard-to-learn groups where spurious correlations are not present. Therefore, the samples that the model mis- classified in the first stage can be treated as proxy for hard-to-leam groups and used to guide the second stage.
We design a bias-ampli scheme in Stage with the aim of identifying a higher-quality error set to guide training in Stage 2. In particular, we introduce a trainable auxiliary variable for each example and add it to the output of the network. Let fe X + IRC be the neural network with parameters 0, where C' = |V is the total number of classes. We use the following objective function in Stage 1:
R1 (0, B)
1 Σ
Σ {(fo(zz) + y1).
(1)
Here, E C is the auxiliary variable for the i-th example in the training set, and the collection of auxiliary variables B = (b).... b,.) is learnable and is learned together with the network parameters 0 (B is initialized to be all 0). variables. We adopt the squared loss V(2, = || ey where eg RCC is the one-hot encoding for the label y. Below we explain the main ideas behind our method.
hyperparameter that controls the strength of the auxiliary
The introduction of auxiliary variables makes it more difficult for the network fo to learn, because the auxiliary variables can do the job of fitting the labels. We expect this effect to be more pro- nounced for hard-to-learn examples. For example, if in normal ERM training it takes a long time for the network fo to fit a hard-to-learn example (I,, yi). after introducing the auxiliary variable b, it will be much easier to use b; to fit the label yı. thus making the loss {(fo(z:) + Ab;, y:) drop relatively faster. This will prohibit the network fo itself from learning this example. The such effect will be smaller for casy-to-leam examples, since the network itself can still quickly fit the labels without much reliance on the auxiliary variables. Therefore, adding auxiliary variables amplifies the bias toward easy-to-learn examples, making hard examples even harder learn.
4

---

Under review as a conference paper at ICLR 2023
We note that the auxiliary variable was first introduced in Hu et al. (2020) for different motivation (learning with noisy labels). Hu et al. (2020) proved that it recovers kernel ridge regression when using the squared loss and when the neural network is in the Neural Tangent Kernel (Jacot et al. 2018) regime, and provided a theoretical guarantee for the noisy lahel learning problem. Empirically. Hu et al. (2020) showed that squared loss works better than cross-entropy loss, which is why the squared loss in (I. Additional evidence that squared loss can achieve competitive performance for classification was summarized in Hui & Belkin (2020).
At the end of Stage 1, we evaluate the obtained model (nims()) Sanm ()) identify error ser: (note that auxiliary variables now removed)
on the training set and
E = {( Wi): frias + v=).
4.2 STAGE 2: REBALANCED TRAINING
In Stage 2. we continue training the model starting from the parameters 4mas from Stage 1, using a rebalanced loss that upweights the examples in the error set E:
(2 = p Σ (cE(fo(z).))
Σ
hcE (cB()(()))))
(3)
(#.g)E
(x.))ED(E
where lc the cross-entropy loss and y a hyperparamter (upweight factor).
We note that more complicated approaches have been proposed for Stage 2, e.g. Zhang et al. (2022).
but we stick with the simple rebalanced training method in order to focus on the bias amplification effect in Stage 1.
4.3 STOPPING CRITERION WITHOUT ANY GROUP ANNOTATIONS CLASS DIFFERENCE
When group annotations are available in a validation set, we can simply use the worst- worst-group valida- tion accuracy as stopping criterion and
hyperparameters, similar prior approaches (Nam et al. 2020 Liu et al. 2021 Creager et al. 2021 Zhang et al. (002)). When no group annotations available, naive approach use the validation average accuracy as a proxy, but this results in poor worst-group accuracy (Liu et al. 2021 drissi et al. 2022).
We identify simple heuristic when no group annotations are available, using minimum class dif- ference, which we find to be highly effective and result in little or no loss in worst group accuracy. For classification problem with C classes, we calculate the sum of pairwise validation accuracy differences between classes as
ClassDiff = |cc(cllass i) Acc(class j)
ClassDiff can be calculated on a validation set without any group annotations. In all the datasets (where 2 or 3) we experiment with, we observe that ClassDiff inversely correlates with worst- group accuracy (see Section 5.4). Therefore, we can use ClassDiff as stopping criterion and completely remove the need for any group annotations.
Our algorithm summarized in Algorithm 1 It has three hyperparameters: T (number of epochs in Stage 1), À (auxiliary variable coefficient), and y (upweight factor). We provide full training details in Appendix B
5 EXPERIMENTS
In this section, we first briefly explain the experiment setup (Section 5.1 main results and show that BAM improves worst-group accuracy compared to prior methods that are trained without spurious attributes the same benchmark datasets. BAM achieves state-of-the-ar performance when group annotations are either available or unavailable the validation set (Section 5.2). Then, conduct thorough ablation studies to verify the effectiveness of every component of our proposed model (Section 5.3). Finally, we provide more detailed analyses of BAM'S behavior (Section 5.4
Next, we present our
5

---

Under review as a conference paper at ICLR 2023
5.1 SETUP
We conduct our experiments on four popular benchmark datasets containing spurious correlations. Two of them are image datasets: Waterbirds and CelebA, and the other two are NLP datasets: MultiNLI and CivilComments- W...eent--W..DS..... WILDS. The full dataset details are in Appendix A BAM is trained the absence of training group annotations throughout all experiments. We obtain the main results of BAM via tuning with and without group annotations on the validation set, following Algorithm
For fair comparison, we adopt the general settings from previous methods (JTT) and stay con- sistent with other approaches without extensive hyperparameter tuning (batch size, learning rate, regularization strength in Stage etc.). We use pretrained ResNet-50 (He et al. (016)) for image datasets, and pretrained BERT (Devlin et al. 2019) for NLP datasets. More details be found in Appendix
Table 1: Average and worst-group test accuracies of different approaches evaluated image datasets (Waterbird and CelebA), We run BAM and JTT (in *) on 3 random seeds based on the highest worst-group validation accuracies and minimum class differences, respectively, and report the mean and standard deviation. Results of EIIL and Group-DRO are reported by Nam et al. (2022). and results of other approaches come from their original papers. The best worst-group accuracies under the same condition are marked in bold.
Annotation only used for tuning i-params?
Annotation free?
Method
Waterbird
CelebA
Avg.
Worst-group
Avg.
Worst-group
SUBG (Idrissi et al 2022
89.1+:.
85 85.6+22
GroupDRO (aeawa
2019
91.8
89.2
88.5
SSA NNm
92.2-0a7
89.0=a.s
92.8
89.8=122
No
ERM
97.3
72.6
95.6
47.2
EIIL Creager et
96.9
78.7
91.9
Yes
CNC Zhhang et
90,9
88.5
89.9
88.8
JIT
89, 89.9=641
86.8 8=1.6
91.3eau
78. 7
BAM
91.4_nas
89.7.am
88.0gam
83.5ga
GEORGE (Shhnni et al. 2020
95.7
76.2
94.8
52.4
ClasTDif
88.5
87.1
11.8
75.4
BAM ClassDiff
91.4gm01
89.2...1s
88.4
80.3
Table 2: Average and worst-group test accuracies of different approaches evaluated on natural lan- guage datasets (MultiNIL and CivilComment-WILDDS following the same conventions in Table
Annotation only used for unning h-params?
Annotation free?
Method
MoliNLI
CivilComments WILDS
Avg.
Worst- group 68.9=ns
Avg.
Worst group
SUBG Idrissi al.
2022
71.8-u4
No
GroupDRU Sugawa el
2019
81 1
76.62841
87.7
69 11.
SSA Nam
79.9
76.6144
88.2
No
ERM
82.4
67.9
92.6
$7.4
Creager et al.
79.4
70.9
90.5
67.0
Yes
CNC Zhang
81.7at5
68.922.
JTT
80.0gn4
68. 1uss
17.2
77.7.mm
BAM
79.6
71.5
88.3
79.3
GEORGE (Sohoni et al. 2020
Yes
JTT Cluss
66.5
87.2
77.7
BAM ClassDiff
80.3
71.2
18.3
79.3
5.2 MAIN RESULTS
Tables and report the average and worst-group test accuracies of BAM and compare it against standard ERM and recently proposed methods under different conditions, including SUBG (Idrissi
et al. 2022 JTT (Liu al. 2021), SSA (Nam et al. 2022], CNC (Zhang et al., 2022). GEORGE

---

Under review as a conference paper at ICLR 2023
BAx Waterbed
Irr Wateliid
6.9
6.9
Wover Cropp Tex Acuseg
Woog Group Tex Accuacy
0.7
BAM One M Tim--M
Jer Two M Jir One-M
21
100 120
20
100
Fpochs T Stage
Fpochs T Stage
Figure 2: (1) Comparison of BAM's performance between training One-M and Two-M. (2) Compar- ison of JTT's performance between training One-M and Two-M. We find that One-M consistently achieves better worst-group test accuracies than Two-M for BAM over all hyperparameter combina- tions, while we do not observe the same result on JTT.
(Shoon et al. 2020), and Group-DRO (Sagawa et al. 2019 We tune BAM and JIT according to the highest worst-group validation accuracy (not annotation free) and minimum class difference (annotation-free).
First, compared with other methods that use group annotations only for hyperparameter tuning, BAM consistently achieves higher worst-group accuracies across all datasets, with the exception of the CelebA dataset on which CNC achieves better performance. We note that CNC primarily focuses on improving Stage 2 with more complicated contrastive learning method, while BAM uses the simple upweighting method. It is possible that the combination of CNC and BAM could lead to better results. Nevertheless, the result of BAM is promising on all other datasets, even surpassing the weakly-supervised method SSA and the fully-supervised method Group-DRO on Waterbirds and CivilComments- WILDS.
Second, if the validation group annotations are not available at all, BAM achieves state-of-the-ee performance on all four benchmark datasets and improves previous methods by a large margin. Notably, BAM recovers a significant portion of the gap in worst-group accuracy between GEORGE (previous state-of-the-arr that requires no group annotations) and Group-DRO/SSA (previous state- of-the-art requiring supervision) by average of 88% on the image classification datasets.
BAM'S improved performance in worst-group accuracy comes at the expense of a moderate drop in average accuracy. The tradeoff' between average accuracy and worst-group accuracy is consistent with the observation made by Liu et al. (2021); Sagawa et al. (2019). We note that our implementa- tion of JTT follows directly from its published code, and obtain much higher performance the CivilComments-II.D dataset than originally reported by Liu et al. (002))
5.3 ABLATION STUDIES
We conduct thorough ablation studies on the Waterbirds dataset to test the effectiveness of every component of our model (the use of auxiliary variable and the squared loss in Stage 1; continued training in Stage 2). For consistent terminology, we define the approach that loads the model from Stage and continues training the same model in Stage 2 One-M. We define the approach that trains a separate ERM model in Stage 2 as Two-M. In this subsection, we first verify the effectiveness of using One-M in Stage Then, we study the necessity of using the combination of the auxiliary variable and the squared loss in Stage 1. We have total of 8 ablations: Train One-M using (1) cross- entropy loss, (2) squared loss, (3) auxiliary variable (AUX.), and cross-entropy loss, (4) auxiliary variable and squared loss in Stage 1; Train Two-M using (5) cross-entropy loss, (6) squared loss, (7) auxiliary variable and cross-entropy loss, (8) auxiliary variable and squared loss Stage 1. We note that (1) & (5), (2)& & (6), (3) & (7), and (4) & (8) share the same model in Stage I and have the same error set to be upweighted. Notably, (4) recovers the standard procedure of BAM and (5) recovers the standard procedure of JTT. For fair comparison,
employ the same hyperparameters throughout our ablation studies. Stage 2 of (1) (8) are tuned with identical procedures. We use the same stopping criterion (highest worst-group validation accuracy) for ablation studies. We tune
7

---

Under review as a conference paper at ICLR 2023
Wandbord
CalbA
CM Conmere WILES
MatNu
MustCrrup
Wert-rrup Val Acceray Abadds Gan D Fusrrs
WerstGra
WarstGraa
2.0
Abeo ise Dava Drfinesss
0.
Ads Can Cils
0.6
Ads sie Cas Citarrncc
0.2
30
25
30
20
57
50
20
offeccis Piigs?
3
of iiage
Figure 3: Relation between absolute valdiation class difference and worst-group validation accuracy in Stage 2 on Waterbirds, CelebA, CivilComments- WILDS, and MultiNLI. It can be observed that minimizing absolute validation class difference is roughly equivalent to maximizing worst-group accuracy in the validation set. Each dataset uses the same hyperparameters as employed in Table and [ Each line represents the value averaged over 3 different seeds and the shade represents the standard deviation.
over T = {10. 15, 20, 25, 30, 40, 50, 60, 80, 100, ]20 and μ {50, 100, 140} for each setting, and fix À~ = 0.5 whenever auxiliary variables involved. To reduce the influence of randomness, we run all experiments using three different seeds and report the mean of the worst-group accuracies in Tables and By analyzing the results, we have the following observations:
One-M is better than Two-M only after bias amplification We compare highest worst-group accuracies between One-M and Two-M approaches under all different combinations of components in Stage I: the use of auxiliary variable and the choice of loss functions. Namely, we compare the performance of the model (1) vs. (5), (2) vs. (6). (3) vs. (7), and (4) vs. (8), as shown in Table The experiment result shows that with
the same error set and hyperparame- ters and using our proposed elements, continuously training one model consis- tently achieves higher worst-group ac- curacies, averaging a performance gain of 2.3%. However, One-M itself does not provide improvement for JTT, as
Table 3: Ablation on One-M versus Two-M
Ablation
Worst Group (Sc.(S)
(1) One Cross- Cross-Entrpyy Luss
86.4
(5) Two- Cruss Entmpy Lass
(+1.4)
(2) One M. Aux. Variable and Cross: Entropy Loss
88.7
(6) Two-M, Aux, Variable and Cross Fonss-Fniropy Loss
86.7 (-2.0)
shown in (I) vs. (5).
(3) One M, Squared Loss
88.8
7) Two-M, Squared Loss
86.6 (-2.2)
Figure compares the highest worst-
(4) One-M, Aux. Variable and Squared Loss
89.7
group test accuracy between One-M
(8) Two-M, Aux. Variable and Squared Loss
87.0 (-2.7)
BAM/JTT and Two-M BAM/JTT over a wide range of Stage epochs T. The re- sult suggests that One-M BAM outperforms Two-M BAM in every single Stage epoch T, while such findings do not apply to JTT. In other words, One-M is better than Two-M only after BAM's 's bias amplification. In particular, BAM outperforms JTT with much fewer epochs T' needed in Stage 1, and the Twvo-M BAM has slightly better worst-group performance than JTT. Furthermore, similar to Liu et al. 2021). the worst-group test accuracy of BAM only stays high for a range of epochs T (10-25) in Stage and degrades as gets large.
Every component in Stage 1 helps with improving group robustness After illustrating the superior perfor- mance of One-M, here we systemati- cally investigate the necessity of the use
Table 4: Ablation on auxiliary variable and loss functions
Ablation
Worst-group (cc.(%)
of our proposed components in Stage 1, namely the combination of the auxiliary variable and the loss function. In Ta- ble 4 we apply One-M approach with
(1) Baseline (Cross (Cross-Entropy Loss)
86.4
(2) Baseline + Aux. Vuriable
88.7 (+2.3)
Baseline Squared Loss
88.8 (+2.4)
(4) Baseline Aux. Variable Squared Loss
89.7 (+3.3)
the same fair setup in all four experi- ments in Stage 1. We consider the sole use of cross-entropy loss in Stage as our baseline model. The addition of the auxiliary variable and the substitution of the loss function is indicated with the sign "+". The results suggest that the isolated use of the auxiliary variable, squared loss and the combination of both all contribute non-trivial improvements in performances over the baseline
8

---

Under review as a conference paper at ICLR 2023
model. In particular, the use of squared loss biases the model faster in Stage achieving its best performance with only T = 20 epochs compared to JTT's 60 epochs in Stage 1, as illustrated by Figure
5.4 FURTHER ANALYSES
Relation between class difference and worst-group accuracy Figure plots the trend of the absolute class difference and the worst-gros group accuracy on the validation set in Stage 2 for the Wa- terbirds, CelebA, MultiNLI and CiviComments datasets. Clearly, there is a strong inverse relation- ship between the absolute class difference and worst-group accuracy in the validation datasets, which justifies the use of class difference as a stopping criterion.
all four
Table 5: Error set specific information. The statistics directly follow from our ablation studies in Section 5.3 AUX or SQ respectively represents the sole use of auxiliary variable or squared loss in Stage 1. Each entry is evaluated by calculating the mean of the error sets from three different seeds. The last row indicates the best worst-group lest accuracy evaluated using One-M and Two- M approaches. It is reasonable
list them together because the one/two model ablation uses the identical error set. We also experiment to train a model that only upweights the "minority examples" (waterbirds in land and landbirds in water) according to our understanding.
Group Size
Group Description
AUX
SQ
BAM
Minority-Onip
3498
Waterbird in Water Waterbird Land
8
17
12
0
184
76
85
184
56
Landbird in Water Landbird in Land
47
46
36
35
56
1057
114
99
96
91
0
Worst-groun Accuracy (One-M/Two-M)
+
86.4/86.8
88.7/86.7
88.8/86.6
89.7/87.0
79.2
Error Set Related Analysis In Table 5 all listed methods identify a large portion of minority examples. After using our proposed components, the set indicates that the classifier becomes more "biased" towards the majority groups. This is manifested by the amount of majority and minority examples misclassified by the classifier in Stage BAM, AUX, and SQ all contribute to more minority group examples (# waterbird in land + # landbird in water) and fewer majority group examples (# waterbird in water # landbird in land) misclassified compared with JTT. However, note that the combination of squared loss and auxiliary variable (BAM) does not guarantee a linearly more "biased" eтrоr set compared to their isolated usages. Nevertheless, seemingly more "biased" enror set does not necessarily guarantee optimal performance. The extreme condition, where only the minority samples are upweighted, contributes worst-group test accuracy that is substantially lower than every method listed in Table
DISCUSSION
In this paper, we present BAM, two-stage method that does not use any training group annotations and achieves state-of-the-ar worst-group performance. We also introduce novel stopping criterion that completely removes the need for any group annotations and is applicable to other methods as well. We conclude our paper by proposing several directions for future work:
First, it is crucial to have better theoretical understanding of our "bias amplification" scheme. While we conduct thorough ablation studies and provide empirical evidence to support our proposed method, the definition of "bias" and its effects on the model still needs to be understood theoretically. is also helpful to systematically investigate the influence of the loss functions.
Second, aside from the success on the four benchmark datasets, it would be interesting if the class difference can be applied to other datasets, especially those with much more classes.
Finally, while with other recently developed methods for Stage 2, such as CNC (Zhang et al. 2022). Any future work that focuses improving Stage 2 could also our general framework.
simple upweighting in Stage 2, it is possible to combine approach in Stage

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456 456-473, 2018.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion pro- ceedings of the 2019 world wide web conference, pp. 491-500, 2019.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In International Conference on Machine Learning, pp. PMLR
PMLR, 2019.
Kaidi Cao, Yining Chen, Junwei Lu, Nikos Arechiga, Adrien Gaidon, and Tengyu Ma. Het- eroskedastic and imbalanced deep learning with adaptive regularization. arXiv preprint 100::20006..766. 2020.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. arXiv preprint arXiv: 100.003909.03883,, 2019.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, pp. 2189-2200. PMLR, 2021.
Jacob Devlin, Ming- Keig-WWiil Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 4171-41888 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 18653/v1/NN19-4425 7/ a antho ogy. wwtp/1/g1//11/1//
URL https:
John Duchi and Hongseok Namkoong. Variance- regularization with convex objectives. The 2019.
Journal of Machine Learning Research,
John C Duchi, Tatsunori Hashimoto, and Hongseok Namkoong. Distributionally robust losses against mixture covariate shifts. Under review, 2, 2019.
R. Geirhos, J. Jacobsen, C. Michaclis, R. Zemel, W. Brendel, M. Bethge, and A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-677) 2020. doi: 10. 101111//422560000000000
Karan Goel, Albert Gu, Yixuan Li, and Christopher Ré. Model patching: Closing the subgroup performance gap with data augmentation. arXiv preprint arXiv:2008.0677: 06775, 2020.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 2 (Short Papers), pp. 107-112, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/N18- 1017220 URL
Haibo He and Edwardo Garcia. Learning from imbalanced data. IEEE Transactions on Knowl- edge and Data Engineering, 1263) 2009. doi: 10. 1211099KKDEE2208223553
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp- 770-778, 2016. doi: 10.1109/CVPR.20169990
Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training noisily labeled data with generalization guarantee. arXiv: Learning, 2020.
Chen Huang, Yining Li. Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5375-5388. 2016.
10

---

Under review as a conference paper at ICLR 2023
Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss cross- entropy in classification tasks. arXiv preprint arXiv:2006.07322, 2020.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-accurna. In Conference Causal Learning and Reasoning, pp. 336-351. PMLR, 2022.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernl: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data, 6(1)11-54, 2019.
Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE transactions on neural networks and learning systems, 29(8):3573-33)) 2017.
Fereshte Khani, Aditi Raghunathan, and Percy Liang. Maximum weighted loss discrepancy. arXiv preprint 1r0iv:/906.0333 035/8, 2019.
Nayeong Kim. Schyun Hwang, Sungsoo Ahn, Jaesik Park, and Suha Kwak. Learning debiased classifier with biased committee. arXiv preprint arXiv:2206. 10843, 2022.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re- re-training is sufficient for robustness to spurious correlations. arXiv preprint wwt.i::224.0299.0 2022.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 5637-5664. PMLR, 18-24 Jul 2021. URL htttps://proceediing igs ml press 1ttk339//kh222 httt//hhhh2aaamm
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from under- specified data, arXiv preprint arXiv:2202.03:: 03418 2022.
Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities. Artificial life, 26(2):274-306 2020.
Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distribution- ally robust optimization. Advances Neural Information Processing Systems, 33:8847-8860, 2020.
Evan Z. Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedingg of Machine Learning Research, pp. 6781-6792. PMLR, 18 18-24 Jul 2021. URL htt.tt::/ppocceeellllaa
mwr. ptt.ss//ees///1//////
1iu21f. html
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. 2015 IEEE International Conference Computer Vision (ICCV), pp. 3730-3738, 2015.
Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, Jure Leskovee, Anshul Kundaje, et al. Wilds: A benchmark of in-the-wit wild distribution shifts. arXiv preprint arXiv:20/2.0742. 2020.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De- biasing classifier from biased classifier. Advances in Neural Information Processing Systems, 04220673-20884 2020.
11

---

Under review as a conference paper at ICLR 2023
Junhyun Nam, Jachyung Kim, Jacho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. arXiv preprint arXiv:2204.02070, 2022.
Yonatan Oren, Shiori Sagawa, Tatsunori B Hashimoto, and Perey Liang. Distributionally robust language modeling. arXiv preprint /909 02060, 2019.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guil- laume Lajoie. Gradient starvation: learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34: 226111222222 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arXiv: /91 1.08311, 2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterizatioo exacerbates spurious correlations. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference
Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8346-8356. PMLR, 13-18 Jul 2020. URL
https://procecinng wwweeeiiinsssm...... mlr wwt.ssss..r.ppprrss... prwsss wwwpppesssslle.eeeeeeepeppap.ait wttpp////////////////aaaaaaaaaaa www.s/aaaaaaaaa.c..
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-62 1186626, 2017.
Seonguk Seo, Joon-Young Lee, and Bohyung Han. Unsupervised learning of debiased representa- tions with useudo-attributes In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16742-16751, 2022.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- weight-net: Learning an explicit mapping for sample weighting. Advances in neural information processing systems, 32, 2019.
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse- -aarsee classification problems. Advances in Neural 2020.
Information Processing Systems,
Saeid A Taghanaki, Kristy Choi, Amir Hosein Khasahmadi, and Anirudh Goyal. Robust representa- tion learning via perceptual similarity metrics. In International Conference on Machine Learning. pp. 10043-10053. PMLR, 2021.
Fadi Thabtah, Suhel Hammoud, Firuz Kamalov, and Amanda Gonsalves. Data imbalance in classi- fication: Experimental evaluation. Information Sciences, 513:429 441, 2020.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Towards debiasing nlu models from unknown biases. arXiv preprint arXiv: 12303, 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge J. Belongie. The caltech- uesd birds-200-201 dataset. 2011.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122 New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/1I-- 1101. URL httpss:/aclatthooogyy
orq/N18-1101
Yadollah Yaghoobzadeh, Soroush Mehri, Remi Tachet, Timothy J Hazen, and Alessandro Sor- doni. Increasing robustness to spurious correlations using forgettable examples. arXiv preprint 19X 2019.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improv- ing ditt---disttibttion robustness via selective augmentation. arXiv preprint arXiv:2201.0029, 2022.
12

---

Under review as a conference paper at ICLR 2023
Jingzhao Zhang, Aditya Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. arXiv preprint 20XX::20000.223000 2020.
Michael Zhang, Nimit Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Ré. Correct-n- contrast: A contrastive approach for improving robustness to spurious correlations. arXiv preprint arXiv:2203.01517, 2022.
13

---

Under review as a conference paper at ICLR 2023
A DATASET DETAILS
Waterbirds (Wah et al. 2011 Sagawa et al. 2019): Waterbirds is an image dataset consisting of two types of birds different backgrounds. It contains images of birds directly cut and pasted on real-life images of different landscapes. We aim at classifying y = (waterbird, landbird], which is spuriously correlated with the background A (water, land). The train/validdttinntest splits is followed from Sagawa et al. (2019).
CelebA (Liu al. 2015):
CelebA is image dataset consisting of celebrities with types of hair color and different genders. we aim at classifying celebrities' hair color y = (blond, not blond), which is spuriously correlated with the gender A (male, female]. The train/validation/test splits
is followed from Sagawa et al. (2019).
2018): MultiNLI is a natural language processing dataset, where each sample consists of two sentences and the second sentence is entailed by, neutral with, or contra- dicts the first sentence, with the presence or absence of negation words. We aim classifying the sentence relationship ) = |entailment, neutral, contradiction), which is spuriously correlated with (negation, no negation). The train/validation/test splits is followed from
MultiNLI (Williams
the negation words
Sagawa et al. (2019).
CvilComments-WIL.DD (Borkan et al. 2019 Koh et al. 2021) CivilComments WILDS is a nat- ural language processing dataset consisting of online comment that is toxic non-toxic, with or without the mentions certain demographic identities (male, female, White, Black, LGBTQ, Muslim, Christian, and other religions). We aim classifying whether the sentence is toxic Y = ftoxic, non-toxic), which spuriously correlated with the mentions of (demographic) identi- ties A (identity, no identity). The train/validationntest splits is followed from Koh al. (2021).
B TRAINING DETAILS
In this section, we provide details about the model selection and the hyperparameter tuning for different datasets. As claimed in the main text, in order make fair comparison with previous methods, we use the same pretrained models. Namely, we use ResNet-50 pretrained from Image-net weights for Waterbirds and CelebA, and pretrained BERT for MultiNLI and CivilComments. We use the Pytorch implementation for ResNet50 and the HuggingE a cce implementation for BERT. We tune BAM and JTT according to class difference and worst-group accuracies the validation set in Stage 2.
Table 6: Hyperparameters tuned over 4 datasets.
Dataset
Auxiliary coefficient (A)
#Epochs in Stage (T)
Upweight factor (μ²
Waterbirds CelebA
{0.05, 0.5, 5)
{10, 15, 20}
(50, 100, 140)
(0.05, 0.5,
[. 2)
{50, 70, 100}
MultiNLI
(0.05, 0.5, 5)
1. 2
{4. 5, 6}
CivilComments
(0.05, 0.5, 5)
(1.2)
{4. 5, 6]
In general, our setting follows closely from Liu et al. (2021), with some minor discrepancies. For the major hyperparameters, We tuned over À {0.05, 0.5, 5). T (1,2, 10, 15, 60) and I = [4, 5, 6, 50, 70, 100. 140) for BAM. Despite the three choices for X, we actually fix 0.5 throughout our studies since it yields the best result and in fact, all three of X's yield very similar output under similar conditions. We note that BAM fairly insensitive with the choice of X. We tune over 7 21,2} and μ = (4, 5, 6, 50, 70, 100, 140) for JTT for fair comparisons. We tuned the major hyperparameters according to Table 6 More details are provided below:
Waterbirds We use the learning rate le-5 and batch size 64 for two stages of training. We use the stochastic gradient descent (SGD) optimizer with momentum 0.9 throughout the training process. We use & regularization 1 for Stage rebalanced training. We apply the same above setting for both JTT and BAM. Notably, as illustrated by Figure when tuned for the minimum absolute validation class difference, the curve may fluctuate abnormally after the first 30 epochs and it is clear that the
14

---

Under review as a conference paper at ICLR 2023
model is not learning anything useful. We tackle this problem by smoothing out abrupt changes in class difference (neglect the result the difference between consecutive class differences greater than 10%). The best result for JTT occurs when T 60 and μ 140. The best result for BAM occurs when = 20 and = 140. We train for total of 360 epochs
CelebA We use the learning rate le-5 and batch size 128 for two stages of training. We use the stochastic gradient descent (SGD) optimizer with momentum 0.9 throughout the training process. We use l regularization 0.1 for Stage 2 rebalanced training. We apply the same above setting for both JTT and BAM. The best result for JTT occurs when T and μ 70. The best result for BAM occurs when 7 = and 50. We train for total of 60 epochs.
MultiNLI We use batch size 32 for two stages of training. We apply an initial learning rate of 2c-5 for Stage and le-5 for Stage 2. We use the SGD optimizer without clipping for Stage I and the AdamW optimizer with clipping for Stage 2. We use & regularization 0.1 for Stage 2 rebalanced training. We apply the same above setting for both JTT and BAM. The best result for JTT occurs when T = and μ 4. The best result for BAM occurs when T 2 and 6. We train for total of 10 epochs.
CivilComments -IIDDS We use batch size 16 for two stages of training. We apply an initial learning rate of 2e-5 for Stage and le-5 for Stage 2. We use the SGD optimizer without clipping for Stage I and the A optimizer with clipping for Stage 2. We use & regularization 0.01 for Stage rebalanced training. We apply the same above setting for both JTT and BAM. The best result for JTT occurs when T 1 and = 4. The best result for BAM occurs when T 1 and J 4. We train for a total of epochs.
C ADDITIONAL FIGURES AND STATISTICS
C.I SUPPLEMENTARY FIGURES
Figure shows some supplementary plots to Figure where from (1) and (2) it can be observed that using different upweight factors y in Stage will not change our findings that One-M on BAM performs better than Two-M, while Two-M on JTT performs better than One-M. In addition, Figure 5 shows that even in the case without any group annotations, BAM can still have a comparable result with JTT by stopping when achieving minimum class difference the validation set.
BAM Waterbird
JTT Waterbird
0.90
0.90
Wosst- Group Accuracy
0.85
0.85
0.80
0.80
0.75
One-M, 50
0.75
Two 50
One M, 10100
Two M, 10
0.70
M, 14140
Wonst-
0.70
Two 1=14
Two M, 5=
One M. 5=
0.65
Two 1=100
0.65
One 1=100
Two M, 14040
One M. 1=140
0.00
0.60
20
20
60
80
100
120
20
10
60
80 100
120
Epochs in Stage 1
Epochs in Stage 1
Figure 4: (1) Comparison performance of BAM between training one model and models through different upweight factor μ on Waterbirds. (2) Comparison of performance of JTT between training one model and two models through different upweight factor j on Waterbirds.
15

---

Under review as a conference paper at ICLR 2023
Stopping Stage by Minimal Class Difference on Val Data
Robust Worst- Accuracy
0.7-
0.6
HAM
100
120
Epochs T in Stage
Figure Comparison of performance of BAM against JTT by stopping at the minimum class dif- ference on validation data by choosing their best set of hyperparameterss separately. We BAM outperforms JTT with fewer epochs T in Stage
C.2 LESS VALIDATION SET
Table Worst-group accuracy on Waterbirds with varying size validation set with group anno- tations. BAM maintains high worst-group test accuracies even when tuned with very few numbers of group-annotated validation set. However, we note that with fewer size of validation set with annotations, the performance is actually worse than when tuned for full-size validation set without any annotations.
Size of Annotated Validation Set
100%
20%
10%
5%
Worst- Worst-Group Acc.(%)
89.8
89.1
88.4
86.2
16

---

Under review as a conference paper at ICLR 2023
D REBUTTAL UPDATES
D.1
VISUALIZATION OF AUXILIARY VARIABLES
0.10
0.20
Matartird Naterbird In
landbiro
0.05
0.15
0.00
0.10
logtt
logt1
0.05
-9.10
0.00
0.15
0.05
-0.0
0.10
0.05
0.50
0.15
0.20
0.15
0.05
0.00 0.05
(rrateriir))
Figure Distributions of the auxiliary variable w.r.f. the waterbird class (left) and landbird class(right) the training stopping at T 20. We two distinct colors to illustrate dis- tributions of two groups in each class. The coordinates of data sample i relative to the origin show the bias learned by the aux. variable.
watertird water
landbiro
Meterbind
0.15
0.00
lopt (and810)
0.10
-0.55
lopt
0.05
-0.0
0.00
-0.15
0.05
-0.10
-0.05
0.00
0.05
0.10
0.15
-0.20
-0.15
-0.10
-0.05
0.00
0.05
(wsterbrs))
(wsterbind))
Figure Corresponding KDE plot of Figure
In Figure 6 we have two observations regarding the distribution of auxiliary variables on the train- ing set. First, values of data samples in the majority and the minority group have clear distinctions for each class. This shows that the auxiliary variables differentiate between easy-to-learn and hard- to-learn examples. Second, auxiliary variables for majority group examples are in general closer to the origin, while those in minority groups tend to have larger (positive) logit values on the ground truth class they actually belong to and have smaller (negative) logit values the class they do not belong to. The visualization shows that the auxiliary variables help with fitting hard-to-learn sam- ples, which corroborates the intuition described in Section 4.1 We observe the same trend for any
17

---

Under review as a conference paper at ICLR 2023
A 5, 50, 70, 100) well any T in Stage 1 in {20, 50, 100). which strongly supports our claims that "adding auxiliary variables amplifies the bias toward easy-to-learn examples" in Section 41
D.2 How AUX. VARIABLE CHANGES TRAINING PROCEEDS
460
405
-053
-005
-020
-2.0
SEE GE5 OE2
-033
the
64
023-
0:5-
02-
850
000-
028
-58
-02
2.2
Figure 8: Distribution changes of the auxiliary variable w.r.. the waterbird class (upper) and land- bird class(lower) on the training set stopping at T (left), T 20 (middle), and T 50 (right).
In Figure g
we observe that as training proceeds, the magnitudes of auxiliary variables get larger, and the minority group and majority group within each class can be casier to split apart.
D.3 STABILITY OF T AND ROBUSTNESS OF À
First, we find that BAM is robust to the choice of X, and below we present the best test accuracies obtained from a wide range of À using the waterbird dataset.
Table 8: Robustness of As
Best worst group test accuracy
0.5
89.9
10
89.8
30
89.7
35
90.2
40
89.5
50
89.5
70
88.8
100
88.6
The main benefit of using a larger À eliminates the need to carefully tune over T. As originally T increases with small choice
reported in Figure 2 we observe degradation of performance of X. Nevertheless, as we experiment with larger that can yield comparable best worst-group accuracies, we find that it helps with preventing the original model from overfitting. Moreover,
18

---

Under review as a conference paper at ICLR 2023
Table 9: Robustness of worst group test accuracy with different Ts
T
Worst group test accuracy
80
88.6
50
100
88.2
120
86.9
70
80
88.8
70
100
88.8
70
120
88.5
there is longer need to carefully tune over T. Table 2
illustrates that even trained until full convergence, an appropriate choice of À can still guarantee state-of-the-ar performance.
19


Reviews:
[{"review_id": "jUqRQXleZH", "reviewer": "Reviewer_3fdp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Several group robust optimization have achieved remarkable performance but they require group annotations, which limits the practicality. To alleviate this limitation, the authors focus on the scenario where group annotations are not or partially available on a validation set. The proposed method BAM consists of two stages; 1) model is trained using a bias amplification scheme with an auxiliary variable and 2) upweight the samples that are misclassified in 1).", "summary_of_the_review": "- The authors proposed a two-stage methods for learning debiased representations, but however, I don’t think this method is sufficiently novel compared to the existing approaches. For a stronger work, I think it needs to conduct more detailed and thorough analysis about auxiliary variable for debiasing.\n", "strength_and_weaknesses": "### Strength\n- The paper provides performance improvement in most of datasets.\n- The authors also introduce early stopping strategy without group annotations to ensure the unsupervised setting.\n- The writing is easy to follow.\n\n### Weakness\n- The main idea is partially overlapped with LfF (stage 1) and JTT (stage 2), and the performance improvement is not significant compared to theirs. The auxiliary variable aims to make hard examples to be learnt harder, which has similar objective to generalized cross entropy of LfF. What is the advantage of auxiliary variable, compared to GCE of LfF?\n- Why the auxiliary variable is instance-wise, rather than the common one for all samples? If I correctly understand, this is not realistic if there are millions or billions of training images. Also, how does the auxiliary variable changes as training progresses?\n- There are some missing comparisons for unsupervised debiasing methods [LWBC, BPA, CVaR DRO]. Especially, [LWBC] also employs a biased committee to learn a final debiasing classifier.\n\n[LWBC] Learning Debiased Classifier with Biased Committee, NeurIPS 2022\n\n[CVaR DRO] Large-scale methods for distributionally robust optimization, NeurIPS 2020\n\n[BPA] Unsupervised learning of debiased representations with pseudo-attributes, CVPR 2022\n\n### Minor questions\n- Why One-M is better than two-M only after bias amplification? There is no explanation.\n- The results of BAM/BAM+class diff and JTT/JTT+class diff on CivilComments are the same. Is it correct?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cHIuhs4GNV", "reviewer": "Reviewer_CiNy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "I am adding an additional review to aid AC in making a decision for the paper. \n\nThe paper proposes a method to improve the worst-off group accuracies in a dataset when the group annotations are only available during validation. Prior research either uses group annotations during training (such as GroupDRO) or trains in an unsupervised manner as a two-stage method (such as JTT). \n\nA two-stage method is proposed in the paper where an error set of hard-to-learn examples is identified in stage one. These examples are devoid of any spurious correlations and are identified via learnable auxiliary variables and a squared loss. In stage-two, the sample misclassified by stage-one are upweighted. \n\nFor tuning hyper-parameters, a metric called ClassDiff is introduced. This metric captures the pair-wise class difference between the accuracies. Experiments show it to be a good proxy for worst-off group accuracy. \n\nExperimental results show an improvement over the worst-off group accuracy in four benchmark datasets. \n", "summary_of_the_review": "The paper proposes a two-stage approach for improving the worst-off group accuracies. The method bears similarities to JTT, as both the approaches first identify and error set and later upweight the samples based on the error set. On the experimental side, the method is comparable to the baselines when evaluated for both average group accuracy and worst-off group accuracy. While it is interesting to see an unsupervised metric, ClassDiff, being for hyper-parameter tuning, it is unclear when such a metric is applicable and when should one avoid using ClassDIff. Based on these reasons, I am leaning towards a reject. ", "strength_and_weaknesses": "**Strengths**\n\nA strength in the paper is the introduction of the ClassDiff metric. The authors verify the efficacy of the metric in being used as a proxy for worst-off group accuracy in their experiments. Prior works like JTT and EIIL use group annotations for hyper-parameter tuning, however, obtaining group annotations is often an expensive task. For this reason, it is essential for a method to avoid any group annotations during training or validation. \n\nThe ClassDiff metric is analogous to a fairness measure called demographic parity. Additionally, Worst-off group accuracies is captured by the fairness measure called Rawlsian Criterion. The experiments demonstrate that on the datasets used in the paper demographic parity can be used in place of the Rawlsian Criterion for hyper-parameter tuning. \n\nThe auxiliary variable technique effectively identifies error sets and improves the worst-off group accuracy. This is demonstrated via experiments where the overall accuracy and worst-off group accuracy are comparable to the baselines. \n\nThe paper provides a good list of ablation experiments. The most helpful ones are the One-M / Two-M model comparisons and the experiments identifying inverse relationship between ClassDiff and Worst-off Group Accuracy. \n\n**Weaknesses**\n\nThere is an inherent trade-off between the average group accuracy and the worst-off group accuracy. The experiments demonstrate that the proposed method provides a small drop in the overall accuracy at the cost of improving the worst-off group accuracy. It maybe required to observe the pareto-frontier of the proposed method and the other baselines to asses if the proposed method provides better metrics for a wide range of hyper-parameters. At the moment, the experiments suggest that the proposed method is comparable (and not strictly better) than the baselines when evaluated for both average accuracy and worst-offf group accuracy. \n\nWhile the paper suggests that ClassDiff (or the demographic parity) measure can be used as a proxy for Worst-off group accuracy (or the Rawlsian Criterion), it is unclear if such a proxy would work always. A more thorough study, either through theoretical analysis or experimental evidence, is needed to identify scenarios in which ClassDiff is to be (and is NOT to be) used. \n\nThe method proposed in the paper, similar to JTT, is a two-stage approach. Any two-stage approach requires 2X training time relative to ERM. It is a weakness of the method that it requires 2X training time relative to ERM. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BDNeLYCf7o", "reviewer": "Reviewer_bEzt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of (in-distribution) subpopulation generalization when facing potential subgroups shift in the training data, without knowing the group annotations. The paper proposes a bias amplification scheme, which consists of two training stages. In the first stage, auxiliary variables are introduced for each input sample during model optimization, which are used to identify subgroups. Then the model is further trained using a re-weighting scheme based on the identified subpopulations of data points. It also proposes an early stopping method that does not need any subgroup annotations even in the validation set. Experiments on several datasets show that the method might be effective over several baselines on popular subpopulation shift datasets.", "summary_of_the_review": "The paper studies an interesting yet important problem, model generalization when facing potential subgroups / subpopulations in the training data without knowing the group annotations.\n\nThe overall idea of leveraging a bias amplification scheme for subgroup discovery and robustness seems to be novel and not explored before in this field, which is plausible.\n\nHowever, currently there are several drawbacks and weaknesses in terms of Methods and Exepriments, making me less confident on its stability, scalability, and working rationale.\n\nIn summary, my rating is borderline for now. I'm willing to change my score if my concerns/questions are well addressed.", "strength_and_weaknesses": "# Strengths\nOverall the strengths of the paper are pretty clear. They can be summarized into the following points.\n\n+ The idea of introducing learnable auxiliary variables with a bias amplification scheme for subgroup discovery and robustness seems to be novel and not explored before in this field.\n\n+ The paper also introduced a simple yet effective method - class accuracy difference - to avoid the need of a balanced validation set with group annotations.\n\n+ The author did a comprehensive experimental validation. They tested the BAM method on 4 popular subpopulation shift datasets, and explore the effects of the proposed 3 components via a thorough ablation study. The latter part is especially plausible as it makes the contribution of different components clearer.\n\n+ The writing is overall good and easy to follow. The paper is well structured.\n\n---\n\n# Weaknesses\nThere are, however, several major weaknesses exist in the current paper.\n\n## Method\n- The authors claimed that \"adding auxiliary variables amplifies the bias toward easy-to-learn examples\". This is however not supported in the experiments.\n1. What do the final learned variables look like after stage 1? It would be interesting and important to see the distribution of learned variables.\n2. Are there any differences between that of minority groups versus majority groups? Is the bias really amplified toward easy-to-learn examples?\n\n- The bias amplification scheme introduces an auxiliary variable for **each example** in the dataset. Moreover, by its definition, each variable has a size of $C$, which is the total number of classes. Therefore, the size of the auxiliary variables is propotional to the dataset size (total number of samples) as well as the total number of classes. It is doubtful that whether the method could scale to large-scale real-world datasets with large number of samples and fine-grained classification requirements (e.g., iNaturalist dataset has > 8,000 classes and tens of millions of images).\n\n- Related to the above point, the idea of introducing auxiliary variables is plausible, but it also introduces potentially high computational costs. For example, ERM does not need any tracking or storing intermediate variables. I would like to see the actual computational cost w.r.t. time and memory costs. Wall-clock training time comparison as well as memory consumption to other baseline methods (e.g., ERM, JTT) would be good.\n\n## Experiments\n- Using squared loss instead of CE loss is interesting. The main intuition (motivation) comes from Hu et al. (2020). However, in Hu et al. (2020), using squared loss is reasonable as it recovers kernel ridge regression formulation. When it comes to the subpopulation setup in this paper, it is unclear why using squared loss is a good choice in the first place.\n1. Can the authors explain, theoretically or hypothetically, that using squared loss is a good choice compared to CE loss in your setup?\n2. The main change from CE to squared loss is using a soft regression loss instead of a hard classification loss. What about other (regression) losses, e.g., L1 or Huber loss? Is it actually the **squared loss** that leads to the improvement, or any of the aforementioned **regression losses** can lead to improvements? More comparisons are needed to confirm the argument.\n\n- In Figure 2, regardless of whether One-M  or Two-M is used, the final performance of BAM degrades as $T$ gets large. This contrasts with the intuition, and also indicates that BAM training might not be stable and highly depends on the number of epochs used in stage 1.\n1. Can the authors explain why performance of BAM degrades as $T$ gets larger?\n2. Given its sensitivity to $T$, for a new dataset, how should one empirically choose the value of $T$? Does this indicate that BAM still needs a validation set to choose proper hyper-parameters?\n\n---\n\n# Additional questions\n\n- Re-weighting is used for balanced training in stage 2. I wonder why not use data re-sampling (i.e., upsample the identified minority group or downsample the majority group). Is there a specific reason using only re-weighting?\n\n- In Table 2, for the  CivilComments-WILDS dataset, the performances of BAM & JTT are exactly the same with or without annotations, but this does not hold for other datasets. I wonder is there a reason behind this? Intuitively with group annotations should improve the performance (at least for the worst group).\n\n- Intuitively, the learned \"bias\" would be more stable when the model training is converged, which is however not the case (Figure 2). What if in stage 1, the network is trained longer enough? What would be the overfitting pattern? Will the model overfit to all training samples?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7h45ZIf2AW-", "reviewer": "Reviewer_UsCc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a two-stage training approach for improving worst-group performance, BAM (Bias Amplication), where in the first stage a bias amplified model is trained (Hu et al. 2020) to replace that of JTT, and the second stage still follows JTT. The paper also proposes a validation metric that works well without any group annotation on the validation set. Empirical results are reported and abundant ablations are performed to support the model.", "summary_of_the_review": "The paper is one of the few that explicitly links the worst-group improvement literature with learning with label noise (LLN), and explores another understudied scenario where no group annotations are available at all. The paper can be improved if some comments above can be addressed.", "strength_and_weaknesses": "Strengths:\n\n1. The paper is one of the few that explicitly links the worst-group improvement literature with learning with label noise (LLN). A particular LLN model (Hu et al. 2020) is used to replace stage 1 of JTT. I think the idea can be quite interesting for many readers to further explore the links between the two research areas.\n2. The paper explores worst-group improvement without any group annotations at all which is an understudied problem in the literature.\n3. Abundant ablations are performed to support the proposed model. Some can be quite interesting to other readers, such as One-M vs Two-M (Fig 2).\n\nWeaknesses: The method is clearly presented, and I only have some comments/clarification questions.\n\n1. I find it a bit hard to understand the motivation for the ClassDiff validation metric for model selection. Why pairwise class perf diff should work? For example, is it  somehow related to worst-class accuracy (as opposed to worst-group that requires annotation)? How robust is it if one varies class imbalance or (group,class)-imbalance (as a tuple) in the val set? Some intuitions/ablations can help.\n2. Is lambda in Eq. 1 fixed to 0.5 instead of tuned as a hparam (as said in Sec 5.3, but not sure how it's done in Sec 5.1/2?)\n3. The paper discussed, and compared empirically through ablation, the choice of MSE vs CE as loss function in Stage 1. In fact, a relatively well known robust loss for LLN is simple MAE loss (https://arxiv.org/pdf/1712.09482.pdf). Given its implementation simplicity, it would be interesting to see how it compares to MSE/CE to suit this task.\n4. As a minor point, despite being the an interesting and well-motivated idea to use a noise-robust bias amplifying model to replace JTT's stage 1 model, I still think the contribution can be much stronger if more LLN models are tested (and of course significantly more amount of work needed). Or at least I think it'd be good to make the point clear from the beginning that in fact many other LLN models (e.g. curriculum learning, robust losses, label transition) can be candidates to identify possibly group-conflicting training samples. (Perhaps another LLN model may suit the task even better with better performance.)\n5. As common evaluation criteria of LLN models, one could also report the clean/noisy ratio (i.e. identified clean/noisy samples in the training set) to demonstrate how well the method works. A similar idea could be employed to show how well bias amplifying model compares to a simple early-stopped ERM-trained model (JTT) in terms of identifying group-conflicting training examples in another experiment/ablation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject