Bias Amplification Enhances Minority Group Performance
Authors: Gaotang Li
Jiarui Liu
Wei Hu
Venue: Accepted by TMLR ()

Abstract: Neural networks produced by standard training are known to suffer from poor accuracy on rare subgroups despite achieving high accuracy on average, due to the correlations between certain spurious features and labels. Previous approaches based on worst-group loss minimization (e.g. Group-DRO) are effective in improving worse-group accuracy but require expensive group annotations for all the training samples. In this paper, we focus on the more challenging and realistic setting where group annotations are only available on a small validation set or are not available at all. We propose BAM, a novel two-stage training algorithm: in the first stage, the model is trained using a bias amplification scheme via introducing a learnable auxiliary variable for each training sample; in the second stage, we upweight the samples that the bias-amplified model misclassifies, and then continue training the same model on the reweighted dataset. Empirically, BAM achieves competitive performance compared with existing methods evaluated on spurious correlation benchmarks in computer vision and natural language processing. Moreover, we find a simple stopping criterion based on minimum class accuracy difference that can remove the need for group annotations, with little or no loss in worst-group accuracy. We perform extensive analyses and ablations to verify the effectiveness and robustness of our algorithm in varying class and group imbalance ratios.

Published in Transactions on Machine Learning Research (03/2024)
Bias Amplification Enhances Minority Group Performance
Gaotang Li University of Michigan Ann Arbor, MI
gaokangSumicch.cmm eduu
Jiarui Liu
jiarui550cs. cmau. edu
Carnegie Mellon University Pittsburgh, PA
Wei Hu
wwh@umich edu
University of Michigan Ann Arbor, MI
Reviewed OpenReview: https: httpe://openrevieal openreview. net/ forum? id= 750
Abstract
Neural networks produced by standard training are known to sulfer from poor accuracy on rare subgroups despite achieving high accuracy on average, due to the correlations between certain spurious features and labels. Previous approaches based on worst-group loss minimization (e.g. Group-DRO) are effective in improving worse-group accuracy but require expensive group annotations for all the training samples. In this paper, we focus on the more challenging and realistic setting where group annotations are only available small validation set training algorithm: in the first stage, the model is trained using bias amplifiention scheme via introducing a learnable auriliary variable for each training sample; in the second stage, we upweight the samples that the bias-amplified model misclassifies, and then continue training the same model on the reweighted dataset. Empirically, BAM achieves competitive performance compared with existing methods evaluated on spurious correlation benchmarks in computer vision and natural language processing. Moreover, we find a simple stopping criterion based ménimum class accuracy difference that remove the need for group loss in worst-group accuracy. We perform extensive analyses
not available at all. We propose BAM, novel two-stage
annotations, with little and ablations to verify the offectiveness and robustness of our algorithm in varying class and group imbalance ratios
1 Introduction
The presence of spurious correlations in the data distribution, also referred to as "shorteuts" (Geirhos et al. 2020), is known to cause machine learning models to generate unintended decision rules that rely on spurious features. For example, image classifiers can largely use background instead of the intended combination of object features to make predictions (Beery et al. 2018).
Similar phenomenon is also prevalent in natural language processing (Gururangan et al. 2018 and reinforce- ment learning (Lchman et al. 2020). In this paper, we focus on the group robustness formulation of such problems (Sagawa et al. 2019), where assume the existence of spurious attributes in the training data and define groups to be the combination of class labels and spurious attributes. The objective is to achieve high worst-group accuracy test data, which would indicate that the model is not exploiting the spurious attributes.
"Equal contribution.
"Our code available bttps://github. http://at////tttte.o/a/ot/iiiiie http://hhhb...cmm/....eeetee///s

---

Published in Transactions on Machine Learning Research (03/2024)
Water
Land
Water
Land
Water
Land
Waterbird
Bias Amplification
Rebalanced Training
R (0,
RS
Landbled
Figure 1: Using Grad-CAM (Selvaraju et al. 2017) to visualize the effect of bias amplification and rebalanced training stages, where the classifier heavily relies on the background information to make predictions after bias amplification but focuses on the useful feature (bird) itself after the rebalanced training stage.
Under this sctup, one type of methods use distributionally robust optimization framework to directly minimize the worst-group training loss (Sagawa et al., 2019). While these methods are effective in improving worst-group accuracy, they require knowing the group annotations for all training examples, which is expensive and oftentimes unrealistic. In order to resolve this issue, line of recent work focused on designing methods that do not require group annotations for the training data, but need them for a small set of validation data (Liu et al., 2021 Nam et al. 2020 2022 Zhang et al., 2022 h. A common feature shared by these methods is that they all consist of training two models: the first model is trained using plain empirical risk minimization (ERM) and intended to be "biased" toward certain groups; then, certain results from the first model are utilized to train a debiased second model to achieve better worst-group performance. For instance, a reprosentative method is JTT (Liu et al. 2021 which, after training the first model using ERM for few epochs, trains the second model while upweighting the training examples incorrectly classified by the first model.
The core question that motivates this paper is: Since the first model is intended to be biased, can we amplify its bias in order to improve the final group robustness? Intuitively, a bias-amplified first model can provide better information to guide the second model to be debiased, which can potentially lead improving group robustness. To this end, we propose novel two-stage algorithm, BAM (Bias AMplification), for improving worst-group accuracy without any group annotations for training data:
Stage 1: Bias amplification. We train a bias-amplified model by introducing a trainable auxiliary variable for each training example.
Rebalanced training. We upweight the training examples that are misclassified in Stage 1, and contimue training the same model instcad of retraining a new model P
Stage 2:
Evaluated on various standard benchmark datasets for spurious correlations, including Waterbirds (Wah) ct al. 2011; Sagawa et al., 2019), CclebA (Liu et al.) 2015 Sagawa et al. 2019). MultiNLI (Williams et al.
2018: Sagawa et al. 2019). and CivilComments- WILDS (Borkan et al. 2019 Koh el al., 2021), we lind that
BAm achieves competitive worst- worst-group accuracy compared to existing methods in the setting where group annotations are only available on a validation set.
Digging deeper into the mechanism of BAM, we observe that auxiliary variables learned in Stage exhibit clearly different magnitudes between majority and minority group examples, thus confirming their effectiveness in bias amplification. We also find that BAM achieves robust performance across hyperparameter choices. In addition, ablation studies demonstrate the clear advantage of continued training in Stage 2 over training separate model from scratch and the effectiveness of each of our proposed components.
Furthermore, we explore the possibility of completely removing the need for group annotations. We find that low class accuracy difference (which does not require any group annotations to evaluate) is strongly correlated
PIn Figure we use Grad-CAM visualization illustrate that our bias-amplified model from Stage focuses more on the image background while the final model after Stage 2 focuses on the abject target.
2

---

Published in Transactions on Machine Learning Research (03/2024)
with high worst-group accuracy. Using minimum class accuracy difference the stopping criterion, BAM achieves the best overall performance on benchmark datasets in the complete absence of group annotations. We also perform extensive controlled experiments and find that this approach is robust across different datascts, varying dataset sizes, and varying imbalance ratios.
2 Related Works
A variety of recent work discussed different realmns of robust for instance, class imbalance (He & Garcia
2009; Huang et al.] 2016 Khan et al.) 2017 Johnson & Khoshgoftaar 2019 Thabtah et al. 2020), and robustness in distribution shift, where the target data distribution is different from the source data distribution
(Clark et al.) 2019 Zhang ct al., 2020 Marklund et al. 2020 Lee et al. 2022 Yao et al. 2022). In this paper, we mainly focus on improving group robustness. Categorized by the amount of information available for training and validation, we discuss three directions below.
Improving Group Robustness with Training Group Annotations. Multiple works used training
group annotations to improve worst-group accuracy (Byrd & Liptonn 2019 Khani et al., 2019: Goel et al.
2020
Cao et al. 2020 Sagawa ct al., 2020). Other works include minimizing the worst-group training loss using distributionally robust optimization (Group-DRO) (Sagawa et al., 2019), simple training data balancing (SUBG) (Idrissi et al. 2022), and retraining the last layer of the model on the group-balance,
dataset (DFR) Kirichenko et al. 2022). These methods achieve state-of-the-art performance on all benchmark datascts. However, the acquisition of spurious attributes of the entire training set is expensive and often unrealistic.
Improving Group Robustness with Validation Group Annotations Only. Acknowledging the cost of obtaining group annotations, many recent works focus on the setting where training group annotations are not available (Duchi & Namkoong 2019; Oren al., 2019 Levy et al., 2020 Pezeshki et al.) 2021 Taghanaki et al. (2021) proposed transformation network to remove the spurious correlated ueatures from image datasets and then choose classifier architectures according to the downstream task. Shh et al. (2019) utilized small set of unbiased meta-data to reweight data samples. CVaR. DRO (Duchi et al.) 2019) introduced a variant of distributionally robust optimization that dynamically reweights data samples that have the highest losses. Jeon et al. (2022) proposed a method that utilizes the multi-variant and unbiased characteristics of CNNs through hierarchical feature extraction and orthogonal regularization. In particular, the most popular recent methods of achieving high worst-group accuracy involve training two models. CNC (Zhang et al.) 2022) first trains an ERM model to help infer pseudo-group labels by clustering output features and then adopts standard contrastive learning approach to improve robustnoss. SSA (Nam et al., 2022 uses a subset of the validation samples with group annotations for training to obtain pseudo spurious altributes, and then trains a robust model by minimizing worst-group loss, the same as Group-DRO. Similarly, DFRY, Val (Kirichenko
iet al. 2022) uses validation data with group annotations for training and tuning hyperparameters, though it just requires retraining the last layer of the model. Kim et al. (2022) introduces a multi-model approach that identifies hard-to-learn samples and obtains their weights based on the consensus of member classifiers of a committee, and simultaneoully trains main classifier through knowledge distillation.
Most related to this paper are the approaches that use model to identify minority samples and then train second model based on the results predicted by the first model (Yaghoobzadeh et al., 2019 Utama et al. 2020). LfF (Nam et al. 2020) trains two models concurrently, where one model is intentionally biased, and the other one is debiased by reweighting the gradient of the loss according to a relative difficulty score. JrT [Lu et al. (021) first trains an ERM model to identify minority groups the training set (similar to FIIT, Creager et al. 2021)), ), and then trains sccond ERM model with these selocted samples to be upweighted. Hwang et al. 2022) trains an auxiliary model with generalized supervised contrastive loss to learn biased features, and then implicitly upweights of minority groups based on the mixup method (Zhang et al. 2017]. p. Our algorithm improves over these methods by introducing trainable per-example auxiliary variables for bias amplification in the first model, as well as using continued training instead of training a separate second model.
Improving Group Robustness without any Group Annotations. Relatively less work has been done under the condition that no group information is provided for either training or validation. Idrissi al. (2022); Liu et al. (202T) observed significant drop (10%- 25%) in worst-group test accuracy if using the highest.
3

---

Published in Transactions on Machine Learning Research (03/2024)
average validation accuracy as the stopping criterion without any group annotations. GEORGE (Shooni et al. 2020) generates pseudo group annotations by performing clustering in the neural network's & feature space, and uses the pseudo annotations to train & robust model via distributionally robust optimization. Seo ct al. )022 clustered the pscudo-attributes based on the embodding feature of a naively trained model, and then defined a trainable factor to reweight different clusters based on their sizes and larget losses. However, there a considerable performance gap between group annotation-free methods and those that use group annotations.
The method of per-example auxiliary variables first introduced in Hu et al (020) for different motivation (lcarning with noisy labels). Hu et al. (2020) proved that it recovers &2 regularization when the neural network is in the Neural Tangent Kernel (Jacot et al.) 2018) regime, and provided a theoretical guarantee for the noisy label learning problem. Our result shows that this method is also applicable to improving worst-grout group accuracy through its bias amplification effect.
3 Preliminaries
We adopt the group robustness formulation for spurious correlation (Sagwwa et al. 2019 Consider a classification problem where each sample consists of an input x E X, label y E V, and a spurious attribute A. For example, in CelebA, X contains images of human faces and we want to classify hair color into the labels y (blonde, not blonde). Hair color may be highly correlated with gender A {male, female) which is spurious feature that can also predict the label. We say that each example belongs to group (y, a. E G = x A.
Let f X' + y be a classifier learned from a training dataset D = overly rely on the spurious feature a to predict the label. To this end, we evaluate the model through its worst-group error:
{( W)) We hope that f does not
Errwg(f) = max Ea.x (egls + vll.
gEg
We focus the setting where group annotations available in the training dataset. We consider two cases under this setting: (1) group annotations are available in validation set solely for the purpose of hyperparameter tuning, and (2) no group annotations are available at all. We will distinguish between these cases when comparing them with existing methods.
4
Our Approach: Bam
Algorithm BAM
Input:
Training dataset D, number of epochs T in Stage 1, auxiliary coefficient X, and upweight factor μ
Stage Bias Amplification
1. Optimize (0, B) equation for T epochs and save the model parameters A
2. Construct the error set E equation 2 misclassified by fuias = ff (+).
Stage Rebalanced Training
3. Continue training the model starting from Oriss to optimize R2) equation
4. Apply stopping criterion:
If group annotations are available for validation, stop at the highest worst-group validation accuracy;
If no group annotations are available, stop at the lowest validation class difference equation
We now present BAM, a two-stage approach to improving worst-group accuracy without any group annotations at training time. In Stage 1, we train bias-amplified model and select examples that this model makes mistakes on. Then, in Stage we continue to train the same model while upweighting the samples selected from Stage
4

---

Published in Transactions on Machine Learning Research (03/2024)
4.1 Stage 1: Bias Amplification
The key inluition behind previous Iwo-stage approaches (e.g. JrT) is that standard training via. ERM tends to first fit easy-to-learn groups with spurious correlations, but not the other hard-to-learn groups where spurious correlations are not present. Therefore, the samples that the model misclassified in the first stage can be treated as a proxy for hard-to-learn groups and used to guide the second stage.
We design a amplliyin scheme in Stage 1 with the aim of identifying a higher-quality error set to guide training in Stage In particular, we introduce trainable auxiliary variable for each example and add it to the output of the network. Let fo X' - RC be the neural network with parameters 9, where C' = | is the total number of classes. We use the following objective function in Stage 1:
1 - Σ (Vo(xx)) + Abi, ys). 7
Ri (0, B)
(1)
Here, h, E C is the auxiliary variable for the i-th example in the training set, and the collection of auxiliary variables B = (bi....,bb, b,) is learnable and is learned together with the network parameters via gradient- based optimization (B is initialized to be all 0). E is the loss function. À is hyperparamcter that controls the strength of the auxiliary variables: if x = O, this reduces to standard ERM and auxiliary variables are not used; the larger À is, the more we are relying
auxiliary variables to reduce the loss.
The introduction of auxiliary variables makes it more difficult for the network fo to learn, because the auxiliary variables can do the job of fitting the labels. We expect this effect to be more pronounced for hard-to-learn examples. For example, if in normal ERM training takes a long time for the network fo to fit a hard-to-learn example (Ii Wi), after introducing the auxiliary variable 6,, it will be much easier to use b, to fit the label ye: thus making the loss (zf) + Ab, W) drop relatively faster. This will prohibit the network So itself from learning this example. Such effect will be smaller for easy-to-learn examples, since the network itself can still quickly fit the labels without much reliance on the auxiliary variables. Therefore, adding auxiliary variables amplifies the bias toward easy-to-learn examples, making hard examples even harder to learn. We note that the trivial solution where the model achieves zero loss without actually learning will not occur with a proper choice of X.
At the end of Stage 1, we evaluate the obtained model fsias()) = Sgmm (-) on the training set and identify an error set: (note that auxiliary variables now removed)
E ((F)) folna(x))) # vif.
4.2 Stage 2: Rebalanced Training
In Stage 2, we continue training the model starting from the parameters Aoiss from Stage using rebalanced loss that upweights the examples in the error set E:
R2( = Σ ((f(z)),)+
Σ ((o(o)) (E.))DςE
(E.D)R
where j is a hyperparamter (upweight. factor). For fair comparison, our implementation of upweighting follows that of JIT (Liu et al., 2022 b, i.e., we construct an upsampled dataset containing examples in the error set y times and all other examples once.
We note that more complicated approaches have been proposed for Stage 2, e.g. Zhang et al. (002)), but we stick with the simple rebalanced training method in order focus on the bias amplification effect in Stage
4.3 Stopping Criterion without any Group Annotations Class Difference
When group annotations are available in a validation set, we can simply use the worst-group validation accuracy as stopping criterion and to tune hyperparameters, similar prior approaches (Nam al. 2020
Liu et al. 2021
Creager et al. 2021 When group annotations are available, naive approach to use
5

---

Published in Transactions on Machine Learning Research (03/2024)
the validation average accuracy proxy, but this results in poor worst- porst-group accuracy Liu et al. 2021 Idrissi et. al. 2022).
We identify a simple heuristic when no group annotations are available, using minimumm class difference, which we find to be highly effective and result in little or no loss in worst-group accuracy. For a classification problem with classes, we calculate the average of pairwise validation accuracy differences between classes as
CiassDiff
1 (
Σ
|Aco(class i) Acc(class j)}-
ISiSiSC
ClassDiff can be calculated on validation set without any group annotations. The main intuition for ClassDiff is that expect the worst-group accuracy to be (near) highest when all group accuracics have relatively small gap belween each other. Below we present simple claim showing that ClassDiff being small is a necessary condition for having a small group accuracy difference.
Claim 4.1 (proved in Appendix Cj. If the accuracies of any two groups differ by at most c, then ClassDiff S
In all the datasets (where C or 3) we experiment with, as well for varying dataset size, class imbalance ratio, and group imbalance ratio, observe that ClassDiff inversely correlates with worst-group accuracy (see Section 6.3 long as ClassDiff does not fluctuate around the same value. Our results suggest that ClassDiff is a useful stopping criterion when no group annotations are available.
Our algorithm is summarized in Algorithm It has three hyperparameters: T (number of epochs in Stage 1), (auxiliary variable coefficient), and μ (upweight factor). We provide full training details Appendix
5
Experiments
In this section, we demonstrate the effectiveness of BAM in improving worst-group accuracy compared to prior methods that are trained without spurious altributes on standard benchmark datasets.
5.1 Setup
We conduct experiments on four popular benchmark datasets containing spurious correlations. Two of them are image datasets: Waterbirds and CelebA, and the other two are NLP datasets: MultiNLI and GivlCCommens-WILDD.. The full dalaset details are in Appendix A BAM is trained in the absence of training group annotations throughout all experiments. We obtain the main results of BAM via tuning with and without group annotations on the validation set, following Algorithm 1
For a fair comparison, we adopt the general seltings from previous methods (JTT) and stay consistent with other approachee without extensive hyperparameter tuning (batch size, learning rate, regularization strength in Stage 2, ctc.). We use pretrained ResNet-50 (He et al., 2016) for image datascts, and pretrained BERT Devlin et al. 2019 for NLP datasets. More details can be found in Appendix B
5.2 Results
Tables and report the average and worst-group test accuracies of BAM and compare against standard ERM and recently proposed methods under different conditions, including SUBG (Idrissi et al., 2022). ) Jrr
Liu et al.) 20221, SSA (Nam et al.) 2022)) CNC (Zhang et. al., 2022), GEORGE (Sohoni et al. 2020), BPA (Seo et al. 2022), and Group-DRO (Sagawa et al. 2019). We tune BAM and JTT according to the highest worst-group validation accuracy (not group annotation-free) and minimum class difference (group annotation-free).
First, compared with other methods that use group annotations only for hyperparameter tuning, BAM consistently achieves higher worst-group accuracics across all datasots, with the exception of the CelebA dataset on which CNC achieves better performance. We note that CNC primarily focuses on improving Stage with a complicated contrastive learning method, while BAM uses the simple upweighting method.

---

Published in Transactions on Machine Learning Research (03/2024)
Table Average and worst-group test accuracies of different approaches evaluated on image datasets (Waterbird and CelebA). We run BAM and JTT (in ") on random seeds based on the highest worst-ga corst-group validation accuracies and minimum class differences, respectively, and report the mean and standard deviation. Results of EIIL and Group-DRO are reported by Nam et al. (2022), and results of other approaches come from their original papers. The best worst-group accuracies under the same condition are marked in bold.
Group annotation free?
Annotation only used for tuning h-params?
Method
Waterbird
CelehA
Avg.
Worst-group 89.1+1a
Avg.
Worst-group
SUBG (Idrisii 2022
65.
GroupDRO Sagawa of al.
2019
91 91.820.4 is 92. 22
89.2
98. a
88. 88.521116
SSA Nam et al.
2022
89.02ass
92.8
89.8
No
ERM
97.3
72.6
95.6
47.2
EMIL Creager et al.
2021
96.9
78.7
91.9
83.3
Yes
CNC
Zhang ot al. 2022
90.9
88.5
89.9
88.8
JTT
89. 49.9
861881
91.3
78, 781.1
BAM
91.4+0.44
89.2
88.0+o.57
83.5
GEORGE (Sohoni al.
2020
95.7
76.2
94.8
52.4
BPA
Seo
2022
71.4
82.5
Yes
JTT Class
88.5+1.47
87.1+n.24
91.8+a.7
75.4+3.2s
BAM ClassDiff
91.4+o.3
89.1+o.s
88.4+2.32
80.1-3..
Table 2: Average and worst-group test accuracies of different approaches evaluated natural language datasets (MultiNIL and CivilComments-WIDn WILDSS, following the same conventions in Table
Group annotation free?
Annotation only
used for tuning h-params?
Method
MultiNLI
CivilComments- WILDS
Avg.
Worst-group 68.92a
Avg.
Worst-group
SUBG
Idrissi al.
2022
71.8
No
Group Sagawa el al.
2019
81.421. a0
76.62ma
87.7
1
SSA
Nam el al.
79.9
76. G6.ss
88.221.0
69.922.z
No
ERM
82.4
67.9
92.6
57.4
EIIL
Creager
79.4
70.9
90.5
67.0
Yes
CNC
Zhang et al.
2022
81.7=oa
68.9+2a
Jrr
80.0=c.4
68.1
87.2+1.s
77.7+1m0
BAM
79.6-1.11
71.2+1.0
88.3+ 0.70
79.3+2.0
GEORGE (Shoni et al.
2020
BPA Seo et al. 2022
Yes
JTT ClassDif
81.2
66.5
87.2
77.7
BAM ClassDiff
80 80.3
70.8
88.3
79.3
It is possible that the combination of CNC and BAM could lead to better results. Nevertheless, the result of BAM is promising on all other datasets, even comparable to the weakly-supervisee method SSA and the fully-supervised method Group-DRO on Waterbirds and CivilComments- W...D.
Second, if the validation group annotations are not available at all, BAM achieves the best performance on all four benchmark datasets among all the baseline methods. While BPA slightly outperforms BAM the CelebA dataset, BAM still achieves the best overall worst-group accuracy on the image classification datasets.
BAM'S improved performance in worst-group accuracy comes al the expense of a moderate drop in average accuracy. The tradeoff between average accuracy and worst-group accuracy is consistent with the observation made by Liu et al. (2021); Sagawa et al. (2019). We note that our implementation of JTT follows directly from its published code, and we obtain a much higher performance on the CivilComments. WILDS dataset. than originally reported by [iu et al. (2021

---

Published in Transactions on Machine Learning Research (03/2024)
0.10
0.20
0.05
0.15
Logit 1 (landbird)
Logit (landbird)
0.00
-0.05
0.05
-0.10
0.00
-0.15
Waterbird in water
Landbird in water
Waterbird in land
-0.05
Landbird in land
-0.20
-0.1
0.1 Logit (waterbird)
0.0
-0.2
-0.1 0.0
Logit (waterbird)
Figure 2: Distributions of the auxiliary variable w.r.t. the waterbird class (left) and landbird class (right) the training set at T == =0. We use two distinct colors to illustrate the distributions of two groups in each class. The coordinates of data sample i relative to the origin show the bias learned by the auxiliary variable.
6 Analyses and Ablations
In this section, perform detailed analyses and ablations to better understand various components of BAM, including auxiliary variables (Section 6.2 ClassDiff (Section 6.3 sensitivity of hyperparameters À and T
(Section 6.4 and the advantage of continued training in Stage 2 (Section 6.5
6.1 Additional Controlled Experiments Setup
Despite the popularity of the benchmark datasets, they have significantly different major-versus-minor class/grone groupp ratios and uncommon Icarning patterns (Idrissi al., 2022). For a more rigorous analysis of our proposed auxiliary variable and classDiff, introduce three additional controlled datasets where adjust different class/group imbalance ratios, including ColOred-MNIST, Controlled- Haatroled-Wattebirdee and CIFARIO-MNIST. See Appendix A for details.
6.2 Analysis of Auxiliary Variables
We verify the intuition behind the auxiliary variables by visualizing their learned values and comparing between majority and minority examples. Recall that we expect minority examples to rely more on auxiliary variables (see Section 4.1
6.2.1 Visualization of Auxiliary Variables
In Figure 2 we visualize the distribution of the learned auxiliary variables on each group of the original Waterbirds dataset at T 20. We have two observations. First, auxiliary variables for examples in majority and minority groups have clear distinctions. Second, auxiliary variables for majority group examples are in general closer to the origin, while those in minority groups tend to have larger (positive) logit values on the ground truth class they actually belong to and have smaller (negative) logit values on the class they do not belong to. The visualization shows that the auxiliary variables help with fitting hard-to-lear samples, which corroborates the intuition described in Section 4.1 We observe the same trend for any A E (0.5, 5, 20, 50, 70, 100} as well as any T in Stage in {20, 50, 100). This supports our intuition that "adding auxiliary variables amplifies the bias toward easy-to-learn examples" in Section 4.1

---

Published in Transactions on Machine Learning Research (03/2024)
Group
2.004
Wateebird rs Land
Wanehur
12
Wottrred Wuter
Wett rd on Lad Lanie and on Matirr aaard
0.15
Landiird so saror
2.30
3.002
3.0
2.08
02
3.00
0.2
-1.1
-100
Group
-02
Grou
-2.002
-1.0
Chee Maier drep
Miger Gropp
Minor smep
-04
-115
-3.002
Caceh Stage
0.022
0004
Epach 100 Stage
-420
1 Group
Epech Stage
Logt Q Epect Stage
Figure 3: Epochs 1 and 100 in Stage 1 on Waterbirds. Logit 0 corresponds to the prediction the water- bird class, and logit 1 corresponds to landbird. The group sizes are 1800, 200, 200, and 1800 in order.
Figure 4: Epochs and 4 in stage 1 on Colored- MNIST. Logit 0 corresponds to the prediction on class O, and logit 1 corresponds to class 1.
6.2.2 Visualization as Training Progresses
To further explore if the auxiliary variables keep such trends in different scenarios, we take a deeper look into the training progression of the auxiliary variables in our controlled experiments, where we set a variety of different imbalance sizes/ratios. In Figures 3 and 4 scatter plots show how the values of auxiliary variables change as Stage 1 proceeds. Carefully controlling all other parameters and hyperparameters, clear patterns on Controlled-Waterbirdde and Colored-MNIST.: First, as T grows, the logits become larger and increasingly influence the model predictions. Second, minority and majority groups are more separated from each other as training progresses, resulting in the bias amplification effect. Appendix D illustrates the values of at more intermediate epochs T and clearly shows the trends.
observe
6.3 Results of Class
Worss Group Val Accuracy Abxolute Class thrence
Wers: Group Val Accuracy Absolute Dawx fffeencce
Worst-Gruup Val Accuracy Abvuliee Clecs Diference
GonstGroup Val Accuracy
Abenute Class Dificence
0.4
0.4
0.4
0.2
0.2
0.2
10
20
0.0g
10
0.0
0.
Num Epocit
Num Epachs
Eoodis Stage
Epochs
Figure 5: Relation between absolute valdiation class difference and worst-group validation accuracy in Stage 2 on Waterbirds, CelebA, CivilComments- WILDS, and MultiNLI. It can be observed that minimizing absolute validation class difference roughly equivalent to maximizing worst-group accuracy in the validation set. Each dataset uses the same hyperparameters as employed in Table [] Each line represents the value averaged over 3 different seeds and the shade represents the standard deviation.
We perform extensive experiments on the four benchmark datasets, as well as on our controlled datasets by altering class and group imbalance ratios. We find that ClassDiff works well on every single experiment, which shows the effectiveness and robustness of ClassDiff across different scttings.
World Benchmark Datasets Figure plots the trend of the absolute class difference and the worst- group accuracy on the validation set in Stage 2 for the Waterbirds, CclebA, MultiNLI, and CiviComments. Clearly, there is a strong inverse relationship between the absolute class difference and worst-group accuracy in the validation set on all four datasets, which justifies the use of class difference as a stopping criterion.
Controlled Datasets On our controlled datasets (COIS,, CIFARIO-MNIST, and Controlled- Waterbirds), we vary the dataset sizes, class size ratios, and group size ratios. Tn Appendix P we show
9

---

Published in Transactions on Machine Learning Research (03/2024)
the performance of ClassDiff in several randomly selected settings (we explain our procedure of selecting settings that do not involve cherry-picking). In every single selling we have tested, we observe similar negative correlation as in Figure 5 These findings suggest that ClassDiff could be generally effective when no group annotation is available, as it robust across different datascts, varying dataset sizes, and imbalancod characteristics.
6.4 Sensitivity of Hyperparameters
We find that BAM is robust to the choice X, as Table 8 below presents the best worst-grous grost accuracies for wide range of X on Waterbirds. Clearly, BAM demonstrates resilience against the choice of A. We also find that there noticeable performance drop when 0 compared with 0, further demonstrating its efficacy.
Table Sensitivity of
À
0
0.5
10
30
35
40
50
70
100
Test acc
87.1
89.9
89.8
89.7
90.2
89.5
89.5
88.8
88.6
Table 4: Robustness of worst group test accuracy with different T's
A
20
20
30
30
40
50
50
70
70
T
80
100
120
80
100
120
80
100
120
80
100
120
80
100
120
80
100
120
80
100
120
Test ace
82.2
75.2
68.9
83.5
77.4
75.7
84.7
81.8
77.2
86.0
86.9
81.8
87.1
86.7
85.3
88.6
88.2
86.9
88.8
58.8
88.5
Furthermore, using larger À can eliminate the need to carefully tune the epoch number T in Stage 1. This resolves major drawback of previous methods such as JTT, whose performance is sensitive with respect to the choice of T. Table g shows that even trained until full convergence in Stage 1, an appropriate choice of X can still guarantee a competitive performance. addition, as f increases, there are clear performance boosts with larger Ts'. This strong correlation shows the effect of μ.
Table 5: Sensitivity of μ
P Test acc
40
60
70
80
90
100
110
120
130
140
150
160
170
180
80.1
85.8
83.8
85.6
87.6
88.8
88.7
88.9
88.9
89.6
89.0
89.7
89.2
89.0
89.2
90.1
88.4
Finally, Table very competitive performance. This result further illustrates that BAM is not very sensitive to the choice of hyperparameters..
5 examine the sensitivity of μ. We find that a large range of μ (70-180) can guarantee a
6.5 Ablation Studies on One-M vs. Two-M
We conduct ablation studies the Waterbirds dataset to test the effectiveness of our proposed method of continued training in Stage 2. For consistent terminology, Stage and continues training the same model in Stage 2 as One-M. We define the approach that trains a separate ERM model in Stage 2 as Two-M. For a fair comparison, we employ the same hyperparameter settings throughout this section. We use the same stopping criterion here (highest worst-group validation accuracy). We tune over T (10, 15, 20, 25, 30, 40, 50, 60, 80, 100, 120) and l = (50, 100, 140) for each setting, and fix 50. Each experiment is replicated thrice with varying soeds counteract randomness, subsequently averaging results.
define the approach that loads the model from
Figure t compares the performance between One-M and Two-M over wide range of Stage epochs T. Notably, One-M and Two-M share the same error sets in cach T. The result suggests that One-M BAM outperforms Two-M BAM in every single Stage 1 epoch T, further verifying our intuition that the biased knowledge accumulated from Stage helps with improving the final robustness. Additionally, as explained in Scction 6.4 and also shown here Figure D BAM demonstrates a stable performance, regardless of the choice of T's. By stark contrast, JTT's performance falters with escalating T, as documented in Figure of Liu al. 20011
10

---

Published in Transactions on Machine Learning Research (03/2024)
BAN on Waterbird
0.9
Wore: Accuracy
0.88
0.7
0.6
Ore-M
0.3
0
20
40
00
50
100 120
Epochs T in Stage
Figure 6: Comparison of BAM'S performance between training One-M M and Two-M. We find that One-M consistently achieves better vorst-group test accuracies than Two-M for BAM over all hyperparameter combinations and BAM One-M demonstrates a stable performance regardless of the choice of T.
7 Conclusion
In this work, we introducod BAM, a method that can effectively improve the worst-group accuracy across different NLP and CV benchmarks. Further experiments suggest the elfectiveness of the bias amplification scheme of the auxiliary variable and ClassDiff through comprehensive analysis and under various experiment. settings. Future work can further leverage the idea of ВAМ and apply the auxiliary variables to other applications, such as continual learning and curriculum learning. Conducting thcoretical analysis of the bias amplification ion scheme can also be of great interest.
8 Ethical Statement
BAM is designed to effectively enhance worst-group robustness, and it represents step towards addressing the marginalization under-reprosented groups in the training data. We believe that a training procedure without learning spurious correlation essential for the responsible development. deep learning tools and models. In addition, while we evaluate BAM on four well reoognized benchmark datasets, we acknowledge that these may not fully encapsulate the complexity and biases present in real-world data. This limitation underscores the need for more comprehensive and diverse datasets in future works. Furthermore, the bias-amplified model trained in Stage should not be deployed for real-world meployment and the model must be continued trained in Stage 2 to correct the bias.
11

---

Published in Transactions on Machine Learning Research (03/2024)
References
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arxiv preprint wrXii:1977.02893 2019.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456-473, 2018.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference, pp. 491-500, 2019.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In International Conference on Machine Learning, pp. 872-881. PMLR, 2019.
Kaidi Cao, Yining Chen, Junwei Lu, Nikos Arechiga, Adrien Gaidon, and Tengyu Ma. Heteroskedastic and imbalanced deep learning with adaptive regularization. arXiv preprint arXiv:2006.157) 2020.
Dan C Cireşan; Ueli Meier, Jonathan Masci, Luca M Gambardella, and Jürgen Schmidhuber. High-performance neural networks for visual object classification. arXiv preprint arXiv: 022. 0183, 2011.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based
methods for avoiding known dataset biases. arXiv preprint (tte::999000688. 2019.
Elliot Creager, Jörn-Heurik Jacobsen, and Richard Zemel. Enviroument inference for invariant learning. International Conference on Machine Learning, pp. 2189 2200. PMLR, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding. Tn Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Humon Language Technologies, Volume (Long and Short. Papers), pp. 4171 4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 11186533v1/1199 1423. URL https:///aclaattoooogg 1ttp:/ooooooy..o/11-11-1111 org/N19-1423
John Duchi and Hongseok Namkoong. Variance- based regularization with convex objectives. The Journal of Machine Learning Research, 20(1):2450-251) 2204, 2019.
John C Duchi, Tatsunori Hashimoto, and Hongseok Namkoong. Distributionally robust losses against mixture covariate shifts. Under review, 2, 2019.
Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning deep neural networks. Nature Machine Intelligence, 2(11):665-673, 2020. doi: 10.1038/ +422556-000-02257-z
Karan Goel, Albert Gu, Yixuan Li, and Christopher Ré. Model patching: Closing the subgroup performance gap with data angmentation. arXiv preprint arXiv:2088. 06775, 2020.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 2 (Short Papers), pp. 107-112, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.
Haibo He and Edwardo A. Garcia. Learning from imbalanced data. IEEE Transoctions on Knowledge and Data Engineering, 21()::1263-128; 2009. doi: httpps///ccaattooooooggg////--
URL https://aclaathhollgg
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pallern Recogrition (CVPR), pp. 770 778, 2016. doi: 10.1109/CVPP 201..000..00
Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training on noisily labeled data with generalization guarantee. Xiv: Learning, 2020.
12

---

Published in Transactions on Machine Learning Research (03/2024)
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference on computer vision and pallern recognilion, pp- 5375 5384, 2016.
Inwoo Hwang, Sangjun Lee, Yunhyeok Kwak, Seong Joon Oh, Damien Teney, Jin Hwa Kim, and Byoung-Tas Tak Zhang. Selecmix: Debiased learning by contradicting-pin sampling. Advances in Neural Information Processing Systems, 35:14345 14357, 2022.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-accuracy. In Conference on Causal Learning and Reasoning, pp. 336-351. PMLR, 2022.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Myeongho Jeon, Dackyung Kim, Woochul Lee, Myungjoo Kang, and Joonscok Lee. A conservative approach for unbiased learning unknown biases. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 16752-16760, 2022.
Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Dala, 6(1):1 54, 2019.
Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. Cost- sensitive learning of deep feature representations from imbalanced data. IEEE transactions on newral networks and learning systems, 29(8):373 3587, 2017.
Fereshte Khani, Aditi Raghunathan, and Percy Liang. Maximum weighted loss discrepancy. ar preprint arXiv:1906.03337 2019.
Naycong Kim, Schyun Hwang, Sungsoo Ahn, Jacsik Park, and Suha Kwak. Learning debiasod classifier with biased commillee. arXiv preprinl Xix:22206. 10843, 2022.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training sufficient for robustness to spurious correlations. arXiv preprint 1tti::2224.09933 2022.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma. Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. In Marina Meila and Tong Zhang (cds.), Proceedings of the 38th International Conference on Machine Leurning, volume 139 of Proceedings of Machine Learning Research, pp.
PMLR, 18 24 Jul 2021.
URL https://proceeddinga mlr, wwwppreeeevee.ee. wttsss//v3///oohhaa..m html
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yoonho Loc, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from undcrspocified data. arXiv preprint Xi::22200 18,18, 2022.
Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary compulation and artificial life research communities. Arlificial life, 26(2):274-33) 272--666,66-2 2020.
Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization. Advances in Neural Information Processing Systems, 33:8847-8888 8860, 2020.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6781-6792. PMLR, 18-24 Jul 2021. URL http:::/pproeeeengggss mlr. mlr. wttppp/eees//iil22//pp html
13

---

Published in Transactions on Machine Learning Research (03/2024)
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. 2015 IEEE International Conference on Computer Vision (TCCV), pp. 3730 1730--738, 2015.
Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, et al. Wilds: A benchmark of widl distribution shifts. aiv preprint orXiu:2012.07422 2020.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahm, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Newral Information Processing Systems, 14-20633-00685 2020.
Junhyun Nam, Jachyung Kim, Jacho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. arXiv preprinl Xiv:22200 0200000 2022.
Yonatan Oren, Shiori Sagawa, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust language modeling. arXiv preprint Xiw::999.000000 2019.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaune Lajoie. Gradient starvation: A learning proclivity in neural networks. Advonces in Neural Information Processing Systems, 14:1256- 1272, 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Perey Liang, Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint 1rXii:1911.0879/ 2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameteri- zation exacerbates spurious correlations. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, html
pp. 8346 8356. PMLR, 13- 18 Jul 2020. URL https::/prooceedings mlr. attess//v19//agga/ae
Ramprasaath R Sclvaraju, Michael Cogswell, Abhishck Das, Ramakrishna Vodantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp. 618 118-626,, 626, 2017.
Sconguk Sco, Joon- Lee, and Bohyung Han. Unsupervised learning of debiased representations with pseudo-attributes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16742-16755 16751, 2022.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. weig-wwightt-et:: Learning an explicit mapping for sample weighting, Advances in newral information processing systems, 32, 2019.
Nimit Sohoni, Jared Dunnmon, Geolfrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine- grained robustness in coarse-grr grarind classification problems. Advances in Neural Information Processing Systems, 13:19239-1935,, 2020.
Saeid A Taghanaki, Kristy Choi, Amir Hosein Khasahmadi, and Anirudh Goyal. Robust representation learning via perceptual similarity metrics. In International Conference on Machine Learning, pp. 10043 10053. PMLR, 2021.
Fadi Thabtah, Suhel Hammoud, Firuz Kamalov, and Amanda Gonsalves. Data imbalance in classification: Experimental evaluation. Information Sciences, 513:429 441, 2020.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Tryna Gurevych. Towards debiasing nlu models from unknown biases. arXie preprint arXio:2009.12303 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge J. Belongic. The caltech-ucsd birds-200-2011 dataset. 2011.
14

---

Published in Transactions on Machine Learning Research (03/2024)
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapler of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1112-1122, New Orlcans, Louisiana, June 2018. Association for Computational Linguistics. doi: URL htttp:://allanthol.a 1ttp://cchooooooooo///1111000 org/N18-1101
Yadollah Yaghoobzadehı Soroush Mehri, Remi Tachet, Timothy J Hazen, and Alessandro Sordoni. Increasing robustness to spurious correlations using forgettable examples. arXiv preprint arXiv:1911 0.86611,1 2019.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. arXiv preprint arXiv:2201.0020) 2022.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.0044 2017.
Jingzhao Zhang, Aditya Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. arXiv preprint orXiv:2010. 1 12230, 2020.
Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Ré. Corroct-n-contrast. A contrastive approach for improving robustness to spurious correlations. orXie preprinl ar urXiv:2233001117, 2022.
15

---

Published in Transactions on Machine Learning Research (03/2024)
A
Dataset details
Waterbirds is image dataset consisting of two types of birds different backgrounds. It contains images of birds directly cut and pasted real-life images of different landscapes. We aim at classifying } (waterbird, landbird], which is spuriously correlated with the background A = [water, land]. The train/ /eeivvvaaaaiiio/eeees splits is followed from Sagawa et al. (2019).
Waterbirds (Wah al. 2011: Sagawa et al. 2019h:
CelebA [iu et al. 2015): CelobA is an image datasct consisting of celebritics with two types of hair color and different genders. we aim al classifying celebrities' hair color ) (blond, not blond), which is spuriously correlated with the gender
{male, female). The train saain/valiiatioo/teer splits followed from Sagawa
ct al. 2019).
MultiNLI (Williams et al. 2018 MultiNLI is natural language processing dataset, where each sample consists of two sentences and the second sentence is entailed by, neutral with, or contradicts the first sentence, with the presence or absence of negation words. We aim at classifying the sentence relationship y = (entailment, neutral, contradiction), which is spuriously correlated with the negation words (negation, no negation). The train/ vaai/vllaaiioo/tesstt splits is followed from Sagawa et al. (2019).
natural language processing dataset consisting of online comment that is toxic non-toxic, with or without the mentions of certain demographic identities (male, female, White, Black, LGBTQ, Muslim, Christian, and other religions). We aim at classifying whether the sentence is toxic y = {toxic, non-toxic), which is spuriously correlated with the mentions of (demographic) identities A = (identity, no identity). The train/ /aaa/vvaaaatotteeeeeeeeeee test splits followed from Koh et al. 2021
CivilComments- Weroommentts-WWDDS [Borkan al. 2019 Koh et al., 2021 ): CivilCommetts-
Red
Green
Water
Land
4
J
Class 0
Waterbird
Car
6
8
Class
Landbird
Truck
Figure 7: Examples of each group in Colored-MNIST, Controlled-Watterbinee and CIFARIO-MNIST.
MNIISEd-MNIST Arrovsky et al. 2019 Cireşan et al.
2011 ) Colored-MNIST is image dataset that
colors each grayscale MNIST images either green or red and make it spuriously correlates with the digit values. We aim at classifying binary class labels y = {class (digits 0-4), class 1 (digits 5-9)} without relying on the color AA = (red, green). Examples are shown in Figure 7 We shuffle the original data and generate dataset splits with train/vall aaaii/vvluaatinn/omommmm stetiittt/ttttttee lest sizes = 0.7/0.15/0.155 In Figure 4 we set the total dataset size to be 10000, class imbalance ratio 1.0, and the group imbalance ratio within each class to be 0.1.
Controlled- Waterbirds (Wah et al. 2011 Sagawa et al. 2019:: :: We manually construct this dataset from the original Waterbirds benchmark. We follow the previous definition that sets Waterrbird- Water sizes = 0.7/0.115/0.000 Examples are shown in Figure 7 To show how the ClassDiff performs in various settings in Appendix E we consider the following three circumstannces: (1) We vary the group imbalance ratios, while the two classes are balanced. Due to the original datasct size limits, we fix the size of two majority groups to 1800, and vary the minority group sizes in (200, 400, 600, 800}. (2) The two classes are imbalanced. We fix sizes of the two groups in the smaller class 200 and 1 1400, and choose the class imbalance ratio waterbird landbird
and Landbird-Land as majority groups. We regenerate the datasct splits as train ioain/valudation/tee
16

---

Published in Transactions on Machine Learning Research (03/2024)
in (1.0, (3) Both group and class sizes are imbalanced. We consider random setting of four group sizes, where Snherbird-Water Waterbird-Land Hlandbird-Watte Landbird-Land = 1800 600 200 3600.
CIFARIO-MNIST Krizhevsky et al.) 2009 Cireşan et al. 2011 by concatenating MNIST images onto CIFAR-10 images. We aim at classifying binary class labels y = {car, truck), which is spuriously correlated with the MNIST digits A = {0. 1}. We shuffle the original data and generate dataset splits with train/valli valudation//ens sizes 0.7/0.15/0.15. Examples are shown in Figure 7 To show how the ClassDiff performs in various settings in Appendix E consider the following two circumstances: (1) We vary the group imbalance ratios, while the two classes are balanced. Due to the original dataset size limits, we fix the size of two majority groups to 4000, and vary the minority group sizes in {400, 600, 800). (2) The two classes are imbalanced. We fix sizes of the two groups in the larger class as 800 and 4000, and choose the class imbalance ratio car truck in (4/3, 2/2, 4/13.
We manually construct this dataset
B Training details
In this section, we provide details about the model selection and the hyperparameter tuning for different datasets. As claimed in the main text, in order to make a fair comparison with previous methods, we use the same pretrained models. Namely, we use ResNet-50 pretrained from Image-net weights for Waterbirds and CelebA, and pretrained BERT for MultiNLI and CivilComments. We use the Pytorch implementation for 5e and the HuggingFace implementation for BERT. We tune BAM and JTT according to class difference and worst-group accuracies in the validation set in Stage 2.
Table 6: Hyperparameters tuned over 4 datasets.
Dataset
Auxiliary coefficient (A)
#Epochs in Stage 1 (2)
Upweight factor (n)
Waterbirds
{ 0.5, 5, 50}
{10, 15, 20}
{50, 100, 140)
CelebA
{0.5, 5, 50]
{1, 2}
{50, 70, 100]
MultiNLI
{0.5, 5, 50)
{1, 2}
{4, 5, 6}
CivilComments
{0.5, 5, 50
{1, 2)
{4, 5, 6)
In general, our setting follows closely from Liu et al. (0021 with some minor discrepancies. For the major hyperparameters, We tuned over X = {0.5.5. 50), T [1,2, 10, 15, 60) and μ. {4, 6, 50, 70, 100, 140] for BAM. We note that BAM is fairly insensitive with the choice of X as mentioned in Section 6.4| We tune over T = 21, and f = (4,5, 6, 50, 70, 100, 140} for JIT for fair comparisons. We tuned the major hyperparameters according to Table 6 More details are provided below:
Waterbirds We use the learning rate 1e-5 and batch size 64 for two stages of training. We use the stochastic gradient descent (SGD) optimizer with momentum 0.9 throughout the training process. We use & regularization for Stage 2 rebalanced training. We apply the same above sctting for both JTT and BAM. Notably, illustrated by Figure 5 when tuned for the minimun absolute validation class difference, the curve may fluctuate abnormally after the first 30 epochs and it is clear that the model is not learning anything useful. We tackle this problem by smoothing out abrupt changes in class difference (neglect the result if the difference between consecutive class differences is greater than 10%). The best result for JTT occurs when T = 60 and μ = 140. The best. result. for BAM occurs when T = 20 and = 140. We train for a total of 360 epochs
CelebA We use the learning rate le-5 and batch size 128 for two stages of training. We use the stochastic gradient descent (SGD) optimizer with momentum 0.9 throughout the training process. We use lz regulariza- tion 0.1 for Stage 2 rebalanced training. We apply the same above setting for both JrT and BAM. The best result for JTT occurs when T = 1 and μ = 70. The best result for BAM occurs when T and μ = 50. We train for total of 60 epochs.
17

---

Published in Transactions on Machine Learning Research (03/2024)
MultiNLI We use batch size 32 for two stages of training. We apply initial learning rate of 2e-5 for Stage and le-5 for Stage 2. We use the SGD optimizer without clipping for Stage 1 and the AdamW optimizer with clipping for Stage 2. We use l regularization 0.1 for Stage 2 rebalanced training. We apply the same above setting for both JTT and BAM. The best result for JTT occurs when T = 2 and μ = 4. The best result for BAM occurs when 7 and μ = 6. We train for a lotal of 10 epochs.
We use batch size 16 for two stages of training. We apply an initial learning rate of 2e-5 for Stage and 1e-5 for Stage 2. We the SGD optimizer without clipping for Stage and the AdamW optimizer with clipping for Stage 2. We
CivilComments-Wee
l2 regularization 0.01 for Stage 2 rebalanced training. We apply the same above setting for both JIT and BAM. The best result for JTT occurs when T = 1 and Jt = 4. The best result for BAM occurs when T = 1 and l = 4. We train for total of 10 epochs.
Colored-MNIST We use batch size 64 and learning rate le-4 for two stages of training. We use the stochastic gradient descent (SGD) optimizer with momentum 0.9 throughout the training process. We use simple convolution network with two convolutional layers followed by fully connected layer. As toy datasct, the datasct is only used for visualizations. In Figure
20.
Waterbirds Contrblled-Wateebbrrds We fix À = 50 and T = 100. For circumstances (1), we test different E {70, 100, 120). For circumstance (2), we test different {10, 50, 70, 100, 120, 150). For ciscumstance (3), we test different j E {10, 50, 70, 100, 120, 150}. All other hyperparameters and scttings follow what is used in the original Waterbirds dataset.
CIFAR1O-MNIST We use the learning rate 1e-5 and batch size 64 for two stages of training. We use the stochastic gradient descent (SGD) optimizer with momentum 0.9 throughout the training process. We use & regularization 1 for Stage 2 rebalanced training. We pretrained ResNet-18 as the training model. We fix À = 50, and test different T E (5, 10, 25} and μ E {5, 10, 20,50 50} for circunstances (1) and (2).
C Proof of Claim 4.1
Proof of Claim 4.1
Suppose we have C classes, and each class i contains two groups i(a) and i(b). We have C(b)) (to,
that all group accuracies (i.e. 1(a), 1(b), 2(),
are within of each other, then
CifssDiff
Σ
AAce(class i) Acc(class j)]
(2) Isisise
Σ Isiciso
lagAcc(ia) + +(1 a:) Acc(ii) - Acc(ja) - (1 ac)Ac((jb))
(c)
Σ
E=E,
Isisic
where refers to the group ratio of group a in class i.
D Further Analysis on Auxiliary Variables
Figure and Figure g show the auxiliary variable logits with different T's in Stage 1, and clearly illustrate the trends of changes.
E Class Difference Plots and Explanations
We conducted additional experiments on two datasets to verify the robustness of BAM by changing the relative group sizes and class sizes: listed in Appendix A and following the training details in Appendix B we conduct the experiments, and randomly show some of the ClassDiff plots in Figure 13
and CIFARIO-MNIST. Considering the circumstances
18

---

Published in Transactions on Machine Learning Research (03/2024)
0.3
3.4
0.2
0.1
02
0.00
Lagkt
0.0
0,5
-0.05
Group
Class Hajor Group Class o Minor croup
Class
-0.15
Clas 1 Majo Group
Cuss Majer Droup
Niner
-0.4
a Winer
2 Mager Group
Class
Class Mauer Group
-0.00
Clas 1 Minor Group
Cuss Ninor Group
Class 1 Minor Group
Class Miner Group
6.0 Spech 1 Siage
0.1
Epoch Stage
-50
Epoch Stage
10072
Loglt Eoorh Stage
Figure 8: Epoch 1 to 4 in stage 1 on a toy dataset, Colored-MINIST. Logit corresponds to the prediction on class 0, and logit corresponds to class
Group
Group
0.004
Watarbird on Water Watrotlddinn on
0.00
Watartard Land
Watarbird Water
Landord Wuuer
Logit
3.032
0.02
Legit
0.00
0.00
2,000
Group
Untertind on Water
Water ind on Land
-1.04
-0.00
Landted
Lavittid Water
-1.00
-62
5.000 Epool Stage
2.24
-0.02
5.00
0.02
0.04
-0.25
0,00
2.03
5.0
02
Logit
Stage
Epoch528 Stage
Spocs 150 Stage
Figure 9: Epoch to 100 in Stage on Waterbirds. Logit corresponds to the prediction on the waterbird class, and logit corresponds landbird. The group sizes are 1800, 200, 200, 1800 in order.
To avoid charry picking, we first classify each experiment into one of the settings in Appendix A The number of experiments are 12, 18, 6, 36, 36 for each scenario in order. Then, we sort the following parameters and hyperparameters by the order of importance: dataset parameter values shown each setting in the dataset details > > T > μ.. Finally, we set random seed to 0 and use the function NumPy random. choice to select. independent runs for each circumstance.
The following figures illustrate that both ClassDiff and BAM robust to varying group and class ratios on the manually generated datasets. In addition, we observed every single experiment of both these two new established datasets that the class difference is negatively correlated with highest robust test accuracy, regardless of their group sizes and class sizes.
F Smaller Validation Set
Table 7 shows the worst-group accuracies on Waterbirds with varying sizes of group-annolated validation set.
Table 7: Worst-group accuracy on Waterbirds with varying size of validation set with group annotations. BAM maintains high worst-group test accuracies even when tuned with very few nunbers of group-anotaaled validation set. However, we note that with a fewer size of validation set with group annotations, the performance is actually worse than when tuned for a full-size validation set without any group annotations.
Size of Annotated Validation Set.
100%
20%
10%
5%
Worst-Group A%c.
89.8
89.1
88.4
86.2
19

---

Published in Transactions on Machine Learning Research (03/2024)
14
Ner decorer
Figure 10: 5 random experiments of the circumstance (1) on the Controlled-Wateee Waterbirds dataset.
344
40000 Mo
wwwctppote nangge
suwcctepdee Kings2
Figure 11: 5 random experiments of the circumstance (2) on the Controllad-Waterbirrds dataset.
1.4
Nawetippte nangge
Figure 12: 5 random experiments of the circumstance (3) on the Controlled- Waterbirds dataset.
ale Care Dilenise
ABFADA CIE OPHirie
2.44
Sanaltpots Istaged
Figure 13: 5 random experiments the CIFARIO-MNIST dataset.
20

---

Published in Transactions on Machine Learning Research (03/2024)
G
Supplementary Figures
0.20
Group
Group
0.05
Waterbird in water
Landbird in water
0.15
Logit 1 (landbird)
Waterbird in land
Logit 1 (landbird)
Landbird in land
0.00
0.10
-0.05
0.05
-0.10
0.00
-0.15
-0.05
-0.1
0.0
0.1
-02
-0.1
0.0
Logit 0 (waterbird)
Logit (waterbird)
Figure 14: Corresponding KDE plot of Figure
2
21


Reviews:
[{"review_id": "PZ1ks9fTcr", "reviewer": "Reviewer_FBUU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1. The whole paper is well-written and easy to follow.\nS2. The group-annotation-free approach for spurious correlations lacks further work to achieve close performance to the group-annotation-dependent approach and it is of great significance in practice as well.\nS3. The idea of bias amplification and the introduction of learnable auxiliary variables are absorbing.\nS4. The experiments of the ablation study are well organized. The results are also well presented.\n\nW1. Stage 1 aims to amplify the bias from spurious correlations. However, this paper does not prove that stage 1 indeed amplifies such bias in experiments. More accurate evaluations can be introduced to assess whether the bias is indeed amplified.\nW2. Using the simple stopping criterion classdiff cannot be reliable through an intuitive thought. For example, if the bias amplification leads to a heavy drop in the performance of the major group and such performance gets close to the performance of the minor group. The classdiff between the major and minor groups can also be small.\nW3. The author mentioned two related works that improve group robustness without any group annotations, i.e. [1] and [2]. \nIn experiments, this paper only compares itself with [1] and does not compare with [2] or other works that do not require group annotations. In addition, this paper does not provide the selection criteria for the comparison approaches.\nW4. Samely in the comparison results, some results are provided std while some only contain the mean values or do not provide exact results. This paper does not mention the reasons for the lack of results and std.\n\n[1] Sohoni, et al. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. NeuIPS 2020.\n\n[2] Seo, et al. Unsupervised learning of debiased representations with pseudo-attributes. CVPR 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. Target on the spurious correlations, this paper proposes a group-annotation-free approach. This paper uses a two-stage method, firstly amplifying the bias from spurious correlations and then holding rebalanced training.\n2. In the Bias Amplification stage, this paper introduces learnable auxiliary variable to realize the Amplification.\n3. This paper also proposes a stopping criterion classdiff to select the best model during the rebalanced training", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The term of “Annotation free“ should be specific as “Group annotation free”\n2. The Bias Amplification in stage 1 should be proved by a more intuitionistic approach.\n3. The blank results and missing std in Table 1 and Table 2 should be filled. \n4. The comparison method [2] and more group-annotation-free approaches (if possible) should also be added. \n\n[1] Sohoni, et al. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. NeuIPS 2020.\n\n[2] Seo, et al. Unsupervised learning of debiased representations with pseudo-attributes. CVPR 2022.", "broader_impact_concerns": "No ethical issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eftoTiWdHu", "reviewer": "Reviewer_eZXj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed method tackles an important challenge.\n\n2. Experiments are performed on both CV and NLP datasets and demonstrate a consistent advantage in all benchmarks.\n\n3. The implementation is provided and the results seem reproducible. \n\nWeaknesses:\n\n1. The theoretical contribution of the paper is weak and quite trivial.\n\n2. Comparison is mostly based on performance which may not be enough to judge model bias. In the absence of such a comparison, it is not easy to conclude that the proposed method is helpful in reducing bias.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces BAM which is a two-stage training algorithm to improve neural network predictions for minority subgroups when group annotations are limited or missing. In the first stage, BAM uses a bias amplification mechanism to consider biases in the data in model training. In the second stage, called rebalanced training, BAM focuses on misclassified samples from the first stage for further improvement. BAM competes well with existing methods in addressing spurious correlations in computer vision and natural language processing benchmarks. The study also identifies a stopping criterion that reduces the need for group annotations without affecting the accuracy of disadvantaged subgroups. Extensive analyses confirm BAM's effectiveness in addressing bias and improving model accuracy when group annotations are scarce or unavailable.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In Table 3, it seems that the performance is not sensitive with respect to lambda. I would like to see what happens if we set lambda = 0. \n\n2. In Table 4, please include smaller values for lambda, starting from lambda = 0.\n\n3. Since the focus of this paper is on bias, I think it is appropriate that bias metrics such as demographic parity are also reported for comparison.\n\n4. An experiment about the mu parameter can be helpful.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ek9Mtr0IId", "reviewer": "Reviewer_PXZp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: 1. The novel method Bam improves worst-group accuracy in machine learning models and does not require group annotations for training data.\n2. The article provides comprehensive experimental results on various benchmark datasets.\n3. The use of Grad-CAM visualization helps illustrate the impact of bias amplification and rebalanced training stages.\n\nWeaknesses:1.The theoretical basis of the bias amplification scheme can be further explored and analyzed.\n2. The BAM model is under-motivated and the article should provide more details about the experimental setup and a detailed analysis of the model to present a more argumentative summary.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The article introduces a method called Bias Amplification (Bam) that aims to improve worst-group accuracy in machine learning models. The method consists of two stages: bias amplification and rebalanced training. The method can remove the need for group annotations by using a simple stopping criterion based on minimum class accuracy difference.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tWhat are the theoretical foundations behind the Bias Amplification (Bam) method? Could you explain the process of bias amplification and rebalanced training in more detail?\n\n2.\tHow does the Bam method address the issue of covariate shifts and distributional differences between the source and target data?\n\n3.\tHow does the Bam method handle cases where the spurious attributes are not known or are difficult to identify?\n\n4.\tCould you explain the process of upweighting misclassified examples in the rebalanced training stage?\n\n5.\tHow does the Bam method handle cases where the training data is imbalanced in terms of both group and class sizes?\n\n6.\tThe paper starts with the premise that amplifying bias in the first model can improve final group robustness. How did you determine that this approach is more effective than directly trying to reduce the bias?\n7.\tHow does the proposed Bam algorithm differ significantly from other two-stage algorithms or existing methods that aim at bias mitigation?\n\n8.\tHow does Bam ensure that it isn't simply overfitting to the \"low class accuracy difference\" when it's used as a stopping criterion in the absence of group annotations?\n\n9.\tFor the rebalanced training in Stage 2, how does the choice of upweighting factor µ impact the model? Is there a risk of over-emphasizing the error set from Stage 1?\n\n10.\tPaper mentioned a tradeoff between average accuracy and worst-group accuracy. Could you provide a more detailed analysis on why this tradeoff occurs and its implications?\n\n11.\tWhat are the benefits of introducing controlled datasets like Colored-MNIST, Controlled-Waterbirds, and CIFAR10-MNIST over the standard benchmarks?\n\n12.\tWhy is Bam's performance, unlike Jtt's, unaffected by full convergence in Stage 1?\n\n13.\tHow does Bam's computational cost and training time compare with the other mentioned methods?\n\n14.\tEffect of Imbalance: How does the class or group imbalance in datasets affect the performance of Bam?", "broader_impact_concerns": "1.\tDo the datasets used for the experiment (e.g., Waterbirds, CelebA, MultiNLI, and CivilComments-WILDS) truly reflect real-world distributions and biases? What are the potential risks if the model is applied in the real world?\n\n2.\tThe experiment mentions an improvement in the accuracy of \"worst-group\". Have the authors considered whether such improvements might exacerbate the marginalization of niche groups in some cases?\n\n3.\tBam has been compared to CNC methods using contrast learning in some experiments. Considering that contrast learning may amplify the importance of certain features, how might this affect the model's performance in terms of diversity or inclusion?\n4.\tAre there potential risks for the choice of hyperparameters such as λ and T, such as overfitting or instability due to over-tuning in real-world applications?\n\n5.\tBam uses auxiliary variables to aid in training. Are these auxiliary variables likely to lead to some unforeseen bias or misrepresentation in practical applications?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper underwent a through reviewing process. The revisions made by the authors generally addressed the reviewers' concerns, and finally, all the three reviewers recommended to (weakly) accept this submission. The AE concurs with the reviewers' recommendations.
Decision: Accept (poster)