Mitigating Spurious Correlations via Disagreement Probability
Authors: Hyeonggeun Han
Sehwan Kim
Hyungjun Joo
Sangwoo Hong
Jungwoo Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: Debiasing
Spurious correlation
Group robustness
Primary area: fairness
TL;DR: We propose a novel training objective and method designed to promote consistent model performance irrespective of spurious correlations.

Abstract: Models trained with empirical risk minimization (ERM) are prone to be biased towards spurious correlations between target labels and bias attributes, which leads to poor performance on data groups lacking spurious correlations. It is particularly challenging to address this problem when access to bias labels is not permitted. To mitigate the effect of spurious correlations without bias labels, we first introduce a novel training objective designed to robustly enhance model performance across all data samples, irrespective of the presence of spurious correlations. From this objective, we then derive a debiasing method, Disagreement Probability based Resampling for debiasing (DPR), which does not require bias labels. DPR leverages the disagreement between the target label and the prediction of a biased model to identify bias-conflicting samples—those without spurious correlations—and upsamples them according to the disagreement probability. Empirical evaluations on multiple benchmarks demonstrate that DPR achieves state-of-the-art performance over existing baselines that do not use bias labels. Furthermore, we provide a theoretical analysis that details how DPR reduces dependency on spurious correlations.

Mitigating Spurious Correlations via Disagreement Probability
Hyeonggeun Han Sehwan Kim Hyungjun Joo
Sangwoo Hong! Jungwoo L2c-.2.3 12,3+
EECE & "NextQuantum, Seoul National University Hoddoo Al Labs fhygnhan, sehwankim, joohj911, tkddn0606, junglee) acc .rr
Abstract
Models trained with empirical risk minimization (ERM) are prone to be biased towards spurious correlations between target labels and bias attributes, which leads to poor performance on data groups lacking spurious correlations. It is particularly challenging to address this problem when access to bias labels is not permitted. To mitigate the effect of spurious correlations without bias labels, we first introduce a novel training objective designed robustly enhance model performance across all data samples, irrespective of the presence of spurious correlations. From this objective, we then derive debiasing method, Disagreement Probability based Re- sampling for debiasing (DPR), which does not require bias labels. DPR leverages the disagreement between the target label and the prediction of biased model identify bins-connlicting samples those without spurious correlations and up- samples them according to the disagreement probability. Empirical evaluations on multiple benchmarks demonstrate that DPR achieves Maate-of-the-ar performance over existing baselines that do not use bias labels. Furthermore, we provide theo- retical analysis that details how DPR reduces dependency spurious correlations.
Introduction
In the realm of machine learning, many classification models em- ploy Empirical Risk Minimization (ERM) [46]. which optimizes average performance. However, this approach has been found to underperform on certain groups of data [4. [I] due the preva- lence of spurious correlations within training datasets [31] 38 (39]
Class label
Cow
Camel
Backgrund
Spurious correlations refer to the strong correlations between get labels and easy-to- lusy-to-learn attributes (i.e., bias attributes), which are present in majority of the training examples. ERM-trained models often rely on these bias attributes [12, leading to biased predictions and poor generalization on minority groups where spu- rious correlations are absent. For example, consider the cow/camel classification task illustrated Figure T A majority of camel images feature desert backgrounds while a majority of cow im- ages feature pasture backgrounds. Models trained via ERM might learn to recognize animals based their backgrounds-dee desert for camels and pasture for cows-rather than on their distinctive features. This reliance can result in misclassifications. such erroncously identifying a camel in a pasture as cow. Addressing
Protumm:
Figure An illustration of the cow/camel classification task. Red dotted boxes indicate sam- ples where spurious comellatins do hold.
these spurious correlations is
*Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

critical issue across various applications, including medical imaging [6], algorithmic faimesss [T]] and education (37.
There have been extensive efforts to reduce the effects of spurious correlations. Numerous studies have initially relied on the assumption that bias attributes given in the training dataset (38
18 24 35
However, annotating bias attributes is labor-intensive and expensive, rendering methods dependent on such annotations impractical. Consequently, research has shifted towards developing debiasing I
techniques that do not require bias labels during training (34, 27
29
28 49]
In this line of
research, samples with spurious correlations are called bias- cla--al samples, whereas those lacking such correlations are termed bias- ctas-confficting samples [34] 27 23). In the absence of explicit bias labels, many existing methods focus on identifying coafliinfi samples and employ strategies such as upweighting or upsampling these samples to counteract the negative impacts of spurious correlations (3 28 I. Despite their straightforward nature, questions persist regarding the optimal scale for the weight of each sample to effectively reduce dependence on spurious correlations.
We thus formulate a debiasing objective for improved robustness against spurious correlations and derive a practical resampling-basse algorithm. Specifically, our debiasing objective is built on two groups: the bias-aligned group and the bias-conflicting group, each consisting of bias-aligned and bias-conflicting samples, respectively. This objective encourages the model to perform equally well across both groups. The key insight behind the objective that models relying on spurious correlations exhibit worse performance on the bias-conflicting group compared to the bias-aligned group, whereas models that do not rely on spurious correlations should exhibit strong performance across both groups. Then, under a simple condition, we derive an objective function as a weighted loss, with weights proportional to the bias-confficting probability for each training example. Since we consider cases where bias labels are not provided, we use the disagreement probability between the target label and the prediction an intentionally biased model as a substitute for the bias-conflicting probability. Our proposed approach is simple yet effective in mitigating spurious correlations.
The main contributions of this paper can be summarized as follows:
We present debiasing objective for mitigating reliance on bias attributes. This objective aims to guide the model to have similar performance over two groups: the bias-aligned group and the bias-conf group. This approach differs from previous works, which typically define groups based on combinations of target labels and bias labels.
We propose a new method, coined Disagreement Probability based Resampling for debiasing (DPR), which is derived from the proposed objective. DPR leverages the disagreement probability between the target label and the prediction of a biased model to determine the weight of each training example.
DPR achieves state-of-the-an performance across six benchmarks with spurious correlations, surpassing existing methods that do not use bias labels. Notably, on the bias-confficting test set of Biased FFHQ (BFFHQ)-which whhic contains only 0.5% of bias-conflicting samples in the training set tee-theepp proposed method significantly improves accuracy by 20.87% compared to ERM and by 6.2% compared to the best baseline.
We thcoretically demonstrate that DPR encourages consistent model performance across both bias-aligned and bias-confficting groups, thus promoting strong performance across all samples regardless of spurious correlations.
2 Related work
Debiasing with bias annotations. Numerous previous works utilize bias labels for debiasing
44 [3]. For example, Group DRO [8) employs bias labels define groups and directly enhances worst-group accuracy to mitigate the effect of spurious correlations; meanwhile, 48 mixes two samples with the same label but different domains, or with different labels but the same domain, canceling out spurious correlations and learning invariant predictors. These methods have demonstrated their effectiveness across multiple benchmarks with spurious correlations. However, annotating bias labels for each training example is labor-intensive, and obtaining such labels can sometimes be challenging due to privacy concerns as well [51] Consequently, in more recent studies, some researchers have utilized only small set of bias annstannttated data to reduce reliance on
[8 48 18
LISA

---

bias labels 24 35 19. For example, DFR [24] uses a small group- group- balanced dataset with bias labels retrain the last layer of the ERM-trained model.
Debiasing without bias annotations. In effort to eliminate reliance on bias annotations, recent studies predominantly focus on reducing bias without explicit bias labels [4 22
I 29, 28
49 45]. Given the unavailability of bias labels, these methods commonly employ two-stage strategy: (1) identifying bias- aaiigdd and bias conflicting samples, and (2) training the debiased model by leveraging information obtained from (1). The identification of bias-confiicingg samples is accomplished through the use of deliberately biased model, trained either with generalized
cross-entropy loss [34]] 22 27 29 standard cross-entropy loss (8
49 45). For instance, LfF
[34] and DFA 27 regard samples as bias-conflicting samples if the biased model exhibits higher cross-entropy losses these samples compared to the debiased model. Subsequent efforts then focus on enhancing classification accuracy for these identified bias-conflicting samples. Numerous studies have proposed diverse debiasing methods [14 52 8 23 including those based on contrastive learning [49 (20] and unsupervised clustering 44 42
Many debiasing methods that do not require bias labels aim to enhance performance on the bias- conflicting group, yet the optimal extent of this enhancement remains uncertain. We contend that a debiased model should exhibit consistent performance across both bias-aligned and binfcconn groups. However, to the best of our knowledge, no existing training objective is explicitly designed for this purpose. Therefore, introduce novel training objective tailored for this purpose, from which we develop DPR-s a debiasing method that does not rely on bias labels.
3 Problem formulation
We consider a multi-class classification problem with a dataset D {(x,, v:) where each I; E X represents an input, and each yi y is the corresponding label from K possible classes. These examples are presumed to be sampled from a training distribution P. Given a classification model fo X - R that maps imput to K logits, and convex loss é X x - Rz0, ERM aims to find a model that minimizes the expected loss ((a.f)=(((fo(t)) v)]. To this end, we typically
minimize surrogate loss Laugi
1 Σ ((fo(z).v).
Lavg
(2.1))D
The cross-entropy (CE) loss is commonly used for training classification models. It is defined as (ce(.fe (c ing the y-th class label.
{x) + log Eyg exp(foxx)(v)) where fe [x)[x) denotes the logit correspond-
We assume there is a spurious correlation presence indicator E B (correlated, uncorrelated) in the dataset. As this indicator denotes whether spurious correlations are present within each sample, it can also be considered as bias-aligned or bhas-confficting group indicator. Similar to the Group DRO setting [38 [17), adopt the latent prior probability change assumption indicator b, we assume
[43). With group
P(z, Q(z, vlb), P(b) + Q(b),
(2)
where P and Q denote the training and test data distributions, respectively. Under this assumption, P(r, 3) and Q(z, y) are represented as mixtures of the conditional distributions P(æ, y|b). Our goal is find models that are robust against spurious correlations. Regardless of the presence of spurious correlations within each data example, model that does not rely on these correlations should exhibit strong performance across all examples. other words, the debiased model should perform consistently well on both bias- bias-aligned and bias- chas-confiictig groups To this end, we propose the following training objective to avoid spurious correlations:
min max B BEB
L, A= Thb
Σ ((fo()).))
(x.g.b)EG.
where Gb is subset of the training data composed of samples drawn from P(x, y/b) and nı the size of Gg. The proposed objective aims to minimize the maximum average loss over bias-aligned and
3

---

bias-conftictimg groups, thereby reducing the reliance of classification models on spurious correlations. However, to utilize the above objective, we must know the information about bias attributes. In the next section, we describe practical method to train the debiased model using Equation 3 bias labels.
without
DPR: Disagreement Probability based Resampling for debiasing
We present DPR, a resampling method derived from the proposed objective, which does not require bias annotations during training. First, we provide walk-through of how the objective in Equation (3) can be reformulated the weight of each training example, along with proxies for both bias-aligned and bias-conflicting groups. Finally, we provide full description of our algorithm.
weighted loss minimization problem. Next, we detail method for calculating
4.1 Problem reformulation
To utilize the objective in Equation 3 without bias annotations, we reformulate this objective as weighted loss minimization problem. For this purpose, we introduce the following assumption:
Assumption Let ba B and b. B represent the bias-aligned and bias-conflicting groups, respectively. The neural network, parameterized by 0, satisfies that Lt. < Lb.-
In Assumption assume that the model, parameterized by 0, exhibits higher average loss the bias-confiitting group compared to the bias-aligned group in the training dataset. Under this assumption, the maximum average loss over groups in Equation (3) can be expressed as follows:
max Lo BEB The (e.
Σ
{(fo(r).))
Σ
p(b = be b=l=))((((()))
(5)
The (z,y.))E
p(b = belz) ((f(r)).))
Σ
(6)
p(b b.)
(E.
When Assumption 1 is satisfied, Equation 4 holds. Given that all samples in G2 have p(b = xx)
equal to 1. whereas all samples in Go., have p(b = bo(z) equal to O, Equation (5) is derived from
Equation (4). As p(b = be) is equal to Réc. Equation (6) is obtained. By denoting p(b=bela) as
p(b=b.)
r(x, y. b). we obtain a weighted loss minimization as follows:
min Σ (D.D
r(x, U. ()((f(()))))
Note that the weight r(z, y, b) can be interpreted as the sampling probability.
4.2 Sampling probability with group proxy
In order to train the model using Equation (7), it is necessary to compute the sampling probability r(x, y. b) for each training example. However, directly calculating the probabilities p(b = be(z) and p(b = be) is unfeasible without bias labels. To overcome this limitation, we first introduce proxies for the bias-aligned and bias-conflicting groups We then derive substitutes for the probabilities p(b = be(zz) and p(b be) using biased model.
Group proxy. We focus on the characteristies of the biased model for the group proxies. Following Nam et al. 34 the biased model fø is trained using ERM with the generalized cross- cross-entropy (GCE)
loss [5
lacE (ocE(f(()), 1 - (2(
(8)
q
where (y(px) represents the probability assigned to the target label y by the neural network after softmax layer, and (0, 1] is hyperparameter. The GCE loss amplifies the bias of the model

---

Algorithm 1: Disagreement Probability based Resampling for debiasing (DPR)
Input: training set D, biased model fa, debiased model Jo, learning rate 1), the total number of iterations T3 and Ta. calibration parameter т, GCE parameter
** Train the biased model
for i = to T3 do
Sample mini-batch {(r, y)} from D
Update by training on mini-batch using Equation (8) end
/* Compute the sampling probability
Compute f(x, y) for all (x,y) E following Equation
/* To satisfy Assumption 1 Initialize the debiased model fa with the biased model f
*/
/* Train the debiased model
for 1 to Ta do
Sample mini-batch {(r, y)} from D according to f(xr, y) Update 0 by training mini-batch using cross-entropy loss
end
by up- up-weighting the gradient of the cross-entropy loss for samples with high probability (y) thereby training the model to rely on spurious correlations. Consequently, the biased model tends to predict correctly for bias-aligned samples and incorrectly for bias-confficting samples (34). Building
on this insight, leveraging the predictions Ybins of the biased model, we employ the agreement between the label and the biased model's prediction (i.e., y = Yvias) proxy for the bias bias-aligned group ba. and the disagreement (i.e., y # Yoias) as proxy for the conffccnnfi group be-
Sampling probability. We now discuss how compute the sampling probability r(x, y, b). Using the group proxies mentioned above, we substitute ply * Wbias(r) for p(b = b.lx). For given example (2r, y). the disagreement probability ply Yoiss (r) can be computed follows:
ply * Yhias (x))
Σ
p(u, Yhias (x) ply = Ubins [x) = 11 Phins (3)lz),
(9)
where Phins (w/x²) exp(fa(e)lul/r) is the probability assigned to label y by the biased model, (()) ((0)
Ey exp(fe ( (z)l)/ the partition function, and T is temperature hyperparameter. Note that Phias (y ()) is used to approximate p(b = 6, (z)).. It is crucial for the biased model to accurately capture the spurious correlation structure; hence, the probabilities of the biased model should be appropriately
calibrated [13 33 32). We also substitute p(y * Ybins) for p(b = be). This probability can be
estimated using all the training data, as prior works (29 45):
ply 4 Uoias) RJ
Σ
p(u + Whins (x)
1
Σ
(1 Prins(y/(3)
(10)
(2.9.))D
(m.y.h)ED
With Equations 9 and (0), we compute
f(z, v) = p(y # Uhizs (x)
1 Poias (1/(2=
(11)
plu F Ybius)
E Exz.g,ie
Poius (y)zz))
Instead of r(æ, y, b), we use f(x, y) as the sampling probability. We are now ready to train the debiased model.
4.3 Training algorithm
We outline the entire training process for our method as follows. First, we train the biased model fg. Next, we compute the sampling probability (zx,y) using the pretrained biased model. Before proceeding to train the debiased model fo, it is worth noting that Equation 7 is derived under Assumption Thus, it is essential to fulfill Assumption I when training the debiased model using
5

---

We leverage the characteristics of the biased model for this purpose. The biased model typically exhibits higher loss on consfconn samples and lower loss on bias-aligned samples. thereby fulfilling Assumption [ Consequently, we initialize the model so with the biased model Sa and then train fo using training examples sampled with the probability f(x, y)- We also employ data augmentation to enhance the diversity of bias-conflicting samples. Simply oversampling these samples without enhancing their diversity does not effectively mitigate bias [27]. Therefore, we enhance the diversity of bias-conflicting samples through data augmentation techniques such as random color jitter and random rotation. The complete training procedure is outlined Algorithm [
Equation 7p.
Theoretical analysis
In this section, theoretically demonstrate that DPR minimizes losses for both bias-aligned and bias-conflicting groups while reducing the disparity between their losses, All proof' are deferred to Appendix A Let Lavg be the expected average loss:
Lang == R ((
(12)
Let Cb be the average loss of group b:
Lo (( ((fox())
(13)
where Ps P(x,, b)b) denotes the training distribution conditioned on b, for any E B. In this setting, we derive the following inequality for the loss gap between the bias-aligned and biascconflicting groups.
Theorem 1. Suppose that the loss function {()(())))) y) is upper-bounded by a constant C > 0. Given two distinct groups b, B and b, B such that ba * be, the following inequality holds with probability at least 1 - ō, for any 8 >> 0:
8log (18)/5) 7
Lb.. - Lo,1 S 2 max Lo + C' max
(14)
BEB
DEB
Theorem T |pecifies that the upper bound on the disparity between losses for bias-aligned and bias- conflicting groups is determined by the maximum average loss over groups and term dependent on the size of the smaller group. Additionally, we derive an inequality associated with the expected average loss.
Theorem 2. In the same setting as Theorem the expected average loss is bounded above with probability least 1- ō:
Laug < max L. + C. /2 bEB
2 log
(15)
According to Theorems and our proposed training objective not only closes the loss gap between bias-aligned and bias-conflicting groups but also reduces the expected average loss. However, in Equation (14), there remains a loss gap due to a term inversely related to the square root of the size of the smaller group. Note that DPR is a resampling method derived from the proposed objective when Assumption iis fulfilled, and it identifies and upsamples Stas-cnfflicting samples. Thus, it efficiently minimizes both terms of the upper bound described Equation (14). Given that Lavy expressed as a weighted sum of Lb, and Lb., these theorems indicate that DPR enhances performance across both blas-aligned and bias-conflicting groups while reducing the performance gap between them.
Experiments
6.1
Datasets
We evaluate the debiasing performance of DPR using six benchmark datasets that exhibit spurious cor- relations. Colored MNIST and Multi-bias MNIST are synthetic datasets designed under the premise that models learn bias attributes first. Conversely, BAR, BFFHQ, CelebA, and CivilComments- WILDS are real-world datasets where inherent biases degrade model performance. We follow the evaluation protocols of previous studies in Appendix B.I
[ 27
and provide a detailed description of these datasets

---

Colored MNIST. Colored MNIST (C-MNIST) is synthetic dataset designed for digit classification, comprising ten digits, each spuriously correlated with a specific color. Following the protocols in Ahn et al. [1 we set the ratios of bias-conflicting samples, denoted p. at 0.5%, 1%, and 5% for the training set, and 90% for the unbiased test set. We report the accuracy on this unbiased test set.
Multi-bias MNIST. Multi-bias MNIST (MB-MNIST) [ is synthetic dataset designed to in- corporate more complex patterns compared to C-MNIST and biased-MNIST [41]. MB-MNIST comprises eight attributes: digit (26), alphabet 77. fashion object [47], Japanese character [6] digit color, alphabet color, fashion object color, and Japanese character color. The digit shape is the target attribute, while the other seven serve as bias attributes. In MB-MNIST, bias is introduced by aligning the digit with each of the other seven attributes, each with a probability of 1 In our experiments, p is set 10%, 20%, and 30% for the training set and 90% for the unbiased test set, as in Ahn et al. [1] We report the accuracy on this unbiased test set.
Biased action recognition. The biased action recognition (BAR) dataset [34], designed for action classification, comprises six action classes such as climbing and fishing, each spuriously correlated with a specific place. The training set of BAR contains only bias-aligned samples, while the test set consists solely of bias-conflicting samples. We report the accuracy this bias-conflicting test set.
Biased FFHQ (BFFHQ) is real-world facial dataset, which has age (young or old) as a label and gender (male or female) as bias attribute. Predominantly, females are young and males are old in this dataset. We use bias-conflicting ratio of 0.5% in the training set and report accuracies on both an unbiased test set with p 50% and clas-conflicting test set with 100% (27].
Biased FFHQ.
CelebA. CelebA 30 is a dataset for facial classification. The goal is to classify celebrity hair color blond non-blond, which is spuriously correlated with gender. Notably, only a few blond-haired celebrities are male. Following prior studies [49 (38] we report both average and worst- worst-group accuracies on the test set, where groups are defined combinations of class labels and bias labels.
Certoommmnts-WI..DD 5 25 is a text classification dataset aimed at identifying whether online comments are toxic or non-toxic. The label is spuriously correlated with demographic identities such gender, race, and religion. Following previous works [28] 25 49). we report both average and worst-group accuracies on the test set, where groups are defined as combinations of class labels and bias labels.
6.2 Experimental setup
Baselines. We compare our method with six baselines on various benchmarks: ERM, JTT (28). DFA (27). CNC [49), PGD (I. and LC [29]. ERM denotes conventional training without any consid- erations for debiasing, while the others are debiasing methods that do not require bias annotations during training.
Implementation details. For all datasets except CelebA and CivilComment-WILDS. we follow the experimental settings of Ahn et al. M. Specifically, for CMNIST and MB-MNIST, we employ two distinct types of simple CNN models, respectively. For BAR and BFFHQ utilize ResNet18 [66] pretrained on ImageNet 19. In the case of CelebA, we use a pretrained ResNet50, following the experimental settings of Zhang al. [49]. For CivilComments WILDS, we deploy a pretrained BERT model and follow the experimental setup detailed in Liu et al. (28). Moreover, we apply data augmentation techniques-incluceee rotation-for all vision datasets except CelebA, as discussed in Section Further details on model architectures, hyperparameters, and image processing are provided in Appendix B.2
random resize crop, random color jitter, and random
6.3 Experimental results and analysis
Classification accuracy. Table presents the average accuracies unbiased test sets for C-MNIST and MB-MNIST. DPR consistently outperforms or matches the performance of other baselines across all experiments with varying bias-conflicting ratios. Notably, for the MB-MNIST dataset with
7

---

Table Average accuracies and standard deviations over three trials on two synthetic image datasets, C-MNIST and MB- MB-MNIST, under varying ratios (%) of bias-conflicting samples. Except for LC, the results of baselines reported in Ahn et al. are provided. The best performances are highlighted in bold.
C-MNIST
MB MNIST
Ratio (%)
0.5
5
30
ERM
60,94 (0.97)
79.13 (0.73)
95.12 (0.24)
25.23 (1.16)
62.06 (2.45)
87.61 (1.60)
JTT
85.84 (1.32)
95.07 (3.42)
96.56 (1.23)
(1.34(1145)
68.02 (3.23)
(3,44)
DFA
94.56 (0.57)
96.87 (0.64)
98.35 (0.20)
25.75 (5.38)
61.62 (2.60)
88.36 (2.06)
PGD
96.88 (0.28)
98.35 (0.12)
98.62 (0.14)
61.38 (4.41)
89.09 (0:97)
90.76 (1.84)
LCC
97.25 (0.21)
97.34 (0.16)
97 44 (0.37)
25.86 (8.68)
71.23 (1.71)
89.57 (2.50)
DPR (Ours)
97.52 (0.33)
98.40 (0.03)
98.62 (0.12)
62.21 (4.02)
89.11 (1.65)
94.04 (0.26)
Table 2: Classification accuracies (%) and standard deviations over three trials on four real- world datasets. Conflicting refers to accuracy on bias-conflicting test sets, while Unbiased indicates accuracy on unbiased test sets. Average and Worst denote average test accuracy and worst-group accuracy, respectively. Results of all baselines except LC are taken from Ahn al. M, with the exception of bias-conflicting accuracy on BFFHQ. The best performances are highlighted in bold.
BAR
BFFHQ
ColebA
CivilCommens-WLL
Accuracy (%)
Conflicting
Unbiased
Conflicting
Average
Worst
Average
Worst
ERM
63.15 (1.06)
77.77 (0.45)
55.93 (0.64)
94.9 (0.3)
47.7 (2.1)
92.1 (0.4)
58.7(1.)
JTT
63.62 (1.33)
77.93 (2.16)
56.13 (0.83)
88.1 (0.3)
(1.7)
91.1
69.3
DFA
64.70 (2.06)
82.77 (1.40)
66.00 (2.00)
CNC
89.9 (0.5)
88.8 (0.9)
81.7 (0.7
68.9 (2.1)
PGD
65.39 (0.47)
84,20 (1.15)
70.07 (2.00)
88.6 (-)
88.8 (-)
92.1
70.6 (-)
(2.14)
83,97 (0.83)
70.60 (0.60)
88.1 (0.8)
70.3 (1.2)
DPR (Ours)
66.11 (3.29)
87.57 (1.22)
76.80 (2.51)
90.7 (0.6)
88,9 (0.6)
82.9 (0.7)
70.9 (1.7)
100
Samole ranto per group 196
Blas-eigned Blas contlicting
Semohe retto group (96
Blas-allgned
Samooee ratio per group 1%6
Blar aligned Blas conticting
Bla conflicting
03
03
-0
01 amtr in p0.00, aoss, pu.6, (KS.
0000000
pa.se, nast pass, B.9 (AS.31
19 (.50)
[0.355, BURSi [88%,
Diaggrement probabilty
Disagrement probabilty
Disagreement probabilitt
(a) C-MNIST (0.5%)
(b) C-MNIST (1%)
(c) C-MNIST (5%)
Figure 2: Distributions of disagreement probabilities for each sample within bias-aligned and bias- conflicting groups.
ratio of 30%, DPR achieves an unbiased test accuracy of 94.04 04.0,, outperforming PGD by 3.28%. Even for MB-MNIST with ratio of 10%, where all baselines except PGD fail to achieve higher accuracy, DPR exhibits the highest accuracy of 62.21%. In the more complex setting of MB-MNIST. compared to C-MNIST, the effectiveness of DPR is even more pronounced. The superiority of our method further demonstrated on real-world benchmarks, as shown in Table 2 Our method consistently achieves the best performance for each real-world dataset. Specifically, for the BFFHQ dataset, DPR achieves an accuracy of 87.57% on unbiased test sets, which is 3.37% higher than PGD, and accuracy of 76.80% on bias-conflicting test sets, which is 6.20% higher than LCC For the CelebA and CivilComments- WOLLD datasets, our method achieves the highest worst-group accuracy compared to other baselines, with groups defined the combination of the target and bias labels. We especially highlight the results for the BFFHQ benchmark, where our method improves accuracy on both unbiased and bias-conflicting test sets. This result supports our claim that DPR enhances performance on both bias-aligned and bias-conflicting groups
Identifying group via disagreement probability. To discern whether each sample belongs to the bias blis-aligned or blas-conflicting group, we check if the target label disagrees with the prediction of the biased model and use its probability as an up-weight. To evaluate whether the disagreement probability between the target label and the prediction of the biased model distinguishes bias- conflicting samples from bias-aligned samples effectively, we plot the distributions of disagreement probabilities ply A Sbias/z) [2) for each bias-aligned and bias-conflicting sample. The experiment conducted on the C-MNIST dataset. As illustrated in Figure bias-aligned samples generally exhibit
8

---

Table Ablation studies of the proposed method on the C-MNIST and BFFHQ datasets. We report the average test accuracies and standard deviations over three trials unbiased and cins-conn test sets. checkmark (v)) indicates the case where each component of the proposed method applied. The best performances are highlighted in bold.
C-MNIST (0.5%)
C-MNIST (5%)
BFFHQ
Initialization
GCE
Augmentution
Unbiased
Unbiased
Unbiased
Confiicting
x
x
66.13 (0.51)
95.32 (0.42)
77.07 (1.16)
54.60 (2.43)
/
/
66.47 (1.74)
95.21 (0.14)
77.60 (0.79)
55.60 (1.64)
x
X
89. 06 (0.62)
97.36 (0.29)
78.50 (0.50)
57.47 (0.90)
X
95.94 (0.45)
97.66 (0.17)
80,93 (1.33)
63.40 (3.67)
V 97.52 (0.33)
98.62 (0.12)
87.57 (1.22)
76.80 (2.51)
smaller disagreement probabilities compared to bias-conflicting samples. This result demonstrates that disagreement probability effectively differentiates bias-aligned and bias-confficting samples. Moreover, the relatively high disagreement probability associated with bias-conflicting samples enables DPR to effectively identify and upsample bias-conflicting samples, suggesting that the disagreement probability p(y * Voias/ ()) serves as an efficient substitute for the bias-conflicting probability p(b bz(z)).
Group losses of different models. As stated carlier in Sec- tion 4.3 our debiasing objective Equation ( should be used only when Assumption D satisfied. A natural next question is: which type of model satisfies this assumption? We test three types of models: a randomly initialized model, a model pretrained on ImageNet, and biased model. Specifically, we examine the average loss of randomly initialized model on the bias-aligned and bias-conflicting groups of C-MNIST with bias-conflicting ratio of 0.5%. Additionally, we examine the average loss of pretrained model on the bias-aligned and bias-conflicting groups of BFFHQ. We also investigate the average loss of a biased model on the bias-aligned and bias-conflicting groups of C-MNIST and BFFHQ. The results, shown in Figure indicate that both the randomly initialized and pretrained models have similar average losses the bias aligned and mlas-confficting groups In contrast, the biased model has a higher average loss on the coasfconfii group compared the bias-aligned group for both C-MNIST and validity of initializing the debiased model with the biased model.
C-MNIST (0.5%)
BFFHO
Blaz-ligned Blac-oor ficting
Average loss
0:05
6:03
800
0:00
Randomlyy Initialized
Blased
Pretrained
Blased
Figure Average loss of ran- domly initialized, pretrained, and biased models on bias-aligned and bias-conflicting groups. The eтror bars represent the standard deviations over three trials.
BFFHQ. This result supports the
Ablation study. We conduct ablation study model initialization, GCE, and data augmentation. Model initialization refers to initializing the debiased model with the biased model. The evaluation is performed C-MNIST with bias bias-confticting ratios of 0.5% and 5%, as well as BFFHQ Table demonstrates the importance of each component. Comparing the first and second rows, we observe that GCE and data augmentation bring small improvements when initialization is used. However, from rows 3-5, we observe that GCE and augmentation bring significant improvements when initialization is utilized. For BFFHQ, introducing both GCE and augmentation significantly improves the average accuracies on unbiased and bias-conflicting sets by 9.07% and 19.33%, respectively. Furthermore, observe the performance gap between using and not using initialization. These results suggest that model initialization is crucial and that GCE and augmentation are also important when initialization is introduced. The best performances are consistently achieved when all components are utilized.
7 Conclusions and Limitations
We present DPR, a resampling method that leverages disagreement probability to identify and upsample bias-conflicting samples. This method is derived from novel training objective designed to achieve consistent performance across both bias-aligned and bias-conflicting groups. It does not rely on bias annotations and has demonstrated state-of-the-arr performance across spurious correlation benchmarks. However, DPR has certain limitations: its effectiveness depends on how well the biased model captures the spurious correlation structure, it uses the predictions of this model to infer

---

the group to which each training sample belongs. Moreover, DPR employs a two-stage approach that complicates the training process and introduces additional hyperparameters. Despite these limitations, DPR's simple implementation and strong performance, supported by theoretical analysis illustrating its ability to reduce loss disparity between groups and minimize average loss, underscore its usefulness in mitigating reliance spurious correlations.
Acknowledgements
This work is in part supported by the National Research Foundation of Korea (NRF, RS-2024- 00451435(25%),
Institute of Information & communications Technology Planning & Evaluation (IIT,, 1021-000105(()) (15),, 2021-0-001 1221--0000662056 2021-0-001800 funded by the Ministry of Science and ICT (MSIT), a grant-in-ard of HANHWA SYSTEMS, Institute of New Media and Communications((NMMa Research Program for Future ICT Pioneers, Scoul National University in 2024.
grant
and the BK21 FOUR program of the Education and
References
[1] Sumyeong Ahn, Scongyoon Kim, and Se- Young Yun. Mitigating dataset bias by using per- sample gradient. In The Eleventh International Conference on Learning Representations, 2023. URL http:ttts///perreiiiee www.nneeetedn-aaeeeaenna
7GMv
[2] Jing An, Lexing Ying, and Yuhua Zhu. Why resampling outperforms reweighting for cor- recting sampling bias with stochastic gradients. In International Conference Learning Representations, 2021. URL http::ttp://opeeeevveett http://ppaarreeeeeeeeeeeeuuuraea
[3] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint 1w0X.v:907.02893 2019.
[4] Su Lin Blodgett, Lisa Green, and Brendan O'Connor. Demographic dialectal variation in social media: A case study of african-american english. arXiv preprint arXiv: 088668 2016.
[5] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference, pages 491-500, 2019.
[6] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01770 2018.
[7] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921-2926. IEEE, 2017.
[8] Elliot Creager, Jörn- Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, pages 2189-2200. PMLR, 2021.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, Icee, 2009.
[10] Mengnan Du, Subhabrata Mukherjee, Guanchu Wang, Ruixiang Tang, Ahmed Awadallah, and Xia Hu. Fairness via representation neutralization. Advances in Neural Information Processing Systems, 14:12091-,2103, 2021.
[11] John C Duchi, Tatsunori Hashimoto, and Hongseok Namkoong. Distributionally robust losses against mixture covariate shifts. Under review, 2(1), 2019.
[12] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shorteut learning deep neural networks. Nature Machine Intelligence, 2(11):665-677 661-66666666,,6 2020.
10

---

[13] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. International conference on machine learning, pages 1321-1330. PMLR, 2017.
[14] Zongbo Han, Zhipeng Liang. Fan Yang, Liu Liu, Lanqing Li, Yatao Bian, Peilin Zhao, Bingzhe Wu, Changqing Zhang, and Jianhua Yao. Umix: Improving importance weighting for subpopu- lation shift via uncertainty-awalt mixup. Advances in Neural Information Processing Systems, 35:37704-37716 18, 2022.
[15] Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning, pages 1929-1938. PMLR, 2018.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[17] Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust super- vised learning give robust classifiers? In International Conference on Machine Learning, pages 0029-2037, PMLR, 2018
[18] Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-accuracy In Conference on Causal Learning and Reasoning, pages 336-351. PMLR, 2022.
[19] Sangwon Jung, Sanghyuk Chun, and Taesup Moon. Learning fair classifiers with partially annotated group labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 10348-10357, 2022.
[20] Yeonsung Jung, Hajin Shim, June Yong Yang, and Eunho Yang. Fighting fire with fire: contrastive debiasing without bias-free data via generative bias-transformation. In International Conference on Machine Learning, pages 15435-15450. PMLR, 2023.
[21] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IÉEE/CVF conference on computer vision and pattem recognition, pages 4401 -410, 2019.
[22] Eungyeup Kim, Jihyeon Lee, and Jaegul Choo. Biaswap: Removing dataset bias with bias- tailored swapping augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11992-15011, 2021.
[23] Naycong Kim, Schyun Hwang, Sungsoo Ahn, Jaesik Park, and Suha Kwak. Learning debiased classifier with biased committee. Advances in Neural Information Processing Systems, 35: 18403-18415, 2022.
[24] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is suffi- cient for robustness to spurious correlations. In The Eleventh International Conference on Learn- ing Representations, 2023. URL
[25] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference machine learning, pages 5637-5664. PMLR, 2021.
[26] Yann LeCun, Corinna Cortes, Chris Burges, et al. Mnist handwritten digit database, 2010.
[27] Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debi- ased representation via disentangled feature augmentation. Advances in Neural Information Processing Systems, 34:25123-2515 2021.
[28] Evan Liu, Behzad Haghgoo, Annie Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pages 6781-6792. PMLR, 2021.
11

---

[29] Sheng Liu, Xu Zhang, Nitesh Sekhar, Yue Wu, Prateek Singhal, and Carlos Srrnandez-Grandde Avoiding spurious correlations via logit correction. In The Eleventh International Confer- ence on Learning Representations, 2023. URL fettp::/opennreviee..etttt wwtp:/nneeee./otoeoo.. 5BaqCFVh5qL
[30] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
[31] Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv: (a0:::00200007 2019.
[32] Aditya Menon, Harikrishna Narasimhan, Shivani Agarwal, and Sanjay Chawla. On the statistical consistency of algorithms for binary classification under class imbalance. In International Conference on Machine Learning, pages 603-611. PMLR, 2013.
[33] Rafael Müller, Simon Korbllith, and Geoffrey E. Hinton. When does label smoothing help? Advances neural information processing systems, 32, 2019.
[34] Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems, 53:20673-20684 2020.
[35] Junhyun Nam, Jaehyung Kim, Jaeho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. In International Conference on Learn- ing Representations, 2022. htttps:////::peereee//
fept:/formm?dda... 2ttp://rorommmee_Ppx0r FFpppOrqyXXX
[36] Luke Oakden-Ray Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden strat- ification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM conference on health, inference, and learning, pages 151-159, 2020.
[37] Les Perelman. When "the state of the art" is counting words. Assessing Writing, 21:104-111, 2014.
[38] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL
[39] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterizatioo exacerbates spurious correlations. In International Conference on Machine Learning, pages 8346-8356. PMLR, 2020.
[40] Seonguk Seo, Joon- Young Lee, and Bohyung Han. Unsupervised learning of debiased repre- sentations with secudo-tttributs.. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16742-16751, 2022.
[41] Robik Shrestha, Kushal Kafle, and Christopher Kanan. An investigation of critical issues in bias mitigation techniques. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1943-1954, 2022.
[42] Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. Advances in Neural Information Prucessing Systems, 13:19339-19352, 2020.
[43] Masashi Sugiyama and Amos J Storkey. Mixture regression for covariate shift. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006. https:/ttts:::
necigggg...eurrig neurips.... wttppppp..ppppp.ppppppppppp fites//feeessee//////// 2006/ fil.ae/2cbeeeees800044afefefeff.
pdff
[44] Damien Teney, Ehsan Abbasnejad, and Anton van den Hengel. Unshuffling data for improved generalization in visual question answering. In Proceedings of the IEEF/CVF international conference computer vision, pages 1417-1427, 2021.
12

---

[45] Christos Tsirigotis, Joao Monteiro, Pau Rodriguez, David Vazquez, and Aaron C Courville. Group robust classification without any group information. Advances in Neural Information Processing Systems, 36, 2024.
[46] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.
[47] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 170.07777 2017.
[48] Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Im- proving out-of-distribution robustness via selective augmentation. In International Conference on Machine Learning, pages 25407-25437. PMLR, 2022.
[49] Michael Zhang, Nimit Sohoni, Hongyang Zhang, Chelsea Finn, and Christopher Re. Correct-n-contrast: contrastive approach for improving robustness spurious correlations. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 26484-26516. PMLR, 17-23 Jul 2022.
[50] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances neural information processing systems, 31, 2018.
[51] Chenye Zhao, Jasmine Mangat, Sujay Koujalgi, Anna Squicciarini, and Cornelia Caragea. Privacyalert: A dataset for image privacy prediction. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pages 1352-1361, 2022.
[52] Xiao Zhou, Yong Lin, Renjie Pi, Weizhong Zhang, Renzhe Xu, Peng Cui, and Tong Zhang. Model agnostic sample reweighting for ofitt--distribuutio learning. International Conference on Machine Learning, pages 27203-27222 PMLR, 2022.
[53] Wei Zhu, Haitian Zheng, Haofu Liao, Weijian Li, and Jiebo Luo. Learning bias-invariant repre- sentation by cross-sample mutual information minimization. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15002-15012, 2021.
13

---

Appendix
A Missing proofs from Section 5
Proof of Theorem 7
Consider the two groups ba B and be such that by + be Let d(bas be) be the difference in expected losses between the group by and be:
d(ba. be) = Lb. l.
(16)
By the Hoeffding's incquality, the following incquality holds with probability at least 8, for all groups B,
2 log (151/0)
c. - =c C.
(17)
n2
Note that the loss E(-) upper-bounded by some constant according to our assumption and always non-negative. Accordingly, the following holds with probability at least 1 - 8,
d(ba: be) S Cs. - Cool -oul [c 22 (131/5)
2 log (131/6) Td.
+
(18)
Tbb
Since the equation max y) = 242 18-21 holds for E R and I,, the RHS of Equation (18) most:
2
S max Ls-
(19)
BEB
Therefore, we have shown that the following result holds,
d(ba, be) S 2. max Lo + C c( (V
2 log (151/5)
2 log (131/5)
DEB
Tib.
nbb.
log (181/0)
2 max Lo C max BEB
(20)
BEB
71b
This completes the proof of Theorem
Proof of Theorem
By the Hoeffding's inequality, the following inequality holds with least 1
8:
Lavg Lang SC. V
1log(110)
(21)
n
Accordingly, the expected average loss is bounded with probability - 5 as follows:
2 log(1//) I Lavg I Lavg C
2 2log(1/8)
Lavg C
(22)
Considering that the training distribution is represented by a mixture of group distributions Pa. and Pb. for two distinct groups ba E and b. B, Lavg equal to
kbb Lo. + kbb Lo, max Ls,
(23)
beB
where kt. P(b.) and ls. P(b.) denote the prior probability associated with groups. Therefore, Lang is upper-bounded with probability at least 1 - 8:
2 log (1/8)
Lavg
max C. C. beB
(24)
7
This completes the proof of Theorem 2
14

---

B Experimental details
B.1 Benchmarks
We provide detailed description of the datasets utilized in Section 6 In Figures 4 to g corresponds to a distinct class. The images positioned above the dashed line represent bias-aligned samples, whereas those below represent bias- clasfcoonii samples.
each column
2
3
4
3
6
7
8
9
/
2
3
4
3
6
7
8
9
3
2
2 4
5
7
8
9
Figure Colored MNIST.
Figure 5: Multi-bias MNIST.
Figure 6: Biased action recognition.
Figure 7: Biased FFHQ
Figure 8: CelebA
Colored MNIST (C-MNIST) is synthetic dataset designed for digit classification that introduces controlled spurious correlation between digit shape and color. This dataset consists of ten digits from the MNIST dataset, each associated with a specific RGB color. Following the setting in Ahn et al. M1. ten RGB colors uniformly selected and applied to the grayscale MNIST images. The image generation involves assigning colors based on the predetermined bias-conflicting ratio, p. Specifically, each image is colored to create bias- bias-confficting sample with probability p. and bias-aligned sample otherwise. For bias-aligned samples of a given class y, the digit is colored using (C,, 013x3), where Cy R is the predefined color vector for class y. Conversely, bias-conflicting samples receive a color from any class other than i.e. Cv, E (e and colored using с ~ (Cu,, l3x3). The experiments utilize bias-confficting ratios 0.5%, 1 1%, and 5%, with set to 0.0001. Additionally, we use 10% of training data as validation data, and an unbiased test set with bias-conflicting ratio of 90% is employed for performance evaluation.
Colored MNIST.
Multi-bias MNIST. Multi-bian MNIST (MB-MNIST) is synthetic dataset derived from Ahn et al. 1 to incorporate more complex patterns compared to C-MNIST and biaSed-MNIST
[4 MB-
MNIST comprises eight attributes: digit (26), alphabet 77. fashion object [47]. Japanese character (6). digit color, alphabet color, fashion object color, and Japanese character color. The digit shape is the target attribute, while the other seven attributes serve bias attributes. In MB-MNIST, bias is introduced by aligning the digit and its color with probability 1 - p. Similarly, the other six attributes also independently aligned with the digit, each with probability
p. In our setting, the ratios of bias-cot chas-conffcctng samples are 10%, 20%, and 30% as in Ahn et al. [ As CMNIST, we use 10% of the training data as validation data. Test samples are generated with a bias-conflicting ratio of 90% for all bias attributes to create an unbiased test set.
Biased action recognition. The biased action recognition (BAR) dataset (34] designed for action classification, is real-world dataset that encompasses six action classes: climbing, diving, fishing, racing, throwing, and vaulting. Each class spuriously correlated with a specific place. The training set of BAR consists exclusively of bias-aligned samples, while the test set consists solely of bias- conflicting samples. For example, all the training samples for the climbing class are associated with rock walls, but the test samples for climbing are associated with different settings, such as ice cliffs.
15

---

We utilize the data splits defined in Ahn et al. M. which allocate 10% of the training set to the validation set.
Biased FFHQ. Biased FFHQ (BFFHQ) constructed from the real-world facial dataset FFHQ [2]. which has age (young or old) as a label and gender (male or female) as bias feature. This benchmark was conducted in previous works 27 22). Individuals aged between 10 and 29 labeled as "young", while those aged between 40 and 59 are labeled as "old". In this dataset, the majority of females young, while males are predominantly old. The training set contains only 0.5% bias-conflieting samples. We report accuracies on both an unbiased test set with bias-conflicting ratio of 50% [T] and a bias-conflicinn test [27
CelebA. CelebA [30] is real-world dataset for face classification. Following Sagawa et al. [38.
the goal is to classify celebrities' hair color as either blond or non-blond, which exhibits a spurious correlation with gender (male or female). Notably, only a small fraction of blond-haired celebrities are male, which leads to poor performance of ERM-trained models on this group. We employ the training, validation, and test splits as specified in Sagawa et al. 38 and report both the average accuracy and the worst-group accuracy on the test set. this context, the group is defined as the combination of the class label and the bias label as in prior studies 49 29 38
Coviletmme WILLS... CivilComments-WV WILLS 5 25 is a dataset for text classification prob- lems. The goal is to classify whether an online comment is toxic or non-toxic, which exhibits a spurious correlation with the mention of certain demographic identities, including male, female, white, black, LGBTQ, Christian, Muslim, and other religions. We use the same data splits as described in Koh et al. [25] and report both the average accuracy and the worst-group accuracy on the test dataset. Here, groups are defined as combinations class labels and bias labels, as described in previous
works 28 25 49).
B.2
Implementation details
We provide descriptions of the implementation details. For CelebA and CivilComment-WILDS we follow the experimental settings outlined in Zhang et [49] and Liu al. 28 respectively. For the other datasets CMNIST. MB- -NIITT, BAR, and BFFHQ-We we follow the experimental setups presented in Ahn al. M. The hyperparameter for GCE, denoted as q. to 0.7 as Zhang and Sabuncu [50]. All classification models are trained using an NVIDIA RTX A6000.
B.2.1 Model architectures and hyperparameters
The model architectures and hyperparameters for each dataset are described:
C-MNIST. As in Ahn et al. D, we utilize a simple CNN architecture (SimConv-I). Please see Appendix B of Ahn et al. [ for detailed network implementation. We train the model for 100 epochs with SGD optimizer, a batch size of 128, a learning rate of 0.02, weight decay of 0.001, momentum of 0.9, and learning rate decay of 0.1 at every 40 steps. For C-MNIST with bias-confficting ratios of 0.5% and 1,, we use a temperature of I; for ratio of 5%, we use a temperature L.I.
MB-MNIST. We use different type of simple CNN model (SimConv-22 following Ahn al. M. Please refer to Appendix B in Ahn et al. [T] for network implementation. We train for 100 epochs with SGD optimizer, a batch size of 32, a learning rate of 0.01, and weight decay of le-4, momentum of 0.9. For the MB-MNIST dataset, we use temperatures of 0.9, 1.1, and 1.3 for the ratios of 10%, 20%, and 30%, respectively.
BAR. We ResNet18 pretrained on ImageNet in Kim et al. [22] We train for 100 epochs with SGD optimizer, a batch size of 16, a learning rate of 5e-4, weight decay of le-5, momentum of 0.9, learning rate decay of 0.1 at every 20 steps, and temperature of 0.6.
BFFHQ. We utilize an Imageeet--retrrineed ResNet18 8 as in Lee et al. 27 We train for 160 epochs with Adam optimizer, a batch size of 64, a learning rate of le-4, weight decay of 0, learning rate decay of 0.1 at every 32 steps, and temperature of 0.9.
16

---

CelebA. We utilize ImageNett-rettrre ResNet50 and hyperparameters from Zhang et al. [49]. We train for 50 epochs with SGD optimizer, batch size of 128, a learning rate of le-4, weight decay of 0.1, and temperature of 1.
CivilComments-WWe WILDS. We utilize a pretrained BERT with the number of tokens capped at 300 [5]. We train the biased model using the SGD optimizer without gradient clipping. In contrast, for training the debiased model, we employ the AdamW optimizer with gradient clipping and linearl-deccayin learning rate. Both models are trained for up to 5 epochs with a batch size of 16, a learning rate of le-5, weight decay of 0.01, and a temperature of 2.
following previous works 44 28
B.2.2 Image preprocessing
We employ the same image preprocessing scheme as presented in Ahn et al. [ For CMNIST and MB-MNIST, we use fixed sizes of (28x28) and (56x56), respectively. For the remaining datasets, we use fixed size of (224x224). Additionally, we normalize images from BAR and BFFHQ with a mean of (0.4914, 0.4822, 0,4465) and standard deviation of (0.2023, 0.1994, 0.2010). For vision datasets except for CelebA, we apply random resize crop, random color jitter, and random rotation to increase the diversity of bias-conflicing samples. To ensure fair comparison, the same data augmentation techniques are applied to both the baselines and the proposed DPR.
C Additional ablation study
C.1 Ablation study on GCE
Table 4: Ablation study on q of GCE,
C-MNIST (0.5%)
MB-MNIST (30%)
Biased model
Debiased model
Biased model
Debiased model
0.3
26.27 (2.11)
95.53 (0.74)
79.73 (0.91)
95.32 (0.71)
0.5
21.30 (3.10)
96.83 (0.32)
83.16 (0.56)
94,84 (0.15)
0 0.7
18.55 (2,91)
97.52 (0.33)
85.47 (1.26)
(0.26)
0.9
16.19 (0.46)
97.66 (0.06)
85.26 (0.70)
93.77 (0.19)
We conduct ablation study the C-MNIST and MB-MNIST datasets, which have bias bias-confficting ratios of 0.5% and 30%, respectively, to assess the impact of GCE parameter 9. As depicted in Table 4 varying q demonstrates distinet effects on the performance of biased and debiased models. For C-MNIST, increasing y enhances the debiased model's 's performance while degrading that of the biased model. In contrast, for MB-MNIST, decreasing q enhances the performance of the debiased model but worsens the performance of the biased model. These results suggest that the optimal setting of q varies between datasets. Additionally, consistent observation from both datasets is that as the performance of the biased model decreases, the performance of the debiased model increases. This trend implies that lower performance of the biased model, indicative of a stronger focus on spurious correlations, leads to more accurate group proxies, which in turn boosts the debiased model's performance.
C.2 Ablation study on calibration hyperparameter
Table 5: Ablation study on calibration hyperparameter 7.
C-NNIST
MB-MNIST
Ratio (%)
0.5
5
20
30
0.9
97.48 (0.41)
97.96 (0.28)
88. (1.78)
91.53 (0.87)
97.52 (0.33)
98.38 (0.16)
88.68 (1.69)
92.85 (0.65)
96.48 (0.16)
98.62 (9.12)
89.11 (1.65
92.99 (0.66)
T 1.2
95.84 (0.46)
98.40 (0.18)
87.80 (2.49)
93.49 (0.47)
95.07 (0.36)
98.42 (0.02)
87.73 (1.53)
94.04 (0.26)
Since DPR utilizes the calibration hyperparameter 7 for capturing the spurious correlation structure well, we conduct an ablation study on C-MNIST and MB-MNIST across various bias-conflicting ratios to assess the effect r. Taableejillustraal how performance varies with different settings of т. For C-MNIST, the best performances are achieved at 5 1 and 1.1 for bias-conflicting ratios of
17

---

0.5% and 5%, respectively. For MB-MNIST, the best results are obtained with T 1.1 and 7 = 1.3 for ratios of 20% and 30%, respectively. The optimal r can vary not only between different datasets but also according to the extent of the prevalence of spurious correlations within the datasets. These results suggest that adjusting r is effective in capturing the diverse spurious correlation structures.
C.3 Comparison of resampling and reweighting
Table 6: Comparison of resampling and reweighting.
C-MNIST
Ratio (%)
0.5
Resampling
97.52 (0.33)
98 98.40 (0.03)
98.62 (0.
Reweighting 95.04 (0.35)
96.21 (0.45)
97.84 (0.50)
While reweighting could effectively solve the weighted loss minimization problem presented in Equation ( DPR adopts resampling. To justify this choice, we conduct experiments to compare resampling with reweighting. Table the performance of each method on the C-MNIST dataset across various cinfcconf ratios. The results reveal that DPR using the resampling strategy consistently outperforms DPR using the reweighting approach. This superiority resampling over reweighting has been explored and explained by An al. [2].
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yss]
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with
NeurlPS Code of Ethics httppp///...ee....c/////////ccca
Answer: [Yes]
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
19

---

Answer: [NA]
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
15.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
20


Reviews:
[{"review_id": "iCeF0Ti211", "reviewer": "Reviewer_qVjP", "summary": "This paper proposes a re-sampling approach based on disagreement between bias predictions and target label predictions. First, a biased model is trained using generalized cross-entropy. Then, sample-wise weights are determined by calculating the probability of disagreement between the bias predictions of the biased model and the target label predictions of the target model.", "questions": "Please see the Weaknesses", "limitations": "Please see the Weaknesses", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper tackles the significant and pressing issue of robust learning, which is crucial for effectively using trained models in real-world applications.\n\n2. The proposed sampling method is both simple and straightforward, making it easy to understand and implement.", "weaknesses": "1. The method extends the approach of upweighting misclassified samples in the existing Just Train Twice (JTT) by utilizing generalized cross-entropy (GCE) and the disagreement between bias predictions and target label predictions. However, this extension does not appear novel. Furthermore, do there not exist approaches that use bias predictions? If so, discussing these studies would provide context and underscore the novelty of the proposed method.\n\n2. Color jitter is employed as a data augmentation technique that can directly solve the color bias in CMNIST. However, the ablation study does not show significant performance improvement. What is the reason for this outcome? Additionally, for a fair evaluation, the same data augmentation should be applied to the baselines.\n\n3. CMNIST is a relatively easy dataset, and BAR is not widely used recently. Therefore, additional experiments on other widely used datasets, such as CIFAR-10C, Waterbird, or NICO, are needed. Demonstrating similar performance improvements on these datasets would better support the experimental effectiveness of the proposed method.\n\n4. The related works section does not discuss some recent studies, such as [1, 2].\n\n[1] Lee et al., \"Surgical fine-tuning improves adaptation to distribution shifts.\", ICLR'23\n\n[2] Jung et al., \"Fighting Fire with Fire: Contrastive Debiasing without Bias-free Data via Generative Bias-transformation\", ICML'23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We've carefully considered them and provided responses. Please let us know if you need any clarification or have additional questions.\n\n> **Q1**: The differences between the proposed DPR and other existing approaches using bias predictions. \n\n**A1**: As outlined in Section 2, several debiasing methods utilize biased models to identify bias-conflicting samples, including JTT [1], DFA [2], PGD [3], and LC [4]. Each method employs a unique approach:\n\n1. JTT upweights bias-conflicting samples by specifying their weights as a hyperparameter.\n\n2. DFA generates diverse bias-conflicting samples by mixing features between samples, using these to mitigate spurious correlations.\n\n3. PGD, a resampling-based method, determines upweighting based on the sample gradient of a biased model.\n\n4. LC mitigates spurious correlations by correcting the sample logit.\n\nOur proposed DPR method differs from these existing approaches in several key aspects: \n\n1. DPR is derived from a newly suggested learning objective (equation 3) that aims to improve performance for both bias-aligned and bias-conflicting groups while reducing the performance gap between them.\n\n2. In contrast, JTT's heuristic reweighting strategy may not consistently achieve this objective, potentially limiting its effectiveness in mitigating spurious correlations.\n\n3. PGD interprets debiasing as a min-max problem of minimizing loss for maximally difficult samples, which can be relaxed to minimizing the trace of inverse Fisher Information.\n\n4. LC focuses on debiasing by maximizing group-balanced accuracy.\n\nThese differing objectives distinguish our approach from existing methods. In section 6.3, we provide empirical evidence demonstrating that our proposed method outperforms other baselines including JTT, DFA, PGD, and LC across all six benchmarks.\n\n> **Q2**: Data augmentation setting and the impact of color jitter on mitigating color bias.\n\n**A2**: We would like to clarify that our experiments followed the same setup as Ahn et al. [3], including the data augmentation settings. This ensures a fair comparison, as the same data augmentation techniques were applied to all baseline models as well as our proposed method.\n\nRegarding the C-MNIST dataset results presented in Table 3 of our paper:\n\n1. Our model achieves high performance (95.94% and 97.66%) on C-MNIST with just initialization and the use of GCE loss, even without augmentation. This is due to the effectiveness of the biased model trained with GCE loss in identifying and oversampling bias-conflicting samples within the C-MNIST dataset, thus reducing the impact of color bias.\n\n2. As you correctly pointed out, incorporating data augmentation further addresses the color bias in C-MNIST. This additional step effectively mitigates the remaining bias, leading to significant performance improvements.\n\n> **Q3**: Additional experiments on other widely used datasets.\n\n**A3**: We conducted additional experiments on CIFAR-10C following the experimental settings of Ahn et al. [3] and reported the average accuracies (%) and standard deviations over three trials. The results are as follows.\n\n|            | CIFAR-10C (0.5%) | CIFAR-10C (1%) | CIFAR-10C (5%) |\n|:----------:|:----------------:|:--------------:|:--------------:|\n|     ERM    |   23.06 (1.25)   |  25.94 (0.54)  |  39.31 (0.66)  |\n|     JTT    |   25.34 (1.00)   |  33.62 (1.05)  |  45.13 (3.11)  |\n|     DFA    |   29.96 (0.71)   |  36.35 (1.69)  |  51.19 (1.38)  |\n|     PGD    |   30.15 (1.22)   |  42.02 (0.73)  |  52.43 (0.14)  |\n| DPR (Ours) |   32.20 (0.81)   |  43.77 (0.93)  |  53.10 (0.62)  |\n\nThe ratios shown in parentheses next to the dataset names in the table represent the bias-conflicting ratio in the training set. As shown in the table above, the proposed DPR consistently outperforms other baselines on CIFAR-10C with various bias-conflicting ratios.\n\n> **Q4**: Missing recent studies such as [5, 6] in the related works section.\n\n**A4**: Thank you for letting us know. We will make sure to incorporate these recent studies in the revised version.\n\n[1] Just train twice: Improving group robustness without training group information, ICML 2021\n\n[2] Learning debiased representation via disentangled feature augmentation, NeurIPS 2021\n\n[3] Mitigating dataset bias by using per-sample gradient, ICLR 2023\n\n[4] Avoiding spurious correlations via logit correction, ICLR 2023\n\n[5] Surgical fine-tuning improves adaptation to distribution shifts, ICLR 2023\n\n[6] Fighting Fire with Fire: Contrastive Debiasing without Bias-free Data via Generative Bias-transformation, ICML 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WqabHml1kF", "reviewer": "Reviewer_1RoL", "summary": "This paper addresses the critical issue of spurious correlations in machine learning models, where models often rely on easy-to-learn bias attributes rather than the intended target features, leading to poor generalization on minority groups where these spurious correlations are absent.\n\nThe authors propose a novel method called Disagreement Probability based Resampling for debiasing (DPR), which aims to mitigate the effects of spurious correlations without requiring explicit bias labels during training. The key contributions of this paper are:\n\n1. A new training objective designed to achieve consistent model performance across both bias-aligned and bias-conflicting groups, encouraging robustness against spurious correlations.\n\n2. Development of DPR, a practical resampling method derived from the proposed objective. DPR leverages the disagreement probability between the target label and the prediction of a biased model to identify and upsample bias-conflicting samples.\n\n3. Theoretical analysis demonstrating that DPR minimizes losses for both bias-aligned and bias-conflicting groups while reducing the disparity between their losses.\n\n4. Extensive empirical evaluation on six benchmark datasets (including both synthetic and real-world data) showing that DPR achieves state-of-the-art performance compared to existing methods that do not use bias labels.\n\n5. Ablation studies and analyses that provide insights into the effectiveness of various components of the proposed method, such as model initialization, generalized cross-entropy loss, and data augmentation.", "questions": "Nothing", "limitations": "Authors have discussed the limitations sufficiently.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "### Originality:\n\n1. Novel objective formulation: The authors introduce a new training objective designed to achieve consistent performance across bias-aligned and bias-conflicting groups. This approach differs from previous works that typically define groups based on combinations of target labels and bias labels.\n2. Creative use of disagreement probability: The method leverages the disagreement between target labels and biased model predictions as a proxy for identifying bias-conflicting samples. This is an innovative way to address the challenge of not having explicit bias labels.\n3. Unique combination of existing ideas: DPR creatively combines ideas from biased model training, resampling techniques, and theoretical analysis to create a cohesive and effective debiasing method.\n\n### Quality:\n\n1. Comprehensive empirical evaluation: The authors test their method on six diverse benchmarks, including both synthetic and real-world datasets, providing a thorough assessment of DPR's performance.\n2. Theoretical foundations: The paper includes a rigorous theoretical analysis that supports the empirical results, demonstrating how DPR reduces loss disparity between groups and minimizes average loss.\n3. Ablation studies: The authors conduct detailed ablation studies to understand the contribution of each component of their method, showing a commitment to thorough scientific investigation.\n4. Comparison with state-of-the-art: DPR is compared against multiple recent baselines, consistently showing superior or comparable performance.\n\n### Clarity:\n\n1. Clear problem formulation: The authors provide a clear definition of the problem and their approach to solving it.\n2. Step-by-step derivation: The development of DPR from the initial objective to the final algorithm is presented in a logical, easy-to-follow manner.\n3. Visual aids: The paper includes helpful figures and tables that illustrate key concepts and results.\n4. Detailed experimental setup: The authors provide comprehensive information about their experimental setup, facilitating reproducibility.\n\n### Significance:\n\n1. Addressing a crucial challenge: Spurious correlations are a major issue in machine learning, and this work provides a novel approach to mitigating them without requiring bias labels, which is often a practical constraint in real-world scenarios.\n2. Broad applicability: The method is shown to be effective across various types of data (image and text) and problem setups, suggesting its potential for wide adoption in different domains.\n3. Theoretical and practical relevance: By providing both theoretical guarantees and strong empirical results, the paper bridges the gap between theory and practice in addressing spurious correlations.\n4. Potential impact on fair ML: The proposed method could contribute to the development of fairer and more robust machine learning models, which is a critical goal in the field.", "weaknesses": "Nothing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for pointing out the strength and originality of DPR, which uses a loss function directly derived mathematically from setting an objective. We also appreciate your high regard for the various experiments, theoretical analyses, and ablation studies that support the effectiveness of DPR. We would like to express our deep gratitude once again for your positive review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3YbdaJqScA", "reviewer": "Reviewer_Het1", "summary": "The authors mainly target fairness without accessing bias labels. They suggest a new learning objective that minimizes the loss of the bias group showing the highest ERM and demonstrate that minimizing this objective decreases the upper bound of the expected average loss. To utilize this loss when the bias labels are not accessible, they derive the oversampling method from the new learning objective. In experiments, they show that their method outperforms baselines on various datasets.", "questions": "* According to Section 4.3, the authors introduce additional augmentation for the diversity of minor class samples. Did the other algorithms compared in the experiment also use these augmentation techniques?", "limitations": "The authors properly address the limitations in Section 7.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* They suggest a new learning objective and provide theoretical support for the proposed objective.\n* The authors derive the sampling weights from the perspective of the new learning objective.\n* They demonstrate that their method outperforms baselines on various benchmark datasets.", "weaknesses": "* Since the sampling weights are derived from the proposed learning objective, it's important to demonstrate the effectiveness of the proposed learning objective itself. Could you confirm if it is superior to GroupDRO in scenarios where bias labels are provided on CelebA?\n* The effectiveness of the proposed method compared to baselines seems weak. Except for the synthetic datasets and BFFHQ, their performance gain is marginal.\n* There is no clear explanation of how to choose hyperparameters or select the model from a specific epoch during training. In the absence of an unbiased validation set, such as BFFHQ, how were these selections made?\n* In situations where the ratio of bias-conflicting samples is low, using oversampling methods might make training unstable. Could you show how the test accuracy of bias-conflicting samples changes every 10 epochs during the whole training process on BFFHQ (160 epochs as mentioned in the Appendix)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We have provided answers to each comment. Please let us know if you need any clarification or have additional questions.\n\n> **Q1**: The effectiveness of the proposed learning objective itself.\n\n**A1**: As you mentioned, to demonstrate the effectiveness of the proposed learning objective, we conducted comparative experiments with GroupDRO in a scenario where bias labels are provided for CelebA. We followed the setting of CNC [1] and reported the average accuracies (%) and standard deviations over three trials. The results are as follows.\n\n|          | CelebA Average | CelebA Worst |\n|:--------:|:--------------:|:------------:|\n| GroupDRO |   93.9 (0.1)   |  88.9 (1.3)  |\n|   Ours   |   92.3 (0.8)   |  90.9 (0.6)  |\n\nAs shown in the table above, when bias labels are provided, the proposed learning objective (Ours) demonstrates excellent performance on CelebA. This clearly demonstrates the effectiveness of the objective.\n\n> **Q2**: The effectiveness of the proposed method compared to baselines.\n\n**A2**: The proposed DPR consistently outperforms or matches other baselines across 11 metrics on all six benchmarks. Considering that evaluations were conducted on different settings with various bias-conflicting ratios, types of bias, and data modalities, DPR's consistently highest performance across all benchmarks clearly demonstrates that it is more effective than any of the other baselines.\n\n> **Q3**: Hyperparameter and model selection criteria.\n\n**A3**: We followed the experimental settings of CNC [1] for CelebA, JTT [2] for CivilComments-WILDS, and PGD [3] for the remaining datasets, which include their hyperparameters. In other words, all hyperparameters are the same as those of the aforementioned works [1, 2, 3], except for the temperature hyperparameter newly introduced by our proposed method, DPR.\n\nFollowing existing debiasing papers [1, 2, 4, 5, 6], we selected the temperature and performed early stopping based on the best worst-group validation accuracy for CelebA and CivilComments, and the best validation accuracy for the remaining datasets.\n\n> **Q4**: The test accuracy on BFFHQ bias-conflicting samples every 10 epochs.\n\n**A4**: As you requested, we conducted an experiment to examine the changes in the test accuracy on bias-conflicting samples of BFFHQ. We reported the average accuracies (%) and standard deviations over three trials. The results are as follows.\n\n|      Epoch     |       1      |      10      |      20      |      30      |      40      |      50      |      60      |      70      |      80      |      90      |      100     |      110     |      120     |      130     |      140     |      150     |      160     |\n|:--------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|\n| BFFHQ Conflict | 76.67 (0.31) | 72.40 (3.80) | 74.87 (2.53) | 76.40 (0.92) | 76.60 (4.01) | 75.67 (3.07) | 75.00 (0.80) | 75.87 (1.68) | 75.60 (1.51) | 75.87 (1.36) | 76.07 (1.42) | 75.80 (1.51) | 75.87 (1.42) | 75.87 (1.86) | 75.80 (1.78) | 76.07 (1.62) | 76.07 (1.50) |\n\nAs can be seen in the table above, although the test accuracy shows somewhat unstable patterns during the training process, it achieves a test accuracy close to 76% for many epochs. This figure consistently outperforms other baselines.\n\n> **Q5**: Whether the same data augmentation was applied to other baselines.\n\n**A5**: We conducted our experiments using the same experimental setup as Ahn et al. [3], which includes the data augmentation explained in Section 4.3 of the paper. In other words, the same data augmentation technique was applied not only to our proposed DPR but also to all other baselines.\n\n[1] Correct-n-contrast: A contrastive approach for improving robustness to spurious correlations, ICML 2022\n\n[2] Just train twice: Improving group robustness without training group information, ICML 2021\n\n[3] Mitigating dataset bias by using per-sample gradient, ICLR 2023\n\n[4] Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, ICLR 2020\n\n[5] Learning from failure: De-biasing classifier from biased classifier, NeurIPS 2020\n\n[6] Learning debiased classifier with biased committee, NeurIPS 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1T77ZYw09S", "reviewer": "Reviewer_GDUK", "summary": "To address the issue of spurious correlations when bias labels are unavailable, the work proposes a new method to mitigate the spurious correlations by minimizing the maximum average loss over bias-aligned and bias-conflicting groups. Additionally, they introduce the disagreement probability/sampling probability, which weights samples in the training objective, to achieve group-robust learning.", "questions": "see my weaknesses.", "limitations": "see my weaknesses.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The organization of this paper is good, and it is written in a clear and understandable manner.\n2. This work is comprehensive as it covers both theoretical and experimental analyses.\n3. The experiments are extensive, considering a rich variety of datasets and baselines.", "weaknesses": "1. The method proposed in this paper, DPR, is not novel. Although the authors introduce several concepts such as sampling probability and group indicator, the essence of DPR can be seen as first using ERM with GCE to predict pseudo spurious attribute labels, and then applying the Group DRO method. Similar approaches (pseudo group labels from ERM + invariant learning) can be found in several methods like JTT [1] and CnC [2]. The essence of the author's emphasis on disagreement probability seems to be the predicted probability of ERM that a sample belongs to the minority group (i.e., bias-conflicting groups). Similarly, the reweighting of samples using the predicted probabilities by ERM can be seen in methods like EIIL [3] (the main difference is EIIL focuses more on the regularization term).\n\n2. The use of data augmentation helps alleviate spurious correlations, while other baseline models JTT do not utilize data augmentation, which may lead to unfair comparisons in experiments.\n\n3. The experimental section lacks unity, as evidenced by inconsistent metrics and experimental datasets. It is recommended to focus on the worst group (such as C-MNIST, MB-MNIST) to illustrate the effectiveness of DPR in mitigating spurious correlations. Additionally, I noticed that the average accuracy of CivilComments-WILDS decreases significantly compared to other baselines, which contradicts the theoretical claim in the article of simultaneously reducing the loss for bias-aligned and bias-conflicting groups. Can the authors provide insights into this?\n\n4. I appreciate the authors' demonstration of section  Identifying group via disagreement probability, which shows the effectiveness of biased models in predicting bias-conflicting groups. This is an important evidence for measuring the effectiveness of DPR. However, showcasing it only on C-MNIST is not sufficient. Can you provide experimental results on other datasets, especially real-world datasets?\n\nIf authors could provide reasonable replies, I am willing to further increase the score.\n\n[1] Liu, Evan Z., et al. \"Just train twice: Improving group robustness without training group information.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Zhang, Michael, et al. \"Correct-n-contrast: A contrastive approach for improving robustness to spurious correlations.\" arXiv preprint arXiv:2203.01517 (2022).\n\n[3] Creager, Elliot, Jörn-Henrik Jacobsen, and Richard Zemel. \"Environment inference for invariant learning.\" International Conference on Machine Learning. PMLR, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for your constructive comments. We have provided answers to each comment. Please let us know if you need any clarification or have additional questions.\n\n> **Q1**: Originality of DPR.\n\n**A1**: The following differences exist between JTT, CNC, and our proposed DPR. JTT identifies misclassified samples and uses heuristic reweighting to upweight them, which does not guarantee to encourage consistent model performance between bias-aligned and bias-conflicting groups, thus having little impact on mitigating the effect of spurious correlations. CNC is a debiasing method using contrastive learning, which aims to close the gap between worst-group loss and average loss. In contrast, our proposed method DPR aims to mitigate the impact of spurious correlations by reducing the performance gap between bias-aligned and bias-conflicting groups.\n\nThere is a difference in group definition between EIIL and DPR. Unlike EIIL, which partitions the dataset by maximizing the IRM objective, DPR distinguishes between bias-aligned and bias-conflicting groups based on the presence of spurious correlations, inspired by previous debiasing works [3, 5, 6]. Additionally, there is a difference in implementing a practical algorithm to train a debiased model without bias labels. EIIL optimizes the invariant learning objective, while DPR uses a weighted loss function directly derived from the proposed training objective with easily achievable assumption to reduce performance differences between groups and mitigate spurious correlations. The table below shows the performance of EIIL and DPR, with experiments following the CNC [1] setup, reporting average accuracies and standard deviations over three trials. \n\n|            | CelebA Average | CelebA Worst | CivilComments-WILDS Average | CivilComments-WILDS Worst |\n|:----------:|:--------------:|:------------:|:---------------------------:|:-------------------------:|\n|    EIIL    |   85.7 (0.1)   |  81.7 (0.8)  |          90.5 (0.2)         |         67.0 (2.4)        |\n| DPR (ours) |   90.7 (0.6)   |  88.9 (0.6)  |          82.9 (0.7)         |         70.9 (1.7)        |\n\nAs shown, DPR outperforms EIIL across various data modalities.\n\n> **Q2**: Data augmentation setting on other baselines.\n\n**A2**: The experimental setup we used for our experiments is the same as the experimental settings of CNC [1] for CelebA, JTT [2] for CivilComments-WILDS, and PGD [3] for the remaining datasets, which include data augmentation. Therefore, the same data augmentation technique was applied to all other baselines, including JTT, as well as our proposed algorithm.\n\n> **Q3**: lack of unity in metrics and contradictory results on CivilComments-WILDS.\n\n**A3-1**: Lack of unity in metrics\n\nWe appreciate your recommendation. As you suggested, we could focus on evaluating the worst group to show how effective DPR is in mitigating spurious correlations. We believe we can demonstrate this using existing experimental and dataset settings that focus on the worst group.\n\nTo demonstrate the effectiveness of DPR, we selected six datasets (two synthetic and four real datasets) widely used in existing debiasing papers. The metrics used in our experiments are those commonly employed in previous debiasing studies [1-4], chosen according to each dataset's specific setting.\n\nFor datasets not focused on the worst group, the bias-conflicting ratio in the training dataset is relatively low (such as 0.5%, 5%, or 10%), while the bias-conflicting ratio in the test set used for performance reporting is relatively high (such as 50%, 90%, or 100%). Therefore, we believe this provides a sufficient metric to evaluate how effectively a debiasing method mitigates spurious correlations.\n\n**A3-2**: Contradictory results on CivilComments-WILDS.\n\nThe practical algorithm DPR, derived from the training objective with an assumption, initializes the debiased model with a biased model and then trains it by oversampling bias-conflicting samples to obtain the debiased model. If, during the training process of oversampling bias-conflicting samples, the model forgets previously learned information about bias-aligned samples, this could result in an accuracy drop and consequently lower average accuracy. Unlike CelebA, where the average accuracy of DPR is higher than other baselines except ERM, a relatively lower average accuracy is reported in CivilComments-WILDS. This could be due to certain characteristics of the BERT model and language data. Analyzing and addressing these causes will be part of future work.\n\n> **Q4**: Group identification experimental results on real-world datasets.\n\n**A4**: We conducted an experiment on group identification via disagreement probability for the real-world dataset BFFHQ. The figures showing the experimental results have been uploaded as a PDF file in the global response. Please refer to it. As shown in the figures, for BFFHQ, bias-aligned samples generally exhibit smaller disagreement probabilities compared to bias-conflicting samples, effectively distinguishing between bias-aligned and bias-conflicting samples. Additionally, the relatively large disagreement probability of bias-conflicting samples allows DPR to effectively identify and upsample these bias-conflicting samples.\n\n[1] Correct-n-contrast: A contrastive approach for improving robustness to spurious correlations, ICML 2022\n\n[2] Just train twice: Improving group robustness without training group information, ICML 2021\n\n[3] Mitigating dataset bias by using per-sample gradient, ICLR 2023\n\n[4] Learning debiased representation via disentangled feature augmentation, NeurIPS 2021\n\n[5] Learning from failure: De-biasing classifier from biased classifier, NeurIPS 2020\n\n[6] Learning debiased classifier with biased committee, NeurIPS 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are very grateful to all the reviewers for their valuable comments.

The additional figures showing the experimental results of group identification for BFFHQ are in the PDF file.

Meta-review: The paper proposes Disagreement Probability based Resampling for debiasing to mitigate the effects of spurious correlations without requiring explicit bias labels. The procedure uses a biased model to identify bias-conflicting samples and upsamples them. 

The reviewers agreed the paper addresses a significant problem and appreciated the simplicity and practicality of the proposed solution. The procedure comes with both theoretical guarantees and extensive empirical results. The reviewers' concerns were sufficiently addressed in the author response and all were in favor of acceptance.
Decision: Accept (poster)