Mitigating Dataset Bias by Using Per-Sample Gradient
Authors: Sumyeong Ahn
Seongyoon Kim
Se-Young Yun
Venue: ICLR 2023 poster (2023)
Keywords: Dataset bias
Debiasing
Gradient-norm based debiasing

Abstract: The performance of deep neural networks is strongly influenced by the training dataset setup. In particular, when attributes with a strong correlation with the target attribute are present, the trained model can provide unintended prejudgments and show significant inference errors (i.e., the dataset bias problem). Various methods have been proposed to mitigate dataset bias, and their emphasis is on weakly correlated samples, called bias-conflicting samples. These methods are based on explicit bias labels provided by humans. However, such methods require human costs. Recently, several studies have sought to reduce human intervention by utilizing the output space values of neural networks, such as feature space, logits, loss, or accuracy. However, these output space values may be insufficient for the model to understand the bias attributes well. In this study, we propose a debiasing algorithm leveraging gradient called Per-sample Gradient-based Debiasing (PGD). PGD is comprised of three steps: (1) training a model on uniform batch sampling, (2) setting the importance of each sample in proportion to the norm of the sample gradient, and (3) training the model using importance-batch sampling, whose probability is obtained in step (2). Compared with existing baselines for various datasets, the proposed method showed state-of-the-art accuracy for the classification task. Furthermore, we describe theoretical understandings of how PGD can mitigate dataset bias. 


Published as a conference paper at ICLR 2023
MITIGATING DATASET BIAS BY USING PER-SAMPLE GRADIENT
Sumyeong Ahn KAIST Al sumysongghn@haisst.cckkk
Seongyoon Kim KAIST ISysE
Se-young Yun
KAIST AI
yunseyoung@kisst.ac.kk
ABSTRACT
The performance of deep neural networks is strongly influenced by the training dataset setup. particular, when attributes with a strong correlation with the target attribute are present, the trained model can provide unintended prejudgments and show significant inference errors (i.e., the dataset bias problem). Various methods have been proposed to mitigate dataset bias, and their emphasis is on weakly correlated samples, called bias-confficting samples. These methods are based on explicit bias labels provided by humans. However, such methods require human costs. Recently, several studies have sought to reduce human intervention by utilizing the output space values of neural networks, such feature space, logits, loss, or accuracy. However, these output space values may be insufficient for the model to understand the bias attributes well. In this study, we propose a debiasing algorithm leveraging gradient called Per-sample Gradient-basse Debiasing (PGD). PGD is comprised of three steps: (1) training model on uniform batch sampling, (2) setting the importance of each sample in proportion to the norm of the sample gradient, and (3) training the model using importance-batce
sampling, whose probability is obtained in step (2). Compared with existing baselines for various datasets, the proposed method showed state-of-her the-ahr art accuracy for the classification task. Furthermore, describe theoretical understandings of how PGD mitigate dataset bias.
INTRODUCTION
Dataset bias (Torralba & Efros 2011 Shrestha al. 2021 is bad training dataset problem that occurs when unintended casier-to-leam attributes (i.e., bias attributes), having high correlation with
the target attribute, are present (Shah et al. 2020 Ahmed et al., 2020). This due the fact that the model can infer outputs by focusing the bias features, which could lead to testing failures. For example, most "camel" images include a "desert background," " and this unintended correlation can provide false shortcut for answering "camel" on the basis of the "desert." In (Nam et al.) 22020] Lee ct al. 202T), samples of data that have strong correlation (like the aforementioned desert/camel) are called "bias-aligned samples," while samples of data that have weak correlation (like "camel on the grass" images) are termed "bias-conflictnng samples."
To reduce the dataset bias, initial studies (Kim al. 2019 McDuff et al. 2019 Singh al. 2020
Li & Vasconcelos, 2019) have frequently assumed a case where labels with bias attributes are provided, but these additional labels provided through human effort expensive. Alternatively, the bias-type, such as "background," " is assumed in (Lee et al., 2019 Geirhos et al. 20188] Bahng al. 2020 Cadene et al., 2019 Clark et al. 2019). However, assuming biased knowledge from humans is still unreasonable, since even humans cannot predict the type of bias that may exist in large dataset (Schäfer 2016). Data for deep learning is typically collected by web-crawling without thorough consideration of the dataset bias problem.
"gqual contribution

---

Published as a conference paper at ICLR 2023
Recent studies (Le Bras et al. 2020 Nam al. 2020 Kim al. 2021 Lee al. 2021: Seo et al. 2022] Zhang et al. 2022b) have replaced human intervention with DNN results. They have identified bias-conflicting samples by using empirical metrics for output space (e.g.. training loss and accuracy). For example, Nam et al. (2020) suggested a "relative difficulty" based on per-sample training loss and thought that sample with high "relative difficulty" was bias-conflicting sample. Most of the previous research has focused on the output space, such as feature space (penultimate layer
output) (Lee et al., 202T Kim et al., 202T; Bahng et al. [2200] [eco al 2022 Zhang et al.) 2022b), loss Nam et al. 2020), and accuracy (Le Bras et al. [2000] Liu et al. (021). However, this limited
output space can impose restrictions on describing the data in detail.
Recently, as an alternative, model parameter space (e.g., gradient (Huang et al. 2021 Ki Kilssteety
jtt al. 2021b Mirzasoleiman et al. 2020) has been used to obtain high-performance gains compared
to output space approaches for various target tasks. For example, HHuang et al.] (2021) used gradient-
norm to detect sutoo-diistrbbutinn detection samples and showed that the gradient of FC layer E Ax
could capture joint information between feature and softmax output, where h and are the dimensions of feature and output vector, respectively. Since the gradients of each data point Rhxe constitute high-dimensional information, it is much more informative than the output space, such logit R and feature E R However, there is no approach to tackle the dataset bias problem using a gradient norm-based metric.
In this paper, we present a resampling method from the perspective of the per-sample gradient norm to mitigate dataset bias. Furthermore, we theoretically justify that the gradient-norm-baase resampling method can be an excellent debiasing approach. Our key contributions can be summarized as follows:
We propose Per-sample Gradient-norm based Debiasing (PGD), a simple and efficient gradient- norm- based debiasing method. PGD is motivated by prior research demonstrating (Mirzasoleiman 2020: Huang et al. [202]] Killamsetty et al.) (2021b) that gradient effective finding rare ct al.
samples, and it is also applicable to finding the bias-conflicting samples in the dataset bias problem (See Section g and Appendix FD..
PGD outperforms other dataset bias methods on various benchmarks, such as colored MNIST (CM- NIST), multi-bias MNIST (MBMNIST), corrupted CIFAR (CCIFAR), biased action recognition In particular, for the colored MNIST case, the proposed method yielded higher unbiased test accuracies compared with the vanilla and the best methods by 35.94% and 2.32%, respectively. (See Section
(BAR). biased FFHQ (BFFHQ), CelebA, and CivilComments-WII
We provide theoretical evidence of the superiority of PGD. To this end, we first explain that minimizing the trace of inverse Fisher information is good objective to mitigate dataset bias. In particular, PGD, resampling based on the gradient norm computed by the biased model, is a possible optimizer for mitigating the dataset bias problem. (See Section
2 DATASET BIAS PROBLEM
Classification model. We first describe the conventional supervised learning setting. Let us consider the classification problem when training dataset D, { 3/)) with input image x; and corresponding label y4, is given. Assuming that there are c E IN \ {1} classes, y/ is assigned to the one element in set C = {1, .1,,.... c). Note that focus on situation where dataset D, does not have noisy samples, for example, noisy labels or outtr--distributtin samples (e.g.. SVHN samples when the task CIFAR-IO). When input x; is given. f zz; 0) represents the softmax output of the classifier for label y;- This is derived from the model parameter 0 E I2, The cross-entropy (CE) loss LCE is frequently used to train the classifier, and it is defined CCu( D6 0) log ( J.,, when the label is one-hot encoded.
Dataset bias. Let us suppose that training set, Dn, is comprised of images, as shown in Figure []] and that the objective is to classify the digits. t.s. Each image can be described by a set of attributes, (e.g., for the first image in Figure it can be (digit O, red, thin....)). The purpose of the training classifier is to find model parameter 0 that correctly predicts the target attributes, (e.g., digit.). Notably, the target attributes are also interpreted as classes. However, we focus that is strongly correlated to the target exists, and we call these attributes bias attributes. For example, in Figure 1 the bias attribute is color Furthermore, samples
2
&
O
2
3
4
5
7
8
2
3
4
6
7
6
3
5
6
7
9
case wherein another attribute
Figure 1: Target and bias attribute: digit shape, color.
2

---

Published as a conference paper at ICLR 2023
whose bias attributes are highly correlated to the target attributes are called bias-aligned (top three rows in Figure D. Conversely, weakly correlated samples are called bias-confiicting (see the bottom row of Figure IT. Therefore, our main scope is that the training dataset which has samples whose bias and target attributes are misaligned. [ According to (Nam et al. 2020), when the bias attributes easier-to-learn than the target attributes, dataset bias is problematic, prioritize the bias attributes over the target attributes. For example, for a model trained on the images Figure I it can output class when the (Orange, 0) image (e.g,, left bottom image) given, due to the wrong priority, color which is an easier-to-leam attribute (Nam et al. 2020
the trained model may
3 PGD: PER-SAMPLA
GEADIENT-NORM-BASE DEBIASING
In this section, we propose a novel debiasing algo- rithm, coined PGD. PGD consists of two mod- els, biased fo and debiased fa, with parameters 0, and 8a. respectively. Both models trained se- quentially. Obtaining the ultimately trained debiased model fa involves three steps: (1) train the biased model, (2) compute the sampling probability of each sample, and (3) train the debiased model. These steps are described in Algorithm] !
Algorithm PGD: Per-sumple Gradient-norm based Debiasing
I: Input: dataset Dr,, leurning rate n, iterations Ti. Ta. Butch
size B. Data augmentation operation A()). Initial parame-
ter 9b. GCE parameter cx
/** STEP 1: Train fo **/
2: for t= 1,2, ... .T, do
Construct mini-batch B, = ((2), B.]] ~ U. Update 0, as:
8i-1 V. E (c
5: end for
/*a STEP 2: Calculate *a/
6: Calculate h(f;. for all (s. E D.. [
Step 1:
Training the biased model. In the first step, the biased model is trained on the mini-batches 8: sampled from uniform distribution U. similar to conventional SGD- based training. with data augmen-
/++ STEP 3: Train fa based on *+/
for 1,,,, .T, do
Construct mini 8 = (F h.
9:
Update 01,.. as:
0n # &o CeE(A(s), v 07.4-1)
tation A. The role of the biased model twofold: 10: end
it detects which samples are bias bias-conflicting and calculates how much they should be highlighted. Therefore, we should make the biased model uncertain when it faces bias-conflicting samples. In doing so, the biased model, fo. is trained on the generalized cross-entropy (GCE) loss LOCE to amplify the bias of the biased model, motivated by (Nam et al. 2020). For an input image x, the corresponding true class y. Loce is defined as COCE(æ, y; 0, a) = - Note that E (0, 1] a hyperparameter that controls the degree of emphasizing the easy-to-learn samples, namely bias- aligned samples. Note that when a + 0, the GCE loss LOCE is exactly the same as the conventional CE loss LCE. We set a = 0.7 as done by the authors of Zhang & Sabuncu (2018), Nam et al. (2020) and Lee et al. (2021).
Step 2: Compute the gradient-based sampling probability. In the second step, the sampling probability of each sample is computed from the trained biased model. Since rare samples have large gradient norms compared to the usual samples at the biased model (Hsu et al. 2020), the sampling probability of each sample is computed to be proportional to its gradient norm so that bias-confficting samples are over-sampled. Before computing the sampling probability, the per-sample gradient with respect to LCe for all (ar, y4) D, is obtained from the biased model. We propose the following sampling probability of each sample h( 3i) which is proportional to their gradient norms, as follows:
[ ys: 0b)ll;
h(1s, vi) = - (E..))) V VoLce( ye: 00)l
(1)
where || denotes square of the T., norm, and 0, is the result of Step 1. Since, hx,,, y:) is the sampling probability on D,, h(2,, y:) is the normalized gradient-norm. Note that computing the gra- dient for all samples requires huge computing resources and memory. Therefore, we only extract the gradient of the final FC layer parameters. This is a frequently used technique for reducing the computa- tional complexity (Ash al. 2019 Mirzasoleiman et al., 2020 Killamsetty al.]
2021b 2020). In I[V, Las MC..6
other words, instead of h(,,, y.), we empirically utilize h(,, v.) PCp IV nn Len(=..6:00)1
where 0R is the parameters of the final FC layer. We consider r = 1 and & = 2 (Le. T2), and deliver ablation studies on various and s in Section
Step 3: Ultimate debiased model training. Finally, the debiased model, fa. is trained using mini-batches sampled with the probability h(s2, yi) obtained in Step 2. Note that, as described in
Note that bias-alignment cannot always be strictly divisible in practice. For case of explanation, we use the notations bus-cooffieiinn/baa-aiggged
3

---

Published as a conference paper at ICLR 2023
Algorithm| 1 our debiased model inherits the model parameters of the biased model pz.- However, Lee et al. (2021) argued that just oversampling bias-conflicting samples does not successfully debias, and this unsatisfactory result stems from the data diversity (i.e., data augmentation techniques required). Hence, we used simple randomized augmentation operations A such as random rotation and random color jitter to oversample the bias-conflicting samples.
4 EXPERIMENTS
In this section, demonstrate the effectiveness of PGD for multiple benchmarks compared with previous proposed baselines. Detail analysis not in this section (e.g., training time, unbiased case study, easier learn target attribute, sampling probability analysis, reweighting with PGD) are described in the Appendix E
4.1 BENCHMARKS
To precisely examine the debiasing performance of PGD, we used the Colored MNIST, Multi-bias MNIST, and Corrupted CIFAR datasets synthetic datasets, which assume situations in which the model learns bias attributes first. BFFHQ, BAR, CelebA, and CivilComments- WWILD datasets obtained from the real-world are used to observe the situations in which general algorithms have poor performance due to bias attributes. Note that BFFHQ and BAR are biased by using human prior knowledge, while CelebA and SenCoomments-WLLDDD are naturally biased datasets. A detailed explanation of each benchmark are presented in Appendix A
Colored MNIST (CMNIST). CMNIST is modified version of MNIST dataset (LeCun et al. 2010). where col color or is the biased attribute and digit t. serves as the target. We randomly selected ten colors that will be injected into the digit. Evaluation was conducted for various ratios p E {0.5%, 1 1%, 5%), where denotes the portion of bias-conflicing samples. Note that CMNIST has only one bias attribute: color.
Multi-bias MNIST (MB-MNIST). The authors of (Shrestha et al. 202 stated that CMNIST is too simple to examine the applicability of debiaising algorithms for complex bias cases. However, the dataset that Shrresha et al. (2021) generated is also not complex, since they did not use an real-world pattern dataset (e.g.. MNIST) and used simple artificial patterns (e.g., straight line and triangle). Therefore, we generated a MB- MB-MNIST; we used benchmark to reflect the real-worled better than (Shrestha et al. (022)) D. MB-MNIST consists of eight attributes: digit (LeCun et al. 2010), alphabet (Cohen et al. (017). fashion object (Xiao et al.) 2017). Japanese character (Clanuwat et al., 2018), digit color, alphabet color, fashion object color, Japanese character color. Among the eight attributes, the target attribute is digit shape and the others are the bias attributes. To construct MB- MNIST, we follow the CMNIST protocol, which generates bias by aligning two different attributes (i.e., digit and color) with probability (1 p). MB-MNIST dataset is made by independently aligning the digit and seven other attributes with probabity (1 p). Note that rarest sample is generated with probability p². When p is set as the CMNIST case, too low generate sufficient misaligned samples. Therefore, we use E {10%, 20%, 30%} to ensure the trainability.
Corrupted CIFAR (CCIFAR). CIFARIO (Krizhevsky et al. 2009) is comprised of ten different objects, such as an sirplane and a car. Corrupted CIFAR are biased with ten different types of texture bias (e.g., frost and brightness). The dataset was constructed by following the design protocol of (Hendrycks & Dietterich 2019), and the ratios p E (0.5%, 1%, 5%} are used.
Biased action recognition (BAR). Biased action recognition dataset was derived from (Nam et al. 2020). It comprised six classes for action, (e.g., cl climbing and di diving), and each class is biased with place. For example, diving class pictures are usually taken underwater, while a few images are taken from the diving pool.
Biased FFHQ (BFFHQ). BFFHQ dataset was constructed from the facial dataset, FFHQ Karras et al., 2019). It was first proposed in (Kim al.) 2021) and was used in (Lee et al. 2021) ). It is comprised of two gender classes, and each class is biased with age. For example, most female pictures young while male pictures are old. This benchmark follows p 0.5%.
CelebA. CelebA (Liu et al. 2015) is a common real-world face classification dataset. The goal is classifying the hair color ( ("blond" and "not blond") of celebrities which has spurious correlation with the gender ("male" or "female") attribute. Hair color of almost all female images is blond. We report the average accuracy and the worst-group accuracy on the test dataset.
4

---

Published as a conference paper at ICLR 2023
Table 1: Average test accuracy and standard deviation (three runs) for experiments with the MNIST variants under various bias conflict ratios. The best accuracy is indicated in bold for each case.
Dataset
e
Vanilla
LIF
JTT
Disen
PGD (Ours)
0.5%
60.94 207
91.35=
85.84=
94.56=as
96.88=
CMNIST
1%
79.13= 073
96.88 020
95.07
96.87=
98.35
5%
95.12= 924
98. 18
96.562 1.3
98.35=
98.62 614
10%
25.23= 116
19.18=
25.34 145
2
61.38 4.11
MBMNIST
20%
62.06 2.15
65.721 6:33
61.62.2.M 2m0
89.09 6.9
30%
87,61
160
89,89 1.76
85,44= 3,44
88.36= 2.06
90.76- 1.54
0.5%
23.06 125
28.83 1:30
25.34 1.00
29.96 win
30.15 = 1.21
CCIFAR
1%
25. 94 2.4
33.33 0.15
33.62 1.00
36.35=1.e 1.00
42.02 0.73
5%
39.31 MAs
50.24 1:11
km
51.1911.4 1.76
52.43 0.14
Table 2: Average test accuracy and standard deviation (three runs) for experiments with the raw image benchmarks: BAR and BFFHQ. The best accuracy is indicated in bold, and for the overlapped best performance case is indicated in Underline.
Dataset
Vanilla
LIF
JTT
Disen
PGD (Ours)
HAR
63.15 rss
64.41 130
63.62 a
64. 64g70z nos
65_39_ a.Ir
BETHQ
77.77
82 on
82,77= 2 2.6
140
84,20 LIF
CivilComments-Win
CivilComments-WInD (Borkan al. 2019) is dataset to classify
whether an online comment is toxic or non-toxic. The mentions of certain demographic identities (male, female, White, Black, LGBTQ, Muslim, Christian, and other religion) cause the spurious correlation with the label.
4.2 IMPLEMENTATION.
Baselines. We select baselines available for the official code from the respective authors among debiasing methods without prior knowledge on the bias. Our baselines comprise six methods on the various tasks: vanilla network, LIF (Nam et al. 2020). JTT (Liu et al. 2021 2 Disen (Lee et al.
2021 EIIL (Creager et al. 202T) and CNC (Zhang et al. 2022b).
Implementation details. We use three types of networks: two types of simple convolutional networks (SimConv-1 and SimConv-2) and ResNet18 (He et al.) 2016). Network imeplementation described in Appendix [B] Colored MNIST is trained on SGD optimizer, batch size 128, learning rate 0.02, weight decay 0.001, momentum 0.9, learning rate decay 0.1 every 40 epochs, 100 epochs training, and GCE parameter 0.7. For Multi-bias MNIST, it also utilizes SGD optimizer, and 32 batch size, learning rate 0.01, weight decay 0.0001, momentum 0.9, learning rate decay 0.1 with decay step 40. It runs 100 epochs with GCE parameter 0.7. For corrupted CIFAR and BFFHQ, it uses ResNet18 as backbone network, and exactly the same setting presented by Disen (ee et al. 2021 For CelebA, we follows experimental setting of (Zhang al. 2022b) which uses ResNet50 as backbone network. For CivilComments- -ertommmeettt-LDSS..... we utilize exactly the same hyperparameters of (Liu et al. (021) and utilize pretrained BERT. To reduce the computational cost in extracting the per-sample gradients, use only fully connected layer, similar to (Ash et al. 2019 Mirzasoleiman et al., (2020] Killamsetty ctt al., 22211b at 2020). Except for CivilComments- WILDS and CclebA, we utilize data augmentation, such as color jitter, random resize crop and random rotation. See Appendix B for more details.
4.3 RESULTS AND EMPIRICAL ANALYSIS
Accuracy results. In Table I we present the comparisons of the image classification accuracy for the unbiased test sets. The proposed method outperforms the baseline methods for all benchmarks and for all different ratios. For example, our model performance is 35, 35.94% better than that of the vanilla model for the colored MNIST benchmark with a ratio p = 0.5%. For the same settings, PGD performs better than Disen by 2.32%.
As pointed out in (Shrestha al. 2021), colored MNIST is too simple evaluate debiasing perfor- mance on the basis of the performance of baselines. In Multi-bias MNIST case, other models fail to
In the case of JTT Liu et al. 2021 although the authors used bias label for validation dataset (especially, hias-conflicting samples), we tune the hyperparameters using part of the biased training dataset for fair comparison. Considering that JTT does not show significant performance gain the results, it consistent with the existing results that the validation dataset is important in JTT, as deserihed in [Idrissi et al. (202)..
Lee al. 2021) only reported bias-conflicting case for BFFHQ, but we report the unbiased test result.
5

---

Published as a conference paper at ICLR 2023
Table Average and worst test accuracy with the raw image benchmark: CelebA and raw NLP task: CoiCCommmtt-WLLDSS The results of comparison algorithms are the results reported in (Zhang al. 2022b). The best worst accuracy is indicated in bold.
Vanilla
LIF
EILL
J1T
CNC
Ours
Avg.
94.9
85.1
85.7
88.1
88.9
88.6
CelebA
Worst
47.7
77.2
81.7
81.5
88.8
88.8
Avg.
92.1
92.5
90.5
91.1
81.7
92.1
CivilComments
Worst
58.6
58.8
67.0
69.3
68.9
70.6
B L
L2
100
100
Unblased Acc.
Unbissed Acc.
Unblasod Act.
80
60
30
0.5%
11
5%
10%
20%
30%
0.5%
1%
5%
blas conflicting
Portion contlicting
conflctinng
(a) Colored MNIST
(b) Multi-bias MNIST
(c) Corrupted CIFAR
Figure 2: Average PGD results for various of norms, {L, L2, L2, ). for the feature-injected
benchmarks. The error bars represent the standard deviation of three independent trials.
obtain higher unbiased test results, even though the ratio is high, e.g., 10%. In this complex setting, PGD shows superior performance over other methods. For example, its performance is higher by 36.15% and 35.63% compared with the performance of vanilla model and Disen for the ratio of 10%.
Similar to the results for the jias-feature-ineeted benchmarks, shown in Table 2 and Table PGD shows competitive performance among all the debiasing algorithms on the raw image benchmark (BAR, BFFHQ, and CelebA). For example, for the BFFHQ benchmark, the accuracy of PGD is 1.43% better than that of Disen. As in Table PGD outperforms the other baselines on CivilComments- WILDs, much more realistic NLP task. Therefore, we believe PGD also works well with transformer, and it is applicable to the real-world.
Unbiased test accuracy on various norms. Since, gradient norm can have various candidates. such as order of the norm, we report four configurations of gradient norms. As shown in Figure 2 all norms have significant unbiased test performance. Amongst them, the L2-morm square case shows lower unbiased performance than the other cases. Therefore, it is recommended that any first power of L4,,2,00 -norms be used in PGD for overcoming the dataset bias problem. This is quite different from the results in (Huang et al. 2021), which suggested that -nom is the best choice in the research field of di-striuittittt detection.
--horms
Table 4: Ablation studies on GCE and data augmentation (v for applied case).
Ablation study. Table 4 shows the importance of each module in our method: generalized cross entropy, and data augmentation modules. We set the ratio P as 0.5% and 10% for CMNIST and MB-MNIST, respectively. We ob- serve that GCE is more important that data augmentation for CMNIST. However, data augmentation shows better performance than GCE for MB-MNIST. In all cases, the case where both are utilized outperforms the other cases.
GCE
Aug. CMNIST (0.5%)
MB ENIST (10%)
84,93 1.2
40 40.58 2321
93.18 2:
45.70 1:1
46.70
96.88 ga
61.38 aan
5 MATHEMATICAL UNDERSTANDING OF PGD
This section provides a theoretical analysis of per-sample gradient-nore based debiasing. We first briefly summarize the maximum likelihood estimator (MLE) and Fisher information (FI), which are ingredients of this section. We then interpret the debiasing problem as a min-max problem and deduce that solving it the min-max problem can be phrased minimizing the trace of the inverse FL. Since handling the trace of the inverse Fl is very difficult owing to its inverse computation, we look at glance by relaxing it into one-dimensional toy example. In the end, conclude that the gradient-norm based re- remmannling method is an attempt to solve the dataset bias problem.
5.1 PRELIMINARY
Training and test joint distributions. The general joint distribution P(x, y/() is assumeed to be factored into the parameterized conditional distribution [w) ( and the marginal distribution P(x:)

---

Published as a conference paper at ICLR 2023
which is independent of the model parameter 8, (i.e., P(z, y/0) = P(z)f 8)). We refer to the model [y/2 )*) that produces the exact correct answer, as an oracle model, and to its parameter @" the oracle parameter. The training dataset D,, is sampled from {[I,, B:) where the training and test marginal distributions are denoted by p(z) and g(z), respectively. Here, assume that both marginal distributions are defined on the marginal distribution space M (P(z))]eez P(x) dr = 13, where X' means the input data space, i.e., p(z), ((x) E M.
The space H. of sampling probability h. When the training dataset D. is given, we denote the sampling probability as h(x) which is defined on the probability space 7
H {h(x)) Σ E(s...)D, h(xz)) = 1, h(x)) 2 V( Vo) E D.).
(2)
Maximum likelihood estimator (MLE). When h(x) is the sampling probability, we define MLE
(n(x).D. follows:
(n(a)),.. = arg min - E (2..:)) h(x,) log (2,]),))))
Note that MLE (.()), is variable controlled by two factors: (1) change in the training dataset
Dn and (2) the adjustment of the sampling probability h(x)) If hi(z)) is uniform distribution Ux).
then (u()).D, is the outcome of empirical risk minimization (ERM).
Fisher information (FT). FI, denoted by (0)))()), is information measure of samples from given distribution P(x,, v/0). It defined as follows:
Ip(=) (0) ((ce:))-P(ee//(lloo)) [Va log f(vl,, V log [yll,, 0)].
FI provides a guideline for understanding the test cross-entropy loss of MLE (r. When the training set is sampled from p(x) [(yy 0*) and the test samples are generated from q()) ( we can understand the test loss of MLE (z) by using Fl as follows.
Fad Theorem 1. Suppose Assumption 1 in Appendix and Assumption 2 in Appendix G Chold, then for sufficiently large n = 'Dul, the following holds with high probability:
In Tr [ote)) Tr [[ae))(0))] (4) (aaa))--((a)(ull...)))) ED.-sta)/iil..or [- log loggff (w) (uc)), <
Here is the proof sketch. The left-hand side of (4) converges to the Fisher information ratio (FIR) Tr (0*) (0at))()))(()((0))))) -related term. Then, FIR can be decomposed into two trace terms with respect to the training and test marginal distributions p(x) and q(x)). Finally, we show that the term Tr(4p(x)(0*) (0+)- which is defined in the oracle model parameter can be replaced with Tr(4p(=) (.)(.), ".. The proof of Theorem is in Appendix D. Note that Theorem I means that the upper bound of the test loss of MLE (1) can be minimized by reducing (t(pp(a) (Pu()))).)))
Empirical Fisher information (EFI). In practice, the exact FI 33) cannot be computed since we do not know the exact data generation distribution ffv x. 0). For practical reasons, the empirical Fisher information (EFI) is commonly used (Jastrzębski et al. 2017 Chaudhari et al., (2099) to reduce the computational cost of gathering gradients for all possible classes when is given. In the present study, we used a slightly more generalized EFI that involved the sampling probability h(x) E as follows:
(5) Note that the conventional EFI is the case when h(x) is uniform. EFI provides guideline for understanding the test cross-entropy loss of MLE (n(e).
- (E,,) h(x,) V log f(y T log f(yllz,,)) 0).
5.2 UNDERSTANDING DATASET BIAS PROBLEM VIA MIN-MAX PROBLEM
Debiasing formulation from the perspective of min-max problem. We formulate the dataset bias problem as described in Definition 66 is a min-max problem formula, a type of robust optimization. Similar problem formulations for solving the dataset bias problem can be found in (Arjovsky et al. 2019 Bao et al. 2021 Zhang et al. 2022a). However, they assume that the training data is divided into several groups, and the model minimizes the worst inference error of the reweighted group dataset. In contrast, the objective of (6) minimizes the worst-case test loss without explicit data groups where the test distribution can be arbitrary.
NNte that for simplicity, we abuse the notation h( v) used in Section 3 h(z). This is exactly the same for given dataset D, situation.
7

---

Published as a conference paper at ICLR 2023
Definition 1. When the training dataset D, ~ (1) is given, the debiasing objective is
| log flyl Bn()).
min max (e)a)) M q())eM
(6)
The meaning of Definition is that we have to train the model O,( that the loss of the worst case test samples (m)X)(a)) is minimized by controlling the sampling probability h(() (min
Note that since we cannot control the given training dataset D, and test marginal distribution g(z), the only controllable term is the sampling probability h(z)) Therefore, from Theorem and EFI, design practical objective function for the dataset bias problem as follows:
min TF [mocs(0
h(z)EH
5.3 MEANING OF PGD TERMS OF
In this section, we present an analysis of PGD with respect to 7. To do so, we try to understand which is difficult to directly solve. because computing the trace of the inverse matrix is computa- tionally expensive. Therefore, we intuitively understand (7) in the one-dimensional toy scenario.
One-dimensional example. We assume that D, comprises sets M and m such that elements in each set share the same loss function. For example, the loss functions of the elements in set M and m are (0 + a)² and 2 (0 - a)² with a given constant a, respectively. We also assume that each sample of M and m has the set dependent probability mass M (x) and hmm respectively. With these settings, objective is to determine h" (ar) arg mi)e( Tr( (..))) Thanks the model's simplicity, we can easily find h (x) in closed form with respect to the gradient at (,),)) for each set, i.e., gM (00()))) and gm (0U.).)).
Theorem 2. Under the above setting, the solution of (hir(z), (x)) =
arg m Tr [i/ (O((s))
(x) = 1gM (D((0 (x) = 19m (0mm(00cs)
where Z M 19 ( (..))) + |m| 1gm (fur())))))) and [M] | and |m] denote the cardinality of M
and m, respectively.
The proof of Theorem is provided in Appendix E. Note that hir(x) (x) and h (r) are computed using the trained biased model with batches sampled from the uniform distribution U Uxx). It is the same with the second step of PGD.
PGD tries to minimize 77). Theorem implies that (7) can be minimized by sampling in propor- tion to their gradient norm. Because the basis of PGD is oversampling based on the gradient norm from the biased model, we can deduce that PGD strives to satisfy (7). Furthermore, we empirically show that PGD reduces the trace of the inverse of EFI in the high-dimensional case, as evident in Figure]
Vanills
PGD
mlla (aan I
Begg detge
201 Meter Catoe
contict
Da corfiid meto
(a) Colored MNIST
(b) Multi-bias MNIST objective
Figure (O )- PGD h(x) = h(x).
Target
6 RELATED WORK
and vanilla: h(x) = Ux))
Debiasing with bias label. In (Goya et al. 2017 2020), a debiased dataset was generated using human labor. Various studies (Alvi et al.) 2018 Kim al. 2019) McDuff et al.) 2019 Singh et al.) 2020: Teney et al.] 2021) have attempted to reduce dataset bias using explicir bias labels. Some of these studies (lvi al. 2018, Kim et al.) 2019 McDuff et al., 2019, Singh al.) 2020 Li et al. 2018 Li & Vasconcelos 2019). used bias labels for each sample to reduce the influence of the bias labels when classifying target labels. Furthermore, Tartaglione et al. (02II) proposed the EnD regularizer, which entangles target correlated features and disentangles biased attributes. Several studies (Alvi et al. 2018) Kim et al. 2019 Teney et al., 202T) have designed DNNs as a shared feature extractors and multiple classifiers. In contrast to the shared feature extractor methods, MMfDfff et al. (2019) and Ramaswamy et al. (202T) fabricated a classifier and conditional generative adversarial networks, yielding test samples to determine whether the classifier was biased. Furthermore, (ingh
8

---

Published as a conference paper at ICLR 2023
et al. 22200) proposed new overlap loss defined by class activation map (CAM). The overlap loss reduces the overlapping parts of the CAM outputs of the two bias labels and target labels. The
authors of (Li & Vasconcelos 2019 Li et al.
2018) employed bias labels to detect bias-conflicting samples and to oversample them to debias. In Liu et al. 2021). a reconstructing method based on the
sample accuracy was proposed. [Liu et al. (022)) used bias labels in the validation dataset to tune the hyper- paper-parameters. On the other hand, there has been a focus on fairness within each attribute (Hardt al. 2016 Woodworth et al. 20177; Pleiss al. 2017 Agarwal et al. 2018). Their goal is to prevent bias attributes from affecting the final decision of the trained model.
Debiasing with bias context. In contrast to studies assuming the explicit bias labels, a few stud-
ies (Geirhos et al. 2018: Wang et al. (008,Lees et al.) 2019 Bahng et al., [2020] Cadene et al. 2019
Clark et al. (019) assumed that the hias context is known. In (Geirhos et al. 2018:] Wang et al. 2018
Lee et al. 2019), debiasing was performed by directly modifying known context bias. In particular,
the authors of (Geirhos et al. 2018) empirically showed that CNNs trained on ImageNet (Deng et al.)
2009) were biased towards the image texture, and they generated stylized ImageNet to mitigate the texture bias, while Lee et al. 2019) and Wang et al. (2018) inserted a filter in front of the models so that the influence of the backgrounds and colors of the images could be removed. Meanwhile, some studies (Bahng et al.) 2020 Clark et al. 2019; Cadene et al.] 2019), , mitigated bias by reweighting bias-conflicting samples: Bahnn al. (2020) used specific types of CNNs, such BagNet (Brendel & Bethge| 2018). to capture the texture bias, and the bias was reduced using the Hilbert-Schmidt independence criterion (HSIC). In the visual question answering (VQA) task, Clark al. (2019) and Cadene et al. (2019) conducted debiasing using the entropy regularizer or sigmoid output of the biased model trained on the fact that the biased model was biased toward the question.
Debiasing without human supervision. Owing to the impractical assumption that bias information given, recent studies have aimed to mitigate bias without human supervision (Le Bras et al.. 2020
Nam et al. 2020: Darlow et al., 2020: Kim et al. [022]] Lee et al. 202I). Le Bras et al. (2020)
identified bias-confiitinn samples by sorting the average accuracy of multiple train-test iterations
and performed debiasing by training on the samples with low average accuracy. In (Ahmed et al.)
2020), each class is divided into two clusters based on IRMv1 penalty (Arjovsky et al. 2019) using the trained biased model, and the deibased model is trained so that the output of two clusters become similar. Furthermore, Kim et al.] (2021) used Swap Auto-Encoder (Park et al. 2020) to generate bias- conflicting samples, Darlow et al. 2020) proposed the modification of the latent representation to generate bias confficting samples by using an auto-encoder. [ee et al. (202I) and [Nam et al.
(2020) proposed debiasing algorithm weighted training by using relative difficulty score, which is measured by the per-sample training loss. Specifically, Lee et al. (202T) used feature-mixing techniques to enrich the dataset feature information. Seo et al. (2022) and Sohoni et al. (2020) proposed unsupervised clustering based debiasing method. Recently contrastive learning based method (Zhang et al., (0226) and self-supervised learning method (Kim et al. 2022) are proposed.
On the other hand, there have been studies (Li & Xu, |021]] Lang et 20.1 Krishnakumar et al.) 2021) that identify the bias attribute of the training dataset without human supervision.
7 CONCLUSION
We propose a gradient-norm-based dataset oversampling method for mitigating the dataset bias problem. The main intuition of this work is that gradients contain abundant information about each sample. Since the bias-conflicting samples are relatively more difficult-to-learn than bias-aligned samples, the bias-conflicting samples have higher gradient norm compared with the others. Through various experiments and ablation studies, we demonstrate the effectiveness of our gradient-norm based oversampling method, called PGD. Furthermore, we formulate the dataset bias problem as min-max problem, and show theoretically that it can be relaxed by minimizing the trace of the inverse Fisher information. We provide empirical and theoretical evidence that PGD tries to solve the problem of minimizing the trace of the inverse Fisher information problem. Despite this successful outcome and analysis, we are still working on two future projects: release approximations, such as a toy example, for understanding PGD and cases where the given training dataset is corrupted, such as with noisy labels. We hope that this study will help improve understanding of researchers about the dataset bias problem.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work was supported by Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea governmen((MSrT) [No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST), 10%] and Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea govemmen((MSITT [No.2022- 0-00641, Vooiee Multi-Modal Voice Meta Learning, 90%]
REFERENCES
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reduc- tions approach to fair classification. In International Conference on Machine Learning, pp. 60-69. PMLR, 2018.
Faruk Ahmed, Yoshua Bengio, Harm van Seijen, and Aaron Courville. Systematic generalisation with group invariant predictions. In International Conference Learning Representations, 2020.
Mohsan Alvi, Andrew Zisserman, and Christoffer Nelláker. Turning a blind eye: Explicit removal of biases and variation from deep neural network embeddings. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 0-0, 2018.
Jing An, Lexing Ying, and Yuhua Zhu. Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients. In International Conference on Learning Representations, 2020.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint 1xXiv:1907.02893, 2019.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint urXiv:1906.0344 2019.
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Scong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning, pp. 528-539. PMLR, 2020.
Yujia Bao, Shiyu Chang, and Regina Barzilay. Predict then interpolate: A simple algorithm to learn stable classifiers. In International Conference on Machine Learning, pp. 640-650. PMLR, 2021.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. CoRR, wwss//90..0m.m 000660451, 2019. URL http:
org/ abs abs 110//999...00 04561
Wieland Brendel and Matthias Bethge. Approximating cnns with bug-oo--locll-ectture models works surprisingly well on imagenet. In International Conference on Learning Representations, 2018.
Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, al. Rubi: Reducing unimodal biases for visual question answering. In Advances in Neural Information Processing Systems, pp. 839-850, 2019.
Pratik Chaudhari. Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12); 2019.
Kamalika Chaudhuri, Sham Kakade, Praneeth Netrapalli, and Sujay Sanghavi. Convergence rates of actíve learning for maximum likelihood estimation, 2015.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature, 2018.
10

---

Published as a conference paper at ICLR 2023
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNI.PIICNILP), pp. 4069-4082, 2019.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. 2017 International Joint Conference on Neural Networks (IJCNN), pp. 2921-2926. IEEE, 2017.
Elliot Creager, learning. International Conference on Machine Learning, pp. 2189-2200. PMLR, 2021.
Jacobsen, and Richard Zemel. Environment inference for invariant
Luke Darlow, Stanislaw Jastrzębski, and Amos Storkey. Latent adversarial debiasing: Mitigating collider bias in deep neural networks. arXiv preprint 1. 2020.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2018.
Ankit Goyal, Kaiyu Yang, Dawei Yang, and Jia Deng. Rel3d: A minimally contrastive benchmark for grounding spatial relations in 3d. Advances Neural Information Processing Systems, 33, 2020.
Yash Goyal, Tejas Khot, Douglas Summers-Stapy Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6904-6913, 2017.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning, Advances in neural information processing systems, 29, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 120.. 2019.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of- distribution image without learning from out-of-distribution data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 10951-10960 2020.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34, 2021.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-accuracy. In Conference on Causal Learning and Reasoning, pp. 336-351 PMLR, 2022.
Stanislaw Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint urXiv:I711 1XX1::171.006220 1.04623, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401 4410- 2019.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. 2020.
Krishnateja Killamsetty, S Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference Machine Learning, pp. 5464 5474. PMLR, 2021a.
11

---

Published as a conference paper at ICLR 2023
Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust semi semi-supervised learning. Advances in Neural Information Processing Systems, 34, 2021b.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not learn: Training deep neural networks with biased data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9012-9020, 2019.
Eungyeup Kim, Jihyeon Lee, and Jaegul Choo, Biaswap: Removing dataset bias with bias-tailored swapping augmentation. arXiv preprint arXiv:2108. 10008, 2021.
Naycong Kim, Schyun Hwang, Sungsoo Ahn, Jaesik Park, and Suha Kwak. Learning debiased classifier with biased committee. arXiv preprint arXiv:2206, 10843, 2022.
Arvindkumar Krishnakumar, Viraj Prabhu, Sruthi Sudhakar, and Judy Hoffmann Udis: Unsupervised discovery of bias in deep visual recognition models. In British Machine Vision Conference (BMVC), volume 1, pp. 3, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Oran Lang, Yossi Gandelsman, Michal Yarom, Yoav Wald, Gal Elidan, Avinatan Hassidim, William Freeman, Phillip Isola, Amir Globerson, Michal Irani, al. Explaining in style: Training a gan to explain a classifier in stylespace. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 693-702, 2021.
Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. Adversarial filters of dataset biases. In International Conference on Machine Learning, pp. 1078-1088. PMLR, 2020.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann..ee http://yann.lecum.com www.::llclcccccccoc.. coccum..om//eub/nniss,, 2, 2010.
Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. Advances in Neural Information Processing Systems, 34, 2021.
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. International Conference on Learning Representations, 2019.
Erich L. Lehmann and George Casella. Theory of Point Estimation. Springer- Verlag, New York, NY,
USA, second edition, 1998.
Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9572-9581, 2019.
Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representa- tion bias. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 513-528, 2018.
Zhiheng Li and Chenliang Xu. Discover the unknown biased attribute of an image classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14970-14979, 2021.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In 2015 IEEE International Conference Computer Vision, ICCV 2015. Santiago, Chile, December 7-13, 2015, pp. 3730-3738 3738, IEEE Computer Society, 2015. doi: 10.1109/CC2.0015..222. URL https: /ttts::/ddo..ooo .rgg 10. 11e.1119/// 1009/TCCV 201 5. 425.
12

---

Published as a conference paper at ICLR 2023
Daniel McDuff, Shuang Ma, Yale Song, and Ashish Kapoor. Characterizing bias in classifiers using generative models. In Advances in Neural Information Processing Systems, pp. 50045404444 5415, 2019.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approxima-
tion and projection for dimension reduction, 2018. URL http://arxiv.org/ 1802. 03426 cite arxiv: 001.880...222mmmmmeee Reference implementation available at http://gitub.cco//neimmmiii
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960. PMLR, 2020.
Jun Hyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. In 34th Conference Neural Information Processing Systems (NeurIPS) 2020. Neural Information Processing Systems, 2020.
Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. arXiv preprint wrXiv:2007.00653 2020.
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and calibration. Advances in neural information processing systems, 30, 2017.
Vikram V Ramaswamy, Sunnie SY Kim, and Olga Russakovsky, Fair attribute classification through latent space de-biasing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1301-9310, 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang, Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: 191 2019.
Roland Schäfer. On bias-fre free crawling and representative web corpora. In Proceedings of the 10th web as corpus workshop, pp. 99-105, 2016.
Seonguk Seo, Joon- Young Lee, and Bohyung Han. Unsupervised learning of debiased representations with pseudo-an attribbuee In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 16742-16751, June 2022.
Robert J. Serfing. Approximation Theorems of Mathematical Statistics. John Wiley Sons, 1980.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585, 2020.
Robik Shrestha, Kushal Kafle, and Christopher Kanan. An investigation of critical issues in bias mitigation techniques. arXiv preprint arXiv:2104.00170, 2021.
Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jae Lee, Matt Feiszli, and Deepti Ghadiyaram. Don't judge an object by its context: Learning to overcome contextual bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11070-11078, 2020.
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-graines robustness in coarse- coarse-grained classification problems. Advances in Neural Information Processing Systems, 13:19339-19352 2020.
Jamshid Sourati, Murat Akcakaya, Todd K. Leen, Deniz Erdogmus, and Jennifer G. Dy. Asymptotic analysis of objectives based on fisher information in active learning, 2016.
Enzo Tartaglione, Carlo Alberto Barbano, and Marco Grangetto. End: Entangling and disentangling
deep representations for bias correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13508-13517 2021.
13

---

Published as a conference paper at ICLR 2023
Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton den Hengel. Evading the simplicity bias: Training a diverse set of models discovers solutions with superior ood generalization. arXiv preprint arXiv:2105.05612, 2021.
Antonio Torralba and Alexei A Efros. Unbiased look dataset bias. In CVPR 2011, pp. 1521-1528. IEEE, 2011.
Haohan Wang, Zexue He, Zachary C Lipton, and Eric P Xing. Learning robust representations by projecting superficial statistics out. In International Conference on Learning Representations, 2018.
Larry Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer Texts in Statistics. Springer, New York, 2004. ISBN 978-1-4419-2322-6. doi: 10.10077978.00388--1136--99
Blake E. Woodworth, Suriya Nathan Srebro. URL
Gunasckar,
Mesrob I.
Ohannessian, and
Learning non-discriminatory predictors.
In COLT.
2017.
https://www. httpss//www..enrrrcccccoo.......
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnish novel image dataset for benchmarking
machine learning algorithms, 2017.
Jianyu Zhang, David Lopez-Paz, and Léon Bottou. Rich feature construction for the optimization- generalization dilemma. arXiv preprini arXiv:2203. /55 2022a.
Michael Zhang, Nimit S Sohoni, Hongyang Zhang, Chelsea Finn, and Christopher Ré. Correct-n- contrast: A contrastive approach for improving robustness to spurious correlations. arXiv preprint arXiv:2203.015517, 2022b.
Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss training deep neural networks with noisy labels. In 32nd Conference Neural Information Processing Systems (NeurIPS), 2018.
14

---

Published as a conference paper at ICLR 2023
Appendix- Mitigating Dataset Bias by Using Per-sample Gradient
Due to the page constraint, this extra material includes additional results and theoretical proofs that are not in the original manuscript. Section deemonsttatssssssss how to create datasets. Section B.1 contains implementation details such as hyperparameters, computer resources, and a brief explanation of the baseline. Section and Section D include case studies and empirical evidence of PGD. Section D demonstrates additional experiment results. In Section F first provide notation summary and some assumptions for theoretical analysis. Section G and Section include proofs of Theorem [ and Theorem with lemmas, respectively.
A BENCHMARKS AND BASELINES
We explain the datasets utilized in Section 4 In short, we build MNIST variants from scratch, while others get them directly from online repositories BAR CCIFAR and BFFHO
A.1
CONTROLLED BIASED BENCHMARKS
D
2
O
2
3
4
7
D
2
4
0
2
3
The Figure 5: Multi-bias MNIST: Figure 6: Corrupted CIFAR: cor- single bias attribute is color, Multiple colors and objects bias, ruption is the bias attribute, and the target attribute is shape. with digit shape the target. while target attribute is object. The top 3 rows represent bias-The top 3 rows represent bias-The top three rows are bias- aligned samples, and the bottom aligned samples, and the bottom aligned samples, while the bot- row samples are biascconfiicting row samples are bias-confiictin tom row are bias-conflicting ex- examples.
Figure 4: Colored MNIST:
examples.
amples.
Colored MNIST (CMNIST) The MNIST dataset (LeCun et al. 2010 is composed of 1-dimensional grayscale handwritten images. The size of the image is 28 x 28. We inject color into these gray images to give them two main attributes: color and digit shape. This benchmark comes from related works (Nam al., 2020 Kim al. 2021 Lee et al. 2021 Bahng al.) (020). At the beginning of the generation, ten uniformly sampled RGB colors are chosen, {C.Jiejoo E R When the constant a ratio of bias-confficting samples, is given, each sample (æ, y) is colored by the following steps: (1) Choose bias-confficting or bias-aligned samples: take random sample and set it to bias-conflicting set when u < p where ~ Ld(0, 1), otherwise bias blaseed In experiments, we use p E (0.5%, 1%, 5%). (2) Coloring: Note that each C. 3 (i E [10]) bias-aligned diree-dimensional color vector for each digit i {10}. Then for bias- bias-aligned images with the arbitrary digit color the digit with NC, a I3x3). In the case of bias confiictigg images with the arbitrary digit y. first uniformly sample Cu, (E:):e( and color the digit with c ~ (CC,, Tax3). In the experiments, we set a as 0.0001. We use 55, 000 samples for training 5, 000 samples for validation (i.e., 10%). and 10, 000 samples for testing. Take note that test samples are unbiased, which means p = 90%.
Multi- Bias MNIST Multi-bias MNIST has images with size 56 x 56. This dataset aims to test the case where there are multiple bias attributes. To accomplish this, we inject total of seven bias
"hats://gihub.com/aaaaaaaaa hhang://gtthhhccc//aakceeeenerse www.a//eeeeeeaaanaaaeeeeee Dehiissd---ii--------ae Disentangled
Dehiased-Diseenangled
15

---

Published as a conference paper at ICLR 2023
attributes: digit color, fashion object, fashion color, Japanese character, Japanese character color, English character, and English character color, with digit shape serving as the target attribute. We inject each bias independently into each sample, injecting bias). We also set p = 90% for all bias attributes to generate an unbiased test set. As with CMNIST, use 55, 000 samples for training and 5, 000 samples for validation, and 10, 000 samples for testing.
with the CMNIST case (i.e., sampling and
Corrupted CIFAR This dataset was generated by injecting filters into the CIFARIO dataset (Krizhevsky et al. 2009). The work (Nam et al., 22000; Lee et al. 2021) inspired this benchmark. In this benchmark, the target attribute and the bias attribute are object and corruption. respectively. (Snow, Frost, Fog. Brightness, Contrast, Spatter, Elastic, JPEG, Pixelate, Saturate| are examples of corruption. We downloaded this benchmark from the repository of the official code of Disen (Lee et al., (021). This dataset contains 45, 000 in training samples, 5, 000 in validation samples, and 10, 000 testing images. As with prior datasets, the test dataset is composed of unbiased samples (i.e., P 90%)
A.2 REAL-WORLD BENCHMARKS
Figure 7: Biased Action Recognition: The biased attribute is background, while the target attribute is action. The top 2 rows are bias-aligned samples, and the bottom row is bias-conffict samples.
stands for biased FFHQ. Target attribute: gender, biased attribute: age. The sam- ples in the top two rows are bias-aligned, while the samples in the bottom row are bias-conffict.
Figure 8: BFFHQ:
Biased Action Recognition (BAR) This dataset comes from the paper (Nam et al., 2020) for real-world image testing. TThis benchmark aims classify six actions (Climbing, Diving, Fishing, Racing, Throwing, Vaulting] even though the places are biased. Target and bias attribute pairs (Climbing.Rocks Wall), (Diving, Underwatrr), (Fishing, WaterSurface), (Racing, APavedTrack), (Throwing, PlayingField), and (Vaulting. Sky). Bias-conflicting samples, for example, are (Climbing, IceCliff), (Diving, Indoor), (Fishing, Forest), (Racing, OnIce), (Throwing, Cave), (Vaulting, Beach). There are 1, 941 samples for training and 6, 54 samples for testing. To split the training and validation samples, we used 10% validation samples, i.e., 1, 746 images for training and 195 validation. We download training datasets from the online repository of BAR.
Biased FFHQ This BFFHQ benchmark was conducted in (Lee et al. 202 Kim et al. 202 Target and bias attributes for bias-aligned samples are (Female, Young), and (Male, Old). Here, "Young" refers to people aged to 29, while "old" refers to people aged 40 to 59. The bias-conflicting samples are (Female, Old) and (Male, Young). The number of training, validation, and test samples are 19, 200, 1, 000, and 1, 000, respectively.
CelebA CelebA (L.iu et al. 2015) is a common real-world face classification dataset, and each image has 40 attributes. The goal is to classify the hair color ("blond" and "not blond") of celebrities, which has spurious correlation with the gender ("male" or "female") attribute. In fact, only 6% of blond hair color images are male. Therefore, ERM shows poor performance on the bias-conflicting samples. We report the average accuracy and the worst-group accuracy on the test dataset.
CivilComments - CivilComments WILDS (Borkan et al. 2019) is a dataset to classify whether an online comment is toxic non-toxic. Each sentence a real online comment, curated on the Civil Comments platform, a comment plug-in for independent news sites. The mentions of certain demographic identities (male, female, White, Black, LGBTQ, Muslim, Christian, and other religion)
16

---

Published as a conference paper at ICLR 2023
cause the spurious correlation with the label. Table indicates the portion of toxic comments for each demographic identity.
Identity
Male
Female
White
Black
LGBTQ
Muslim
Christian
Other religions
Portion %) of toxic
14.9
13.7
28.0
31.4
26.9
22.4
9.1
15.3
Table 5: For each demographic identity, the portion of toxic comments in the DoviCoomment--WI.DD
A.3 BASELINES
In this section, we briefly describe how the baselines, such as LfF, JTT, Disen, GEORGE, BPA, CNC, and EIIL.. Please refer to each paper for a detailed explanation because we briefly explain the algorithms.
(1) LfF (Nam et al. 2020) trains the debiased model by weighting the bias- ctussconflictin samples based on the "relative diffculty", computed by the two loss values from the biased model and the debiased model. To amplify the bias-cone bias-conflicting samples, the authors employ generalized cross-entropy loss with parameter a = 0.7. We implement the LfF algorithm following the code officially offered by the authors. The loss functions that this work proposes are follows:
LLF (1)) (C + (Cocr(Cs(2,
(cn(C((z),))
W (z) CCE(CM(z).)) LCE(CC()).))
Note that (z)) is a relative difficulty and that GCE is generalized cross-entropy. x denotes feature, which is the output of the penultimate layer, and C. a fully connected layer.
(2) JTT (Liu et al. (021) aims to debias by splitting the dataset into correctly and incorrectly learned samples. To do so, JTT trains the biased model first and splits the given training dataset as follows:
Demor-set {zx y) s.t. Mgiien # arg max (x)
The ultimate debiased model is then trained by oversampling Derrorset with Xup times. We set Aup for all experiments as 1/p. We reproduce the results by utilizing the official code offered by the authors. The main strength of PGD compared to JTT is that PGD does not need to set hyperparameter Xup-
(3) Disen (Lee et al. 2021) aims to debias by generating abundant features from mixing features between samples. To do so, the author trains the biased and debiased model by aggregating features from both networks. This work also utilize the "relative difficulty" that is proposed in (Lee et al.) 202Tb. We reproduced the results utilizing the official code offered by the authors. The loss function proposed in this work is as follows:
Lioall = Ldis + Asswap
where
Cswap (C)))CC(C 8) + Aswap, COC Ø))
Lai ((x)
(c,((())),))
W(2) LEE (c)(CC))),)) + Leu(Ca(=),
Except for the swapped feature, Aswap all terms are identical to those in LfF explanation.
(4) GEORGE [Sohoni et al.) 2020) aims to debias by measuring and mitigating hidden stratification without requiring access to subclass labels. Assume there are n data points æ1, Fn. E and associated superclass (target) labels y1...., yn E {1,- C). Furthermore, each datapoint x; is associated with latent (unobserved) subclass label z;. George consists of three steps. The author trains the biased model using ERM. Next, to estimate an approximate subclass (latent) label, apply UMAP dimensionality reduction (McInnes et al. 2018) to the features of given training dataset at the ERM model. Here, GEORGE cluster the output of the reduced dimension for the data of each superclass into K clusters, where K is chosen automatically. The original paper contains detailed
17

---

Published as a conference paper at ICLR 2023
description of the clustering process. Lastly, to improve performance these estimated subclasses, GEORGE minimizes the maximum per-cluster average loss (i.e., (Ir, y) Ps), by using the cluster as groups in the G-DRO objective (Sagawa et al. 2019). The loss function proposed in this work is as follows:
minimize L.fe
max
I (a fa(æ),
ISESK
where fa and are parameterized feature extractor and classifier, respectively.
(5) BPA (Seo et al. 2022) aims to debias by using the technique of feature clustering and cluster reweighting. It consists of three steps. First, the author trains the biased model based on ERM. Next, at the biased model, cluster all training samples into K clusters based on the feature, where K is the hyperparameter. Here, h(z, 0) K {1.-- .K} denote the cluster mapping function of data (xr, y) derived by the biased model with parameter 0. At the last step, BPA computes the proper importance weight, Wi for the k-th cluster, where & E IC and the final objective of debiasing the framework is given by minimizing the weighted empirical risk as follows:
minimize (E(x,))> ("(a.) (0)1(z, y: 0]]
Concretely, for any iteration number T, the momentum method based on the history set Hr, which defined as:
Hr { SISTI Ban [1(x,, 0.)1
N%
where N₁ is the number of the data belonging k-th cluster.
(6) CNC (Zhang et al. 2022b) aims to debias by learning representation such that samples in the same class are close but different groups are far. CNC is composed of two steps: (I) inferring pscudo group label, (2) supervised contrastive learning. Get the ERM-based model f and the pseudo prediction first, then standard argmax the final layer outputs of the model f. Next, CNC trains the debiased model based on supervised contrastive learning using pseudo predictiong. The detailed process of contrastive learning for each iteration follows:
From the selected batch, sample the one anchor data (x.y).
Construct the set of positives samples { yth) which is belong the batch, satisfying vm = y and fm # ÿ.
Similarly, construct the set negative samples {(# B7 which is belong to the batch, satisfying Vn # and In ÿ.
With the loss of generality, assume the cardinality of the positive and negative sets are M and N, respectively.
Weight update based on the gradient of the loss function L(fe; xr,eex,,,0 2r, y), the detail is like below:
i, 2.,y xi i (x, t J ( I. fenc) (1 E (So; y)-
Here, À E [0, 1] is hyperparameter and Temes(Jo;; v) is an average cross-entropy loss over Jr the M positives, and N negatives. Moreover, fene the feature extractor part of fo and the detail formulation of Lean (Ix, { AM {2, J fens) is like below:
M
exp(fanc (x) fene (x
log
M
M Im=1 exp(feec(x) (1th)//)7 + Σ exp(fenc(x)) fenc (enc((5)) (/T)
(7) EIIL (Creager et al.
2021) proposes a novel invariant learning framework that does not require prior knowledge of the environment. EIIL is composed of three steps: (i) training based ERM, (ii) environment inference (EI), and (iii) invariant learning (IL). In the first step, EIIL gets a biased model by minimizing ERM. Next, based on the feature extractor trained in (i), optimize the El objective to infer environments from each training dataset. The object of EI is to sort training examples that maximally separate the spurious features so that they facilitate effective invariant learning. Lastly, to get debiased model, optimize the classifier and feature extractor by minimizing the invariant learning objective.
18

---

Published as a conference paper at ICLR 2023
B EXPERIMENT DETAILS AND ADDITIONAL ANALYSIS
B.1 SETTINGS
This section discusses how our experiment was set up, including architecture, image processing, and implementation details.
Architecture. For the colored MNIST, we simple convolutional networks consisting of three CNN layers with kernel size 4 and channel sizes {8, 32, 64) for each layer. Also, we utilize average pooling at the end of each layer. Batch normalization and dropout techniques are used for regularization. Detailed network configurations are below. Similarly, for the multi-biat MNIST, we use four CNN layers with kernel size (7, 7, 5, 3) and channel size {8, 32, 64, 128), respectively. For corrupted CIFAR, BAR, and BFFHQ, we utilize ResNet-18, which provided by the open-source library torchvision. For CelebA, follow the experimental setting of CNC (Zhang al. 2022b), which uses ResNet-50 as backbone network. For CivilCOmments-WiL.. WILDs, we use pretrained-BRET for the backbone network and exactly the same hyperparameters as in (Zhang al. 2022b
SinCo..
(convl): Conv2d (3, 8, kernel_size= 4), stride= 1)) (bnl): catchhorm2d (8, eps-1e-05, moment 0m- Effin-TTuee, www_rrnninngsstttsnire (relul): ()LU()
(dropout1): Dropout (p=0.5, inplace=False)
(avgpool1): AvgPool2d ( stride=2, padding=0)
(conv2) : (8nv2d 32, kernel_size=(4 4), stride= (1, 1))
(bn2) : BatchNorm2d (32, eps-le-05, momentun=0.1, affine=True,
Bocc__rumiin_stttttttte (relu2) ReLU()
(dropout2): Dropout (p=0.5, inplace=False)
(avgpoo12): AvgPcol2d (kernee_iez,,,, stride=2, padding=0)
(conv3) Conv2d (32, 64, kernel_size=((4 (4, 4), stride= (1, 1)) (relu3): ReLU()
(bn3): BatchNorm2d (64, eps=1e-05, momentum=0.i affine=True, Beack_running sanc___rnnnng_sttss-rreee
(dropout3): Dropout (p=0.5, inplace=False) (avgpoo13):
(output_size 1)) (fc) : Linear fentureere,s44 fnt_ class, bias=True)
SimConv-2.
(convl): Conv2d 8, kermel_size==1, 7), stride=(1. (1, 1))
(bnl): SatchNorm2d (8, eps=1e-05, momentum=0.1. affine=True,
(relul): ReLU()
(dropoutl): Dropout (p-0.5, inplace-False)
(avgpool1): (keenne
stride=3, padding=0)
(conv2) Conv2d 32, kernel_size=fe 7), stride= (1, 1))
(bn2): BatchNorm2d (32, eps=1e-05, momentum=0.1, affine=True, crack_runniig_ssttssTTT (relu2): ()LU()
(dropout2): Dropout (p=0.5, inplace=False)
:avgpoo12): AvgFool2d (kernel ( stride=3, padding=()
(conv3) Con, 64, ternel_size= 5), stride= (1, 1))
(relu3): ReLU()
(bn3): BatchNorm2d165 (64, eps-1e-05, momentum=0.1, affine=True, brack_ruming_sttts-TTe
(dropout3): Dropout (p=0.5, inplace=Palse)
(conv4) Conv2d(64, 128, kernel size= 3), stride=(1, 1))
(relu4): : ReLU()
(bn4): : RatchNorm2d (128, eps=1e-05, momentum=0.1, affine=True,
19

---

Published as a conference paper at ICLR 2023
crack_runnne sumiing_stats=Irue)
( Dropout (p=0.5, 5, inplace=Palse) (avapool)
(iteputtsiz=(1,, 1))
(fc) : Linear (in_ (in_features=1122 out_featuues=$num class, bias=True)
B.2 PGD IMPLEMENTATION DETAILS
Image processing We train and evaluate with fixed image size. For colored MNIST case (28 x 28), multi-bias MNIST (56 x 56), corrupted CIFAR (32 x 32), and the remains (224 x 224). For the CMNIST, MBMNIST, CCIFAR, BAR, and BFFHQ, we utilize random resize crop, random rotation, and color jitter to avoid overfitting. We use normalizing with mean of 19,4914, 0.4822, 0.4465), and standard deviation of (0.2023, 0.1994, 0. 0.2010) for CCIFAR, BAR, and BFFHQ cases.
Implementation For table and Table reported in Setion 4 reproduce all experimental results referring to other official repositories: 9 The differences compared to the baseline codes are network architecture for CMNIST and usage of data augmentation. Here, use the same architecture for CMNIST and data augmentation for all algorithms for a fair comparison. Except for JTT, all hyperparameters for CCIFAR and BFFHQ follow previously reported parameters in repositories. We grid-search for other cases, MNIST variants, and BAR. We set the only hyperparameter of PGD, a 0.7, as proposed by the original paper (Zhang & Sabuncu 2018). A summary of the hyperparameters that we used is reported in Table
Colorel MINIST
Malti-bias MNIST
Comupted CIFAR
BAR
Biased FFJQ
ColobA Adam
CiviComments WILDS
Uptimizer
SGD
SGD
Adam
SGD
Adam
SGD
Ratch
128
32
256
16
Learning
Weight deaa
0.011
(0001
00001
1c-3
BVD
0.01
0.01
Momentum
0.9
0.9
0.9
Le decay
Cosine anncaling
Decay step
40
Epoch
100
100
200
100
160
100
$
GCE a
0.7
0.7
0.7
0.7
Table 6: Hyperparameter details
For Table reported in Section we follow the implementation settings of CelebA and CivilComments-WII. suggested by [Soo et al.) (2022) and Liu et al. (202T), respectively. A summary of the hyperparameters that we used reported T6 We conduct our experiments mainly using single Titan XP GPU for all cases.
"http:://github.c http:://gtttiii.oooo/oo////aaeaa
htttps//gituuh.oo/mmnsssss
20

---

Published as a conference paper at ICLR 2023
C CASE STUDIES ON PGD
In this section, we analyze PGD in many ways. Most analyses are based on the CMNIST dataset, and the experimental setting the same as the existing setting unless otherwise noted. For example, all experiments used the same data augmentation, color jitter, resize crop, and random rotation.
C.I STUDY 1: ABLATION STUDY ON GCE PARAMETER a
The only
used PGD is the GCE parameter We experimented with this value at 0.7 according to the protocol of LfF (Nam et al.) 2020 However, compare the various cases of a. To analyze this, run PGD with various and report the per- formance in T As in Table 7 the debiased model performs best when the GCE parameter is 0.9. This is because the biased model is fully focused on the bias feature, rather than the target feature, which can be seen from the unbiased test accuracy of the biased model, as in the bottom of Table
Colored MNIST
= 0.3
Debiased model
0.5
a = 0.7
0-= 0.9
= 0.5%
94.70: oz
96.79
p=
96.32a
a7 98.82 we
97.35amm
p=
98.80 oxe
98.62
98.78 cas
need to
Bussed model
P 0.5%
19.86: ws
10.70 wns
18
17.39. cas
1%
21.40
21.04 49.462
19.
19:
53.24
43.97
39.40 tas
Table 7: Ablation study GCE parameter
C.2 STUDY 2: MULTI-STAGE VS SINGLE-STAGE
PGD computes the per-sample gradient norm only once, between training the biased model and the debiased model. However, an update of the per-sample gradient can be performed repeatedly at each epoch (i.e., single-stage). In other words, PGD can be modeled run the following loop: updating the biased model, updating the sampling probability, and updating the debiased model. this section, we justify why we use the multi-stage approach. We report the performance of multi-stage and single- stage PGD on the colored MNIST dataset. As in Table & the single-stage method has two characteristics: (1) it requires more training time than the multi-stage method. (2) It has lower unbiased accuracy compared to the multi-stage method. The longer training time is due to the high computational resources required to compute the per-sample gradient norm. Moreover, because the Ainleestage method's ' sampling probability changes the training distribution over epochs, the debiased model suffers from unstable training and loses debiasing performance.
Triiing
Test Ace:
Sungle-stage stage
Malti-tage
Bangle-stage
Multi stage
= 0.8%
2h 53m d0s
33m 39%
92.19 a12 97 23: aM
96.88
= 15%
2h 5im 45:
32m 28s
97.35
5%
2h 49m 31s
34m 136
98.44,
98 22.
Table 8: Multi-stage vs Single-stage
C.3 STUDY 3: RESAMPLING VS REWEIGHTING
To support our algorithm design, we provide further ex- perimental analysis, i.e., resampling versus reweighting. Reweighting (Nam al. 2020 Lee et al. (02T) and re- sampling (Liu et al. (021)) ) are the two main techniques
Colored MNIST
Resampling 96
Reweighting 94.70
0.5%
0 1%
=1
97.20+
0
98.62=a14
98.51-au oue
to debias by up- weightigg bias-conflicting samples. PGD
Table 9: Reweighting resampling
an algorithm that modifies the sampling probability by
using the per-sample gradient norm. To check whether PGD works with reweighting, we examine
the results of PGD with reweighting colored MNIST dataset and report them in Table We
compute the weight for each sample as follows: w(r,, yt) [D.]]
As in Table 9 PGD with resampling slightly outperforms PGD with reweighting. As argued in (An et al. 2020). this gain from resampling can be explained by the argument that resampling is more stable and better than reweighting.
21

---

Published as a conference paper at ICLR 2023
C.4 STUDY 4: ANALYSIS OF THE PURE EFFECT OF GRADIENT-NORM BASED SCORE
PGD performance improvement comes not only from the two-stage and resampling modules that we wrote about in Appendix [22 and Ap- pendix C.3] but also from the gradient score. To verify this, we report the following two results: (1) resample based on per-sample loss rather than per-sample gradient norm in PGD, and (2) change the relative difficulty score of LfF to gra- dient norm. As shown in Table TO we can conclude the following two results: (1) loss of the last epoch of the first stage in a two-stage approach is not suitable for resampling, and (2) the results of replacing the relative difficulty metric of LfF with gradient norm show that the gradient norm has better discriminative properties for bias-conf samples than loss.
Method
p= 0.5%
= 1%
0 5%
LfF
91.35 135
96.88+
98,18 0.s
PGD + Loss
30.63 223
34.04 300
78.48:
LIF + Gradient
93.29 0.39
97.55=
98.37
PGD
96,88 0.28
98.35= 0.12
98.62
Table 10: loss score vs gradient norm score
C.5 STUDY 5: COMPUTATION COST
Debiasing algorithms require an additional com- putational cost. To evaluate the computational cost of PGD, we report the training time in Ta- We conduct this experiment by using the colored MNIST with p = 0.5%. As in the top of [1 report the training time of four methods: vanilla, LfF, Disen, and PGD. Here, PGD spends a longer amount of training time. This is because there is module for computing per-sample gradient in batch manner. At the bottom of Table we report part-by part-by-part costs to see which parts consume the most time. Note that Steps 1, 2. and represent training the biased model, computing the per-sample gradient-norm, and training the debiased model, respectively. We can conclude with the following two facts. (1) Step 2 (computing the per-sample gradient norm and sampling probability) spends 4.3% of training time. (2) Resampling based on the modified sampling probability h(r) requires an additional cost of Im 27s by seeing the difference between the computing times of Step and Step
Vanilla
LtF
Disen
Compulation time
14m 59s
21m 358
23m 18s
33m 31s
Step
Step
Step 3
ble [1]
Computation time
Im
16m 46s
Table 11: Computation cost
C.6 STUDY WHAT PGD RUNS ON UNBIASED DATASETS
We examine whether PGD fails when an un-
Vanilla
LfF
Disen
Ours
biased dataset is given. To verify this, we re- port two types of additional results: (1) unbi-
P 90%
Colored MNIST
99.04 0.s
98.75=
99.31+ aur
98.43 an
ased CMNIST (i.e., p = 90%) and (2) conven- tional public dataset (i.e., CIFAR10). We follow the experimental setting of CMNIST for the un- biased CMNIST case. On the other hand, we train ResNet18 (He et al. 2016) for CIFARIO with the SGD optimizer, 0.9 momentum, 5e weight decay, 0.1 learning rate, and Cosine Annealing LR decay scheduler. As shown Table 12 PGD does suffer significant performance degradation in unbiased CMNIST. Furthermore, it per- forms better than the vanilla model on the CIFAR10 10 dataset. This means that the training distribution that PGD changes do not cause significant performance degradation. In other words, PGD works well, regardless of whether the training dataset is balanced or unbiased.
Natrual CIFARTO
94.242001
94.79 002
Table 12: Results on unbiased CMNIST and natu- ral CIFARIO cases.
22

---

Published as a conference paper at ICLR 2023
D EMPIRICAL EVIDENCE OF PGD
As same with the setting of Appendix C in this section, we also use the existing CMNIST setting, such data augmentation, hyperparameters.
D.1 CORRELATION BETWEEN GRADIENT NORM AND BIAS- ALIGNMENT OF THE CMNIST
To check if the per-sample gradient norm efficiently separates the bias-conflicting samples from the bias-aligned samples, we plot the gradient norm distributions of the colored MNIST (CMNIST). For Va comparison, we normalized the per-sample gradient norm as follows: (y ).P. II Ce (x:::))) As in Figure 9. the bias- blas-aligned sample has a lower gradient norm (blue bars) than the bias-conflicting samples (red bars).
C184ST pm 05%
CWREST pa 1%
CMREST pm
100
Blawiigeed Bio oon-onn kding
100
Ran-sigeed corlorlitingg
100
Bond igredd
Bis-oerlitig
50
12
10
Bia
B2
13722
88
(a) Colored MNIST 0.5%
(b) Colored MNIST 1%
(c) Colored MNIST p 5%
Figure 9: Histogram of per-sample gradient norm.
D.2 PGD DOES NOT LEARN ONLY THE SECOND-EASIES FEATURE
We provide the results of the following experimental settings: The target feature is color, and the bias feature is digit shape, i.e., the task is to classify the color, not the digit shape. Let give an example of this task. When one of the target classes is red, this class is aligned with one of the digits (e.g.. "0"). In other words, the bias-aligned samples in this class are (Red, "O"), and the bias-conficting samples are (c.g., (Red. "I"), (Red, "2")..., (Red, "9")). Note that, as shown in LfF (Nam et al. 2020), color is empirically known be easier learn than digit shape; we think that the above scenario reflects the concern: whether PGD is only targeting the second-easies feature (digit shape). Therefore, if the concern is correct, PGD may fail in this color target MNIST scenario since the model will learn digit shape. However, as shown in the table below, vanilla, PGD, and LiF perform well in that case.
Vanilla (Digit)
Vanilla (Color)
LfF (Digit)
LfF (Color)
PGD (Digit)
PGD (Color)
= 0.3%
60.94
90.33
91.35
91.16
96.88
98.92
1%
79.13
92.53
96,88
96.12
98.35
99.58
5%
95.12
96.96
98.18
99.11
98.62
99.7
Table 13: Digit target MNIST vs Color target MNIST
We can also support this result by seeing the distribution of the normalized gradient norms,
[[VoCE yı; 0b) max H..D, VVLCCE(, Mi; 0b)]l2 [0, 1]. extracted from the biased
model 0g (trained in Step of Algorithm of Section 3
(0.0,0.1)
(0.1.02)
(0.2.03)
[0.3,0.4)
(0.4.0.5)
(0.5.0.6)
(0.6.0.7)
(0.7.088
(0.8.099
[0.9.10]]
Bias-aligned
53504
21
16
17
8
Bias-confieting
212
18
10
4
0
I
Table 14: Number of samples at each bin: Color target MNIST (p = 0.5%)
The numbers filled in Table 14 are the number of data items belonging to each bin category. We can check that there no bias-core confticting samples whose gradient norm is significantly larger than the bias-aligned samples. In other words, PGD does not force the debiased model to learn the digit shape (i.e., the second-easiest feature) in this scenario. This scenario brings similar performance to Vanilla.
23

---

Published as a conference paper at ICLR 2023
p= 0.5%
g= 1%
p= 5%
Disen
ML
Disen PGD
Disen PGD
# friils
6l.
#f trials
#o trials
20
M
10
0
90
92
94
96
98
100
96
99
100
Accuracy
Accuracy
Accuracy
Colored MNIST 0.5%
Colored MNIST
1%
(c) Colored MNIST p 5%
Figure 10: Histogram of unbiased test accuracy among 25 trials for each.
D.3 HISTOGRAM OF THE RESULTS OF TRIALS IN CMNIST
For in-depth analysis, we provide the results of 25 tests CMNIST in Figure 10 We compare with Disen, which shows the best performance except for PGD. However, very few cases overlap, as shown in Figure I0] We conduct I-test for more in-depth analysis of this. When the alternative hypothesis is established that PGD is superior to Disen, the p-value for it has values of 0.01572, 0.01239, and 0.29370, respectively. In other words, be said that as the bias becomes worse, the superiority of PGD stands out.
E ADDITIONAL RESULTS ON OTHER DATASETS
E.1 CELEBA WITH RESNETI8 SETTING.
Vanilla
LfF
GEORGE
BPA
Ours
Avg.
80.52
84.89
83.13
90.18
89.27
CelebAt
Worst
41.02
57.96
65.45
82.54
82.73
Table 15: Average and worst test accuracy with CelebA setting (Sco et al. 2022).
The results of
comparison algorithms for CelebA are the results reported in (Seo et al. 2022). accuracy is indicated bold.
The best worst
We reported the results of CelebA in Table 3 of section 4 following the settings of (Zhang et al.
2022b). For comparison with more diverse algorithms, we further report the CelebA results according to the settings of (Seo et al. 2022). Note that the (Zhang al. 2022b) and (Seo et al. 2022 used a different model, each using ResNet50 and ResNet18, respectively. As in Table 15 competitive performance among the other baselines.
PGD shows
24

---

Published as a conference paper at ICLR 2023
F BACKGROUNDS FOR THEORETICAL ANALYSIS
F.1 NOTATIONS SUMMARY.
For convenience, we describe notations used in Section 5
Appendix F
G
and H
Table 16: Notation Table
Notation
Description
Remark
Variables
(P) 9)
(image, label)
x K' y = [.....
Mtruc (2r)
the true lahel of image or
laheled by the oracle model fll 9*)
model parameter
8"
oracle model parameter
satislying (x)]a, 0*) = 1 for any composed of ((,,,,)) to p(r.a/0")
training dataset
b((r)
sampling probability of each sample in Dr
satisfying A(r) E 9
Distributions
P(ar)
general distrhuttion of impui image ir distribution training image
p(x)
q(x)
distribation of test image
conditional distriburion with model parameter general joint distributiinn with model parameter 0 joint distribution of the training dataset
p(r.y(9)
Plai 0)
pla, 8*)
plx) fly 2. q(z) 2,6"
q(3. (
joint distribution of the test datasct
Estimators
Oupe D.
MLE solution on the D, with
m1x solution of FRM
Σ Max) log fini 8)
MLF E. solution on the D, with uniform distribution E
Fisher Information
10
Information
P flv Va log Ay log flyl,,0) Ei Va log flala,, V log flul.,)))
(nts((0)
Empirical Fisher Information
Set
H
set of possible a(a)) on 2,
(Ma) Lir. Max; and h(z,) 20 V(ra)) P.)
M w
set of possible marginal P(z) on input space X set of all possible (rt (x))
(P(x) Jex. P(z)
(x vl0)
Support set of (())
{(x.g) ex c] P(x, v10) * 0z, v(a,)))))
Order notations in probahility
Op
Big O, stochastic boundedness Small convergence in probahility
Toy example (Appendix
set of majority (iee, bias-aligned) sumples
set of minority (E.e., bias-conflicting) samples
(3)(8)
gradient of samples in M at a
(m( (9)
gradient samples
(ar(
Optimal sampling probubility of samples in Af
Optimal sampling probability samples in
F.2 MAIN ASSUMPTION
Here, we organize the assumptions that are used in the proof of Theorems. These are basically used when analyzing models through Fisher information. The assumptions are motivated by (Sourati et al. 2016).
Assumption 1.
(A0). The general joint distribution P(z, y00) is factorized into the conditional distribution fyylx, 0) and the marginal distribution P(x), not depend on model parameter 0, that is:
P(x, vl0) P())f(y) 8).
Thus, the joint distribution is derived from model parameter 0 and the marginal distribution P(x), which is determined from the task that we want to solve. Without loss of generality, we match the joint distribution's 's name with the marginal distribution.
(AI). (Identifiabiliit):: The CDF Py (whose density is given by y()) is identifiable for different parameters. Meaning that for every distinct parameter vectors 01 and 0g in Ω Po. and Po, are also distinct. That is,
vO, * 0₂ E Ω 3A cf s.t. Pa, (A) # Po,
where X, (1.... .c) and Ω are input, label, and model parameter space, respectively.
25

---

Published as a conference paper at ICLR 2023
(A2). The joint distribution Pe has common support for all 0 E Ω
(A3). (Model Faithfulness): For any E X, we assume oracle model parameter 0* that generates Vlnee (z), true label of input with the conditional distribution [ (2) xx,0))
1.
(A4). (Training joint): Let ((x) denote the training marginal with no dependence on the parameter. Then, the set observations D, _ ((11,2)).... (on, 1/)) are drawn independently from the training/proposal joint distribution of the form p(x, y/0*) _ f( (yla, 8*), be- cause do not think the existence of mismatched label data situation in the training data.
(A5). (Test joint): Let g(x) denote the test marginal without dependence on the parameter. The unseen test pairs are distributed according to the test/true joint distribution of the form g(x, y)0*) & (yla,, 0*), because we do not think the existence of mismatched label data situation in the test task.
(A6). (Differentiability): The log-conditional log f(yl,, 0) is of class ( for all (ær, y) x (1,2,-.. c), when being viewed as a function of the parameter
(A7). The parameter space is compact, and there exists an open ball around the true parameter of the model 6** E Ω.
(A8). (Invertibility): The arbitrary Fisher information matrix (x()) (0) is positive definite and therefore invertible for all 0 E Ω
(A9). {zx, y) E supple(x, y)0*))
V log q(z, y/0*) is singular) is measure zero set.
In contrast to (Sourati et al. 2016), we modify (A3) so that the oracle model always outputs a hard label, i.e., (yttue (x) xr 0r) = I and add (A9) which is not numbered but noted the statement of Theorem and Theorem 11 in (Sourati et al. 2016).
F.3 PRELIMINARIES
We organize the two types of background knowledge. maximum likelihood estimator (MLE) and Fisher information (FI), needed for future analysis.
F.3.1 MAXIMUM LIKELIHOOD ESTIMATOR (MLE)
In this section, derive the maximum likelihood estimator in classification problem with sampling probability ((x). Unless otherwise specified, training set Dn. {( 3: H"= is sampled from p(a, 0)) For given probability mass function (PMF) h(x) on Dn., we define MLE O1(2), as follows:
(N(a).D. A arg max
arg min
h(x;)) p(o,
(3)
arg min
h((;)) log (u: 0)
(4)
n
arg min
M(xx)) (ou
(5)
In (3) and 44. (A0) and (A4) of Assumption I in Appendix F were used, respectively. It is worth noting that MLE (,()),,. variable that influenced by two factors: (1) a change the training
We say that function f X + Y is of CP(X), for an integer p >> O, if its derivatives up to p-th order exist and continuous all points of X.
26

---

Published as a conference paper at ICLR 2023
dataset D and (2) the adjustment of the sampling probability h(x)). If h(x) is uniform distribution (z),, then the result of empirical risk minimization (ERM) is (s).
F.3.2 FISHER INFORMATION (FI)
General definition of FI. Fisher information (FI), denoted by (0), is a measure of sample information from given distribution P(x, yl - P 0). It is defined as the expected value of the outer product of the score function V log ((x, v/0) with itself, evaluated at some 0 E Ω.
((x) (0) A E(x.w))((aa11)) Vo log P(z, v log P(z, y(0)..
(6)
Extended version FI. Here, we summarize the extended version FI, which can derived by making some assumptions. These variants of FI utilized in the proof of Theorems.
(Hessian version) Under the differentiability condition (A6) of Assumption Din Appendix F Fl can be written in terms of the Hessian matrix of the log- ligeliiei:ooo::
(()) (0) (((e.)))PPPP V log y/0]-
(Model decomposition version) Under the facttorzaaioon condition (A0) of Assumption [ Appendix
in
and 7 can be transformed as follows:
(0) ((eea))-Ple)f(lll....) Vo log flylæ, 0)V7 log f(y2,,
(8)
P(aaaa--())))/ V log f(yl2, 0)].
(9)
Specifically, 8 and can be unfolded as follows:
[0)=)(0) = P(x)
[f 0) vo log f(ylæ, T log f(ylz, 0)] dz (10)
øEX
= =- DEX
P(x)
[f 0) v log f(yl,, dr
y=1
From now on, we define p(a) (0) and (a(m) (f) as the Fl derived from the training and test marginal, respectively.
F.3.3 EMPIRICAL FISHER INFORMATION (EFI)
When the training dataset D, is given, we denote the sampling probability as h(()) which is defined on the probability space H:
H = (1(x) Σ E(EWDEED M(x;)) 1, M(xx)) 2 O V(zi, (k) E D.,
(11)
Practically, the training dataset D, is given deterministic. Therefore. { can be refined empirical Fisher information (EFI). This reformulation is frequently utilized, e.g., in (Jastrzebski et al. 2017] Chaudhari al. 2019), to reduce the computational complexity of gathering gradients for possible classes (i.e., expectation with respect to fi f (z, 0) as in (8)) Refer the Σ term of (10). Different from prior EFI, which is defined on the case when h(x) is uniform, Ux:), generalize the definition of EFI in terms of (xx)) E H as follows:
In(m) (0) = Ex(x) [ log f(y|z, 8) V log f(ylx,
E(e.WeE, Vx log f(u Fr, V log f(y xr, 0).
(12)
Note that (a) holds owing to
12 Note that for simplicity, we abuse the notation h(x,,)) v) used in Section ass h(z). This is exactly the same for given dataset D, situation.
27

---

Published as a conference paper at ICLR 2023
F.3.4 STOCHASTIC ORDER NOTATIONS Op AND O2
For set of random variables X,, and a corresponding set of constant on, the notation X,, - op means that the set of values Xa/an converges to zero in probability as n approaches an appropriate limit. It is equivalent with X,/a, = (1), where X, = op(1) is defined as:
lim p[ 2 c) 7-0
Ae 0.
The notation X, Op(an) means that the set of values X,, a, is stochastically bounded. That is v > 0, 3 finite M > 0, N > S.t.t /l/ > M) < for any n > N.
28

---

Published as a conference paper at ICLR 2023
G
THEOREM 1
In this section, we deal with some required sub-lemmas that are used for the proof of Lemma 8 which is the main ingredient of the proof of Theoorem I
G.1 SUB-LEMMAS
Lemma ( (Lehmann & Casella 1998
Theorem 5.1). When E, denotes convergence in probability, in Appendix hold, then there exists a sequence of MLE 6 as n oo, where 0 is the 'true' unknown parameter
and if (AO) to (A7) of the Assumption
solutions (x) D.. JaeN that (z), of the distribution of the sample.
Proof. We refer to [ehmmann Casella 1998) for detailed proof.
Lemma 2 ((Lehmann & Casella 1998), Theorem 5.1). Let ( JnEN be the MLE based on the training data set Da. If (A0) to (A8) of the Assumption D in Appendix hold, then the MLE
(U(w),D has a zero-mean normal asymptotic distribution with a covariance equal to the inverse Fisher information matrix, and with the convergence rate of 1/2.
Vu(Ou()), 0-) 2, N(,, (0")
where D represents convergence in distribution.
Proof. We refer to [ehhmann & Casella 1998) for detailed proof, based on Lemmal
1
Lemma 3 ( (Wasserman 2004). Theorem 9.18). Under the (AO) to (A8) of the Assumption 7 in
Appendix hold, we get
T (OU())))) (frr,), (0r1,) - -8) 24 N
where D represents convergence in distribution.
Proof. We refer to Wasserman 2004) for detailed proof, based on Lemma]
Lemma 4. ([Serfling 1980), Chapter 1) Let {0n} be a sequence of random vectors. If there exists
random vector 8 such that 0, Dg 0, then |0, A||2 Op(1), where || 1112 denote the L2 norm.
Proof. We refer to (Serfling
1980 for detailed proof.
2016,, Theorem 27) Let {0,) be a sequence of random vectors in Lemma 5. ((Sourati et al.
convex and compact where a, + 0 (as + oe). g Ω IR is C3 function, then
Rd and 0* Ω be constant vector such that ||0, 0*1|2 = Op((a)
g(Om) = g000000))))))))))))) (1))( 0-) + 2 (On -0) "O'"")((0))() - -0") + op(an
Proof. We refer to (Serfling 1980) for detailed proof.
Lemma 6. If(AO) and (A3) of the Assumption 7 in Appendix hhold, then Vo log P(x,, Herue (x))(²)))))))))))))))))))))))(
Ô for any joint distribution P(z, y
Proof.
Va log P(x, (true(x) = V log ['ttrue 0*)
V o log
On the first equality, (A0) of Assumption i Appendix F is used. At the second equality, (A3) of Assumption in Appendix Fiis used.
29

---

Published as a conference paper at ICLR 2023
Lemma 7. If (A0) (A8) of the Assumption 7 in Appendix hold and the case
V log p(a, (z) is non non-singular for given data (æ, ytrue (x)) satishes, then the asymptotic distribution of the log- nigllikeldoodd ratio is mixture of firs-order Chi-square distributions, and the convergence rate is one. More specifically:
log p(z, ytrue (2) (x))
-
M1 ytrue (z)) xi.
(13)
p(t, ytr (x) ( (U(G),
i=1
where (Mi(1r, Utrue (1))) are eigenvalues (1w) {
V log p(a, Vtrue (1) Ioce) (0*)
Proof. The proof based on the Taylor expansion theorem. Remind that we deal with the data (z, Vtrue (z)) satisfying V2 log p(z, Vtrue (x)) (0) is non-singular. From the property vπ((Ou=))D, -
0*) 2, NO, T (0') derived from Lemma one concludes that vull@uce),,. - 0 2l2 - Op(1)
and therefore (.).), 0* 112 Opl 1 by the Lemma
Thus, by the Lemma
log p(z, Utrue (x)) / (.())
D..) = log p(,, Virue (x))0-) + (fu()),,, 0-) Vo log p(ar, Vtrue (3+)
+ 2 (01000 0*) T log p(z, Vtrue (z) (=U))))) 0²) Op
holds. By the Lemma obtain
Strue (x) = Ō derived by Lemma 6 we can and the property Vo log Var,e
log p(z, (trae (z)10*) loog p(x, Verue (x) (x) (0000)))))
- [ -
V log pla, Verue 10 (Vi(Ou(a),,, 00] +
D, N (ō. (Cc
T
V log p(a, verwe (x)10*)] N (ō. [(o)
= N (, (ō,
Us) [oo (0)) 2 V log p(a, verne (z) (()) N (d.
Define [(x,, Verue( (I)) as - Ip(c V log p(æ, Blrue (xn and rewrite the right- hand-side dlement-wisd
1 N (G. [(x, Merue (x)) (d.
dela,, Verue (x)) (N -N0,
Σ
(x x1
2
1=1
where (11 Btrue (x)) are eigenvalues of [(x, ( Thus, the desired property obtained.
G.2 MAIN LEMMA
In this section, we derive the main Lemma, which represents the test cross-entropy loss and can be understood as Fisher information ratio (FIR) (Sourati et al. 2016).
"Suppoee I = UTUT and 2 diag(Ai, Xa). Then, N (o. u ~ ( -N (d. Thus, /V (o. TN (0, mz) =W =N( (0.II) EN Io. Ed AcN(0.1).
30

---

Published as a conference paper at ICLR 2023
G.2.1 MAIN LEMMA STATEMENT AND PROOF
Lemma 8 (FIR in expected cross entropy loss with MLE). the Assumption 1 in Appendix holds, then
ED. (p(o))/uuu) log (yla, (ayl2;000cen (o())
lim g(e 10000
(14)
Ir (-"
2
Proof. We prove Lemma 8 via two steps. First we show that the expected cross- coss-entropy loss term can be rewritten in terms of the log-likelihe likeelhodood ratio. Then, we prove that the expected log- liglliillhood ratio can be asymptotically understood as FIR.
Step Log-likelihood ratio We show that the expected oog-likelhhod ratio can be formulated as the expected test cross-entrope loss as follows:
This property holds because,
log plz, y
E ((x))l))(()) ED, (ule)
p(r, (10
log ful22,0))
(15)
log Oule),0)
E (p.
1 supplo(zx210-)
(16)
[ulo,, (U()),))
log flulz,0*)
E =p(())))(()..0
1 SSppla(aay)00-)
flylæ, (u(()))).)
E (ul
log (ulæ, (.)()) 1 supplatasll (17)
log (y]a, (u()),))))))))))
Since (A0) of Assumption in Appendix
155 and (6 hold. At (17). the properties, (i) (x, y)
Supp(g(a, y)0*)) C W and (ii) f(ylz: 0+)
W. used which is derived by
(A3) the Assumption in Appendix
Step 2: FIR Here, show that the expected test loss of MLE can be understood as FIR.
By (A0) in Assumption Tin Appendix
trivially
((x.v) v) ESupp(a(x.
V log a(x, y|0") singular)
(18)
[{ y) E Supp(y(z, ((T)
V log p(æ; (0")) singular).
holds. Since 18 and (A9) Assumption in Appendix Supp(g(z, y y/0²)) can be replaced by,
S a Supp(q(z,, y)0*)) {(x,g) E W V log p(z, y/ x00") is singular)
(19)
when calculate expectation.
We can get a result of Lemma follows:
lim nE (x.)
E D. (p
[ log (uyl,,00(e)),))
log p(z, v(0)
lim nEEpe..)) n-000
E (p..p(e)\/ol...0
1 Supp(g(=.
(20)
p(x, vl0u()).D.)
log p(a,
lim nF (
F Fp..ple)//(ll.ee>
1g
(21)
n=330
p(r, vl0
p(z,
(e
lim ED,, Ep,-p(e)f(0)..)) n+000
nog plr, p(z,vlOur
1s
31

---

Published as a conference paper at ICLR 2023
n log plz, Mtrue (x)
E (e)
lim (pp.pte)//(ll))))) n-000
18
plar, yerue (x) (z) (e(())
(22)
d
(e))
Ae( Verue (x)) [xi] 1s
(23)
Ae(2, (truc(())
1s
(24)
2 1=1
1
Tr (o)=) (-Va log p(x, (x) Ip(=) (0*)
1s
2
(25)
Tr (0(=)
{ v log p(r, y|0") (0))) 1s
(26)
2
1
[00)
$ -FL.w)
v log p(a, y 00) 1s] (0=))
2
1
Tr (00
v log a( y 0*) 1s (1(=)
(27)
2
1
Tr (all -ECeN)
v log q( q( v00)] 1 (p(z) -b]
2
(00) - 1cc0 (0 p(c)) (0-)-3
2
1
(28)
2
20 holds from Step 1. From 19) 21 satisfied. 22 and 26 holds because (z, y) is sampled from
( f(y fyl.00)) From 23) 25p. the result of Lemma is used. (27) can be obtained thanks (A0).
Lastly, (28) holds because trace satisfies the commutative law about matrix multiplication.
The final term, Tr [ (0*) )00) (a()) (8*) known as the Fisher information ratio (FIR) because can be expressed as ratio in the scalar case.
G.3 THEOREM
In this section, we finally prove Theorem [ et al. 2016).
To do so, we additionally follow assumptions in (Sourati)
G.3.1 ADDITIONAL. ASSUMPTION
Assumption 2. We assume exist four positive constants L1, L2, La, La such that following properties hold Vx E X,y (1,....c} and 0 e Ω:
T(0,x)) - log [ly|, 0) is independent of the class labels y-
Vo lolog f(ylz, 8 (0*) Vo log f2y|20, 8*) s L1
(o(=)(00)) 1/2 (0t) 1/2 _ L2
(1))) .x) I(0""x)) (8)) s Lg(0' - (0') )")
100100-''1'((00000 3 T(0,x)) T(0*,.)) TTll0 0 0'()T((',50)
32

---

Published as a conference paper at ICLR 2023
G.3.2 REPLACING 8* BY (a))).
Lemma 9. Suppose Assumption in Appendix high probability:
F and Assumption in Appendix G hold, then with
Tr [0))
lim Tr
(29)
Proof. is shown in the proof of Lemma 2 in (Chaudhuri et al. 2015) that under the assumptions the following inequalities hold with probability 1 - 8(n):
mentioned in Assumption 2
(m)) I(0", .)) 1 [(QU(()) D., x) ((m) I(0", x),
(30)
8(n)
8(n)
where B(n) and 8(n) are proportional to n, which is the size of the training set Dz,-
Because of the independence for the class labels of ((0,2), P(z), (0)) = Pa) [[))
holds for any marginal distribution P(x)
Taking the expectation to the (30) with respect to the marginal p(x) and q(x), then:
3(n) - (0*))(00) 1 [ 3(n) + B(n)
(0))
(31)
B(n)
3(n) - I((=)(0") I (, B(n)
3(n) + 3(n) (()))
Since (0*) and L (x) (Ou())) are assumed to positive definite, we can write 31 in terms of inverted matrices
B(n) B(n) = [=0))(0(0)000)) = 3(n) - (
3(n))
3
(32)
32 is equivalent to
3(n) -1
3(n) Tp(o) (O0 3 (0t) 3(n) +
(33)
3(n)
From 33
((n) Tr (0(o0)0())) Tr (00)
3(n) + Tr [[0=0 (34)
3(n)
B(n)
satisfies 16
Thus,
lim Tr (O((,))) =T
=
=Tr
(35)
holds when taking n to the 34). Note that 3(n) is proportional n.
14 (l) = B(x.))=((a.ve))) v3 log flylz,0) 0)] = Ex-P(H) (By~/(olero) [-V3 log fwya,, 0]]]
EpwP(s) ((ytso)) (1(0, x)] = EewPtH) [1 x)).
"For V two positive definite matrices A and B, we have that B 1- B-
1or < B, [[] x Tr [B] holds.
(:) < = B- A O and
== U.UT, where ... |ua]-
Then, Tr(B - A) =d ( B AjuF < because of the positive definite property of B A. Tr(A).
Tr(B A) < 0
Br
33

---

Published as a conference paper at ICLR 2023
G.4 STATEMENT AND PROOF OF THEOREM 1
Theorem 1. Suppose Assumption 7 in Appendix I and Assumption 2 in Appendix G hold, then for sufficientll large Do.), the following holds with high probability:
E (eD.ppte)s(ll.. [-
[-log log ('ul,, Ou(e),,
(36)
Tr (oo)) Tr [
- 2n
Proof. Because of the (A8) of Assumption
in Appendix (0t and Ts(=) positive
definite matrix. Thus, TT [ (0-)
(zx)
[*)] s Tr (0*) Tr (m)) (0*)
holds
From the result of Lemma 8 and in Appendix G
lim n (e.)))))e(())))lle n-
[ llog f(vl,, ((l,,,,
Ir 11 gzxz) (0*)
lim Tr (O(r)) ) Tr [0*)
m-000
holds with high probability.
It is worth noting that Theorem states that the upper bound of the MLE (e), D, test loss be minimized by lowering Tr (ptee (0(e),)) when training marginal p(z), the only tractable and controllable variable.
H THEOREM 2
In this section, we introduce the motivation of gradient norm-based importance sampling. To show why this important, we introduce the debiasing object problem for a given Dn, under sampling probability h(x) and show how to solve it in toy example because the problem difficult.
H.1 PRACTICAL OBJECTIVE FUNCTION FOR THE DATASET BIAS PROBLEM
Remark that the right-hand side term of 366 are controlled by the training and test marginals p(x), and q(x). Since we can only control the training dataset D,, p(x) and g(z), we can design practical objective function for the dataset bias problem by using EFI and Theorem aas follows:
min Tr (A h(z)EH
(37)
where (0) an empirical Fisher information matrix. Remark that EFI defined as:
(0) = Σ h(z,)Volog f 0) log f(ul=,,0)
(38)
Here, h(z) describes the sampling probability D,, which is the only controllable term. We deal with (37) in the toy example because of the difficulty of the problem.
H.2 ONE-DIMENSIONAL TOY EXAMPLE SETTING
For simplicity, we assume that D, comprises sets and m, and the samples in each set share the same loss function and the same probability mass. The details are as follows:
17 For Y two positive definite matrices A and B, Tr [AB] < Tr [A] [T(B] satisfies.
34

---

Published as a conference paper at ICLR 2023
For the given E IR, the model parameter 0 R, (0 + aa² and 2 a)² loss function arise for all data in M and m, respectively.
Dn.D, denote the trained model from the arbitrary PMF h(aa)) E H which has a constraint having degree of freedom 2, (Parr (x)) km
Concretely, each samples of M and m has probability mass ha (x) and hm respectively.
i.e.. |M] - ha (z) + [m]] Am (z) = 1, where |M and [ml] denote the cardinality of M and
m, respectively.
Let gaz (0) and (0) denote the gradient of each sample in M and at 0 E R. respectively.
Then, (MM(()) 9M (On()),D.) + |m| hm (x) Im (On()) = hold by the definition of (.()).
In these settings, our objective can be written arg M())E Tr (O(G))()))) and this is equivalent to find (1iss km))))
finding h²))))
H.3 STATEMENT AND PROOF OF THEOREM
2
In this section, we introduce the motivation for the gradient norm-based importance sampling in the toy example setting.
Theorem 2. Under the above setting, the solution of (Mar hi (x)) arg min Ir (nte))(nte))..))) is:
(x) = 19N (DU().)))) h (x)
Im (OU(w))
Z
Z
where - M (Qu()) + |ml 19m (Ou )1, and [M] and [m] denote the cardinality of M and m. respectively.
Proof. The trained model (s(e),,, [-a, a] holds trivially for any h(x) E H. By the loss function definition in the toy setting, 9M (0) = 0 + a and 9m (m) = 0 - a, V Thus, [g + (f) = 2a satisfies for V 0 [-a, a). Since the gradient is scalar in the toy setting, [n(a (On(o))) is also scalar and the same as the unique eigenvalue, that is,
Tn(ee) (P.())...) = [M] hx)())) {ag (On(e),D.)) + |mll hom (x) fgm (O.)).,). Yh(x) E H.
Thus, problem is deciding has (x) and hm (x) that maximize |M| hM(2)) (x) {gM (On(e))) |m]] (x) [gm (OMG)))()))
Because of the toy setting, three constraints are held for arbitrary 0 e [-a, , a] and h(x) E H.
1. M JM (x) + |m/] (x) 1. (probability definition)
2. M (M gM ( + |mm] (m Im (Onts)) ).)) 0. Note that convex linear sum of the sample's 'ss gradient w.r.t. h(()) = (har(z) hm (x)) is zero at the trained model ()(z).D..
3. gg (0) + 9mm (0) 2a + (0) (0) 2a (0) 2a + Im (0).
Note that this is derived by the property of predefined loss function at 0 E (-a, a).
2nd constraint is equivalent to JM (x)) (2a + 9m (On(G))))).))) + |m] hm (z) Im (fn(x)) + 9m (Qn(z).D.) 2a| M| (zr). Because of the 3rd constraint, gM (On()) 2a(1
[M] ha (z)). Then the objective is, maximizing
MM] (x) (2a(1
|M (x)) + |M] (r (x)) (x))
(39)
= |M] h=r(=)(1 [M] hxs
39 is maximized when |M|- hx(x))
2- and it means |m| hm(x)) 3.- Thus, Ma(00
35

---

Published as a conference paper at ICLR 2023
[m] M h (x)
2/ m This result is related with the trained model (o(=)) where UM (2)
JM
[ At D., [Mlg (O.())) |mlgm (DU()) satisfies and this is equivalent to | |m] 19m (D.)))))) IgM (OU()))
Um
Thus, it is consistent with our intuition that setting the sampling probability h for set M and m in proportion to 1 (0v())) and 9m (0U(a))) helps to minimize the trace of the inverse empirical Fisher information.
36


Reviews:
[{"review_id": "skX5uvr_Qu", "reviewer": "Reviewer_oD49", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Machine Learning models tend to take shortcuts when learning, taking advantage of dataset biases (or spurious correlations) within the data features, that make the task considerably easier to solve. However, these correlations are, indeed, nothing but spurious, and they may not be present during testing, making the model brittle and prone to errors. Existing approaches to solve this dataset bias problem focus either on leveraging human-labelled information, or on using attributes of the network such as their output, loss values, or feature space. This work, instead, focuses on the gradient of the samples with respect to the model parameters, and proposes a novel approach which: i) trains a biased model; ii) detects the biased samples based on their gradient; and iii) trains a new model with importance sampling to correct these biases. Theoretical and empirical results show the effectiveness of the proposed method.", "summary_of_the_review": "I think this is an interesting paper that addresses an important problem. However, I feel that it is a result of a number of systematic steps starting from LfF, and it is not entirely clear whether all the steps are necesssary to obtain good results (specially the gradient norms, which have not been tested in isolation).\n\nThis, combined with a lack of clarity, and the need to clarify some experimental results, lean me towards rejection for now. I am sure a polished version of the current work will be a pretty good paper.", "strength_and_weaknesses": "**Strengths:**\n- S1. The algorithm is clear, simple, effective, and therefore of interest to the community.\n- S2. Experimental results are strong, and different ablation studies were performed.\n- S3. Theoretical justification is appealing and sensible.\n\n**Weaknesses:**\n- W1. Different ablation studies would be needed to understand where does the improvement come from. Specifically, the proposed approach is clearly inspired by LfF, and the difference are: i) sequential training of both models, instead of interleaved steps; ii) different weighting function; iii) re-sampling instead of re-weighting gradients (+ data augmentation). A proper ablation study should shed some lights on which steps are giving an edge to the proposed approach with respect to LfF.  \n   - **Edit.** I do not want to remove the point, as I think it is still important to clarify in the main paper, but I have just read Appendix C6 and C7 where part of these studies are done. After reading them, it is unclear to me whether using the gradient norm improves the results, as the proposed method with single-stage and reweighting performs similar to LfF. \n- W2. I am not sure how fair it is to introduce the data augmentation as part of the approach. Table 4 clearly shows that data augmentation is really helpful, and I think a fair comparison would apply data augmentation to the other methods as well.\n- W3. I don't see a reason to run only three seeds on these experiments, as they are not that heavy. Also, there are not statistical tests performed, but rather the best on-average results are marked in bold.\n- W4. While sensible, the theoretical analysis makes quite some assumptions. I am particularly concerned with how realistic is the first point of assumptions 2 ($\\nabla_\\theta^2 \\log f(y|x,\\theta)$ is independent of the class labels $y$). However, while I find the toy example really confusing (more on that later), Figure 3 is quite helpful.\n\n**Questions/Comments:**\n- Q1. Results on CCIFAR, CMNIST, and BAR on Vanilla and LfF are significantly different from those reported by LfF. Could the authors explain where do the differences come from?\n- Q2. Are you sure the cross-entropy loss is right? I think it is missing terms. See, for example. [here](https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html).\n- Q3. What is \"an official dataset (e.g. MNIST)\" supposed to mean? What does a dataset \"official\"?\n- Q4. Was data augmentation used in the appendix C7?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XxR6nfCYEs_", "reviewer": "Reviewer_jyHT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper debiases models by resampling with probabilities proportional to gradient norms. It uses the generalized cross-entropy loss from LfF to train a biased model and debiases the main model by sampling probabilities from gradient norms. It leverages the observation that rarer samples tend to have higher gradient norms, so their resampling scheme assigns higher weights to rarer samples.", "summary_of_the_review": "Overall, I think the approach is simple and has practical applications -- from the results it seems to be the SoTA amongst all the implicit de-biasing methods. \nHowever, the paper does not report all their comparison methods on all the datasets, which is really strange.\nI am willing to raise the score if the authors properly justify why that is the case.", "strength_and_weaknesses": "Strengths:\n\n[S1] The approach is simple and logical. It leverages the fact that rarer samples have higher gradient norms. This seems to be the first paper to use gradient norm instead of the commonly used output space-based measures for re-sampling/re-weighting.\n\n[S2] Mathematically, the paper interprets the debiasing problem as that of min-max (minimizing loss) of maximally difficult samples, which can be relaxed to minimize the trace of inverse Fisher Info. They show how the gradient norm minimizes this theoretically for 1D and empirically for higher dimensions. This provides a strong justification to the choice of gradient norms.\n\n[S4] The results on image classification datasets used by previous debiasing methods and Civil Comments (from WILDs) show gains over existing methods. However, I have a concern in [W1]\n\nWeaknesses:\n\n[W1] Sec 4.2 mentions 8 baselines, but not all comparison methods are run on all the datasets. For example, EIIL seems to be reported only for CivilComments and BPA only for CelebA. I did not find a good justification for this in the paper; it seems like all the methods can potentially be run on all the datasets.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7wiY0nCSKdq", "reviewer": "Reviewer_Lo2Z", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a interesting approach to achieve group robustness without group label by re-sampling training data based on last-layer gradients, the work is simple, comes with rigorous theoretical justification and extensive experiments.", "summary_of_the_review": "As stated in the summary, I find the work to be simple, comes with rigorous theoretical justification and extensive experiments. Therefore I believe it carries sufficient scientific novelty and empirical significance to justify an acceptance. ", "strength_and_weaknesses": "Strength: \n* Comparing to previous approach, this current paper proposed a distinct signal (gradient magnitude) for quantifying the learning difficulty of examples, and proposed a simple learning-time resampling approach to debiasing. \n* The proposed methodology is simple and general purpose, and therefore worthwhile put it out there for the community.\n* The method is comprehensively evaluated across multiple benchmarks and comes with sufficient ablation study.\n\nWeakness:\n* None.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster