Mitigating Dataset Bias Using Per-Sample Gradients From A Biased Classifier
Authors: Sumyeong Ahn
Se-Young Yun
Venue: ICLR 2022 Submitted (2022)
Keywords: dataset bias
debiasing
representation bias

Abstract: The performance of deep neural networks (DNNs) primarily depends on the configuration of the training set. Specifically, biased training sets can make the trained model have unintended prejudice, which causes severe errors in the inference. Although several studies have addressed biased training using human supervision, few studies have been conducted without human knowledge because biased information cannot be easily extracted without human involvement. This study proposes a simple method to remove prejudice from a biased model without additional information and reconstruct a balanced training set based on the biased training set. The novel training method consists of three steps: (1) training biased DNNs, (2) measuring the contribution to the prejudicial training and generating balanced data batches to prevent the prejudice, (3) training de-biased DNNs with the balanced data. We test the training method based on various synthetic and real-world biased sets and discuss how gradients can efficiently detect minority samples. The experiment demonstrates that the detection method based on the gradients helps erase prejudice, resulting in improved inference accuracy by up to 19.58\% compared to the other state-of-the-art algorithm.

Under review as a conference paper at ICLR 2022
MITIGATING DATASET BIAS USING PER-SAMPLE GRADIENTS FROM A BIASED CLASSIFIER
Anonymous authors Paper under double-blind review
ABSTRACT
The performance of deep neural networks (DNNs) primarily depends the figuration of the training set. Specifically, biased training sets can make the trained model have unintended prejudice, which causes severe errors in the inference. Although several studies have addressed biased training using human supervision, few studies have been conducted without human knowledge because biased infor- mation cannot be easily extracted without human involvement. This study proposes a simple method to remove prejudice from a biased model without additional information and reconstruct balanced training set based on the biased training set. The novel training method consists of three steps: (1) training biased DNNs, (2) measuring the contribution to the prejudicial training and generating balanced data batches to prevent the prejudice, (3) training de-biased DNNs with the balanced data. We test the training method based various synthetic and real- weal-world biased sets and discuss how gradients can efficiently detect minority samples. The experi- ment demonstrates that the detection method based on the gradients helps erase prejudice, resulting in improved inference accuracy by up to 19.58% compared to the other tatte-ofthheaan algorithm.
INTRODUCTION
Over the past decade, deep neural networks (DNNs) have been a focus of research owing to human- like performances in various tasks, e.g., image classification (Krizhevsky et al., 2012). object de- tection (Girshick, 2015), and image generation (Goodfellow et al., 2014). Despite these impressive results, deploying DNNs directly real-world problems remains significant challenge due to the difficulties in obtaining well-curated training sets. Specifically, unintendedly biased information in training sets causes prejudice, resulting in wrong decisions at inference time (Torralba & Efros, 2011; Shrestha al., 2021). For instance, most "ski" images contain "skier." This unintended correlation can recommend a wrong shortcut for "ski" by examining the person.
Prior studies have employed human supervision (i.e., providing additional labels related to the bias) (Kim et al., 2019; McDuff et al., 2019; Singh et al., 2020) or giving a specific information about biased domain (Lee al., 2019; Geirhos et al., 2018), which is very expensive, reduce the influence of a "skier" on "ski" class images. Recently, studies on replacing human labor with DNNs have been actively discussed (Li & Vasconcelos. 2019; Nam et al., 2020; Cadene et al., 2019; Bahng et al., 2020; Clark et al., 2019; Le Bras et al., 2020). Typically, two separate networks have been used: the biased network and the de-biased network. biased model learns to replace human intervention and teaches a de- di-biased model using the prejudiced knowledge of the biased model.
Previous studies based on the two mentioned networks approaches have had two directions: adjusting objectives and resumpling. Adjusting the objective refers to method that obtains weighted loss or additional regularizers for each sample differently, where the weights or regularizers are obtained from the biased model (Nam et al., 2020; Cadene al., 2019; Bahng et al., 2020; Clark et al., 2019). Methods that adjust the objective have been proposed frequently due to their several advantages, such the implementational simplicity. However, according to (An et al., 2020), adjusting--bjeecte approaches suffers instability with stochastic gradient descent (SGD) type optimizers owing to the high variance of learning weights per sample.
Alternatively, resampling methods could reconstruct balanced sets. Previous research (Le Bras et al., 2020; Li & Vasconcelos, 2019; Li et al., 2018b; Kim et al., 2021a) has focused which samples

---

Under review as a conference paper at ICLR 2022
should be oversampled to build a balanced set. However, almost all prior works has focused on expensive human labor to tune hyperparameters related to the resampling densities of data points. For instance, it is difficult to set the amount of resampling of each sample without knowing how much biased the training set is.
The de- he-biasing methods, belonging both adjusting-objectivv and resampling approaches designed to emphasize samples that are hard for a model to learn. However, a few noisy labels (i.e., mislabeled samples) in the training could significantly interfere with such de-biasing because might be difficult to discriminate between two types of hard samples: noisy samples that the model should not learn and rare samples that have learn. Although training sets have noisy labels in practice (Natarajan et al., 2013), to the best of our knowledge, no study considers the de- te-bbasing problem under noisy label cases.
Contribution. In this study, we propose a score-based resampling scheme for de-biasing, which does not require hyperparameters related to training set reconstruction. Instead of hyperparameters. we construct two types of scores that leverage gradients the biased model trained on the given biased set. The proposed scores determine the required proportions for each sample in training set reconstruction.
Before designing scores, explain hypothesis that gradients have remarkable differences between samples generating prejudice and the others, and check it empirically. The hypothesis is based on the following observations. First, the gradient magnitudes (especially the Euclidian norm) of the samples which contributes to the prejudice are relatively smaller than those of the others. Second, samples that make prejudice and the others do not have significantly different gradient directions. measured by the estimated likelihood of the von Mises-Fisher (vMF) distribution (Banerjee et al., 2005; Lee al., 2018).
Based on these observations, we propose two types of scores and a resampling- based le-biasing method. The proposed method consists of three steps similar to previous resampling studies (Li & Vasconcelos, 2019; Le Bras et al., 2020; Kim et al., 2021a). The first step is to train biased model based on a biased raw training set. Afterward, the scores are obtained by the biased model. The mini-batch sampler generates balanced mini-batches using the scores, and it feeds them into an ultimate model for de-biased training.
In addition, we investigate the side effects of de-biasing methods when incorrectly labeled samples are in the training set. We observe significant performance degradations of de-biasing methods with a small portion of noisy data. To alleviate the negative side effects of noisy data, we first de-noise the training set using loss values from the our observation that noisy label samples have higher losses than clean samples and then run de-noising algorithms. The de-noising step successfully protects the de-biasing performance.
Finally, demonstrate the effectiveness of method using various biased benchmarks (Col- ored MNIST (Nam et al., 2020; Kim et al., 2019; Li & Vasconcelos, 2019; Bahng et al., 2020), Watermarked MNIST, Cartoon (Royer et al., 2020), CelebA (Liu al., 2015), Biased action recogni- tion (Nam et al., 2020), and ImageNet/ / ImageNet-A (Bahng et al., 2020; Deng et al., 2009; Hendrycks al., 2019)). In most experiments, the proposed method outperforms the current state-of ofate-ofthee-eee f-tthe-art methods. In particular, the proposed method improves the accuracy of the unbiased test by 68. 18.00% from the baseline with a small performance degradation of the biased test of 0.91%. Moreover, we compare the proposed method with the previous resampling method, REPAIR (Li & Vasconcelos, 2019). The proposed method improves the average accuracy by 71.04% > 98.14%.
RELATED WORK
Mitigating bias with human supervision. In (Goyal et al., 2017; 2020), generated a de-biased dataset using human labor. A collection of studies (Alvi et al., 2018; Kim et al., 2019; McDuff et al., 2019; Singh et al., 2020; Teney et al., 2021; Ramaswamy et al., 2021; Tartaglione et al., 2021; Geirhos et al., 2018; Wang et al., 2018; Lee et al., 2019) aimed to mitigate bias based on one of two types of human supervision: explicit bias labels and implicit bias information. Other studies (Alvi et al., 2018; Kim et al., 2019; McDuff et al., 2019; Singh et al., 2020), have used bias labels for each sample to reduce the in influence of the bias labels when classifying target labels. (Tartaglione et al.,
2

---

Under review as a conference paper at ICLR 2022
2021) proposed the EnD regularizer, which entangles target correlated features and disentangles biased features. Several authors (Alvi et al.. 2018; Kim et al., 2019; Teney et al., 2022) 1) have designed DNNs shared feature extractor and multiple classifiers. In contrast to the shared feature extractor methods, (McDuff et al., 2019; Ramaswamy et al., 2021) constructed a classifier and conditional generative adversarial networks, generating test samples to check whether the classifier is biased. (Singh et al., 2020) proposed a new overlap loss defined by a class activation map (CAM). The overlap loss reduces the overlapping parts of the CAM outputs of two bias labels and target labels. Different approaches (Geirhos et al., 2018; Wang et al., 2018; Lee et al., 2019) have assumed that the characteristics of bias features are known and prevent learning the implicit bias features. To avoid the known texture bias of ImageNet (Deng et al., 2009), (Geirhos et al., 2018) generated stylized ImageNet, and (Lee et al., 2019; Wang et al., 2018) inserted filter in front of the models so that the influence of the backgrounds and colors of the images can be removed.
Mitigating bias without human supervision. To reduce human intervention, recent research (Clark et al., 2019; Cadene et al., 2019; Bahng et al., 2020; Nam et al., 2020; Li & Vasconcelos, 2019; Li al., 201 8b; Le Bras et al., 2020; Darlow et al., 2020) has used ensemble-based methods, using two separate networks with different purposes. The biased model learns biased information and even empowers it. Then, the biased model is used to train the de-biased model as a pathfinder, which provides information about the bias features.
The ensemble idea has been used as a key component in several studies, (Clark et al., 2019; Cadene et al., 2019; Bahng et al., 2020; Nam et al., 2020). (Cadene et al., 2019) proposed a method known as RUBi, which multiplies the sigmoid output of the biased model by the softmax output of the de-biased model. (Clark et al., 2019) summed the output of the two models to train the de-biased model with an entropy regularizer. (Bahng et al., 2020) proposed ReBias algorithm using the filleet--Shhiit independence criterion (HSIC) as an objective of the de-biased model, by independent of the biased model. Learning from failure (LiF) (Nam et al., 2020) used generalized cross-entropy-best based training that emphasizing the usual samples but ignoring unusual samples.
In contrast, biased and de-biased models are trained sequentially (Li & Vasconcelos, 2019; Li et al., 201 80; Le Bras et al., 2020; Darlow et al., 2020). Researchers computed the weights to reconstruct the training set from the biased model and trained the de-biased network on the reconstructed training set. The REPAIR (Li & Vasconcelos, 2019) and RESOUND (Li et al., 2018b) methods proposed cnntualliiffrmation--o based weights to create a downsampled dataset. In addition, AFL.ite (Le Bras al., 2020) proposed accuracy-based weights from multiple biased models, unlike LAD (Darlow et al., 2020) and BiaSwap (Kim et al., 202 which generated new images using an autoencoder based on the latent representation of the biased model.
3 BIAS PROBLEM FOR TRAINING A CLASSIFIER
3.1 UNINTENDEDLY BIASED TRAINING
Suppose that training set D is composed of images x; as illustrated in Figure 1. Each image can be described by a set of attributes {a ..., dk.,.. }. (e.g., ("digit 0", "digit I",... "red", "green", ... "thick" J). The goal of the train- ing classifier is to find a model fe that correctly predicts the C' target attributes "digits." y {at,..., ate 1 =I ("digi o", "digit I"....). Remark that target attributes are also interpreted as class. However, we focus on the case in which another attributes exists, "colors," where b b= {Ab, ..... Ube } = "ord"""gree", ...) a y. which is highly correlated to the target "digits" (i.e., H(t, (4b,) ) R 0). Such an correlated attributes and training set are denoted as the bias attributes and biased training set, respectively. In addition, the "color-digit" correlated and uncorrelated samples, in the top two rows and the bottom row in Figure 1, are called the majority and minority. We use the notations M and m for the majority and minority sets, respectively. For example, "red-digit 0" image is majority sample, while "orange-digit 0" one is minority sample.
2
3
4
5
6
&
0
2
3
4
5
6
7
8
2
3
S
Figure 1: Colored MNIST.
When we train a model based on the biased training set D, the model may infer the bias attributes b, not target attributes y, which is problematic because the target is y, not b, and refer to this as an
3

---

Under review as a conference paper at ICLR 2022
unintended bias problem. For example, if the model trained on the images in Figure 1 suffers from this problem, this model outputs 4 when orange.0 image is given.
3.2
WHEN DOES THIS PROBLEM HAPPEN AND INTENSIFY?
ImiDDi 0.11
Iml 0.05
100
08
1.00
1.00
Target altriibute
altribules
Accuracy
6
60
-0.90
44
09 09
0.90
60
00
Major Minor Average
12
0.85
44
0.85
40-
20
0.5
2
ImMDI (%)
attributes
Blas attributes
Figure 2: Accuracy vs. |ml/|D|
Figure 3: Confusion matrices for |M]|D] = 99% and 95%.
(Nam et al., 2020) argued that this problem arises when two conditions are met simultaneously: (C1) bias attributes is easier to learn than target y, and (C2) highly correlated b to y. We focus on observing the effect of (C2) on the unintended bias problem among the two conditions when (C1) holds. To understand (C2) precisely, we check the effect of the minority ratio |ml/|DD on the accuracy. A detailed description of the training setting is provided in Appendix A.2.
We investigate the test accuracy of the samples in the majority and minority sets. As in Figure 2, the average performance drop of the minority samples intensifies when the minority ratio |ml/ [] decreases. On the other hand, the accuracy of the majority samples for all cases are not dropped. More precisely, we plot the confusion matrix to how poor the performance is when the bias attributes b and the target attributes y are not aligned according to the minority ratio |m|/| D) As in Figure 3, the increments of the minority ratio does not make the performance change of the aligned samples located in the diagonal, but other entries change a lot. These results can be interpreted as when the ratio /ml/) becomes small, the model highly likely infer the bias attributes 6 rather than target attributes
4 DE-BIASING USING PER-SAMPLE GRADIENT INFORMATION
When a given training is biased, the model learns b over the intended target y, and it strengthened when |ml/|D| approaches zero. If this is the case, it is desirable to modify the training set to a larger |ml/ / D so that the model learns the target y rather than the biased attribute b. To do so, we considered constructing a new training set D' [(1%, v() such that |m' increases.
Similar prior resampling- based work (Li & Vasconcelos, 2019), we sequentially train two separated networks: biased model fo, and de-biased model fe.- The biased model, fo, is trained based on the raw training set D. After sufficiently converging point t, - 6.. the model computes gradients V0,- = VoLCn ys; f) of all the samples (Ri, y:) E D. Then, we compute the sampling probability (i)) of the ith sample using the per-sample gradient V0, break (C2) by balancing between M and m. Finally, we train the de-biased model fe, using a rejection sampler that creates a balanced training set D' based on the sampling probability (i).
4.1 PER-SAMPLE GRADIENT-BASED SCORES
As the first step of de-biasing, we train the biased model without modifying training set D. The converged model fg. (.) contains in-depth information about the distribution of the training set in its model (especially, through the per-sample gradient). Therefore, we leverage the per-sample gradient, a.. of (I, ys). which is dissolved in the biased model.
V8,- 3C.z (3.))))))
80
Why the per-sample gradient? The majority and minority samples exhibit a clear difference in the per-sample gradient for the model fě, owing the following reasons. (R1) Each sample has its own loss landscape, and similar representation samples have similar loss landscapes (see Figure 4(a)). In other words, the minority samples have different loss landscapes from those of the majorities, which implies that local minimum cluster is composed of similar representation samples. (R2)
4

---

Under review as a conference paper at ICLR 2022
Portin Diudd'
100
Bir Indled
Porten
100
(a) Sample images and the per-sample loss landscape
(b) Do E. Lce
(c) Histogram
Figure 4: Loss landscapes of the model trained the biased training set. (a) and (b) represents the per-sample and aggregated loss landscapes. Top and bottom row depict the results of the case. The red arrows the convergence point and minimal point. We draw the loss landscapes based on ||V The values are gradient direction and loss difference (Li et al., 2018a). See Appendix A.2 for the detailed experiment setting. (c) represents the ratio of majority and minority samples at each bin respect to the directional and magnitude.
From the averaging property of the batch-based optimization methods, the influence of the minority samples is diluted of according to their ratio. Thus, the model trained the biased training set converges to proximity point the minimum point the majorities and is reflected the gradient of each sample. The red arrows in Figure 4(a) and 4(b) indicate that the majority samples (top) have shorter gradients than the minorities (bottom) for both per-sample and aggregated cases. Similarly, the direction of both cases is significantly different. Additionally, seen in Figure 4(c), the portion of minority samples in the lower directional likelihood and higher magnitude bins much larger than the majority samples.
Gradient-based score. Based on the intuition of per-sample gradient and empirical results, design gradient-basee based score S(i S (1r,, v.): which is computed for each class separately.
The score is computed using two base scores, the magnitude score M (i) and direction score D which are defined as follows:
(i) ド (1/1108 (111 )
(1/1100.12)
(i) E EaeiPan
D(I)
(1) Magnitude score (i): The contribution to prejudicial training is inversely proportional to the magnitude of its gradient (x 1/1]V[;|l). We used Euclidian distance to measure the magnitude.
(2) Direction score D(i): The contribution to prejudicial training is also proportional to the directional likelihood (x Pai Par(t)). (i)). For the directional likelihood, the estimated von Mises-Fisher (vMF) distribu- tion is used to understand the concentration of the directional distribution. The vMF distribution is defined as follows:
Definition 4.1 (von Mises- Fisher Distribution). The pdf of the vMF K) is given by
fa(u; u, k) = (C(())e(nu
on the hypersphere Sd-1 c. R2, where determines how much samples are concentrated based on the distribution along the mean direction u, and Ca(x) is a normalization constant determined by dimension and the concentration parameter
To estimates the parameters of the assumed vMF distribution,
approximated maximum likelihood estimates (MLE) solutions and Å. For normalized gradient vectors V0,
V
[V
the MLE solutions î and & can be obtained from à N and R ~ (1-F2)*
vo.
F(d-F) where
Em voll
EN vo.1l (see Appendix G).
Then, the normalized directional score D(i) is defined in (I), where Pai = fa (V,;; ú, R) is the approximated vMF distribution.
To relax the computational constraint, we select k-dominant sub-dimensions, denoted by the set K arg max Σjek (E, V (k), where V corresponds to the gradient of the j-th dimension.
KKN
5

---

Under review as a conference paper at ICLR 2022
4.2 DE-BIASING USING GRADIENT-BASE BASED SCORES
Balanced batch selection. By the definitions of the scores, the sample in the majority set M (or the minority set m) must have higher (or smaller) (i) and D(i), respectively. Each sample must interact inversely with the two scores to treat the samples in the minority set m similarly to the samples in the majority set. To do so, we design aggregated score S(i) based on the harmonic mean, and sampling probability pa (i( as follows:
1 M(i D(1) )
C
S(i)
(t)
S(i)
where À is the balancing hyperparameter emphasize specific score between M and D(i), and C is the normalization constant defined as C = min; (i)), which makes ps (i) = for the smallest ((i) sample. From the harmonic-mean, S()) has higher value when one of the scores is high.
To construct a balanced mini-batch, our algorithm use a rejection- sampling- based uniform sampler, Algorithm 1. Each sample is rejected with a probability/ 1 - (i),,, which implies that the rejection is proportionally to the scores. From the property of rejection sampling, such sampler generates uniform batches.
Algorithm 1 Uniform Sampler
I: Sompling prob. P... Batch size & while [B < do
< (0,, p- (4(0, 1) p S (.))) then; - BUi end if 6. end while
Gradient-based de-biasing. The ultimate de- biasing algorithm consists of three steps. First, the biased model Sa, is trained on the raw data samples with GCE loss (Zhang & Sabuncu, 2018; Nam et al., 2020). The GCE loss con- trolled by q guides the biased model to ignore the minority samples in the training set. Then, all raw samples are input into the biased model train the de- biased model fea using a balanced mini-batch sampler. This sampler generates balanced mini-
Algorithm 2 De biasbisiig using uniform sampler
A(-).samliing 2: /a* STEP 1: Train fo. **/
I: D, A. I. T. aug. alg. A(-), sampling alg St-)
3: for t=1.2..... do
4: Draw a mini-batch (X.Y) (lx, WilL, from D
5: Update Og + 8. 15) Vo Exx.) g.
6: end for
7: /** STEP 2: Calculate p.(i) **/
8: Calculate pa(t) for allali
to compute ps (i). The last step
9: /** STEP 3: Train fo, **/
1=1,2...,T
1I: Draw mini-batch ((r.wilf from the Alg. 1: (())(0)
12: Update 02 +- Og L(x.)) (Af()))) y)
batch based on the sampling probability Ps (i).
13: end for
We use the typical image data augmentation method A(-) (e.g., rotate, resize) to avoid over-fitting.
DE-NOISING A SMALL PORTION OF THE NOISY LABELS
ImMDI
1%
Imi 1%, noisy label ratio 10%
ImMDI = 10%, noisy label ratio 1%
wio
1%
5%
10%
Loss (Nomallized)
Clean, M Clean,
Noise, M
Loss (Nomalliee)
Clean, M Clean,
Noise, M Noise,
100
1.0
Noise, m
1.0
Aocuracy
0.5
0.5
RCBLLo
REFUIR
Ous
0.5
1.0
0.5 Entropy (Normalized)
1.0
Algorihhm
Entropy (Normalized)
Figure 5: Acc. drops.
Figure 6: Entropy vs Loss.
Why is noisy label a problem for de-biasing algorithms? All de-biasing algorithms work by emphasizing rare samples. For example, the proposed method operates by reconstructing mini- batches in proportion the rarity. However, when small portion of incorrectly labeled samples, called noisy labels, is in the training set, the de-biasing algorithms accentuate the noise labels due to their mechanism of enlarging the influence of the rare samples. Therefore, as presented in Figure all de-biasing algorithms suffer a performance decline.
Distinguishing minority and noisy labels via compensated loss. Recent de-noising methods (Yu et al., 2019; Yi & Wu, 2019; Li et al., 2019; Kim et al., 2021b) have mainly focused on ignor- ing unusual sample, which may be noisy labels. However, by comparing de-noising mechanism with (Kim et al., 2021b). previous de-noising algorithms also ignores minority samples which must be highlighted not discarded, see Appendix I for details. Therefore, delete noisylabels in which the minority samples are preserved. To do so, we observe the loss statisties of the minority samples and noisy labels under various Colored MNIST settings with
design module to

---

Under review as a conference paper at ICLR 2022
(mml/ [mm/|DD,, noisy label ratio) = (0.01.0.) 0.1) and (0.1, 0.1) (see Appendix B.3). As depicted in Figure 6, noisy labels have a higher loss than clean labels, regardless of whether the sample is in the majority or the minority set. The trained model that ignores rare samples, outputs the correct results, whereas the given label for calculating the loss value is incorrect.
However, the loss of the clean minority samples increases, owing to the rarity. To compensate, suggest the compensated loss L,, reducing the influence of the rarity, and the splitting criteria to distinguish between the clean samples and noisy labels:
< <6. noise = {(0,, v.). 2 0} where L. LoE(E,, H(fa(zi))
Delean = {( V.lC,
(2)
Note that H(fo(z:)) represents the softmax entropy of sample Is. and 8 EisN Ce/ EieN H (fo(z)).. In Figure 6, the black lines represent the threshold ö, and the lower part of this line represents Detean. whereas the upper part represents Dnoise, respectively. Other discrimination results are depicted in Appendix E.
De-noising module. We design a de-noising module that is applicable in front of all prior de-biasing algorithms from the proposed criteria. As presented in Algorithm 3, this module comprises three simple steps. First, a noise model parameterized by 6n trained on the raw biased and noisy training sets. Then, we split Detean and Doois according to the criteria (2). After splitting, we run the de-biasing algorithm using Detean-
Algorithm 3 De-noising & De-biasing
I: Train network fu, with CE loss fee Split Daem and Dusie based on (2) 3: De-binsing using Daes
6 EXPERIMENTS
We evaluated the proposed model for two types of benchmarks: synthetically generated bias (referred as controlled biased benchmarks) and raw bias (real-world biased benchmarks). We compared the results with the officially available recent de-biasing methods, RUBi (Cadene et al., 2019), Learned- MixinH (Clark et al., 2019), ReBias (Bahng et al., 2020), AFLite (Le Bras et al., 2020), LiF (Nam al., 2020), REPAIR (Li & Vasconcelos, 2019) and the vanilla model. Each baseline was reproduced using the official codes for each algorithm (see Appendix C). Morcover, we optimized all algorithms based on the 10% of the biased training set. For the experiments, three types of convolutional neural networks were used with various regularization methods (batch normalization (loffe & Szegedy, 2015), dropout (Srivastava et al., 20, and weight decaying (Moody & Hanson)). We report the detailed settings in Appendix 1. In addition, we determined the test under flipped labels in MNIST variants with various noise ratios of 1 10%, 5%. and 0% to verify the side effects from noisy labels and check the protecting performance of the de-noising module (see Appendix A).
6.1 BENCUMARKS
6.1.1
CONTROLLED BIASED BENCHMARKS
2
4
R
0
2
3
4
5
6
f
3
4
S
8
0
2
4
5
6
7
3
O
2
S
6 Z
3
Figure 7: Examples of MNIST variants (Left: CM, Right: WM).
Figure 8: Cartoon.
To precisely examine the de-biasing performance, we generated controlled biased benchmarks using well-known data: MNIST (LeCun et al., 2010) and Cartoon (Royer et al., 2020). Various bias ratios, D E {0.995, 0.99, 0.95). were used for all the controlled tests. All benchmarks were divided into a biased training set D.r. biased validation set D (10% of Dtr), and test set Drc comprising the majority M and minority m. (see Appendix B.1 for detailed data construction rules.)
MNIST Variants. The MNIST dataset comprises a grayscale digit data with a handwritten style. We injected two different biases, color and different objects (specifically, Fashion MNIST (Xiao et al., 2017)), which are colored MNIST (CM) and watermarked MNIST (WM), respectively. This main task is classifying the shape of the digit without examining the biased attribute color of the
7

---

Under review as a conference paper at ICLR 2022
99.5%
99%
95%
type
Peat
Ma jor
Minor
Moat
Ho jor
Vall Oeai
Major
MNIST
Vanilla
99.49 sm.
99
26.58s im
99.37
-9.97
51.97
99. 00
29.87
90.92- 140
Lammeiiiiini
99.61
99 09.93gmm
39.55274
99.53gem
99.87g
65. 65.28
99 48
99 80g
91. 01.67:91
RUBBI
99 49
99 99.95gmm
29, 39.36e11
99 99,43
99.90.
36.97-19
99, 99.56
99 90
92. 190-1
98.35aa1
98.05
58.36=..
95.75-
81.42=xx
-9.15-um
96.55-44
1M
99.47..0
09.87..0
51
AFI
30
08.78
29.592.1
08.97g
09.272-00
14:
99 04
37
REPAIR
99 09.65.amm
99.93
42. 42.1
90.52gmm
99, vm
70.3 m
99 20
91.39 1x
Outs
99.13 zon
99.07
96,50, sua
98.91 eas
99.24
97.76 93s
98. ou
99.37-
97.42
Watermarked MNIST
Vanilla
98.70a
51,72s 99.57
98,80=
99.40z-us
58.02-144
29.40..a3
80.56
LoamedMixinH
65,95,
99.174
99,74..uz
99.50-4..
99.77
RUBL
98.80
c2
52.53: is
98,43
99.67.
62.32
98.53
99.30
83.22
445
LFF
98.60k 2M
98. 98.74
47.26623.
97.40l
97. m
54.02=
83. 83.19
ReBiaa
98.93+117 1m7
99 c9
48.82:442
98.53 emn
99.34-um
54.98
98.83
99.54-13)
80.62
1.9
AFLite
98.40:
64.80:14 con
98.67 oz
99.21
75.37
98.60
99.01
90.05
REPAIR
99
99.57amm
55.4623.
98.67=.m
-9.57
60.75
98.
13.19-m0
99.53
99.70.
85.99 2.
99.10
99.67
91.98
99.03
99.30...tx
96.19..
Cartoon
Vamilla
99. 99.80 104
99 19.99.cm co
16.6412 1X2
99. 0V.422.
99. --8
$6.25
99. 19,110 02:
20.95_00r
78. 26.266:25
LoamedMixinH
99.90s sia
99.99 ca
60.93.
99.58 cus
99.87
73.88
99.57-uux
99.99.uu12
88. 58.48
RUB:
99.72.
99.
43.88
98.97
99,71 ozz
52.61
98.27..04
99.57..44
70.11
99.72.104
45,42:1.
55.17 99.91..4
99.11..a
99.81:am
81.02.
ReBias
99.792mm
100.02.co
55,47
V9.5K
100.02:000
66. =z0s
99. 10
99:47.mm
37.56-2211
AFLite
99.38 sos
99,36 om
51.49, 1.53
98.96 cun
99,54 ou-
55,77
98.28.-ox our
99.70-
72.17-1M 1.9
REPAIR
99.87
100.0. ea
47.59; 2:11
99,87)
100.01
55.41 1.4
99. 19.16..44
99.941.00
80.34 547
Ours
99.01
98 98.622.0
80,22
99.06.
98. 08.962.04
83.28
98. 98.54..0z
29:000
94.23
Table 1: Average test accuracy and standard deviation (three runs) on experiments with the MNIST variants under various bias ratios. The best accuracy is indicated by bolded for each case. digit or Fashion MNIST objects. As an example of the CM, class 0 is divided into major and minor sets colored in red and the others. Similarly, in WM, the major set of class I contains rouser, whereas the minor set contains other objects as watermarks at the top-left comnr. Examples are depicted in Figure 7.
Cartoon. A Cartoon (CT) (Royer et al., 2020) is an avatar image with 18 corresponding attributes (e.g., hair style). We select two of them, hair color and face color, for the target and biased attributes. To create a bias, the major set is whose two attributes are bonded was subsampled. For example, in Figure 8, the face color of blond hair is black, except for few.
6.2.2 REAL-WORLD BIASED BENCHMARKS
We used three well-known biased benchmarks to determine the de- de-biasing performance on com- plex benchmarks: Biased Action Recognition (BAR), CelebA, and ImageNet/ImageNet-A (see Appendix B.2 for details).
Biased Action Recognition. Biased action recognition (Bahng et al., 2020; Nam et al., 2020) is motion classification set with a background bias. For example, in the class cl limiing,, major and Minor samples have rockwall 11 and ice cliff backgrounds, respectively.
CelebA. (Nam et al., 2020; Liu et al., 2015) CelebA has facial images with 40 binary attributes. Among these, we used heavy ma makeup and gender attributes for target and biased attributes. In the heavy makeup class, major is the fema le attribute, whereas ma le is the minority attribute.
ImageNet/ImageNett ImageNet (Deng et al., 2009) is a well- image classification dataset. However, (Hendrycks et al., 2019) claimed, a model trained on the ImageNet data easily fails to classify unusual backgrounds. This is because ImageNet has a biased background (e.g., almost all Frog images are considered in the swamp, which means major sample). As a de-biasing performance evaluation benchmark, we used ImageNet for training and to test ImageNet-A images with unusual backgrounds.
6.2 EVALUATION
Controlled benchmarks without noisy labels. The accuracy values of two major and minor sets reported in Table 1. In particular, we first examined the accuracy of the vanilla algorithm, the ultimate baseline. Overall, the vanilla algorithm achieved nearly 100% accuracy for the major case, the vanilla failed to infer with an accuracy similar that of the major case.
8

---

Under review as a conference paper at ICLR 2022
Die Eal Nite rib
W7774
Wer
We lire
Bossking
Wakest dosebii
Wiraaz de-saang
Win docaliis
Wakker dorabiny
Wab de
syp
Mojorr
Calorel MNINT
Vaslaa
29 50
51,77
25 B
72
30
1005
120 Doan
31.2+10
S1/02
4883)mm
2031
56 Meue
51
15
99 2m
11.9
120 fl
3/
Watensarked MNIST
Vastaa
99:6
5602
47.37x10
16.
52586
53.2-45
52.5
L.P
mm
$6.02 10
1600
17
66.48cm
10:2
REPAIR
59.7amm
00.55
99
31
klarm
06 066
58
99 52
56. km
Owis
91.66
69.75
44.6.
50.37
17,00 mm
26:48.1
2611
Table 2: Average test accuracy and standard deviation (three runs) on experiments with the MNIST variants under 1% bias ratio and {0%, 5%, 10%) noisy labels. The best accuracy is reported in bolded for the minority case.
De-biasing algorithms correctly classified the major case, similar to the vanilla case. However, for the minor case, these al algorithms obtained higher accuracy values than that of the vanilla results (at most 31. 18% t). Despite these de-biasing performances, they have a higher gap between major and minor cases (at least 36.69% under a 99.5%% bias ratio). In contrast, the proposed method obtained the highest accuracy value for the minor case (at most 69.92% 1) compared to the vanilla results. These results were obtained from the hyperparameter tuning with biased validation set. In appendex D, we report the oracle results obtained with the optimal hyperparameters, which were tuned by using the majority-minorii split validation set.
Controlled benchmarks with noisy labels. In Table 2, almost all algorithms failed to improve the minority accuracy, when labels are flipped, compared to the 0% noisy label case. For example, the proposed model's average performance declined from 98. 50% to 84.05%, and the majority performance declined from 99.24% to 80.60% for the CM case with 5% noisy labels. However, these performance decrements were recovered from the de-noising module for all cases. For the 5% noise CM case, all cases obtained close to 100% accuracy for the majority case and improved the performance for the minority case. Among all methods, the proposed pradient-bbsed method obtained 92.94% on average of the majority and minority test sets for the 5% noise of CM case, and improved by 7.83% in performance from the state-of-the-r checked from the 0% noise case that the adverse effect of the de-noising module on de-biasing is -3.9% (99.50% + 95.60%) on CM case for the proposed method.
algorithm, LfF at 85. 1 %. Moreover,
Real-world benchmarks. In Table 3, we report that real-world benchmarks can be biased by observing the lousy performance of the vanilla algorithm for all cases. The BAR case indicates that the proposed method gained the best performance on average among classes (a 5.06% increase from the vanilla algorithm and 2.53% increase from the second- besond-bestaaa accuracy). As listed in other columns, the proposed method has helped improve vanilla by 3.64%
Benchmarks
BAR
2N/ 1N-A
Celeba
Vanilla LFF
58 58.08g
34.55g
82.36g
57 on
36
83
REPAIR
60.61
35.462 nm
83.01
Ours
63.14 mm
37.42=
86.01=ae
Table 3: Test accuracy of real-world benchmarks averaged under three runs.
The best one is reported in bold font. cases, respectively. It also outperforms other methods by up to 1.11% and 3.00%, respectively. A detailed analysis is provided in Appendix F.
and 2.87% for the CelebA and ImageNet/ImageNeer
Analysis. We report the detailed various studies in Appendix H.
7 CONCLUSION
We presented a novel de-biased method by generating a mini-batch so that the trained model does not have prejudice the inference. The proposed sampling metric is based on the gradient of each sample, which is mainly correlated with its familiarity. Through extensive experiments using various biased datasets, we demonstrated the effectiveness of the proposed method in several cases. Furthermore, we proposed de-noising module that is easily applicable to all de- algorithms and protects from the side effects of noisy labels. We hope this study proposed exploration to understand the feature bias problem, especially for resampling-bar based approaches and tradient-based data valuation areas of machine learning.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Mohsan Alvi, Andrew Zisserman, and Christoffer Nellaker. Turning a blind eye: Explicit removal of biases and variation from deep neural network embeddings. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 0-0, 2018.
Jing An, Lexing Ying, and Yuhua Zhu. Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients. In International Conference on Learning Representations, 2020.
Hyojin Bahng. Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In International Conference Machine Learning, pp. 528-539. PMLR, 2020.
Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. Clustering on the unit hypersphere using mises-fisher distributions. Journal of Machine Learning Research, 6(9), 2005.
Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. In Advances in Neural Information Processing Systems, pp. 839-850, 2019.
Kaidi Cao, Yining Chen, Junwei Lu, Nikos Arechiga, Adrien Gaidon, and Tengyu Ma. Heteroskedas- tic and imbalanced deep learning with adaptive regularization. In International Conference Learning Representations, 2020.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.P-JCNIIP), pp. 4069-4082, 2019.
Luke Darlow, Stanislaw Jastrzębski, and Amos Storkey. Latent adversarial debiasing: Mitigating collider bias in deep neural networks. arXiv preprint arXiv:2011.11486 2020.
Deng, Dong, R. Socher, L.-J. Li, Li, and I.. Fei-Fei. ImageNet: Large-Scale Hierarchical
Image Database. In CVPR09, 2009.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis. Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2018.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- 2014.
tion processing systems. pp.
Ankit Goyal, Kaiyu Yang. Dawei Yang, and Jia Deng. Rel3d: A minimally contrastive benchmark for grounding spatial relations in 3d. Advances in Neural Information Processing Systems, 33, 2020.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Blevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6904-6913, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv: 000000007120 174, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference Machine Learning-Volle Volume 37, pp. 448 456, 2015.
10

---

Under review as a conference paper at ICLR 2022
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not learn: Training deep neural networks with biased data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9012-9020, 2019.
Eungyeup Kim, Jihyeon Lee, and Jaegul Choo. Biaswap: Removing dataset bias with bias-tailored swapping augmentation. arXiv preprint arXiv:2108. 10008, 2021a.
Tachyeon Kim, Jongwoo Ko, Sangwook Cho, Jinhwan Choi, and Se-Young Yun. Fine samples for learning with noisy labels. arXiv preprint arXiv:2102. 2021b.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. Adversarial filters of dataset biases. In International Conference on Machine Learning, pp. 1078-1083 PMLR, 2020.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecu www.nn/wenn.ccm wwt.exxb/////isstt 2, 2010.
Cheolhyung Lee, Kyunghyun Cho, and Wanmo Kang. Directional analysis of stochastic gradient descent via von mises-fisher distributions in deep learning. In Thirty-second Conference on Neural Information Processing Systems. Neural Information Processing Systems (NIPS) Foundation, 2018.
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. In International Conference Learning Representations, 2019.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6391-6401, 2018a.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In International Conference on Learning Representations, 2019.
Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9572-9581, 2019.
Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representa- tion bias. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 513-528, 2018b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Daniel McDuff, Shuang Ma, Yale Song, and Ashish Kapoor. Characterizing bias in classifiers using generative models. In Advances in Neural Information Processing Systems, pp. 5404-5415, 2019.
John Moody and Stephen Hanson. A simple weight decay can improve generalization.
Jun Hyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. In 34th Conference on Neural Information Processing Systems (NeurIPS) 2020. Neural Information Processing Systems, 2020.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26:11966 196-12044 2013.
Vikram V Ramaswamy, Sunnie SY Kim, and Olga Russakovsky. Fair attribute classification through latent space biasing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9301-9310, 2021.

---

Under review as a conference paper at ICLR 2022
Amélie Royer, Konstantinos Bousmalis, Stephan Gouws, Fred Bertsch, Inbar Mosseri, Forrester Cole, and Kevin Murphy. Xgan: Unsupervised image-to-ing saageetttimagg -magg translation for many-to-man to-mmny mappings. In Domain Adaptation for Visual Understanding, pp. 33 33-49. Springer, 2020.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-628860 626, 2017.
Robik Shrestha, Kushal Kafle, and Christopher Kanan. An investigation of critical issues in bias mitigation techniques. arXiv preprint urXiv:2104.000 2021.
Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jae Lee, Matt Feiszli, and Deepti Ghadiyaram. Don't judge object by its context: Learning to overcome contextual bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11070-11078, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()):1999-1988, 2014.
Enzo Tartaglione, Carlo Alberto Barbano, and Marco Grangetto. End: Entangling and disentangling deep representations for bias correction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13508-13517, 2021.
Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton van den Hengel. Evading the simplicity bias: Training a diverse set of models discovers solutions with superior ood generalization. arXiv preprint arXiv:2 10f.::2055.06122.. 2021.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pp. 1521-1528. IEEE, 2011.
Haohan Wang, Zexue He, Zachary C Lipton, and Eric P Xing. Learning robust representations by projecting superficial statistics out. In International Conference Learning Representations, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Bishion-mnist a novel image dataset for benchmarking machine learning algorithms, 2017.
Kun and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 7017-7025, 2019.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning, pp. 7164-7173. PMLR, 2019.
Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018.
12

---

Under review as a conference paper at ICLR 2022
A
EXPERIMENT DETAILS
A.1
BASIC SETTINGS
Simple ConvNet -1
Simple ConvNet -2
4 4 xi Conv
4x4 x 32 Conv
x x 64 Conv
6x6x Conv
6 x 16 Conv
6x6x 16 Conv
5 5 16 Conv
5 5 x 32 Cone
x 4 32 Conv
pool
2 xx2 pool
Avg pool
2 x2 pool
xx2 pool
2 x2 pool
pool
Avg
2
(a)
Simple ConvNet-1
(b) Simple ConvNet-2
Figure 9: Simple convolutional networks for the variants of the MNIST task and Cartoon data.
Network architecture. We use three types of convolutional neural networks (CNNs) (see Figure for two of them and the other is ResNet- 18 (He et al., 2016)) for all benchmarks, For MNIST variants, three convolution layers with filter size 4 and average pooling with size 2 are used (see Simple ConvNet-1), For Cartoon, 5 convolution layers with filter size [6, 6, 5, 5, 5.44 for each layer with average pooling size are used (see S Simple ConvNet-2) The real-world benchmarks tested under pre-trained ResNet- 18 provided by Pytorch torchvision without freezing feature extractor. We apply batch normalization (loffe & Szegedy, 2015), dropout (Srivastava et al., 2014) for regularization, and utilize SGD optimizer with momentum and weight decay (Moody & Hanson).
Partsettes
Coloeed MNIST
Wateomakedd MINIST
Canoon
Bissed Axtion Recogaition
CoRbA
Netwosk Type Searning rate decay
Simple ConvMet-11
Sisple Conviee-1
Eirgle Cooviett22
Pretriindd Rechve
Premaied Reshee-11
Pietzined Reshee-15 00001
(0.00.0......)))
(020466003))
[0440..0055)
[0.00.5]]
(02.0.........
Momentum
0.9
0.1
0.5
01.
Neight dacay
BU033
0,001
0,003
Se.4
0.1
0.3
Batch side
256
255
256
64
64
64
Epoche
008 paranotez
Table 4: Hyperparameters for each task
Hyperparameters. The hyperparameters that we used are summarized Table 4. We share basic hy- perparameters (including Network Type, Learning rate, Learning rate decay, Learning rate decay epoch, Nomentum, Weight decay, and Batch size and Number of Epoch) for all algorithms. Precisely, for the proposed method, set GCE parameter y and Dimension size K for our method, as in Table 4.
Validation. We optimize hyperparameters using a validation set of 10% of the training set and unseen at the training phase. Reported our results are obtained from the hyperparameters which shows a small loss for the validation set.
Data augmentation. Image data augmentation operations have been used frequently to improve gen- cralization performance. We utilize three operations: RandomRes zedCrop with scale (0.9, 1.1), ColorJitter with 0.05 hue and 0.05 saturation, and RandomRot ioon with (-10, 10) degree using NEAREST filling criterion. These augmentation operations are from trans forms module in torchvision.
A.2 TEST SETTINGS SECTION 3, 4, AND
Ratio minority samples analysis. analyze the correlation between [m]/[D] and accuracy statistics, we generated a bias training set based on CM data generation in Appendix B.1. This tests are conducted on the experiment with difficulty 0.0001.
Gradient property test To test our intuitions about gradients, we utilize part CM data (includes class 0 and 1) with bias ratio of 99.5% and difficulty 0.0001. Finally, we draw red arrows using the
13

---

Under review as a conference paper at ICLR 2022
lowest point and center point. This center point is the convergence point, and other points are random permuted loss values. All loss landscape figures are plotted by following (Li et al., 2018a).
B BENCHMARKS
B. CONTROLLED BIASED BENCHMARKS
MNIST Variants. MNIST variants are modified data from gray-scaled hand-written digit MNIST images (LeCun et al.. 2010). To inject biased attributes into the MNIST, we use two different attributes: color and object.
Colored MNIST (CM) has been frequently used as benchmark due to their simplicity (Kim et al., 2019; Bahng et al., 2020; Nam et al., 2020). The target of this benchmark is classifying the target attributes (shape of digits) not the biased attributes (color of digits). To make biased training set, all images are colored after up-scaling dimension from gray IR 28228 to 3D) RGB IR 28x28x3, Uniformly sampled 3-dimensional 10 colors C. R.. Ge: Bel²_ allocated to each class c. The samples in class of set M are colored by their allocated color Co. The other samples are colored with the color vector Ce'ste where e' A c is uniformly sampled. Furthermore, to deviate correlation between the target and biased attributes, color for each sample (z), y/) is deviated using 3-dimensional Gaussian distribution C, N NCC,, aT). This deviation is controlled by the difficulty parameter E (0.0001, 0.0005).
Watermarked MNIST (WM) has a different type of bias. The target of this benchmark is the same with generic MNIST, classifying hand-written digits, while the unintended object (Fashion MNIST (Xiao et al., 2017) data) is located biased attributes. To locate Fashion MNIST, all images from MNIST are size-up scaled from R 28 to 556556 Set M and m includes samples (I,, y/) where image 2; is made with digit image Trarget and fashion object Tbias whose label indices follow y: = Marget = Ybias and yi = Yurget * Yhias respectively. To make deviation, we put images from 5 pixels far from top and bottom of the canvas. 5 is sampled from uniform distribution 8 ~ U( a). We test two cases, a E {8, 16).
Cartoon. Cartoon data composed of synthetically generated face image samples which have 18 attributes (e.g.. hair color, face color) with multiple labels (e.g., hair r color = {1...., 10)). Among 18 attributes, larget attributes and the biased attributes. For simplicity, the size of each label is reduced from 9 to 4 and 10 4 by mapping 2 classes (similar colors) into 1 class, (e.g., original classes 1, are mapped to the new class 1). We make set M whose new target class and biased attributes following Orarget Obiss- Also, set m are sampled among images following Caarget o Obias.
choose hair color and face color attributes as the
B.2 REAL-WORLD BIASED BENCHMARKS
Biased Action Recognition. Biased action recognition data has 1,941 motion images for training with 6-motions [cl imming, Diving, Fishing, Racing, Throwing, Vaulting on various types of backgrounds (e.g., (Climbing, Rockwall)), The images in this benchmark include the frequent background of that action as a biased attributes for set M and rare backgrounds for set (e.g., (Climbing, Rockwall) for M and (Climbine iig Ice cliff) for m). This benchmark is originated from (Bahng et al., 2020; Nam et al., 2020; Kim et al., 2021a).
CelebA. CelebA has 202, 599 face images with 40 binary attributes from 10, 177 identities. Among 40 attributes, we select two attributes (HeavyMa up. e) for target and biased attributes. The size of training subsets M and m are 130, 410, and 32, 360, respectively. Remains are used for evaluation. Splitting training and evaluation sets follow (Liu al., 2015). This task reflects that real-world benchmark is easy to be biased in some specific attributes, unintendedly. This benchmark originated from (Nam et al., 2020).
TougeeNe//ImagNNee-Aa ImageNet data has 1, 000 classes, and we utilize a subset of ImageNet, which has 9 super-classes for scalability (Bahng et al., 2020). Also, evaluate using ImageNet- (Hendrycks et al., 2019), which gathers the failure cases of ImageNet from web. As described in (Bahng al., 2020; Hendrycks et al., 2019) dataset, "frequent background can be biased
14

---

Under review as a conference paper at ICLR 2022
attributes." For example, frogs in the swamp have higher proportion than other backgrounds (e.g., underwater). It indirectly represents that a de-biased model trained under ImageNet has better de-biasing performance when it has higher accuracy on ImageNet-A samples.
B.3 NOISY LABELS
We inject 10% noise labels into the basic configuration conducted in Section 5. Noise labels are uniform randomly flipped by following rules. At first, we randomly sampled a set of noise label candidates regardless of whether the source of the data samples are majority set or minority set. Flip the labely; + Oz uniform randomly, so that the label is not equal to the ground truth di * yi. Three noise label ratios, (90%, 95%, 1%0%]. were used. In order train noised model fa.. we utilize cross-entropy loss (not generalized cross-entropy loss). Also, to avoid over-fittings we utilize three data augmentation operations: RandomResizeddCoop. ColorJitter, and Random Rotation which are parameterized by (0.9, 1.1), (0.05, 0.05) and (-10, 10), respectively.
C BASELINES
Vanilla As an ultimate baseline, which means without de-biasing module, we train a network called vanilla. As summarized before, we utilize frequently used regularization techniques and simple data augmentation operations (jitter, crop, and rotation).
LearnedMMixiH (Clark et al., 2019) LeamneMMixiH was proposed by Clark et al for VQA task. This method of de-biasing is composed of biased (on the question) and unbiased (on both the image and the question) networks. We generate a biased network in our experiment by simply training on the biased data set. This is based on the argument LIF (Nam et al., 2020), which states that biased models learn biased information first, because it is easier to learn than the target attribute. The biased model outputs two values: its softmax output and bias prediction value for each given sample. Ultimately, the de-biased model trained under the cross-entie cross-entropy loss with regularizer of two above values from the biased model. We implement this algorithm by inheriting the authors' official code and unofficial code made by the authors (Bahng et al., 2020).
RUBi (Cadene et al., 2019) Similar with LearnedMixinH, RUBi also targets to solve VQA tasks. We also utilize the argument from LfF (Nam et al., 2020), to train the biased network. Instead using two values of the biased model, the authors of this paper utilized sigmoid outputs of the biased model and give weights to the de-biased model's output, when calculating the cross-entropy loss. This weighting mechanism gives weighted punishements by followings. The biased model outputs highly and lower biased sigmoid output when ma major and minor samples are given. By multiplying sigmoid values the softmax output of the de-biased model, the de-biased model suffers higher loss from the mi nor samples. We implement this method using the authors official code and unofficial code from (Bahng et al., 2020).
ReBias (Bahng et al., 2020) ReBias is a bi-level optimization-ban based de-biasing algorithm proposed by Bahng et al. ReBias trains the biased and de-biased model simultancously. The de-biased model updated its cross-entropy loss and additional regularizer using Hilbert-Schmid Independence Criterion (HSIC). Here, the biased model is composed of large convolutional filters to capture the bias information better, but we ignore this for a fair comparison with same network. We utilize official code from the authors.
LfF (Nam et al., 2020) The main module of this algorithm is two-fold: (i) The biased model trained on Generalized cross-entree entropy (GCE), which highlights the ma jor samples. (ii) The de-biased model trained on the weighted loss where weight computed by the cross-entropy outputs from each. From GCE, the cross entropy loss loss of the mi nor samples increases. We utilize official code from authors.
AFLite (Le Bras et al., 2020) AFLite is another type of resampling approach. AFLite trains the biased model multiple times and selects samples whose accuracy is low. At the de- de-biasing phase, the de-biased model is trainend on the sub-sampled new training set. Because authors offer the official code for synthetic data, manually reproduce their idea.
15

---

Under review as a conference paper at ICLR 2022
REPAIR (Li & Vasconcelos, 2019) REPAIR is re-sampling-basee based de- de-biasing algorithm proposed by Li et al. REPAIR reconfigure the training set based on the weight parameter w obtained by the adversarial training. As first step. the biased model and the adversary are trained simultaneously. The adversary computes per sample weight w so that the biased model gets a larger value for the minorities. REPAIR utilized w to reconstruct the repaired training set. The authors proposed four types of re-configuring methods with given parameter k: REPAIR-T (Thresholding, sampling above k), REPAIR-R (Ranking, Sampling top-k%). REPAIR-PR (Per-class Ranking), REPAIR-S (Sampling, reject with probability w). We utilize REPAIR-S among these variants to fairly compare our sampling mechanism (without hyperparameter case). We implement REPAIR using the official code from the authors. Additionally, note that we obtained w by training the classifier at the adversarial training phase without using biased information, such as RGB values in the case of CM, where as the official code used the biased information. This is enabled by the LfF (Nam et al., 2020) argument, which state that the network is biased toward a biased attribute that is easier to learn than the target.
D RESULT: CONTROLLED BIASED BENCHMARKS (ORACLE VALIDATION)
These results are obtained from the tuned hyperparameters, which are optimized by using the labels related to the biased attributes b. As we can see, the proposed method obtains the best performance in terms of the average accuracy of majority and minority cases. Moreover, other works also successfully improves their results. However, it is difficult to obtain a validation set which is discriminated between majority and minority samples. Therefore, we can conclude that the proposed method obtains when we can access to the discriminated dataset between majority and minority cases, which is difficult in practice.
milo
59%
52%
type
Hoijox
Kiaor
foz
Mlace
Mo.joz
Minor
Mo.joz
Milnor
Difficulty: 0,0001
Vanilla
09.077amx
56.09
89.87.mm
LesmeeMiixinl (Clazk al.. 2019)
36.27.123
10000:
96.77149
(Cadene 2019)
99.97
31.65:112
57.23:
77.77.00
99.57:
al., 2020))
99:63
16.00-102
97.43
92.23-134
98.03- 021
REPAIR Vasconcetos, 2019)
99.774x17
64.97.00
72:12-40
99.1-
-7.74-0
99.77 141
99.06
97.22
99.19
98,57
99.36 Km3
Difticult:
99.97
55
73,1622e
89, 07.522.7)
Lourn-MMixiaH (Cluk 2019)
99.93
26.42-12
10000)
60.44
96.774
74.47-114
96.58
(Cadene 2019)
99.00: ss
57.21
77.78:
99.57
al., 2020)
001
07.46,
05.02
92.22
08.001
20
REPAIR Vasconcclos, 2019)
09.77
09.53. am
am
57.81-
Am
97.23 am
98.99, w
98.21
98.64
58 ouz
99.06 w.1
Table 5: Average test accuracy and standard deviation (3 independent runs) on experiments with Colored MNIST data under various bias ratio. The best accuracy on average of Ma jor and Minor test is reported in bold font.
type
fox
foz
Difficulry:
Vanilla,
98 (9:99)
47.52.1.9
[5
53. 03d7
99. 10.99..2
66, 77277
99.55
79. 34:.5:::
LenmedMixinll (Clark 2019)
99.48
69.
99.41
74, 74.50
99.77 09.712x0
84,02
00.51:21
91. nv
RUB: (Cadene 2019)
99.48
99.00.
56.93
64.31
99.40
LIF (Num al. 2020)
99.221
49.05
98.47
56.13
98.61
67.21
56.43
REPAIR Vascuncelos, 2019)
98.96
70.75.
77.02
99
83.94.122
99.00
91,30
k3.10-1.
99.45
91.09
98.57
93.700..0
Difficulty: 16
Vasilla.
99.09
48.18.
54.15
99.
65.61.
99.61
LeamedMixinli (Clar 2019)
99.41
69.80
73,13
99.77
$5.17.12
99.65:2
92.
RUB: (Cadene 2019) (Nuss
99.15
$4.04-
09 39.35..00
65.99
98.37
10,401-12
9844.
98 48.45..0
K5.
REPAIR Vascuncelns, 20
00.00....
16.76.14
64.20
14.07=11
98.86;
98.46:041
99.26
94.25
Table 6: Average test accuracy and standard deviation (3 independent runs) on experiments with Watermarked MNIST data under various bias ratio. The best accuracy on average of Major and Minor test is reported in bold font.
16

---

Under review as a conference paper at ICLR 2022
Bias mioo
96.556
99%
96%
9896
Test
Me joz
Kinoo
joz
Kiince
Me joz
joz
Minoor
Vanilla
10000:
33,82.25
99.97 sxz
24.55
99. u.v
72.45
99.84 x21
LeurnedMixiii (Cluke 2019)
19010s
53,82
99.96,
48, o0
99.92.
70.27=
59.56
55, 15
RUBi (Cadene 2019)
100L0)
22.51
99.99%
46, 7o
99.87
68.56
99.91
57,01
1/F (Nim al. 2010)
90 00.56G.75
5688511
81.77a
76 97
78.84;
77.25
80.0 80,02, 85.93
REPAIR (Li Vasconeclos, 2019)
99.55s
60, 60.36
59 99.05s
60.0
99 99.96s
88.74
50 59.90s30
94. 14.08
Ours
97.98
92.53:
98.33:
93.78
96.95.
92.57.1m
9841.
95.24:00
Table Average test accuracy and standard deviation (3 independent runs) on experiments with Cartoon data under various bias ratio. The best accuracy on average of Major and Minor test is reported in bold font.
E SPLITTING PERFORMANCE OF DE-NOISING MODULE
Discriminativeness power of the de-noising module between majority and minority cases on various noisy label ratio and bias ratio |m|/DD|. As compensated loss criterion can distinguish the minority samples and noisy labels. In addition, when there no noisy cases, (0% cases) the proposed de-noising module lose some valuable clean minority samples. All cases are run with difficulty parameter 0.0001 and 8 for CM and WM cases, respectively.
for all cases, our de de-noising module with
Imv/CCI 10%, noisy
noisy label
ImM noisy
Locs (Nomalized)
Loss (Nomalized)
Loos (Nommlizee)
Clean,
Noisn.
1.0
1.0
0.5-
0.5
0-
0.5 Entropy malizal
1.0
0.5
1.0
0.5 Entropy (Normalized)
1.0
Entropy (Normalizedy
(a) Noise 10%, Bias 90%
(b) Noise 10%, Bias 95%
(c) Noise 10%, Bias 99%
ImwiDI 10%, noisy label ratio
ImMiDI 0.5%, noley label ratio 0%
Loos (Normalized)
(Nommalized)
Clean, M Clean,
Noise,
Clean, Clean,
Noise. M Nolee,
1.0-
Noles,
1.0-
0.5
1.0
0.5 Entropy (Normalized)
1.0
Entropy (Normalized)
(d) Noise 1%, Bias 90%
Noise 0%, Bias 99.5%
Figure 10: De-noising module and threshold plots for various conditions. Noise= (0%, 1%, 10%} and Bias=(99.5%, 1%, 5%, 10%).
Data= CM,
F REAL-WORLD BENCHMARK: DETAIL ACCURACY
As in all cases, the proposed method obtains the best performance on average of all cases, e.g., CelebA on average of majority and minority cases. Also, our BAR results fails to obtain state-of-the-art performance in some classes, but get the best average of all classes.
Benchmark
CelebA
Type
Ma jor
Minor
Avg-
Vanilla
93.16
82.36 71
L.F
92.60
69.
81.13
REPAIR
92.54.mm
83.01.00 13
Our
88.18
83.83 124
86.01 oa
Table 8: Test accuracy real-world benchmarks averaged under three runs. The best accuracy reported in bold font.
17

---

Under review as a conference paper at ICLR 2022
ImV/CCI
10%, nolsy label ratio 10%
ImiiiD 5%, noley label ratio 10%
ImMDI 1%, naisy label ratio 10%
Lozs (Normallzzd))
(Normalized)
(Normalized)
Clean. M
Noise,
Clean.
Noise. M
Clean. M
Noise. M Noise,
1.0
Clean,
Nolse,
1.0
Cloan,
Noise,
1.0
Clean,
0.5
0.5
0.5
0.5 Entropy (Normalized)
1.0
1.0
Entropy (Normalized)
Entropy (Normalize)
(a) Noisc 10%, Bias 90%
(b) Noise 10%, Bias 95%
(c) Noise 10%, Bias 99%
InviiDi 10%, noiey label ratio =
ImM 0.5%, noley label ratio 0%
Loos (Normalizd)
Loos (Nomalized)
Clean, M Cleen, m
Noise,
Clean, Clean,
Noise, M Noise, m
1.0
Nolee,
0.5-
0.5
0.5 Entropy (Normalized)
0.5 Entropy (Normalized)
1.0
(d) Noise 1%, Bias 90%
(c) Noise 0%, Bias 99.5%
Figure 11: De-noising module and threshold plots for various conditions. Noise=(0% 1%, 10%} and Bias=(99.5%, 1%, 5%, 10%].
Data= WM,
Benchmark
IN/N-A
Type
Aor.
Vanilla
34.55
LIF
36.31 oo1
REPAIR
35.46= uos
Ours
37.42 was
Table Test accu- racy of real-world bench- marks averaged under three runs. The best accu- racy is reported in bold font.
Benchmark
Biased action recognition
Type
Climbing
Diviny
Fishing
Racing
Throwing
VaulLing
Avg-
Vanilla
67.62 4m
29.98
66.67 19s
80.30-211 210
35.29 1.0
69.72
58.08+ sm
LIF REPAIR
68.57=15
61.11
78.28=29
32.94 188
70.48=13 1.7
57.35 ore
38.36=
70.63+
80.81=a4
37.65+
71.25
60.61
Ours
71.43 am
45.
72.22 1me
79.80
42.35 wm
68.70 330
63.14
Table 10: Test accuracy of real-world benchmarks averaged under three runs. The best accuracy is reported in bold font each column.
G VON MISES-FISHED DISTRIBUTION FOR GRADIENTS
The von Mises-Fisher (vMF) distribution is probability distribution on the hypersphere. S C IR.. The probability density function of the vMF distribution for the unit vector is given by:
fa(e; k) = C v)
where x and u are called concentration parameter and mean direction, respectively. The normalization -/2--1
constant Ca(ss) is equal to Ca(k) of the first kind at order t.
(2m)p/31, 1,/2 1(6) where I denotes the modified Bessel function
As in (Banerjee al., 2005), when N samples from the vMF distribution given as U; the maximum likelihood estimates solutions á and & can be obtained from à
N us and A s F(d-F") where
ME v.ll
>> mall
18

---

Under review as a conference paper at ICLR 2022
To interpret gradient vectors as vMF distribution, given vectors have be unit vectors: V8, Therefore, the approximated vMF distribution of the gradients be obtained as follows:
78.
||00,1|
Σ
VO
r(d 7²
12 ii=1 VO.
fa(VO;; ú, R) where ú
and
V0.
(2-
N
H ANALYSIS
We conducted a few studies to analyze our method: hyperparameter sensitivity, (targe,,bias)-paivii accuracy, pairwise sampling probability, ablation study on X, adjusting objective results based on metric, class activation map, unbiased case result, training time, results on naive metrics, and impact of GCE. Controlled benchmarks were used for analysis.
Varilla
Miän
03-
LF
06-
Repai
85
AFLiN
(a) Validation
(b) Majority
(c) Minority
Figure 12: Hyperparameter sensitivity. We train models on CM with diffeulty 0.0001 under learning rate 1/ E {0.005, 0.01, 0.02) and epoch e E {50, 100].
Hyperparameter sensitivity. In Figure 12, accuracy results of the models trained on the various learning rate 1) E {0.005, 0.01, 0.02] and epochs E {50, 100} were depicted. It shows that the prediction performance of all algorithms has higher accuracies on biased validation sets and majority set for various learning rates and epochs. However, in minority case 12(c), ours have less sensitive result on minority cases by seeing error bar's length.
Augment 63.0.
REPAIR
Labbl Indlox
Label Index
Label Index
1000 POD
CA 0000000 0110
8.7 53 08 CAll 046.
Label
111
0380 0101
01 07.0000 0- 1-100000 012
Color Index
Color Index
Color Index
Color Index
(a) Vanilla
(b) LfF
(c) REPAIR
(d) Ours
Figure 13: Accuracy each (color, label)-pair. Red color corresponds to bad accuracy, while blue
color denotes higher accuracy.
(Target, Baisssp accuracy. In Figure 13, pairwise accuracy matrices were plotted using various de-biasing methods. This represents the prediction result of 10 Major (diagonal) and 90 Minor pairs. First, vanilla fails to infer Minor pairs, but it infers Major accurately. Similar to vanilla, other algorithms, including ours, achieve almost accurate results for the Major pairs. On the other hand, based on the perspective of minority, other algorithms fail to infer. Among the de- biasing methods, our method has been approximately 100% accurate for almost all the pairs.
Pairwise sampling probability. To understand the previous pairwise result, we checked the sam- pling probability of our method. In Figure 14, each subfigure indicates pairwise sampling probability 1/M(s) for all the metrics: (i) (I/) (i) Σ, (i)D and p. (i) 1/S(i). In Figure 14(a), the raw case is highly bonded between the shape and color (see diagonal). In par and pn(() cases, both are well mixed compared to without case. In Figure 14(d), p+(i) has the most uniformized sampling probability. Red and yellow indicators denote complementary effect of both scores.
1/D(i)
Ablation study on X. In the definition of the score S(1), we set the hyperparameter À to make balance between source scores (i) and (i)) In order confirm the effect of X, the experimental results
Acc.
974679741 9797797.00
10
100
Acc.
97.7796.99
97.167 95.64
Table 11: Ablation study on X
19

---

Under review as a conference paper at ICLR 2022
Label Index
Label Index
Label Index
P ( i
Label
Index
Color
Color Index
Index
(a) w/o, 99.5%
(b) par (1). 99.5%
(c) pD()), 99.5%
(d) pa(i), 99.5%
Figure 14: Pairwise sampling probability under colored MNIST with 0.0001 difficulty. Blue color indicates higher sampling probability.
reported from various values of À {0, 1, 10, 100} on CM with 99% bias without noise. As in Table 11, there is no remarkable per- formance difference depending on the X, could be observed that it robust to the hyperparameter A.
Adjusting objective vs Resampling, As ablation study, we clari- fied the power of our gradient-based metric by reporting the results of adjusting objective leveraged by ps (i). We conducted tests under two controlled biased benchmarks, colored MNIST and watermarked MNIST with difficulty 0.0001 and 8. (t),), normalized by the mean
Benchmark
W.
Type
Avg.
Avg.
Vanilla
64.56
73.26
L.F
74,13
REPAIR
82.37
84,86
value, ENPp (G)/N' used for the weights. Our gradient-based score
P_(0]
Ours
95.33
86.67
outperforms others in the adjusting objective (dagger mark and resam- pling cases. Compared with our gradient-based ways, the adjusting objective methods achieve lower performance. This implies that, argued in (An et al., 2020), the adjusting objective-based approach suffers from training sensitivity.
Ours
98.14
91.23
(a) Vanilla, . (b) LfF, M (c) Repair, M(d) Ours, M (c) Vanilla, LfF, m (g) Repair, mh) Ours,
Figure 15: Plots of class activation map (CAM) for each method of same inputs, sampled from the
Major (M) and Minor (m) sets.
What de-biased model learn.
The main goal of de- de-biasing algo- rithms is make the de-biased model not learn the biased feature. To check whether the biased models satisfy this goal, we evaluate two tests: (1) random color test for CM, and (2) CAM for WM.
Algorithm
Accuracy
Vanilla
50.80
LfF
85.04
ReBias REPAIR
51.49
We construct test dataset by uniformly randomly sample colors for all samples RGB (0, 1)3. : As in Table 12, the result of ours show that outputs based on the shape whether the training set biased or not.
53.28
Ours
95.44
Table 12: Random color test
Additionally, conduct CAM test for WM dataset. CAM (Sclvaraju et al., 2017) has been used as a tool to explain "where a model is
results on CM
focusing on?" In Figure 15, state the CAM results for watermarked MNIST samples obtained from the Major (Left) and Minor (Right) sets. In the top row (for Major), our model better ignores the biased object (fashion object) than the other methods, Even for Minor, only our model determines the target digit, while the rest focuses on the biased object ( (Ankle boots).
Unbiased / Rarely biased case. To analyze the gen-
Beadlannk
eral side effect of additional modules, performance degra- dation, we check the accuracy of unbiased (especially. 10% = 1/## of classes) biased case) and rarely biased (70%. 80%, 90%) cases, under MNIST variants with dif- ficulties of 0.0001. As listed in the table, all models have similar accuracy to the vanilla case. This means that,
Testope
A4
84
10
M
A2
Vault
07.41
47
99
LIF
1003
$9.33
$3.01
16.57
9111
98-11
19.21
90.15
KERAIR
09.73
08.72
95.81
08.17
1019
$3.08
06.20
99.33
Daaxs
08.01
Table 13: Unbiased Colored MNIST ac-
curacy results.
20

---

Under review as a conference paper at ICLR 2022
gardless of whether the training set is biased, all models, including ours, do not suffer from severe side effects. It shows that our algorithm can be applied without performance loss. In short, our algorithm needs not to utilize human knowledge about bias in the training set, an on/off knowledge.
Training Time. To see the drawbacks of training time for de-biasing, we measure the training time of related works on WM case. As in Table 14, the proposed model spends x x1.45 time compared to RUBi which is the fastest de- de-biasing method, while x0.72 faster than the other re-sampling algorithm, REPAIR.
Vanilla
RUBi
Mixin
12 26 13'
29 22
LIF REPAIR
Ours
27 15" 08"
38' 00"
Table 14: Training time
Other metries. This study shows other metrics-based resampling results, (e.g., softmax response, loss, and logit value), based on sampling probability. These results show empirical justification of our gradient-based scores for de-biasing. All scores are defined to make the minority samples (unfamiliar samples) have higher sampling prob- ability. Each score is defined as follows:
0.010
Samp ing probatilty
92
92.8311
50.90 s8.31
90.87
0.11
0.00-
0.036
0.034
0.002
Loge
1 (IJ W:))
17 [ Me)
C( vs)
Suuftmax (1) = E, - S(f(z,, (i)gi(() E, - (f(I). Stase (1)
Σ, ((zj, Uj)
where S(-) is softmax response and m m D (z y4)). As depicted in Figure H. our gradient- based approach has the highest performance on the average accuracy of majority and minority results. Moreover, the gradient method has short error bar than the others, which works reasonably (softmax and loss case).
Vanilla
RUBi
Impact of GCE. We checked the proposed method without GCE (trained with CE loss) under 99.5% WM case. The proposed method w/o GCE achieved (M/m) 99.15% / 88.72%, which outperforms the others, while lower than the proposed method without GCE case (99.35% / 83.10%).
98.84 / 94.78 98.847 94.66
REPAIR
Ours
98.84 / 95.56
98.55 96.30
Table 15: |m|/[D| [D 0.7
I DE-NOISING
In recent noisy label cases, there have been two major trends: (1) con- structing robust objective functions or regularizers (Cao et al., 2020; Yi & Wu, 2019), and (2) cleaning the noisy data (Li et al., 2019; Kim et al., 2021b). Those strategies are mostly used to reduce the influence of rare samples, such as out-of-cluster samples in (Kim et al., 2021a). However, the case of dataset bias problem, minority samples can be omitted when rare samples are discarded, resulting in de-biasing algorithms' performance degrading. As result, crasing noisy label samples while maintaining minority samples is crucial in the dataset bias problem with noisy labels to preserve the de-biasing performance. Table 16 shows the number of preserved samples after cleansing steps. We compare the proposed de de-noising module to the state-of of-th-ot- method, FINE. It cutting-out noisy labels by using eigenvectors. As shown in Table 16, while the proposed de-noising module does not perfectly eliminate noisy samples, it does preserve clean minority samples, whereas FINE does not.
Clean/Major
Clean/Minor
Noisy/Majer
Noisy/Minor 26
Betore denoising FINE Ours
50586
368
2820
47455
50577
331
Table 16:
Comparison between the state-of-thh-ar noisy label algorithm and the proposed de-noising module in terms of the number of preserved samples after cleansing.
21


Reviews:
[{"review_id": "tP9BoMfbIOh", "reviewer": "Reviewer_3HUA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method differs from prior work in the novel scoring function. However, the proposed scoring function is not well-motivated. Despite the success of better performances, it is hard to be convinced that the per-sample gradients are able to differentiate between the samples drawn from majority or minority group. The authors show only 4 examples in Color-MNIST in figure 4, which are not representative enough.\n\n**Strength**\n- Extensive experimental results across different setting and different baselines.\n- Strong performance compared to the baselines\n\n**Weakness**\n- Lack of motivation and analysis on the connections between per-sample gradients and the majority/minority splits (in more complex datasets)\n\n**Others**\n- For table 3, I would be interested to see the accuracy of the minority as well since it help validate if the proposed method works under real-world correlation.", "summary_of_the_paper": "The paper propose to use the per-sample gradients from the biased classifier to tackle the dataset bias. Following prior work, this work adopts a three-step approach: 1) train a classifier on biased dataset 2) identify which samples are from the minority and which are not 3) train a classifier with re-sampling. The proposed per-sample gradient-based scores are used in step 2. The motivation is purely from the empirical observation that samples from the majority group tend to have similar gradient magnitude and direction. From the observation, they derive a scoring function for re-sampling. The proposed method work pretty well empirically compared to various baselines.\n\nAside from the novel scoring function, since the noisy labeled data might be accidentally considered as minority group, the paper introduces a de-noising step to tackle the potential noise in the dataset. The de-noising step further improves the performances.", "summary_of_the_review": "The paper shows good empirical results on mitigating dataset bias by devising a novel scoring function for re-sampling. The scoring function is the key in this work, but is not well-motivated enough. The paper can gains a lot of values by providing more analysis on the connections between per-sample gradients and the majority/minority sets.\n\n\n---\n\nUpdated on Dec. 6. \nThough the paper still lacks of theoretic motivations, it does provide convincing empirical findings. I raised my score from 5 to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UWfJPtWYDPW", "reviewer": "Reviewer_BR78", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed idea of using gradient-based scoring for de-biasing seems novel and effective. I have the following concerns and questions. \n\nThis work is based on the hypothesis that the “gradients have remarkable differences between samples generating prejudice and the others”. By observing that the label noise can degrade the performance of the proposed gradient-based model, they added the de-noising module to alleviate this issue. However, I’m not convinced if the noisy label is the only critical factor that can negatively affect the performance when using the proposed gradient-based scheme. I wonder what other factors (e.g., adversarial attacks) can degrade the performance and how to tackle them. Can you add more analysis and discussion on this? \n\nFigure 4 is drawn using the Colored MNIST. This result may be useful to illustrate the proposed hypothesis, but it may not be enough to show the generality because the dataset has a very simple structure and variation. I wonder how the plots for more natural datasets would look like, and if any theoretical analysis can be included.  \n\nThe model assumes that the biased samples make up the majority of the training set. The experiments are also based on this assumption, and the proportions of the unbiased (minority) samples in the experiments are 5% or less in all the experiments. I wonder how realistic this scenario is and how the proposed method performs on a dataset that contains a higher portion of unbiased (minority) training data (e.g., 10%, 20%, 30%,…). \n\nIn Table 2, comparing the accuracies for the Minor group, which I think is more important in the performance comparison, the performance of the proposed model “with de-noising”, compared to the case without it, is improved at the 99%/10% setting, but not for the remaining cases of 99%/5% and 99%/0% (except for Watermarked MNIST, 99%/5%, showing very low accuracy). Therefore, this result is not sufficient to validate the proposed de-noising idea. More experiments and/or analyses should be provided. \n\nMinor comments:\n\n- Figure 3: |m|/|D| -> |M|/|D|\n- Page 6, Section 4.2: “From the harmonic-mean property has a higher value” (need to revise)\n- In Table 2 - 99%/5% - Without de-noising – Major M: \n-- Colored MNIST: LfF performance is written as 00.00. Should it be 100.00?\n-- Watermarked MNIST: the performance from “Ours” is 33.26, which is much lower than the other methods (over 90). Is it a typo or any reason for this?   \n- In Table 1, the method names of “Proposed” and “Ours” are both used. It’s better to make it consistent. \n- Page 9, Section 6.2\n-- ... an accuracy similar to that o the major case. -> … an accuracy similar to that of the major case.\n-- de-basing -> de-biasing\n", "summary_of_the_paper": "This paper proposes a gradient-based resampling scheme for de-biasing. The authors construct two types of scores that leverage gradients of the biased model: the magnitude and the direction of the gradients. In addition, to mitigate the side effects from the noisy labels, the authors also propose a de-noising module, which can be easily applied to any de-biasing algorithms. Extensive experimentation validates the proposed method against other state-of-the-art methods. ", "summary_of_the_review": "This paper present a novel idea for de-biasing and its performance is validated on limited scenarios. More validation and analysis of the used hypothesis and the model would strengthen the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AUWrVG0em5Y", "reviewer": "Reviewer_sFXQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n- This work studies an important problem in machine learning. The proposed method has the benefit of not requiring supervision of biased attributes, which makes it potentially scalable to large-scale datasets with bias.\n- Extensive experimental results are reported, where the proposed method showed superior performance to prior state-of-the-art on the both synthetic benchmarks and real-world datasets. It is particular interesting that all baseline methods are vulnerable to label noise and that the denoising module improves results universally.\n- The writing of the paper is easy to follow. Implementation details are presented with full transparency which contributes to the reproducibility of the work.\n\n## Cons / Questions\n- The key assumption which motivates the proposed method, namely that minority samples have different gradient distributions than majority ones, deserves a more rigorous validation. For example, in addition to the aggregate measure in Figure 4b, it would be more convincing to compare histograms of per-sample gradient norms $||\\nabla\\theta_i||$ for both groups of training data, to ensure that the difference between the two groups is not the consequence of a few outliers.\n- Despite its positive performance under synthetic settings, the significance of the proposed label denoising module is not validated on real-world datasets. In addition, the paper makes no reference to the literature on training with noisy labels (some recent work includes [A, B, C]). It is unclear how the proposed method compares to these approaches both gconceptually and in empirical results.\n- Experimental results on real-world datasets are relatively weaker compared to synthetic ones. For example, the reimplemented LfF model reports significantly lower accuracy on BAR dataset (57.35% vs. 62.98%). Can the authors comment on factors that may contribute to this discrepancy? For ImageNet experiments, it would be cool if similar improvements can be observed on ImageNet-C [D] and Stylized-ImageNet [E] test sets. Finally, among the six baseline debiasing methods used in this work, only LfF and REPAIR are compared against using real-world datasets. It would be helpful if explanations are provided on the choice of baselines for these tasks.\n- Results in Table 1 are presented in a slightly misleading way, as it seems to suggest that the proposed method achieves highest accuracy on both majority and minority groups. I believe the clarity can be improved by additional columns for the average of major and minor accuracies and making bold only the best numbers in each column.\n\n## Minor comments\n- Table 1 caption seems to have grammatical error: \"...bolded for the on average of the majority and minority tests\".\n- In table 3, column titles `CelebA` and `IN/IN-A` should be swapped.\n\n[A] Yu, Xingrui, et al. \"How does disagreement help generalization against label corruption?.\" *International Conference on Machine Learning*. PMLR, 2019.\n\n[B] Yi, Kun, and Jianxin Wu. \"Probabilistic end-to-end noise correction for learning with noisy labels.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2019.\n\n[C] Li, Junnan, Richard Socher, and Steven CH Hoi. \"Dividemix: Learning with noisy labels as semi-supervised learning.\" *arXiv preprint arXiv:2002.07394* (2020).\n\n[D] Hendrycks, Dan, and Thomas Dietterich. \"Benchmarking neural network robustness to common corruptions and perturbations.\" *arXiv preprint arXiv:1903.12261* (2019).\n\n[E] Geirhos, Robert, et al. \"ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.\" *arXiv preprint arXiv:1811.12231* (2018).", "summary_of_the_paper": "The paper proposes a de-biasing method for neural network without requiring explicit annotations of biases. The authors propose a resampling-based approach that automatically detects underrepresented samples based on the direction and magnitude of their loss gradients, then balances the dataset by rejection sampling. The paper further proposes a denoising module to identify and filter out noisy samples under the presence of label noise. The gradient-based de-biasing method is empirically shown to outperform prior work on both controlled synthetic and real-world datasets.", "summary_of_the_review": "The paper presents a novel and simple debiasing approach for neural networks, as well as a denoising module for training with label noise. The proposed debiasing method shows promising results especially on synthetic benchmarks, comparing against various state-of-the-art algorithms; however, I believe the concerns listed in the section above needs to be addressed to fully justify the contributions of the work. I am setting my initial score to borderline reject but would be happy to update it based on the reviewers' response.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6jKtRI7sqYp", "reviewer": "Reviewer_WQAv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There is a lot to like about this paper. I find the hypothesis about the \"per sample\" gradient can be used to identify \"biased\" samples interesting. The authors also correctly discuss the accompanying argument that this has close ties to label noise. The results also show a remarkable improvement in the commonly used datasets. However, I am increasingly wary of testing bias mitigation algorithms on the same toy settings, and their ability to actually quantify the degree to which these algorithms are actually working is under increasing suspicion. \n\nStrengths:\n\nS1.  Great results: The results for varying levels of \"bias\" as well as the success of the proposed \"denoising\" setup is remarkable for the datasets tested. The results are vastly superior to other comparable methods in three different datasets and even better for extreme levels of bias (99.5%). \n\nS2. Interesting Hypothesis and approach: The paper proposes an interesting hypothesis about the differences between gradient magnitude and direction (as measured by its proximity to an \"average\" gradient direction for all samples) look different for biased as compared to \"regular\" data sample. This discussion would be incomplete without a discussion of its relationship with label noise which the authors rightly address. The proposed denoising module is shown to mitigate the effects of label bias on various debiasing algorithms including the proposed one. While I am not fully convinced of the hypothesis is universally valid, and whether the denoising module is truly capable of dealing with label noise per se, the empirical validation, at least in the datasets tested, is hard to refute in the context of debiasing.\n\nWeaknesses:\n\n\nW1. Unclear how bias model is implemented: In prior works, such as LearnedMixinH and Rubi for VQA, the bias-only model takes question-only model as a biased model which assumes that if an answer could be guessed only using the question, then that sample can be assumed to be biased. There isn't a straightforward parallel for this in color-MNIST. For color-MNIST, REPAIR proposes to train a classifier using only the RGB value of the input. It is unclear how 1) the biased model is trained for the proposed algorithm, 2) How were they trained for Rubi?\n\nW2: Serious doubts about \"scalability\" and assumption about known bias:  There are various questions about the \"scalability\" of the proposed method. One common theme across these datasets is that they can be \"learned\" (at least the biased version) with a much smaller amount of data than is present in the training set. Hence, a rejection sampling-based method can work even when the minority set (m) diminishes. Next, the proposed algorithm falls under the category called variously as  \"supervised\", \"known\", or \"explicit\" bias mitigation algorithm. In other words, the algorithm assumes knowledge about which of the factors were biased so that a suitable \"bias-only\" model can be trained by leveraging only the \"bias\". Such methods can be unscalable to all but a single, well-defined type of bias. This could be addressed by demonstrating the algorithm's success in more difficult scenarios (https://arxiv.org/abs/2104.00170).", "summary_of_the_paper": "The paper proposes a new bias-mitigation technique. The main hypothesis of the paper is that there are differences in gradients for \"biased\" samples (or samples that are \"rare\") compared to majority patterns in the training data. Using this, the paper devises a rejection sampling method that tries to balance samples in a minibatch. However, a sample with a noisy label can appear to be a \"biased\" sample (with a correct label) which can affect the proposed method (as well as many other bias mitigation methods). To this end, the paper also proposes a denoising module that successfully eliminates the effects of noisy labels on the debiasing algorithm proposed.", "summary_of_the_review": "Overall, I think the paper shows lots of properties of a solid paper. It has an interesting hypothesis, a solid experimental section, and some valuable insights with great results. Taken together, I find that it clearly surpasses the bar for acceptance. However, it still leaves a few things to be desired, which prevents me from giving a higher score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript proposes a method to adjust a biased model without requiring explicit annotations of biases. The main hypothesis of the manuscript is that there are differences in the direction and magnitude of the loss gradients for underrepresented samples compared to majority patterns in the training data. Based on this hypothesis, the manuscript proposes a rejection sampling method that tries to balance samples in a minibatch. However, a sample with a noisy label can appear to be an underrepresented sample with a correct label which can affect the proposed method. To tackle this, the manuscript also proposes a denoising module that successfully eliminates the effects of noisy labels on the debiasing algorithm proposed. Experiments are performed on various synthetic and real-world biased sets. 

Positive aspects of the manuscript includes:
1. The results for varying levels of "bias" as well as the success of the proposed "denoising" setup is remarkable for the datasets tested;
2. An interesting hypothesis about the differences between gradient magnitude and direction (as measured by its proximity to an "average" gradient direction for all samples) look different for underrepresented samples as compared to "regular" data sample.

There are also several major concerns, including:
1. Lack of motivation and analysis on the connections between per-sample gradients and the majority/minority splits in more complex datasets;
2. The key assumption which motivates the proposed method, namely that minority samples have different gradient distributions than majority ones, deserves a more rigorous validation;
3. The "scalability" of the proposed method. One common theme across these datasets is that they can be "learned" (at least the biased version) with a much smaller amount of data than is present in the training set. Hence, a rejection sampling-based method can work even when the minority set diminishes; 
4. Assumption about known bias. The proposed method assumes knowledge about which of the factors were biased so that a suitable "bias-only" model can be trained by leveraging only the "bias". 

Post-rebuttal, reviewers stayed with borderline ratings, and they have suggested further improvements: more details about Biased MNIST numbers (to address concerns about known bias), and ablation studies on real datasets (e.g. compare to results without denoising, or denoised using FINE) to fully justify the practical importance of proposed denoising module.
Decision: Reject