Year: 2021

Abstract:
The performance of deep neural networks (DNNs) primarily depends on the configuration of the training set. Specifically, biased training sets can make the trained model have unintended prejudice, which causes severe errors in the inference. Although several studies have addressed biased training using human supervision, few studies have been conducted without human knowledge because biased information cannot be easily extracted without human involvement. This study proposes a simple method to remove prejudice from a biased model without additional information and reconstruct a balanced training set based on the biased training set. The novel training method consists of three steps: (1) training biased DNNs, (2) measuring the contribution to the prejudicial training and generating balanced data batches to prevent the prejudice, (3) training de-biased DNNs with the balanced data. We test the training method based on various synthetic and real-world biased sets and discuss how gradients can efficiently detect minority samples. The experiment demonstrates that the detection method based on the gradients helps erase prejudice, resulting in improved inference accuracy by up to 19.58% compared to the other state-of-the-art algorithm.

heading: 1 INTRODUCTION
text: Over the past decade, deep neural networks (DNNs) have been a focus of research owing to humanlike performances in various tasks, e.g., image classification (Krizhevsky et al., 2012), object detection (Girshick, 2015), and image generation (Goodfellow et al., 2014). Despite these impressive results, deploying DNNs directly in real-world problems remains a significant challenge due to the difficulties in obtaining well-curated training sets. Specifically, unintendedly biased information in training sets causes prejudice, resulting in wrong decisions at inference time (Torralba & Efros, 2011; Shrestha et al., 2021). For instance, most “ski” images contain “skier.” This unintended correlation can recommend a wrong shortcut for “ski” by examining the person.
Prior studies have employed human supervision (i.e., providing additional labels related to the bias) (Kim et al., 2019; McDuff et al., 2019; Singh et al., 2020) or giving a specific information about a biased domain (Lee et al., 2019; Geirhos et al., 2018), which is very expensive, to reduce the influence of a “skier” on “ski” class images. Recently, studies on replacing human labor with DNNs have been actively discussed (Li & Vasconcelos, 2019; Nam et al., 2020; Cadene et al., 2019; Bahng et al., 2020; Clark et al., 2019; Le Bras et al., 2020). Typically, two separate networks have been used: the biased network and the de-biased network. A biased model learns to replace human intervention and teaches a de-biased model using the prejudiced knowledge of the biased model.
Previous studies based on the two mentioned networks approaches have had two directions: adjusting objectives and resampling. Adjusting the objective refers to a method that obtains weighted loss or additional regularizers for each sample differently, where the weights or regularizers are obtained from the biased model (Nam et al., 2020; Cadene et al., 2019; Bahng et al., 2020; Clark et al., 2019). Methods that adjust the objective have been proposed frequently due to their several advantages, such as the implementational simplicity. However, according to (An et al., 2020), adjusting-objective approaches suffers instability with stochastic gradient descent (SGD) type optimizers owing to the high variance of learning weights per sample.
Alternatively, resampling methods could reconstruct balanced sets. Previous research (Le Bras et al., 2020; Li & Vasconcelos, 2019; Li et al., 2018b; Kim et al., 2021a) has focused on which samples
should be oversampled to build a balanced set. However, almost all prior works has focused on expensive human labor to tune hyperparameters related to the resampling densities of data points. For instance, it is difficult to set the amount of resampling of each sample without knowing how much biased the training set is.
The de-biasing methods, belonging to both adjusting-objective and resampling approaches are designed to emphasize samples that are hard for a model to learn. However, a few noisy labels (i.e., mislabeled samples) in the training set could significantly interfere with such de-biasing because it might be difficult to discriminate between two types of hard samples: noisy samples that the model should not learn and rare samples that have to learn. Although training sets have noisy labels in practice (Natarajan et al., 2013), to the best of our knowledge, no study considers the de-biasing problem under noisy label cases.
Contribution. In this study, we propose a score-based resampling scheme for de-biasing, which does not require hyperparameters related to training set reconstruction. Instead of hyperparameters, we construct two types of scores that leverage gradients of the biased model trained on the given biased set. The proposed scores determine the required proportions for each sample in training set reconstruction.
Before designing scores, we explain our hypothesis that gradients have remarkable differences between samples generating prejudice and the others , and check it empirically. The hypothesis is based on the following observations. First, the gradient magnitudes (especially the Euclidian norm) of the samples which contributes to the prejudice are relatively smaller than those of the others. Second, samples that make prejudice and the others do not have significantly different gradient directions, measured by the estimated likelihood of the von Mises-Fisher (vMF) distribution (Banerjee et al., 2005; Lee et al., 2018).
Based on these observations, we propose two types of scores and a resampling-based de-biasing method. The proposed method consists of three steps similar to previous resampling studies (Li & Vasconcelos, 2019; Le Bras et al., 2020; Kim et al., 2021a). The first step is to train a biased model based on a biased raw training set. Afterward, the scores are obtained by the biased model. The mini-batch sampler generates balanced mini-batches using the scores, and it feeds them into an ultimate model for de-biased training.
In addition, we investigate the side effects of de-biasing methods when incorrectly labeled samples are in the training set. We observe significant performance degradations of de-biasing methods with a small portion of noisy data. To alleviate the negative side effects of noisy data, we first de-noise the training set using loss values from the our observation that noisy label samples have higher losses than clean samples and then run de-noising algorithms. The de-noising step successfully protects the de-biasing performance.
Finally, we demonstrate the effectiveness of our method using various biased benchmarks (Colored MNIST (Nam et al., 2020; Kim et al., 2019; Li & Vasconcelos, 2019; Bahng et al., 2020), Watermarked MNIST, Cartoon (Royer et al., 2020), CelebA (Liu et al., 2015), Biased action recognition (Nam et al., 2020), and ImageNet / ImageNet-A (Bahng et al., 2020; Deng et al., 2009; Hendrycks et al., 2019)). In most experiments, the proposed method outperforms the current state-of-the-art methods. In particular, the proposed method improves the accuracy of the unbiased test by 68.00% from the baseline with a small performance degradation of the biased test of 0.91%. Moreover, we compare the proposed method with the previous resampling method, REPAIR (Li & Vasconcelos, 2019). The proposed method improves the average accuracy by 71.04%→ 98.14%.

heading: 2 RELATED WORK
text: Mitigating bias with human supervision. In (Goyal et al., 2017; 2020), generated a de-biased dataset using human labor. A collection of studies (Alvi et al., 2018; Kim et al., 2019; McDuff et al., 2019; Singh et al., 2020; Teney et al., 2021; Ramaswamy et al., 2021; Tartaglione et al., 2021; Geirhos et al., 2018; Wang et al., 2018; Lee et al., 2019) aimed to mitigate bias based on one of two types of human supervision: explicit bias labels and implicit bias information. Other studies (Alvi et al., 2018; Kim et al., 2019; McDuff et al., 2019; Singh et al., 2020), have used bias labels for each sample to reduce the influence of the bias labels when classifying target labels. (Tartaglione et al.,
2021) proposed the EnD regularizer, which entangles target correlated features and disentangles biased features. Several authors (Alvi et al., 2018; Kim et al., 2019; Teney et al., 2021) have designed DNNs as a shared feature extractor and multiple classifiers. In contrast to the shared feature extractor methods, (McDuff et al., 2019; Ramaswamy et al., 2021) constructed a classifier and conditional generative adversarial networks, generating test samples to check whether the classifier is biased. (Singh et al., 2020) proposed a new overlap loss defined by a class activation map (CAM). The overlap loss reduces the overlapping parts of the CAM outputs of two bias labels and target labels. Different approaches (Geirhos et al., 2018; Wang et al., 2018; Lee et al., 2019) have assumed that the characteristics of bias features are known and prevent learning the implicit bias features. To avoid the known texture bias of ImageNet (Deng et al., 2009), (Geirhos et al., 2018) generated stylized ImageNet, and (Lee et al., 2019; Wang et al., 2018) inserted filter in front of the models so that the influence of the backgrounds and colors of the images can be removed.
Mitigating bias without human supervision. To reduce human intervention, recent research (Clark et al., 2019; Cadene et al., 2019; Bahng et al., 2020; Nam et al., 2020; Li & Vasconcelos, 2019; Li et al., 2018b; Le Bras et al., 2020; Darlow et al., 2020) has used ensemble-based methods, using two separate networks with different purposes. The biased model learns biased information and even empowers it. Then, the biased model is used to train the de-biased model as a pathfinder, which provides information about the bias features.
The ensemble idea has been used as a key component in several studies, (Clark et al., 2019; Cadene et al., 2019; Bahng et al., 2020; Nam et al., 2020). (Cadene et al., 2019) proposed a method known as RUBi, which multiplies the sigmoid output of the biased model by the softmax output of the de-biased model. (Clark et al., 2019) summed the output of the two models to train the de-biased model with an entropy regularizer. (Bahng et al., 2020) proposed ReBias algorithm using the Hilbert-Schmidt independence criterion (HSIC) as an objective of the de-biased model, by independent of the biased model. Learning from failure (LfF) (Nam et al., 2020) used generalized cross-entropy-based training that emphasizing the usual samples but ignoring unusual samples.
In contrast, biased and de-biased models are trained sequentially (Li & Vasconcelos, 2019; Li et al., 2018b; Le Bras et al., 2020; Darlow et al., 2020). Researchers computed the weights to reconstruct the training set from the biased model and trained the de-biased network on the reconstructed training set. The REPAIR (Li & Vasconcelos, 2019) and RESOUND (Li et al., 2018b) methods proposed mutual-information-based weights to create a downsampled dataset. In addition, AFLite (Le Bras et al., 2020) proposed accuracy-based weights from multiple biased models, unlike LAD (Darlow et al., 2020) and BiaSwap (Kim et al., 2021a), which generated new images using an autoencoder based on the latent representation of the biased model.

heading: 3 BIAS PROBLEM FOR TRAINING A CLASSIFIER
text: 3.1 UNINTENDEDLY BIASED TRAINING
Suppose that a training set D is composed of images x, as illustrated in Figure 1. Each image can be described by a set of attributes {a1, ..., ak, ...}, (e.g., {“digit 0”, “digit 1”,... “red”, “green”, ... , “thick”,...}). The goal of the training classifier is to find a model fθ that correctly predicts the C target attributes “digits,” y = {at1 , ..., atC} ={“digit
0”, “digit 1”,...}. Remark that target attributes are also interpreted as class. However, we focus on the case in which another attributes exists, “colors,” where b = {ab1 , ..., abC} = {“red”,“green”, ...} 6= y, which is highly correlated to the target “digits” (i.e., H(at1 |ab1) ≈ 0). Such an correlated attributes and training set are denoted as the bias attributes and biased training set, respectively. In addition, the “color-digit” correlated and uncorrelated samples, in the top two rows and the bottom row in Figure 1, are called the majority and minority. We use the notations M and m for the majority and minority sets, respectively. For example, “red-digit 0” image is majority sample, while “orange-digit 0” one is minority sample.
When we train a model based on the biased training set D, the model may infer the bias attributes b, not target attributes y, which is problematic because the target is y, not b, and we refer to this as an
unintended bias problem. For example, if the model trained on the images in Figure 1 suffers from this problem, this model outputs 4 when orange,0 image is given.

heading: 3.2 WHEN DOES THIS PROBLEM HAPPEN AND INTENSIFY?
text: 1.0 0.9 0.7 0.8 0.9 0.9 0.4 0.5 0.5 0.8 0.9 1.0 0.8 1.0 0.9 0.9 0.9 1.0 0.8 0.8 0.9 0.2 1.0 0.2 1.0 0.9 0.8 0.3 0.7 0.9 0.9 1.0 0.8 1.0 0.8 0.2 1.0 0.4 0.5 0.2 1.0 0.8 1.0 0.8 1.0 0.8 0.6 0.8 0.9 0.1 1.0 0.4 0.9 0.3 0.9 1.0 0.4 0.9 0.4 0.5 0.6 1.0 0.8 1.0 0.6 0.6 1.0 1.0 0.7 0.7 0.8 0.6 0.1 0.5 0.7 0.9 0.9 1.0 1.0 0. 0.5 0.3 0.2 0.2 0.7 0.1 0.5 0.7 1.0 0. 0.8 0.6 0.9 0.5 0.2 0.7 0.8 0. 0.2 1.0
0.80
0.85
0.90
0.95
1.00
Ta rg
et a
ttr ib
ut es y 0 1 2 3 4 5 6 7 8
9
Bias attributes b
RedGreen Yellow Turquoise Orange Purple Cyan Beige Lime Brown
|m|/|D| = 0.01
1.0 1.0 1.0 1.0 1.0 1.0 0.8 1.0 0.9 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.9 1.0 0.8 1.0 1.0 1.0 0.7 1.0 1.0 1.0 1.0 0.9 1.0 0.9 0.9 1.0 0.9 1.0 1.0 1.0 0.9 1.0 0.9 1.0 1.0 0.9 0.9 1.0 0.8 1.0 0.9 0.9 0.8 1.0 1.0 0.9 1.0 0.9 0.9 0.9 1.0 1.0 1.0 1.0 0.9 1.0 1.0 0.9 1.0 0.9 0.8 0.8 0.8 0.8 1.0 1.0 1.0 1.0 0.8 0.9 0.9 0.8 0.8 0.9 0.9 0.7 0.9 1.0 0.6 1.0 0.9 0.9 0.9 0.7 0.9 0.9 0.9 0.8 1.0
0.80
0.85
0.90
0.95
1.00
Ta rg
et a
ttr ib
ut es y 0 1 2 3 4 5 6 7 8
9
Bias attributes b
RedGreen Yellow Turquoise Orange Purple Cyan Beige Lime Brown
|m|/|D| = 0.05
Figure 3: Confusion matrices for |M |/|D| = 99% and 95%.
(Nam et al., 2020) argued that this problem arises when two conditions are met simultaneously: (C1) bias attributes b is easier to learn than target y, and (C2) highly correlated b to y. We focus on observing the effect of (C2) on the unintended bias problem among the two conditions when (C1) holds. To understand (C2) precisely, we check the effect of the minority ratio |m|/|D| on the accuracy. A detailed description of the training setting is provided in Appendix A.2.
We investigate the test accuracy of the samples in the majority and minority sets. As in Figure 2, the average performance drop of the minority samples intensifies when the minority ratio |m|/|D| decreases. On the other hand, the accuracy of the majority samples for all cases are not dropped. More precisely, we plot the confusion matrix to see how poor the performance is when the bias attributes b and the target attributes y are not aligned according to the minority ratio |m|/|D|. As in Figure 3, the increments of the minority ratio does not make the performance change of the aligned samples located in the diagonal, but other entries change a lot. These results can be interpreted as when the ratio |m|/|D| becomes small, the model highly likely infer the bias attributes b rather than target attributes y.

heading: 4 DE-BIASING USING PER-SAMPLE GRADIENT INFORMATION
text: When a given training set is biased, the model learns b over the intended target y, and it is strengthened when |m|/|D| approaches zero. If this is the case, it is desirable to modify the training set to a larger |m|/|D| so that the model learns the target y rather than the biased attribute b. To do so, we considered constructing a new training set D′ = {(x′i, y′i)}Ni=1 such that |m′|/|D′| increases. Similar to prior resampling-based work (Li & Vasconcelos, 2019), we sequentially train two separated networks: biased model fθb and de-biased model fθd . The biased model, fθb is trained based on the raw training set D. After sufficiently converging to point θb → θ̂b, the model computes gradients ∇θi = ∇θLCE(xi, yi; θ) of all the samples (xi, yi) ∈ D. Then, we compute the sampling probability (ps(i)) of the ith sample using the per-sample gradient ∇θi, to break (C2) by balancing between M and m. Finally, we train the de-biased model fθd using a rejection sampler that creates a balanced training set D′ based on the sampling probability ps(i).

heading: 4.1 PER-SAMPLE GRADIENT-BASED SCORES
text: As the first step of de-biasing, we train the biased model without modifying training set D. The converged model fθ̂b(·) contains in-depth information about the distribution of the training set in its model (especially, through the per-sample gradient). Therefore, we leverage the per-sample gradient, ∇θi = ∂LCE(x,y;θ̂b)∂θ |xi,yi of (xi, yi), which is dissolved in the biased model.
Why the per-sample gradient? The majority and minority samples exhibit a clear difference in the per-sample gradient for the model fθ̂b owing to the following reasons. (R1) Each sample has its own loss landscape, and similar representation samples have similar loss landscapes (see Figure 4(a)). In other words, the minority samples have different loss landscapes from those of the majorities, which implies that a local minimum cluster is composed of similar representation samples. (R2)
From the averaging property of the batch-based optimization methods, the influence of the minority samples is diluted of according to their ratio. Thus, the model trained on the biased training set converges to a proximity point of the minimum point of the majorities and it is reflected in the gradient of each sample. The red arrows in Figure 4(a) and 4(b) indicate that the majority samples (top) have shorter gradients than the minorities (bottom) for both per-sample and aggregated cases. Similarly, the direction of both cases is significantly different. Additionally, as seen in Figure 4(c), the portion of minority samples in the lower directional likelihood and higher magnitude bins is much larger than the majority samples.
Gradient-based score. Based on the intuition of per-sample gradient and empirical results, we design a gradient-based score S(i) = S ( (xi, yi); θ̂b ) , which is computed for each class separately.
The score is computed using two base scores, the magnitude score M(i) and direction score D(i), which are defined as follows:
M(i) = ( (1/‖∇θi‖2)∑N j=1(1/‖∇θj‖2) ) , D(i) = ( pdir(i)∑N j=1 pdir(j) ) . (1)
(1) Magnitude score M(i): The contribution to prejudicial training is inversely proportional to the magnitude of its gradient (∝ 1/‖∇θi‖). We used Euclidian distance to measure the magnitude. (2) Direction scoreD(i): The contribution to prejudicial training is also proportional to the directional likelihood (∝ pdir(i)). For the directional likelihood, the estimated von Mises-Fisher (vMF) distribution is used to understand the concentration of the directional distribution. The vMF distribution is defined as follows: Definition 4.1 (von Mises-Fisher Distribution). The pdf of the vMF(u, κ) is given by
fd(v;u, κ) = Cd(κ)e (κu>v),
on the hypersphere Sd−1 ⊂ Rd, where κ determines how much samples are concentrated based on the distribution along the mean direction u, and Cd(κ) is a normalization constant determined by dimension d and the concentration parameter κ.
To estimates the parameters of the assumed vMF distribution, we use approximated maximum likelihood estimates (MLE) solutions û and κ̂. For normalized gradient vectors ∇θi = ∇θi‖∇θi‖ ,
the MLE solutions û and κ̂ can be obtained from û = ∑N
i=1∇θi ‖ ∑N i=1∇θi‖ and κ̂ ≈ r̄(d−r̄ 2) (1−r̄2) , where
r̄ = ‖ ∑N
i=1∇θi‖ N (see Appendix G).
Then, the normalized directional score D(i) is defined in (1), where pdir(i) = fd(∇θi; û, κ̂) is the approximated vMF distribution.
To relax the computational constraint, we select k-dominant sub-dimensions, denoted by the set

heading: K = arg max
text: K⊂N,|K|=k
∑ j∈K (∑ i |∇θj |i ) , where ∇θj corresponds to the gradient of the j-th dimension.

heading: 4.2 DE-BIASING USING GRADIENT-BASED SCORES
text: Balanced batch selection. By the definitions of the scores, the sample in the majority set M (or the minority set m) must have a higher (or smaller) M(i) and D(i), respectively. Each sample must interact inversely with the two scores to treat the samples in the minority set m similarly to the samples in the majority set. To do so, we design aggregated score S(i) based on the harmonic mean, and sampling probability ps(i) as follows:
S(i) =
( λ
M(i) +
1
D(i)
)−1 , ps(i) =
C S(i) ,
where λ is the balancing hyperparameter to emphasize a specific score between M(i) and D(i), and C is the normalization constant defined as C = mini S(i), which makes ps(i) = 1 for the smallest S(i) sample. From the harmonic-mean, S(i) has a higher value when one of the scores is high.
Algorithm 1 Uniform Sampler 1: Sampling prob. ps, Batch size k 2: while |B| < k do 3: i ∼ U(1, N), p ∼ U(0, 1) 4: if p ≤ ps(i) then; B = B ∪ i 5: end if 6: end while To construct a balanced mini-batch, our algorithm use a rejectionsampling-based uniform sampler, Algorithm 1. Each sample i is rejected with a probability 1 − ps(i), which implies that the rejection is proportionally to the scores. From the property of rejection sampling, such a sampler generates uniform batches.
Algorithm 2 De-biasing using the uniform sampler 1: D, λ, q, lr η, T , k, aug. alg. A(·), sampling alg S(·) 2: /** STEP 1: Train fθb **/ 3: for t=1,2,...,T do 4: Draw a mini-batch (X,Y ) = {(x, y)}ki=1 from D 5: Update θb ← θb − η|D|∇θ ∑ (X,Y ) LGCE(x, y; q)
6: end for 7: /** STEP 2: Calculate ps(i) **/ 8: Calculate ps(i) for all i ∈ D 9: /** STEP 3: Train fθd **/
10: for R do t=1,2,...,T 11: Draw a mini-batch {(x, y)}ki=1 from the Alg. 1: S(ps(i)) 12: Update θd ← θd − η|D|∇θ ∑ (X,Y ) LCE(A(x), y) 13: end for Gradient-based de-biasing. The ultimate debiasing algorithm consists of three steps. First, the biased model fθb is trained on the raw data samples with GCE loss (Zhang & Sabuncu, 2018; Nam et al., 2020). The GCE loss controlled by q guides the biased model to ignore the minority samples in the training set. Then, all raw samples are input into the biased model to compute ps(i). The last step is to train the debiased model fθd using a balanced mini-batch sampler. This sampler generates balanced minibatch based on the sampling probability ps(i). We use the typical image data augmentation method A(·) (e.g., rotate, resize) to avoid over-fitting.

heading: 5 DE-NOISING A SMALL PORTION OF THE NOISY LABELS
text: Why is noisy label a problem for de-biasing algorithms? All de-biasing algorithms work by emphasizing rare samples. For example, the proposed method operates by reconstructing minibatches in proportion to the rarity. However, when a small portion of incorrectly labeled samples, called noisy labels, is in the training set, the de-biasing algorithms accentuate the noise labels due to their mechanism of enlarging the influence of the rare samples. Therefore, as presented in Figure 5, all de-biasing algorithms suffer a performance decline.
Distinguishing minority and noisy labels via compensated loss. Recent de-noising methods (Yu et al., 2019; Yi & Wu, 2019; Li et al., 2019; Kim et al., 2021b) have mainly focused on ignoring unusual sample, which may be noisy labels. However, by comparing de-noising mechanism with (Kim et al., 2021b), previous de-noising algorithms also ignores minority samples which must be highlighted not discarded, see Appendix I for details. Therefore, we design a module to delete noisylabels in which the minority samples are preserved. To do so, we observe the loss statistics of the minority samples and noisy labels under various Colored MNIST settings with
(|m|/|D|, noisy label ratio) = (0.01, 0.1) and (0.1, 0.1) (see Appendix B.3). As depicted in Figure 6, noisy labels have a higher loss than clean labels, regardless of whether the sample is in the majority or the minority set. The trained model that ignores rare samples, outputs the correct results, whereas the given label for calculating the loss value is incorrect.
However, the loss of the clean minority samples increases, owing to the rarity. To compensate, we suggest the compensated loss L̄i, reducing the influence of the rarity, and the splitting criteria to distinguish between the clean samples and noisy labels:
Dclean = { (xi, yi)|L̄i < δ } , Dnoise = { (xi, yi)|L̄i ≥ δ } , where L̄i =
LCE(xi, yi; θ̂n) H(fθ(xi)) . (2)
Note that H(fθ(xi)) represents the softmax entropy of sample xi, and δ =∑ i≤N L̄i/ ∑ i≤N H(fθ(xi)). In Figure 6, the black lines represent the threshold δ, and the lower part of this line represents Dclean, whereas the upper part represents Dnoise, respectively. Other discrimination results are depicted in Appendix E.
Algorithm 3 De-noising & De-biasing 1: Train network fθn with CE loss LCE 2: Split Dclean and Dnoise based on (2) 3: De-biasing using Dclean De-noising module. We design a de-noising module that is applicable in front of all prior de-biasing algorithms from the proposed criteria. As presented in Algorithm 3, this module comprises three simple steps. First, a noise model parameterized by θ̂n is trained on the raw biased and noisy training sets. Then, we split Dclean and Dnoise according to the criteria (2). After splitting, we run the de-biasing algorithm using Dclean.

heading: 6 EXPERIMENTS
text: We evaluated the proposed model for two types of benchmarks: synthetically generated bias (referred as controlled biased benchmarks) and raw bias (real-world biased benchmarks). We compared the results with the officially available recent de-biasing methods, RUBi (Cadene et al., 2019), LearnedMixinH (Clark et al., 2019), ReBias (Bahng et al., 2020), AFLite (Le Bras et al., 2020), LfF (Nam et al., 2020), REPAIR (Li & Vasconcelos, 2019) and the vanilla model. Each baseline was reproduced using the official codes for each algorithm (see Appendix C). Moreover, we optimized all algorithms based on the 10% of the biased training set. For the experiments, three types of convolutional neural networks were used with various regularization methods (batch normalization (Ioffe & Szegedy, 2015), dropout (Srivastava et al., 2014), and weight decaying (Moody & Hanson)). We report the detailed settings in Appendix A.1. In addition, we determined the test under flipped labels in MNIST variants with various noise ratios of 10%, 5%, and 0% to verify the side effects from noisy labels and check the protecting performance of the de-noising module (see Appendix A).

heading: 6.1 BENCHMARKS

heading: 6.1.1 CONTROLLED BIASED BENCHMARKS
text: To precisely examine the de-biasing performance, we generated controlled biased benchmarks using well-known data: MNIST (LeCun et al., 2010) and Cartoon (Royer et al., 2020). Various bias ratios, |M |/|D| ∈ {0.995, 0.99, 0.95}, were used for all the controlled tests. All benchmarks were divided into a biased training set Dtr, biased validation set Dval (10% of Dtr), and test set Dte comprising the majority M and minority m (see Appendix B.1 for detailed data construction rules.)
MNIST Variants. The MNIST dataset comprises a grayscale digit data with a handwritten style. We injected two different biases, color and different objects (specifically, Fashion MNIST (Xiao et al., 2017)), which are colored MNIST (CM) and watermarked MNIST (WM), respectively. This main task is classifying the shape of the digit without examining the biased attribute color of the
digit or Fashion MNIST objects. As an example of the CM, class 0 is divided into major and minor sets colored in red and the others. Similarly, in WM, the major set of class 1 contains trouser, whereas the minor set contains other objects as watermarks at the top-left corner. Examples are depicted in Figure 7.
Cartoon. A Cartoon (CT) (Royer et al., 2020) is an avatar image with 18 corresponding attributes (e.g., hair style). We select two of them, hair color and face color, for the target and biased attributes. To create a bias, the major set is whose two attributes are bonded was subsampled. For example, in Figure 8, the face color of blond hair is black, except for a few.

heading: 6.1.2 REAL-WORLD BIASED BENCHMARKS
text: We used three well-known biased benchmarks to determine the de-biasing performance on complex benchmarks: Biased Action Recognition (BAR), CelebA, and ImageNet/ImageNet-A (see Appendix B.2 for details).
Biased Action Recognition. Biased action recognition (Bahng et al., 2020; Nam et al., 2020) is a motion classification set with a background bias. For example, in the class climbing, major and Minor samples have rockwall and ice cliff backgrounds, respectively.
CelebA. (Nam et al., 2020; Liu et al., 2015) CelebA has facial images with 40 binary attributes. Among these, we used heavy makeup and gender attributes for target and biased attributes. In the heavy makeup class, major is the female attribute, whereas male is the minority attribute.
ImageNet/ImageNet-A. ImageNet (Deng et al., 2009) is a well-known image classification dataset. However, (Hendrycks et al., 2019) claimed, a model trained on the ImageNet data easily fails to classify unusual backgrounds. This is because ImageNet has a biased background (e.g., almost all Frog images are considered in the swamp, which means major sample). As a de-biasing performance evaluation benchmark, we used ImageNet for training and to test ImageNet-A images with unusual backgrounds.

heading: 6.2 EVALUATION
text: Controlled benchmarks without noisy labels. The accuracy values of two major and minor sets are reported in Table 1. In particular, we first examined the accuracy of the vanilla algorithm, the ultimate baseline. Overall, the vanilla algorithm achieved nearly 100% accuracy for the major case, the vanilla failed to infer with an accuracy similar to that of the major case.
De-biasing algorithms correctly classified the major case, similar to the vanilla case. However, for the minor case, these algorithms obtained higher accuracy values than that of the vanilla results (at most 31.78% ↑). Despite these de-biasing performances, they have a higher gap between major and minor cases (at least 36.69% under a 99.5% bias ratio). In contrast, the proposed method obtained the highest accuracy value for the minor case (at most 69.92% ↑) compared to the vanilla results. These results were obtained from the hyperparameter tuning with a biased validation set. In appendex D, we report the oracle results obtained with the optimal hyperparameters, which were tuned by using the majority-minority split validation set.
Controlled benchmarks with noisy labels. In Table 2, almost all algorithms failed to improve the minority accuracy, when labels are flipped, compared to the 0% noisy label case. For example, the proposed model’s average performance declined from 98.50% to 84.05%, and the majority performance declined from 99.24% to 80.60% for the CM case with 5% noisy labels. However, these performance decrements were recovered from the de-noising module for all cases. For the 5% noise CM case, all cases obtained close to 100% accuracy for the majority case and improved the performance for the minority case. Among all methods, the proposed gradient-based method obtained 92.94% on average of the majority and minority test sets for the 5% noise of CM case, and improved by 7.83% in performance from the state-of-the-art algorithm, LfF at 85.11%. Moreover, we checked from the 0% noise case that the adverse effect of the de-noising module on de-biasing is −3.9% (99.50%→ 95.60%) on CM case for the proposed method.
Real-world benchmarks. In Table 3, we report that real-world benchmarks can be biased by observing the lousy performance of the vanilla algorithm for all cases. The BAR case indicates that the proposed method gained the best performance on average among classes (a 5.06% increase from the vanilla algorithm and 2.53% increase from the second-best accuracy). As listed in other columns, the proposed method has helped improve vanilla by 3.64%
and 2.87% for the CelebA and ImageNet/ImageNet-A cases, respectively. It also outperforms other methods by up to 1.11% and 3.00%, respectively. A detailed analysis is provided in Appendix F.
Analysis. We report the detailed various studies in Appendix H.

heading: 7 CONCLUSION
text: We presented a novel de-biased method by generating a mini-batch so that the trained model does not have prejudice in the inference. The proposed sampling metric is based on the gradient of each sample, which is mainly correlated with its familiarity. Through extensive experiments using various biased datasets, we demonstrated the effectiveness of the proposed method in several cases. Furthermore, we proposed a de-noising module that is easily applicable to all de-biasing algorithms and protects from the side effects of noisy labels. We hope this study proposed exploration to understand the feature bias problem, especially for resampling-based approaches and gradient-based data valuation areas of machine learning.

heading: A EXPERIMENT DETAILS
text: A.1 BASIC SETTINGS
Network architecture. We use three types of convolutional neural networks (CNNs) (see Figure 9 for two of them and the other is ResNet-18 (He et al., 2016)) for all benchmarks. For MNIST variants, three convolution layers with filter size 4 and average pooling with size 2 are used (see Simple ConvNet-1). For Cartoon, 5 convolution layers with filter size [6, 6, 5, 5, 4] for each layer with average pooling size 2 are used (see Simple ConvNet-2). The real-world benchmarks are tested under pre-trained ResNet-18 provided by Pytorch torchvision, without freezing feature extractor. We apply batch normalization (Ioffe & Szegedy, 2015), dropout (Srivastava et al., 2014) for regularization, and utilize SGD optimizer with momentum and weight decay (Moody & Hanson).
Hyperparameters. The hyperparameters that we used are summarized in Table 4. We share basic hyperparameters (including Network Type, Learning rate, Learning rate decay, Learning rate decay epoch, Momentum, Weight decay, and Batch size and Number of Epoch) for all algorithms. Precisely, for the proposed method, we set GCE parameter q and Dimension size K for our method, as in Table 4.
Validation. We optimize hyperparameters using a validation set of 10% of the training set and unseen at the training phase. Reported our results are obtained from the hyperparameters which shows a small loss for the validation set.
Data augmentation. Image data augmentation operations have been used frequently to improve generalization performance. We utilize three operations: RandomResizedCrop with scale (0.9, 1.1), ColorJitter with 0.05 hue and 0.05 saturation, and RandomRotation with (−10, 10) degree using NEAREST filling criterion. These augmentation operations are from transforms module in torchvision.
A.2 TEST SETTINGS IN SECTION 3, 4, AND 5
Ratio of minority samples analysis. To analyze the correlation between |m|/|D| and accuracy statistics, we generated a bias training set based on CM data generation in Appendix B.1. This tests are conducted on the experiment with difficulty 0.0001.
Gradient property test To test our intuitions about gradients, we utilize part of CM data (includes class 0 and 1) with a bias ratio of 99.5% and difficulty 0.0001. Finally, we draw red arrows using the
lowest point and center point. This center point is the convergence point, and other points are random permuted loss values. All loss landscape figures are plotted by following (Li et al., 2018a).

heading: B BENCHMARKS
text: B.1 CONTROLLED BIASED BENCHMARKS
MNIST Variants. MNIST variants are modified data from gray-scaled hand-written digit MNIST images (LeCun et al., 2010). To inject biased attributes into the MNIST, we use two different attributes: color and object.
• Colored MNIST (CM) has been frequently used as a benchmark due to their simplicity (Kim et al., 2019; Bahng et al., 2020; Nam et al., 2020). The target of this benchmark is classifying the target attributes (shape of digits) not the biased attributes (color of digits). To make biased training set, all images are colored after up-scaling dimension from gray R28×28 to 3D RGBR28×28×3. Uniformly sampled 3-dimensional 10 colors Cc = {Rc, Gc, Bc}9c=0 are allocated to each class c. The samples in class c of set M are colored by their allocated color Cc. The other samples are colored with the color vector Cc′ 6=c where c′ 6= c is uniformly sampled. Furthermore, to deviate correlation between the target and biased attributes, color for each sample (xi, yi) is deviated using 3-dimensional Gaussian distribution Ci = N (Cyi , αI). This deviation is controlled by the difficulty parameter α ∈ {0.0001, 0.0005}.
• Watermarked MNIST (WM) has a different type of bias. The target of this benchmark is the same with generic MNIST, classifying hand-written digits, while the unintended object (Fashion MNIST (Xiao et al., 2017) data) is located as a biased attributes. To locate Fashion MNIST, all images from MNIST are size-up scaled fromR28×28 toR56×56. Set M and m includes samples (xi, yi) where image xi is made with digit image xtarget and fashion object xbias whose label indices follow yi = ytarget = ybias and yi = ytarget 6= ybias, respectively. To make deviation, we put images from δ pixels far from top and bottom of the canvas. δ is sampled from uniform distribution δ ∼ U(0, α). We test two cases, α ∈ {8, 16}.
Cartoon. Cartoon data composed of synthetically generated face image samples which have 18 attributes (e.g., hair color, face color) with multiple labels (e.g., hair color = {1, ..., 10}). Among 18 attributes, we choose hair color and face color attributes as the target attributes and the biased attributes. For simplicity, the size of each label is reduced from 9 to 4 and 10 to 4 by mapping 2 classes (similar colors) into 1 class, (e.g., original classes 1, 2 are mapped to the new class 1). We make set M whose new target class and biased attributes following ctarget = cbias. Also, set m are sampled among images following ctarget 6= cbias.
B.2 REAL-WORLD BIASED BENCHMARKS
Biased Action Recognition. Biased action recognition data has 1, 941 motion images for training with 6-motions {Climbing, Diving, Fishing, Racing, Throwing, Vaulting} on various types of backgrounds (e.g., (Climbing, Rockwall)). The images in this benchmark include the frequent background of that action as a biased attributes for set M and rare backgrounds for set m (e.g., (Climbing, Rockwall) for M and (Climbing, Ice cliff) for m). This benchmark is originated from (Bahng et al., 2020; Nam et al., 2020; Kim et al., 2021a).
CelebA. CelebA has 202, 599 face images with 40 binary attributes from 10, 177 identities. Among 40 attributes, we select two attributes (HeavyMakeup,Male) for target and biased attributes. The size of training subsets M and m are 130, 410, and 32, 360, respectively. Remains are used for evaluation. Splitting training and evaluation sets follow (Liu et al., 2015). This task reflects that a real-world benchmark is easy to be biased in some specific attributes, unintendedly. This benchmark is originated from (Nam et al., 2020).
ImageNet/ImageNet-A. ImageNet data has 1, 000 classes, and we utilize a subset of ImageNet, which has 9 super-classes for scalability (Bahng et al., 2020). Also, we evaluate using ImageNetA (Hendrycks et al., 2019), which gathers the failure cases of ImageNet from web. As described in (Bahng et al., 2020; Hendrycks et al., 2019) dataset, “frequent background can be a biased
attributes.” For example, frogs in the swamp have a higher proportion than other backgrounds (e.g., underwater). It indirectly represents that a de-biased model trained under ImageNet has better de-biasing performance when it has higher accuracy on ImageNet-A samples.
B.3 NOISY LABELS
We inject 10% noise labels into the basic configuration conducted in Section 5. Noise labels are uniform randomly flipped by following rules. At first, we randomly sampled a set of noise label candidates regardless of whether the source of the data samples are majority set or minority set. Flip the labelyi → ŷi uniform randomly, so that the label is not equal to the ground truth ŷi 6= yi. Three noise label ratios, {90%, 95%, 1%0%}, were used. In order to train noised model fθ̂n , we utilize cross-entropy loss (not generalized cross-entropy loss). Also, to avoid over-fitting, we utilize three data augmentation operations: RandomResizedCrop, ColorJitter, and Random Rotation which are parameterized by (0.9, 1.1), (0.05, 0.05) and (−10, 10), respectively.

heading: C BASELINES
text: Vanilla As an ultimate baseline, which means without de-biasing module, we train a network called vanilla. As summarized before, we utilize frequently used regularization techniques and simple data augmentation operations (jitter, crop, and rotation).
LearnedMixinH (Clark et al., 2019) LearnedMixinH was proposed by Clark et al for VQA task. This method of de-biasing is composed of biased (on the question) and unbiased (on both the image and the question) networks. We generate a biased network in our experiment by simply training on the biased data set. This is based on the argument in LfF (Nam et al., 2020), which states that biased models learn biased information first, because it is easier to learn than the target attribute. The biased model outputs two values: its softmax output and a bias prediction value for each given sample. Ultimately, the de-biased model trained under the cross-entropy loss with regularizer of two above values from the biased model. We implement this algorithm by inheriting the authors’ official code and unofficial code made by the authors (Bahng et al., 2020).
RUBi (Cadene et al., 2019) Similar with LearnedMixinH, RUBi also targets to solve VQA tasks. We also utilize the argument from LfF (Nam et al., 2020), to train the biased network. Instead using two values of the biased model, the authors of this paper utilized sigmoid outputs of the biased model and give weights to the de-biased model’s output, when calculating the cross-entropy loss. This weighting mechanism gives weighted punishements by followings. The biased model outputs highly and lower biased sigmoid output when major and minor samples are given. By multiplying sigmoid values to the softmax output of the de-biased model, the de-biased model suffers higher loss from the minor samples. We implement this method using the authors official code and unofficial code from (Bahng et al., 2020).
ReBias (Bahng et al., 2020) ReBias is a bi-level optimization-based de-biasing algorithm proposed by Bahng et al. ReBias trains the biased and de-biased model simultaneously. The de-biased model updated its cross-entropy loss and additional regularizer using Hilbert-Schmidt Independence Criterion (HSIC). Here, the biased model is composed of large convolutional filters to capture the bias information better, but we ignore this for a fair comparison with same network. We utilize official code from the authors.
LfF (Nam et al., 2020) The main module of this algorithm is two-fold: (i) The biased model trained on Generalized cross-entropy (GCE), which highlights the major samples. (ii) The de-biased model trained on the weighted loss where weight is computed by the cross-entropy outputs from each. From GCE, the cross entropy loss loss of the minor samples increases. We utilize official code from authors.
AFLite (Le Bras et al., 2020) AFLite is another type of resampling approach. AFLite trains the biased model multiple times and selects samples whose accuracy is low. At the de-biasing phase, the de-biased model is trainend on the sub-sampled new training set. Because authors offer the official code for a synthetic data, we manually reproduce their idea.
REPAIR (Li & Vasconcelos, 2019) REPAIR is a re-sampling-based de-biasing algorithm proposed by Li et al. REPAIR reconfigure the training set based on the weight parameter w obtained by the adversarial training. As a first step, the biased model and the adversary are trained simultaneously. The adversary computes per sample weight w so that the biased model gets a larger value for the minorities. REPAIR utilized w to reconstruct the repaired training set. The authors proposed four types of re-configuring methods with given parameter k: REPAIR-T (Thresholding, sampling above k), REPAIR-R (Ranking, Sampling top-k%), REPAIR-PR (Per-class Ranking), REPAIR-S (Sampling, reject with probability 1 − w). We utilize REPAIR-S among these variants to fairly compare our sampling mechanism (without hyperparameter k case). We implement REPAIR using the official code from the authors. Additionally, note that we obtained w by training the classifier at the adversarial training phase without using biased information, such as RGB values in the case of CM, where as the official code used the biased information. This is enabled by the LfF (Nam et al., 2020) argument, which state that the network is biased toward a biased attribute that is easier to learn than the target.

heading: D RESULT: CONTROLLED BIASED BENCHMARKS (ORACLE VALIDATION)
text: These results are obtained from the tuned hyperparameters, which are optimized by using the labels related to the biased attributes b . As we can see, the proposed method obtains the best performance in terms of the average accuracy of majority and minority cases. Moreover, other works also successfully improves their results. However, it is difficult to obtain a validation set which is discriminated between majority and minority samples. Therefore, we can conclude that the proposed method obtains when we can access to the discriminated dataset between majority and minority cases, which is difficult in practice.

heading: E SPLITTING PERFORMANCE OF DE-NOISING MODULE
text: Discriminativeness power of the de-noising module between majority and minority cases on various noisy label ratio and bias ratio |m|/|D|. As we can see for all cases, our de-noising module with compensated loss criterion can distinguish the minority samples and noisy labels. In addition, when there is no noisy cases, (0% cases) the proposed de-noising module lose some valuable clean minority samples. All cases are run with difficulty parameter 0.0001 and 8 for CM and WM cases, respectively.

heading: F REAL-WORLD BENCHMARK: DETAIL ACCURACY
text: As in all cases, the proposed method obtains the best performance on average of all cases, e.g., CelebA on average of majority and minority cases. Also, our BAR results fails to obtain state-of-the-art performance in some classes, but get the best on average of all classes.
G VON MISES-FISHER DISTRIBUTION FOR GRADIENTS
The von Mises-Fisher (vMF) distribution is a probability distribution on the hypersphere Sd−1 ⊂ Rd. The probability density function of the vMF distribution for the unit vector v is given by:
fd(v;u, κ) = Cd(κ)e (κu>v),
where κ and u are called concentration parameter and mean direction, respectively. The normalization constant Cd(κ) is equal to Cd(κ) = κ p/2−1
(2π)p/2Ip/2−1(κ) , where It denotes the modified Bessel function
of the first kind at order t.
As in (Banerjee et al., 2005), whenN samples from the vMF distribution are given as vi, the maximum likelihood estimates solutions û and κ̂ can be obtained from û = ∑N i=1 vi
‖ ∑N i=1 vi‖ and κ̂ ≈ r(d−r 2) 1−r2 , where
r = ‖ ∑N
i=1 vi‖ N .
To interpret gradient vectors as vMF distribution, given vectors have to be unit vectors: ∇θi = ∇θi‖∇θi‖ . Therefore, the approximated vMF distribution of the gradients can be obtained as follows:
fd(∇θi; û, κ̂) where û = ∑N i=1∇θi
‖ ∑N i=1∇θi‖ , κ̂ ≈ r̄(d− r̄ 2) (1− r̄2) , and r̄ = ‖ ∑N i=1∇θi‖ N .

heading: H ANALYSIS
text: We conducted a few studies to analyze our method: hyperparameter sensitivity, (target,bias)-pairwise accuracy, pairwise sampling probability, ablation study on λ, adjusting objective results based on our metric, class activation map, unbiased case result, training time, results on naive metrics, and impact of GCE. Controlled benchmarks were used for analysis.
Hyperparameter sensitivity. In Figure 12, accuracy results of the models trained on the various learning rate η ∈ {0.005, 0.01, 0.02} and epochs e ∈ {50, 100} were depicted. It shows that the prediction performance of all algorithms has higher accuracies on biased validation sets and majority set for various learning rates and epochs. However, in minority case 12(c), ours have a less sensitive result on minority cases by seeing error bar’s length.
(Target, Bias)-pairwise accuracy. In Figure 13, pairwise accuracy matrices were plotted using various de-biasing methods. This represents the prediction result of 10 Major (diagonal) and 90 Minor pairs. First, vanilla fails to infer Minor pairs, but it infers Major accurately. Similar to vanilla, other algorithms, including ours, achieve almost accurate results for the Major pairs. On the other hand, based on the perspective of minority, other algorithms fail to infer. Among the de-biasing methods, our method has been approximately 100% accurate for almost all the pairs.
Pairwise sampling probability. To understand the previous pairwise result, we checked the sampling probability of our method. In Figure 14, each subfigure indicates pairwise sampling probability for all the metrics: pM (i) =
1/M(i)∑ i(1/M(i)) , pD(i) = 1/D(i)∑ i(1/D(i))
, and ps(i) = 1/S(i). In Figure 14(a), the raw case is highly bonded between the shape and color (see diagonal). In pM (i) and pD(i) cases, both are well mixed compared to without case. In Figure 14(d), ps(i) has the most uniformized sampling probability. Red and yellow indicators denote complementary effect of both scores.
Ablation study on λ. In the definition of the score S(i), we set the hyperparameter λ to make balance between source scores M(i) and D(i). In order to confirm the effect of λ, the experimental results
are reported from various values of λ = {0, 1, 10, 100} on CM with 99% bias without noise. As in Table 11, there is no remarkable performance difference depending on the λ, so it could be observed that it is robust to the hyperparameter λ.
Benchmark C. M. W. M.
Type Avg. Avg.
Vanilla 64.56 73.26 LfF 77.82 74.13
REPAIR 82.37 84.86
Ours† 95.33 86.67 Ours 98.14 91.23
Adjusting objective vs Resampling. As an ablation study, we clarified the power of our gradient-based metric by reporting the results of adjusting objective leveraged by our ps(i). We conducted tests under two controlled biased benchmarks, colored MNIST and watermarked MNIST with difficulty 0.0001 and 8. ps(i), normalized by the mean value, ps(i)∑N
i ps(i)/N , is used for the weights. Our gradient-based score
outperforms others in the adjusting objective (dagger mark †) and resampling cases. Compared with our gradient-based ways, the adjusting objective methods achieve lower performance. This implies that, as argued in (An et al., 2020), the adjusting objective-based approach suffers from training sensitivity.
What de-biased model learn. The main goal of de-biasing algorithms is to make the de-biased model not learn the biased feature. To check whether the biased models satisfy this goal, we evaluate two tests: (1) random color test for CM, and (2) CAM for WM.
We construct test dataset by uniformly randomly sample colors for all samples RGB ∼ U(0, 1)3. As in Table 12, the result of ours show that it outputs based on the shape whether the training set is biased or not.
Additionally, we conduct CAM test for WM dataset. CAM (Selvaraju et al., 2017) has been used as a tool to explain “where a model is
focusing on?” In Figure 15, we state the CAM results for watermarked MNIST samples obtained from the Major (Left) and Minor (Right) sets. In the top row (for Major), our model better ignores the biased object (fashion object) than the other methods. Even for Minor, only our model determines the target digit, while the rest focuses on the biased object (Ankle boots).
Unbiased / Rarely biased case. To analyze the general side effect of additional modules, performance degradation, we check the accuracy of unbiased (especially, 10% = 1/(# of classes) biased case) and rarely biased (70%, 80%, 90%) cases, under MNIST variants with difficulties of 0.0001. As listed in the table, all models have similar accuracy to the vanilla case. This means that, re-
gardless of whether the training set is biased, all models, including ours, do not suffer from severe side effects. It shows that our algorithm can be applied without performance loss. In short, our algorithm needs not to utilize human knowledge about bias in the training set, an on/off knowledge.
Training Time. To see the drawbacks of training time for de-biasing, we measure the training time of 6 related works on WM case. As in Table 14, the proposed model spends ×1.45 time compared to RUBi which is the fastest de-biasing method, while ×0.72 faster than the other re-sampling algorithm, REPAIR.
99.27 / 96.92 99.83 / 11.00 99.87 / 38.31 99.87 / 10.27
0
0.002
0.004
0.006
0.008
0.010
0.012
Gradient Softmax Logit Loss Sampling probability Other metrics. This study shows other metrics-based resampling results, (e.g., softmax response, loss, and logit value), based on sampling probability. These results show empirical justification of our gradient-based scores for de-biasing. All scores are defined to make the minority samples (unfamiliar samples) have a higher sampling probability. Each score is defined as follows:
Ssoftmax(i) = 1− S(f(xi, yi))∑ j(1− S(f(xj , yj))) , Slogit(i) = m− f(xi, yi)∑ j(m− (f(xj , yj)) , Sloss(i) = L(xi, yi)∑ j L(xj , yj) ,
where S(·) is softmax response and m = maxk∈D(f(xk, yk)). As depicted in Figure H, our gradientbased approach has the highest performance on the average accuracy of majority and minority results. Moreover, the gradient method has a short error bar than the others, which works reasonably (softmax and loss case).
Impact of GCE. We checked the proposed method without GCE (trained with CE loss) under 99.5% WM case. The proposed method w/o GCE achieved (M /m) 99.15% / 81.72%, which outperforms the others, while lower than the proposed method without GCE case (99.35% / 83.10%).
I DE-NOISING
In recent noisy label cases, there have been two major trends: (1) constructing robust objective functions or regularizers (Cao et al., 2020; Yi & Wu, 2019), and (2) cleaning the noisy data (Li et al., 2019; Kim et al., 2021b). Those strategies are mostly used to reduce the influence of rare
samples, such as out-of-cluster samples in (Kim et al., 2021a). However, in the case of dataset bias problem, minority samples can be omitted when rare samples are discarded, resulting in de-biasing algorithms’ performance degrading. As a result, erasing noisy label samples while maintaining minority samples is crucial in the dataset bias problem with noisy labels to preserve the de-biasing performance. Table 16 shows the number of preserved samples after cleansing steps. We compare the proposed de-noising module to the state-of-the-art method, FINE. It cutting-out noisy labels by using eigenvectors. As shown in Table 16, while the proposed de-noising module does not perfectly eliminate noisy samples, it does preserve clean minority samples, whereas FINE does not.

Review summaries:
The paper propose to use the per-sample gradients from the biased classifier to tackle the dataset bias. Following prior work, this work adopts a three-step approach: 1) train a classifier on biased dataset 2) identify which samples are from the minority and which are not 3) train a classifier with re-sampling. The proposed per-sample gradient-based scores are used in step 2. The motivation is purely from the empirical observation that samples from the majority group tend to have similar gradient magnitude and direction. From the observation, they derive a scoring function for re-sampling. The proposed method work pretty well empirically compared to various baselines.  Aside from the novel scoring function, since the noisy labeled data might be accidentally considered as minority group, the paper introduces a de-noising step to tackle the potential noise in the dataset. The de-noising step further improves the performances.

The paper proposes a new bias-mitigation technique. The main hypothesis of the paper is that there are differences in gradients for "biased" samples (or samples that are "rare") compared to majority patterns in the training data. Using this, the paper devises a rejection sampling method that tries to balance samples in a minibatch. However, a sample with a noisy label can appear to be a "biased" sample (with a correct label) which can affect the proposed method (as well as many other bias mitigation methods). To this end, the paper also proposes a denoising module that successfully eliminates the effects of noisy labels on the debiasing algorithm proposed.

This paper proposes a gradient-based resampling scheme for de-biasing. The authors construct two types of scores that leverage gradients of the biased model: the magnitude and the direction of the gradients. In addition, to mitigate the side effects from the noisy labels, the authors also propose a de-noising module, which can be easily applied to any de-biasing algorithms. Extensive experimentation validates the proposed method against other state-of-the-art methods. 

The paper proposes a de-biasing method for neural network without requiring explicit annotations of biases. The authors propose a resampling-based approach that automatically detects underrepresented samples based on the direction and magnitude of their loss gradients, then balances the dataset by rejection sampling. The paper further proposes a denoising module to identify and filter out noisy samples under the presence of label noise. The gradient-based de-biasing method is empirically shown to outperform prior work on both controlled synthetic and real-world datasets.