Fighting Fire with Fire: Contrastive Debiasing without Bias-free Data via Generative Bias-transformation
Authors: Yeonsung Jung
Hajin Shim
June Yong Yang
Eunho Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: debiasing
contrastive learning
image-to-image translation

Abstract: Despite their remarkable ability to generalize with over-capacity networks, deep neural networks often abuse bias instead of using the actual task-related information for discriminative tasks. Since such shortcuts are only effective within the collected dataset, the resulting biased model underperforms on real-world inputs. To counteract the influence of bias, existing methods either exploit auxiliary information which is rarely obtainable in practice, or sift bias-free samples to exploit them for debiasing. However, such presumptions about the availability of the auxiliary information or bias-free samples are not always guaranteed and the existing methods could break down due to the unmet presumptions. In this paper, we propose Contrastive Debiasing via Generative Bias-transformation (CDvG) which is capable of operating without exploiting bias labels and bias-free samples explicitly. Motivated by our observation that not only discriminative models but also image translation models tend to focus on the easy-to-learn bias, CDvG employs a image translation model to transform the bias to another mode of bias while preserving task-relevant information. Through contrastive learning, we set transformed biased views against another, learning bias-invariant representations. Especially, as the bias has a stronger correlation or is easier to perceive compared to the signal, the translation model is more likely to be a bias translation model, resulting in better debiasing effect. Experimental results demonstrate that CDvG outperforms the state-of-the-arts, especially when bias-free samples are extremely scarce.

Under review as a conference paper at ICLR 2023
FIGHTING FIRE WITH FIRE: CONTRASTIVE DEBIASING WITHOUT BIAS-FREE DATA VIA GENERATIVE BIAS- TRANSFORMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Despite their remarkable ability to generalize with over-capacity networks, deep neural networks often abuse bias instead of using the actual task-related information for discriminative tasks. Since such shortcuts are only effective within the collected dataset, the resulting biased model underperforms on real-world inputs. To coun- teract the influence of bias, existing methods either exploit auxiliary information which is rarely obtainable in practice sift handful bias-free samples emphasize them for debiasing. However, such presumptions are not always guaranteed, and the existing methods could break down due to the unmet presumptions. In this pa- per, we propose Contrastive Debiasing via Generative Bias-transformation (CDvG) which is capable of operating without exploiting bias labels and bias-free sam- ples explicitly. Motivated by our observation that not only discriminative models but also image translation models tend focus on the easy-to-learn bias, CDvG employs image translation model to transform the bias to another mode of bias while preserving task-relevant information. Through contrastive learning, we set transformed biased views against another, learning bias- inaa-invaiiant representations. Especially, as the bias has a stronger correlation or is easier to perceive compared to the signal, the translation model is more likely to be bias translation model, resulting in better debiasing effect. Experimental results demonstrate that CDvG outperforms the sfute-ofthh-aeee especially when bias-fre samples extremely scarce.
INTRODUCTION
Recent advances in deep learning have showcased that deep neural networks (DNN) are capable of reaching state-of oh-the-art performance in various fields of machine learning, such as computer vi- sion (He et al., 2015), natural language processing (Brown et al., 2020), reinforcement learning (Mnih et al., 2016) and more. However, it is also known that the over-parameterizze nature of DNNs not only exposes them to general overfitting but also renders them susceptible to biases present in collected datasets (Torralba & Efros, 2011) which are detrimental to the generalizability. In supervised learning. neural networks tend to prefer shortcut solutions based on biases rather than real signal (Zhu et al., 2017b; Li et al., 2018). Since spurious correlations do not provide task-related information, DNNs that use these biases will ultimately fail on future data. For instance, classifier trained to identify car racing images using a dataset dominated by track will exploit the track road information. However, the classifier will fail to exhibit the same performance on images of off-road rallies. To this end, debiasing is imperative in utilizing DNNs for real-world applications.
tautological solution to the bias problem to construct a bias-free dataset from the start. However, curating dataset devoid of all bias is extremely costly at best, and generally infeasible. A more practical attempt at neutralizing dataset bias is to fortify a dataset with explicit supervision with regards the bias (Kim al., 2019a; Sagawa al., 2019). However, additional expenditure of human labor in procuring such information cannot be avoided, which renders the option less appealing.
In most cases where such explicit supervision for bias is absent, the following two lines of works are recently proposed. One line of works mitigates the influence of bias by leveraging the bias type (e.g. texture) (Bahng et al., 2020; Geirhos et al., 2019; Wang et al., 2019; Hong & Yang, 2021)

---

Under review as a conference paper at ICLR 2023
Training StarGAN with biased datasel
Generator
Discriminator
Biased sample
Lead the discriminator be blased
4
Bias-transformed
Domain classificaaionn
Real/Fake
Target class
y'-y
2) Lead the generator transiate blas features
Whole framework
Pretrained G
Encoder E
t2-J
h
z'
Bias-transformed
f fCL
x'
h
z
4
y' y'~y
3) Maximize agreement by Contrative loss
1) Blas-transformed Biased sample View Generation
2) Random Augmentation
x
Figure I: Illustration of our Contrastive Debiasing via Generative Bias Bias-transformation (CDvG).
to design bias-oriented auxiliary models or to augment texture- petturrre samples. However, such prior knowledge of bias is by means guaranteed, and even with such information, designing bias-oriented architectures is not always straightforward. Instead, another line of works leverages an empirical prior knowledge that malignant biases usually learned faster than task-relevant features (Li & Vasconcelos, 2019; Nam et al., 2020; Lee et al., 2021; Kim et al., 2021). However, these methods tend to break down in regimes where bias-free samples extremely scarce absent (See Section 4.
To tackle these shortcomings, we develop general debiasing method capable of operating even in regimes devoid of bias-free data, thus not reliant on any presumptions regarding the bjases. In this paper, we propose Contrastive Debiasing via Generative Bias-transformation (CDvG), which contrasts the bias modes within the dataset against each other to attenuate the influence of bias while effectively learning the task-relevant information contained in all samples. Through preliminary experiments, we find that image-to-image translation models favor learning malignant biases over task-relevant signals, as discriminative models are known to do so (See Section 3.2). Motivated by the observation, we train biased image translation model without bias labels that learns the bias distribution over the signal, obtaining capabilities to transform the bias feature of given input to another bias. Using the learned bias-transie translatinn model, we synthesize novel views with altered bias features. Fighting fire with fire, we pit one bias type against another via contrastive learning. By maximizing agreement between the views with different biases, the model is encouraged to learn bias-invariant representations. Unlike existing methods, CDvG does not require explicit supervision, domain knowledge, other meta-knowledge- the existence of bias-free samples.
Our contributions are three-fold:
We experimentally observe that certain image translation models are also prone to consider conspicuous but incidental information such as background texture first rather than task-related information, as discriminative models do (Sec. 3).
We propose Contrastive Debiasing via Generative Bias- transformation (CDvG) with gen- erative view transformation that transforms bias factors of input images (Sec. 4). CDvG does not presume the existence of bias labels, bias type information, even the existence
2

---

Under review as a conference paper at ICLR 2023
of bias- free samples which is more malignant. Also, CDvG can be integrated with other debiasing methods in a plug-and-pla manner.
With experiments on synthetic and real-world datasets, we demonstrate that CDvG outper- forms baselines in a variety of bias situations (Sec. 5). We show that CDvG is particularly effective compared existing methods when bias-free samples are extremely scarce or absent.
2 RELATED WORKS
In real-world datasets across diverse domains, there exist various kinds of incidental biases strongly but spuriously correlated with the task-related information. However, when the bias factors are more noticeable and easier to learn than the task-related signals, DNNs tend to lean on such biases (Nam et al., 2020), causing failures in generalization. To counter this effect, several lines of works were developed.
Explicit bias supervision can be used screen or mitigate the influence of bias. Kim et al. (2019a) uses known explicit bias supervision to train an auxiliary network that helps in reducing the influence of bias. Sagawa et al. (2019) use the bias supervision to group data samples for grouped distributionally robust optimization. Kim et al. (2019b) address the problem by minimizing the mutual information not to learn biased instances by using bias supervision. Hong & Yang (2021) propose bias-contrastive loss and bias-balanced regression that encourages the model to pull together the samples in the same class with different bias features, with balancing target-bias distribution.
Domain knowledge of biases. When acquiring bias supervision is impractical, we can leverage domain-specific knowledge about bias type. For example, it was shown that ImageNet-trained classifiers exploit texture information in the image rather than information contained in the object of interest (.Gatys et al., 2017; Brendel & Bethge, 2019). Utilizing this fact, Geirhos et al. (2019) construct an augmented dataset by applying various textures to the images for texture debiasing. On the other hand, the following works exploit an auxiliary model which is carefully designed to capture biases over signals. Wang al. (2019) trains eexture-debiase classifter by projecting classifier representations to a subspace orthogonal to the previously learned biased space. Bahng al. (2020) lead the model representation from the biased auxiliary model using Hilbert-Schmin independence criterion. Hong & Yang (2021) further propose soft version of coatrasnte loss for the case where the type of bias is available. However, these methods are grounded on obtaining such knowledge which is often costly, or even impossible. addition, designing bias-oriented auxiliary model may not be so intuitive depending on the type of bias.
be statistically independent the biased one produced
Without domain knowledge. To combat bias without any side information, Li & Vasconcelos (2019) introduced an alternating minimization scheme between bias identification and sample reweighting. Using the fact that biases are usually learned faster than salient task-relevant features, Nam et al. (2020) train an auxiliary model slightly ahead in terms of training iterations using generalized cross entropy (Zhang& & Sabuncu, 2018) to absorb fast- fast-learned bias, and assign more weight to the bias-free samples for the debiased follower. Seo et al. (2022) cluster the training samples into the K groups based on the latent feature of the Empirical Risk Minimization(ERe) each sample in batch according to the assigned groups Lee al. (2021) proposed feature-level data augmentation that disentangles bias features by using biased auxiliary model obtained by following Nam al. (2020) and swaps latent bias features within the mini-batch. Another augmentation- based debiasing method, BiaSwap (Kim et al., 2021), employs task-related features in abundant bias- blas-el samples by synthesizing new image that takes the bias-irrelevant core features from the biased sample and the bias attribute from the free-free sample. However, all these methods presume that bias-free samples do exist in sufficient quantities and can be distinguished, which cannot always be guaranteed.
model and reweight the loss of
Contrastive learning is a self-supervised learning method proven to learn representations substan- tially beneficial to numerous downstream tasks, achieving state-of-the-art performance (Chen et al., 2020a;b; Grill et al., 2020; Chen & He, 2020; Khosla et al., 2020). A representative work (Chen et al., 2020a) defines the contrastive prediction task in two steps: a) augmenting two views from the same image by strong data augmentations (Bachman et al., 2019; Hénaff et al., 2019; Krizhevsky et al., 2017) and b) maximizing agreement between the augmented views on the latent space by
3

---

Under review as a conference paper at ICLR 2023
100
100
80
Bias-aligned
80
80
Blas-free
Acc.
60
Acc.
40
40
20
20
1000
2000
3000
4000
5000
%
5000
10000
15000
20000
5000
10000 15800
20000
lteration
Iteration
Iteration
(a) Colored MNIST (0.5%)
(b) Corrupted CIFAR10 (0.5%)
(c) BFFHQ (0.5%)
Figure Accuracy evaluated on the unbiased datasets during StarGAN training.
employing the contrastive loss (van den Oord al., 2018) capture core features of the image. In the process of maximizing the agreement between the augmented views, the encoder discards the deviating features between the views and learns transformation-invarie example, using color distortion methods such color jitter and gray scale in the augmentation step encourages the encoder to learn color-invariant representations.
representations. For
IMAGE TRANSLATION MODELS FOR GENERATING BIAS-FREE SAMPLES
Previous approaches addressing the bias problem is to either deploy manpower or algorithms to salvage a sufficient amount of bias-free samples from the contaminated dataset. However, under cireumstances where bias-free samples are almost nonexistent, there are not enough bias-free samples to begin with. Instead of salvaging, we first opt to synthesize bias-free samples using image translation models. In other words, we aim to translate bias-aligned samples to bias samples by changing a given bias factor to another.
In this section, we first define our setup in Section 3.1 and describe our observations of how translation models behave on biased datasets, which is key part of our algorithm, in Section 3.2.
3.1 SETUP
To formally define our target task, we introduce the following random variables: input image X, target label Y, signal S, bias B and other attributes O comprising input X (S, B, O). Here, B is feature unrelated to Y, that is, Y and B are independent given S. We further define the random variable for bias label YB which hidden.
Ideally, the image classification model parameterized by 0 predicts the label based on signal as Po(Y B, = (Y However, when the training data consists of highly but spuriously correlated bias and target, i.e., H(Y**IB**) ~ 0, predicting Y based on B is also one of the possible solutions that can be deemed effective in the training phase. In this paper, when the sample bias-aligned and the opposite case is called bias-free. We tackle the case where B is easier to perceive than S, so that the spurious correlation between B and Y is malignant in that the model preferentially takes B as a clue to predict Y over (Nam et al., 2020). This is obviously an unintended consequence and impairs generalizability due the discrepancy of the bias-target correlation Y between the training and test phases.
(s, b, o) consist of a correlated signal and bias (i.e. high (bls)), is called
3.2 BEHAVIOR OF IMAGE TRANSLATION MODELS UNDER BIAS
We first examine whether image translation models are capable of generating bias-free samples. Since discriminative models are known to be susceptible to bias, it not far-fetched to first enquire whether generative models also carry the same frailty. For instance, one might intuitively suspect that the family of Generative Adversarial Networks are susceptible to bias, as their composition by nature involves a discriminator, which is bias bias-pregnable To this end, we investigate the case of a particular member of the GAN family, namely, image translation models.
Image-to tange-to-image translation methods (Isola et al., 2017; Zhu et al., 2017a; Choi et al., 2017) render image from source domain characteristics of the target domain y and combine them with the input image. However, when the dataset is biased, the translation model interacting with the biased domain classifier is also prone
target domain y'. Ideally, they find out the representative
4

---

Under review as a conference paper at ICLR 2023
translate biases rather than task-related domain features to satisfy the domain classifier (See Figure 1). For example, CycleGAN (Zhu et al., 2017a), a representative milestone of image translation, presented number of typical failure modes that when the source domain is apple and the target domain is an orange, a transformed image is not the orange counterpart of the input, but an apple with the color and texture of orange. This implies that the model perceives color and texture rather than shape as the representative traits for the target domain, even without using biased dataset. With this in consideration, is plausible to speculate that this phenomenon would be exacerbated when handling highly biased datasets. To verify this, we examine the behavior of StarGAN (Choi et al., 2017) on biased datasets.
3.2.1 DISCRIMINATORS OF TRANSLATION MODELS ARE ALSO PRONE TO BE BIASED.
StarGAN introduces an auxiliary domain classifier Dats on top of discriminator D enable translation between multiple domains. The domain classifier Dais, trained on the real image with domain labels, learns to classify images with representative traits of the domains by optimizing the domain classification loss of real images C.s = [(a,,))D[- log Dets a (y]=)) However, on the biased dataset, we observed that Dat has absorbed bias attributes as representative traits of the domains during training phase. To quantitatively evaluate whether Data is biased, we measure the classification loss on the unbiased dataset with DAta while training StarGAN on the biased dataset Colored MNIST, Corrupted CIFARIO, and BFFHQ (Figure 4) which have color bias, texture bias, and gender bias, respectively. Figure 2 shows the classification losses of bias-aligned and bias-free samples, respectively. We observed that the accuracy of bias-aligned samples increases to near 100%, however, the accuracy of bias-free samples is low depending on how malignant the bias of each data set is. Therefore, we conclude that Dets utilizes the biases rather than the task-related features.
3.2.2 A BIASED DOMAIN CLASSIFIER INDUCES BIASED TRANSLATION GENERATOR
By optimizing the domain classification loss C/s = D. (y log (v G(a, y/) of translated fake images G(x, v/), biased Dete induces the generator G translate the bias attribute of image x into the other bias correlated to the randomly sampled target domain y' rather than the task-relevant signal. As a result, G becomes bias- transsaans.tto.. This phenomenon becomes more noticeable as the bias is malignant for Dat. that is, the more scarce the bias- free samples are or the easier the bias is to perceive. To quantitatively evaluate whether the translation model truly favor learning biases over task-relevant signals, we measure the classification loss of translated images x' = GG y/) with the bias classifier C3 and the signal classifier Cs while training StarGAN on the biased dataset Colored MNIST and BFFHQ As alternatives of oracles, the classifiers C3 and C's are trained with the bias label yn and the true class label y. respectively, on the unbiased Colored (erreleeeeCCCCCCeCeeooo v')
MNIST and FFHQ. The bias and signal loss are defined and I
(z'), s/)], respectively.
As expected, Figure shows that the bias is more favorable translation model. For Colored MNIST, the bias loss quickly de- clines to zero, but the signal loss increases rapidly since G concentrates on the color biases rather than digits. For BFFHQ, both signal and bias loss are decreased due to the less malignant bias (see vanilla perfor- mance in Table 1), however, G still places more emphasis on the bias.
100
80
CE Loss
60
CE Loss
40
Signal Blas
20
%
100 700
300
500
%
200
400
600
860
1000
Iteration
(a) CalaredMNIST (5%)
(b) BFFHQ (0.5%)
Figure 3: Classification loss evaluated on the bias-translated images during StarGAN training.
Next, we also present the qualitative results of the biased translator G in Figure 4. The leftmost column of each subfigure is of the original x and the other columns are hias-translated results of each class. Interestingly, we found that the translated results retain their contents to some extent and color, texture, background (e.g. rock wall for climbing, water for fishing), and gender characteristics (e.g. makeup for female, beard for male). which are the respective bias features of the datasets, are altered. is worth mentioning that we can take the advantage of the image translation model to identify bias attributes by distinguishing unintentionally translated features from the translated images. For example, when StarGAN has trained on CclebA with blond hair attribute, it not only turns a male photo into blond hair, but also applies makeup which reveals gender bias.
5

---

Under review as a conference paper at ICLR 2023
8
8
8
8
8
8
8
8
8
8
8
0
0
0
0
0
0
0
0
0
0
6
6
8
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
(a) Colored MNIST
(b) Corrupted CIFAR-10
(c) BAR
(d) BFFHQ
Figure 4: Bias-transformed images by our generative biastrrantoormation on synthetic and real-world biased datasets. The leftmost column of each subfigure contains the original images and each column is the transformed images of each target domain. The resulting images show the translated bias attributes such as color (Colored MNIST), texture (Corrupted CIFAR-1O), background (BAR), and gender (BFFHQ) respectivvlly For details about the datasets, we refer the reader Section A in Supplementary.
Although generated images of BFTHQ in Figure 4d look quite different from the real images to humans, the core contents are still discernible while the gender-specific features emerge as pink lips and eye makeup in the second column and thick eyebrow and beard for the third column. Also, note that the biased classifier judges that the ttas-transfommed images well transformed into the target label with high confidence. These observations verify that there is a chance that image translation models trained on biased datasets be employed to synthesize bias-free samples for debiasing.
4 METHOD
In this section, we empirically show that the previous unsupervised debiasing methods break down when the presumption about the availability of bias-free samples does not hold in Section 4.1. Acknowledging this limitation, propose leveraging our empirical findings (Section 3.2) that image translation models are prone to be biased.
debiasing method called CDvG in Section 4.2
4.1 LEARNING WITHOUT BIAS-FREE SAMPLES
Although recent debiasing methods (Bahng et al., 2020; Hong & Yang, 2021; Nam et al., 2020; Lee et al., 2021; Kim et al., 2021) work as intended when they can exploit the bias-free samples, they neglect the case where such bias-free samples are extremely scarce or absent. Thus, in regimes where a presumption that enough number of bias-free samples exist does not hold, their behaviors are practically unknown, and are likely to break down. More specifically, in Table 1, we empirically demonstrate that previous methods fail to effectively debias the model (showing low performance, almost comparable to Vanilla) when the proportion of bias-free samples is 0%. Also, we further analyse the sample reweighting scheme (Nam et al., 2020; Lee et al., 2021) whose key idea is to balance bias-free and bias-aligned samples. To this end, we compare the total weights of bias-free samples and bias-aligned samples the ratio of bias-free samples decreases. We observe that, as the ratio of bias-free samples decreases, the debiasing effect decreases because the bias-free samples are not sufficiently emphasized by the baselines. The detailed descriptions and results are deferred to Section C due to the space constraint. Based on this observation, we consider a more general task of learning-under-l that handles the situations where there are no to only few bias-free samples as well as the cases where such samples are sufficiently provided.
4.2 CONTRASTIVE DEBIASING WITHOUT BIAS-FREE DATA VIA BINSFORANSFORMED VIEWS
Motivated by the findings of Section 3.2, we propose a novel debiasing method called Contrastive Debiasing via Generative Bras-rransooomaoion (CDvG) that employs the biased translation model

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Contrastive Debiasing via Generative Bias-transformation (CDvG)
Input: Encoder E, Projection head H. Classifier C, Biased generator G, Augmentation family T Data: Training set "D = (1. u)) C x y
for minibatch { {r yk) yz)) k=1 )= do
# Generate bras-transformed view
ye ~ (ategorical())))
T = G(
# Augmentation operator t1 ~ TT,z ~T; Tk. Hyy = (xx), t2(x))
Update E, C, H to minimize
Ex(Con(C(B(T))) un) +Lo
+ +Cc +X (E,
end for
to transform an image to have different biases corresponding to other classes and integrate it with the contrastive learning framework. By comparing the diverse bias-translated views generated with the transformation function, the contrastive loss encourages to learn bias-invariant representations without distinguishing bias-free data explicitly. The whole process is provided in the following paragraphs and summarized in Algorithm 1.
First, we train StarGAN (Choi et al., 2017) on training dataset, which is biased in our problem setting, to obtain the Stas-transformation generator G. We observe in Section 3.2 that the image translation model learns biases as the main characteristics of the domain rather than task-related features. Although other multi-class domain translation models can be used, we adopt StarGAN, the representative image-to-image translation model.
worth noting that we do not need bias-free samples at hand in this process. With G, we translate an input image with target label y' to obtain a bias- transformed view = G(z, y'), where y' is uniformly sampled for every iteration (Step in Figure 1, Line 3 in Alg, 1) generate views with diverse biases. Please note that bias-translation with G is applied only for training, not at the test time.
After the trasfrransformtiion step, additionally apply the random augmentation operators and t2 to the original and the bias transformedd images respectively as 2 = f1 H' = t²(x"), where t and t2 sampled from the same augmentation family T (Step 2 in Figure 1, Line 5 in Alg. I). According to Chen et al. (Chen et al., 2020a), strong augmentations are essential to the performance of the contrastive learning framework as they prevent the encoder from easily finding clue about the fact that two views come from the same image. By following (Chen et al., 2020a). T is composed of the following sequential augmentations: random resized cropping and random horizontal flipping. Please note that we did not adopt color distortion methods because most biases are related to color.
Finally, the resulting learning objective is given by the combination of the cross entropy loss and the contrastive loss (Step 3 in Figure 1, Line 7 in Alg. 1). First, the encoder E and the following classifier C are optimized to minimize the cross entropy loss LeE which applied both 7 and ;':
min LCE ( + LCE(C(B(()))))) E,C
Also, we train the encoder E and the projection head H to minimize the contrastive loss Lcr. to tie the original view ă and the bias-transformed view H' as LeI H) = Σ e(2k: - 1, 2k) + f(2k, 2k 1). The loss j(i.j) for positive pair (i,j) is defined
exp(simas/T)
E(i,j) -log Σ l expcexp(simex/7)
Where simi,j 2, =,, ==]=,[[[=<[||==]]]] is the cosine similarity, and 2; = HEE())) is a projected representation of ã; with the base encoder E followed by the projection head H. By tying the original view è and the thas-transformed view 2' as a positive pair, the encoder attempts to attenuate biases while emphasizing the true signals shared by the views via maximizing the mutual information between their latent representations. Please note that we adopt 1.0 for the coefficient of contrastive loss X by default.
7

---

Under review as a conference paper at ICLR 2023
Table 1: The average and the standard deviation of accuracy over runs. exploits auxiliary information of the bias and X does not.
indicates that the model
Vanilla
ReBias
L.F
EnD
BiaSwap
BPA
CDvG
CDvG+LIFF
Dataset (a
x
X
X
X
X
X
0.0
14.64.0.5a +2.53
13.16=1.8
11.65
10.08
96. 16.064..4
96.48+0.19
Colored
0.5
39.1221..1
70.4711.8
as
3..54
13.19
85.76
54.52
20.73
96.20.0.12
1.0
56.0221.81
87.4020.1s
79.83
25.4
83.74
22.63
13.13
96.45.00119
MNIST
2.0
69.32++.22
92.91
82.66
80.08
85,29
78.52+0.00
96,
96.90+0.2a
96.97=00117
5.0
83.93.0..0
96.96.20.0 os
13.30
85.0021.17
90.85
25.30
96.73
96.9820.0s
0.0
16.05
21.93 ST
15.8820.45
18.76::0.8
17
28,44
29.24:10.47
0.5
20.87...34
22.27:0041
25.58
28.62:1.74
29.11
25.50
0..5
39.07:10.27
Computte
1.0
24.05
25.72+0.20 20
30.68
32.3 +:000
32.54
26.86=0.6
33.25+n.20
43.8 +.0.30
CIFAR-10
2.0
29.47:0.20
31.6610.45
37.96
36.51
35,25
27.47
47.45 16.1620.23
5.0
41.1220.16
43.4320.41
48.49
46.41
41.62
34.29 22.20
52.31 42.75
10.13
BFFHQ
0.0
37.9320.96
43.47:2074
1.0
38. 13:213
48. 48.20
140
10.80
49.60
0.5
52.40
56.80
58.0
=0.07
54.33+
51.40=2.08
54.80
62.20
EXPERIMENTS
To validate the effectiveness of CDvG compared to recent debiasing methods, we conduct image classification experiments on standard benchmark datasets for debiasing. We first present the settings for our experiment including datasets and baselines (Section 5.1) then report the comparisons of our method and baselines regarding their debiasing performance (Section 5.2). In addition, we perform ablation studies to demonstrate that each component of CDvG contributes to respective performance improvements (Section 5.3). Also, provide implementation details in Section B.
5.1 EXPERIMENTAL SETTINGS
Dataset. We experiment (Colored MNIST, Corrupted CIFAR-10) and (BFFHQ, ImageNet-9 (IN-9), Waterbirds) which are synthetic datasets injected with synthetic biases and real-world datasets with natural biases, respectively. We provide detailed description of datasets in Section A. By setting the proportion of bias-free samples to (0%. 0.5%, 1 1%. 2%, 5%,, we evaluate the performance considering the highly biased setting and, moreover, the hardest case where bias-free samples are absent. The datasets with 0% bias-free samples are constructed by excluding bias-free samples from the datasetss with the ratio 0.5%.
Baselines. To benchmark the performance of CDvG, we compare CDvG with diverse categories of debiasing methods. We compared with HEX (Wang et al., 2019) and Rebias (Bahng et al., 2020) which leaveraging the domain knowledge about bias and LIF (Nam et al., 2020) which deliberately trains a auxiliary biased classifier to distinguish and utilize bias-free samples. Also, DisEnt (Lee et al., 2021) and BiaSwap (Kim et al., 2021), which based on the bias-free sample selection scheme of (Nam et al., 2020), try to combine the other biases with the instance. For BiaSwap, we borrow the available performance directly from the papers and remain blank for the others because the official code of BiaSwap is unavailable. Also, we compared with BPA (Seo et al., 2022) and EIIL (Creager al., 2021) which are based on ERM model.
5.2 RESULTS
We report the results on the standard benchmark datasets to validate the effectiveness of CDvG for debiasing when bias label is not available. We observed that the CDvG outperforms the baselines especially when the bias-free samples are scarce(0.5%) ()) or absent(0%))
we further proposed to combine CDvG and LfF ("CDyG+LfF' in Table 1) to leverage both bias- aligned and bias-free samples which is consistently superior to the baselines with a low standard deviation in Table 1. In more details, in the training process, we trained CDvG with concurrently applying LfF's sample-wise weighting to CE loss for both original images and bias- transsaaddd images. Therefore, 'CFvG+LfF' allows to utilize both bias bias-aligned and fias-free samples to debias by translating bias- aligned samples into bias-fre samples (CDvG) and giving more weights to bias-free samples (L.fF). Please note that CDvG can be integrated with other debiasing methods in a plug-and-
8

---

Under review as a conference paper at ICLR 2023
Table 2: ImageNet-9 dataset
Dataset
Test type
Vanilla
RUBi
LfF
Rebias
CDvG+Lff
Biased
94.0=a.1
93.920.2
91.22o.1
94.020.2
95.220.
IN-9
Unbiased
92.7=02
92.520.2
89.629.3
92.720.2
94.520.1
IN-a
30.5
31.020.2
29. 29.420.8
30.5
34.620.1
Table 3: Waterbirds dataset. The average of accuracy over runs. We borrow the performance on ResNet- 18 and ResNet-50 directly from (Seo et al., 2022) and (Creager et al., 2021), respectively.
ResNet--18
ResNet-50
Dataset
Test type
Vanilla
LfF
BPA
CDvG+Lif
ERM
ElII.
CDvG+LIF
Unbiased
84.63
85.48
87.05
86,25
97.30
96.90
91.30
Waterbirds
Worst-group
62.39
68.02
71.39
74.92
60.30
78.70
84.80
play manner. In addition, we evaluate the performance when bias-free samples do not exist (ratio 0%) which is likely to occur in real-world settings but not considered in recent studies.
Synthetic datasets In Table I, we observe that the CDvG preforms well on Colored MNIST and Corrupted CIFAR-10 when the bias-free samples are scare or absent, i.e., when the biases are more malignant. CDvG+LIF, which an our integrated method, outperforms the baselines on overall ratio with large margin.
Real-world datasets We conducted experiments on BFFHQ, IN-9 and Waterbirds to evaluate the performance real-world setting. CDvG+LiF shows improved performance with small standard deviations on BFFHQ compared to state-of-the-art Also shows improved or comparable performance on IN-9 and Waterbirds in Table 2 and Table 3, respectively.
To sum up, our method can handle various types of synthetic and real-world biases and works well on wide range of bias ratios, especially when bias-free samples scarce and even absent, compared with state-of-the-art methods Overall consistent results empirically demonstrate that our method debias better than previous approaches that only focus bias-free samples.
5.3 ABLATION STUDY
We study the effectiveness of each component of CDvG. In Table 4, CDvG without bias- transformation generator (CDvG w/o G), which is adding augmentation operator T and contrastive loss to vanilla, shows degradation. It mainly comes from that augmented samples, which are still biased, accelerate the model becomes biased. By adopting our bias-transform generator G only for augmentation (CDvG w/o CL.), there 8.21 % 5.37% %. 4.07%, 2.22%, and 1.4 1.43% improvements for each ratio, respectively. shows that our generator successfully augments bias-tram taassrransormed images which work well for some extent. However, since bias-transform generator G is not always a perfect model that only translate biases, contrastive learning is needed. Our whole framework shows an obvious improvement which means that since bias- trasstrrasforeed images are not perfect, it is essential induce bias-in invariiiveeeeee and capture the true signal shared by the bias-transformar view and original view by contrastive loss. Based on the ablation results, we have shown that our whole framework effectively address the bias problem rather than a single component.
Table 4: Ablation study for taaasrransforeed view generation (w/o G) and for contrastive loss (w/o CL).
Dataset
Ratio %)
Vanilla
CDvG w/o G
CDvG w/o CL
CDvG
0.0
16.04go
18.29
24.25
28.44g0.1
0.5
20.87
20.75
26.24
31.50+osa
Comupted CIFAR-10
1.0
24.05
21.37
28. 12
33,25
2.0
29.47
24.50
31.69
35.16
5.0
22. 16
42.55
42.75

---

Under review as a conference paper at ICLR 2023
CONCLUSION
In this paper, we have proposed Contrastive Debiasing via Generative bias-transformation, general debiasing method that does require presumptions about the data such the existence of bias- free samples or any domain knowledge. Motivated by the observation that not only discriminative models but also image translation models tend to focus on bias, we utilize translation model as bias-tras muusttransformaionn function to generate diverse biased views. With contrastive learning, we compare the biased views to obtaín bias-invariant representations. Our method can be integrated with other debiasing methods in plug-and-pla manner and especially shows good synergy when integrating with models that focus on bias-free samples such as LfF. Our experimental results show that our method outperforms the current state-of-the-are in both synthetic and real-world biases, especially when bias-free samples are scarce or even absent.
One important future work direction, and current limitation, is the extension to datasets that bias-free data are the majority. Including CDvG, many recent debiasing approaches are focus on highly biased setting where bias-aligned are the dominant majority. Although such situations would not require debiasing techniques from the start, suggesting a robust method that work in general is an important area of ongoing research.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. CoRR, 00ss/996.000900, 2019. URL abs/1906.. 19066.000 00910.
Hyojin Bahng. Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 528-539. PMLR, 13-18 Jul 2020. URL http: htttt:::pppooceeeiiiio /ttp///ooooeennses///ee///. mmr. press/ httess/vvl//ae (a119/baang2Caa html.
Wieland Brendel and Matthias Bethge. Approximating CNNs with bag-of-local-feature models works surprisingly well on imagenet. In International Conference on Learning Representations, 2019. URL www.t:ttps/.omm wttp/pppeervveeeeeeeee wwt.eeewwwweeeeeeeeeeeeeeeeeeeee net/ /t/ofortteee Forum? S= AMW
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https: : htttps://proceeeee httpp/ppcceeennnnceeee wtwpeeeddiiin.....iieeeeeee neurips. c0/ htc/pappr/202///
457c0d6bfcbb 9 967 8bfb8aci 42 2aa4644aPapper .dd..
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning. volume 119 of Proceedings Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020a. URL http: /ttp/peoooddnns/////eee mlr.press/viiie sesess/vl190 chwn20jeennccooo. html.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CoRR,
abs/2011.10566, 2020. URL https: ptttpss//arxiv..oo .or.iioorr.....oooo..oo...ooo...
10566.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. CoRR, abs/2003.04297, 2020b. URL https:://rrrw..or////abt 04297.
Yunjey Choi, Min-Je Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. CoRR, abs/1711.00022) 2017. URL http://pp/. / aaaaiiiii.. org/ abs/ 20os/177 7117..1. 09020.
Elliot Creager, Jörn-Henri Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference Machine Learning, pp. 2189-2200. PMLR, 2021.
Lcon A Gatys, Alexander S Ecker, and Matthias Bethge. Texture and art with deep neural networks. Current Opinion in Neurobiology, 46:178-186. 2017. ISSN 0959-4388. doi: http://d....g/11...0016........m setience/artticle//l///////
URL. https: /tt:://wwww/./. sci enced rect com/ Computational Neuroscience.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR. OpenReview.net, 2019. URL http: //dbb//dblppp uni-tries de/áb/ /ttpr//rrrrrr///// html htw.hhtmeeeee.... 4eeerrosssreE
Jean-Bastien Grill, Florian Strub, Florent Altché. Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your latent: A new approach to self supfrviue learning. CoRR, abs/2006.07733. 2020. URL https:
/ //arxiv.orrg/bbbb
07733.
11

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image org abs /ataa/11110 1512. 03385.
1asoss recognition. CoRR, abs/1512.033853 2015. URL
Olivier J. Hénaff, Aravind Srinivas, Jeffrre De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aäron den Oord. Data-efficient image recognition with contrastive predictive coding. CoRR, abs/1905.09272., 2019, URL http: arxiv. org/ abs / /1905. 09272.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. CoRR, abs/1903. 12261, 2019. URL http: http://xxivv..// /bss/ 1903. 12261.
Youngkyu Hong and Eunho Yang. Unbiased classification through bias-contrastive and bias-balanced learning. Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https: /ttp///eepeeeeeeeee/eeee/eeeeeee nnt/ forum? (errumiid=22222Zqxxmn
Andrew Hyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattem recognition, pp. 1125-1134 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 4401-4410, 2019.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661-18677 2020.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not learn: Training deep neural networks with biased data. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019a.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not learn: Training deep neural networks with biased data, 2019b.
Eungyeup Kim, Jihyeon Lee, and Jaegul Choo. Biaswap: Removing dataset bias with bias-tailored swapping augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14992-15001, 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. Commun. ACM, 60(6):84-90, 2017. doi: 10.1 14//366538. URL
http://doi acm. org/10. 1145..161133333866
Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. Advances in Neural Information Processing Systems, 34:25123-25133, 2021.
Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9572-9581. 9581, 2019.
Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representa- tion bias. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference Learning Representations. ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: /pppreeeee.reeee. net/forum? d=Skq899cxxx
12

---

Under review as a conference paper at ICLR 2023
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1928-1937, New York, New York, USA, 20-22 Jun 2016. PMLR. URL http: //proceedings mir press / v48 1/088mmmm mmiiha16666 html.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: Training debiased classifier from biased classifier. arXiv preprint arXiv:2007 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bemsstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 2015.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.
arXiv preprint
2019.
Seonguk Seo, Joon- Young Lee, and Bohyung Han. Unsupervised learning of debiased representations with pseudo-attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 16742-16751, 2022.
A. Torralba and A. A. Efros. Unbiased look at dataset bias. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattem Recognition, CVPR '11, pp. 1521-1528, USA, 2011. IEEE Computer Society. ISBN 9781457703942.
doi: 10. 10.1109//VP.2011 201.559577.. URL
https: htt:://oii.ooo /ttg//oo..ooooo/////////oo/////. http:/1..111/////// CCVPR.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/ 0188807.3747 2018. URL http: / //tp/v/axvvvv///a///////a//aaaaa org/ abs 1377403788....
Haohan Wang, Zexue He, Zachary Lipton, and Eric P Xing. Learning robust representations by projecting superficial statistics out. arXiv preprint arXiv:1903.062555 2019.
Zhilu Zhang and Mert R Sabuneu. Generalized eross entropy loss for training deep neural networks with noisy labels. In 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. CoRR, abs/ 10593 2017a. URL http:// a arxiv. org/ abs 173.. 10593.
Zhuotun Zhu, Lingxi Xie, and Alan Yuille. Object recognition with and without objects. In Proceedings of the Twenty-Sixe International Joint Conference on Artificial Intelligence, IJCAI-17, pp. 16699-3614 2017b. doi: 10.24963/ijca..201777 URL https: /ttp://...////./////633 ijcai 2007/505.
13


Reviews:
[{"review_id": "qNmbxLdqRj", "reviewer": "Reviewer_F8oL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses image dataset bias by using GANs and contrastive learning techniques. By training an image-to-image translation model (StarGAN) on a biased dataset, the model learns the implicit biases on that dataset. This is used by the debiasing model to generate new samples with different types of biases. Then, a debiased classifier can be trained using contrastive learning on {original (and biased) sample} - {generated sample} pairs. The model is evaluated on 5 different image datasets and ablation studies are reported.", "summary_of_the_review": "The proposed method is simple and targets an important task. However, there are some concerns with respect to the limitations that should be clarified during the rebuttal. ", "strength_and_weaknesses": "Strength:\n1. The idea of the paper is simple and can be potentially applied to any dataset. Even if the biases are known, the GAN will supposedly uncover them, so no annotations on biased samples are required.\n2. Experiments are conducted on very diverse datasets, including synthetic and real-world images. \n3. It is very interesting that the proposed method outperforms previous methods by a large margin when no unbiased samples are available.\n\nWeaknesses\n1. Paper clarity should be improved. Details:\n- The paper is not very clear about the proposed method. I did not understand what the proposed method was until page 6 (Section 4.2). The paper would benefit from making the proposed model clearer from the beginning.\n- It is not clear to me if the image-to-image augmentations are also applied at test time.\n\n2. Looks like the method needs the existence of a very strong bias to be useful. The following points need clarification:\n- Why Table 1 only has results for low ratios of unbiased samples? Specially BFFHQ dataset goes up only to 0.5%, while results on other datasets are reported to be up to 5%.\n- It should be clarified whether the method works if the dataset bias is not very strong. What if only 50% of the samples are biased? It may be that StarGAN cannot capture this bias well, making the whole method fail. The paper should be clear and transparent about this point.\n- In Table 1, except when the ratio is very small, some results do not seem to be statistically significant.\n\n3. Given that this method needs to train a StarGAN model and generate various images per sample in the dataset, computation cost and time should be reported and compared against previous work.\n\n4. Other details:\n- Section 5.1 says the model is evaluated on BAR dataset, but results are not reported.\n- Why is the Waterbird dataset the only dataset not reported over 3 runs?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kUcUX3mqJP", "reviewer": "Reviewer_SxSC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a contrastive learning-based debiasing framework to train a unbias classifier under the assumption that there are NO unbias samples existing in the training data. To achieve this goal, the authors utilize a conditional StarGAN approach to ``adjust'' the images and augment the samples. The experiment demonstrates that the method can outperform the existing approaches when combing with LfF debiasing framework.", "summary_of_the_review": "Overall, the paper is well-written and the experiment is solid. But authors should address the comments I mentioned above.\n", "strength_and_weaknesses": "Strength\n1. the paper discusses an interesting debiasing case: when there is no unbias data among the data, how can we train an unbias classifier\n2. the experiment is very solid\n\nWeakness\n1. Detailed experiment setting is not described. For example, how to combine the proposed work with LfF?\n2. Missing reference.\n3. Missing detailed information about StarGAN, for example, how to train StarGAN on the bias data? What is the structure of StarGAN?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "h-wKGdzuzCl", "reviewer": "Reviewer_Xnts", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper uses image translation models to generate diverse biased views of images to debias models. For this, the paper observes that generative models find it easier to latch onto biases instead of the signal, which is consistent with the observation made by previous works for classification models. As such, when asked to translate, they change the bias features, while not altering the core features, thus providing views with different biases.", "summary_of_the_review": "I think the paper cleverly exploits the bias-prone nature of generative models to debias methods, which is an important contribution. \nWhile I am leaning toward acceptance, I cannot see this as a clear accept mainly due to the experimental limitations.\nI would also need confirmation if the authors have fixed Sec 5.3, and included references and the appendix.\n", "strength_and_weaknesses": "**Strengths (S):**\n\n[S1] The approach is neat and sensible. It converts the weakness of DNNs (that image translation models latch onto biases) to a strength (use that to generate diverse biases).\n\n[S2] The results show benefits over existing methods on the datasets tested. However, I have some concerns about the choice of datasets (see weaknesses section).\n\n[S3] The method functions well despite not having bias-free samples in the training set, which is an important result.\n\n**Weaknesses (W):**\n\nI think the experiments leave many open questions about the true generalization abilities of the system:\n\n[W1] The paper claims that it doesn’t require presumptions about the data, but it assumes that the biases are easier to exploit than the signal. What if they are of similar complexities to exploit/learn? Wouldn’t the generative model then alter the core features too and not just the bias?\n\n[W2] ColoredMNIST consists of single source of bias. But how would this method handle multiple types of biases e.g., BiasedMNIST [1]? Does the image translation model uncover all the spurious factors regardless of their ‘ease of exploitation’?\n\n[W3] Waterbirds, BAR, ImageNet-9 all test against foreground-background spurious correlations. Apart from completeness, do they add additional value? \n\n[W4] It is unclear how the method would tackle real world distributional shifts. Examples,  datasets from WILDs benchmark [2] or the different renditions of ImageNet (https://github.com/hendrycks/imagenet-r) could be used, which are not tested here.\n\n[W5] The original submission does not include references or the appendix which is referenced in the main text.\n\n[1] Shrestha, Robik, Kushal Kafle, and Christopher Kanan. \"OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypotheses.\" arXiv preprint arXiv:2204.02426 (2022).\n\n[2]  Koh, Pang Wei, et al. \"Wilds: A benchmark of in-the-wild distribution shifts.\" International Conference on Machine Learning. PMLR, 2021.\n\n[3] Hendrycks, Dan, et al. \"The many faces of robustness: A critical analysis of out-of-distribution generalization.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fuapIVp-oZO", "reviewer": "Reviewer_Fqi9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of a classifier being biased to nuisance factors (shortcuts). The proposed method uses image-to-image translation to synthesize different biases in the dataset to remove the bias without collecting bias-free dataset.", "summary_of_the_review": "This paper builds up upon somewhat true (true in some cases) intuition (bias in translation) and provides reasonable de-biasing technique. However, the presentation and the logical flow should be improved. Furthermore, the idea is incremental in that this paper works with image-to-image translation instead of style transfer in \"imagenet-pretrained cnns are biased toward texture\".", "strength_and_weaknesses": "Strengths\n\n(+) The translation model does not require bias supervision for training.\n\n(+) Using translated images with bias for reducing bias of the dataset is plausible.\n\n(+) Experiments cover many datasets over different classes (numbers, faces, objects, birds), modalities (images and videos) and biases (synthetic, real).\n\n(+) The proposed method can be used in any setting.\n\nWeaknesses\n\n(-) The bias of a discriminative model is not guaranteed to be in the same bias of translation model.\n\n(-) Outdated translation model (stargan 2017). Biased translation may not exist in recent translation models: stargan v2 2020, style-aware \ndiscriminator 2022. Is stargan 2017 chosen among other alternatives to produce biases? Please discuss the trade-off between existence \nof bias and image quality.\n\nEtc.\n\nHow does it work when there is no bias in the dataset?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject