Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of a classifier being biased to nuisance factors (shortcuts). The proposed method uses image-to-image translation to synthesize different biases in the dataset to remove the bias without collecting bias-free dataset.
strength_and_weaknesses: Strengths

(+) The translation model does not require bias supervision for training.

(+) Using translated images with bias for reducing bias of the dataset is plausible.

(+) Experiments cover many datasets over different classes (numbers, faces, objects, birds), modalities (images and videos) and biases (synthetic, real).

(+) The proposed method can be used in any setting.

Weaknesses

(-) The bias of a discriminative model is not guaranteed to be in the same bias of translation model.

(-) Outdated translation model (stargan 2017). Biased translation may not exist in recent translation models: stargan v2 2020, style-aware 
discriminator 2022. Is stargan 2017 chosen among other alternatives to produce biases? Please discuss the trade-off between existence 
of bias and image quality.

Etc.

How does it work when there is no bias in the dataset?

clarity,_quality,_novelty_and_reproducibility: Clarity

(-) Bird-eye view of the proposed framework is missing. Please refer to my summary.

(-) The sentences should be more compact.

(-) The sentences have many redundant descriptions. E.g., "auxiliary information which is rarely obtainable in practice" and "such presumptions about the availability of the auxiliary information or bias-free samples are not always guaranteed"

(-) Sentence flow can be improved. Especially in 3.2 before 3.2.1.

(-) Inconsistent statements:
- "We find that image-to-image translation models favor learning malignant biases over task-relevant signals."
- "We experimentally observe that certain image translation models are also prone to consider conspicuous but incidental information."

(-) It is difficult to guess the expected result in each column in Figure 3. I can see the 2nd and  3rd columns should be female and male. I cannot tell on other datasets.

(-) What do checks and crosses mean in Table 1?

Novelty / Originality

The logical flow is just the same as "Imagenet-pretrained cnns are biased toward texture". Instead of texture, this paper is tackling the biases that can be captured as the domain specific attributes by image-to-image translator.

Quality

This paper is technically sound except that the translators are not always biased.

summary_of_the_review: This paper builds up upon somewhat true (true in some cases) intuition (bias in translation) and provides reasonable de-biasing technique. However, the presentation and the logical flow should be improved. Furthermore, the idea is incremental in that this paper works with image-to-image translation instead of style transfer in "imagenet-pretrained cnns are biased toward texture".
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses image dataset bias by using GANs and contrastive learning techniques. By training an image-to-image translation model (StarGAN) on a biased dataset, the model learns the implicit biases on that dataset. This is used by the debiasing model to generate new samples with different types of biases. Then, a debiased classifier can be trained using contrastive learning on {original (and biased) sample} - {generated sample} pairs. The model is evaluated on 5 different image datasets and ablation studies are reported.
strength_and_weaknesses: Strength:
1. The idea of the paper is simple and can be potentially applied to any dataset. Even if the biases are known, the GAN will supposedly uncover them, so no annotations on biased samples are required.
2. Experiments are conducted on very diverse datasets, including synthetic and real-world images. 
3. It is very interesting that the proposed method outperforms previous methods by a large margin when no unbiased samples are available.

Weaknesses
1. Paper clarity should be improved. Details:
- The paper is not very clear about the proposed method. I did not understand what the proposed method was until page 6 (Section 4.2). The paper would benefit from making the proposed model clearer from the beginning.
- It is not clear to me if the image-to-image augmentations are also applied at test time.

2. Looks like the method needs the existence of a very strong bias to be useful. The following points need clarification:
- Why Table 1 only has results for low ratios of unbiased samples? Specially BFFHQ dataset goes up only to 0.5%, while results on other datasets are reported to be up to 5%.
- It should be clarified whether the method works if the dataset bias is not very strong. What if only 50% of the samples are biased? It may be that StarGAN cannot capture this bias well, making the whole method fail. The paper should be clear and transparent about this point.
- In Table 1, except when the ratio is very small, some results do not seem to be statistically significant.

3. Given that this method needs to train a StarGAN model and generate various images per sample in the dataset, computation cost and time should be reported and compared against previous work.

4. Other details:
- Section 5.1 says the model is evaluated on BAR dataset, but results are not reported.
- Why is the Waterbird dataset the only dataset not reported over 3 runs?

clarity,_quality,_novelty_and_reproducibility: - The paper should be improved in terms of clarity. The details about the proposed method do not appear until page 6.
- Quality, novelty, and reproducibility look reasonable. The method is simple, and it may be able to be reproduced with the details provided in the paper.
summary_of_the_review: The proposed method is simple and targets an important task. However, there are some concerns with respect to the limitations that should be clarified during the rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a contrastive learning-based debiasing framework to train a unbias classifier under the assumption that there are NO unbias samples existing in the training data. To achieve this goal, the authors utilize a conditional StarGAN approach to ``adjust'' the images and augment the samples. The experiment demonstrates that the method can outperform the existing approaches when combing with LfF debiasing framework.
strength_and_weaknesses: Strength
1. the paper discusses an interesting debiasing case: when there is no unbias data among the data, how can we train an unbias classifier
2. the experiment is very solid

Weakness
1. Detailed experiment setting is not described. For example, how to combine the proposed work with LfF?
2. Missing reference.
3. Missing detailed information about StarGAN, for example, how to train StarGAN on the bias data? What is the structure of StarGAN?

clarity,_quality,_novelty_and_reproducibility: Here are several comments:

1. Where is the reference? It seems the article missing the reference.
2. How you trained StarGAN? Did you train it on Color-MINIST data?
3. How to combine the proposed framework CDvG with LfF? Is the training embedding of CDvG used for LfF?
4. Lacking explanation about StarGAN. Please include a detailed explanation of StarGAN in the supplemental materials.


summary_of_the_review: Overall, the paper is well-written and the experiment is solid. But authors should address the comments I mentioned above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper uses image translation models to generate diverse biased views of images to debias models. For this, the paper observes that generative models find it easier to latch onto biases instead of the signal, which is consistent with the observation made by previous works for classification models. As such, when asked to translate, they change the bias features, while not altering the core features, thus providing views with different biases.
strength_and_weaknesses: **Strengths (S):**

[S1] The approach is neat and sensible. It converts the weakness of DNNs (that image translation models latch onto biases) to a strength (use that to generate diverse biases).

[S2] The results show benefits over existing methods on the datasets tested. However, I have some concerns about the choice of datasets (see weaknesses section).

[S3] The method functions well despite not having bias-free samples in the training set, which is an important result.

**Weaknesses (W):**

I think the experiments leave many open questions about the true generalization abilities of the system:

[W1] The paper claims that it doesn’t require presumptions about the data, but it assumes that the biases are easier to exploit than the signal. What if they are of similar complexities to exploit/learn? Wouldn’t the generative model then alter the core features too and not just the bias?

[W2] ColoredMNIST consists of single source of bias. But how would this method handle multiple types of biases e.g., BiasedMNIST [1]? Does the image translation model uncover all the spurious factors regardless of their ‘ease of exploitation’?

[W3] Waterbirds, BAR, ImageNet-9 all test against foreground-background spurious correlations. Apart from completeness, do they add additional value? 

[W4] It is unclear how the method would tackle real world distributional shifts. Examples,  datasets from WILDs benchmark [2] or the different renditions of ImageNet (https://github.com/hendrycks/imagenet-r) could be used, which are not tested here.

[W5] The original submission does not include references or the appendix which is referenced in the main text.

[1] Shrestha, Robik, Kushal Kafle, and Christopher Kanan. "OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypotheses." arXiv preprint arXiv:2204.02426 (2022).

[2]  Koh, Pang Wei, et al. "Wilds: A benchmark of in-the-wild distribution shifts." International Conference on Machine Learning. PMLR, 2021.

[3] Hendrycks, Dan, et al. "The many faces of robustness: A critical analysis of out-of-distribution generalization." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
clarity,_quality,_novelty_and_reproducibility: **Clarity:**
The paper is clearly written except for some sections e.g., Sec 5.3 (Ablation Study) which needs to be revised heavily due to usage of terms such as 

‘there is remarkable improvement’/ ‘obvious improvement’ – better just provide a range of gains and remove terms such as remarkable/obvious

‘bias-transform generator G is not always a perfect model’? → what is a ‘perfect G’? Did you mean to say, it may alter signal/core features too?

**Awkward writing:**
‘... address the bias problem not a single component … ‘

The subsection needs to be re-written, but this poorly written paragraph did not affect my score.

**Novelty:** As noted in the paper, the observation that DNNs find it easier to latch onto biases compared to the signal  (at least for the types of datasets studied in this line of work) is a known observation. However, the application to image translation models is novel.

Reproducibility: It lacks supplementary materials (appendix/code), so reproducibility is questionable.

summary_of_the_review: I think the paper cleverly exploits the bias-prone nature of generative models to debias methods, which is an important contribution. 
While I am leaning toward acceptance, I cannot see this as a clear accept mainly due to the experimental limitations.
I would also need confirmation if the authors have fixed Sec 5.3, and included references and the appendix.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper presents a method to debias image datasets. First, image-to-image translation is used to create multiple views of the data each with a different bias. Then, this multiview data is fed to a contrasative learner that learns a representation invariant to these biases. The method works without having access to labeled biases, instead using the natural biases that emerge during translation, which are assumed (and empirically verified) to overlap with the biases of the downstream classifier.

Strengths:
* Interesting to use the natural biases of the domain translator as a source of supervision for debiasing
* Good results across a range of datasets

Weaknesses:
* There are some strong assumptions that may limit the generality and real world use of the method: 1) assumption that the discriminative model and the translation model share the same biases, 2) assumption that the biases are easier to learn than the core features.
* Performance is weak when the bias is not strong (e.g., when only 50% of samples are biased)
* StarGAN is somewhat out of date and it's not clear if the same findings would hold for more recent translation approaches (the initial experiments on StarGANv2 are a good step)


justification_for_why_not_higher_score: Reviewers and the AC found the approach to be novel and interesting, but were unconvinced about the generality of the method and felt that the present paper is limited by strong assumptions. The paper could be improved either by weakening the assumptions, showing more thoroughly that they are valid assumptions that hold in real scenarios, or better articulating the limitations of the method in the writing.
Decision: Reject