Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new bias-mitigation technique. The main hypothesis of the paper is that there are differences in gradients for "biased" samples (or samples that are "rare") compared to majority patterns in the training data. Using this, the paper devises a rejection sampling method that tries to balance samples in a minibatch. However, a sample with a noisy label can appear to be a "biased" sample (with a correct label) which can affect the proposed method (as well as many other bias mitigation methods). To this end, the paper also proposes a denoising module that successfully eliminates the effects of noisy labels on the debiasing algorithm proposed.
main_review: There is a lot to like about this paper. I find the hypothesis about the "per sample" gradient can be used to identify "biased" samples interesting. The authors also correctly discuss the accompanying argument that this has close ties to label noise. The results also show a remarkable improvement in the commonly used datasets. However, I am increasingly wary of testing bias mitigation algorithms on the same toy settings, and their ability to actually quantify the degree to which these algorithms are actually working is under increasing suspicion. 

Strengths:

S1.  Great results: The results for varying levels of "bias" as well as the success of the proposed "denoising" setup is remarkable for the datasets tested. The results are vastly superior to other comparable methods in three different datasets and even better for extreme levels of bias (99.5%). 

S2. Interesting Hypothesis and approach: The paper proposes an interesting hypothesis about the differences between gradient magnitude and direction (as measured by its proximity to an "average" gradient direction for all samples) look different for biased as compared to "regular" data sample. This discussion would be incomplete without a discussion of its relationship with label noise which the authors rightly address. The proposed denoising module is shown to mitigate the effects of label bias on various debiasing algorithms including the proposed one. While I am not fully convinced of the hypothesis is universally valid, and whether the denoising module is truly capable of dealing with label noise per se, the empirical validation, at least in the datasets tested, is hard to refute in the context of debiasing.

Weaknesses:


W1. Unclear how bias model is implemented: In prior works, such as LearnedMixinH and Rubi for VQA, the bias-only model takes question-only model as a biased model which assumes that if an answer could be guessed only using the question, then that sample can be assumed to be biased. There isn't a straightforward parallel for this in color-MNIST. For color-MNIST, REPAIR proposes to train a classifier using only the RGB value of the input. It is unclear how 1) the biased model is trained for the proposed algorithm, 2) How were they trained for Rubi?

W2: Serious doubts about "scalability" and assumption about known bias:  There are various questions about the "scalability" of the proposed method. One common theme across these datasets is that they can be "learned" (at least the biased version) with a much smaller amount of data than is present in the training set. Hence, a rejection sampling-based method can work even when the minority set (m) diminishes. Next, the proposed algorithm falls under the category called variously as  "supervised", "known", or "explicit" bias mitigation algorithm. In other words, the algorithm assumes knowledge about which of the factors were biased so that a suitable "bias-only" model can be trained by leveraging only the "bias". Such methods can be unscalable to all but a single, well-defined type of bias. This could be addressed by demonstrating the algorithm's success in more difficult scenarios (https://arxiv.org/abs/2104.00170).
summary_of_the_review: Overall, I think the paper shows lots of properties of a solid paper. It has an interesting hypothesis, a solid experimental section, and some valuable insights with great results. Taken together, I find that it clearly surpasses the bar for acceptance. However, it still leaves a few things to be desired, which prevents me from giving a higher score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper propose to use the per-sample gradients from the biased classifier to tackle the dataset bias. Following prior work, this work adopts a three-step approach: 1) train a classifier on biased dataset 2) identify which samples are from the minority and which are not 3) train a classifier with re-sampling. The proposed per-sample gradient-based scores are used in step 2. The motivation is purely from the empirical observation that samples from the majority group tend to have similar gradient magnitude and direction. From the observation, they derive a scoring function for re-sampling. The proposed method work pretty well empirically compared to various baselines.

Aside from the novel scoring function, since the noisy labeled data might be accidentally considered as minority group, the paper introduces a de-noising step to tackle the potential noise in the dataset. The de-noising step further improves the performances.
main_review: The proposed method differs from prior work in the novel scoring function. However, the proposed scoring function is not well-motivated. Despite the success of better performances, it is hard to be convinced that the per-sample gradients are able to differentiate between the samples drawn from majority or minority group. The authors show only 4 examples in Color-MNIST in figure 4, which are not representative enough.

**Strength**
- Extensive experimental results across different setting and different baselines.
- Strong performance compared to the baselines

**Weakness**
- Lack of motivation and analysis on the connections between per-sample gradients and the majority/minority splits (in more complex datasets)

**Others**
- For table 3, I would be interested to see the accuracy of the minority as well since it help validate if the proposed method works under real-world correlation.
summary_of_the_review: The paper shows good empirical results on mitigating dataset bias by devising a novel scoring function for re-sampling. The scoring function is the key in this work, but is not well-motivated enough. The paper can gains a lot of values by providing more analysis on the connections between per-sample gradients and the majority/minority sets.


---

Updated on Dec. 6. 
Though the paper still lacks of theoretic motivations, it does provide convincing empirical findings. I raised my score from 5 to 6.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a gradient-based resampling scheme for de-biasing. The authors construct two types of scores that leverage gradients of the biased model: the magnitude and the direction of the gradients. In addition, to mitigate the side effects from the noisy labels, the authors also propose a de-noising module, which can be easily applied to any de-biasing algorithms. Extensive experimentation validates the proposed method against other state-of-the-art methods. 
main_review: The proposed idea of using gradient-based scoring for de-biasing seems novel and effective. I have the following concerns and questions. 

This work is based on the hypothesis that the “gradients have remarkable differences between samples generating prejudice and the others”. By observing that the label noise can degrade the performance of the proposed gradient-based model, they added the de-noising module to alleviate this issue. However, I’m not convinced if the noisy label is the only critical factor that can negatively affect the performance when using the proposed gradient-based scheme. I wonder what other factors (e.g., adversarial attacks) can degrade the performance and how to tackle them. Can you add more analysis and discussion on this? 

Figure 4 is drawn using the Colored MNIST. This result may be useful to illustrate the proposed hypothesis, but it may not be enough to show the generality because the dataset has a very simple structure and variation. I wonder how the plots for more natural datasets would look like, and if any theoretical analysis can be included.  

The model assumes that the biased samples make up the majority of the training set. The experiments are also based on this assumption, and the proportions of the unbiased (minority) samples in the experiments are 5% or less in all the experiments. I wonder how realistic this scenario is and how the proposed method performs on a dataset that contains a higher portion of unbiased (minority) training data (e.g., 10%, 20%, 30%,…). 

In Table 2, comparing the accuracies for the Minor group, which I think is more important in the performance comparison, the performance of the proposed model “with de-noising”, compared to the case without it, is improved at the 99%/10% setting, but not for the remaining cases of 99%/5% and 99%/0% (except for Watermarked MNIST, 99%/5%, showing very low accuracy). Therefore, this result is not sufficient to validate the proposed de-noising idea. More experiments and/or analyses should be provided. 

Minor comments:

- Figure 3: |m|/|D| -> |M|/|D|
- Page 6, Section 4.2: “From the harmonic-mean property has a higher value” (need to revise)
- In Table 2 - 99%/5% - Without de-noising – Major M: 
-- Colored MNIST: LfF performance is written as 00.00. Should it be 100.00?
-- Watermarked MNIST: the performance from “Ours” is 33.26, which is much lower than the other methods (over 90). Is it a typo or any reason for this?   
- In Table 1, the method names of “Proposed” and “Ours” are both used. It’s better to make it consistent. 
- Page 9, Section 6.2
-- ... an accuracy similar to that o the major case. -> … an accuracy similar to that of the major case.
-- de-basing -> de-biasing

summary_of_the_review: This paper present a novel idea for de-biasing and its performance is validated on limited scenarios. More validation and analysis of the used hypothesis and the model would strengthen the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a de-biasing method for neural network without requiring explicit annotations of biases. The authors propose a resampling-based approach that automatically detects underrepresented samples based on the direction and magnitude of their loss gradients, then balances the dataset by rejection sampling. The paper further proposes a denoising module to identify and filter out noisy samples under the presence of label noise. The gradient-based de-biasing method is empirically shown to outperform prior work on both controlled synthetic and real-world datasets.
main_review: ## Pros
- This work studies an important problem in machine learning. The proposed method has the benefit of not requiring supervision of biased attributes, which makes it potentially scalable to large-scale datasets with bias.
- Extensive experimental results are reported, where the proposed method showed superior performance to prior state-of-the-art on the both synthetic benchmarks and real-world datasets. It is particular interesting that all baseline methods are vulnerable to label noise and that the denoising module improves results universally.
- The writing of the paper is easy to follow. Implementation details are presented with full transparency which contributes to the reproducibility of the work.

## Cons / Questions
- The key assumption which motivates the proposed method, namely that minority samples have different gradient distributions than majority ones, deserves a more rigorous validation. For example, in addition to the aggregate measure in Figure 4b, it would be more convincing to compare histograms of per-sample gradient norms $||\nabla\theta_i||$ for both groups of training data, to ensure that the difference between the two groups is not the consequence of a few outliers.
- Despite its positive performance under synthetic settings, the significance of the proposed label denoising module is not validated on real-world datasets. In addition, the paper makes no reference to the literature on training with noisy labels (some recent work includes [A, B, C]). It is unclear how the proposed method compares to these approaches both gconceptually and in empirical results.
- Experimental results on real-world datasets are relatively weaker compared to synthetic ones. For example, the reimplemented LfF model reports significantly lower accuracy on BAR dataset (57.35% vs. 62.98%). Can the authors comment on factors that may contribute to this discrepancy? For ImageNet experiments, it would be cool if similar improvements can be observed on ImageNet-C [D] and Stylized-ImageNet [E] test sets. Finally, among the six baseline debiasing methods used in this work, only LfF and REPAIR are compared against using real-world datasets. It would be helpful if explanations are provided on the choice of baselines for these tasks.
- Results in Table 1 are presented in a slightly misleading way, as it seems to suggest that the proposed method achieves highest accuracy on both majority and minority groups. I believe the clarity can be improved by additional columns for the average of major and minor accuracies and making bold only the best numbers in each column.

## Minor comments
- Table 1 caption seems to have grammatical error: "...bolded for the on average of the majority and minority tests".
- In table 3, column titles `CelebA` and `IN/IN-A` should be swapped.

[A] Yu, Xingrui, et al. "How does disagreement help generalization against label corruption?." *International Conference on Machine Learning*. PMLR, 2019.

[B] Yi, Kun, and Jianxin Wu. "Probabilistic end-to-end noise correction for learning with noisy labels." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2019.

[C] Li, Junnan, Richard Socher, and Steven CH Hoi. "Dividemix: Learning with noisy labels as semi-supervised learning." *arXiv preprint arXiv:2002.07394* (2020).

[D] Hendrycks, Dan, and Thomas Dietterich. "Benchmarking neural network robustness to common corruptions and perturbations." *arXiv preprint arXiv:1903.12261* (2019).

[E] Geirhos, Robert, et al. "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness." *arXiv preprint arXiv:1811.12231* (2018).
summary_of_the_review: The paper presents a novel and simple debiasing approach for neural networks, as well as a denoising module for training with label noise. The proposed debiasing method shows promising results especially on synthetic benchmarks, comparing against various state-of-the-art algorithms; however, I believe the concerns listed in the section above needs to be addressed to fully justify the contributions of the work. I am setting my initial score to borderline reject but would be happy to update it based on the reviewers' response.
review_title: null

Meta-review: The manuscript proposes a method to adjust a biased model without requiring explicit annotations of biases. The main hypothesis of the manuscript is that there are differences in the direction and magnitude of the loss gradients for underrepresented samples compared to majority patterns in the training data. Based on this hypothesis, the manuscript proposes a rejection sampling method that tries to balance samples in a minibatch. However, a sample with a noisy label can appear to be an underrepresented sample with a correct label which can affect the proposed method. To tackle this, the manuscript also proposes a denoising module that successfully eliminates the effects of noisy labels on the debiasing algorithm proposed. Experiments are performed on various synthetic and real-world biased sets. 

Positive aspects of the manuscript includes:
1. The results for varying levels of "bias" as well as the success of the proposed "denoising" setup is remarkable for the datasets tested;
2. An interesting hypothesis about the differences between gradient magnitude and direction (as measured by its proximity to an "average" gradient direction for all samples) look different for underrepresented samples as compared to "regular" data sample.

There are also several major concerns, including:
1. Lack of motivation and analysis on the connections between per-sample gradients and the majority/minority splits in more complex datasets;
2. The key assumption which motivates the proposed method, namely that minority samples have different gradient distributions than majority ones, deserves a more rigorous validation;
3. The "scalability" of the proposed method. One common theme across these datasets is that they can be "learned" (at least the biased version) with a much smaller amount of data than is present in the training set. Hence, a rejection sampling-based method can work even when the minority set diminishes; 
4. Assumption about known bias. The proposed method assumes knowledge about which of the factors were biased so that a suitable "bias-only" model can be trained by leveraging only the "bias". 

Post-rebuttal, reviewers stayed with borderline ratings, and they have suggested further improvements: more details about Biased MNIST numbers (to address concerns about known bias), and ablation studies on real datasets (e.g. compare to results without denoising, or denoised using FINE) to fully justify the practical importance of proposed denoising module.
Decision: Reject