Towards Real World Debiasing: A Fine-grained Analysis On Spurious Correlation

Authors: Zhibo Wang, Peng Kuang, Zhixuan Chu, Jingyi Wang, Kui Ren
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: spurious correlation, dataset bias, debias

Abstract:
Spurious correlations in training data significantly hinder the generalization capability of machine learning models when faced with distribution shifts in real-world scenarios. To tackle the problem, numerous debias approaches have been proposed and benchmarked on datasets intentionally designed with severe biases. However, it remains to be asked: \textit{1. Do existing benchmarks really capture biases in the real world? 2. Can existing debias methods handle biases in the real world?} To answer the questions, we revisit biased distributions in existing benchmarks and real-world datasets, and propose a fine-grained framework for analyzing dataset bias by disentangling it into the magnitude and prevalence of bias. We observe and theoretically demonstrate that existing benchmarks poorly represent real-world biases. We further introduce two novel biased distributions to bridge this gap, forming a nuanced evaluation framework for real-world debiasing. Building upon these results, we evaluate existing debias methods with our evaluation framework. Results show that existing methods are incapable of handling real-world biases. Through in-depth analysis, we propose a simple yet effective approach that can be easily applied to existing debias methods, named Debias in Destruction (DiD). Empirical results on real-world datasets in both image and language modalities demonstrate the superiority of DiD, improving the performance of existing methods on all types of biases within the proposed evaluation framework.

Decision: reject
Reviewer scores: 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730429877438

thread_type:
official_review

<thread_object>
creator: reviewer_vvhf

<content>
summary:
This paper studies the suspicious correlation problem. The authors first identify the unrealistic suspicious correlation patterns in benchmark datasets. Based on this observation, the authors reveal that the gap between benchmark and real-world datasets mislead the development of debiased methods. The authors further propose a new debiasing method, enhancing exitsing methods across different biases.

strengths:
1.	The authors identify a key problem in existing benchmark datasets, which mislead the development of existing methods.
2.	The analysis on existing metrics provides insight in understanding existing benchmarks and developing better methods to handle realistic bias.
3.	The authors propose a new debiasing method named bias capture with feature destruction to destroy the effect of target features.

weaknesses:
1.	The motivation of different choices of metrics is not clear.
2.	Some of the assumptions in analysis are too strong or unrealistic.

questions:
1.	Line 198, why choosing KL divergence here, but using total variation in later analysis.
2.	Line 198, since the KL divergence is non-symmetric (and probably ill-defined when encountering zero probability), will other metrics, e.g., JS divergence, be a better metric.
3.	Eq.(1): can you switch the role of $y^t$ and $y^s$, i.e., $KL(p(y^s), P(y^s|y^t))? What’s the difference between these two formulations?
4.	Is there any justification on whether the two proposed metrics, i.e., magnitude and prevalence, are (1) complementary/orthogonal, i.e., they depict dataset property from quite different aspects, and (2) complete, i.e., they can completely depict the dataset properties.
5.	Line 269, data distribution: the analysis assumes that both target and suspicious attributes are binary, can you analysis generalized to multi-class/non-binary settings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730533844751

thread_type:
official_review

<thread_object>
creator: reviewer_wryr

<content>
summary:
The authors mainly aim to address the discrepancy between benchmark datasets and real-world datasets. Specifically, they observe that real-world datasets exhibit low prevalence, while commonly used datasets have high prevalence. They theoretically support this observation by demonstrating that high prevalence can occur under two unrealistic assumptions. However, in low-prevalence scenarios, the auxiliary models might fail because bias-neutral and bias-conflicting samples constitute the majority, leading these models to learn target attributes too quickly. To address this issue, the authors propose DiD, which disrupts the auxiliary models from learning target attributes through patch shuffling. In experiments, they demonstrate that their method significantly improves upon existing methods across all settings.

strengths:
* By analyzing the datasets from the perspective of prevalence and magnitude, the authors identify significant differences between real-world datasets and benchmark datasets.
* The authors provide theoretical support for this observation.
* The paper is well-written.

weaknesses:
- Although a theoretical background is provided, the experimental evidence appears limited. It would strengthen the work to demonstrate low prevalence on additional datasets, such as an NLP dataset and CelebA, commonly used in the spurious correlation domain. In particular, CelebA is not suggested for evaluating the methods for handling spurious correlations, yet it has a substantial amount of data and exhibits strong biases despite being a natural dataset. Demonstrating that its prevalence aligns with the claim would strengthen the argument. Furthermore, it would be helpful to show the performance in CelebA and NLP tasks as well.
- For B2T [1], which does not rely on the easy-to-learn property of bias, there seems to be no reason for it to fail in low prevalence settings. A discussion of the advantages of the proposed method over B2T in such scenarios would be required. Additionally, since B2T is a recent approach, it would enhance the paper to include it as a baseline on natural datasets.
- DiD lacks general applicability. Specifically, it relies on the assumption that bias will be insensitive to patch-shuffling, while the target will be sensitive to it. However, this approach may not work for other types of semantic biases beyond background or perturbation. Furthermore, this method seems to be difficult to use in NLP tasks.
- The attempt to improve the performance in low-bias settings while keeping the performance in high-bias settings is not the first [2].
- In Table 3, although the proposed method is designed to perform well in low prevalence settings, it shows lower performance than ERM in the unbiased setting.
- I couldn’t find a clear description of the metric used for Waterbird in Table 2 in the main paper. Given the reported performance, it seems that either a different metric from the commonly reported worst-group accuracy was used, or the setting doesn’t align with prior methods. It would enhance the evaluation to show whether worst-group accuracy improves when using the same experimental setup as JTT [3].

[1] Kim, Younghyun, et al. "Discovering and Mitigating Visual Biases through Keyword Explanation." CVPR, 2024.

[2] Jung, Yeonsung, et al. "Post-Training Recovery from Injected Bias with Self-Influence.".

[3] Liu, Evan Z., et al. "Just train twice: Improving group robustness without training group information." ICML, 2021.

questions:
* Including results from GroupDRO [1] as an upper bound for comparison would be helpful.

[1] Sagawa, Shiori, et al. "Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization." ICLR, 2020.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730632600747

thread_type:
official_review

<thread_object>
creator: reviewer_yhqh

<content>
summary:
This paper examines whether existing benchmarks can effectively capture biases present in real-world datasets. To address this, the authors introduce a new evaluation framework that assesses dataset bias from the perspectives of magnitude and prevalence. Experimental results indicate that current real-world datasets primarily exhibit a low magnitude, low prevalence distribution, highlighting a gap compared to synthetic benchmarks. Furthermore, the authors discuss the debiasing effects of existing DBAM methods under three different bias distributions. They also propose a new debiasing method, DiD, which enhances the bias capture module and improves the accuracy of bias feature learning through a feature destruction approach.

strengths:
1. The motivation for this research is sufficient, as there is widespread concern about whether existing benchmarks can truly reflect the biases present in real-world datasets.
2. This paper presents a fine-grained framework that evaluates dataset bias through the metrics of magnitude and prevalence, offering a multi-dimensional approach.
3. The authors highlight unique characteristics of real data distribution, which is essential for understanding the effectiveness of debiasing methods in practical scenarios.
4. They enrich the diversity of synthetic datasets, adding valuable resources for further research in debiasing.

weaknesses:
1. **Clarity and Presentation Issues**
    * Line 160: Are you referring to "measure spurious correlation according to the probability of the correlated class $a_t$ within samples with biased feature $a_s$"?
    * Figure 2b may contain a scaling error on the x-axis that could impact result interpretation.
    * Some descriptions are ambiguous. For instance, the term “realistic” in Table 2 might misleadingly imply real-world data, whereas it refers to synthetic datasets. Clarifying this distinction would improve reader understanding. In Appendix D.2, the BAR dataset is described as "a real-world dataset," while in Section 2.1, it is referred to as a "semi-synthetic dataset."
2. **Limitations in the Scope of the DiD Method**
   *  The feature destruction in DiD method has a limited scope of applicability. While we acknowledge its rationale in visual contexts, identifying or destructing target features is challenging in more general scenarios. Could the authors provide potential solutions for feature destruction in structured real-world datasets (e.g., Adult and COMPAS) or within the NLP domain?
3. **The experiments are insufficient**
    * The selection of baselines is limited. Why are comparisons made only with DBAM methods? Would it not be beneficial to include comparisons with data generation methods as well?
    * As mentioned in Section 2, real-world data is primarily characterized by LP bias distributions. In the results presented in Table 1, why does the DiD method underperform compared to the ERM method under LP distribution?
    * Considering the importance of BC samples in debiasing, why does the DiD method exhibit negative transfer in accuracy on BC samples in some experiments?

questions:
1. In Section 4.2, although BN samples have lower weights under LP conditions, is it possible that, due to their being learned more frequently than other samples, the model still captures the information contained in BN samples without overlooking it?
2. As mentioned, real-world data primarily exhibits LMLP bias distributions. Are there experimental results for Emphasis on BN samples under LMLP conditions?
3. Could the authors provide experimental results for the DiD method on real-world datasets to validate its effectiveness on actual data?

If the authors are willing to address our concerns, we are open to revising the scores during the rebuttal stage.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732817586537

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Milestone summary and looking forward to further feedback

comment:
As the deadline for the authors to upload a revised paper has passed and the second stage of the discussion starts, we summarize the major revisions of the paper along with the discussions we have had with the reviewers so far.

## Revisions of the paper

- We have further demonstrated the scope and general effectiveness of the proposed method by extending the method from the image to the language domain. (Thanks to reviewers yHqh and WrYr)
- We further included evaluations of the proposed method on 1 real-world image dataset and 2 real-world language datasets. (Thanks to reviewers yHqh and WrYr)
- We have included 3 additional baselines from various domains and tasks as suggested by reviewers yHqh and WrYr, which further strengthens our evaluation.
- We have included additional empirical analysis to address concerns/questions from reviewers yHqh and WrYr. Specifically, BN weights in LMLP distributions, distribution analysis of 3 additional real-world datasets, and evaluation on the WaterBirds dataset in the setting of JTT.
- Additional clarifications are included to avoid misunderstandings similar to the reviewers. Typos and ambiguity are corrected thanks to suggestions from reviewers yHqh and vvhf.

## Feedback from the reviewers so far

Both reviewers yHqh and WrYr considered most of their concerns addressed with no additional concerns/questions and raised the scores.

Reviewer vvhf keeps the score without mentioning any unaddressed or additional concerns/questions.

We briefly sum up the reviews and comments from the reviewers so far as follows:

1. In terms of contribution, all reviewers rated "3: good".

    - Reviewer yHqh considers "*the motivation behind the analysis is sufficient and meaningful*", "*essential for understanding the effectiveness of debiasing methods in practical scenarios*", and "*adding valuable resources for further research in debiasing*". 
    - Reviewer WrYr considers "*the authors identify significant differences between real-world datasets and benchmark datasets*", and "they demonstrate that their method significantly improves upon existing methods across all settings". 
    - Reviewer vvhf considers "*The authors identify a key problem in existing benchmark datasets, which mislead the development of existing methods.*".

2. In terms of presentation, all reviewers rated "3: good". Reviewer WrYr considers "*The paper is well-written.*"

3. In terms of soundness, we carefully addressed all the reviewers' concerns and questions without any further questions or concerns from any of the reviewers after reading the rebuttal. Reviewer WrYr rated "3: good" after the rebuttal.

**We are truly grateful for all the reviewers' valuable advice and the reviewers' appreciation of the work, especially the consistent acknowledgment in terms of contribution, considering our motivation "*sufficient and meaningful*".**

## Looking forward to further feedback

Despite what we and the reviewers have accomplished so far as mentioned above, we want to make sure if there are any further concerns/questions from the reviewers, especially from reviewers yHqh and vvhf since they still hold a negative attitude towards acceptance without mentioning their unaddressed or additional concerns/questions. **If any, we are more than willing to address further questions or concerns regarding the paper**.

**Again, we sincerely appreciate all the reviewers for their time and effort devoted to reviewing and discussing the paper.**
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper examines dataset bias in practical contexts, pinpointing the shortcomings of current benchmarks. The authors put forward a detailed framework for bias assessment, introduce two bias metrics, and a debiasing technique called Debias in Destruction (DiD). After the rebuttal phase, the paper ended up with a borderline score below the acceptance criteria. I have read the comments and the paper. I believe that there are still some issues to be addressed:

1. **Limited Theoretical Scope (Reviewer vvhf)**: The theoretical discussion is mainly confined to binary classification tasks using a simple metric. It's unclear if the theory could be extended to multiclass scenarios, creating a substantial gap between theory and application. The authors did not provide proof or even a sketch to extend the theoretical result in the rebuttal phase. 

2. **Insufficient Novelty and Generalizability (Reviewers WrYr, yHqh)**: Though DiD provides a new viewpoint on debiasing, its dependence on conventional methodologies (CE and GCE), like feature destruction, limits its novelty. Additionally, the paper does not adequately clarify how effective feature destruction is in debiasing compared to theoretical expectations.

3. **Limited Empirical Evidence**: The experimental analysis is relatively narrow, with few and simple competitors. 

Above all, I tend to recommend rejecting this paper. The authors are encouraged to resubmit their work after considering the suggestions.