Severing Spurious Correlations with Data Pruning

Authors: Varun Mulchandani, Jung-Eun Kim
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Spurious Correlations, Data Pruning

Abstract:
Deep neural networks have been shown to learn and rely on spurious correlations present in the data that they are trained on. Reliance on such correlations can cause these networks to malfunction when deployed in the real world, where these correlations may no longer hold. To overcome the learning of and reliance on such correlations, recent studies propose approaches that yield promising results. These works, however, study settings where the strength of the spurious signal is significantly greater than that of the core, invariant signal, making it easier to detect the presence of spurious features in individual training samples and allow for further processing. In this paper, we identify new settings where the strength of the spurious signal is relatively weaker, making it difficult to detect any spurious information while continuing to have catastrophic consequences. We also discover that spurious correlations are learned primarily due to only a handful of all the samples containing the spurious feature and develop a novel data pruning technique that identifies and prunes small subsets of the training data that contain these samples. Our proposed technique does not require inferred domain knowledge, information regarding the sample-wise presence or nature of spurious information, or human intervention. Finally, we show that such data pruning attains state-of-the-art performance on previously studied settings where spurious information is identifiable.

Decision: accept (spotlight)
Reviewer scores: 10, 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1729973779497

thread_type:
official_review

<thread_object>
creator: reviewer_n9k8

<content>
summary:
This paper empirically investigates the phenomenon of spurious correlations being learned by deep learning models.

The authors are mainly interested in the setting where the spurious signals are "weak", in the sense that are not easy to be detected and removed.

In this setting, the authors propose a data pruning approach to limit the effects of the spurious correlations.

The method consists in looking at the (few) training points that are more difficult to classify during training. These points are most likely making the model overfit patterns that are not necessarily causal to the right label, and are therefore a major cause to the learning of spurious patterns. This is confirmed experimentally.

strengths:
The paper is very well written, it is clear, the approach followed by the authors is logically sound, and I personally enjoyed reading it. While I am not certain that it is true that all previous work adresses cases where the spurious signal is stronger than the core features, the contribution seems indeed novel, and the results are reasonable and well explained.

The paper is on point. It proposes mainly one single method, and the writing is structured in a way that the reader can digest the phenomenology before being given the final algorithm / approach.

weaknesses:
It is sometimes not obvious what the authors mean by "strong" or "weak". While providing precise definitions is beyond the purposes of this work, some examples during the introduction could facilitate the reading. For example, I was confused in the paragraph at line 201, where the strength of the signal is defined both in terms of the geometry of the pattern and it's frequency in the data. These two aspects are fundamentally different, and putting them altogether might not result in the best model to investigate this problem...

The reason behing I do not give a higher schore is that some of the results are not entirely surprising. The phenomenlogy described by Figure 2 is interesting, but at the same time - to my understanding - in line with what we expect from the influence of individual samples to the final parameters of the model (see, e.g., [1, 2]). Nevertheless, this might be a new perspective in the community of spurious features / correlations, and I therefore recommend acceptance.

[1] https://arxiv.org/pdf/1906.05271
[2] https://proceedings.neurips.cc/paper_files/paper/2020/file/1e14bfe2714193e7af5abc64ecbd6b46-Paper.pdf

questions:
There should be a typo in line 162 "as shown in..."

I am still confused about the setting used to obtain Figure 3. And what do the authors mean by "polynomial growth" in this case? What does it mean to "scale up the difficulty of the core features of the 100 saples"?

I am also confused by Section 6.3 - What do the authors mean by "group labels" in line 457? Can the authors provide a brief and self contained description of the point that wants to be raised in this section (while I believe this is not the central part of the work)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730131066690

thread_type:
official_review

<thread_object>
creator: reviewer_7bqb

<content>
summary:
This paper study the spurious feature associated with the trained model, focusing on how this correlation is formed during the training process and proposing a data pruning approach to mitigate spurious correlation. Specifically, the authors find that spurious correlation is caused by a few samples that are hard to learn and contains spurious features. As a result, this paper proposes to remove those data points from the training dataset to improve model performance.

strengths:
1. The paper organization is clear and easy to follow.
2. The problem of identifying and mitigating spurious correlation is critial.
3. This paper considers a challenging scenario where spurious features have weak signals and therefore are difficult to be detected.

weaknesses:
**Major Concerns:**

1. Lack of rigorous formulation and solution. Throughout this paper, there is no single equation or definition that clearly states the problem and the proposed solution. Key concepts that heavily mentioned in the paper, such as spurious correlation, core/invariant features, simple/hard features, are not well-defined. For a more readable paper, the authors are encouraged to (1) state a self-contained problem with properly defined concepts (2) write a pesudocode for the proposed data pruning algorithm (3) disclose complete experimental details including but not limited to training/test dataset processing procedure, models, calculation of evaluation metric (which is not well-defined as well), and baseline methods.

2. Lack of clarity in findings. Most figures and tables are not self-explained and neither explained by their titles. As a result, it is confusing how do they support the claims in the main text. This problem is aggravated due to the previous point. For example, I did not understand the message in Figure 2, as concepts like "easy/hard" are undefined, and authors do not explain how they perform the experiments, e.g., how to add spurious features into the training data and how to calculate the misclassification rates. The authors should review all their findings and claims and rewrite their evidence to be more supportive and convincing.

questions:
Please find in the weakness section above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730673041572

thread_type:
official_review

<thread_object>
creator: reviewer_ecbw

<content>
summary:
The paper explores how deep neural networks often rely on spurious correlations present in training data, which can lead to performance drop under distributional shifts. The authors highlight a novel setting where spurious signals are weaker, making their identification challenging. They propose a new data pruning technique that selectively removes some training samples that contribute significantly to the formation of spurious correlations. This technique operates without requiring detailed information on the nature or presence of spurious features. The authors demonstrate that their approach achieves state-of-the-art performance on both standard and challenging benchmarks, including scenarios where spurious features are identifiable and unidentifiable.

strengths:
1) The paper addresses a scenario where the strength of the spurious signal is relatively weaker and thus it is difficult to detect any spurious informationhat spurious signals compared to other works  where the strength of the spurious signal is significantly greater
than that of the core. 
2) The paper provides a thorough experimental design that spans both identifiable and unidentifiable spurious feature scenarios. The use of multiple datasets (e.g., CIFAR-10S, CelebA, Hard ImageNet, Waterbirds, MultiNLI) showcases the method's robustness.
3) The paper is well-organized, detailing the rationale behind the proposed method, experimental setup, and results.

weaknesses:
1) The paper could explore more thoroughly the practicality of applying the proposed pruning method to large datasets. Specifically,even though the proposed method shows promise for datasets of moderate size, an assessment of its computational cost and efficiency on large and real world data would be beneficial.
2) The approach relies on assessing sample difficulty as a proxy for contribution to spurious correlations. Clarifying the robustness of this metric under different training regimes (e.g., varied architectures or optimization strategies) could strengthen the generalizability of the findings.
3) Although the paper discusses state-of-the-art methods, further comparative analysis with recently emerging pruning and robust training techniques that do not rely on explicit spurious feature identification would be helpful.

questions:
1) How sensitive is the pruning method to changes in hyperparameters?
2) The robustness of sample difficulty estimation across different model architectures is not clear at the current version of the paper. Would it be possible to add some results or explanations for that?
3) Table 1 shows that some SOTa methods achieve better results. Could you please give some explanation on that? Also, 
4) could you please open source the code to enhance the reproducibility?
Suggestions for Improvement:
1) It would be helpful to include a section that discusses the potential negative impacts or limitations of the method, such as the risk of pruning samples that are informative but rare.
2)Extending the empirical analysis to more complex and real-world datasets with non-synthetic spurious correlations could further validate the applicability of the method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730776741507

thread_type:
official_review

<thread_object>
creator: reviewer_x6fn

<content>
summary:
This paper aims to mitigate the problem of spurious correlations in deep learning models. Through a sequence of simulation experiments, they authors discover that a small subset of training data containing “hard” core features is primarily responsible for the model learning spurious correlations, even when the spurious signal is weak. The authors then demonstrate through subsequent experiments that pruning this subset of samples effectively severs the link between spurious and core features.

strengths:
This work represents an example of my favorite kind of work. A very important and easy-to-understand problem (spurious correlations learned by models that affect their generalizability) and a very intuitive solution, once the authors explain and demonstrate it. I also felt the work was well written, guided the reader through the gaps in the literature, and step by step demonstrated the veracity of their arguments with simple and compelling experiments.

weaknesses:
Inductive Solution
There I assume many types of spurious correlations can be learned by a model. For example, it is will documted that outliers can overinfluence functional estimation, and create spurious corrleations. The proposed solution addresses only a specific type of spurious correlation (i.e., roughly speaking, where a certain class has an over-representation of a given feature that its not truly indicative of the class label), which arises under the particular conditions the authors generate (i.e., "the spurious feature takes the form of a line running through the center of the images", "images of men with glasses", etc. ), given the distributional properties of the datasets considered, and given deep learning architectures (e.g., mostly ResNet) considered. Moreover, their solution relies heavily on specific empirical observations about how this particular type of spurious correlation manifests and is distributed, given the particular generation process followed. For instance, the authors observe that in their simulations "in settings where the strength of the spurious signal is not significantly greater than the strength of the invariant signal...samples containing spurious features are uniformly distributed across the training distribution...[and] the presence of spurious features does not have a significant impact on the training distribution...[therefore] samples containing hard core features that also contain the spurious feature are primary contributors to the formation of spurious correlations". They therefore, conclude that "to mitigate spurious correlations, one would only have to prune the hardest samples in the training set, as this subset of the data would contain samples with spurious features that have hard core features.” This reasoning appears to be purely inductive and raises questions about the generalizability of the observations and subsequent solution. Specifically, I'd assume that the observations about the distribution of spurious signals and sample difficulty will hold across other types of spurious correlations, which may behave differently under varying dataset characteristics or model architectures. 

Theoretical Justification/Generality
Without a theoretical foundation supporting the general applicability of the inducitve observations, it remains unclear whether the observations that lead to solution are universal and/or if the prunning method can serve as a general approach to mitigating spurious correlations in models. Therefore, the paper could be significantly improve with a theoretical justifications for the generality of the inductive observation. For example, 1) is the particular type of spurious correlation the authors consider representative of all spurious correlations, 2) does the distributional uniformality of samples containing features with spurious correlation across the training samples generalize across types of spurious correlations, 3) is the pruning of this particular type of subset as a solution to (any type of) spurious correlations, and 4) are these justifications architecture/data dependent. 

Related, but unique, I think structuring the pruning solution as a formal model, statistical hypothesis test, etc. would strengthen the theoretical foundation of the proposed pruning technique, and again perhaps shed light on it's generality. A theoretical or statistical clarity on why the samples containing spurious correlation are distributed uniformaly (e.g., I wonder if this is a consequence of the inverse probability transform). A claim that spurious feature reliance experienceing polynomial growth with increasing sample difficulty, would be more trustworthy if supported by some theoretical formulation or model. 

Pruning Consequences
While the authors demonstrate that prunning a particular type of subset of data points will reduce spurious correlations, there is no discussion of what are the consequences. I am inclined to beleive that throwing out data is going to have some sort of negative consequence, and therefore it important to know what trade-off is being made. This will allow a better comparison to other methods that do not simply prune data as well as enable practitioners to understand/determine if the cost of pruning data is worth the benefit in removing spurious correlations.

questions:
My question(s) would simply be can the authors provide theory, models, statistical justification etc. (or fairly robust/general empirics) to address the listed weaknesses?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731971273228

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We thank the reviewers for their comments. We are glad that the reviewers recognize the key contributions of the paper:

**Contributions:**
1) Identifying and targeting novel settings where obtaining spurious information is difficult/impossible and showing the failure of past techniques on these settings.
2) Discovering that spurious correlations are primarily formed from a handful of all samples containing spurious features through extensive empirical investigation.
3) Proposing a novel data pruning solution that severs spurious correlations in the proposed novel settings while attaining state-of-the-art performances on previously studied settings.

We believe we have addressed the concerns of all reviewers by responding to them individually. We summarize the resolution of the primary concerns that were shared by the reviewers:

**Weaknesses Resolved:**

1) Included more results across more architectures and hyperparameters to further reinforce our claims and observations. (Reviewers eCbW, X6Fn)
2) Improved all relevant formal definitions/more detailed figure captions/technical details. (Reviewers 7BQb, n9k8)
3) Released our code through an anonymized github link and will open-source it upon acceptance. (Reviewer eCbW)


However, we strongly believe that the score received from reviewer 7BQb is not a fair assessment of our work. Their only concern was the clarity or presentation of the paper but they have given our contribution and soundness a low score as well. We would like to emphasize that the clarity and writing quality of the paper were well received by all other reviewers. Nevertheless, we have addressed the clarity concerns of reviewer 7BQb and hope that our work is now well received. If not, we are happy to address any further concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731977644816

thread_type:
official_comment

<thread_object>
creator: area_chair_n8q2

<content>
title:
Discussion Phase

comment:
Dear Reviewers,

Please review the authors' replies and consider the feedback from other reviewers. If your concerns remain unresolved, feel free to ask for further clarifications. We have until November 26th for discussion.

Thank you for your efforts.

Best regards,
Area Chair
</content>
</thread_object>

</thread>
<thread>
creation date:
1733246041246

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response for the Area Chair and the Reviewers

comment:
We thank the reviewers for their comments, which have helped improve the quality of this paper. We strongly believe that we have addressed all concerns raised by all reviewers. Below, we summarize the discussion at a high level:

---

>**Summary**

The reviewers recognized that this work tackles an important and critical problem (X6Fn, 7BQb) through a very intuitive solution (X6Fn), contains novel contributions (n9k8, X6Fn), delivers interesting and exciting insights and observations (X6Fn), discovers some valuable generalizable truths (X6Fn), contains simple and compelling experiments (X6Fn), provides a thorough experimental design with multiple datasets (eCbW), and that it was well written (n9k8, X6Fn) and well organized (eCbW, n9k8, 7BQb). While some reviewers had some initial comments regarding clarity of the paper (7BQb, n9k8), these have now been addressed and acknowledged by the reviewers.

---

The following are comments that we have addressed but have **not** heard back regarding:

1) Reviewer **eCbW**: Reviewer eCbW asked us to **open source our code, which we have**. They point out that the settings studied in this paper are synthetic and that we should include settings with real-world spurious features. **This is incorrect** as only 1 out of the 5 datasets studied has synthetic spurious features which is **necessary and intentional** as it enables us to alter the synthetic spurious feature to draw insights. The other four datasets contain **real-world** spurious features in real-world settings. Furthermore, apart from the novel settings proposed, the standard settings studied are consistent with those in the literature. We have also received **no references or settings** that they want us to reproduce our results in. They also asked us to provide **additional results across different architectures and hyperparameters, which we have**. However, we have not heard back from them at all during the rebuttal phase.
 
2) Reviewer **7BQb**: As we have stated in our last response, we believe Reviewer 7BQb may have misunderstood our explanation on what impacts the strength of a feature. There are many factors that can be varied to alter the strength of a feature. In our paper, the strength is **varied** by two factors: frequency and area. They additionally claim that these concepts are not well defined. We emphasize, however, that these concepts are heavily studied in all relevant literature that we have cited.
We also note that discovering and considering all factors that can impact the strength of a feature is beyond the scope of this paper and that this does not impact the generalizability of our findings, as they ultimately alter the same thing: the strength of a feature. We have already shown that frequency and area can be modified interchangeably to alter the strength. One could also modify the strength by making red color in eyeglasses a lighter/darker shade, as the reviewer has pointed out. For instance, one could vary the shade to create the same plots in Figure 5 (a) but this will **not offer any new insights**. It is also unclear why the reviewer believes that their example of eyeglasses impacts generalizability and relates to Reviewer X6Fn’s comments. However, we have not heard back from them.
</content>
</thread_object>

</thread>


Metareview:
metareview:
### Summary

The authors investigate spurious correlations in a controlled environment by introducing them into existing datasets. Through a series of calibrated experiments, they identify how individual samples contribute to spurious correlations. This leads to a pruning strategy that significantly improves worst-group performance without drastically reducing average performance. The proposed method achieves state-of-the-art results compared to other strategies.

### Strenghts

The paper is well-written and easy to follow. 
The authors guide the reader through their experiments, elucidating their findings and the reasoning behind the method.
This paper gives both a contribution to the understanding of the phenomenon and to the development of a mitigation strategy. 

### Weaknesses 

No major concerns.
Marginal improvements over other mitigation strategies.

### Reasons for acceptance

This paper presents a simple yet impactful idea, supported by experiments, that leads to a practical algorithm. All major critiques have been addressed, and I see no reason for rejection.