Trained Models Tell Us How to Make Them Robust to Spurious Correlation without Group Annotation

Authors: Mahdi Ghaznavi, Hesam Asadollahzadeh, Fahimeh Hosseini Noohdani, Soroush Vafaie Tabar, Hosein Hasani, Taha Akbari Alvanagh, Mohammad Hossein Rohban, Mahdieh Soleymani Baghshah
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Spurious Correlation, Group Robustness, Zero Group Annotation, Distribution Shift, Out-of-Distribution Generalization

Abstract:
Classifiers trained with Empirical Risk Minimization (ERM) tend to rely on attributes that have high spurious correlation with the target. This can degrade the performance on underrepresented (or 'minority') groups that lack these attributes, posing significant challenges for both out-of-distribution generalization and fairness objectives. Many studies aim to enhance robustness to spurious correlation, but they sometimes depend on group annotations for training. Additionally, a common limitation in previous research is the reliance on group-annotated validation datasets for model selection. This constrains their applicability in situations where the nature of the spurious correlation is not known, or when group labels for certain spurious attributes are not available. To enhance model robustness with minimal group annotation assumptions, we propose Environment-based Validation and Loss-based Sampling (EVaLS). It uses the losses from an ERM-trained model to construct a balanced dataset of high-loss and low-loss samples, mitigating group imbalance in data. This significantly enhances robustness to group shifts when equipped with a simple post-training last layer retraining. By using environment inference methods to create diverse environments with correlation shifts, EVaLS can potentially eliminate the need for group annotation in validation data. In this context, the worst environment accuracy acts as a reliable surrogate throughout the retraining process for tuning hyperparameters and finding a model that performs well across diverse group shifts. EVaLS effectively achieves group robustness, showing that group annotation is not necessary even for validation. It is a fast, straightforward, and effective approach that reaches near-optimal worst group accuracy without needing group annotations, marking a new chapter in the robustness of trained models against spurious correlation.

Decision: reject
Reviewer scores: 5, 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1730411217750

thread_type:
official_review

<thread_object>
creator: reviewer_3v9e

<content>
summary:
This paper works in the sub-population shift setting, in which the data consists in samples of different groups that shares a property. The goal is to learn a model robust to sub-population shifts, such as class imbalance, attribute imbalance, and spurious correlations. They propose a method (EVaLS) that improves the robustness of models trained using ERM without using any group annotation data. The method is well motivated both empirically and theoretically, and they show that the model has competitive performance with and without using group annotation. Moreover, they show that EVaLS is robust to scenarios where there is an unknown spurious attribute in comparison to the state-of-the-art.

strengths:
- Simple method, that works in multiple scenarios (with and without group annotations).  
- The method do not make strong assumptions about the trained model. The only requirement is a model trained by ERM (the training data or any other training information is not necessary), while the method acts in a post-training phase.  
- It shows competitive performance in comparison to the literature, while it has a less strict set of requirements in comparison to most of them.  
- EVaLS outperforms the DFR (one of the state-of-the-art models) in cases with multiple spurious attributes

weaknesses:
- Make a comparison of EVaLS with more methods (e.g. AFR, since it also doesn't depend on ERM training) in the multiple spurious attributes scenario.
- Have more evidence that EVaLS outperforms the DFR/other methods in cases with multiple spurious attributes (e.g. using more datasets). I believe that this is the strongest part of the results, and it is a clear advantage for EVaLS (besides cases in which group annotation is not available).

questions:
Some questions:
1) Is it feasible to add AFR results to the multiple spurious attributes experiment?   
2) Is it feasible to add an extra dataset to the multiple spurious attributes experiment?

Adding these extra results will address the points mentioned in the weakness, showing stronger empirical evidence about EVaLS advantage in cases with multiple spurious attributes. 

Minor points:
- There are references to Figure 1 (Line 83) and Figure 3 (Lines 172-177) that come before Figure 2 and were a bit confusing to me while I was reading the paper for the first time. In my opinion, removing these references will improve the readability of the paper.  
- Figure 2 instead of figure 2 (Line 205).  
- Have the oracle results as a reference in Table 6 to follow Figure 4 (b).  
- Figure 4 (b) could include the standard deviation (such as in Table 6).  
- Sometimes you use sub-population (Line 131) or subpopulation (Line 139). Keep it consistent across the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730433954361

thread_type:
official_review

<thread_object>
creator: reviewer_9oxu

<content>
summary:
This paper introduces a method called EVaLS, which trains a classifier that is robust to spurious correlations without requiring group labels. The method constructs a balanced dataset based on loss values and utilizes environments for hyperparameter tuning. Experiments on several datasets demonstrate the effectiveness of the proposed method.

strengths:
The method of selecting high-loss and low-loss data proposed in this paper effectively mitigated the problem of group imbalance, with both experiments and theoretical analysis effectively validating this point.

weaknesses:
- The authors may need to provide a more detailed description of the advantages of loss-based sampling over other sampling methods. For instance, it would be beneficial to compare it specifically with methods like SELF, highlighting the unique benefits or efficiencies achieved by the proposed method.

- The authors' claim that their method 'completely eliminates the need for group annotations' as a primary contribution seems somewhat tenuous. The methodology presented in the paper can be categorized into two main components: Environment-based Validation (EV) and Loss-based Sampling (LS). Notably, LS appears to require group labels for hyperparameter tuning, while EV utilizes environment labels generated through Environment Inference for Invariant Learning (EIIL) as a stand-in for group labels. However, EIIL was originally proposed by Creager et al., 2021, and it was inherently designed to address scenarios where group labels are not available. Although Creager et al., 2021 used true group labels for model selection within GroupDRO in their experiments, it appears that environment labels could also be suitably employed for this step.

questions:
- From lines 452-457, the authors state that annotation-free methods can mitigate the impact of both labeled and unlabeled shortcut features more effectively. However, EVaLS-GL, which utilizes group labels, achieved better results than EVaLS. Could the authors provide their insights on this phenomenon?
- 
In the experiments, the performance of EVaLS and EVaLS-GL varied, with each having its strengths and weaknesses. Could the authors discuss the advantages and disadvantages of using inferenced environments versus group labels based on these findings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730594395756

thread_type:
official_review

<thread_object>
creator: reviewer_3i2u

<content>
summary:
Many studies that enhance robustness to spurious correlation require group annotations for training. This paper aims to enhance the robustness with minimal group annotation assumptions. Specifically, the losses from an ERM-trained model are used to construct a balanced dataset of high-loss and low-loss samples, mitigating group imbalance in data. Moreover, using environment inference methods to create diverse environments has been shown to potentially eliminate the need for group annotation in model selection. Experiments demonstrate the effectiveness of the proposed method.

strengths:
- The paper is well-written and easy to follow.

- The paper proposes a practical method to mitigate the reliance on spurious correlations without any group annotations. 

- A new dataset is constructed that demonstrates the effectiveness of the proposed method in mitigating unknown shortcuts.

weaknesses:
- The proposed method is incremental and has limited technical contributions. Retraining the last-layer using group-balanced validation data to mitigate the reliance on spurious correlations has been used in [1,2]. Inferring environment is a direct follow-up of [3].

- The theoretical analysis does not really explain why loss-based sampling within a class can be used to create a group-balanced dataset. The analysis assumes that the losses on the majority and minority samples follow Gaussian distributions. Under this assumption, it is obvious that the loss-based sampling could create two group-balanced sets of data. However, whether this assumption holds in practice is questionable. Moreover, a previous study [2] has found that that model disagreement may effectively upsample worst-group data, or in other words, may create a more group-balanced dataset. Thus, the loss-based sampling may not be as effective as proved in the paper.

- The comparison in Table 1 isn't fair for some methods. EVaLS uses new data, i.e., a part of validation data, for retraining, while methods including GDRO + EIIL, JTT, and ERM do not have access to the new data. Moreover, the existing work [4] also propose a method that aims to mitigate spurious correlations without group annotations. It would be beneficial to compare with this method under the same setting. 

- There are some model selection methods that do not require group annotations, such as minimum class difference [4] and worst-class accuracy [5]. It would be helpful to analyze the effectiveness of the proposed worst environment accuracy in comparison with these techniques.

[1] Kirichenko et al., Last layer re-training is sufficient for robustness to spurious correlations, ICLR 2023.\
[2] LaBonte et al., Towards last-layer retraining for group robustness with fewer annotations, NIPS, 2023.\
[3] Creager et al., Environment inference for invariant learning, ICML, 2021.\
[4] Li et al., Bias Amplification Enhances Minority Group Performance, TMLR, 2024.\
[5] Yang et al., Change is hard: A closer look at subpopulation shift, ICML, 2023.

questions:
- See the weaknesses.
- In Table 1, why the experiments on the CivilComments and MultiNLI datasets are out of the scope of the method?
- In L189, the authors mention that they randomly divide the validation set into $\mathcal{D}^{LL}$ and $\mathcal{D}^{MS}$. What if the random division results in a poor set of $\mathcal{D}^{MS}$ which does not have sufficient samples to represent a minority group of samples?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730883423119

thread_type:
official_review

<thread_object>
creator: reviewer_jr76

<content>
summary:
The paper EVaLS, a method to improve model robustness against spurious correlations without requiring group annotations. EVaLS balances high- and low-loss samples from an ERM-trained model and applies a simple last-layer retraining (on a loss-based sampled dataset), thus enhancing group robustness. The approach also uses worst environment accuracy to for model selection. Experimental results on diverse dataset shows competitive performance to baseline methods.

strengths:
1. The proposed approach EValS, using environment inference and Loss-based sampling, is novel and interesting. 
2. EValS has competitive performance with other methods across diverse datasets.

weaknesses:
1. The assumption that "minority samples are more prevalent among high-loss samples, while majority samples dominate the low-loss category" is questionable. It is easy to construct distributions that does not satisfy this assumption. 
2. The performance of EValS seems to rely on the EIIL to find the correct environment. However, how to find the environments may be a challenging problem itself. 
3. The peformance of EValS does not seem consistently better than other baseline methods across all datasets. It is not clear whether the better performance in specific dataset is by chance.

questions:
See weakness above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732984848497

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response and Final Reminder

comment:
Dear Reviewers, 


Thank you for your time reviewing our work. **Only 3 days remain** until the end of the discussion period. We have addressed all your concerns and questions. Please read them and let us know if you feel further clarification, justification, evidence, or experiments are needed. 

We uploaded a new version of the paper by the ICLR deadline to correct minor errors and include new experiments reported and requested by Reviewer 3v9E. Please review our response to Reviewer 3v9E and Appendix G in the revised paper for the results of the new dataset added to the paper. The line references in our responses have been updated to reflect the revision.

Also, please read the following for a review and clarification of our contribution: 

In continuation of previous efforts in the field for robustness to spurious correlations without group annotations (L57-69, L738-751), requiring group annotations for model selection remains a limitation (L67-69, L749-751). 

We have found that the availability of a set of disjoint samples (i.e., environments) of data depicting group shifts can effectively serve this purpose, using worst environment accuracy as a reliable surrogate for model selection for robustness to spurious correlations. This is particularly important as there are known (e.g., using EIIL) or very simple (e.g., applying a random linear classifier on top of the feature space of the model) ways to obtain such sets for datasets with spurious correlations. However, please note that inferring environments is a different field of research (L722-727). In other words, what we have shown is that the availability of group annotations can be effectively relaxed to a condition that is possible to satisfy using current knowledge (through environment inference methods, simply applying a random linear classifier on the feature space, or other methods — see Appendix F.3).

We have also proposed an enhanced sampling technique, *loss-based sampling*, which is shown to be more effective than other group balancing schemes for robustness to subpopulation shift (see Table 1 and Table 11 for comparisons of EVaLS-GL with methods at a similar level of group supervision, such as AFR and SELF, and Appendix F.2 for another comparison with the sampling method in JTT). We have backed our balancing scheme with practical observations and theoretical analysis to provide better insights into why it could be effective.

It is expected that approaches that do not use group annotations will underperform compared to those that use group-level information (compare results of double-ticked methods with others in Table 1). In other words, when group annotations based on a spurious attribute are available, using them is logically more effective for achieving robustness to that attribute than not using them. However, *“in many real-world applications, the process of labeling samples according to their respective groups can be prohibitively expensive and sometimes impractical”* (L67-69). EVaLS (Environment-based Validation and Loss-based Sampling), without using group annotations, demonstrates **near-optimal** performance on spurious correlation datasets (L32-33, results in Table 1). While this represents important progress in the field, another property arises in EVaLS that is even more significant.

The ability to achieve robustness to spurious correlations without group annotations enables EVaLS to improve robust to ***unknown spurious correlations***. This means EVaLS can effectively make trained models robust to spurious correlations they rely on, whether identified or unidentified. Without this capability, even if a model becomes robust to a known spurious correlation using current approaches, a **persistent concern** remains about the presence of unknown spurious correlations. Such correlations may affect the model’s predictions and remain undetected, posing significant performance and safety risks. As discussed in Figure 4(b) and its corresponding explanations, as well as in our discussion with Reviewer 3v9E (Appendix G in the revision, https://openreview.net/forum?id=8DuJ5FK2fa&noteId=Eh8QvMMS7y), previous methods could not be responsible for achieving robustness to *unknown* spurious correlations. In contrast, EVaLS improves robustness to both known and unknown attributes. 

In other words, when a spurious correlation is unknown, methods requiring group annotations for robustness are not applicable. However, EVaLS, which demonstrates effectiveness in enhancing robustness to spurious correlations without group annotations, improves robustness to all known and unknown spurious attributes and achieves higher worst-group accuracies among all groups (full combinations of core and spurious attributes).

---

Please let us know if you have any other questions. We try to answer them in the remaining time promptly.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper having been borderline during the review process, I have read it and made my own idea of its content.

First, the paper makes the repeated claim, also repeated here in rebuttals, that its approach is near-optimal. But there is absolutely *no* formal result in the paper that allow to state such near optimality (certainly not Proposition 3.1, see below). The paper and rebuttal clearly attach to this claim of near-optimality the results of one *experimental* table (1). Such a claim could eventually be supported *iff* the optimal mark was known for the domains (e.g. if the domains had been simulated, with clear evidence or a proof of the best result). This is clearly not the case here and the authors insistence on this so-called near-optimality give the opposite results of the one intended: it ends up overselling results otherwise fine. 

It is important to insist at this point that absolutely *nothing* in the paper supports *any* claim of near-optimality of the method proposed.

Then, regarding the theory, there is little in the paper that serves the authors claim of generality. Proposition 3.1 is unfortunately poorly formulated and it is hard to understand the meaning of (1), which does not appear in a sentence (reading the appendix, I suspect it is the result of an unfortunate cut in the paper but any fix I can imagine certainly does not bring any substantial part of the generality claimed). The background of Proposition 3.1 is however *extremely* specific and shallow and *certainly* cannot serve as an illustration of the paper's claims about being of broad appeal (weakness #2 of reviewer 3i2u).

I can only stress the importance of *either* a strong formal analysis of the problem, *or* a more thorough experimental section -- indeed, the range of the key values of the approach are so different between domains (Section E.4) that it seems extremely complicated to figure out a simple rule of thumb to give a new user as a starting point to use the technique. In this context, writing "our results demonstrate that for most datasets, multiple hyperparameter combinations yield optimal or near-optimal performance, reducing the need for exhaustive searches" (L1399) makes little sense for *two reasons*, not just for the near-optimality claim (see above), but also because it is clear that the authors have spent a lot of time doing **domain dependent choices** for the parameters used without much details on why / how choices were made.

In fine, I believe the paper starts with an interesting idea -- indeed, a loss can often be viewed as a negative log-likelihood and so the idea of loss sampling can make sense --, but fails to give the proofs that the idea developed justifies the extremely strong claims made here and there in the paper. I can only recommend that the authors redraft the paper, dig in the theory and orient new experiments to show the approach is simple to use and its (hyper)parameters are at least reasonably easy to figure out even for just a reasonable range.