EvA: Erasing Spurious Correlations with Activations

Authors: Qiyuan He, Kai Xu, Angela Yao
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: spurious correlation, compute efficiency, data efficiency

Abstract:
Spurious correlations often arise when models associate features strongly correlated with, but not causally related to, the label e.g. an image classifier associates bodies of water with ducks. To mitigate spurious correlations, existing methods focus on learning unbiased representation or incorporating additional information about the correlations during training. This work removes spurious correlations by ``**E**rasing **wi**th **A**ctivations'' (EvA).  EvA learns class-specific spurious indicator on each channel for the fully connected layer of pretrained networks. By erasing spurious connections during re-weighting, EvA achieves state-of-the-art performance across diverse datasets (6.2\% relative gain on BAR and achieves 4.1\% on Waterbirds). For biased datasets without any information about the spurious correlations, EvA can outperform previous methods (4.8\% relative gain on Waterbirds) with 6 orders of magnitude less compute, highlighting its data and computational efficiency.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730584029449

thread_type:
official_review

<thread_object>
creator: reviewer_ih3n

<content>
summary:
This paper proposes EVA, a method for addressing spurious correlations in datasets. It identifies spurious features using a consistency measure and the evidence energy measure in scenarios with and without unbiased datasets, respectively. Theoretical analyses are provided regarding the relationship between these two measures and a feature’s spuriousness. Experiments are conducted to demonstrate the method's effectiveness.

strengths:
- The method is well-motivated, and I appreciate its simplicity (in terms of both understanding and computation).
- Providing theoretical analysis adds a more principled foundation to the two measures.
- There are experiments showing that the erased features are indeed the spurious ones on CMNIST, and the correlation between evidence energy and consistency, which further supports that the method works as expected.

weaknesses:
- Since the original DFR paper uses ResNet50 as the architecture, it would be a good sanity check to compare results in that same setting. The authors provided results for ResNet50 in Appendix E.2, but they mention that, to mimic real-world scenarios, they tuned hyperparameters based on Mean Accuracy instead of Worst Accuracy. This might be debatable, as the setting where DFR is applicable assumes an unbiased validation set. Additionally, I wonder if the setup is consistent between ResNet18 in the main paper and ResNet50 in the appendix for the Waterbirds dataset. Could you clarify whether hyperparameters for ResNet18 were tuned based on Mean Accuracy or Worst Accuracy?
- Overall, since most existing papers evaluate their methods on ResNet50 while this paper primarily uses ResNet18 (this makes the numbers not comparable to those reported in other papers; in general, the numbers are much lower for all methods than those in the literature potentially due to the architecture), and some baselines (e.g., JTT, SSA [1], AFR [2], and [3]) are not included in the main table, Table 5, it is hard to compare the proposed method to the state of the art and to assess whether it truly stands out among existing techniques. For example, I wonder if the proposed method can outperform JTT on the Waterbirds dataset when ResNet50 is used, which has a reported worst-group accuracy of 86.7%, a relatively high value compared to the numbers in this paper obtained on ResNet18.
- Perhaps a detailed discussion and comparison with [1] is necessary, given that it also achieves the same goal: lightweight reweighting of the last layer without group labels.


[1] Qiu, Shikai, et al. "Simple and fast group robustness by automatic feature reweighting." International Conference on Machine Learning. PMLR, 2023.

[2] Nam, Junhyun, et al. "Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation." arXiv preprint arXiv:2204.02070 (2022).

[3] Liu, Sheng, et al. "Avoiding spurious correlations via logit correction." arXiv preprint arXiv:2212.01433 (2022).

questions:
See the questions raised in the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695565769

thread_type:
official_review

<thread_object>
creator: reviewer_rgwi

<content>
summary:
Existing methods on mitigating spurious correlations typically requires unbiased data and multiple rounds of retraining. The paper proposes a computational- and data-efficient pipeline that detects spurious features before the last linear layer. When there is no unbiased datasets, the contribution of each feature to the network’s prediction
confidence is exploited for detection.  If the unbiased data is available, the consistency of penultimate activations between spurious and unbiased datasets is used for detecting spurious features.  Through channel-based erasure and re-weighting on the final linear layer, reliance on spurious correlations can be effectively mitigated.

strengths:
- Two practical methods, namely EvA-C and EvA-E, are proposed for detecting and mitigating spurious features in the penultimate activations, depending on the existence of an unbiased dataset. The two proposed methods are both efficient and effective in mitigating spurious correlations.

- The two metrics originate from the spurious feature detection methods can be used to measure the spuriousness of features from the penultimate layer of a model. These metrics are also useful to validate the effectiveness of spurious correlation mitigation methods.

weaknesses:
- Contrary to the claim in Line 93, the performance of the proposed method is not the state-of-the-art in comparison with some methods [1,2]. For example, with ResNet50 as the backbone, EvA-E has a worst-group accuracy of 86.6% (Table 7), while [2] achieves 89.1%.

- The unbiased dataset $\mathcal{D}\_{\text{unbiased}}$ is first introduced in Line 156; however, it is unclear what $\mathcal{D}\_{\text{unbiased}}$ can be considered as "unbiased". Based on the descriptions in Line 230 and Line 312, $\mathcal{D}\_{\text{unbiased}}$ is selected from the validation set. In some datasets, such as CelebA, the validation set also has gender bias. If $\mathcal{D}\_{\text{unbiased}}$ is defined as a set of group-balanced data where each class of samples distribute equally across different spurious features, then group annotations are required to obtain $\mathcal{D}\_{\text{unbiased}}$, contrary to the claim in Line 50-51. Further clarification on this point would be beneficial.

- In Eq. (4), why the unknown test distribution $\phi_{\text{test}}^{ik}$ can be approximated via $\phi_{\text{unbiased}}^{ik}$? In some datasets such as CelebA, the test set is also biased. It would be helpful to further clarify this point.

[1] LaBonte et al., Towards last-layer retraining for group robustness with fewer annotations, NIPS, 2023.\
[2] Li et al., Bias Amplification Enhances Minority Group Performance, TMLR, 2024.

questions:
- Why spurious features often lead to high confidence predictions?
- Can the proposed method used for other model architectures beyond convolutional neural networks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716615110

thread_type:
official_review

<thread_object>
creator: reviewer_feuq

<content>
summary:
The paper introduces a method to detect and mitigate spurious correlations learned by an image classifier. They present two measures to identify channels in the penultimate layer of the model which correspond to such spurious correlations, one of them requiring an additional unbiased dataset (“consistency”) while the other can be computed on the the training data (“evidence energy”). After identifying a set of spurious channels, the corresponding weights in the last layer are set to zero and the remaining weights (of the last layer) are retrained.

strengths:
- computational cost are lower compared to similar methods that do not require unbiased data
- their mitigation outperforms other methods on several common spurious correlation benchmarks
- for the variant including unbiased data, the introduced method requires less samples for the same accuracy
- the procedure seems to be well motivated including theoretical analysis

weaknesses:
- the experiments establishing the correlation between their two measures are based only on one class of one of the datasets
- experiments are only based on small toy settings with strong spurious correlations
- the method is strongly inspired by OOD methods, but apart from a brief mention in the conclusion, this is not discussed in the main paper 

minor:  fontsize in Fig. 1, Tab. 3, Tab. 4 are too small, the definition of “energy” actually corresponds to the “free energy” which might be confusing

questions:
In Table 5, how many runs were performed to compute mean and std, and was the same erase ratio used in all of the EvA runs?

Section 4.4 (and Fig. 3 c) discuss the effect of the erase ratio on the worst accuracy. How does this hyperparameter influence the mean accuracy?

In Table 7, mean acc and worst acc are reported for CelebA but unbiased/conflicting acc in Table 5. The results on Waterbirds suggest that your method results in a better worst acc but worse mean acc for the ResNet-50 than for the ResNet-18: Does this pattern hold on CelebA?

Given the computational efficiency and no requirement for unbiased data or annotations of the spurious correlations, it should be quite feasible to evaluate EvA-E on ImageNet scale spurious correlations benchmarks (e.g. [1],[2]). In these more realistic settings, the distinction between spurious and core features become more complex. Can you provide empirical evidence or arguments that your evidence energy measure can still be used for detection and mitigation in such settings?


[1] Singla et al., Salient ImageNet: How to discover spurious features in Deep Learning?

[2] Neuhaus et al., Spurious Features Everywhere -- Large-Scale Detection of Harmful Spurious Features in ImageNet
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717841453

thread_type:
official_review

<thread_object>
creator: reviewer_jy78

<content>
summary:
This paper presents an approach to mitigate biases learned through spurious correlations by erasing dimensions in the embedding space that are more likely to be associated with spurious features. The authors propose two metrics - namely consistency and evidence energy, for cases when additional unbiased data respectively are and are not available for training, which are used to identify spurious dimensions in the embedding space. Experiments reveal significant improvements over state-of-the-art bias mitigation approaches mainly in the absence of unbiased training data, but also to some extent when it is available.

strengths:
1. The authors propose a post-hoc method for developing robustness to spurious correlations, which does not involve any additional training / heavy computational workload, which is a big positive in my opinion. The method operates by identifying specific dimensions in the embedding space that satisfy certain threshold criteria for some forms of spuriosity metrics proposed by the authors, dropping them from the data representations, and solving a logistic regression from this updated embedding space to the label space. None of these operations seem to be computationally expensive.

2. The authors propose ways to measure the degree of spuriosity of a specific dimension, when additional unbiased samples both are and are not available. This is a useful contribution which may also be applicable in other scenarios beyond bias mitigation.

3. The experimental results are strong. Performance improvements over state-of-the-art are quite significant on standard benchmarks, with minimal extra computational overhead.

weaknesses:
1. The calculation of both consistency (eq. 4) and evidence energy (eq 7) are done independently for each feature dimension $i$. However, it is well known that input features do not necessarily map neatly onto independent dimensions in the feature space [a, b]. Given this, the feature erasure in eq. 9 should technically not be achieving what it claims to achieve. By turning full dimensions on/off, it could be - (i) partially turning off some core features that could be in superposition with a spurious feature; and (ii) not fully turning off spurious features, since it may be in superposition with some core feature dimension which is not turned off.

2. In computing $C_{(ik)}$ in eq. 4, there would be significant disparity in the distribution of the features stemming from sampling bias, since the number of unbiased samples is typically << the number of biased samples in practical datasets, which itself could skew the value of $d$, not providing a faithful measure of consistency. How do the authors account for this imbalance when applying the distance metric $d$?

3. Since the idea of EvA is essentially to remove dimensions from the embedding space that correspond to spurious features, an ablation for what constitutes the best approach for selecting the subspace to drop is necessary. EvA employs thresholding on consistency and evidence energy to select spurious subspaces. However, whether consistency and evidence energy are really the best metric to look at when it comes to performing this selection needs to be either theoretically or empirically established. For this, one empirical baseline could be comparison with random feature drop out in the embedding space. I would suggest the authors to explore this in further detail and consider developing such baselines to establish the optimality of their specific feature erasure mechanisms.

4. Table 5 shows that the gains from EvA are more significant in the absence of extra unbiased training data. The authors should provide a discussion on why their method might have a stronger advantage over SOTA when unbiased training data is not present.

5. When extra training data is not available, how does one perform the hyperparameter search for $\epsilon$? Additionally, when training data is available, although a single pass of EvA is computationally inexpensive, the hyperparameter sweep over $\epsilon$ may be time consuming. Discussing these, especially, the former case of selecting $\epsilon$ in the absence of an unbiased train subset is necessary.

[a] Elhage, et al., "Toy Models of Superposition", Transformer Circuits Thread, 2022. \
[b] Radhakrishnan et al., "Mechanism for feature learning in neural networks and backpropagation-free machine learning models", Science 2024.

questions:
Please see the Weaknesses section.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes EwA, a feature selection method for removing reliance on spurious correlations. The method builds on the observations from [1] that information about both core and spurious features is contained in the penultimate layer representations. The authors propose a simple rule for deciding on which features to drop based on (1) difference in feature distributions between train and unbiased validation (when available) or (2) influence on model confidence. The authors show theoretically in a simple setting that these choices are justified. Then, they show that the proposed method performs strongly on benchmarks, especially when the unbiased rebalancing dataset is small or unavailable.

Strengths:
- The method is very simple and makes intuitive sense.
- The method is computationally very cheap.
- The method shows especially strong performance in the setting where the rebalancing data is extremely limited.
- The authors provide theoretical motivation for the proposed method.
- The authors provide detailed ablations on hyper-parameters, as well as understanding experiments showing a correlation between the two feature selection criteria proposed.
- The method performs well across a broad range of tasks.

Weaknesses:
- The method assumes that the "true features" are aligned with dimensions of the penultimate layer representations. This is known to not be true in the context of LLMs [2]. However, this may still be true in image classification models such as ResNet, considered by the authors.
- The feature selection for the setting where rebalancing data is not available relies on the standard assumption that spurious features are the most predictive in the classifier. While true on benchmarks, this may not hold on more realistic problems, where spurious features may be weaker than core features.
- The authors only evaluate the method on image classification tasks. The standard set of evaluations in spurious correlations literature typically also includes CivilComments and MultiNLI NLP datasets. It would be interesting to see results there. Generally, it would be nice to expand the set of evaluation datasets.
- The main results table 5 uses a non-standard setting with ResNet-18 instead of ResNet-50. The authors mention that on ResNet-50, with the full reweighting dataset, the method does not outperform DFR [1], but in Table 5 it does. It may be better to report the results in the standard evaluation setting in the main text.
- Reviewers highlighted some related work that should be included in the paper.

Decision recommendation: The paper proposes a very simple and intuitive method that works well in practically relevant settings. The strong performance when the rebalancing dataset is small is very promising. I believe this paper makes a good contribution and recommend to accept it.


[1] Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations
Polina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson

[2] [Toy Models of Superposition](https://transformer-circuits.pub/2022/toy_model/index.html)