Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work shows that neural network models trained on datasets affected by the presence of simple spurious correlations also learn “core” features (i.e., features that are truly discriminative and non-spuriously correlated with the training labels). Given this observation the authors propose an algorithm named “Deep Features Reweighing” (DFR) that consists of retaining only the last layer using a “clean” validation set (i.e., a set that is not entirely affected by the spurious correlation present in the training set). 

Results are shown on Waterbirds, CelebA, MultiNLI and CivilComments datasets and performance are compared to different states of the art algorithms (Group DRO, JTT, CnC, SUBG, SSA). The results presented show that DFR outperforms many state of the art techniques in terms of worst-group and mean accuracy, and it is on-par with Group DRO. Unlike Group DRO, however, DFR does not require the training set to contain group-information (while this information is required for the validation set for both algorithms).
strength_and_weaknesses: Strength
- The paper is well written and tackles an important problem: the effect of spurious correlation on ML models.
- The analysis is interesting.
- The algorithm is simple yet effective.

Weaknesses
- The biggest weakness in my opinion is the similarity with the idea presented in “DECOUPLING REPRESENTATION AND CLASSIFIER FOR LONG-TAILED RECOGNITION” [Kang2019], concretely the algorithm that Kang et al. refer to as Learnable Weight Scaling (LWS). The authors of the current paper did cite this work and state that the two works focus on different settings. While this might be true (as LWS focuses on long tail while DFR focuses on spurious correlation) in practice the algorithms seem identical. If I am correct, then the novelty of the current paper is mostly in the observations and analysis rather than in proposing a “new” algorithm. This should be stated and in such case there would be no need to introduce a new “name” for an existing algorithm. If the algorithms are similar, but not identical, then the difference with LWS should be described in more detail and the empirical comparison should appear in Table 2.
- Ablation studies are missing and they could make the paper stronger and the finding more insightful, especially given the similarity with [Kang2019] additional ablation studies would increase the contribution. See below for some suggestions.
- Some statements might need to be rephrased to represent the technique and the results more candidly. Se below for some suggestions.

Details
1. “In order to make use of more of the available data, we train logistic regression 10 times using different random balanced subsets of the data, and average the weights of the learned models”. If averaging the model leads to important gain and it is what it is mostly reported in the experiments than it should described in Figure 1 and in the intro. This is actually a missing ablation study to understand the contribution of the weighting vs the simple rescaling. How much of the gain is related to the averaging of multiple models and how much is the use of a balanced unseen dataset? 
2. The authors say “we initialize the model with weights pretrained on ImageNet”. This naturally makes one wonder if the presence of core features is “just” due to this pre-training or if a model will learn core features even without ImageNet pre-training. It seems that this pre-training affects experiments in section 4.1 but possibly not 4.2? If this is the case I’d invite the author to state this important difference in the main paper (rather than differ this detail to the appendix). I’d be curious to see the results of 4.1 without pre-training, if confirmed I think the message would be stronger.
3. The conclusions from 4.1 are too general given the pre-training step “we conclude that while the models trained on the Original data make use of background information to make predictions, they still learn the features relevant to classifying the birds almost as well as the models trained on the data without spurious correlations”. Without proving the point above (what happens when one trains only on the dataset affected by spurious correlation - without pre-training) this should be re-stated as “we conclude that while models that are pre-trained on ImageNet and fine-tuned on the Original data make use of background information to make predictions, they still learn the features relevant to classifying the birds almost as well as the models trained on the data without spurious correlations”
4. Related to the above point. When using a model pre-trained on imagenet, what happens if one only fine-tunes directly on the validation set (skipping the training on the larger but biased set)?
5. “For all DFR variations, the size of the reweighting set Dˆ is small relative to the number of features”  could be stated more precisely: what does small mean in this context?
6. I suggest to de-emphisize if possible  DFR^train_train. It is a good experiment to show the need for the use of a new set of data. However, It is a bit distracting when read in the current order. I’d consider introducing it after the main result, as an ablation study on the use of the new vs old data. Currently it is presented at the beginning and it creates the expectation for this to work comparably to D^Val_Train.
clarity,_quality,_novelty_and_reproducibility: The paper is well written. All the details seem to be present for reproducibility although many are deferred to the appendix. As mentioned above, some statements might need to be rephrased to represent the technique and the results more candidly. The novelty seems limited to the observation rather than a new algorithm. That is still a valuable contribution but it needs to be presented accordingly.
summary_of_the_review: The work is very interesting and it is worth sharing the findings with the community. While the novelty is limited (see [Kang2019]) the analysis are interesting. The paper needs, however, to either highlight the differences with [Kang2019] or change the focus to be on the application of a known technique (LWS) to this problem (and on the analysis already provided) rather than on presenting a new algorithm. I would be happy to upgrade my rating if the authors could provide an explanation about why DFR is different than LWS (including comparison) or change slightly the focus as just mentioned.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a surprisingly simple approach for mitigating spurious correlation via linear probing on a balanced dataset. The paper conducts comprehensive empirical studies on worst group robustness benchmarks and ImageNet variants. 
strength_and_weaknesses: Strengths 
- The proposed idea of retraining the linear head is simple and natural
- The paper conducts extensive well-designed studies and the method demonstrates very competitive performance compared to recent works on worst group robustness. 

Weaknesses
- The phenomenon that pre-trained networks still learn causal features even under severe spurious correlation in the training set is surprising. No theoretical explanations are provided to explain the phenomenon. 
clarity,_quality,_novelty_and_reproducibility: The paper is well written. Most sections are easy to follow and understand.  The experiments are well design and provide sufficient details on the setup for reproducibility. 
summary_of_the_review: The paper provides a simple method for mitigating spurious correlation. Although no theoretical analysis is provided, the paper conducts comprehensive empirical studies and is well written.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the concern that neural networks trained for classification exhibit a bias towards learning spurious correlations that can negatively impact generalisation. In particular, this paper empirically demonstrates that despite learning spurious features and assigning them larger weights, neural networks can also learn more complex features simultaneously. Based on this observation, the authors propose a method, named Deep Feature Reweighting (DFR), to retrain the classification layer of a deep neural network with a small, controlled, balanced data set to reweight the importance of spurious and complex features.
strength_and_weaknesses: ### Strengths

This paper is conceptually simple, therefore easy to follow, and it is also mostly well-written. As an important strength, I highlight that the paper provides convincing empirical evidence that neural networks trained with empirical risk minimisation (ERM) are able to learn feature extractors of complex features, even in the presence of strong correlation of the labels with spurious features and the weights of the output layer mostly depend on the latter. In particular, the evidence come from two results: one, that such models obtain good (but far from optimal) performance on test sets without spurious correlations; two, that reweighting the output layer by training on a balanced data set achieves near optimal results. I think this strong evidence sharpens our understanding on the representation learning of neural networks trained with ERM and may inspire interesting avenues for future improvements on the generalisation of deep learning methods.

### Weaknesses

In view of what I consider the main strength of the paper, it is also fair to highlight that this result is not completely novel, since it has been already suggested by Rosenfeld et al. (2022), a paper that the authors cite and briefly summarise. I belief that the overlap is substantial enough to deserve a more detailed discussion of the similarities and differences between the two works.

Besides this comment on the novelty, in my opinion the paper places much importance on the proposed method to retrain the last layer of a pre-trained model by using a relatively small but controlled and balance data set. While the method proves to be effective at reducing the reliance on spurious features at the classification layer, the wide applicability of the method (and other related methods) is questionable, in that it requires both prior identification of the spurious features, as well as access to a balanced data set for re-training. I would argue that in general the main problem with spurious correlations is that they may be very hard to spot or identify a priori.

Related to the previous concern, in Section 4.2, the authors propose to use the "decoded accuracy" (accuracy after re-training the classification on a group-balanced validation set) as a way to analyse the reliance on "high-quality representations" (as opposed to the accuracy on the core-only set). However, how can the authors guarantee that the balanced validation set does not contain other spurious correlations (as in the example of the zero-vs-ones provided in the paragraph)?
clarity,_quality,_novelty_and_reproducibility: I have already commented on the novelty of the work in the previous section. In a nutshell, there is significant overlap between this work and recent work by Rosenfeld et al. (2022), as well as with, to a lesser extent,  Kang et al. (2019). While to me the similarity does not invalidate this work, I would argue that a more detailed discussion of overlap would make the paper stronger and more comprehensible within the related literature.

Regarding clarity, the majority of the paper is clearly written and easy to follow. However, I noticed a decline in the clarity of the text in Section 7, where the paragraphs become denser and the results harder to interpret. For instance, Table 2 is hard to read due to the large amount of numbers in the table. Furthermore, although I did not carefully reviewed the whole appendix, I also found it substantially less clear than the main paper. Finally, I would like to mention that the mixture of different data sets and methods in different sections of the paper also impacts the clarity of the paper. In order to help the reader navigate the paper, I would consider including a summary figure of table of the experimental setup.

The appendix contains details about the experimental setup, in terms of reproducibility, but I am not aware of any mention of the availability of code and data to reproduce the results.

I judge the quality of the paper as sufficient, in terms of the methods chosen and designed to evaluate the hypothesis, and in my opinion the conclusions are supported by the results.
summary_of_the_review: My overall impression of this paper is positive, since I believe the results provide strong evidence of the capability of neural networks to learn complex representations, even when the signal of spurious features is highly correlated with the labels. While this has been shown before and future work should shed more lights on the conditions when this is the case, among other open questions, I believe that this paper advances our understanding of the representations learnt by neural networks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of spurious correlations. Authors propose to improve robustness against spurious correlations through a simple approach (Deep Feature Reweighting): specifically re-train the last layer of a neural network using a small set of reweighting
data where the spurious correlation does not hold. Further, they also show that despite relying on spurious correlations, the model still learns the core/invariant features. Empirically their proposed method achieves state-of-the-art performance.
strength_and_weaknesses: Strength:

1. The paper is well-written and easy to follow. 
2. The proposed approach is simple to implement yet effective in combating spurious correlations.
3. Experimental analysis in Section 4 is interesting and provides new findings.
4. Empirically achieves SOTA performance.

Weaknesses:
1. Requires access to group-labeled samples. Although Table 5 shows it is a common assumption in the literature.

Q1. It is difficult to understand why using group-labeled samples from training data performs worse than using group-labeled validation data. Are the number of samples in different groups used for reweighting kept similar for both DFR_Tr and DFR_val ?


clarity,_quality,_novelty_and_reproducibility: The clarity and representation of the paper are good. The study is supported by detailed evaluations and also analysis. The findings are also novel according to me. 
summary_of_the_review: The paper is well-organized, well-motivated, and supported by empirical evaluations. Hence, I recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors build on the fact that neural network classifiers can largely rely on simple spurious features to make their predictions and show that, contrary to common belief, even in these cases, the networks still often learn core features associated with the desired attributes of the data. The work shows that a simple last layer retraining can match or outperform state-of-the-art approaches on spurious correlation benchmarks, at a much lower complexity and computational requirements.
The work is novel and well received by all reviewers, it clearly makes a nice addition to ICLR.
justification_for_why_not_higher_score: I have another paper in the batch that has better scores and that I would consider more interesting for the general audience but happy to discuss of course.
Decision: Accept: notable-top-25%