Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem that deep neural nets trained for classification learn spurious correlations that can negatively impact generalization. For this purpose, the authors propose Logit Correction (LC) loss to mitigate the effects of spurious correlations. Further, to synthesize more samples from the minority groups, the authors propose Group MixUp.

strength_and_weaknesses: Strengths: 
1. The proposed method is simple to implement.
2. The authors provided theoretical justification.
3. The study is supported through detailed empirical analysis and ablations

Weakness 
1. I believe the proposed approach shares some similarities with LfF which also uses a two-branch network with generalized cross-entropy loss for mitigating spurious correlations. The authors should provide a detailed discussion of the difference between the two papers.

2. The average accuracy is not reported in Table 1 and 2. Further, the authors also do not report variances over multiple runs, which slightly diminishes the trustworthiness of the results.

3. According to me, writing can be improved. There are some grammatical mistakes that make it hard to understand the concept in some places. (Minor)

Questions:
1. I am confused about how you get the samples from minority groups while doing GroupMixup if the group labels are not assumed. Further, to the best of my knowledge, both JTT and LfF use group-labeled validation sets for hyperparameter tuning. It's not very clear why the authors indicated that they do not require group information. 

2. Can this proposed method be also applied for spuriously correlated NLP datasets such as MultiNLI and CivilComments. It would be interesting to see the performance.


clarity,_quality,_novelty_and_reproducibility: Although the paper provides some insights and good empirical evidence, the clarity can be much improved. I shall suggest shifting the algorithmic block to the main text to better motivate the algorithm. I believe the work is somewhat related to LfF. However, the technical contributions are novel to the best of my knowledge.
summary_of_the_review: Although the contributions are interesting, according to me the paper needs to be refactored to improve the flow and in its current shape, the overall impression is not positive.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors tackle the spurious correlation problem in the setting where spurious attribute values are unknown. They propose a method where a two-branch neural network is trained, one branch with the generalized cross-entropy, and the other with a logit correction loss which depends on estimated class priors computed from the first branch. Unlike prior two-stage methods like JTT, the proposed method only requires training a single network. The evaluate their method on standard spurious correlation datasets, finding that it improves over the baselines.
strength_and_weaknesses: Strengths:
- The paper is generally well-written.
- The proposed method outperforms the baselines on standard benchmark datasets.


Weaknesses:
1. The main weakness of the paper is the novelty of the proposed method, which is limited by two factors:
- The proposed group MixUp method is nearly identical to intra-label LISA [1], which the authors do not reference. It is also similar to [2], which I do not expect the authors to have referenced as it is too recent.
- The GCE loss is the same as the loss used in training the biased model in Learning from Failure, which the authors also do not mention.

2. It seems that, for the method to work, the user needs to know in advance which of the four spurious attribute to label mappings is present within the data. In particular, in Sections 4.2.2-4.2.4, it seems that we would need to know the values of the spurious attribute that are correlated with each value of the label. If so, this seems like a major drawback of the method. The authors should discuss this further, and perhaps conduct experiments in the case where the mapping is misspecified (e.g. we assumed it was one-to-one but it was actually many-to-one).

3. The authors should show the performance of the estimation of the group prior over training. Does this converge to the real values?

4. The authors should evaluate their method on the CivilComments dataset, which is standard in the spurious correlation setting. This might be a tricky setting for the method to define the spurious attribute to label mapping.

5. For transparency, the authors should show the performance of more recent methods such as CNC [3]. They should also show the performance of all methods on all datasets (for example, JTT is missing in Table 1, and there are a couple missing from Table 2). 

6. The authors should conduct an experiment showing the effect of varying $q$ in the appendix. Why have they been set to 0.7 and 0.8?

[1] https://arxiv.org/abs/2201.00299
[2] https://arxiv.org/abs/2209.08928
[3] https://arxiv.org/abs/2203.01517
clarity,_quality,_novelty_and_reproducibility: The paper is generally clear and well-written. The logit correction loss function is novel in this setting to the best of my knowledge, though the originality of the paper is limited by weakness #1 above. The reproducibility of the paper is unknown as the authors have not included their code in the supplementary, though they do provide implementation details in the paper and Appendix D.
summary_of_the_review: Due to inherent issues with the method (Weaknesses #1-2) and issues with the empirical evaluation (Weaknesses #3-6), I recommend rejection at this time pending the authors' rebuttal.

Post-rebuttal update: The authors have addressed my major concerns with the empirical evaluations through new experiments. I am now leaning towards accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of spurious correlations in deep learning, using image classification as a case study.
This problem arises when some attribute can be observed in the training set that is correlated with the class label, however the correlation does not hold in general.
The formulation assumes that there is one discrete spurious "attribute" variable, for which we know (or can determine) the values that frequently co-occur with each class.
In practice, this is generally the possibly-incorrect class predicted by a model trained with an ERM loss.
The proposed method comprises two components: Logit Correction loss and Group Mixup.
Similar to logit correction for class-imbalanced learning, the LC loss seeks a greater margin for examples in the less frequent groups, where groups are defined as a (class, attribute) pair.
Group Mixup replaces each minority example with a convex combination of itself and a majority example with the same label, mixing both the inputs and the logit corrections.
The method is compared to several baselines on both synthetic and realistic datasets.
strength_and_weaknesses: **Strengths**

1. Based on the work of Menon et al., logit correction seems like a highly effective tool for enforcing different margins from different classes in deep learning. Using this to address spurious correlations makes sense.

1. Both Logit Correction and Group Mixup made a significant contribution to the improvement in results.

1. The inference rule in Proposition 1 was unfamiliar to me and the derivation seems good.

1. Empirical evaluations show the method to be highly effective across all datasets.

1. Algorithm 1 helps to understand the method.

**Weaknesses: Experimental**

1. The ablative study was only conducted for one dataset. It would be better if these 3 ablations were added as rows in Tables 1 and 2 for all datasets.

1. DFA missing from Table 2, JTT missing from Table 1.

1. No confidence intervals (standard deviations) in Tables 1 and 2, even for experiments on relatively small datasets (MNIST, CIFAR). This could be especially important for the "Worst Group" metrics, which would probably higher variance than the mean-based metrics.

1. It is stated that the ERM results in Tables 1 and 2 were taken from past papers. To ensure an apples-to-apples comparison, it would be better to run at least the ERM baseline in the exact same pipeline, only disabling your features. (Like the first row of Table 3, but for all the datasets in Tables 1 and 2? Note that Table 3 differs from Table 2, i.e. 56.87 vs 55.47 for MGA with bFFHQ.)

**Weaknesses: Method and text**

1. Was it not necessary to stop the gradients from the ERM branch to the Robust branch?

1. For continuous attributes, it seems like the LC loss could still be used, but Group Mixup would not be directly applicable.

1. The use of the predicted label as the correlated attribute is quite a specific choice (line 6 of Algorithm 1 in appendix). This attribute will always be one-to-one, with ground-truth and predicted labels coinciding in the majority groups. I feel like this particular choice should be given more attention in the formulation. Otherwise, the method seems much more general than the empirical evaluation, since IIUC all experiments in the main paper use this specific configuration. (Only Appendix E contains results that do not.)

1. When the attribute is not the predicted label, it seems necessary to specify which attribute values are correlated with which class labels (Sections 4.2.2, 4.2.3). This is not necessarily a weakness, as sometimes it may be useful to specify this manually. However, it seems that it might also be automatically determined from a class-attribute confusion matrix? How would the accuracy be affected if it was specified incorrectly?

1. It's not clear whether logit correction being more effective than re-weighting/re-sampling is specific to deep learning. For example, Kang et al. (ICLR 2019; "Decoupling Representation and Classifier for Long-Tailed Recognition") found that it was best to learn features using instance-balanced sampling and then learn a linear classifier using class-balanced sampling. Does the LC loss still improve the results when training a linear classifier with frozen features?

1. It is hypothesized that the effectiveness of the LC loss is due to being Fisher consistent. However, it seems that re-weighting samples rather than adjusting the margin might also be Fisher consistent (w.r.t. the group-balanced distribution)? If this is not the case, can you state this in the paper and provide a reference?

**Uncertain**

I'm not an expert on spurious correlations, so I'm not sure whether the choice of datasets is appropriate, in particular the use of datasets with only two classes. I'll defer to other reviewers here.
clarity,_quality,_novelty_and_reproducibility: In general, the paper and appendix are both quite clear.
I believe that the technique is sufficiently novel and that necessary details have been provided to ensure reproducibility.

Will the code be made available after acceptance?

**Nitpicks**

* I found the short-hand in eq. 2, $\mathbf{P}_{\mathbf{x}|(y, a)}(\cdot)$, much less clear than the full expression in the appendix.
* Unclear without Fig. 1 whether GCE loss is applied after softmax.
* It seems like $f^*$ is introduced and defined in equation 4, however this is not clear.
* The set notation in Algorithm 1 seems incorrect (e.g. for-loops, shuffle).
* LC was described as Logit Correlation in one place (not Correction)
* Some grammar issues (associates, assumes, exist)
* Some verbs mis-used (cooperating, explored)
summary_of_the_review: Overall, the approach is well-motivated and the results seem strong. However, I have some concerns about the justifications and the evaluation. Also, it seems that all experiments in the main paper use a particular choice of attribute, and are thus less general than the formulation. I'm leaning towards accept, but I may downgrade my rating if my concerns are not addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a loss function for classification that allows to mitigate the issue of spurious correlations in data where there is a majority group in which correlations are present, and a minority group without spurious correlations.  

The proposed method consists of training two networks: 1) a ERM network that uses the generalized cross entropy loss, and that is therefore biased to be better at predicting elements containing spurious correlations (in the majority group), 2) a robust network trained with the standard cross entropy loss, but after having scaled the logits by a correction term produced from network 1. 

Experiments show that the method is able to reduce geometry and statistical skews, and outperforms competing method in a number of standard benchmarks. 

 
strength_and_weaknesses: **Strengths**
1. Spurious correlations are very common in real data. This paper proposes an interesting solution to learning classifiers that can avoid them, importantly without assuming that the spurious attribute is known during training 

2. The LC loss is theoretically grounded, since as the authors show minimizing the logit-correction loss is equivalent to maximizing the group-balanced accuracy 

3. The proposed method outperforms competing ones in popular benchmarks for the task. Ablation studies show the importance of all the novel components introduced in the paper. 


**Weaknesses**
 
_Clarity_ 

The logic correction term is the key ingredient of the proposed method. As such, from section 4.1 it should be obvious and very clear to the reader how the correction term is computed and why it works, which I believe it is not the case. 

1. While Equations (4) -> (7) provide all the mathematical steps in detail, the authors should provide the reader intuition on what each of those equations represent. I really liked the simple waterbird example from Figure 1, and I believe it could be used as a running example throughout section 4.1 to give a more concrete intuition of the formula and assumptions presented in the section. 

2.  $\Delta_{y, a_x}$ is never even defined explicitly with a formula. 

3. Section 4.2 would also greatly benefit from providing some intuition to the reader 


_Applicability to more realistic datasets_

1. While the authors perform experiments on standard benchmarks, I would have liked more real-world experiment, to make sure that the proposed method generalizes to more realistic tasks where the spurious attributes might be very subtle and hard to infer (as is the case for example in medical imaging datasets). 

2. I have some doubts that the linear combination assumption of the group mixup will work in a real dataset with higher dimensional images and objects that might have different scale/positions. Would you expect it to generalize well to more realistic tasks? 

 

 
clarity,_quality,_novelty_and_reproducibility: I found the paper to provide an interesting solution to an important problem in real world ML applications. To the best of my knowledge the presented work is novel.

As discussed in the previous section, the clarity needs to be improved.

summary_of_the_review: The presented method is interesting and has potential, but in its current state the impact of this paper is limited by the lack of clarity in the theoretical section.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the problem of spurious correlation. In particular, authors propose Logit Correction (LC) loss to mitigate the effects of spurious correlations. Further, to synthesize more samples from the minority groups, the authors propose Group MixUp. Reviewers generally liked the problem studied in the paper but clarity of some parts of the paper needs to be improved significantly. AUthors did a good job in the rebuttal to address some other concerns by the reviewers. 
justification_for_why_not_higher_score: The paper clarity needs significant improvement however given that all reviewers accepted the paper (borderline), I'll advocate for an accept.  
justification_for_why_not_lower_score: All reviewers recommend accepting the work; I think it is an important problem studied in the paper
Decision: Accept: poster