Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of spurious correlations. Specifically, the authors argue that the assumption that high-loss samples correspond to a lack of spurious cues is not always true. To this end, the authors show theoretically that using predictive uncertainty to identify spurious-cue-free samples is a better indicator. Based on this finding, they propose Entropy based Debiasing (END) to mitigate the problem of learning spurious correlations.
strength_and_weaknesses: Strength:

1. The paper is well-written and easy to follow in general.
2. The authors provide theoretical justification for their findings. Although I did not check through the theorem-proof, the findings are interesting.
3. The findings in Section 4 are interesting.

Weaknesses and Questions:

1. The comparison in Table 1 can be made more comprehensive. I suggest comparing with a few recent works such as SSA [1], and CnC [2]. Further, for CelebA, I see a large decline in the average accuracy, although the improvement in worst-group is marginal compared to JTT.
2. Is there any table reporting the accuracies of the identification model? I also wonder if the training time of END is significantly higher than the other compared methods.
3. It would be interesting to see the performance of END against spuriously correlated NLP datasets such as CivilComments, MultiNLI.

[1] Nam, J., Kim, J., Lee, J. and Shin, J., 2022. Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimation. ICLR 2022.

[2] Zhang, M., Sohoni, N.S., Zhang, H.R., Finn, C. and Ré, C., 2022. Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations. ICML 2022.
clarity,_quality,_novelty_and_reproducibility: In general, the paper is clear and easy to understand. To the best of my knowledge, this is a novel work to some extent. The quality of this paper can be further improved through better empirical analysis, which at the moment is not quite comprehensive and significant.
summary_of_the_review: With the findings above, I currently give the paper a borderline reject score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a predictive uncertainty based method to identify spurious-cue-free samples from a training set with noisy labels. Then a new training dataset is by oversampling the SCF samples to train the debiased model. Separating minority group samples and noisy samples is indeed a trickier problem. But this paper seems to solve this problem well.
strength_and_weaknesses: Strengths:
This paper is well-written and easy to follow. The main idea and modtivation is well-presented. 
Weaknesses:
1. I don't think some of the author's claims are accurate enough. For example, the authors argue that an advantage of the proposed model is that it can be extended to regression models. However, other methods such as GroupDRO or JTT can also be extended to regression models with simple modifications.
2. The baseline used in the paper is much lower than that in the JTT paper. For example, on the Waterbirds dataset, the worst-case acc reported by the JTT paper is 86.7% and in this paper it is 84.6%.
3. Some very relevant other papers have not been cited and discussed[1][2]. [1] distinguishes noisy samples in long-tail data, which is relevant to the task of this paper.
[2] also uses uncertainty to solve the subpopulation shift problem. The way it builds uncertainty is similar to the snapshots mentioned in this paper.
[1] Identifying Hard Noise in Long-Tailed Sample Distribution
[2] UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixup
Minor:
At the end of page ten, citations to the paper "Enhancing the reliability of out-of-distribution image detection in neural networks" appear twice.

clarity,_quality,_novelty_and_reproducibility: This paper is well-written and easy to follow. The method is also interesting.
summary_of_the_review: Overall I think this is a good paper, but it can still be improved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose to leverage the predictive uncertainty to build an identification model, thereby discovering SCF samples and further improving the group robustness under the noisy label environment. They first demonstrate some theoretical justifications about how SCF samples can be obtained through computing predictive uncertainty. To eliminate the drawback of simply using ERM, the authors further propose the END framework. To identify the SCF samples, they train a Bayesian neural network based on MAE and confidence regularization. The debiased model can then be trained by considering both SCF samples and the original dataset. Experiments are conducted on a 2-d synthetic dataset, two group robustness benchmark datasets, and two DTA regression datasets. The experimental results demonstrate that the proposed framework END can obtain better WG accuracy and MSE. 
strength_and_weaknesses: Strengths:
* Practical problem settings for noisy label environments.
* Theoretical analysis on the proposed idea of leveraging predictive uncertainty to improve group robustness.
* Experiments on both synthetic and real-world benchmark datasets with improvements over baselines.

Weaknesses:
* The proposed method does not consider the characteristics of uncertainty like aleatoric and epistemic uncertainty. This could be important to figure out whether the uncertainty on specific predictions identifies the proper SCF samples.
* Although the proposed framework outperforms most of the baseline methods in worst-group accuracy, it generally performs worse than baseline methods in average accuracy, especially when the noises are fewer. The authors should elaborate on this phenomenon and justify the trade-off sacrifice.
* The paper lacks qualitative analysis. For example, the authors should demonstrate some examples of identified SCF samples.
* The authors do not provide the material for reproducibility or mention further plans. 

clarity,_quality,_novelty_and_reproducibility: * For clarity, the paper is well-written and easy-to-follow.
* For novelty, the authors propose an interesting and novel idea, but the technical quality is not very satisfactory in terms of experiments and diving deep about uncertainty. 
* As mentioned in weakness, the paper does not provide the information about reproducibility.

summary_of_the_review: In sum, I would recommend “3: reject, not good enough” because the paper is not ready for both of the framework properly using uncertainty and satisfactory experimental results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to improve the worst-case performance in the present of label noise. Because typical high-loss-based approaches for atypical groups of the data may fail under this setting, the paper introduces an uncertainty-based approach to identify spurious-cue-free (SCF) samples, and then trains a model with the SCF set oversampled dataset. This paper tries to theoretically prove the uncertainty is a proper indicator of the SCF samples, and experiments show effectiveness.
strength_and_weaknesses: Strengths:
1. The work focuses on addressing an important and challenging problem. The motivation behind the design choices is clear. All the related works are properly addressed.
2. Very simple method, which can be useful for practitioners.

Weaknesses and questions:
1. This work proves the predictive uncertainty is sufficient to identify SCF samples with label noise by assuming a very specific setting, and that the proof only considers the binary classification problem, while it remains unclear, even intuitively, how to extend it to true multi-class classification case.
2. One key to the problem is to distinguish between SCF samples and noise samples. Assume that the proof of the paper is correct and general that samples with high uncertainty are SCF samples, while samples with high uncertainty/large losses are also commonly considered to be noisy samples in noisy label learning. My question is what is the relationship between high entropy and large loss?
3. Some of the details about theoretical formalism are not clear to me. Equation 4 is difficult to understand and needs further explanation, and why beta in Theorem 1 is the solution of the regression? 
4. For preventing overconfidency of networks, Section 5.1 proposes to use a regularizer to encourage a smaller predictive entropy, but the predictive entropy is also an indicator to discriminate SCF samples, so the reliable entropy is also desirable. How to trade off these two points? 

clarity,_quality,_novelty_and_reproducibility: The paper is well-motivated. However, some unclear descriptions hinder understanding of the paper fully. 
summary_of_the_review: I am not sure whether the proposed method is well supported. If the concerns are properly addressed I am willing to increase my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a technique for coping with spurious correlations in the presence of label noise. A key observation is that prior works that rely on identifying high-loss samples may fail in such settings. 

All reviewers raised critiques of the work, primarily around apparently poor performance on noise-free settings compared to prior work, missing baselines for the empirical comparison, missing citations to relevant prior work, unclear details in the theory, and limitation of the proof to the binary classification setting. The author response clarified some points, but reviewers still maintained concerns on the first and last point in particular.

From my reading, I concur with the reviewer sentiment that the work could have some interesting ideas, but the current execution falls a bit short. We thus do not think the paper suitable for publication at present, but certainly encourage the authors to submit their work to some future venue.
justification_for_why_not_higher_score: As above, the reviewers noted multiple critiques of the work, primarily around apparently poor performance on noise-free settings compared to prior work, missing baselines for the empirical comparison, missing citations to relevant prior work, unclear details in the theory, and limitation of the proof to the binary classification setting. The author response clarified some points, but reviewers still maintained concerns on the first and last point in particular.
Decision: Reject