Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Summary: This paper studies the Out-of-Distribution Generalization classification problem. And it test several simple reweighting methods on several benchmarks. 
Main review: This paper conducts some experiments on the OOD generalization classification tasks. The authors test some sample reweighting methods on four benchmarks and find that simple data balancing baselines can achieve competitive performance. However, the technical contributions of this work are limited since it does not propose any new methods or theoretical analysis, for which I think this paper does not meet with the bar of this conference.
review_title: null

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: An insightful paper comparing important baselines to recently proposed approaches such as JTT and DRO
Main review: Quality: The paper studies the problem that often arises when the data has groups/attributes that can be in the minority and often times get sacrificed at test time as the model can learn to rely on spurious features that work well for majority group at train time. The problem has gotten quite some attention after the work of gDRO from Sagawa et al. (2019) and IRM Arjovsky et al. (2019). 
The paper carries out a careful empirical comparison of natural baselines that can be divided into two categories one that access group information and others do not access group information to recently proposed JTT and gDRO. 
I found the results of the paper quite insightful, which I summarize below.
1. If we do not have access to attribute information, then rebalancing using class label is the go to method and not JTT.
2. If we have access to attribute information, then rebalancing using group label is approximately as good as gDRO barring some cases. Given its other benefits, it should still be off the shelf method to use in these settings. 
3. Very interestingly, in the hardest problem when attribute information is also not used for validation then the benefit of using these simpler baselines is even more pronounced. 
4. Also, in the hardest problem it seems when there is no attribute information (at train or validation time) and we are comparing RWY, SUBY, ERM, and JTT the winner seems to be SUBY.   Hence, the conclusion I draw is that if there is class imbalance, then SUBY is winner for worst group accuracy. So if there was no class imbalance, then SUBY may be still similar to ERM?
5. I would like to ask authors if they have intuitions or comparisons for the setting when there is no attribute information at train time, then methods like https://openreview.net/pdf?id=YygA0yppTR or https://openreview.net/pdf?id=b9PoimzZFJ seem to have shown impressive results. I ask this because authors conclude that it seems it's quite hard to beat these simple baselines. 
6. Also, the authors at the end ask if the idea of worst group accuracy a good one. Does it not fundamentally boil down to asking if there is one classifier that can achieve a reasonably good performance (not far from their individually optimal ones) for all the groups? I believe in water birds and celebA this should be the case but I am not sure about civil environments and multiNLI dataset. A proxy for checking this can be that train a classifier separately on each group and find the individual optimal performance. Find how the classifier performs when transferred to other groups, if the gap is large then perhaps there is no common classifier that performs well across the groups. If the gap is small, then there exists a good classifier and thus worst group accuracy is meaningful. I wonder if authors have similar intuition?

Clarity: The paper is very clear.

Originality: The paper's contributions are new. The methods are natural and important baselines.

Significance: The paper is quite important.
review_title: null

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: Empirical results showing that simple reweighting techniques can perform comparably to sota methods for worst-group accuracy.
Main review: The paper presents some thought-provoking results. If the goal is to do well on worst-group accuracy, then simple reweighting techniques can do as well on benchmark datasets.The set of datasets is comprehensive  and proposed techniques are intuitive enough. I like the direction of this work. 

However, some of the results are underwhelming, and I feel that the paper can be improved by more theory/conjecture/analysis that explains the results. 
1. In table 2, gdro is better on multi-nli. Why? What is it about this dataset that makes reweighting less effective? Same for suby on civilcomments. What can we learn from these exceptions?

2. Given that there are exceptions, should we still use the complicated methods? Under what conditions would reweighting be ideal? This can be answered by either extending the simulated toy dataset or by theoretical analysis on a simple setup.

Overall, I'm unclear what to take away from this paper. Is the proposal that we should adopt reweighing techniques? or that the datasets needs to be more complicated? If the former, you need to explain the exceptions in Table 2 better. If the latter, it will be good to propose a dataset where you clearly see the difference between sota methods and reweighting (even a toy data is fine).
review_title: null

Meta-review: While it lacks theory/methods contributions, the paper provides a valuable and insightful empirical study of out-of-distribution generalization. At the same time, there are still remaining questions that arise from the authors' study that require exploration and explanation, which the reviewers have dutifully brought up. The authors provide some partial answers in the response. The authors should revise their paper to incorporate and build upon this discussion to make the study more complete, as would be expected from empirically-driven paper that focuses on existing methods.
Decision: Accept (Poster)