Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposed the group invariant learning (group-IL) which splits the training data into groups based on spurious correlations. First, they first theoretically analyzed and clarified necessary conditions of group-IL: falsity exposure criterion and label balance criterion. They also discuss that many benchmark datasets violate these conditions. Therefore, they propose a novel methods guided by these conditions. They conduct experiments on image and natural language, and show the effectiveness of their model.
strengths_and_weaknesses: ### Strengths

- The motivation is reasonable and problem formulation is well constructed with related works.
- Their criterion analysis is well explained, and theoretically supported.
- Proposed model can be applied to most of invariant learning methods.
- The experiment results show the effectiveness of proposed method.
- The paper is well-written.

### Weaknesses

- Their methods are based reference model and they use ERM models. It might be give ambiguous information of spurious correlations.
questions: - It would be nice to modify the caption of each figure of Figure 1, instead of reference numbers.
- Is $h$ in Section 4.1 (e.g. Criterion 4.1) different with $h$ in Line 131, which indicates a classifier?  
- How about group inference time compared with baseline?
- Reference model like ERM might be give ambiguous information of spurious correlations. How does it affected?
- Typo
  - Line 72: Classification -> classification
  - Line 207: )) -> )
limitations: They addressed the limitations in Discussions Section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: They claimed that the existing group invariant learning methods are insufficient to prevent classifiers from depending on spurious correlations in the training set.
In detail, they proposed two criteria on judging such sufficient characteristics.
And then, they showed the limitations of the existing works in terms of the criteria. From this point, they designed a new group invariant learning method, i.e. constructing group and reweighting to meet the criteria. The experimental results support the superiority of the proposed method.
strengths_and_weaknesses: **Strengths**
-
* Two criteria that they designed are suitable for describing the sufficiency of the group invariant learning methods.
* The manuscripts are well-written and easy to follow.
* The analysis for the limitations of the existing works are reasonable and the proposed model is properly designed with this point.

**Weaknesses**
-
* Compared to the analysis until section 4, the proposed model in section 5 is quite simple in terms of the technical novelty.
* In terms of dataset, their experiments are quite limited.
* In terms of baselines, it would be good to compare with Group DRO or its variants.
* The detailed experimental analysis are limited since they just provided the quantitative results.
questions: 1. I think that CelebA or CivilComments datasets are suitable for real-world cases. Is there any reasons that this paper does not conduct on these dataset?
2. Group DRO or its variants should be compared. If not necessary, please provide the reasons.
3. There is a hyper-parameter, \lambda, in Equation (3). Is the proposed model sensitive to this value? It would be interesting if there are analysis of the interaction between the value of \lambda and two criteria the authors define. 
limitations: They addressed the limitation and its further direction adequately.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes and evaluates conditions for group invariant learning methods in preventing classifiers from depending on spurious correlations. Motivated by the proposed conditions, the authors develop a new group learning approach with a two-step procedure involving constructing group structures to achieve these conditions and re-weighting samples in model training according to group information.
strengths_and_weaknesses: This paper studies an important question regarding the effectiveness of group invariant learning at the presence of spurious features. The proposed method is compared in the experiment with a good variety of methods and datasets and has demonstrated potential to improve current methods.

However, there are some issues in the technical results and presentations that prevent the readers from fully following the results. For example, In Proposition 4.5, line 217, it is not quite clear why $P(Y=j | B_i) = p_i$, where the right side depends on $i$ alone. In line 215, $r$ is not defined although its meaning can be vaguely guessed. Similarly, in line 260, the $\mathbb{G}$ is not defined. Another common issue is the usage of conditional probability/distributions given group $g$, which seems to be really referring to conditional on the event $\{G=g\}$, where $G$ is the random mapping from the sample $(X, Y)$ to the deterministic group label $\mathcal{G}$ in which $g$ is an element. 

What is more important than the notation issues is that the main results of this paper seem to center around the construction of the random mapping $G$ as a function of $Y$, $X_{inv}$, and $X_{sp}$ to achieve the independent criterions for group invariant learning to work, but the current presentation of results does not completely bridge the gap between the criterion and the final conclusion and nail the requirement of $G$ in terms of the spurious features. Since the probability model framework plays a central role in this paper, it would be helpful to make sure the arguments are rigorous and seamless.

The paper would also benefit from an ablation study to check the importance of the two proposed criteria as well as varying arguments that affect the results including the grouping $\mathcal{G}$ and the weights in the objective $w$.
questions: line 123: please clarify definition of $D'$ 

line 186: please give the definition of $B$, even if appeared in the appendix

Please clarify in the main text the relation between $X$ and $X_{inv}$ and $X_{sp}$. e.g. whether $X$ is generated as a mapping from $(X_{inv}, X_{sp})$

How strong is the condition in Criterion 4.3 and how would the results be affected if the criterion is violated?



limitations: Yes
review_title: null

Meta-review: metareview: This paper presents interesting new theory and algorithms to address group-invariant learning in the presence of spurious correlations between unimportant features and the target. A robust discussion between reviewers and authors lends confidence that the paper should be accepted.

I'd encourage the authors to go over language carefully before the camera-ready deadline; there are many small grammatical issues remaining (eg, plurals).
confidence: Less certain
Decision: Accept