Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a simple method called CROIS for learning a model that can be applied in group-shift settings, i.e. cases where the performance of the group that is underrepresented in the training set is low when there is a distribution shift.
CROIS is motivated by the idea that although ERM-trained DNNs often take advantage of spurious features, previous works show its capability to produce good features.
CROIS consists of 2 steps method 1) train an ERM model on data without the group label information and 2) correct the classifier layer by using the features extracted from stage 1) to train a GDRO model on the data with group labels. Although in practice in the experiments the hyperparameters are fixed, for both steps the model selection can be done on the validation set, the first one optimising the aggregate accuracy and the second one the worst-group accuracy.

strength_and_weaknesses: **Strength**: 

The paper is well written, with simple and relevant notation. CROIS requires fewer labelled data than the GDRO baseline considered, and it does not rely on pseudo-labelling as JTT. The authors perform a good ablation study on different components of their proposed method (e.g. sampling with or without replacement, i.e. using independent splits or not, first and second stage different training strategies, robustness towards the hyperparameters tuning).

**Weaknesses**: 

Fully training group labels baselines could include more recent methods such as SGDRO (non-flat version of GDRO proposed in Goel et al.). 
There is a misleading sentence on page 3 when describing the Waterbirds dataset: “The bird images are then modified with either a water or land background.” There is no “modification” involved, in the dataset the birds are already placed either on water or land background (this is a minor note, but should be revisited).
The main weaknesses are all in the “Experiments” section. 1) Although the hyperparameters are fixed, some ablation is still required (for example the regularisation in the second stage for the CivilComments dataset). 2) In Tables 1 and 2 it is not clear what the results are, is it correct to assume they are the mean and std. performances over three different initialization seeds? 3) The results in Tables 1 and 2 are not correct, CROIS is using the validation set for training in the second stage, not only for model selection. For the datasets considered, the validation set has the same distribution as the test set, it does not seem correct to talk about “group shift” anymore. 4) Table 2 should also include the average test accuracy to show the trade-off between average and worst-case performances when varying the size of the validation set. Also, why is the std. not reported here? 5) Table 3 shows promising results, but the best fraction “p” is a hyperparameter that should be tuned using the worst-group performance on the validation set, like the regularization term for GDRO.  Its behaviour is not linear (e.g. the more the better) nor consistent across datasets. For example, in both text datasets, only one fraction produces better results than the GDRO baseline. Here, I disagree with the statement “In practice, p is not a parameter to choose (there’s no reason to throw away group labels) but rather is limited by the resources available to obtain group labels”: using p=0.5 is often worse than using p=0.3. 6) To obtain robust results, it would have been better to evaluate the methods across different splits of train-val-test, not simply different initialisation seeds. 

clarity,_quality,_novelty_and_reproducibility: **Clarity**: The paper is well written, but the “Experiments” section is not always very clear (e.g. the reported results are the average or the best values? What are the authors varying to create 3 runs? Why is the std. not included in all the Tables?)

**Quality**: As highlighted in the “weaknesses” the quality of the experiments can be improved.

**Novelty**: Although the single components exist in prior works, the idea of combining the two stages is somewhat new.

**Reproducibility**: The source code is provided and the hyperparameters are detailed in the Appendix.

summary_of_the_review: The idea is interesting and promising, but in the current state, I am not very convinced the work is ready to be published. The authors should revise their experiments section to address my doubts.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a novel approach to improve worst-group performance, CROIS, that works well with reduced/limited group annotations on training/val set. The idea is simple yet effective: obtain good feature extractors on group-unlabeled data with ERM loss, and only retrain the last layer on group-labeled data with GDRO loss. Empirical experiments and abundant ablations are performed to support the method.
strength_and_weaknesses: Strengths:

The proposed method is a simple, novel and interesting idea that is orthogonal to the usual pseudo-group-labeling approach taken by other existing methods like JTT.

Weaknesses/comments:

1. The relationship of the CROIS performance to the data size for retraining (group-labeled fraction in training set for CR - Table 3, or reduced val size for CR - Table 2) seems quite inconclusive, as also pointed out by the authors in Sec 4.2 (Page 8). In particular when one cross compares Table 2 and 3, I assume for instance 30% of training size (Table 3) is comparable to 100% of val size (Table 2) for CelebA, which happens to be the sweet CR size for this dataset? IMHO p is the (only) key parameter in Algorithm 1, and it would be good to clarify in the paper, for instance, what's the rule of thumb of choosing p? Would it make sense to consider the CR data size (or p) as a tunable hparam? If group-unlabeled size >> group-labeled size, then should one use 50/50 group-labeled in D_L/D_val or tuning the ratio is still needed?
2. Empirical results are somewhat strong, on par with (but not significantly outperforming) SOTA (e.g. Table 1). Many other baselines (Sec 1.1) should be added to the results (e.g. Table 1) for comparison. Besides, it would be good to include error bars in Table 2 to show that error bar should become much larger when val size becomes smaller despite the "relatively small" drop in mean performance.
3. In Sec 3, the authors claim that independent splits are done to avoid the feature extractors memorizing spurious correlations with class labels. But many modern DL models rely on unsupervised/contrastive/label-free training of feature extractors. For instance, one can continue the pretraining of BERT on the in-domain data on the two language tasks (https://arxiv.org/abs/2004.10964). Perhaps then independent splits still won't be required and more group labels can be used for classifier retraining (in case of limited group annotations)?
4. The abstract/intro clearly separates the two use cases: a) full training and b) val group annotation. Algorithm 1 (Sec 3) does not make such distinction where D_L is listed as an input. It only becomes clear how Alg 1 is applied to case b) in Sec 4.1 where val set is actually split 50/50 into D_L and D_val. For consistency, could consider clarifying this earlier in the paper to avoid confusion.
clarity,_quality,_novelty_and_reproducibility: The paper is well presented (except for the choice of a key hparam of CR data size, see weakness #2 above). The proposed method is a simple, novel and interesting idea that is orthogonal to the usual pseudo-group-labeling approach taken by other existing methods like JTT. I think most readers will find it interesting. The pseudo code is clear and simple enough for interested readers to reproduce the work.
summary_of_the_review: Though the empirical results do not outperform SOTA, the paper still presents a novel and interesting idea in worst-group improvement that is quite different from usual approaches in this area. A key question regarding the group-labeled size for classifier retraining can be better explained for other researchers to build upon the work and/or practitioners to apply the proposed method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a very simple method termed classifier retraining on independent splits to improve the subpopulation shift robustness of the model. The main algorithm of the paper is clearly shown in Algorithm 1 of the paper.
strength_and_weaknesses: Advantages.

This paper is well written and easy to follow. Although the method of the paper is too straightforward, the author explains its principle very clearly.

Weaknesses.

However, I still have the following concerns: 
1. The comparison experiments in Table 1 is unfair. Since the proposed method uses the data from the validation set for training.
2. The paper does not conduct discussion and compare with state-of-the-art methods, e.g., LISA[1], UMIX[2], CnC[3], CGD[4].

[1] Improving out-of-distribution robustness via selective augmentation.

[2] UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixup

[3] Correct-n-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations

[4] Improving out-of-distribution robustness via selective augmentation.
clarity,_quality,_novelty_and_reproducibility: This paper is well written and easy to follow.
summary_of_the_review: This paper proposes a very simple and effective method. But some experiments are unfair.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a simple yet promising method for improving worst group performance of a trained model. The idea is to use group-unlabeled training data to learn good features with an ERM objective, and then to re-train the last layer of the model on group-labeled data using a GDRO objective. The reviewers have raised multiple concerns regarding empirical evaluations and technical novelty. AC agrees with the reviewers that albeit it’s an important study, limited technical contributions and deficient empirical evidence are critical issues that will require too many revisions to be prepared and verified at this stage, and will benefit from another round of reviews. We hope the reviews are useful to improve the manuscript.
Decision: Reject