Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work focuses on the worst group optimization (distributional robust optimization) problem with few spurious attributes are available as a validation set. They aim to achieve similar performance with methods that use spurious attributes. The main contribution is an adaptive thresholding method to ensure balanced sample from each group. Experiments show the proposed pseudo labeling and thresholding are effective in various tasks.

main_review: ## Long summary
Previous methods try to identify and upweight samples from minority groups (spurious attribute joint with class label). Then a validation set of annotated samples are used to tune hyperparameters. The performance of such models are very sensitive to these hyperparameters/validation set, and therefore, fail to perform comparably with methods that use the spurious attribute. This work aims to resolve this issue to achieve performance similar to methods with annotations.

SSA first does pseudo labeling and then does robust training. It trains a domain index predictor and uses a threshold to solve the confirmation bias issue by balancing each group. Then pseudo labels are used on the training set. Then group DRO is used for robust training.

The confidence of the majority group increases faster than the minority group, which is even more severe when the label is pseudo label. This is so-called confirmation bias.

In experiments, it uses the datasets widely used in this field: Waterbirds, CelebA, MultiNLI and CivilComments-WILDS. They show SSA improve the worst group accuracy significantly over existing methods without spurious attribute even with much less valiation data. They also analyze the influence of the validation set size. Finally, they analyze the pseudo labeling process of SSA to verify the adaptive thresholding is useful. They also show SSA works with supervised contrastive learning loss and can be useful in semi supervised learning with imbalanced classes.


## Strength:

1. Very comprehensive experiments.
2. Relatively simple and reproducible method.

## Weakness:

1. Lack of very strong theoretical justification. It is hard to have a straightforward interpretation of why this works, especially based on pseudo labels.
2. Some experiments seem not relevant to the problem this paper aims to solve.

## details and questions
1. Table 6 is kind of redundant since most of it overlaps with Table 1.
2. It is not clear why the supervised contrastive learning part is needed in this work. The same to the semi-supervised learning experiments.
3. How are the supervised and unsupervised loss combined (eq.3)?
4. Why do we need to split the training data in 4.1? They don't have spurious attributes anyway and the method only infers them on training data.
5. I believe it can also be compared with EIIL [1].

[1] Creager, Elliot, Jörn-Henrik Jacobsen, and Richard Zemel. "Environment inference for invariant learning." International Conference on Machine Learning. PMLR, 2021.
summary_of_the_review: ## Strength:

1. Very comprehensive experiments.
2. Relatively simple and reproducible method.

## Weakness:

1. Lack of very strong theoretical justification. It is hard to have a straightforward interpretation of why this works, especially based on pseudo labels.
2. Some experiments seem not relevant to the problem this paper aims to solve.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a technique, spread spurious attribute, for inferring the group annotation for training samples in a dataset. The inferred group information is then used as part of a group DRO minimization scheme or some other worst case scheme. The key insight in this work is to use a small validation set (1-5 percent) of data that has annotations to learn annotations for the entire training set via pseudo-labelling. The labelling scheme presented uses different thresholds for the different spurious groups. This new dataset is then used as part of a new DRO pipeline.  
main_review: This paper tackles an important problem and provides a convincing demonstration that current SOTA models are susceptible to reliance on spurious correlation in the training set. I detail key strengths, weaknesses, and additional questions below.

### Strengths
- Reducing the amount of annotations required worst-group loss minimization is an important problem, and this paper presents a simple procedure to help with that problem. 
- The empirical portion of the paper seems quite thorough, and the work compares against several recent work in this line.
- This paper includes ablations of different components of the proposed formulation to help understand what each one contributes.
- It seems like the SSA approach might be useful when applied with robust training approaches in general.

### Weaknesses, Concerns, & Additional Questions
- My biggest issue is that the paper has several moving parts in Sec. 4.1 and 4.2, which are really the key contributions of this work. I am not sure how to improve the writing here, but there are several pieces like the different validation and training sets, the two different losses, and then the different group thresholds. Even though it seems clear, it was challenging to be sure that all the pieces fit together.
- I am surprised that the approach works so well given the data size disparity between D_train and D_val. I am not that familiar with the performance of standard unsupervised methods. Is such size disparity usually the case?
- In section 4.2, what does "pseudo-group population ratio of the highly-confident samples" mean?
- It seems like the whole point of the group specific thresholding is to help make sure that the model for learning the spurious/group attributes perform well on the small sample groups. First, I don't think I would call this 'confirmation bias' as this paper does, unless this is a standard term in the unsupervised learning literature. Second, it is unclear to me why the approach used for setting this threshold should be effective. The minimum threshold is initially set on the basis of the size of the smallest group, however, consider the unlikely case where that small group is just n copies of the same sample or somehow homogeneous and low variance. This small group would be more easy to learn than a larger group with larger 'variance' or more 'diversity'. Essentially, I am hoping the authors can clarify why this scheme should be effective on the basis of intuition or a toy example. To be more specific, equation 6 seems critical here. I don't understand why this requirement was chosen, and I worry that the dependence on size of the groups is somewhat unusual.
- To clarify, pseudo-group is $\hat{a}(x)$ for the points in the training set?
- There is an important yet simple experiment that I think is missing. Why can't the authors check the accuracy of their SSA scheme for a dataset where we have the ground truth spurious attributes like the waterbirds and the other datasets that they use? Is this what Table 4 is showing? If yes, then it might help to clarify the caption and add more clarification.

Post Rebuttal
Satisfied with the author response, so recommending that the accept.
summary_of_the_review: Overall, this paper conducts thorough empirical analysis of the SSA approach, which can be incorporated with robust training methods. I recommend a weak accept because there are portions of the scheme that I think needs better justification.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the group robustness problem in the setting of spurious correlations, when only a small number of samples have spurious attribute annotations. They present a pseudolabeling algorithm based on FixMatch to pseudolabel the remaining examples, and then use worst-case loss minimization algorithms such as Group DRO to train a more robust model.
main_review: Strengths:
- The empirical results on worst-group accuracy are strong, even with a small number of attribute-annotated examples.
- The proposed semi-supervised approach seems promising, both for the spurious attribute setting and even the general class-imbalanced SSL setting, as shown in Section 5.4.

Weaknesses:
- The language of training and validation is confusing and imprecise, as the "validation" samples are actually used for training as well. The authors should instead clarify that some training examples have attribute annotations and some do not, and similarly for the validation set.
- The methods description is hard to understand in some parts. Specifically, why are predictions not made in D_{train}^{circ} (as mentioned at the end of Section 4.1)? In principle, the model could make predictions for these, but no clear reason is given why this is not done.
- Related to the previous point, one would expect the proposed algorithm to be very computationally intensive. First, the semi-supervised learning stage will likely be expensive. Next, since this is repeated K=3 times, that makes it even more expensive. (And finally, the Group DRO model still needs to be trained after that.) Computational cost / runtime results are not provided.
- The paper is unclear about some hyperparameter details such as the number of epochs for semi-supervised learning.
- It would be great to better understand why the proposed approach performs so well. Based on Table 3 there is almost no drop in performance until we reach 5% of the original val set size, and even then the drop is small. In this case, on Waterbirds, we would only have 3 annotated examples for worst group for training the spurious attribute predictor (and perhaps fewer if any of these are actually used for validation, rather than training itself?). What is the accuracy of the spurious attribute predictor on the worst group - if it is high, how and why might this be the case (despite the extremely low number of annotations)? If it is low, why does the robust model still attain good worst-group performance?

Questions:
In Table 6, are the SupCon results actually vanilla SupCon or the procedure from Zhang et al. (2021)? Based on the writing, it seems the latter, but the table suggests the former.
summary_of_the_review: Overall, this paper has strong empirical results. The paper could be improved with more precise / clear descriptions of the methods and design decisions, and additional ablation experiments or metrics that help explain how the methods actually perform so well even in the extremely low-annotation setting (in lieu of theoretical analysis, which the paper does not have). Thus, my current score is a weak acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a pseudo-attribute-based algorithm, coined Spread Spurious Attribute (SSA), for improving the worst-group accuracy. The proposed method leverages samples both with and without spurious attribute annotations to train a model to predict the spurious attribute, then uses the pseudo-attribute predicted by the trained model as supervision on the spurious attribute to train a new robust model having minimal worst-group loss. Experimental results on various benchmark datasets show that the algorithm consistently outperforms the baseline methods using the same number of validation samples with spurious attribute annotations. 
main_review: Strengths：

1. This paper is well-written and well-motivated. The proposed spread spurious attribute method is novel. 

2. Comprehensive experimental results demonstrate the effectiveness of the proposed approach. 

3. The proposed approach is a general yet effective framework that can be applied to lots of tasks.

 
summary_of_the_review: Overall, this paper proposes a novel and general yet effective framework. The paper is well-motivated and well-written. Moreover, comprehensive experiments have been conducted to demonstrate the effectiveness of the proposed approach.
review_title: null

Meta-review: This paper presents a new method to decrease the supervision cost for learning spurious attributes using worst-group loss minimization. Their method uses samples both with and without spurious attribute annotations to train a model to predict the spurious attribute, then use the pseudo-attribute predicted by the trained model as supervision on the spurious attribute to train a new robust model having minimal worst-group loss. The experiments show promising results in this domain for reducing annotation cost. 

The reviewers vote to accept the paper, and some of them increased their scores during the discussions since the authors have addressed their concerns.
Decision: Accept (Poster)