Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work deals with the problem of groups (bias) robustness under distribution shift between train and test set. The authors propose a mixup strategy where samples with i) same label but different groups, and samples with ii) different labels but similar bias are mixed without having access to bias (or group) annotation. Cosine similarity is used as a proxy to find clusters of different groups training an additional model via supervised contrastive learning. The authors provide experiments on vision datasets to corroborate their method.
strengths_and_weaknesses: strengths

* Important work that explores the use of mixup strategy in distribution shift with possible applications to foundational models.
* The paper is well written and clear with a solid set of experiments and ablation studies.
* The method is effective and experiments are provided on vision benchmarks.

weaknesses
 
* Cosine similarity to cluster samples from the same group can be ineffective for more complex datasets with a large number of classes or a large variety of biases.
* Method limited to images.
questions: * Mixup-based augmentation has been employed to handle distribution shifts finetuning large models, for example in this recent work - https://openreview.net/forum?id=HI2ilxFli0W - class-conditional mixup is employed to improve group robustness in presence of group shift between train and test in absence of group annotation. How would your method, in particular, `i) the same label but dissimilar bias` part of SelecMix, compare to this or a similar approach for transfer learning or distribution shift?

* For SelecMix, other than i) you also mix `ii) samples with different labels but similar bias`. How important is this second mixing for SelecMix to work? I would assume that using standard mixup on a train set with highly unbalanced groups (99% vs 1% or 95% vs 5%) the results should be identical to applying ii) and shouldn't help group robustness. Can you elaborate on why using ii) is helpful compared to just applying mixup in Table 4?

* Do you think you could use your method to de-bias language or a different modality?

* What is the metric in Table 1? Average accuracy? It would be interesting to see the worst-group (label, bias) accuracy for the different methods.
limitations: * The effectiveness of the method is based on how well the cosine similarity can cluster samples from the same group together. This is a reasonable assumption for images but not necessarily for other modalities, and for this reason, the method is (or could be) limited in applicability.
* Additionally I think this approach will have issues with larger datasets with more variety and number of classes and biases.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes SelectMix for debiased laerning. Contradicting pairs are selected for mixup with different training loss to achieve the learning goal. The experiments show improvement over previous methods.
strengths_and_weaknesses: Strengths
1. Comprehensive experiments and analysis are provided. The results also show improvement.

Weaknesses
1. The organization of the paper is not very easy to follow.
2. The core contribution of the paper is SelectMix, but also a contrastive objective is included in training. The analysis and improvement of adding the contrastive loss is not shown. 

questions: see above
limitations: Limitations and societal impact are presented.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed selective mixup for learning robust classifiers. Specifically, they apply mixup to 
1. input pairs that have the same label but different bias attribute values.
2. input pairs with different label but with similar bias attribute values.
Since the bias attribute values are usually not given, they estimate it by a bias-amplified model (that favors easy-to-learn shallow features during training). On MNIST, CIFAR10 and BFFHQ, combining selective mixup with previous methods consistently improves the robust performance.
strengths_and_weaknesses: Strength:
1. Adapting input mixup for de-biasing is interesting.
2. Consistently good performance across three datasets.

Weakness:
1. The idea is clearly presented (nice figure) but the paper is poorly written.
    + For example, you cannot just simply name *bias attributes* as *bias*. Bias often refers to unwanted correlations between the input and the output. The sentence "the same label but dissimilar bias [line 11]" is just very confusing.
2. A few missing references.
    + [1] uses group DRO to learn a robust classifier from the training mistakes. It outperforms LfF by a significant margin, I am very curious to see how does SelecMix compare to this method.
    + [2] proposes to use a biased model to guide the training of a debased model. I think they are one of the first among this line and it should definitely be cited.
    + [3] uses different data environments to learn a metric space that encodes examples based on their bias attributes. 


[1] Liu, Evan Z., et al. "Just train twice: Improving group robustness without training group information." International Conference on Machine Learning. PMLR, 2021.

[2] Sanh, Victor, et al. "Learning from others' mistakes: Avoiding dataset biases without modeling them." International Conference on Learning Representations. 2020.

[3] Bao, Yujia, Shiyu Chang, and Regina Barzilay. "Learning Stable Classifiers by Transferring Unstable Features." ICML 2022.

questions: 1. I think comparing with JTT [1] is necessary and will strengthen the paper.
limitations: Yes.
review_title: null

Meta-review: metareview: This paper proposes to handle so-called spurious / undesirable signals that are correlated with but do not entail the label (and where it is considered undesirable that a classifier should rely on these signals). The authors propose a variation of the mixup training heuristic where for each example, one selects a bias-conflicting pair. Because the bias-conflicting pairs are rare, they are oversampled to form the mixup pairs and intuitively, this makes the biased signal less predictive. The authors compare against other methods in the case where the "spurious feature" is known and propose a further heuristic for automatically providing pseudo "bias labels" based on the intuition that spurious features of concern are often "easy to learn" and thus examples tend to be grouped together by their spurious features earlier in training. This seems to work well on some toy datasets but the degree to which guesses are piled upon guesses here is of concern. Overall, this is a borderline paper, with 2 of 3 reviewers liking it and one championing it for acceptance. 
confidence: Less certain
Decision: Accept