Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper propose a mixup-style data augmentation method under the data distribution shift context. In particular, data distributions are formulated as mixture of distributions (i.e., domains), and two distribution shift scenarios are considered: (1) domain shift, where the test domain and train domain are disjoint. (2) subpopulation shift, where test distribution has different mixture proportion than train distribution. It's  assumed that domain identification spuriously correlates with labels. To tackle this problem, this paper proposes two mixup strategies: (I) mixup two examples with same label but different domains; (II) mixup two examples with same domain but different labels. It's claimed that such mixup could cancel out the spurious correlations. Extensive experiments on a variety of datasets show its superiority compared to empirical risk minimization (ERM) and alternative data augmentation methods. The paper further provide theoretical analysis that under certain conditions, the proposed method has asymptotically smaller worst case classification errors than ERM and vanilla mixup. 
main_review: the idea is quite simple and intuitively reasonable, and empirical results seem extensive and significant, and theoretically justified to some extent

some of the results analysis is a bit confusing to me
(1) in 4.1 "evaluating robustness to domain shifts", the best strategy was to always mixup same label with different domains, and the potential reason given is that the datasets actually have weak or even no spurious correlation between domain and label. Two questions follow:
(a) From early text, seems both selection strategies are motivated by the spurious correlation, but here why do we still observe advantage over ERM or vanilla mixup? It would be great if you could clarify the different motivations (if any) of the two selection strategies
(b) the reasoning about "weak or no spurious correlation" seems to be contradicting with claim in 4.3, where it's stated that "compared with vanilla mixup, ...LISA...improve the OOD robustness by canceling out the spurious correlations..". or did I misunderstand something? Is it easy to quantify such spurious correlation? if so, why not present the actual correlation metrics for these datasets? 

(2) for the ablation study, I think a more convincing way would be: first test LISA and mixup on a dataset that is known to have NO spurious correlations, then we expect neutral results; then test them on a data that is known to have spurious correlations, and we could give quantitative metrics of such correlations if possible, and show LISA is better than mixup; further more, the stronger the correlation, the more advantage LISA has. Is that what you're trying to demonstrate here? 

(3) In table 8, vanilla mixup shows much worse performance than ERM in terms of learning invariant representations under the defined metric, which doesn't seem quite reasonable to me. Shouldn't we expect the contrary? 
 


Another minor question: In Theorem 1, is p the dimension of x? If so it's better to state that explicitly in the theorem instead of relying on readers to go back to text and only to find in the superscript notation. 
summary_of_the_review: the idea is quite simple and intuitively reasonable, and empirical results seem extensive and significant, and theoretically justified to some extent, but the results analysis and some experimental design could be more insightful or improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers the model robustness under distribution shift brought by domains and subpopulations. Specifically, based on the interpolation scheme in mixup, the authors propose two selection strategies to perform data augmentation, aim at eliminating the spurious correlations and learning an invariant representation.
main_review: Strength:
(1) The authors address a critical point that prevent models from generalization, namely spurious correlation.
(2) The proposed method is simple and easy to implement, and the empirical results are within expectation.

Weakness:
(1) Maybe the biggest concern is the contribution over previous work. The proposed method can be seen as a heuristic extention of mixup. Though simple and easy to follow, the contribution is marginal. 
(2) There ara growing trends on investigating OOD generalization under missing domain label, it is better to at least include such work (e.g. [1]) for discussion.
(3) There is a hyper-parameter p_sel that controls the probability of performing different strategy, is there a rule of thumb or we need to tune it for every task?
[1] Qiao, F., Zhao, L., & Peng, X. (2020). Learning to Learn Single Domain Generalization. In CVPR.
summary_of_the_review: This paper address the spurious correlation by augmenting the data via interpolation. Although intuitions are provided and empirical effectiveness is illustrated accordingly, the contributions over previous work (e.g. mixup) are marginal.

=========After Response==========

The response from authors has addressed my major concerns, so I raise my score accordingly.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors introduced approaches aimed at learning invariant predictors across data sources. Rather than using distribution/risk matching schemes as often done by previous work, they propose to train models against mixtures of data points as a means to avoid that models rely on spurious correlations between domain and class labels, since such correlations observed during training might not hold at testing time. The proposed setting uses the idea of mixup to combine data instances in two different schemes: I-combine data points from the same class but different domains, and II-combine data points from the same domain but from different classes.
main_review: Strengths:

+ The proposed approach is simple and efficient, and can be directly incorporated in or combined with other invariance-inducing approaches;

+ Prediction performance is shown to improve over a number of recent baselines under challenging benchmarks.

Weaknesses/suggestions: 

- The proposal requires assumptions that are not discussed in the manuscript. In [1], it was shown that domain-invariant approaches can only improve out-of-distribution generalization if data-conditional label distributions P(y|x) are fixed across domains; i.e., observing x suffices in order to determine y, regardless of the domain according to which x was observed.

- My main concern lies in the reported evaluation, which is focused on showing improvements in terms of downstream performance, and presented results consist of comparing the proposed approach with alternative methods. While improving downstream performance is of course our ultimate goal, and results are strong in this sense, doing so does not explain the sources of improvements. In particular, authors claim that the mixup strategies they introduce yield some type of domain invariance, which is not verified empirically. To verify that learned representations are invariant, that could be achieved via domain-prediction experiments; i.e., train domain classifiers on top of representations learned by different methods. The higher the accuracy of such a classifier, the less invariant are representations. For the case of prediction-level invariance, authors could perhaps evaluate the range of estimated risks across domains. Improvements on either one of these notions of invariance would then explain observed improvements in terms of prediction accuracy.

- "We argue that the failure of other methods in some datasets may be caused by their regularizers limiting model capacity to some extent". That's another case where the evaluation lacks in supporting authors' claims. This hypothesis can be verified via in-domain prediction performance, i.e., overly regularized underfitting models should result in accuracy degradation in the training domains. Alternatively, one could rule out the underfitting effect by using higher capacity model classes.

- Conclusions from the risk bounds provided in theorems 1 and 2 are a bit unrealistic given the strong assumptions that imply the results. In particular, the model assumed for the data generating process is overly simplified and, although it enables theoretical analysis, it's unclear to which extend the conclusions hold in practice.

- Finally, regarding novelty, it seems different recent approaches introduce methods that use some sort of mixup across domains in similar settings. For the domain adaptation/generalization cases, there are, for instance, [2,3,4]. For the multi-domain case, cross-domain mixup was studied in [5]. Authors do compare results against [4], but it's unclear how the proposed approach differs from those other recent applications of mixup under similar settings, and the related work section should include such a discussion.

Other comments:

- On page 2, the setting described was not originally introduced by Koh et al. (2021). To my knowledge, it was first discussed in [6] and later on in [7].

- The definition of mixup for labels in the rightmost term in eq. 2 seems to require labels y are one-hot encoded, which is not mentioned in the text.

- While in the title authors claim to be improving out-of-distribution robustness, a large part of the work focus on the multi-domain learning (or fairness) setting, where one's goal is to find predictors with uniform risks across a set of domains that doesn't change from training to testing. Technically, that wouldn't be out-of-distribution. Perhaps there should be a sentence or two in the introduction indicating what authors refer to as out-of-distribution.

References:

[1] Zhao, Han, et al. "On learning invariant representations for domain adaptation." International Conference on Machine Learning. PMLR, 2019.

[2] Shu, Yang, et al. "Open Domain Generalization with Domain-Augmented Meta-Learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[3] Wang, Yufei, Haoliang Li, and Alex C. Kot. "Heterogeneous domain generalization via domain mixup." ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020.

[4] Xu, Minghao, et al. "Adversarial domain adaptation with domain mixup." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.

[5] Chuang, Ching-Yao, and Youssef Mroueh. "Fair mixup: Fairness via interpolation." arXiv preprint arXiv:2103.06503 (2021).

[6] Muandet, Krikamol, David Balduzzi, and Bernhard Schölkopf. "Domain generalization via invariant feature representation." International Conference on Machine Learning. PMLR, 2013.

[7] Albuquerque, Isabela, et al. "Generalizing to unseen domains via distribution matching." arXiv preprint arXiv:1911.00804 (2019).

summary_of_the_review: The paper is well-written, the approach is efficient and observed to work well on a number of benchmarks. However, experiments supporting key claims are lacking, and it's unclear whether the observed improvements in terms of invariance (either at feature- or prediction-level) hold true since no supporting experiments are reported. Contextualization of the proposal relative to past literature also needs improvements since cross-domain data mixup was studied in the past under both settings considered by the authors. The provided discussion on related work does not clarify what and how the authors' proposal improves upon previous work.
review_title: null

Meta-review: The manuscript focuses on model robustness under distribution shift, specifically domain shifts and subpopulation shifts. Domain shift is where the test domain and train domain are disjoint. Subpopulation shift is where test distribution has different mixture proportion than train distribution. The assumption is that domain identification spuriously correlates with labels. The proposed framework learns an invariant representation by using mixup strategies and interpolates samples either with the same labels but different domains or with the same domain but different labels to. Experiments are performed on a variety of domain shift and subpopulation shift benchmarks, and results showed that the proposed framework is better than empirical risk minimization (ERM) and alternative data augmentation methods. Theoretical analysis is also provided and it is shown that, under certain conditions, the proposed framework has asymptotically smaller worst case classification errors than ERM and vanilla mixup.

Reviewers agreed on several positive aspects of the manuscript, including:
1. The manuscripts addresses a critical point that prevent models from generalization, namely spurious correlation; 
2. The proposed method is simple and easy to implement, and the empirical results are within expectation.

Reviewers also highlighted several major concerns, including:
1. Different recent approaches introduce methods that use some sort of mixup across domains in similar settings;
2. Ablation study on datasets without spurious correlations are missing;
3. Evaluation of domain invariance representations and prediction-level invariance needs clarifications;

Authors clarified different motivations of the two selection strategies in relation to spurious correlation between domains and labels, and provided an ablation study on datasets with no spurious correlation. Post-rebuttal, reviewers stayed with borderline ratings, and they have suggested further improvements: improving results analysis and the conclusion that “existing domain information may not fully reflect the spurious correlation”, understanding the implication and the reasons that invariance is achieved at the prediction level instead of at the representation level despite the original goal is to learn an invariant representation, and improving presentation of the manuscript including settings and assumptions.
Decision: Reject