Improving Out-of-Distribution Robustness via Selective Augmentation
Authors: Huaxiu Yao
Yu Wang
Sai Li
Linjun Zhang
Weixin Liang
James Zou
Chelsea Finn
Venue: ICLR 2022 Submitted (2022)
Keywords: out-of-distribution robustness
distribution shifts
selective data augmentation

Abstract: Machine learning algorithms typically assume that training and test examples are drawn from the same distribution. However, distribution shifts is a common problem in real-world applications and can cause models to perform dramatically worse at test time. In this paper, we specifically consider the problems of domain shifts and subpopulation shifts, where learning invariant representations by aligning domain-specific representations or balancing the risks across domains with regularizers are popular solutions. However, designing regularizers that are suitable for diverse real-world datasets is challenging. Instead, we shed new light on addressing distribution shifts by directly eliminating domain-related spurious correlations with augmentation, leading to a simple technique based on mixup, called LISA (Learning Invariant Representations via Selective Augmentation). LISA selectively interpolates samples either with the same labels but different domains or with the same domain but different labels. Empirically, we study the effectiveness of LISA on nine benchmarks ranging from subpopulation shifts to domain shifts. The results indicate that LISA consistently outperforms other state-of-the-art methods with superior invariant representations. The empirical findings are further strengthened by our theoretical analysis.

Under review as a conference paper at ICLR 2022
IMPROVING OUT-OF-DISTRIBUTIOO ROBUSTNESS VIA SELECTIVE AUGMENTATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Machine learning algorithms typically assume that training and test examples are drawn from the same distribution. However, distribution shifts is common problem in real-world applications and can cause models to perform dramatically worse at test time. In this paper, we specifically consider the problems of domain shifts and subpopulation shifts, where learning invariant representations by align- ing domain-specific representations or balancing the risks across domains with regularizers are popular solutions. However, designing regularizers that are suit- able for diverse real-world datasets is challenging. Instead, we shed new light on addressing distribution shifts by directly eliminating domain-related spurious cor- relations with augmentation, leading to simple technique based on mixup, called LISA (Learning Invariant Representations via Selective Augmentation). LISA se- lectively interpolates samples either with the same labels but different domains or with the same domain but different labels. Empirically, we study the effective- ness of LISA on nine benchmarks ranging from subpopulation shifts to domain shifts. The results indicate that LISA consistently outperforms other tate-of-the- art methods with superior invariant representations. The empirical findings are further strengthened by our theoretical analysis.
INTRODUCTION
To deploy machine learning algorithms in real-world applications, must pay attention distri- bution shifts, i.e. when the test distribution is different from the training distribution, which sub- stantially degrades model performance. In this paper, we refer this problem as out-o--distribution (OOD) generalization and specifically consider performance gaps caused by two kinds of distribu- tion shifts: domain shifts and subpopulation shifts. In domain shifts, the test data sampled from different domains than the training data, which requires the trained model to generalize well to test domains without seeing training data from those domains. Take health risk prediction as an exam- ple. We may want to train a model on patients from a few sampled hospitals and then deploy the model to a broader set hospitals (Koh et al. 202T). In subpopulation shifts, the proportions of subpopulations in the test distribution differ from the proportions in the training distribution. When subpopulation shift occur, models perform poorly when they falsely rely on spurious correlations, which may occur when some subpopulations in financial risk prediction, a machine learning model trained on the entire population may associate the labels with demographic features (e.g., religion and race), making the model fail on the test set when such an association does not hold in reality.
under-represented in the training
For example,
To improve model robustness under these two kinds of distribution shifts, methods for learning invariant representations have shown effectiveness in various applications. These methods learn features or prediction mechanisms that are invariant different domains while still containing suf- ficient information for the targeted task (Li et al.) 2018 Arjovsky et al, 2019). Concretely, some prior works learn invariant representations by aligning and regularizing the domain-specifie repre- sentations (Li et al.) 2018 [Sun & Saenko 2016). Other works aim to find invariant representations by balancing the risk across domains using regularizers (Arjovsky et al.) 2019 Krueger et al. 2021 Rosenfeld et al. 2021). which further increases the dependency between the invariant representa- tions and labels. However, designing regularizers that are widely suitable to datasets from diverse domains is especially challenging and insuitable regularizers may adversely limit the model's ex- pressive power, leading to inconsistent performance among various real-world datasets. For exam- ple, on the WILDS datasets (Koh et al. 2021), invariant risk minimization (IRM) (Arjovsky et al.,

---

Under review as a conference paper at ICLR 2022
2019)
outperforms empirical risk minimization (ERM) on CivilComments, but fails to improve bustness on a variety of other datasets like Camelyon17 and RxRx1. A similar phenomenon is also reflected in the performance of CORAL (Sun & Saenko 2016).
Instead of explicitly imposing regularization to learn invariant representations, we turn towards an implicit solution. Inspired by mixup (Zhang et al. 2018), we aim alleviate the effects of domain- related spurious information through data interpolation, leading to a simple algorithm called LISA (Learning Invariant Representations with Selective Augmentation). Concretely, LISA linearly in- terpolates the features for a pair of samples and applies the same interpolation strategy on the cor- responding labels. Critically, the pairs are selectively chosen according to two sample selection strategies. In selection strategy I, LISA interpolates samples with the same label but from different domains, aiming to eliminate domain-related spurious correlations. In selection strategy II, LISA interpolates samples with the same domain but different labels, where the model should to ignore the domain information and generate different predicted values as the interpolation ratio changes. predictors without explicitly con-
In this way, LISA encourages the model to learn straining or regularizing the representation.
The primary contributions of this paper are as follows: (1) We develop a method that tackles the problem of distribution shifts by canceling out the domain-related spurious correlations via data in- terpolation. (2) We conduct broad empirical experiments to evaluate the effectiveness of LISA on nine benchmark datasets from diverse domains. In these experiments, we make the following obser- vations. First, find that LISA consistently outperforms seven prior methods in addressing both domain shifts and subpopulation shifts. Second, we identify that the performance gains of LISA are indeed caused by canceling out domain-specific information and learning invariant representations, rather than simply involving more data via interpolation. Third, when the degree of distribution shift increases, LISA achieves more significant performance gains. (3) Finally, we provide theoretical analysis of the phenomena distilled from the empirical studies, where we provably demonstrate that I. LISA can achieve smaller worst-domain error compared with ERM and vanilla mixup. We also note that to the best of our knowledge, this is the first theoretical analysis of how mixup (with or without the selection strategies) affects miisclassificatton error.
PRELIMINARIES
In this paper, consider the setting where one predicts the label y E y based on the input feature x E X. Given a parameter space θ and loss function f, we supposed to train a model fe under the training distribution Ptr. where 0 E O. In empirical risk minimization (ERM), as- sume the empirical distribution over training data is Per. ERM optimizes the following objective:
Y1: Waterbird
V/: Landbird
dy: Water
72.9% of train data
3.8% of train data
== arg min der
((a.)))(((((())..)))
(1)
Land
In a traditional machine learning setting, a test set, sam- pled from a test distribution Pts. is used to evaluate the generalization of the trained model 0", where the test dis- tribution is assumed be the same as the training distri- bution, i.c., ptr = pts In this paper, we are interested in the setting when distribution shift occurs, i.c., ptr * pts
1.2% of train data 22.1% of train data
Group illustration of Water- bird data. Domains and labels are rep- resented by background and bird type. Most training samples are drawn from waterbird in water (di, y1) and landbird in land (dg, y2). The trained model is spuriously biased by the background.
Figure 1:
Specifically, follow Muandet et al. (2013) Albuquerque al.] ( Koh et al. (021)) we regard the overall data distribution containing D {1...., , D} domains and each domain d D is associated with a data distribu- tion Pa over set (X, Y, d) = {(x+,, Mi, d) J.ve where Nd is the number of samples in domain d. Then, we formu- late the training distribution as the mixture of D domains, ie., ptr = Edep r Pa. where (x² denotes the mixture probabilities in training set. Here, the training domains are defined as Dtr = {d E Dirr"" > 0). Similarly, the test distribution could be represented as prs EdeD r/z Pa. where {r(a] } is the mixture probabilities in test set. The test domains are defined Dis = E Dirff 0}.
2

---

Under review as a conference paper at ICLR 2022
In domain shifts, we investigate the problem that the test domains are disjoint from the training domains, i.c., Der n D's = 0. In general, we assume the test domains share some common prop- erties with the training domains. For example, in Camelyon17 (Koh et al. (021) data, we train the model on some hospitals and test it in a new hospital. We evaluate the worst-domain or average performance of the classifier among all test domains.
In subpopulation shifts, the test set have domains that have been seen in the training set, but with different proportion of subpopulations, i.e., Dís C Der but {rd}} + {r2(). Under this setting, follow Sagawa et al. (2020a), we specially consider group-based spurious correlations, where each group 9 E G is defined to be associated with a domain d and label y, ie., 9 (d. y). We assume the domain identification is spuriously correlated with the label. For example, we illustrate the Waterbirds dataset in Figure[[ where the background d (water or land) is spuriously correlated with the label (waterbird or landbird). Based on the group definition, we evaluate the model via the v)), where fo-1 represents the 0-1 loss.
worst test group error, i.e., max,
LEARNING INVARIANT REPRESENTATIONS WITH SELECTIVE AUGMENTATION
This section presents LISA, simple way to improve robustness to domain shifts or subpopulation shifts. The key idea behind LISA is to encourage the model to alleviate the effects of domain-related spurious correlations by selective data interpolation. Before detailing how to select interpolated samples, we first provide a general formulation for data interpolation.
In LISA, perform linear interpolation between training samples. Specifically, given samples (r;; yi. di) and (rj, Vj. dz) drawn from domains d, and dj. we apply mixup (Zhang et al. 2018). a simple data interpolation strategy, separately on the input features and corresponding labels as:
Tmii Ar; + (1 \)m Umiz xy/ (1 X)ys
where the interpolation ratio À [o, 1] is sampled from Beta distribution Beta(a, 3) and 3 and By are one-hot vectors for classification problem. Notice that the mixup approach in equation 2 can be replaced by CutMix (Yun et al. 2019). which shows stronger empirical performance in vision- based applications. In text- based applications, we replace the feature interpolation in equation 2 with Manifold Mixup (Verma et al. 2019), where the interpolation strategy is performed the output representation of a pre-trained model, e.g., the output of BERT (Devlin et al. (019)).
After obtaining the interpolated features and labels, we replace the original features and labels in ERM with the interpolated ones. Then, the optimization process in equation | is reformulated as:
8 := arg min E ((rs,,.dd)) AEB
(Z ymiz)]
(3)
Without additional selection strategies, vanilla mixup will regularize the model and reduce overfit- ting (Zhang et al.) 2021b), allowing it to attain good in-distribution generalization. However, vanilla mixup may not be able to cancel out spurious correlations, causing the model to still fail at attaining good OOD generalization (see empirical comparisons in Section 4.3 and theoretical discussion in Section 5). In LISA, we instead adopt new strategy where mixup only applied across specific domains or groups, which leans towards learning invariant representations and thus better OOD performance. Specifically, the two kinds of sample selection strategies are presented as follows:
Selection Strategy I: Interpolating samples with the same label. In selection strategy I, LISA interpolates samples with the same label but different domains (i.e., d: o dj. y/ = 1j). This produces datapoints that have both domains partially present, effectively eliminating spurious cor- relations between domain and label in cases where the pair of domains correlate differently with the label. Additionally, if domain information does not fully reflect the spurious correlations in some datasets, relations by only interpolating samples within the same class regardless domain information (i.e., yi = yj). As a result, LISA with selection strategy I should learn domain-invariant representations for each class and thus achieve better OOD robustness.
also enlarge the interpolation scope to cover more potentially spurious cor-
Selection Strategy II: Interpolating samples with the same domain. Supposing domain infor- mation is highly spuriously correlated with the label information, selection strategy II applies the
3

---

Under review as a conference paper at ICLR 2022
interpolation strategy on samples with the same domain but different labels, i.e., d, = dj. Ye Vj- Intuitively, even within the same domain, the model is supposed to generate different predicted labels since the interpolation ratio À is randomly sampled, corresponding to different labels Ymir- This causes the model to make predictions that are less dependent on the domain, again improving OOD robustness.
In LISA, we randomly perform selection strategies I II during the training process with probability Pset and 1-pset for each batch of data, where Psel is treated as hyperparameter in our experiments. The choice of Psel depends on the number of domains and the relation between domain information and spurious correlations. Empirically, using selection strategy I only brings much more benefits when there are more domains, and/or the domain information can not fully reflect the spurious correlations. LISA with selection strategy II can benefit the performance when domain information highly spuriously correlated with the label, where we find balanced ratio (i.e., Psel 0.5) performs the best. The pseudocode of the training procedure of LISA shown in Algorithm [
Algorithm Training Procedure of LISA
Require: Training data, Step size ?)
1: while not converge do
2: Sample
(aeta(a,3) 3)
3: Sample set of samples Bi uniformly from the training data
4: Randomly select a sample selection strategy or II with the probability psel and - Pael
5: if use selection strategy I then
6: For each sample (Ii, Vt. d:), find another one (z 3/- d,) from the dataset with the same label (y: yj) but different domains (d, * dz), and construct set B2.
7: else if use selection strategy II then
8: For each sample (I,, yı. de), find another one (Iz, yy, dz) from the same domain (d, = dz)
but different labels (y: # gj). constructing set 132.
9: Update 9 with data XB1 + (1 B2
4 EXPERIMENTS
In this section, conduct comprehensive experiments to evaluate the effectiveness of LISA. Specif- ically, we aim to answer the following questions: Q1: Compared to prior methods, can LISA im-
prove robustness to domain shifts and subpopulation shifts (Section 4.1 and Section 4 ? Q2:
Which aspects of LISA are the most important for improving robustness (Section 4.3 Q3: How
does LISA perform with varying degrees of distribution shifts (Section 4.4 cessfully produce invariant representations (Section(4.5)
Q4: Does LISA suc-
To answer Q1, compare to ERM, IRM (Arjovsky al. 2019) MMD Li et al. 2018
DRNN (Ganin & Lempitsky 2015), GroupDRO (Sagawa et al. 2020a). DomainMix (Xu et al, 2020), and Fish (Shi all 2021). Upweighting (UW) is particularly suitable for subpopulation shifts, so we also use it for comparison. We adopt the same model architectures for all approaches.
4.1 EVALUATING ROBUSTNESS TO DOMAIN SHIFTS
Experimental Setup. To study domain shifts, we study five datasets. Four datasets (Camelyon 17, FMoW, RxRx1, and Amazon) are selected from WILDS (Koh et al. (002)). covering real-world dis- tribution shifts across diverse domains (e.g., health, natural language process, and vision). Besides the WILDS data, we also apply L.SA on the MetaShift datasets (Liang & Zou (02T). constructed using the real-world images and natural heterogeneity of Visual Genome (Krishna al., 2016). We summarize the datasets in Table g of Appendix I. tion metric, model architecture, and the number of classes. Detailed dataset descriptions and other respectively.
including domain information, evalua-
training details are discussed Appendix A. and A.1.2
We report the results of domain shifts in Table I where the full results that include vali- dation performance and other metrics are listed in Appendix [A...d The optimal strategy selection probability Pset is set as 1.0 for these domain shifts problems, i.e., only selection strategy is used.
Results.
4

---

Under review as a conference paper at ICLR 2022
In addition, only interpolate samples with the same labels regardless the domain information, which empirically leads to the best performance. According to Table we have two key findings:
There no significant performance differences between ERM and other invariant representation learning methods (e.g. IRM, CORAL, MimainMix) most datasets, which is consistent with the reported results on WILDS (Koh et al. 202T). potential reason is that the existing domain information may not fully reflects the spurious correlation. For example, in Camelyon17-wild dataset (Koh et al. 2021). the presence of tumor tissue (i.e., label) mainly depends on the de- mographic of patients (e.g., race, gender), which shows significant difference across hospitals (i.e., domain information). This could also explain why only interpolating samples with the same labels regardless the domain information achieves the best performance, which also corroborates our claims in the description of the selection strategy in Section
The consistent superiority of LISA outperforms prior methods on all five datasets regardless of the model architecture and dataset types (i.e., image or text), demonstrating its effectiveness in improving OOD robustness by canceling out spurious correlations with augmentation.
Table Main domain shifts results. LISA outperforms prior methods on all five datasets. Following the instructions of Koh et al. (202I). report the performance of Camelyon17 over 10 different seeds and the results of other datasets are obtained over different seeds.
Camelyon17
FMoW
RxRx1
Amazon
MctaShift
Avg. Acc.
Worst Acc.
Avg. Acc.
10-th Per. Acc.
Worst Acc.
ERM
70,3 + 6.4%
32.3 +:t 1.25%
29.9 + 0.4%
53.8 1 0.8%
52.1 + 0.4%
IRM
64.2 : 8.1%
30.0 + 1.37%
8,2 = 1.1%
52.4 + 0.8%
51.8 + 0.8%
CORAL
59.5 + 7.7%
31.7 = 1.24%
28.4 = 0.3%
52.9 + 0.8%
47.6 - 1.9%
GroupDRO
68.4 7.3%
30.8 0.81%
23.0 = 0.3%
53.3 0.0%
51.9 0.7%
DomainMix
69.7 + 5.5%
34.2 + 0.76%
30.8 + 0.4%
53.3 + 0.0%
51.3 0.5%
Fish
74.7 + 7.1%
34.6 + 0.18%
10.1 + 1.5%
53.3 + 0.0%
49.2 + 2.1%
LISA (ours)
77.1 + 6.5%
35.5 + 0.65%
31.9 = 0.8%
54.7 1 0.0%
54.2 + 0.7%
4.2
EVALUATING ROBUSTNESS TO SUBPOPULATION SHIFTS
Evaluation Protocol. In subpopulation shifts, we evaluate the performance on four binary clas- sification datasets, including Colored MNIST (CMNIST), Waterbirds (Sagawa et al., (2020a). CelebA (Liu et al. (015)). and Civilcomments (Borkan et al. (019). We summarize brief data statis- ties in Table 14 of Appendix A.2.T covering domain information, model architecture, and class information. Full dataset descriptions of subpopulation shifts are also presented in Appendix 1 Following Sagawa et al. (2020a), in subpopulation shifts, we use the worst-group accuracy to evalu- ate the performance of all approaches and the domain identifications are highly spurious correlated with the label information. For example, as suggested in Figure I 95% images in the Waterbirds dataset have the same background and bird type, i.e., waterbirds in water landbirds in land. Full hyperparameter settings and training details are listed in Appendix A..222
Results. In Table 2 report the overall performance of LISA and other methods. Similar to the observations in domain shifts, LISA consistently outperforms prior methods in CMNIST, CelebA, and CivilComments. In Waterbirds, LISA outperforms other invariant representation learning meth- ods (c.g., IRM, CORAL, DomainMix, Fish) and shows similar performance to GroupDRO. These results demonstrate the effectiveness of LISA in improving OOD robustness. In CMNIST, Water- birds, and CelebA, we find that Pse! = 0.5 works well for choosing selection strategies I and II, surprising because it might be more beneficial to use the strategy I more often to eliminate domain effects when there are more domains, i.e., eight domains CivilComments v.S. two domains others.
while Psel is set as 1.0 in CivilComments. This
4.3 ABLATION STUDY: IS THE PERFORMANCE GAIN FROM DATA AUGMENTATION?
In LISA, we apply selective interpolation strategies samples either with the same label but dif- ferent domains or with the same domain but different labels. Here, we explore two substitute in- terpolation strategies: (1) Vanilla mixup: In Vanilla mixup, we do not add any constraint the
5

---

Under review as a conference paper at ICLR 2022
Table Results of subpopulation shifts. Here, we show the average and worst group accuracy. We repeat the experiments three times and put full results with standard deviation in Table 16
CMNIST
Waterbirds
CelebA
CivilComments
Avg.
Worst
Avg.
Worst
Avg.
Worst
Avg.
Worst
ERM
27.8%
0.0%
97.0%
63.7%
94.9%
47.8%
92.2%
56.0%
UW
72.2%
66.0%
95.1%
88.0%
92.9%
83.3%
89.8%
69.2%
JRM
72.1%
70.3%
87.5%
75.6%
94.0%
77.8%
88. 8%
66.3%
CORAL
71.8%
69.5%
90.3%
79.8%
93.8%
76.9%
88.7%
65.6%
GroupDRO
72.3%
68.6%
91.8%
90.6%
92.1%
87.2%
89.9%
70.0%
DomainMix
51.4%
48.0%
76.4%
53.0%
93.4%
65.6%
90.9%
63.6%
Fish
46.9%
35.6%
85.6%
64.0%
93.1%
61.2%
89.8%
71.1% %
LISA (ours)
74.0%
73.3%
91.8%
89.2%
92.4%
89.3%
89.2%
72.6%
sample selection, i.c., the mixup is performed on any pairs of samples; (2) In-group mixup: This strategy upplies data interpolation on samples with the same labels and from the same domains. No- tice that all the substitute interpolation strategies use the same mixup types (e.g.. mixup/Manifold Mixup/CutMix) as LISA. Finally, since upweighting (UW) small groups significantly improves per- formance in subpopulation shifts, we also evaluate UW combined with Vanilla/In-groow mixup.
The ablation results of domain shifts and subpopulation shifts are in Table 3 and Table respec- tively. Furthermore, we also conduct experiments on datasets without spurious correlation in Ta- 18 of Appendix A.3 From the results, we make the following three key observations. First, compared with Vanilla mixup, the performance of LISA verifies that selective data interpolation does improve the out-of- -isdistribtttin robustness by canceling out the spurious correlations and en- couraging invariant representation learning rather than simply data augmentation. This findings further strengthened by the results in Table T8] where Vanilla mixup outperforms LISA and ERM without spurious correlations but LISA achieves the best performance with spurious correlations. Second, the superiority of LISA over In-group mixup verifies that only interpolating samples within each group is incapable of eliminating out the spurious information, where In-group mixup still performs the role of data augmentation. Finally, though incorporating UW significantly improves the performance of Vanilla mixup and In In-group mixup subpopulation shifts, LISA still achieves larger benefits than these enhanced substitute strategies, demonstrating its stronger power in improv- ing OOD robustness.
Table 3: Compared LISA with substitute mixup strategies in domain shifts.
Camelyon17
FMoW
RxRx1
Amazon
MetaShift
Avg. Acc.
Worst Acc.
Avg. Acc.
10-th Per. Acc.
Worst Acc.
ERM Vanilla mixup In- in-group mixup
70.3 1 6.4%
32.8 = 0.45%
29.9 + 0.4%
$3.8 i 0.8%
52.1 + 0.4%
71.2 + 5.3%
34.2 = 0.45%
26.5 + 0.5%
53.3 1 0.0%
51.3 :: 0.7%
75.5 + 6.7%
32.2 1.18%
24.4 + 0.2%
53.8 + 0.6%
52.7 + 0.5%
LISA (ours)
77.1 1. 6.5%
35.5 i 0.65 %
31.9 1. 0.8%
54.7 + 0.0%
54.2 1 0.7%
Table 4: Compared LISA with substitute mixup strategies in subpopulation shifts. UW represents upweighting. Full results with standard deviation listed in Table 17
CMNIST
Waterbirds Avg.
CelebA
CivilComments
Avg.
Worst
Worst
Avg.
Worst
Avg.
Worst
ERM
27.8%
0.0%
97.0%
63.7%
94.9%
47. 37.8%
92.2%
56,0%
Vanilla mixup
32.6%
3.1%
81.0%
56.2%
95.8%
46.4%
90.8%
67.2%
Vanilla mixup UW
72.2%
71.8%
92.1%
85.6%
91.5%
88.0%
87.8%
66.1%
In-group mixup
33.6%
24.0%
88.7%
68.0%
95.2%
58.3%
90.8%
69.2%
In In-group mixup UW
72.6%
71.6%
91.4%
87.1%
92.4%
87.8%
84.8%
69.3%
LISA (ours)
74.0%
73.3%
91.8%
89.2% %
92.4%
89.3%
89.2%
72.6%
4.4
EFFECT OF THE DEGREE OF DISTRIBUTION SHIFTS

---

Under review as a conference paper at ICLR 2022
We further investigate the performance of LISA with respect to the degree of distribution shifts. Here, we use MetaShift to evaluate performance, where the distance between training and test do- meta-
Table 5: Effects of the degree of distribution shifts w.r.t the performance. Distance repre- sents the distribution distance between train- ing and test domains.
mains is measured as the node similarity graph (Liang & Zou 202I). To vary the distance between training and test domains, we change the backgrounds of training objects (see full experi- mental details in Appendix A. The perfor- mance with varied distances is illustrated in Table[5] where the top four best methods (i.e., ERM, Group- DRO, IRM, DimainMix) are reported for compar- ison. We observe that LISA consistently outperforms other methods under all scenarios. Addi- tionally, another interesting finding is that LISA achieves more substantial improvements with the increases of distance. A potential reason is that the models may rely more heavily on domain corre- lations when there is larger distance between training and test domains.
Distance ERM
0.44
0.71
1.12
1.43
80.1%
68.4%
52.1%
33.2%
IRM
79.5%
67.4%
51.8%
32.0%
DomainMix GroupDRO
76.0%
63.7%
51.3%
30.8%
77.0%
68.9%
51.9%
34.2%
LISA (ours)
81.3%
69.7 %
54.2%
37.5%
4.5 ANALYSIS ABOUT LEARNED INVARIANCE
Finally, we analyze the invariance learned by LISA. Specif- ically, we investigate representation-level and prediction- level invariances below.
Table 6: Results of representation- level invariance (x10"), where smaller values denote stronger invariance w.r.t. labels.
Representation-level Invariance. For each label y, as- sume the hidden representation for each domain d as Hý The representation-leve invariance IV, is measured by the pairwise KL divergence of distribution P(H)) among all domains as IV, yللpア Evey Ee LdD
CMNIST
MetaShift
FRM
1.683
0.632
Vanilla mixup IRM
4.392
0.634
.dE KL(P(H) D D= D == d').,, where smaller IV, values indi- cate that the learned representations are more invariant with respect to the labels. We report the results on CMNIST and MetaShift Table 6 Our key observations are: (1) Com-
1.905
0.627
DomainMix
2.155
0.614
LISA (ours)
0.421
0.585
pared with ERM, the invariance of LISA indicates its promise in improving the OOD robustness by encouraging invariant representation learning. (2) LISA has greater invariance than vanilla mixup,
validating that the invariant representations are not caused by naive data interpolation. (3) LISA pro- vides more invariant representations than regularization based methods, i.e., IRM and DomainMix.
Prediction-level Invariance. Motivated by Arjovsky et al. 2019) Krueger et al.] 202T), the prediction-leve invariance Nv is measured by the variance of test risks across all do- mains, i.e., IV, Var( (ar((R, (O), R (D))))) where D rep- resents the number of test domains. A small IV, represents strong prediction-lewe invariance. The results on CMNIST and MetaShift are reported in Table where the superiority of LISA verifies that it could also improve the prediction-level invariance.
Table 7: Results of the analysis of prediction- invariance. Smaller values denote stronger invariance.
CMNIST
MetaShift
ERM Vanilla mixup IRM
12,0486
1.8824
0.2769
0.26559
0.01 12
0.8748
DomainMix
0.1674
1.1158
LISA (ours)
0.0012
0.2387
THEORETICAL ANALYSIS
In this section, we provide some theoretical understandings that explain several of the empirical phenomena from the previous experiments and theoretically compare the worst-group errors of three methods: the proposed LISA, ERM, and vanilla mixup. Specifically, we consider a Gaussian mixture model with subpopulation and domain shifts, which has been widely adopted in theory to shed light upon complex machine learning phenomenon such as in Montanari al. (2019); Zhang et al. (021c). We also note here that despite the popularity of mixup in practice, the theoretical analysis of how mixup (with without the selection strategies) affects the misclassification error is still largely unexplored in the literature even in the simple models. As discussed in Section 2, here, we define y (0, 1} the label, and d E (B,G) G} the domain information. For y (0, 1) and
7

---

Under review as a conference paper at ICLR 2022
d E {B.G), consider the following model:
21/y/ y, d, = d ~ N(llvdd, (.ll)) ) == -.....,,, n(l.d)
(4)
where (lvdd) E RR is the conditional mean vector and ( E IR*x the covariance matrix. Let _0e(o.)).EEBB.o) n(v.d) Let x(3.d) Pm y, dl d), x(0) = Py: y). and (()) = (d, - d).
To account for the spurious correlation brought by domains, we consider ((s. * μ (v. in general for y E {0, 13 and the imbalanced case where (00,B), 7 (21,G) < 1/4. Moreover, we assume there exists some invariance across different domains. Specifically, we assume
je (1 p (0. p (2.)) / (0.G) = A and g =[G)=({B)
According to the theory of Fisher's s linear discriminant analysis rule (Anderson, 1962). the optimal classification rule is linear with slope E-1 The assumption above implies that (> A) T x the (unknown) invariant representation in model equation 4
Suppose we use some method A and obtain linear classifter xb' + by from training data, we will apply it to a test data and compute the worst-group misclassification error, where the mis- classification error for domain d and class y is E(v.)) (b, (b) := P(1 (z? b + bo > 2 vd
d, y/1 y), and we denote the worst-group error with the method as
E(west
E(v.d) (bA, b0.4),
max de (0..)
where ba and A are the slope and intercept based on the method A. Specifically, A = ERM denotes the ERM method (by minimizing the sum of squares loss on the training data altogether), A = mix denotes the vanilla mixup method (without any selection strategy), and A = LISA denotes ((wst)
the mixup strategy for LISA. We also denote its finite sample version by E
Ler A = Elzily 1] E[rrly 0] denote the marginal difference and g A A denote the | |
correlation operator between the domain-specific difference A and the marginal difference A with respect to Σ. We see that smaller & indicates larger discrepancy between the marginal difference and the domain-specific difference and therefore implies stronger spurious correlation between the domains and labels. We present the following theorem showing that our proposed LISA algorithm outperforms the ERM and vanilla mixup in the subpopulation shifts setting.
Theorem 1 (Worst-group error comparison with subpopulation shifts). Consider independent samples generated from model (4). (B) (!1) 1/2. x00.B) -(1.G)
1/4,
maxy,d (| ||2 < C, and E is positive definite. Suppose (5, a) satisfies that & S mint
-للق: 1)
TAIls
for some large enough constant C and || V max(3aar(x)))174
2F
Then for any Pset E [0, 1).
E(ust) LISA
S min E(wsst m
B(wstt mix
+ +Op
p plog n
P
n
where p is the dimension of x. Theorem implies that when & is small (indicating that the domain has strong spurious correlation with the label) and p p= o(an). the worst-group classification emrors of LISA are asymptotically smaller than that of ERM and vanilla mixup. In fact, our analysis shows that LISA yields classification rule closer to the invariant classification rules by leveraging the domain information.
In the next theorem, we present the classaassiiination error comparisons with domain shifts. That is, consider samples from a new unseen domain:
(0,0))) ((),(0,)), Σ), xt (1.4) ~ N(u E).
Let A = (0x( [(ss)), where E[s;;] is the mean of the training distribution, and assume
(1.+) - (0,+) A. Let S* Ahx; A and 7 = ATx 1 denote the correlation for (Ā+, A) الل
|
and for (Ā., A), respectively, with respect to E-1, Let E (wsste) its sample version be (waas))
ma. (bv bo.a) and
8

---

Under review as a conference paper at ICLR 2022
Theorem 2 clasffissiiiiation error comparison with domain shifts). Suppose n samples in-
dependently generated from model (4) T(B) = =1() = =1/2, ( T[
< 1/4,
maxy,d 1| 1l2 I C' and E is positive definite. Suppose that (5,5",7) ,)) satisfy that 0 I
5* I 75 and & S minf
| 13 - Ca for some large enough constant C' and || S A
2
Then for any Paet [0,
max (axxa
E(va()))
S <min{ E(wate) E(wat+)) + +p
pplog n
P OWT2
71
Similar to Theorem D this result shows that when domain has strong spurious correlation with the label (corresponding small (). such spurious correlation leads to the downgraded performance of ERM and vanilla mixup, while our proposed LISA method is able to mitigate such an issue by selective data interpolation. Proofs of Theorem T and Theorem
2 are provided in Appendix B
RELATED WORK AND DISCUSSION
In this paper, we focus on improving the robustness of machine learning models to domain shifts and subpopulation shifts. Here, we discuss related approaches from the following three categories:
Learning Invariant Representations with Domain Alignment. Motivated by unsupervised do- main adaptation (Ben-David et al. 2010; Ganin et al.) 2016), the first category of works learns invariant representations by aligning representations across domains. The major research line of this category aims to eliminate the domain dependency by minimizing the divergence of feature distri- butions with different distance metrics, e.g., maximum mean discrepancy (Tzeng et al.) 2014 Long et al. 2015), adversarial loss (Ganin et al., 2016 Li et al.) 2018), Wassertein distance (Zhou al.
2020a). Follow-up works applied data augmentation to (1) generate more domains and enhance the
consistency of representations during training (Yue et al. 2019 Zhou et al. 20206 Xu al. 2020
Yan et al. 2020 Shu et al. 2021: Wang et al.) 2020) or (2) generate new domains in an adversar
ial way to imitate the challenging domains without using training domain information (Zhao al.]
2020 Qiao et al.) 2020 Volpi et al. 2018). Unlike these latter methods. LISA instead focuses on
canceling out correlations the dataset between the domain and the label through selective data
interpolation, leading to stronger empirical performance.
Learning Invariant Representations with Invariant Predictors. Beyond using domain alignment to learning invariant representations, recent work aims to further enhance the correlations between the invariant representations and the labels (Koyama Yamaguchil (020). Representatively, moti- vated by casual inference, invariant risk minimization (IRM) (Arjovsky et al. 2019) aims to find predictor that performs well across all domains. After IRM, the following works propose stronger regularizers by penalizing the variance of risks across all domains (Krueger et al. 2021 by align- ing the gradient across domains (Koyama & Yamaguchi] 2020), by smoothing the cross-domain interpolation paths (Chuang & Mrouch 2021). or through game-theoretic invariant rationalization criterion (Chang et al. 2020). Instead of using regularization, LISA eliminates spurious correlations in the data directly via data interpolation.
Group Robustness. The last category of methods combats spurious correlations and are particularly suitable for subpopulation shifts. These approaches include directly optimizing the worst- group performance with Distributionally Robust Optimization (Sagawa et al.) 2020aa Zhang et al.) 202 Zhou et al., 202T), generating samples around the minority groups (Goel et al. (021). and balancing the majority and minority groups via reweighting (Sagawa et al. (2200b) or regularizing (Cao et al.) 2019 2020). Here, LISA proposes a more general strategy based on data augmentation that is suitable for both domain shifts and subpopulation shifts.
7 CONCLUSION
To tackle the distribution shifts, we propose LISA, a simple and efficient algorithm, to improve the out-of-distribution robustness. LISA aims to eliminate the domain-related spurious correlations among the training set by selective sample interpolation. We evaluate the effectiveness of LISA on nine datasets under subpopulation shifts and domain shifts settings, demonstrating its promise. Our theoretical results further strengthen the superiority of LISA by showing smaller worst-group mis-classification error compared with ERM and vanilla mixup.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We conduct experiments under the setting of domain shifts and subpopulation shifts problems. In terms of the domain shifts, the results are reported in in Table [ The dataset details are pro- vided in Appendix A.1.I and the training details along with the hyperparameter settings are in appendix [A..2] Then, for the subpopulation shifts, we have included the full results including the error bounds in Table [6] The dataset details are discussed in Appendix AA.2II] while the train-
ing details and the hyperparameter settings are Appendix A.2.2 We will release the code upon publication. Besides, The detailed proof of Theorem 1 and Theorem
2 are provided in Appendix
REFERENCES
Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and loannis Mitliagkas. Generalizing to unseen domains via distribution matching. arXiv preprint arXiv: 191 1.00804, 2019.
Theodore Wilbur Anderson. An introduction to multivariate statistical analysis. Technical report, Wiley New York, 1962.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez- Pazz Invariant risk minimization. arXiv preprint arXiv:1907.0289) 2019.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 2018.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory learning from different domains. Machine learning, 19(1)1151-175, 2010.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion pro- ceedings of the 2019 world wide web conference, pp. 491-500, 2019.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution--wwaee margin loss. NeurlPS, 2019.
Kaidi Cao, Yining Chen, Junwei Lu, Nikos Arechiga, Adrien Gaidon, and Tengyu Ma. Het- eroskedastic and imbalanced deep learning with adaptive regularization. arXiv preprint arXiv:2006.15766 2020.
Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. Invariant rationalization. In International Conference on Machine Learning, pp. 1448-1458. PMLR, 2020.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2018.
Ching-Yar Yao Chuang and Youssef Mroueh. Fair mixup: Fairness via interpolation. ICLR, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference machine learning, pp. 1180-1189, PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarte training of neural net- works. The journal of machine learning research, 17(1):20662003 2016.
Karan Goel, Albert Gu, Yixuan Li, and Christopher Ré. Model patching: Closing the subgroup performance gap with data augmentation. In ICLR, 2021.
10

---

Under review as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Pang Wei Koh, Shiori Sagawa, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-
5664. PMLR, 2021.
Masanori Koyama and Shoichiro Yamaguchi. Out-of-distribution generalization with maximal in- variant predictor. arXiv preprint arXiv:2008.04 0/888, 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. 2016. URL
07332
David Krueger, Ethan Caballero, Joemn- Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribation generalization via risk extrapo- lation (rex). In International Conference on Machine Learning, pp. 5815-5826. PMLR, 2021.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adver- sarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5400-5409, 2018.
Weixin Liang and James Zou. Metadataset: A dataset of datasets for evaluating distribution shifts and training conflicts. In /CML202] MLAdata Workshop, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In I,CV,,,211 2015.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105. PMLR, 2015.
Andrea Montanari, Feng Ruan, Youngtak Sohn, and Jun Yan. The generalization error of max- margin linear classifiers: High-dimensional asymptotics in the overparametrized regime. arXiv preprint arXiv:1911.0154: 2019.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. International Conference on Machine Learning, pp. 10-18. PMLR, 2013.
Jianmo Ni. Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly- labeled reviews and fne-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPP-JCCIPP) 2019.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12556 12565, 2020.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. In ICLR, 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. In ICLR, 2020a.
11

---

Under review as a conference paper at ICLR 2022
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In ICML, pp. 8346883668880 8356. PMLR, 2020b.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 1xXX:::910.011000 2019.
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.09937) 2021.
Yang Shu, Zhangjie Cao, Chenyu Wang. Jianmin Wang, and Mingsheng Long. Open domain gen- eralization with -ummin--uggmeted meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9624-9633, 2021.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference computer vision, pp. 443-450. Springer, 2016.
J. Taylor, B. Earnshaw, B. Mabey, M. Victors, and J. Yosinski. RxIxI: An image set for cellular mor- phological variation across many experimental batches. In International Conference on Learning Representations (ICLR), 2019.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 2014.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pp. 1438-6447. PMLR, 2019.
Riccardo Volpi, Hongseok Namkoong. Ozan Sener, John Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. arXiv preprint 1805 2018.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-00 California Institute of Technology. 2011.
Yufei Wang. Haoliang Li, and Alex C Kot. Heterogeneous domain generalization via domain mixup. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 3622-36222 IEEE, 2020.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6502-6509, 2020.
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint wrXiv:2011.00677 2020.
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni- Sungiovammi-Vincenellle Kurt Keutzer, and Bo- qing Gong. Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2100-21 2019.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. 2018.
Jingzhao Zhang, Aditya Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. In ICLR, 2021a.
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In ICLR, 2021b.
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration. arXiv preprint arXiv:2102.06289, 2021c.
12

---

Under review as a conference paper at ICLR 2022
Long Zhao, Ting Liu, Xi Peng, and Dimitris Metaxas. Maximum- euxtopy adversarial data augmen- tation for improved generalization and robustness. arXiv preprint arXiv:2010.08001, 2020.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10
million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6): 1452-14644 2017.
Chunting Zhou, Xuezhe Ma, Paul Michel, and Graham Neubig. Examining and combating spurious features under distribution shift. In ICML, 2021.
Fan Zhou, Zhuqing Jiang, Changjian Shui, Boyu Wang, and Brahim Chaib-draa, Domain general- ization with optimal transport and metric learning. arXiv preprint 200:::000..1075: 2020a.
im- Kaiyang Zhou, Yongxin Yang. Timothy Hospedales, and Tao Xiang. Deep age generation for domain generalisation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 13025-13032, 2020b.
A
ADDITIONAL EXPERIMENTS
A.1 DOMAIN SHIFTS
A.1.1 DATASET DETAILS
In this section, we provide detailed descriptions of datasets used in the experiments of domain shifts and report the data statistics in Table g
Camelyon17 We Camelyon17 from the WILDS benchmark (Koh al. 2021 Bandi al., 2018), which provides 450, 000 lymph-node scans sampled from 5 hospitals. Camelyon17 is a medical image classification task where the input is 96 96 image and the label y is whether there exists tumor tissue in the image. The domain d denotes the hospital that the patch was taken from. The training dataset is drawn from the first hospitals, while out-of-distribtiion validation and out-of- distriibbuiion test datasets are sampled from the 4-th hospital and 5-th hospital respectively.
FMoW
The FMoW dataset is from the WILDS benchmark (Koh et al. 2021 Christie et al. 2018)
satellite image classification task which includes 62 classes and 80 domains (16 years regions). Concretely, the input æ is a 224 x 224 RGB satellite image, the label y is one of the 62 building land use categories, and the domain d represents the year that the image was taken as well as its corresponding geographical region Africa, the Americas. Oceania, Asia, or Europe. The train/testvalldaation splits are based on the time when the images are taken. Specifically, images taken before 2013 are used as the training set. Images taken between 2013 and 2015 are used as the validation set. Images taken after 2015 are used for testing.
RxRx1 RxRx1 (Koh al. 2021 Taylor et al., 2019) from the WILDS benchmark is cell image classification task. In the dataset, some cells have been genetically perturbed by siRNA. The goal of RxRx1 to predict which siRNA that the cells have been treated with. Concretely, the input x an image of cells obtained by fluorescent microscopy, the label indicates which of the 1, 139 genetic treatments the cells received, and the domain d denotes the experimental batches. Here, 33 different batches of images are used for training, where each batch contains one sample for each class. The -ut-t--distiibution validation set has images from 4 experimental batches. The outtt--distriuution test set has 14 experimental batches. The average accuracy
test set is reported.
Amazon Each task in the Amazon benchmark (Koh et al., 2021 Ni et al.) 2019) is multi-class sentiment classification task. The input x is the text a review, the label y is the corresponding star rating ranging from 1 to 5, and the domain d is the corresponding reviewer. The training set has 245, 502 reviews from 1, 252 reviewers, while the -istt--diittibutiin validation set has 100, 050 reviews from another 1, 334 reviewers. The out-of-distributte from the rest 1, 252 reviewers. We evaluate the models by the 10th percentile of per-user accuracies in the test set.
test set also has 100, 050 reviews
13

---

Under review as a conference paper at ICLR 2022
MetaShift We use the MetaShift (Liang & Zou 2021). which is derived from Visual Genome (Kr- ishna et al. 2016). MetaShift leverages the natural heterogeneity of Visual Genome to provide many distinct data distributions for given class (e.g. "cats with cars" or "cats in bathroom" for the "cat" class). A key feature of MetaShift that it provides explicit explanations of the dataset correlation and distance score measure the degree of distribution shift between any pair of sets. We adopt the "Cat vs. Dog" task in MetaShift, where we evaluate the model on the "dog(shelf)) domain with 306 images, and the "cat(shelf)" domain with 235 images. The training data for the "Cat" class is the cat(sofa + bed), including cat(sofa) domain and cat(bed) domain. MetaShift provides 4 different sets of training data for the "Dog" class in an increasingly challenging order, i.e. increasing the amount of distribution shift. Specifically, dog(cabinet + bed), dog(bug box), dog(bench + bike), dog(boul + surfboard) are selected for training, where their corresponding distances to dog(shelf) are 0.44, 0.71, 1.12, 1.43.
Table 8: Dataset Statistics for Domain Shifts.
Datasets
Domains
Metric
Base Model
Num. of classes
Camelyon17
hospitals
Avg. Acc. Worst-group Ace. Avg. Acc.
DenseNet- 121
FMoW
16 years x 5 regions
DenseNet- 121
62
RxRx1
51 experimental batches
ResNct-50
1,139
Amazon
7, 7,676 reviewers
10th Percentile Acc.
DistilBERT
5
MetaShift
backgrounds
Worst-group Acc.
ResNet-50
A.1.2 TRAINING DETAILS
Follow WILDS Koh et al. 2021). we adopt rre-trained DenseNet121 (Huang et al. 2017) for Came- lyon17 and FMoW datasets, ResNet-50 (He et al. 2016) for RxRx1 and MetaShift datasets, and DistilBert (Sanh et al. 2019) for Amazon datasets.
In each training iteration, we first draw a batch of samples B1 from the training set. With B1, we then select another sample batch B, with same labels as B1 for data interpolation. The interpolation ratio À is drawn from the distribution Beta(2, 2). We use the same image transformers as Koh al. 2021), and all other hyperparameter settings are listed in Table
Table 9: Hyperparameter settings for the domain shifts.
Dataset
Camelyon17
FMoW
RxRx1 1c-3
Amazon
MetaShift
Learning rate Weight decay
lc-4
lc-4
2c-6
1e-3
0
0
le-5
0
le-4
Scheduler
n/a
n/a
Cosine Warmup
n/a
n/a
Batch size
32
32
72
8
Type mixup
CutMix
CutMix
CutMix ResNet50
ManifoldMix DistilBert Adam 3
CutMix
Architecture
DenseNet121
DenseNet121 Adam 5
ResNet50
Optimizer
SGD
Adam 90
SGD
Maximum Epoch
2
100
Strategy sel. prob. Psel
1.0
1.0
1.0
1.0
1.0
A.1.3 FULL RESULTS OF WILDS DATA
Follow Koh al. (202I). we reported more results on WILDS datasets in Table 10 Table 13 including validation performance and the results of other metrics. According to these additional results, we could see that LISA outperforms other baseline approaches in all scenarios. Particularly, we here discuss two additional findings: (1) In Camelyon dataset, the test data is much more visually distinctive compared with the validation data, resulting in the large gap (~ 10%) between validation and test performance of ERM (see Table [0). However, LISA significantly reduces the performance gap between the validation and test sets, showing its promise in improving OOD robustness; (2) In Amazon dataset, though LISA performs worse than ERM in average accuracy, it achieves the best accuracy at the 10th percentile, which is regarded as a more common and important metric to evaluate whether models perform consistently well across all users (Koh al. 2021
14

---

Under review as a conference paper at ICLR 2022
Table 10: Full Results of Camelyon17. We report both validation accuracy and test accuracy.
Validation Acc.
Test Acc.
ERM
84.9 + 3.1%
70.3 + 6.4%
IRM
86.2 + 1.4%
64,2 +: 8.1%
Coral
86.2 + 1.4%
59.5 + 7.7%
GroupDRO
85.5 2.2%
68.4 + 7.3%
DomainMix
83.5 - 1.1%
69.7 + 5,5%
Fish
83.9 + 1.2%
74.7 +: 7.1%
LISA (ours)
81.8 = 1.3%
77.1 + 6.5%
Table 11: Full Results of FMoW. Here, we report the average accuracy and the worst-domain accu- racy on both validation and test sets.
Validation
Test
Avg. Acc.
Worst Acc.
Avg. Acc.
Worst Acc.
ERM
59,5 0.37%
48.9 0.62%
53.0 0.55
32.3 1.25%
IRM
57.4 0.37%
47.5 + 1.57%
50.8 + 0.13%
30.0 + 1.37%
Coral
56.9 + 0.25%
47.1 + 0.43%
50.5 + 0.36%
31.7 + 1.24%
DRO
58.8 + 0.19%
46.5 + 0.25%
52.1 + 0.50%
30.8 + 0.81%
DomainMix
58.6 0.29%
48.9 = 1.15%
51.6 = 0.19%
34.2 + 0.76%
Fish
57.8 0.15%
49.5 : 2.34%
51.8 + 0.32%
34.6 0.18%
LISA (ours)
58.7 0.92%
48.7
0.74%
52.8 + 0.94%
35.5 0.65%
A.2 SUBPOPULATION SHIFTS
A.2.1 DATASET DETAILS
We detail the data descriptions of subpopulation shifts below and report the detailed data statistics
Table 14
Colored MNIST (CMNIST): We classify MNIST digits from 2 classes, where classes 0 and indicate original digits 10,1,2,3,4) and (5,6,7,8,9). The color is treated as a spurious attribute. Con- cretely, in the training set, the proportion between red samples and green samples is 8:2 in class O, while the proportion is set as 2:8 in class 1. In the validation set, the proportion between green and red samples is 1:1 for all classes. In the test set, the proportion between green and red samples is 1:9 in class O, while the ratio is 9:1 in class 1. The data sizes of train, validation, and test sets are 30000, 10000, and 20000, respectively.
Waterbirds (Sagawa et al., 2020a): The Waterbirds dataset aims to classify birds as "waterbird" or "landbird"", where each bird image is spuriously associated with the background "water" or "land".. Waterbirds synthetic dataset where each image is composed by pasting a bird image sampled from CUB dataset (Wah al., 2011 to a background drawn from the Places dataset Zhou et al. (017). The bird categories in CUB are stratified as land birds or water birds. Specifically, the tollowing bird species are selected construct the waterbird class: albatross, auklet, cormorant, frigatebird, fulmar, gull, jaeger, kittiwake, pelican, puffin, tern, gadwall, grebe, mallard, merganser, guillemot, or Pacific loon. All other bird species are combined as the landbird class. We define (land background, waterbird) and (water background, landbird) are minority groups. There are 4,795 training samples while only samples are "waterbirds on land" and 184 samples are "landbirds on water". The remaining training data include 3,498 samples from "landbirds land", and 1,057 samples from "waterbirds on water".
CelebA (Liu et al. 2015 Sagawa al.) 2020a): For the CelebA data (Liu et al. 2015), we follow the data preprocess procedure from Sagawa et al. (2020a). CelebA defines a image classification task where the input is face image of celebrities and the classification label is its corresponding hair color - "blond" or "not blond." The label is spuriously correlated with gender, i.c., male or female. In CelebA, the minority groups are (blond, male) and (not blond, female). The number of
15

---

Under review as a conference paper at ICLR 2022
Table 12: Full Results RxRxI. ID: in-distribution; OOD: out-of-distrriutiin
Validation Acc.
Test II) Acc.
Test OOD Acc.
ERM
19.4 + 0.2%
35,9 0.4%
29.9 1 0,4%
IRM
5.6 + 0.4%
9.9 + 1.4%
8.2 + 1.1%
Coral
18.5 + 0.4%
34.0 + 0.3%
28.4 + 0.3%
GroupDRO
15.2 0.1%
28.1 0.3%
23.0 + 0.3%
DomainMix
19.3 19.3 0.7%
39.8 0.2%
30.8 + 0.4%
Fish
7.5 1 0.6%
12.7 1.9%
10.1 + 1.5%
LISA (ours)
20.1 + 0.4%
41.2 + 1.0%
31.9 + 0.8%
Table 13: Full Results of Amazon. Both the average accuracy and the 10th Percentile accuracy reported.
Validation
Test
Avg. Acc.
10-th Per.
Avg. Acc.
10-th Per. Acc.
ERM
72.7 0.1%
55.2 0.7
71.9 0.1 %
53.8 = 0.8%
IRM
71.5 + 0.3%
54.2 0.8%
70.5 + 0.3%
52.4 = 0.8%
Coral
72.0 + 0.3%
54.7 + 0.0%
70.0 + 0.6%
52.9 + 0.8%
DRO
70.7 + 0.6%
54.7 + 0.0%
70.0 + 0.6%
53.3 + 0.0%
DoxainMix
71.9 + 0.2%
54.7 + 0.0%
71.1 0.1%
53.3 0.0%
Fish
72.5 0.0%
54.7 0.0%
71.7 0.1%
53.3 = 0.0%
LISA (ours)
71.2 0.3%
55.1 0.61 %
70.6 0.3%
54.7 = 0.0%
samples for each group are 71,629 "dark hair, female", 66.874 "dark hair, male". 22,880 "blond hair, female", 1,387 "blond hair, male".
CivilComments (Borkan al.) 2019 Koh et al. 2021 We use CivilComments from the WILDS benchmark (Koh et al. 2021). CivilComments is text classification task, aiming to predict whether an online comment is toxic or non-toxic. The spurious domain identifications are defined as the demographic features, including male, female, LGBTQ, Christian, Muslim, other religion, Black, and White. CivilComments contains 450,000 comments collected from online articles. The number of samples for training, validation. and test are 269, 269,038, 45,180, and 133,782, respectively. The readers may kindly refer to Table 17 in Koh et al. (2021) for the detailed group information.
Table 14: Dataset Statistics for Subpopulation Shifts. All datasets are binary classification tasks and use the worst group accuracy the evaluation metric.
Datasets
Domains
Base Model
Class Information
CMNIST
2 digit colors
ResNet-50 ResNet-50
digit (0,1,2,3,4) v.s. (5,6,7,8,9) waterbirds v.s. landhirds
Waterbirds
2 backgrounds
CelehA
hair colors demographic identities
ResNet-50
man v.s. women toxic non-toxic
CivilComments
DistilBEERT-uncased
A.2.2 TRAINING DETAILS
We adopt pre-trained ResNet-50 (He et al. 2016) and BERT (Sanh et al.] 2019) as the model for image data (i.e., CMNIST, Waterbirds, CelebA) and text data (i.c., CivilComments), respectively. In each training iteration, we sample batch of data per group. For sample selection strategy I, we randomly apply mixup on sample batches with the same labels but different domains. For sample selection strategy II, instead apply mixup sample batches with the same domain but different labels. The interpolation ratio is sampled from the distribution Beta(2, 2). All hyperparameters listed in Table 15
16

---

Under review as a conference paper at ICLR 2022
Table 15: Hyperparameler settings for the subpopulation shifts.
Dataset
CMNIST
Waterbirds
CelebA
CivilComments
Learning rate
le-3
le-3
le-4
le-5
Weight decay
le-4
le-4
le-4
0
Scheduler
n/a
n/a
Batch size
16
16
Type of mixup
mixup
mixup
CutMix
ManifoldMix
Architecture
ResNet50
ResNet50
ResNet50 SGD
DistilBert Adam
Optimizer
SGD
SGD
Maximum Epoch
300
300
Strategy sel. prob. Pool
0.5
0.5
0.5
1.0
A.2.3 ADDITIONAL. RESULTS
In this section, we have added the full results of subpopulation shifts in Table and Table 17
Table 16: Full results of subpopulation shifts with standard deviation. All the results are performed with three random seed.
CMNIST
Waterbirds
Avg.
Worst
Avg.
Worst
ERM
27.8 + 1.9%
0.0 + 0.0%
97.0 + 0.2%
63.7 + 1.9%
UW
72.2 = 1.1%
66.0 + 0.7%
95.1 = 0.3%
88,0 - 1.3%
IRM
72.1 1.2%
70.3 0.8%
87.5 0.7%
75.6 3.1%
Coral
71.8 - 1.7%
69.5 + 0.9%
90.3 = 1.1%
79.8 = 1.8%
GroupDRO
72.3 + 1.2%
68.6 1 0.8%
91.8 + 0.3%
90.6 1.1 %
DomainMix
51.4 + 1.3%
48.0 + 1.3%
76.4 + 0.3%
53.0 + 1.3%
Fish
46.9 + 1.4%
35.6 + 1.7%
85.6 + 0.4%
64.0 + 0.3%
LISA
74.0 + 0.1%
73.3
0.2%
91.8 + 0.3%
89.2 + 0.6%
CelebA
CivilComments
Avg.
Worst
Avg.
Worst
ERM
94.9 + 0.2%
47.8 + 3.7%
92.2 + 0.1%
56.0 +: 3.6%
UW
92.9 + 0.2%
83.3 + 2.8%
89.8 + 0.5%
69.2 + 0.9%
IRM
94.0 0.4%
77.8 + 3.9%
88.8 = 0.7%
66.3 = 2.1%
Coral
93.8 0.3%
76.9 + 3.6%
88.7 =
65.6 1.3%
GroupDRO
92.1
87.2 : 1.6%
89.9 + 0.5%
70.0 + 2.0%
DomainMix
93.4 + 0.1%
65.6 + 1.7%
90.9 + 0.4%
63.6 + 2.5%
Fish
93.1 + 0.3%
61.2 + 2.5%
89.8 + 0.4%
71.1 + 0.4%
LISA (ours)
92.4 +: 0.4%
89.3 + 1.1%
89.2 1 0.9%
72.6 + 0.1%
A.3 RESULTS ON DATASETS WITHOOUT SPURIOUS CORRELATIONS
In order to analyze the factors that lead to the performance gains of LISA, we conduct experiments on datasets without spurious correlations. To be more specific, we balance the number of samples for each group under the subpopulation shifts setting. The results of ERM, Vanilla mixup and LISA on CMNIST, Waterbirds and CelebA are reported in Table 18 The results show that LISA per- forms similarly compared with ERM when datasets do not have spurious correlations. If there exists any spurious correlation, LISA significantly outperforms ERM. Another interesting finding is that Vanilla mixup outperforms LISA and ERM without spurious correlations, while LISA achieves the best performance with spurious correlations. This finding strengthens our conclusion that the perfor- mance gains LISA are caused by eliminating spurious correlations rather than data augmentation.
17

---

Under review as a conference paper at ICLR 2022
Table 17: Full table of the comparison between LISA and other substitute mixup strategies in sub- population shifts. UW represents upweighting.
CMNIST
Waterbirds
Avg.
Worst
Avg.
Worst
ERM
27.8 1.9%
0.0 0.0%
97.0 0.2%
63.7 1.9%
Vanilla mixup
32.6 2.6::30 3.1%
3.1 2.4%
81.0 0.2%
56.2 0.2%
Vamilla mixup + UW
72.2 + 0.7%
71.8 + 0.1%
92.1 + 0.1%
85.6 + 1.0%
In-group Group
33.6 1.9%
24.0 + 1.1%
88.7 0.3%
68.0 + 0,4%
In-group + UW
72.6 0.1%
71.6
0.2%
91.4 0.6%
87.1 0.6%
LISA (ours)
74.0 0.1%
73.3 + 0.2%
91.8 0.3%
89.2 + 0.6%
CelebA
CivilComments
Avg.
Worst
Avg.
Worst
ERM
94.9 + 0.2%
47.8 + 3.7%
92.2 + 0.1 %
56.0 + 3.6%
Vanilla mixup
95.8 + 0.0%
46.4 + 0.5%
90.8 0.8%
67.2 + 1,2%
Vanilla mixup UW
91.5 1 0.2%
88.0 1 0.3%
87.8 1.2%
66.1 1.4%
Within Group
95.2 0.3%
58.3
0.9%
90.8 0.6%
69.2 0.8%
Within Group + UW
92.4 0.4%
87.8 1 0.6%
84.8 0.7%
69.3 1.1%
LISA (ours)
92.4 0.4%
89.3 1.1 %
89.2 0.9%
72.6 0.1%
Table 18: Results on Datasets without Spurious Correlations
Dataset
ERM
Vanilla mixup
LISA
CMNIST 73.67%
74.28%
73.18%
Waterbirds 88.07%
88. 23%
87.05%
CelebA 86.11%
88.89%
87.22%
B PROOFS OF THEOREM 1 AND THEOREM 2
Outline of the proof. We will first find the mis-classification errors based on the population version of OLS with different mixup strategies. Next, we will develop the convergence rate of the empirical OLS based on n samples towards its population version. These two steps together give us the empirical mis-classification errors of different methods. We will separately show that the upper bounds in Theorem [ and Theorem 2hold for two strategies of LISA and hence hold for any Pael E [0, 1]. Let L1 denote selection strategy I of LISA method and L.2 denote selection strategy II of LISA method.
Let 1 (y, 1) and To (w 0) denote the marginal class proportions in the training sam- ples. Let л = (d, = B) and TG = PP (d, = G) denote the marginal subpopulation proportions in the training samples. Let ae (d, Gl y/ = 1) and define Tyjo- TBI1 and T similarly.
We consider the setting where a == x (1.G) न (). relatively small and (1) =0)) xG) (B) 1/2.
B.I DECOMPOSING THE LOSS FUNCTION
Recall that A (1.G) (0,00 = ((l B) (0.B), We further define Á = م() - ()) g(a) = (0,G) B[r;]] and ((B) μ (0, E[x,).
For the mixup estimators, we will repeatedly use the fact that 2g has a symmetric distribution with support (0. 1).
18

---

Under review as a conference paper at ICLR 2022
For ERM estimator based on (X, v). where by = - E[x,,'b, we have
(n(0.G)) + bo =(100.) ]F(s:))
= =(0lo)) Ely]]
( bo (41.0)- -
= A (0(G))) Elysl
Notice that based on the estimator b, bo
gl,d) (b, bu) Φ = (B
and (t,.d) (o),bb)) 2 (0(o))Tb)
Eb
Vo- Σb
B.2
CLASSIFICATION ERRORS OF FOUR METHODS WITH INFINITE TRAINING SAMPLES We first provide the limit of the classification errors when + cc.
B.2.1 BASELINE METHOD: ERM
For the training data, it is easy to show that
var var(r) = E [[var(x]])) y)] + var(E(x))))
Σ + [Eu D1l1)] + var((u(1) -1(0))))
+ Flvar(u (0. n1 (0.0)))( = B)lw)l (1)
Σ 7 - ( (TBnTTTe) Tooo + 00
1
coc(a, v) = cov( [[<|by),y)
= cov(u (0) Ây y)
cou(Ay, y) Åm (1)
For an being
2 (TBITTG11 + T and Do ((0000) - (), the ERM has slope and intercept
b var(x) cou((1y))
(E + 20 40 222 -
an N
Σ Σー Do
1 4
by = Elyy FF b].
In the extreme case where 80,B = GG 0, we have
A = (1. μ (0.G) (( = - ((B)
2 = - A
A,
and Ao = A A.
2
Hence,
(+ā (HÂ- A)Tb
-
Efwatt = maxf#
}.
(5)
VbTEb
VBTEb
where b is computed via ERM.
B.2.2 BASELINE METHOD: VANILLA MIXUP
The vanilla mixup does not use the group information. Let é be a random draw from (1....,2} n.) Let ig be random draw from (1, (1,...,m) n) independent of i, Let
ft Aati, + (1
19

---

Under review as a conference paper at ICLR 2022
and
i = X, + +(1- - A1)
We can find that
con Ui) cou(A + - X1) Aiyli, + (1 - >1)
= cou(A,Ti A:yi; + cov((1 - 2,) (1 - )1)8)sa) (1o[A + [B((1 - ) (oov(I..) v.). (0, = (EaAa) + [ - Ax) coo((())
Hence, the population-level slope is the same as the slope in the benchmark method. It is easy to show that the population-Ieve intercept is also the same. Hence,
E Elust) mix
(wws))
E
B.3 LISA WITH SELECTION STRATEGY (LISA-II): MIXUP WITHIN BACH CLASS Define
2 = A, + - -Xc)arr
(1).0)
(1.
where i is a random draw from {1 : 1/1 = ye Dj = G} and in is a random draw from (1 : y
y/, Dj BJ. Then we perform OLS based on (a (2)
:B),, i .l....,n
We can calculate that
con(x! y/i = cor (E[E (A) 19i1, U:) = coul 1 Fu + (o. yi)
var(v;) 1 π (1)) A
coo(2 (A)) = E[cov( (1) Ui X;)] + cow (E|z) 1Bs
1B5. A:l)
21E[AAE can (4 (0.0) - (... + Ay:
= [TEEA + (ar():) - ().B8)) + π (0) 82
Notice that
(1) ] 1 (110000) + +1(1.00
1
+ (0.B) + (10
2 (1 + (10,0)) = F(a:))
1
For OL1 varvar (2T[ and Do = (0,0) - 10,. the OLS has slope and intercept being
b = varls; (A)) cono (,,
x ( var (di)
(11 (0,0) 11 10.8))82
2|[
OLI A Dn
Aーート Σー
1 OLi A A
bp = Elyl - (1(()A))TT)).
E(we() maxf (33 (クD-A)7b) A)Tb)
-1A
Vb-'bb Eb
Vor
where b is computed based on (x! (A) vy). i ,....,n
B.4 LISA WITH SELECTION STRATEGY II (LISA-II): MIXUP WITHIN EACH DOMAIN
The interpolated sample can be written
(0., ..) (As, X:2 (1.
(1,0) + +(1 - 210 if G
(0.. = (Ai, Apar
+ (1 11)a () if = B,
20

---

Under review as a conference paper at ICLR 2022
where i is random draw from {I d; = di y: = 1} and ig is random draw from {i : d₃ d, y/4 = 0}.
We consider regress yi on Ty..
cou(T;, Bild, G) = cov dj = G) Id; = G) = (1
var(ā;id, = G) Lilvar A,, D; G)ld; G] + var var (Bi d1, = GT|D; G]
= = + var (A111 (1.) +(1- - 11) (0. GG)
2 + var (7 1 82
We further have
cov(E,, ū+) Elcov( [s])) + con (E(E; dd), El (d,)
cou(i!!) .g.0) )=())
(G) + cor(= ()) y! (1) (BB) (B)
var(ū.) (ir (T. var(ū ( (1 (1
var (ar(
Moreover,
vār( B[var (.. [d:] + var (L(; var(z!
(0) Yom (G) + var( oo (B) + (E] B[ (0))
2TE[A + var(A:) 82 (1 , (C)
Slope:
b -ar(5;)))co0((.. cor( Ki)
α (I + 4,2
(oo) 7 1+ (0)) TE--A
= Σ A
T(8) (G) 2
where 0L2
Moreover, bo E
E[5:] J7 b- - - B[.:|'' Notice that
E[ā.]
(1 (000)) + م (1.0) + (0..B) + (1(1111)
4 (2 + A + 24 A)
2 (1 + (1(.BB)) = B|xx]]
Hence,
( ク) T)
F(ustt max
VoTEb
where 6 is computed based on (ã,, V.), i = I,...,
Method comparison. We only need compare 5) 16. and 7
For the ERM, 0 < og S 2a and
a0 ATE Do
an A T 'oo -I'A
b b= (1
- A
1+ ag
A Σ 1
1+ A7 E 1o00
á AĀT Do
x -
1 0 A TA Ao + a0 A A-'
a0 y AAD 'A
1+ ag
T T Eー1
21

---

Under review as a conference paper at ICLR 2022
Let co
00.3 y An I+ A 1oo
and c1 |colll | 2/ ||| g. For simplicity, let |/l||
-. We first
lower bound via
AT A COATE-IA A
cor(BERM. Α)
| L b Eb
A 2 V 26
A Σーン
I A
A [|| A |1 + (col|A|1x)
|| E6 E6
1
co A E
+
||>|||/
| Σ
co 1|
A|E
1 (1 E) of
Ca.
Similarly, we have
A
AE A CA A
cor OOERR, A)
|A
b7 Eb
| El A ATΣ-AA
AT I
A ||x A 115: + coll
(I >: col coll |b:) A|1s
co | A / A ||>
+ col |/|///////||11
co A / /|
> 1x>
+ c1
1
Hence,
( 5 - - (E-a)\l))(()))))))s))s,)) (((- +ca))|/1>))
ERRM max
(8)
for some constant C' depending the true parameters.
For method LISA-I, using the fact that Ao
A A
ØLI A T Σ Ao
OLD ALA Do ΣーI
bL (1 1 01 A E 'A0
DΣ A
1 01 A A.-IAA
α Σ --' 11.-14 AA
for
1+ 1+ -., A
9.1
ALITES Ao
Hence,
A TTh h1
| + A
cor(bl1,))
||ا| b V
b Ebu
A + +
b A
/| 1 A + A A
cor(bL1,
| v
Eb.
To have E(wst) Ca)ll A| and
< FRM it suffices require that Ca)l S -a)) - -E+
cor(bri, |||| cor(br,, A)|A||>> &
1 Co)l| (E+ Cc)|AA
cor A A S 1
-c))
(a) (E Ca)
22

---

Under review as a conference paper at ICLR 2022
A sufficient condition is
$ 5 2 cor(bbi. A))
A
Ca, cor(bLI. A) 2 + Ca, cor(bLi, A) S -Ca.
|
We can find that a further sufficient condition is
A
A + CL 4
| b
Ca, en1
0,55
61a CLi A
(9)
A
E
CLi A|ls
A
+ 9. A
x A E E
61
(10)
A + a. A ||x
||
( 1 1 + cor (bLI, A))
A x
cortbbi
Ca.
(11)
We first find sufficient conditions for the statements in
9 and 10
Parameterizing t
a.llA
|| further simplify the condition in (9) and 10
د|اا s Ca, t > 0,
t SESE
2
1 + 12 + 21 -1
+ + 1 + 2
c1a, &
610
+ 25
We only need to require
| x
t 2 and & S min
1}
Ca.
| >>
Some tedious calculation shows that 2 2 can be guaranteed by
- x < A x S
/3ai or 2
14
||
is left to consider the constraint in Notice that it holds for any & 0. When & 0, we can see
|| و s: + Eq.1 (AIl
1 1 18
cor(bu,, A)
+ 9.1 1x:
V1+ 22g
1 + t
2g
f
Hence, it suffices to guarantee that
- > )5 2
1
|
(1
Ca.
||>| Σ
2
|
If |/ 1 2 2, then LHS is negative and it holds. If S ||
2, then the inequality
becomes & < 1 Ca. If Á
x
A|E 1, then the inequality becomes & <
Ca. Because
A
Å l
we have required & S min
1 Ca for some large enough C, the constraint
11
always
A bs
1 Ca for some large enough C holds. To summarize, EL < ΣERM given that & S min A
and A > <
V/Sui
م|ل 1} Ca
For method LISA-II, we can similarly show that Faz S FERM given that & S min for some large enough C' and | S
A
V3ara
B.5 FINITE SAMPLE ANALYSIS
The empirical loss can be written as
PP (11 TT in
} x y/ (6))
(12)
1 G) 10 in
(((
1 (G)
M/
0) +
((r,
P
G)
bo <
//
(G)
1),
2
2
22
23

---

Under review as a conference paper at ICLR 2022
where
1 (G) = 0) 1 2 (1 b0))
P((2) + bo >
2
2
VirEb
1 (G) 1) $( 2 (((1.00) -
P( ( + 8n < 2g1b/
First notice that
8o = -- - b.b.
We have
(n (n..GG))T+++ bo (110000)5) 5) +
= (410000) F( T + -17- xT6)
[F [Fey [F( {8)}
(0.00078+ bo = (110000 - =) 인
= A (110,0) E[x,)) Tb b++
2
Therefore, according
12
1 (u10,0))28-
80
Φ 1/2 -nul.01778- Bo,
2
2
VirEb
(B(G)) TT + R
A (A(G))TT + R
+
2
VOTEB
2
VBTEb
(o(C))(T IT + R
1
(o(C)) T Ri
VOTE8
2
VOTEb
(BIC)) TO
R
A + (B(G) (191())T- T6 R
2
ViTTi
2
ViTei
(T R
AT (010) 6 R1
2
VITEB
2
ViTEB
Then the mis-classification error can be written as
1
(O(G)TTB + R
Φ (B(G))T8- - AT + R
Φ
(13)
2 2
Virzi
VirEb
i(6)
Larger the 2(6), smaller the mis-classification error. We first find that
i(6) L(b) sCI (
(B(1))
R1
(B
(0(01TT8 - A 8 Rı
(B( ATb
+C|
ViTEi
VorΣb
ViTEi
Vb-Eb
T1
In the event that
| /2 1/bl12
o(1)
max || S C, E is positive definite.
24

---

Under review as a conference paper at ICLR 2022
for the denominator, have Ib - 82508 - -1 -) b)ll -
+ 2
= b)ll2
16 16"22~ 6"
Viri- -VEE6 v
v 4b+ + vb
- 2( o(1)))] 6)]12-
For the numerator, we have
1/
1 A
Rı
- 2
A bl v Ral
+ 2
- ||| ///////2( -b)ll2-
We arrive at
R1 + 1 - 1/2
12 1/ b)112
AT
|| -b)ll
Ti S (1+
+ +1++
|E!/2b|12
V
VBTEb
T: {1 + o(1) R2 + 3(11-112 + ||21) /2 b)ll2
|
+ +1 o(1)) 1/
A | bll2
4b
VoTΣbb
Moreover R3 S 16 b1|2 + Op( To summarize.
[T(] L(b) $ (1 + +()) b +
Vn
In the following, we will upper bound ||6 - bll2 for each method. For the ERM method,
b = {(X - 8)" x)} - Xx) 5).
It is easy to show that
|| b112 = -Op( Opp P_ var(vilza) =
N2
For the vanilla mixup method, we first that
1 - - (Nrz,, + (1-
+ 1pp0n 0p(n-1/3)
1=1
V/ = T(1 (1) + (p
1=1
Next,
(2 Vi, + (1- - X2) Via + Ax(1 Ad)a,, Via + A,(1 - As)xr
1=1
211
n
n
E E(.0
Σ
K E E(
X,
E
2 X. yl E[
E2
For Eg,
2
Eg
E[a.ū.] = 21E 2 1E [x..).
25

---

Under review as a conference paper at ICLR 2022
Hence,
| Op (r("). 7
For E1, conditioning on (X, 3), 22 M/.
E
Σ ElA
T.yl are independent sub-Gaut Gaussian vectors.
The sub-Gaussian norm of 1 N -i=1 ,6
E Zi (conditioning on (X, y)) can be
upper bounded by emax [i,j] // Vn. Hence
Cynt² max m Lijj
P( 2 y) S 2expf
}.
2
As Wig are Gaussian distributed, we know that
P
max max 2 plogn) S expl-c3 log
Hence, with probability at least
exp( ci log
Cplogn
E1 v
n
To summarize,
P plogn
3z) Σ
cou( V.)
Opp
i=1
1=1
1=1
Similarly, we can show that
1
1 Σ 2²)²²
p poo n
Σ 2
az)(
cov(T:)
=Op(
1=1
1m1
2m1
Hence,
p Op(plogn
-b6-b13 Op((
n
For the LISA-I. we first see that
1
(A)
(1,G)
+(1 A1)ar -11->i)a!! (1
1 Σ
(1
(U,G) (1 (.))
n
===1
=:=0
(1
+ (11 A 1 (
+ (1) too
We have
1
(x(a) y-
7 n
1
Σ
x
(2)
co (2) 11)
1 (XrAA)TT 1 n
17 n
Σ x (2) (2)
,v, v)
n
n
coo(x (2)
.2/ X, ,,,(X,,) v)
cou (2, (A)
Fg
For Eg.
Eg (f(1.)) + +(LBB) - 216 + ((1.B))
11
+ (2(0.0) + =Z00B)))0o) coo(2 (A) v.)
2
2
1 (1(1))) (1188 - 00.0) O). u (1)00)) A.
2
It is easy to show that
E2 = Op
p
min e n(nee)
26

---

Under review as a conference paper at ICLR 2022
(A) For Ei, conditioning on X and y, x; yi Ela; y) XX v] are independent sub-Gaussian vectors with mean zero. The sub-Gaussian norm of 1 2im1 Zi, yi (conditioning on X and y) can be upper bounded by emax,gn |xtsl/VN.
(A)
(A)
P( 2 (1 = IP
1. Ui Fl M. xu F|X,3)
1=1
Cynt²
2exp
Hence,
Σ maxisn Lig
2
plog n
E = Op((
= Op(
To summarize,
(2) vil12 -
p
p pog n
= Op(
+
y.e n (v.e)
We can use similar analysis to bound
W (X x(A) - Flar (A)
(z; (x: (
The sub-exponential norm of Ei maxigN / VN. We can show that
(A) (2) Fig Jish (conditioning on X) can be upper bounded by
== (1 = Op( p
(x())) (x,
plog n
ming.e n(v.e) +
n
For the LISA-II, we first see that
1
Σ
(Aaa (1,G) +(1 - -1)aa (0,6)) + Σ
(Ara
(1,B) + (1 Ar)a ()
n
1=1
D.=G
D.=B
= (1(.0)) + f(0.0)) - [ + (00.
1
-- 2
1/
Next,
Σ
12 A, 1r)
(.G)
1 Σ
(220(11B1)
+ xx(11 2,)0;001))
n
n
iz
n
D:=G
D.=B
n
n
Łÿ - cou(ū, 9)
Tcy xy cou( R. 81 y) + cou J, - cor(ā i1
71 i=1
For Eg.
Eg = 4 (E)A)
+ _ 2 + * (7) (EE _ =(00 + 2
1
1 x (0
} (1 00000) 1 (a(1.)) (0. n var (2,)
= = (>,) (2 - x(0.0)) + $ BB) var (A+) - =: (0..)) - var car(A,) (A
Notice that Eg is sub- Gaussian vector with sub- Sab-Gaussian norm upper bounded by
#
#/
TB
G1,
+
(1.B)
n2 (0.B)
max n v.d
T yld
27

---

Under review as a conference paper at ICLR 2022
Using tub--aussian concentration, we can show that
P max n
E2 =2=0r(
v.d Tyld
Notice that maXy,d Hyld
2 1. For E1, conditioning on X and y Z. - - E[ [X, y] are independent
Lub-Gaussian vectors with mean zero. The sub-Gaussian norm of 1 Σ 0=1 3t,, Is conditioning on X
and y can be upper bounded by emax,,j | Similar analysis on E leads to
plog n
p max Wd ydd yyd
S:ū: 5:ū= Tÿ coolā, 5) Op(y
+
For the sample covariance matrix, we can also show that
5:)
n _ T 2z)²
coo(ās)
=Op(p
p plogn
P max n
Td Tyld
n
n
v.d
12
B.6 DOMAIN SHIFTS: PROOF OF THEOREM
It still holds that A 2(,x group mis-classification error for this new environment is
(
20x [[ It easy to show that the worst
1 2 (A
4 2 1(A
- - )}
E (wst,>) max
+
(14)
DAIbA
bgΣba
where A {ERM, mix, L2). Notice that
Ä 2p -4e + (I.B)) 7 + 4 (0 (0.C)
We assume A 1|2 A 2. Lot cor cor(a, A and 7 car A 7*). We have
cor(bERM, A² >|1 || A+ و co || A|
A >> + A| |
>
looS" || -|
7 Ca.
A s = ||o
| 1|
Hence,
(wst) ERM
max (()
Cl) (E- (a))Aa))) 8((-2 -Ca)) -
(15)
for some constant C' depending on the true parameters.
Hence,
(Ā-)TOL) PL
>1 l + +1i5""IAAA
cor cor(bun Corbb,,,")
|| A | V oif Eby,
|l + AlLIIAlls
To have [wwtt)) LI
<
(wats) ERM
it suffices to require that (-3 - -ca)ll 5(3 Co)ll (5+
Ca)
xx and
cor (bLI. A")| corcbrrrrroooo A)IAlI S 12 - -C)) - ( (E-C)]AAll
cor (bLI. | 2 Ca)l Al|S (E+ AC)
A sufficient condition is
f S ( + cor -³)
A
Ca, cor (br,, A) 2 { Ca, cor(bui, 5²) S 2Ca.
A g
28

---

Under review as a conference paper at ICLR 2022
We can find that further sufficient condition is
1 7 A
Σ
Ca, CLI 0055 = 2014 + CLI A E - -1))
E1a
(16)
A 5
9.1 |
+ + E 2 ه E. 5
61
(17)
| + 11 1
||ل| || ||||
S 2 + (bb ))) 2
Ca.
(18)
We first find sufficient conditions for the statements in 9 and (TO aull ||
Parameterizing \ =
we further simplify the condition in (16 and
17 as
1 7
+ 21g 1) ||| Ca, t > 0, 5 7
5 2 |
6
t
1 SESt, s3
1 1 11+2+ 215
610
2
t +
We only need to require
M 2 and { S min 1 7
+ 3
13 -Ca, 75.
2 | 3
Some tedious calculation shows that t 2 2 can be guaranteed by
A x < A |/13 x: S
2 ||
or
V
/ali
left to consider the constraint in Notice that it holds for any & S O. When 5 0, we can see
cor(bu,, A*) 1| x + +
7
| CLI Als
V1 1 12 21
7 + tt"
+ t
Hence, it suffices to guarantee that
2|| 13
s* カ
g+ Ca.
|
To summarize, it suffices to require
7 ||
| I
05 S S 33 min
1} Ca.
2 A
For LISA-II, can similarly show that Efwstx
FRRM
given that
|| |
x <
0 05 E S 3 min
11
Co.
V
2 A
29


Reviews:
[{"review_id": "_ChCHY2Eycq", "reviewer": "Reviewer_4YgJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n+ The proposed approach is simple and efficient, and can be directly incorporated in or combined with other invariance-inducing approaches;\n\n+ Prediction performance is shown to improve over a number of recent baselines under challenging benchmarks.\n\nWeaknesses/suggestions: \n\n- The proposal requires assumptions that are not discussed in the manuscript. In [1], it was shown that domain-invariant approaches can only improve out-of-distribution generalization if data-conditional label distributions P(y|x) are fixed across domains; i.e., observing x suffices in order to determine y, regardless of the domain according to which x was observed.\n\n- My main concern lies in the reported evaluation, which is focused on showing improvements in terms of downstream performance, and presented results consist of comparing the proposed approach with alternative methods. While improving downstream performance is of course our ultimate goal, and results are strong in this sense, doing so does not explain the sources of improvements. In particular, authors claim that the mixup strategies they introduce yield some type of domain invariance, which is not verified empirically. To verify that learned representations are invariant, that could be achieved via domain-prediction experiments; i.e., train domain classifiers on top of representations learned by different methods. The higher the accuracy of such a classifier, the less invariant are representations. For the case of prediction-level invariance, authors could perhaps evaluate the range of estimated risks across domains. Improvements on either one of these notions of invariance would then explain observed improvements in terms of prediction accuracy.\n\n- \"We argue that the failure of other methods in some datasets may be caused by their regularizers limiting model capacity to some extent\". That's another case where the evaluation lacks in supporting authors' claims. This hypothesis can be verified via in-domain prediction performance, i.e., overly regularized underfitting models should result in accuracy degradation in the training domains. Alternatively, one could rule out the underfitting effect by using higher capacity model classes.\n\n- Conclusions from the risk bounds provided in theorems 1 and 2 are a bit unrealistic given the strong assumptions that imply the results. In particular, the model assumed for the data generating process is overly simplified and, although it enables theoretical analysis, it's unclear to which extend the conclusions hold in practice.\n\n- Finally, regarding novelty, it seems different recent approaches introduce methods that use some sort of mixup across domains in similar settings. For the domain adaptation/generalization cases, there are, for instance, [2,3,4]. For the multi-domain case, cross-domain mixup was studied in [5]. Authors do compare results against [4], but it's unclear how the proposed approach differs from those other recent applications of mixup under similar settings, and the related work section should include such a discussion.\n\nOther comments:\n\n- On page 2, the setting described was not originally introduced by Koh et al. (2021). To my knowledge, it was first discussed in [6] and later on in [7].\n\n- The definition of mixup for labels in the rightmost term in eq. 2 seems to require labels y are one-hot encoded, which is not mentioned in the text.\n\n- While in the title authors claim to be improving out-of-distribution robustness, a large part of the work focus on the multi-domain learning (or fairness) setting, where one's goal is to find predictors with uniform risks across a set of domains that doesn't change from training to testing. Technically, that wouldn't be out-of-distribution. Perhaps there should be a sentence or two in the introduction indicating what authors refer to as out-of-distribution.\n\nReferences:\n\n[1] Zhao, Han, et al. \"On learning invariant representations for domain adaptation.\" International Conference on Machine Learning. PMLR, 2019.\n\n[2] Shu, Yang, et al. \"Open Domain Generalization with Domain-Augmented Meta-Learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[3] Wang, Yufei, Haoliang Li, and Alex C. Kot. \"Heterogeneous domain generalization via domain mixup.\" ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020.\n\n[4] Xu, Minghao, et al. \"Adversarial domain adaptation with domain mixup.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.\n\n[5] Chuang, Ching-Yao, and Youssef Mroueh. \"Fair mixup: Fairness via interpolation.\" arXiv preprint arXiv:2103.06503 (2021).\n\n[6] Muandet, Krikamol, David Balduzzi, and Bernhard Schölkopf. \"Domain generalization via invariant feature representation.\" International Conference on Machine Learning. PMLR, 2013.\n\n[7] Albuquerque, Isabela, et al. \"Generalizing to unseen domains via distribution matching.\" arXiv preprint arXiv:1911.00804 (2019).\n", "summary_of_the_paper": "Authors introduced approaches aimed at learning invariant predictors across data sources. Rather than using distribution/risk matching schemes as often done by previous work, they propose to train models against mixtures of data points as a means to avoid that models rely on spurious correlations between domain and class labels, since such correlations observed during training might not hold at testing time. The proposed setting uses the idea of mixup to combine data instances in two different schemes: I-combine data points from the same class but different domains, and II-combine data points from the same domain but from different classes.", "summary_of_the_review": "The paper is well-written, the approach is efficient and observed to work well on a number of benchmarks. However, experiments supporting key claims are lacking, and it's unclear whether the observed improvements in terms of invariance (either at feature- or prediction-level) hold true since no supporting experiments are reported. Contextualization of the proposal relative to past literature also needs improvements since cross-domain data mixup was studied in the past under both settings considered by the authors. The provided discussion on related work does not clarify what and how the authors' proposal improves upon previous work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZU9Dq3GC-Xp", "reviewer": "Reviewer_VX2N", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "the idea is quite simple and intuitively reasonable, and empirical results seem extensive and significant, and theoretically justified to some extent\n\nsome of the results analysis is a bit confusing to me\n(1) in 4.1 \"evaluating robustness to domain shifts\", the best strategy was to always mixup same label with different domains, and the potential reason given is that the datasets actually have weak or even no spurious correlation between domain and label. Two questions follow:\n(a) From early text, seems both selection strategies are motivated by the spurious correlation, but here why do we still observe advantage over ERM or vanilla mixup? It would be great if you could clarify the different motivations (if any) of the two selection strategies\n(b) the reasoning about \"weak or no spurious correlation\" seems to be contradicting with claim in 4.3, where it's stated that \"compared with vanilla mixup, ...LISA...improve the OOD robustness by canceling out the spurious correlations..\". or did I misunderstand something? Is it easy to quantify such spurious correlation? if so, why not present the actual correlation metrics for these datasets? \n\n(2) for the ablation study, I think a more convincing way would be: first test LISA and mixup on a dataset that is known to have NO spurious correlations, then we expect neutral results; then test them on a data that is known to have spurious correlations, and we could give quantitative metrics of such correlations if possible, and show LISA is better than mixup; further more, the stronger the correlation, the more advantage LISA has. Is that what you're trying to demonstrate here? \n\n(3) In table 8, vanilla mixup shows much worse performance than ERM in terms of learning invariant representations under the defined metric, which doesn't seem quite reasonable to me. Shouldn't we expect the contrary? \n \n\n\nAnother minor question: In Theorem 1, is p the dimension of x? If so it's better to state that explicitly in the theorem instead of relying on readers to go back to text and only to find in the superscript notation. ", "summary_of_the_paper": "This paper propose a mixup-style data augmentation method under the data distribution shift context. In particular, data distributions are formulated as mixture of distributions (i.e., domains), and two distribution shift scenarios are considered: (1) domain shift, where the test domain and train domain are disjoint. (2) subpopulation shift, where test distribution has different mixture proportion than train distribution. It's  assumed that domain identification spuriously correlates with labels. To tackle this problem, this paper proposes two mixup strategies: (I) mixup two examples with same label but different domains; (II) mixup two examples with same domain but different labels. It's claimed that such mixup could cancel out the spurious correlations. Extensive experiments on a variety of datasets show its superiority compared to empirical risk minimization (ERM) and alternative data augmentation methods. The paper further provide theoretical analysis that under certain conditions, the proposed method has asymptotically smaller worst case classification errors than ERM and vanilla mixup. ", "summary_of_the_review": "the idea is quite simple and intuitively reasonable, and empirical results seem extensive and significant, and theoretically justified to some extent, but the results analysis and some experimental design could be more insightful or improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AIjTwOXgDz_", "reviewer": "Reviewer_iP5p", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n(1) The authors address a critical point that prevent models from generalization, namely spurious correlation.\n(2) The proposed method is simple and easy to implement, and the empirical results are within expectation.\n\nWeakness:\n(1) Maybe the biggest concern is the contribution over previous work. The proposed method can be seen as a heuristic extention of mixup. Though simple and easy to follow, the contribution is marginal. \n(2) There ara growing trends on investigating OOD generalization under missing domain label, it is better to at least include such work (e.g. [1]) for discussion.\n(3) There is a hyper-parameter p_sel that controls the probability of performing different strategy, is there a rule of thumb or we need to tune it for every task?\n[1] Qiao, F., Zhao, L., & Peng, X. (2020). Learning to Learn Single Domain Generalization. In CVPR.", "summary_of_the_paper": "The paper considers the model robustness under distribution shift brought by domains and subpopulations. Specifically, based on the interpolation scheme in mixup, the authors propose two selection strategies to perform data augmentation, aim at eliminating the spurious correlations and learning an invariant representation.", "summary_of_the_review": "This paper address the spurious correlation by augmenting the data via interpolation. Although intuitions are provided and empirical effectiveness is illustrated accordingly, the contributions over previous work (e.g. mixup) are marginal.\n\n=========After Response==========\n\nThe response from authors has addressed my major concerns, so I raise my score accordingly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript focuses on model robustness under distribution shift, specifically domain shifts and subpopulation shifts. Domain shift is where the test domain and train domain are disjoint. Subpopulation shift is where test distribution has different mixture proportion than train distribution. The assumption is that domain identification spuriously correlates with labels. The proposed framework learns an invariant representation by using mixup strategies and interpolates samples either with the same labels but different domains or with the same domain but different labels to. Experiments are performed on a variety of domain shift and subpopulation shift benchmarks, and results showed that the proposed framework is better than empirical risk minimization (ERM) and alternative data augmentation methods. Theoretical analysis is also provided and it is shown that, under certain conditions, the proposed framework has asymptotically smaller worst case classification errors than ERM and vanilla mixup.

Reviewers agreed on several positive aspects of the manuscript, including:
1. The manuscripts addresses a critical point that prevent models from generalization, namely spurious correlation; 
2. The proposed method is simple and easy to implement, and the empirical results are within expectation.

Reviewers also highlighted several major concerns, including:
1. Different recent approaches introduce methods that use some sort of mixup across domains in similar settings;
2. Ablation study on datasets without spurious correlations are missing;
3. Evaluation of domain invariance representations and prediction-level invariance needs clarifications;

Authors clarified different motivations of the two selection strategies in relation to spurious correlation between domains and labels, and provided an ablation study on datasets with no spurious correlation. Post-rebuttal, reviewers stayed with borderline ratings, and they have suggested further improvements: improving results analysis and the conclusion that “existing domain information may not fully reflect the spurious correlation”, understanding the implication and the reasons that invariance is achieved at the prediction level instead of at the representation level despite the original goal is to learn an invariant representation, and improving presentation of the manuscript including settings and assumptions.
Decision: Reject