Diversity Boosted Learning for Domain Generalization with a Large Number of Domains
Authors: XI LENG
Xiaoying Tang
Yatao Bian
Venue: Submitted to ICLR 2023 (2023)
Keywords: Domain Generalization
Spurious Correlation

Abstract: Machine learning algorithms minimizing the average training loss typically suffer from poor generalization performance. It inspires various works for domain generalization (DG), among which a series of methods work by $O(n^2)$ pairwise domain operations with n domains, where each one is often costly. Moreover, while a common objective in the DG literature is to learn invariant representations against spurious correlations induced by domains, we point out the insufficiency of it and highlight the importance of alleviating spurious correlations caused by objects. Based on the observation that diversity helps mitigate spurious correlations, we propose a Diversity boosted twO-level saMplIng framework (DOMI) to efficiently sample the most informative ones among a large number of domains and data points. We show that DOMI helps train robust models against spurious correlations from both domain-side and object-side, substantially enhancing the performance of five backbone DG algorithms on Rotated MNIST and Rotated Fashion MNIST.

Under review as a conference paper at ICLR 2023
DIVERSITY BOOSTED LEARNING FOR DOMAIN GENER- ALIZATION WITH A LARGE NUMBER OF DOMAINS
Anonymous authors Paper under double-blind review
ABSTRACT
Machine learning algorithms minimizing the average training loss typically suffer from poor generalization performance. inspires various works for domain gener- alization (DG), among which a series methods work by O(n") pairwise domain operations with domains, where each one is often costly. Moreover, while a common objective in the DG literature is to learn invariant representations against spurious correlations induced by domains, we point out the insufficiency of it and highlight the importance of alleviating spurious correlations caused by objects. Based the observation that diversity helps mitigate spurious correlations, we propose a Diversity boosted twO-level saMplIng framework (DOMT) to efficiently sample the most informative ones among a large number of domains and data points. We show that DOMI helps train robust models against spurious correlations from both domain-side and object-side, substantially enhancing the performance of five backbone DG algorithms on Rotated MNIST and Rotated Fashion MNIST.
INTRODUCTION
The effectiveness machine learning algorithms that minimize the average training loss relies on the IID hypothesis. However, distributional shifts between test and training data usually inevitable. Under such circumstances, models trained by minimizing the average training loss are prone to sink into spurious correlations. These misleading heuristics only work well some data distributions but can not be generalized to others that may appear in the test set. In domain generalization (DG) tasks. the data distributions are denoted as different domains. The goal is learn a model that can generalize well to unseen ones after training on several domains. For example, an image classifier should be able to discriminate the objects whatever the image's background is. While lots of methods have been derived to efficiently achieve this goal and show good performance, there are two main drawbacks.
Scalability. With unprecedented amount of applicable data nowadays, many datasets contain tremendous amount of domains, or massive data in each domain, or both. For instance, WILDS (Koh et al., 2021) is curated collection of benchmark datasets representing distribution shifts faced the wild. Among these datasets, some contain thousands of domains and OGB-MoIPCBA (Hu al., 2020b) contains even more than one hundred thousand. Besides WILDS, DrugOOD (Ji et al., 2022) is an out-of-distribun Datasets of DrugOOD contain hundreds to tens of thousands of domains. In addition multitudinous domains, domain augmentation, leveraged to improve the robustness of models in DG tasks, can also lead significant increase in the number of domains. For example, HRM (Liu et al., 2021a) generates heterogeneous domains to help exclude variant features, favoring invariant learning. Under such circumstances. training on the whole dataset in each epoch is computationally prohibitive, especially for methods such as MatchDG (Mahajan et al., 2021) and FISH (Shi et al., 2021b), training by pairwise operations, of which the computational complexity is Onn²) with training domains. Objective. Numerous works the DG field focus entirely on searching domain-independern correlations to exclude or alleviate domain-side impacts (Long et al., 2015; Hoffman et al., 2018; Zhao et al., 2018, 2019; Mahajan et al., 2). We state that this objective insufficient, and a counterexample is given follows. We highlight the importance of mitigating spurious correlations caused by the objects for training a robust model.
dataset curator and benchmark for Al-aided drug discovery.
data with
Suppose our learning task is training model to distinguish between cats and lions. The compo- sition of the training set is shown in Figure 1, and the domain here refers to the images' back- grounds. In this example, the correlation between features corresponding the body color of

---

Under review as a conference paper at ICLR 2023
the objects and class labels is undoubtedly independent of domains. accuracy in the training set by simply taking the tan objects as lions and the white ones as cats. Unfortunately, if this cor- relation is mistaken for the causal correlation, the model is prone to poor per- formance once cat breed distribution shifts in the set. To tackle these two issues, we propose a diversity boosted two-level sam- pling framework named DOMI with the following major contributions: 1) To our best knowledge, this is the first paper to take im- pacts from the object side into account for achieving the goal of DG. We propose DOMI, diversity-boostr two-level sampling framework to select the most informative domains and data points for mitigating both domain-side and object-side impacts. 3) We demonstrate that DOMI substantially enhances the test accuracy of the backbone DG algorithms on different benchmarks.
Moreover, it helps get high
Golden British
Cats
Objects
Silver British
Lions
Dataset
Grassland
Domains
Indoor
Figure I: The training of the counterexample. Cats are mainly silver British shorthair (body color of which is silvery white), rarely golden British shorthair (tan), and lions are all tan. As for the background, most lions are on the grassland while most cats are indoors.
2 RELATED WORK
Domain Generalization. DG aims to learn model that can generalize well to all domains including unseen ones after training on more than one domains (Blanchard et al., 2011; Wang et al., 2022; Zhou al., 2021; Shen al., 2021). Among recent works domain generalization, Ben-Tal al. (2013); Duchi et al. (2016) utilize distributionally robust optimization (DRO) to minimize the worst-case loss potential test distributions instead of the average loss of the training data. Sagawa et al. (2019) propose group DRO to train models against spurious correlations by minimizing the worst-case loss over groups to avoid suffering high losses some data groups Zhai al. (2021) further use distributional and Outlier Robust Optimization (DORO) to address the problem that DRO is sensitive to outliers and thus suffers from poor performance and severe instability when faced with real, large-scale tasks. On the other hand, as Peters et al. (2016) and Coras-Carullaa et al. (2018) state that the predictor should be simultaneously optimal across all domains, (Arjovsky et al., 2019; Javed et al., 2020; Shi et al., 2021a; Ahuja et al., 2020a) leverage Invariant Risk Minimization (IRM) to learn features inducing invariant optimal predictors over training domains. However, Guo et (2021); Rosenfeld et al. (2020); Kamath et al. (2021); Ahuja et al. (2020b) point out that works with IRM lack formal guarantees, and IRM does not provably work with non-linear data. Koh et al. (2021) and Gulrajani & Lopez-Paz (2020) present analysis demonstrate that IRM fails generalize well even when faced with some simple data models and fundamentally does not improve over standard ERM. Risk Extrapolation (V-REx) (Krueger et al., 2021) instead hold the view that training risks from different domains should be similar and achieves the goal of DG by matching the risks. Some works explore data augmentations to mix samples from different domains (Wang al., 2020; Wu al., 2020) or generate more training domains (Liu et al., 2021a,b) to favor generalization. Another branch of studies assume that data from different domains share some "stable" features whose relationships with the outputs are causal correlations and domain- immain-nndeeendent given certain conditions (Long et al., 2015; Hoffman et al., 2018; Zhao al., 2018, 2019). Among this branch of work, Li et al. (2018c); Ghifary et al. (2016); Hu et al. (2020a) hold the view that causal correlations are independent of domain conditioned on class label, and Muandet et (2013) propose DICA to learn representations marginally independent of domain.
MatchDG. Mahajan et al. (2021) state that learning representations independent the domain after conditioning the class label is insufficient for training a robust model. They propose MatchDG to learn correlations independent of domain conditioned on objects, where objects can be seen as clusters within classes based similarity. To ensure the learned features invariant across domains, a term of the distance between each pair of domains is added to the objective to be minimized.
FISH, MMD, CORAL. Another line of works promote agreements between gradients with respect to network weights (Koyama & Yamaguchi, 2020; Parascandolo et al., 2020; Rame et al., 2022; Mansilla al., 2021; Shahtalebi et al., 2021). Among these works, FISH (Shi et al., 2021b) augments
2

---

Under review as a conference paper at ICLR 2023
the ERM loss with auxiliary term of gradient inner product between domains. By minimizing the loss and matching the gradients simultaneously. FISH encourages the optimization paths to be the same for all domains, favoring invariant predictions. MMD (Li et al., 2018b) and CORAL (Sun & Saenko, 2016) are another two matching methods besides MatchDG and FISH. MMD matches the distributions among different domains using Maximum Mean Discrepancy measure. In this way, the learned representation is supposed to be invariant for the training domains. MMD further aligns the matched distribution to an arbitrary prior distribution via adversarial feature learning, aiming to prevent the representation from overfitting to the training domains. Then the learned representation is expected to generalize well on the test domains. CORAL instead matches the second-orde statisties of the distributions across domains. Specifically, CORAL concurrently minimizes the ERM loss and the difference in learned feature covariances across domains. As a simple yet effective method, CORAL shows state-of-the-art performance on various tasks for OOD generalization (Gulrajani & Lopez-Paz, 2020).
DANN. Besides gradients, some approaches enforce agreements between features and align the features with adversarial methods (Li et al., 2018a; Gong et al., 2016). As one of these approaches, DANN (Ganin et al., 2016) incorporates the structure named domain discriminator to implement adversarial training based on the theory that an ideal classifier for cross-domain shifts should be able to distinguish different classes while cannot learn to identify the domain. DOMI takes use of an inverse version of DANN denoted as inVDANN to learn domain-side features and help select the most informative domains.
DPP, Determinantal Point Process (DPP) (Kulesza et al., 2012) is a point process that mimics repulsive interactions. A draw from a DPP yields diversified subsets based on similarity matrix (DPP kernel) of samples to be selected. While it shows powerful performance in selecting heterogeneous data, DPP sampling relies on an eigendeccompoition of the DPP kernel, whose cubic complexity is huge impediment. To address this problem, Li et al. (2016) suggest first construct an approximate probability distribution to the true DPP and then efficiently samples from this approximate distribution. As one choice of diversity sampling, DPP sampling is incorporated into DOMI to help select the most informative domains and data points. It can be replaced with other diversity sampling schemes.
Discussions. Although MatchDG, FISH, MMD and CORAL perform well in domain generalization tasks, the matching procedure between domains means their computational complexity is ((n²) ) with n training domains. When n is large, it will inevitably slow down the training process. Therefore, we must select the most informative domains from all the training domains. Inspired by Liu et al. (2021a) that heterogeneous training domains help to learn invariant features since more variant features can be excluded, we conduct analysis of diversity and spurious correlations to further state it.
DIVERSITY HELPS MITIGATE SPURIOUS CORRELATIONS
Under the cireumstance of imbalanced data where specific clusters contain the majority of data points while the others are only tiny fraction of the training set, there likely to exist an anti-causal path, i.e., a spurious correlation only catching some properties of the large clusters of the data, and algorithms minimizing the average loss like ERM may simply take this correlation as the causal correlation. For example, when faced with chest X-ray dataset -oakden-Rayner et al., 2020) where many images of patients with pneumothorax contain thin drain used for treating the disease, classifier trained by minimizing the average loss can erroneously identify the drains as a predictive feature of the disease. When we sample diverse data, we in fact re- re-balance them to help mitigate spurious correlations. We verify this observation with a toy example. For the task and dataset mentioned above (Figure 1), we further suppose our featurizer extracts 4 features with a binary value as shown in Table 1. Then
Table 1: Details of the features and the label. X, to X3 correspond to features of the object and X, corresponds to features the domain.
X1 Mane
X2 Proportion of face
Xg
Body color
N4 Background
0 no mane
short face long face
white
indoors
cat lion
1 have mane
tan
grassland
X, X₂ 1 = y 1 is the causal correlation since the proportion of lions' faces is longer than that of cats, and X2 may be wrongly computed to 0 for male lions because of the existence of mane. X3 1 y = is the object-induced spurious correlation (Abbrev. OSC) and X, 1 y is
3

---

Under review as a conference paper at ICLR 2023
the domain- inddueed spurious correlation (Abbrev. DSC). Details of our simulated dataset is shown in Appendix A.
Suppose we get 6 of these 12 data samples for training where 3 of 6 come from cats and the other 3 are from lions. There are 4 sampling methods denoted as S to S2 to be picked: random sampling, diverse sampling with respect to the object features (X X2 and Xa), diverse sampling with respect to (X²), ) and diverse sampling with respect to all 4 features. For S2 to Sq, we use Manhattan Distance on the corresponding features to measure the similarity between data. After conducting the similarity matrix, we finally use DPP sampling to select data points, Table 2 shows the average training accuracy of OSC and DSC. When the spurious correlations get lower training accuracy, they are less likely to be mistaken for causal correlation, favoring exploration of the causal correlations.
Table 2: We use each sampling method to select 30 batches of data for training, on which the average accuracy of two kinds of spurious correlations is computed. When spurious correlations get lower accuracy during training, they are more likely to be excluded. Here we use DSC and OSC to denote domain-induced spurious correlation and object-induced spurious correlation, respectively.
Sampling Method
Accuracy of OSC
Accuracy of DSC
0.86
0.68
S
0.77
0.66
S3
0.85
0.50
S4
0.78
0.49
We take the data batches sampled by S₁ as base-batches. Random sampling preserves the imbalance of data since data point is more likely to be sampled into subset when it appears more often in the whole dataset. For base-batches sampled by Sj. both OSC and DSC get high accuracy and are thus prone to be wrongly treated as causal correlations. S₂ selects diverse data pertaining object features. Data batches sampled by S2 get lower accuracy for OSC than mase-batches, which means S2 reduces the probability of taking OSC as causal correlation. However, data batches sampled by S2 get almost the same result for DSC. S3 selects diverse data on domain-feature X4- For these batches of data, DSC gets lower accuracy than base-batches and is less likely to be taken as causal correlation, while OSC has a similar result, S4 selects data with heterogeneity with regard to all 4 features. Compared to pase-batches, the data batches selected by S4 get lower accuracy on both spurious correlations.
4 METHODS
Figure 2 shows the sampling procedure of DOMT, diversity boosted two-level sampling framework.
Random sampling
lavDANN
Initial model
ERM
AA
featurizen
Batches inside sumpled domains
fenturizer,
Ground set of domains
Level-one-samlling
Tevel-tvo-sampling
Figure 2: Illustration of the sampling procedure of DOMI. The solid arrow indicates the actual sampling flow, while the dotted arrow is only used to demonstrate the difference between random sampling and DOMI.
4.1 DIVERSITY BOOSTED SAMPLING FRAMEWORK
Preliminaries. Consider the universal set of domains D, where each domain d E D corresponds to distribution Pa over X' x V. with X' being the space of inputs and y that of outputs. Our goal is to find a predictor f X 5 while we can only access the domains in Der where Der C D. We measure the quality of a prediction with a loss function l: e: e:Dxy y - Rzo: and the quality of a predictor by its population loss on domain d E DD given by La(f) = P(
Definition (Correlation). We define a correlation as a predictor f = w o & where : Z is a data representation and w' : :Z + y is classifier: The causal correlation f satisfies o elicits a invariant predictor (Arjovsky et al., 2019) on D: w" simultaneously optimal for all domains, i.e. dd E D arggin o*))
4

---

Under review as a conference paper at ICLR 2023
Notably, Definition requires that and unrestricted in the space of all (measurable) functions. However, we learn á and w being restricted to only access domains in Dry, a small subset DD For this to be feasible, it is natural to add restriction that & d and w f for suitable classes & of functions mapping X' Z and W of functions mapping 2 D. Assumption 1. samminn.z-yyylloe o 6) = (wlL Φ) s 6), where > 0 is a constant.
Definition 2. Consider domain set a., which the set of invariant predictors, I(D.), is the set of all predictors satisfies following: f = w & with (w, Ω Φ; for all d E D,. argmina:2--yLa(d Φ).
Lemma 4.1. Based on Definition and Definition 2, we can trivially derive: for any nonempty set D, I(D).
Definition (Diversity). We use Integral Probability Metric (Mäller, 1997) to measure the diversity between domains. For domain d and d, the diversity is defined as:
Die(Pa Pa: Pa) = Div(( Pa P).9)= ssup E2, ()] - F.,g[((x,
gEğ
Where G is class of bounded functions. When we let ((, ((f(x),))))))))))))))))) ())),)))))))) and = F= 4, the diversity is:
Div Pa. Pa)
Div( Pas 2,, = sup wERLEED
[Fp,[[( o ((x), u)] Fp.[ (())) ((x),,)]]
= Di F) sup
-
lef
Consider we have domain set Dx = {d1. d2.... d&} and the corresponding I(D;) {/1, f2... fmn And now we get more domain dx+1 to form Dx+1 . According to Lemma 4.1, the causal correlation f E I(Dx+1), so a informative domain dx+1 which helps exclude spurious correlations leads to [[(Dx+1)]]
Proposition 1 (Diverse domains help exclude spurious correlations.). If dx+1 satisfies that: max Dio(dx+1.d) + La(f)) 6, then I(D;) I(D++1).
(ED,.,,E((D)))
Proof. Without loss of generality, first conduct analysis on S1 of I(Dx). For fú:
maz [Caen (fi) La(fi) + Ca(fi) s mar Dio(de+1. d) Ca(ft) dED
deD; maa Dio(der1;d) La(fe) s mar deDt
Div(der1, d) + Ca(f)) s 5 dED(.,,EI(D.)
When
Cduer (fi) La(si) <0:
Cann (ft) < Lx(ft) d
Case (ft)) Ca(fi) 2 2:
Ld (ft) = max Cedi (fi) - La(s)) + La(s)) = mmar (Ld, (f)) - La(si) + Ca(s) 8 dED,
dED,
Ede. (ft) ô, get fo I(Da-1) for any E (1,2..., m), thus I(D;) = I((D+1)
4.1.1 DIVERSITY SAMPLING METHOD
As Proposition states that diversity helps mitigate spurious correlations, DOMI is diversity boosted sampling framework and the sampling scheme to obtain a heterogeneous subset is critical part of DOMI. Determinantal Point Process (DPP) sampling is powerful diversity sampling method. Based on the similarity matrix between the samples, a draw from a DPP yields diversified subsets. Thus we incorporate DPP sampling into DOMI. As one option for the diversity sampling method in DOMI, DPP sampling can also substituted with other sampling methods.
4.2 REVEL--NN-SAAMPLLE
In the level-one-same -aapllin., we select diverse domains to help mitigate domain- induced spurious correlations. Since aim sample diverse domains, have learn about the domains. We propose an inverse version of DANN denoted as inVDANN to train model to capture the domain information.
5

---

Under review as a conference paper at ICLR 2023
4.2.1 INVDANN
Domain-Adversarial Neural Networks (DANN) proposed by (Ganin et al., 2016) is composed by a featurizer, a classifier and discriminator. The featurizer extracts features of data samples, the classifier learns to classify class labels of data, and the discriminator learns to discriminate domains. Since DANN aims to obtain a model that can not differentiate domains to ensure the featurizer captures domain-independent features, the discriminator is connected to the the featurizer via a gradient reversal layer that multiplies the gradient by a certain negative constant during backpropagation Gradient reversal ensures that the feature distributions over the two domains are made similar, thus resulting in domain- nomain-independenn features. Using the architecture of DANN, we let the classifier learn to classify domain labels of data while the discriminator learns to discriminate class labels. As an inverse version of DANN, invDANN trains model that can classify domains while not distinguishing class labels. Thus we can get featurizer extracting only domain-side features.
4.2.2 SAMPLING PROCEDURE
In the level-one-sampling of DOMI, we first use invDANN to train a featurizer. As mentioned in Section 4.2.1, the featurizer only extracts domain-side features. Then we use the featurizer to capture the information of domains and construct a similarity matrix between them. Based on the similarity matrix, DPP sampling selects the diverse domains.
4.3
Observation 1. Excluding lomain-induced spurious correlations is insufficient for learning a robust model.
Mahajan et al. (2021) have proposed that correlations independent of domain conditional on class
(*(z) IL D|Y) are not necessarily causal correlations P() changes across domains. Here ((z) is featurizer to extract features and 전 represents the causal features. We now further propose that the condition is insufficient even if & is consistent across domains. correlation incorporating features entirely from the objects can still be spurious correlation. Figure 3 shows structural causal model (SCM) that describes the data-generating process for the domain generalization task. The SCM divides data into two parts: domain-side and object-side. FF of domain-sidk is the reason for domain-induced spurious correlations. For object- side, feature is further divided into è and & where T. is the reason for object-induced spurious correlations, just like the body color of objects in the toy example. The three parts together make up the observed data. Thus even if we exclude all the domain-induced spurious correlations, i.e., entirely remove the effect from I we may still obtain object-inducee spurious correlations resulting from T.
Object
Domain
X
X
Y (Classes)
X (Pixels)
Figure 3: The Structural Causal Model for the data-generating process with a node leading to object-induced spurious corre- lations.
4.3.1 SAMPLING PROCEDURE
As observation I shows that excluding only induuced spurious correlations is insufficient, select diverse data batches among the selected domains to help mitigate object-induced spurious correlations in the level-two-sampling In the level-two-sampling, since we do not have available labels just like domain labels in the level-one-sampling it is infeasible to utilize invDANN again to train featurizer. So we instead use an ERM model since ERM prone to taking shortcuts and learning spurious correlations (Zhang et al., 2022). Zhang et al. (2022) also leverage an ERM model to infer the spurious attributes in the unsupervised DG field. Moreover, since domains attained by the level-one-sanne sampling contain diverse data with respect to the domain side, ERM can avert learning domain-induced spurious correlations. Combining these two, the ERM model is prone to relying on object-induced spurious correlations and thus can extract their information. Then similarity matrix between data batches is constructed with respect to this information. Based on which DPP sampling selects the data batches helping exclude object-induced spurious correlations.
4.4 DOMI
We present DOMI in Algorithm 1. Combining the two levels, DOMI finally gets subset of the dataset to tackle the issue of scalability under the setting of tremendous domains and training on which helps obtain robust models against impacts from both domain-side and object-side.

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Sampling Procedure of DOMI
Input: The whole training dataset:
9 for d, in D do
T [[(2.. )):- for d D]
for d; in D do
the proportion of domains (3) and batches (6) to be
L LaD+] ||feat. feat,, 11
sampled
Level-one-sampling
Obtain Q2 DPP(L B IDI) = [[(,] Di Di5 for
Train an inVDANN featurizer fo on a randomly sampled subset of ;
d E DI, (D D, |D] 3 (DI::
Level-two-sampling
for dl in D do
Divide into R = [[( )), for b B]: 15 Train an ERM featurizer fa on R:
feat,- + Ō; for from 1 to aa do feats + featz + fa(x();
for b in B do
Compute feat, in the same way computing feata in Level-one-sampling
featz- feats- na
Computing similarity matrix Ls;
Initialize similarity matrix L Oppp|p|
Return S DPP( T, 5 |B) |B):
5 EXPERIMENTS
We have investigated the performance of DOMI with different backbone DG algorithms on four benchmarks, which show that DOMI can help substantially achieve higher test accuracy. We also conduct experiments on iwildcam. Due to space constraints, the results and analysis are listed in Appendix B.3. The experimental settings and results are shown as follows.
5.1 CONFIGURATIONS
Backbones. We take MatchDG (Mahajan al., 2021), FISH (Shi et al., 2021b), CORAL (Sun & Saenko, 2016), MMD (Li et al., 2018b) and DANN (Ganin et al., 2016) as backbone algorithms. The former four algorithms work by pairwise domain operations, leading to O(n²) computational complexity with n domains and thus prohibitive to be scaled DG tasks with multitudinous domains. It is essential for them to sample the most informative domains. We further incorporate DANN as one of the backbone algorithms since DOMI can not only efficiently select domains by its first level of sampling but can help deal with cireumstances where each domain contains massive data by the second level of sampling.
Baselines. For each one of the backbone algorithms, we set the baseline as training on domains selected by the random sampling scheme and denote it as levelg. compared to the level-one-sampling of DOMI and the full version of DOMI represented as level, and levely, respectively. We sample domains, i.c., 9 = 5/61 for training on Rotated MNIST and Rotated Fashion MNIST. The proportion of minibatches selected level-two-sampling (5) is a hyperparameter valued from 0 to 1. When 5 equals 1, levela shrinks level,- Within each backbone algorithm, we keep factors including learning rate, batch size, choice of optimizer and model architecture the same for levely, level and levely highlight the effect of different sampling schemes. It's worth noting that we do no comparison between the backbone algorithms since we do not conduct meticulous hyperparameter tuning for them.
Model selection. During training, we use validation set to measure the model's 's performance. The test accuracy of the model is updated after an epoch it shows better validating performance. That is, we save the model with the highest validation accuracy after the training procedure, obtain its test accuracy and report results. For Rotated MNIST and Rotated Fashion MNIST, data from only source domains (rotation degree from 15 9 to 75 ") are used to form the validation set.
5.2
EXPERIMENTS ON MNISTs.
We first conduct experiments with five backbone DG algorithms on two simulated benchmarks (Rotated MNIST and Rotated Fashion MNIST).
5.2.1 ROTATED MNIST AND ROTATED FASHION MNIST
To satisfy the setting of large number of domains, extend the original simulated benchmarks on MNIST and Fashion MNIST by Piratla et al. (2020) from rotating images 15t through 75% in intervals of 15t intervals of 1 in the training set, i.e., 61 domains in total. And we get test accuracy on the test set which rotates images either 0% or 90º. Moreover, while the original datasets rotate the same images for different degrees, extend them to fit the real cases in DG tasks. We generate indices
7

---

Under review as a conference paper at ICLR 2023
Table Average test accuracy five algorithms. We repeat the experiment for 5 times on FISH and 20 times on the other algorithms with random seeds.
Dataset
Sampling scheme
DANN
MatchDG
FISH
MMD
CORAL
levelg
74.5
81.5
65.2
84.2
85.6
Rotated MNIST
level
76.5 120.0
13.6112.1
1.3
87.2 1.3.0
89.2 1.6.6
level2
78.6 1.4.1
84.2 122.7
66.6 1.4
87.7 3.5
89.6 4.0
levelo
40.3
38.2
33.2
39.0
38.7
Rotated Fashion MNIST
level1
42.8 1.2.5
39.7
1.5
34.5
1:3
41.8 1.2.8
40.8 1.2.1
level2
43.5 1/3.2
40.7
1.5
35.8
1.2.6
42,8 1.3.8
42.1 1.4
using different random seeds select images from MNIST and Fashion MNIST for each domain before rotating. Appendix C gives examples to show how spurious correlations can occur in MNISTs.
5.2.2 EMPIRICAL RESULTS AND ANALYSIS ON MNISTS
Table 3 shows the empirical results and we make the following observations:
Strong performance across datasets and algorithms. Considering results of 5 backhone DG algorithms on MNISTs, levelj gives constant and apparent improvement over levelg- While levelz may lead to slower growth in accuracy at the initial part of training shown in Figure 4 because of using a smaller number of minibatches, it keeps outperforming (evel, and levely later epochs.
35
Accusscy
AccUrac
Level
Level,
Level,
30
10
10
(a) Rotated Fashion MNIST
(b) Rotated MNIST
Figure 4: Average test accuracy of 5 experiments with random seeds during 50 epochs under different sampling schemes of FISH.
The gap between test accuracy and maximal accuracy. During training we observe that the test accuracy first rises the peak value and then begins decline along with the increase of validation accuracy. This reduction indicates a certain degree of overfitting to spurious correlations. Thus further record the peak value of the test accuracy in each experiment and denote it maximal accuracy. The distribution of test accuracy and maximal accuracy on MatchDG under different sampling schemes shown in Figure 5. While the test accuracy of level, scatters, that of levelz centers and level, shrinks the gap between the test accuracy and the maximal accuracy.
accurary
Accuratcy value
dmall aearwry text accuracy
Acainry
80
maaumma axzuracy
75
25
levelo
Level
levels
levelo
feveli
fevely
(a) Rotated Fashion MNIST
(b) Rotated MNIST
Figure Boxplot of test accuracy and maximal accuracy among repeated experiments with random seeds leveraging different sampling levels MNISTs. Among training epochs, the test accuracy rises to the peak value and then declines with the increase of validation accuracy. In this figure, maximal accuracy represents the peak value. Each tiny circle represents time of experiment, of which the vertical location corresponds to the accuracy value. The horizontal line inside each box indicates the mean value.
8

---

Under review as a conference paper at ICLR 2023
The choice of 8. A smaller ó helps efficiently mitigate strong object-indueed spurious correlations and speed up training, but when the impact from object-side is weak, a small leads to a waste of training data. In the experiment observe that relatively small 8 is more beneficial for Rotated Fashion MNIST while a large works better on Rotated MNIST. Figure 6 shows the results of different 6.
75
Accoscy
4-066
0.7
0.0
1.0
30
10
25
20
25
30
25
32
20
25
30
60
45
Epochs
Egpcht
(a) Rotated Fashion MNIST
(b) Rotated MNIST
Figure Average test accuracy 20 experiments with random seeds during 50 epochs with different ó on MNISTs of DANN. 5 = 1.0 corresponds to DOMI with only level one.
5.3 EXPERIMENTS ON CIFARS.
We then extend our experiments to two more challenging benchmarks (CIFARIO-C and CIFAR100-C) with three backbone DG algorithms.
5.3.1 CIFAR AND CIFAR100-C.
To inspect the robustness of Neural Networks to common corruptions and perturbations, Hendrycks Dietterich (2019) add noises such as frost and fog effects to images by 19 corruption functions with different severity. In experiments, modify the original CIFAR1O-C, CIFAR100-4 in (Hendrycks & Dietterich, 2019). After generating indices using different random seeds to select images from CIFARIO and CIFAR100, data. As for training domains, we leverage the left 17 corruption functions where each function holds three severity: light, medium, and hard, i.e., there are 51 domains in the training
use 2 of 19 functions to transform images test domain
5.3.2 EMPIRICAL RESULTS AND ANALYSIS CIFARS
The empirical results in Table 4 show the improvement of DOMI across datasets and algorithms. Detailed settings and more results are listed in Appendix B.2. We further conduct experiments
Table 4: Average test accuracy of three algorithms. We repeat the experiment for 20 times with random seeds.
Dataset
Sampling scheme
DANN
MMD
CORAL
levelo
63.4
65.0
68.5
CIFARIO-C
level,
64.2 0.8
66.5 1.5
70.1
levely
64.6
66.9
70.7
levelo
33.9
33.8
35.1
CIFARI00-C
level,
34.9 1.0
35.2
36.3 1.2
levelz
35.3 1.4
35.7
11.9
37.0
to compare sampling and non-sampling lines. The experimental settings and results are shown in Appendix B.4. Combined with Appendix B.3, it would be a significant future work to tackle two issues: extremely imbalanced data; computational overhead for algorithms that need do sampling for multi-times. Moreover, parameter sharing between the trained model and the model used to sample likely be practical means of reducing computational overhead.
CONCLUSION
Under the setting of large number of domains and domains with massive data points, propose diversity-boosse two-level sampling algorithm named DOMI help sample the most informative subset of dataset. Empirical results show that DOMI substantially enhances the out-of-domain accuracy and gets robust models against spurious correlations from both domain-side and object-side.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
This study does not involve any of the following: human subjects, practices to dataset releases, potentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, discimiiationnbus/aainnsss concerns, privacy and security issues, legal compliance, and research integrity issues.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our empirical results, we present the detailed experimental settings in Appendix B.1 in addition to the main text. Besides, we will further provide the source codes for reproducing results in our paper.
REFERENCES
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In International Conference Machine Learning, pp. 145-155. PMLR, 2020a.
Kartik Ahuja, Jun Wang, Amit Dhurandhar, Karthikeyan Shanmugam, and Kush R Varshney. Empirical or invariant risk minimization? sample complexity perspective. arXiv preprint arXiv:2010. 164 2020b.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. 2019.
arXiv preprini
Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions optimization problems affected by uncertain probabilities. Management Science, 59(2):341-357 2013.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. tasks to a new unlabeled sample. Advances in neural information processing systems, 24, 2011.
Generalizing from several related classification
John Duchi, Peter Glynn, and Hongseok Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. arXiv preprint arXiv:1610.03422. 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research,
2016.
Muhammad Ghifary, David Balduzzi, W Bastiaan Kleijn, and Mengjie Zhang. Scatter component analysis: A unified framework for domain adaptation and domain generalization, IEEE transactions on pattern analysis and machine intelligence, 39(7): 1414-1141111 1430,, 2016.
Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. In International conference on machine learning, pp. 2839-2848. PMLR, 2016.
Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Ruocheng Guo, Pengchuan Zhang, Hao Liu, and Emre Kiciman. Out-of-distribution prediction with invariant risk minimization: The limitation and an effective fix. arXiv preprint arXiv:2101.07332. 2021.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pp. 1989-1998. Pmlr, 2018.
10

---

Under review as a conference paper at ICLR 2023
Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multidomain discriminant analysis. In Uncertainty in Artificial Intelligence, pp. 292-302. PMLR, 2020a.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning graphs. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Khurram Javed, Martha White, and Yoshua Bengio. Learning causal models online. arXiv preprint arXiv:2006.0 07461, 2020.
Yuanfeng Ji, Lu Zhang. Jiaxiang Wu, Bingzhe Wu, Lai Huang, Tingyang Xu, Yu Rong, Lanqing (ood) dataset curator and benchmark
Li, Jie Ren, Ding Xue, et al. Drugood: for ai-aided drug discovery-a focus affinity prediction problems with noise annotations. arXiv preprint arXiv:2201.094 ..0937,0977 2022.
Pritish Kamath, Akilesh Tangella, Danica Sutherland, and Nathan Srebro. Does invariant risk minimization capture invariance? In International Conference on Artificial Intelligence and Statistics, pp. 4069-4077. PMLR, 2021.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wit wild distribution shifts. In International Conference on Machine Learning.
pp. 5637-5664. PMLR, 2021.
Masanori Koyama and Shoichiro Yamaguchi. Out-of-distribution generalization with maximal invariant predictor. 2020.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In International Conference on Machine Learning, pp. 5815-5826. PMLR, 2021.
Alex Kulesza, Ben Taskar, et al. Determinantal point processes for machine learning. Foundations and Trends in Machine Learning, 5(2-3): 2012.
Chengtao Li, Stefanie Jegelka, and Suvrit Sra. Efficient sampling for k-determinantal point processes. In Artificial Intelligence and Statistics, pp. 1328-1333 1337. PMLR, 2016.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 5400-5409, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial the IEEE conference on computer vision and pattern recognition,
feature learning. In Proceedings the pp. 5400-5409, 2018b.
Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018c.
Jiashuo Liu, Zheyuan Hu, Peng Cui, Bo Li, and Zheyan Shen. Heterogeneous risk minimization. In International Conference on Machine Learning, pp. 6814- PMLR, 2021a.
Jiashuo Liu, Zheyuan Hu, Peng Cui, Bo Li, and Zheyan Shen. Kernelized heterogeneous risk minimization. arXiv preprint arXiv:2110. 12425, 2021b.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105. PMLR, 2015.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International Conference Machine Learning, pp. 7313-7324, PMLR, 2021.
Lucas Mansilla, Rodrigo Echeveste. Diego H Milone, and Enzo Ferrante. Domain generalization via gradient surgery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6630-6638, 2021.
11

---

Under review as a conference paper at ICLR 2023
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. International Conference on Machine Learning. pp. 10-18. PMLR, 2013.
Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429 443, 1997.
Luke Dakden-Rayned Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM conference on health, inference, and learning, pp. 151-159, 2020.
Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. arXiv preprint arXiv:2009.0033 2020.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series (Statistical Methodology), 7854)947711122 2016.
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common- specific low-rank decomposition. International Conference on Machine Learning, pp. 7728- 7738. PMLR, 2020.
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out- of-distribution generalization. In International Conference Machine Learning, pp. 18347-18377. PMLR, 2022.
Mateo Rojas-Carulle Bernhard Schölkopf. Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):1309-1342 2018.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. arXiv preprint arXiv:2010.05761, 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprini 1rXiv:1911.088 2019.
Soroosh Shahtalebi, Cean-Christophe Gagnon-Audet, Touraj Laleh, Mojtaba Faramarzi, Kartik Ahuja, and Irina Rish. Sand-mask: An enhanced gradient masking strategy for the discovery of invariances in domain generalization. arXiv preprint arXiv:2106.02200 2021.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of- -utt----distiiuttin generalization: A survey, arXiv preprint arXiv:2108. 13244, 2021.
Claudia Shi, Victor Veitch, and David Blei. Invariant representation learning for treatment effect estimation. In Uncertainty in Artificial Intelligence, pp. 1546-1555. PMLR, 202
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprini vrXiv:2104.09937 2021b.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In
European conference computer vision, pp. 443-450. Springer, 2016.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng. and Philip Yu. Generalizing to unseen domains: A survey domain generalization. IEEE Transactions on Knowledge and Data Engineering. 2022.
Yufei Wang. Haoliang Li, and Alex C Kot. Heterogeneous domain generalization via domain mixup. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3622-3626. IEEF, 2020.
Yuan Wu, Diana Inkpen, and Ahmed El-Roby. Dual mixup regularized learning for adversarial domain adaptation. In European Conference Computer Vision, pp. 540-555. Springer, 2020.
12

---

Under review as a conference paper at ICLR 2023
Runtian Zhai, Chen Dan, Zico Kolter, and Pradeep Ravikumar. Doro: Distributional and outlier robust optimization. In International Conference on Machine Learning, pp. 12345-12355. PMLR, 2021.
Michael Zhang, Nimit S Sohoni, Hongyang Zhang, Chelsea Finn, and Christopher Ré. Correct-n- contrast: A contrastive approach for improving robustness to spurious correlations. arXiv preprint arXiv:2203.01517, 2022.
Han Zhao, Shanghang Zhang. Guanhang Wu, José MF Moura, Joao Costeira, and Geoffrey Gordon. Adversarial multiple source domain adaptation. Advances in neural information processing systems,
31, 2018.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pp- 7523-7532. PMLR, 2019.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. 2021.
13

---

Under review as a conference paper at ICLR 2023
Appendix of DOMI
CONTENTS
A The Simulated Dataset
14
Experiments
14
B.1 Settings and results on MNISTs
14
B.2 Setting and results on CIFARS
15
B.3 Experiments iwildcam
16
B.4 Why sampling?
16
C How can spurious correlations occur in the two datasets?
18
A THE SIMULATED DATASET
Table 5:
The simulated dataset of the toy example. From these 12 data points sample 6 for
training.
D1
D2
D3
Da
D₂
Do
Dy
Dg
Dg
D10
D1
D12
X1
0
0
0
0
0
1
1
X2
0
1
X3
0
0
0
0
1
1
X4
0
1
B EXPERIMENTS
B. SETTINGS AND RESULTS ON MNISTS
For DANN, the training epochs are set to be 50. MatchDG is two-phase method, and in experiment we set 30 epochs training for phase 1 and 25 epochs for phase 2. The training epochs of FISH set to be 5. Each epoch contains 300 iterations and 30 iterations. And in Figure we slightly abuse epoch to mean the time we obtain test accuracy. Unlike MatchDG and DANN, fish needs to sample domains in each iteration instead of training one list of domains. Sampling domains in each iteration will result in great computational overhead compared randomly sampling. Thus we just sample 30 domain lists containing diverse domains using level-one-sane -am--lomlliiggg of DOMI and repeatedly train the model on these domain lists(one list for one iteration) for level,. As for levelz, we further utilize weree-two-sampping to sample data batches of each domain in the domain lists for training. The former 3 DG algorithms utilize SGD optimizer with learning rate 0.01, weight decay 5e-4 and momentum 0.9. The training epochs of MMD and CORAL are set as 30. These two algorithms leverage Adam optimizer with learning rate 0.001 and weight decay All five algorithms use Resnet18 model. Figure show test accuracy and maximal accuracy among 20 times of repeated experiments with random seeds leveraging different sampling levels on Rotated Fashion MNIST and Rotated MNIST. Among training epochs, the test accuracy rises to the peak value and then decline along with the increase of validation accuracy. In this figure maximal accuracy represents the peak value. Each tiny circle represents one times of experiment, of which the vertical location corresponds to the accuracy value. The horizontal line inside each box indicates the mean value.
observe test accuracy every
14

---

Under review as a conference paper at ICLR 2023
56
46
70
65
60
Leselo
Tevel,
levely
Ievelo
lecel
levely
(a) DANN on Rotated MNIST
(b) DANN on Rotated Fashion MNIST
90
levela
level,
levely
lecedo
levely
levelg
CORAL Rotated MNIST
(d) CORAL on Rotated Fashion MNIST
06
56
Assurato
46
40
36
70
30
leceio
lecel,
Levelz
fevela
levely
levely
MMD Rotated MNIST
(f) MMD Rotated Fashion MNIST
Figure 7: Boxplot of DANN, CORAL and MMD.
64.5
0.22
33.7
34.8
35.2
35,00
63.2
84.1
62
34.75
0.07
63.4
642
54.1
31.9
34.50
0.06
63.4
63.6
63.6
6.18
33.9
34.9
34.9
36.22
0.17
6.03
59.6
59.8
58
0.02
54.1
563
55.9
0.14
23.2
34.3
33.7
33.50
0.01
52.6
5422
53.1
0.13
33.2
33.5
333.25
(a) DANN on CIFARIO-C
(b) DANN CIFAR100-C
Figure 8: Average test accuracy 20 experiments with random seeds using different learning rates under three sampling schemes of DANN.
B.2 SETTING AND RESULTS ON CIFARS
The training epochs are set to be 60 for DANN and 30 for MMD and CORAL. DANN utilize SGD optimizer with weight decay 5e-4 and momentum 0.9. Figure 8 shows the results of DANN using different learning rates. MMD and CORAL leverage Adam optimizer with learning rate 0.001 and weight decay 0. The loss of MMD and CORAL is the sum of ERM-loss term and term of weighted distance between domains. The weight set as 1.0 on two CIFAR datasets for CORAL while 1.0 CIFAR 10-C and 0.2 CIFAR 100-C for MMD. All three algorithms use Resnet18
15

---

Under review as a conference paper at ICLR 2023
15
65
Accumar
Level
.vel
50
Lovel
48
Leve
Levil
Level
45
43
20
15
20
Epochs
(a) CORAL on CIFARIO-C
(b) MMD CIFARIO-C
Figure 9: Average test accuracy of 20 experiments with random seeds during 30 epochs leveraging different sampling levels.
Table 6: Macro F1 of FISH iwildcam under three sampling schemes.
levelo
level1
level2
Iwildcam 22.0
22.8
23.4
model, hyperparameters 8 0.9, 5/51 on two CIFAR datasets. Figure 9 show average test accuracy of 20 experiments with random seeds during 30 epochs leveraging different sampling levels on CIFARIO-C.
B.3 EXPERIMENTS ON IWILDCAM
WILDS (Koh et al., 2021) is a curated collection of benchmark datasets representing distribution shifts faced in the wild. As dataset in WILDS, iwildcam contains photos of wild animals and 324 different camera traps are taken as domains. The data of iwildcam is extremely unbalanced, while part of the domains contain less than 20 photos, some domains contain over 2000 ones. In the original experiments of Shi et al. (2021b), iwlidcam is divided into batches in each domain. FISH samples 10 batches from different domains for training in each iteration. The sampling probability of one batch in domain is proportional to the number of batches left in this domain. This sampling scheme is taken as levely here and we refer to the result (Shi et al., 2021b). each iteration, level, samples 10 batches based on DPP using invDANN, level, first samples 10 batches in the wvel-one-sampling and among them selects 6 batches in the level-two-san -wae--li.......g................ Under the same setting the original experiments, the results on iwildcam of FISH are shown in Table 6.
Although DOMI gets higher Macro F1 score, it leads to much larger computational overhead since needs to do sampling in each iteration. Moreover, for DANN and MatchDG, Macro Fl of diverse domains may be significantly lower than randomly sampled domains because of the unbalanced data, i... the diverse domains may contain much fewer data compared to the randomly sampled domains.
B.4 WHY SAMPLING?
We conduct this experiment CIFAR10-C and CIFAR100-C with two backbone algorithms: DANN and CORAL. We set training on all 51 domains in each epoch as the base line.
One-shot sampling & non-sampling. We first do comparison between one-shot sampling lines., ie.. sample subset of domains and data for training for all epochs, and non-sampling line on accuracy and wall-clock time required. Specifically, Table 4 and add experiments on non-sampling line with settings as follows: the training epochs are set to be 15 for DANN and 10 for CORAL: the deployed Neural Network is Resnet18; 8; the experiment with DANN utilize SGD optimizer with learning rate 0.2, weight decay 5e-4 and momentum 0.9; the experiment with CORAL leverage Adam optimizer with learning rate 0.001 and weight decay 0. Table 7 shows the accuracy and wall-clock time of each line.
supplement the spent wall-clock time to
16

---

Under review as a conference paper at ICLR 2023
Table Accuracy(%) and wall-clock time(seconds).
Dataset
Backbone Algorithm
Sampling Scheme
Accuracy
Wall-clock Time
levelo
63.4
916(0.22x)
level1
64.2
1325(0.32x)
DANN
level2
64.6
1523(0.37x)
Non con-sampling
89.3
4164(1.00x)
CIFARIO-C
levelo
68.5
461(0.05x)
level1
70.1
863(0.09x)
CORAL
levelz
70.7
1074(0.11x)
Non- Non-sampling
94.2
10152(L.00x)
levelo
33.9
945(0.22x)
level1
34.9
1427(0.33x)
DANN
levelz
35.3
1664(0.38) (8x))
Non- Non-sampling
71.0
4368(1 (000())00))
CIFAR100-C
levelo
35.2
496(0.04X)
level1
36.3
902(0.08X)
CORAL
levely
37.0
1158(0.10x)
Non- non-sampling
85.3
11361 (1.00x)
Table 8: Mean accuracy(%) and wall-clock time(seconds). We repeat the experiment of each line for 3 times with different random seeds.
Dataset
Backbone Algorithm
Sampling Scheme
Accuracy
Wall-clock Time
levelo
82.2
923(0.22x)
level
83.5
(031()
DANN
levelz
83.9
6712(1.61x)
Non-sampling
89.3
4164(1.00X)
CIFARIO-C
levely
87.8
461(0.05x)
level,
89.2
1004(0.10x)
CORAL
level2
89.7
6175(0.61x)
Non- Non-sampling
94.2
10152(1.00x)
levelo
61.7
945(0.22x)
level
62.9
1674(0.38X)
DANN
levelz
63.4
6864(1.577
Non- Non-sampling
71.0
4368(1.00X)
CIFARI00-C
levelp
76.3
496(0.04x)
level,
78.1
1142(0.10x)
CORAL
level2 78.4
6678(0.59x)
Non- Non-sampling
85.3
11361 (1.00x)
Multi-times sampling & non-sampling. We further do comparison between lines sampling in each epoch and non-sampling line. In this experiment: levelp randomly samples 5 domains in each epoch; level, train an invDANN model only once to get the one-shot similarity matrix among domains based on which select 5 domains in each epoch; levely further select different batches in each epoch compared level,- The training epochs are set to for sampling lines while 15 and 10 respectively for the non-sampling line. For all lines, the deployed Neural Network is Resnet 8; DANN utilize SGD optimizer with learning rate 0.2, weight decay 5e-4 and momentum 0.9; CORAL leverage Adam optimizer with learning rate 0.001 and weight decay O. Table 8 shows the mean accuracy and wall-clock time of each line and we make the following observations.
for DANN and 50 for CORAL
Time overhead of pairwise domain operations. Notice the proportion of the costed wall-clock time of non-sampling line and levelp- The proportion of CORAL, method work by pairwise operations between domains. apparently larger than that of DANN. Sampling(levelp, level,) conspicuously reduce the time overhead of CORAL.
17

---

Under review as a conference paper at ICLR 2023
0
O
1
/
/
/
2
2
a
3
2
>
2
2
1
2
2
2
3
3
8
3
3
5
5
3
3
8
441
P
4
4
4
4
4
4
4
4
55
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
12345
7
7
7
7
7
7
7
7
9
7
7
7
8
8
(a) Hypothetical training set
(b) Hypothetical test set
Figure 10: Two figures to illustrate the impact of object-induced spurious correlations on MNIST.
The efficiency and drawbacks of DOMI. For one-shot sampling lines, DOMI substantially improves over random sampling at a acceptable extra overhead. Nevertheless, one-shot sampling leads to large gap from non-samee sampling line since the model is consistently trained on the same small subset. For multi-time sampling lines, using the one-shot similarity matrix between domains, level, improve the performance over random sampling at a relatively small cost. However, levely need to sample batches from different subset of domains in each epoch, which means levely has to compute the similarity matrix between batches in each epoch, leading great time overhead.
Discussions Combined with Appendix B.3, would be a significant future work to tackle two issues: extremely imbalanced data; computational overhead for algorithms that need do sampling for multi-times. Moreover, parameter sharing between the trained model and the model used to sample likely be practical means of reducing computational overhead.
C How CAN SPURIOUS CORRELATION OCCUR IN THE TWO DATASETS?
It's much easier to differentiate the rotation degree than discriminating the objects. This can be empirically verified since it only needs about 30 epochs for model to achieve over 98% validation accuracy of classifying 61 different degrees while 50 epochs to achieve no more than 97% and 88% validation accuracy of classifying 10 different objects on rotated MNIST and Fashion MNIST, respectively. Thus if certain class label is closely associated with a certain rotation degree in the training set, recognizing objects by actually recognizing rotation degree can be a shortcut and nomain-inducee spurious correlation, just like classifying cats and lions using the background in the toy example. As for object-induced spurious correlation, rotated MNIST, the handwriting is the feature of the object, however, can also be the spurious correlation. For example, in Figure 10, let's focus the number "1" and "7", After training Figure 10a, can the model correctly recognize "I" in Figure 10b instead of wrongly taking it as "?"
On Fashion MNIST, assume we take the data in Figure 11 as the training set. Majority of the data points Shirt are darker than Coat. When differentiate between Shirt and Coat, a model can simply
Dousippian
Exaroles
Sunduh
Beg
Avkle
Figure 11: The figure to illustrate the impact object-induced spurious correlations on Fashion MNIST.
18

---

Under review as a conference paper at ICLR 2023
takes the bright ones coats and the dark ones shirts to obtain high training accuracy. However, what the color of Shirt and Coat is similar in the test set?
Thus our proposed framework set two levels of sampling to mitigate the impacts of domain-side and object-side, the sampling fact a rebalance procedure of data.
19


Reviews:
[{"review_id": "xHlphxJlE2T", "reviewer": "Reviewer_xNqP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper:\n* solves the domain generalization problem\n* presents several observations that diversity helps mitigate serious correlations\n* proposes a sampling method that helps train robust models\n* conducts experiment on Rotated MNIST and Rotated Fashion MNIST to show the effectiveness of the proposed algorithm", "summary_of_the_review": "I think the paper is not ready for publication at this point. If the authors can show more evidence and support, I am happy to raise my scores.", "strength_and_weaknesses": "The paper is easy to follow and the proposed algorithm is easy to understand. The idea is not totally new but seems promising. Since the work is likely considered as adopting and improving existing method, I believe the experimental results should be strong. However, I am not fully convinced by the current experiments. Here are my complaints:\n* The paper can be more professionally written. For example, the definition of set $C$ is very vague. What is precisely a \"good\" correlation?\n* The experiments are very limited. Why not try some large scale experiments mentioned in DomainBed? It is important to show the proposed algorithm on larger and tougher dataset. I am also curious to see how it works on ColoredMNIST.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_KFi1l1Xze", "reviewer": "Reviewer_eKSv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary\nThe authors consider the problem of domain generalization (DG). In specific, they are addressing the issues of scalability and objective.\n Scalability: The state-of-the-art DG methods involve computational\n complexity of 𝒪 (n2) corresponding to pairwise domain operations with n\ndomains. In addition, each domain may contain a large number of data points. This is computationally prohibitive.\nObjective: The existing methods entirely focus on excluding domain-side impacts, i.e., spurious correlations, and don’t consider the data-side impacts.\nThe authors address the above two issues by proposing a two-level sampling method called the Diversity boosted two-level sampling framework (DOMI).\nThe authors argue that diverse domains of data help exclude spurious correlation (observation 1). In regard to this, they propose to sample diverse domains using invDANN as featurizer and DPP as a sampler in the first level.\nThe authors observe excluding domain-induced correlations is insufficient for learning a robust model (observation 2). To alleviate this, they propose to sample diverse data batches from the selected domains (domains obtained from first-level sampling) using ERM and DPP in the second level.\nThe authors validate their algorithm with five backbone DG algorithms on two simulated benchmarks (Rotated MNIST and Rotated Fashion MNIST).", "summary_of_the_review": "The authors consider the problem of domain generalisation (DG). In specific, they are addressing the issues of scalability and objective.", "strength_and_weaknesses": "Strengths\nThe problem of DG considered and the issues highlighted such as scalability and objective in this work is very relevant.\nThe authors provide an important counter-example, i.e., cats and lions to draw attention to the issue of data-side impacts. They also present the comparison of various sampling methods such as random sampling, and different diversity-boosted sampling methods to illustrate the importance of their work and their effect on both object-side spurious correlations and domain-side spurious correlations.\n       \nThe authors give reasonable arguments for observation 1 using concepts such as good, casual, and spurious correlations. In addition, observation 2 is well explained using the structural causal model.\nThe solution proposed, i.e., diversity-boosted two-level sampling to mitigate the issues highlighted seems intuitively well grounded.\nThe experimental results show the validity of their algorithm. The proposed method outperforms level 0 and level 1 sampling schemes across 5 backbone DG methods and two simulated benchmark datasets.\nThe authors provide a decent analysis of the gap between test accuracy and maximal accuracy as well as the impact of the choice of the hyper-\nparameter δ (proportion of data points) . \n\nDrawbacks:\n\nNo analysis regarding the choice of the hyper-parameter β (proportion of domains) is presented.\nThe statement “spurious correlations essentially result from imbalance data” is not entirely true. The spurious correlations result from the existence of anti-causal paths. You could have the same data and different causal models, some of which result in spurious correlations.\nIt would be better if the authors elaborate on causality-related concepts such as causal correlations and unobserved confounders or give proper references.\nThere are a couple of typos (is composed by Featurizer, Classifier ..., while the test accuracy of level 0 .... that of level 2 centers, and ...)\nThe experiments on large-scale datasets are missing. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7U0QqWPTw57", "reviewer": "Reviewer_3y4o", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides a new sampling strategy for domain and datapoint sampling in the domain generalization context. The method encourages sampling diverse domains and diverse datapoints within each domain. To achieve domain diversity, the authors train a class-adversarial neural network (the inverse of the DANN method in [1]) and use a Determinantal Point Process (DPP) with the features produced by this network. To achieve datapoint diversity, the authors train standard ERM on the datapoints sampled randomly from the sampled domains, and also use a DPP.", "summary_of_the_review": "This is an interesting paper, with promising results. Above, I highlighted a few questions and concerns about the presentation, methodology and experiments. The limited empirical evaluation is the main issue in my opinion.\n\n## References\n\n[1] [_Domain-Adversarial Training of Neural Networks_](https://arxiv.org/abs/1505.07818);\nYaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, Victor Lempitsky\n\n[2][_Domain Generalization using Causal Matching_](https://arxiv.org/abs/2006.07500);\nDivyat Mahajan, Shruti Tople, Amit Sharma", "strength_and_weaknesses": "## Strengths\n\n**S1**: The proposed method makes intuitive sense.\n\n**S2**: Empirical results show improvements compared to the baselines with multiple base domain generalization methods.\n\n## Weaknesses\n\n**W1**: I think some of the presentation regarding the \"object spurious correlations\" vs \"domain spurious correlations\" is somewhat imprecise\n\n**W2**: Some of the design decisions are not very clearly motivated and not ablated, in particular the use of invDANN\n\n**W3**: The Empirical evaluation is focusing primarily on Rotated MNIST and Rotated FashionMNIST, which are small-scale synthetic datasets, rather than more standard domain generalization datasets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3f6hX2bmV8", "reviewer": "Reviewer_kTC5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors looks into Domain Generalization for a large number of domains. They propose to sample (using two stage procedure) both the domains and the data within each domain, in order to help the model to filter out spurious correlation.  They show a motivational example that the assumption that domains usually have shared causal features and differ in spurious correlation features does not always hold. For the first stage sampling (choosing the domains), they  train an inverse DANN (classify domains while not being able to assign the class labels correctly) to get to extract only domain specific features. Then using these features, they construct domain similarity matrix . For the second stage sampling,  DPP samples diverse domains and diverse data points within the domains (using similarity matrix constructed by ERM)\n", "summary_of_the_review": "Update:\nRaising my score a bit, since authors included new experiments. TLDR: without subsampling the performance does improve but on some datasets the drop with sampling is small enough considered time saved\n\n\nOverall my main complains about this paper are\n1) multi step procedure with each step step being expensive (train inverse DANN, construct matrix of domain similariities, construct data points similarities matrix etc). Each step will require its own tuning. Unclear how your method compares in terms of time/computation to any methods you labeled as expensive\n2) No comparison with no sampling at all for the table 3. Without this, it is impossible to tell whether the sampling is required at all\n3) You state that DANN works with pairwise domains (n^2)??? You have an adversarial head that classifies into the number of domains, how is it n^2??? Also it is misleading to state this for  MMD too - it will indeed work on pairs, but only on pairs from the batch, so it is batch_size^2 max\n\nMinor: DPP related work - repulsive (interactions) is a strange word to use here. Spurious? Non intended?\n", "strength_and_weaknesses": "Pros:\n- good observations (spurious correlations)\n- interesting idea to choose diverse domains and data points\n- clarity - pretty easy to follow\nCons:\n- overly complicated multi-stage method that requires many expensive computations (train inverse DANN, construct matrix of pairwise domain similarity, construct matrix of data points similarities, train ERM etc) \n- no comparison with no sampling whatsoever - so impossible to say if it was needed in the first place\n- only image experiments. Is it because you can't find non image DG datasets with such a large number of domains? is it possible to generate some synthetic data? why not to try WILDS and OGB-MolPCBA that you mention? Or DRUG OOD", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject