In Search of Lost Domain Generalization
Authors: Ishaan Gulrajani
David Lopez-Paz
Venue: ICLR 2021 Poster (2021)
Keywords: domain generalization
reproducible research

Abstract: The goal of domain generalization algorithms is to predict well on distributions different from those seen during training.
While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions---datasets, network architectures, and model selection criteria---render fair comparisons difficult.
The goal of this paper is to understand how useful domain generalization algorithms are in realistic settings.
As a first step, we realize that model selection is non-trivial for domain generalization tasks, and we argue that algorithms without a model selection criterion remain incomplete.
Next we implement DomainBed, a testbed for domain generalization including seven benchmarks, fourteen algorithms, and three model selection criteria.
When conducting extensive experiments using DomainBed we find that when carefully implemented and tuned, ERM outperforms the state-of-the-art in terms of average performance.
Furthermore, no algorithm included in DomainBed outperforms ERM by more than one point when evaluated under the same experimental conditions.
We hope that the release of DomainBed, alongside contributions from fellow researchers, will streamline reproducible and rigorous advances in domain generalization.

Published as a conference paper at ICLR 2021
IN SEARCH OF LOST DOMAIN GENERALIZATION
Ishaan Gulrajani" Stanford University igu1222 gmmil.com
David Lopez-Paz
Facebook Al Research d com com
ABSTRACT
The goal of domain generalization algorithms is to predict well on distributions different from those seen during training. While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions datasets, network architectures, and model selection mriteri-reende fair comparisons difficult. The goal of this paper is to understand how useful domain generalization algorithms are in realistic settings. As a first step, we realize that model selection is non-trivial for domain generalization tasks, and we argue that algorithms without a model selection criterion remain incomplete. Next we implement DOMAIN testbed for domain generalization including seven benchmarks, fourteen algorithms, and three model selection criteria. When conducting extensive experiments using Do- MAINBED we find that when carefully implemented and tuned, ERM outperforms the state-of-the- art in terms of average performance. Furthermore, cluded DOMAINBED outperforms ERM by more than one point when evaluated under the same experimental conditions. We hope that the release of DOMAINBED, alongside contributions from fellow researchers, will streamline reproducible and rigorous advances in domain generalization.
algorithm in-
INTRODUCTION
Machine learning systems often fail to generalize out-of-dissrribttinn, crashing in spectacular ways when tested outside the domain of training examples (Torralba and Efros, 2011). The overreliance of learning systems on the training distribution manifests widely. For instance, self-driving car systems struggle to perform under conditions different to those of training, including variations in light (Dai and Van Gool, 2018). weather (Volk et al., 2019), and object poses (Alcorn et al., 2019). As another example, systems trained on medical data collected in one hospital do not generalize to other health centers (Castro et al., 2019; AlBadawy et al., 2018; Perone et al., 2019; Heaven, 2020). Arjovsky et al. (2019) suggest. that failing generalize out-of-distribution is failing capture the causal factors of variation in data, clinging instead to casier-to-fit spurious correlations prone to change across domains. Examples of spurious correlations commonly absorbed by learning machines include racial biases (Stock and Cisse, 2018). texture statistics (Geirhos et al., 2018), 1), and object backgrounds (Beery et al., 2018). Alas, the capricious behaviour of machine learning systems -iitr---iitritttion roadblock to their deployment in critical applications.
Aware of this problem, the research community has spent significant efforts during the last decade to develop algorithms able to generalize Generalization (DG) assumes access to multiple datasets during training, each of them containing examples about the same task, but collected under different domain experimental condition (Blanchard et al., 2011; Muandet et al., 2013). The goal of DG algorithms is to incorporate the invariances across these training domains into classifier, in hopes that such invariances will also hold in novel test domains. Different DG solutions assume different types of invariances, and propose algorithms to estimate them from data.
In particular, the literature in Domain
Despite the enormous importance of DG, the literature scattered: a plethora of different algorithms appear yearly, each of them evaluated under different datasets, neural network architectures, and model selection criteria. Borrowing from the success standardized computer vision benchmarks
Iphahetiaal order, equal contrihution. Work done while IG was at Facebook Al Research. This paper is living benchmark, always refer the latest version available at https: / //rxi wtwrg/bb///aae 01 434

---

Published as a conference paper at ICLR 2021
Table 1: Our ERM baseline outperforms the state-of-the-art in terms of average domain generalization performance, even when picking the best competitor per dataset.
Dataset algorithm
DG accuracy per test domain
Average
Rotated MNIST (full)
0'
15'
30
45
60
75
DIVA (lise et al., 2019)
95.3
98.7
98.7
98.4
97.7
94.5
97.2
Our ERM
95.9
98.9
98.8
98.9
98.9
96.4
98.0
VLCS
C
L
V
G2DM (Albuquerque et al., 2019)
95.5
67.6
69.4
71.1
75.9
Our ERM
97.7
64.3
73.4
74.6
77.5
PACS
S
RSC (Huang et al., 2020)
87.9
82.1 1
97.9
83.4
87.8
Our ERM
84.7
80.8
97.2
79.3
85.5
OfficeHome
C
R
DDAIG (Zhou et al., 2020)
59.2
52.3
74.6
76.0
65.5
Our ERM
61.3
52.4
75.8
76.6
66.5
All datasets
Best SOTA competitor
81.6
Our ERM
81.9
such as ImageNet (Russakovsky et al., 2015), the purpose of this work is to perform rigorous comparison DG algorithms, well open-source our software for anyone replicate and extend our analyses. This manuscript investigates the question: How usefud are different DG algorithms when evaluated a consistent and realistic setting?
To answer this question, implement and tune fourteen DG algorithms carefully, to compare them across seven benchmark datasets and three model selection criteria. There are three major takeaways from our investigations:
Claim 1: A careful implementation of ERM outperforms the state-of-the-ar in terms of average performance across common benchmarks (Table full list in Appendix A.5).
Claim 2: When implementing fourteen DG algorithms consistent and realistic setting, no competitor outperforms ERM by more than one point (Table 3).
Claim 3: Model selection is non-trivial for DG, yet affects results (Table 3). As such, we argue that DG algorithms should specify their own model selection criteria.
As a result of our research, we release DOMAINBED, framework to streamline rigorous and reproducible experimentation in DG. Using DOMAINBED, adding new algorithm or dataset is matter of few lines of code. A single command runs all experiments, performs all model selections, and -etteregerrsss the result tables included in this work. DOMAINBED is living project: we welcome pull requests from fellow researchers update the available algorithms, datasets, model selection criteria, and result tables.
Section 2 kicks off our exposition with review of the DG setup. Section 3 discusses the difficulties of model selection in DG and makes recommendations for a path forward. Section 4 introduces DOMAINBED, describing the features included in the initial release. Section 5 discusses the exper- imental results of running the entire POMAINBED suite, illustrating the competitive performance of ERM and the importance of model selection criteria. Finally, Section 6 offers view future research directions in DG. Appendix A reviews one hundred articles spanning a decade research in DG, summarizing the experimental performance of over thirty algorithms.
2

---

Published as a conference paper at ICLR 2021
Table 2: Learning setups. La and U denote the labeled and unlabeled distributions from domain d.
Setup
Training inputs
Test inputs
Generative learning
U
Unsupervised learning
U
U
Supervised learning
L
U
cemi-supervised learning
L',U"
U
Multitask learning
L
U1
Udu
Continual (or lifelong) learning
L
L
U U
Domain adaptation
L
Ldde Udu++
U
Transfer learning
U1
Uder, I 1
Ud++1
Domain generalization
L
ddr
Ud+t1
THE PROBLEM OF DOMAIN GENERALIZATION
The goal supervised learning is to predict values y E 3 of a target random variable Y, given values E X' of an input random variable X. Predictions ú f(x) about originate from predictor f X - y. We often decompose predictors as f = w o o. where we call ó X + H the featurizer, and w H + y the classifier. To solve the prediction task collect the training dataset D = ( y: } which contains identically and independently distributed (i.i.d.) examples from the joint probability distribution P P(X, Y). Given loss function e: y y [0, oo) measuring prediction error, supervised learning seeks the predictor minimizing the risk [(x.p)~<(((ff
v)]- Since we only have access to the data distribution P(X,Y Y) via the dataset D, we instead search a predictor minimizing the empirical risk Σ ((f)),), Va) (Vapnik, 1998).
The rest of this paper studies the problem of Domain Generalization (DG), an extension of supervised learning where training datasets from multiple domains (or environments) are available to train our predictor (Blanchard et al., 2011). Each domain d produces a dataset D {( vf") ) d= containing i.i.d. examples from some probability distribution P(X yd), for all training domains d E (l,....,,du dez J. The goal of DG is out-of-distributioon generalization: learning predictor able to perform well at some unseen test domain du + 1. Since no data about the test domain is available during training, we must assume the existence of statistical invariances across training and testing domains, and incorporate such invariances (but nothing else) into our predictor. The type of invariance assumed, as well as how to estimate it from the training datasets, varies between DG algorithms. We review hundred articles in DG spanning decade of research and thirty algorithms in Appendix A.5.
DG differs from unsupervised domain adaptation. the latter, unlabeled data from the test domain available during training (Pan and Yang, 2009; Patel et al., 2015; Wilson and Cook, 2018). Table 2 compares different machine learning setups to highlight the nature of DG problems. The causality literature refers to DG learning from multiple environments (Peters al.. 2016: Arjovsky et al., 2019). Although challenging, the DG framework can capture some the difficulty of real prediction problems, where unforeseen distributional discrepancies between training and testing data are surely expected. At the same time, the framework can be limiting: in many real world scenarios there may be external variables informing about task relatedness (space, time, annotations) that the DG framework ignores.
MODEL SELECTION AS PART OF THE LEARNING PROBLEM
Here we discuss issues surrounding model selection (choosing hyperparameters, training checkpoints, architecture variants) in DG and make specific recommendations for path forward. Because we lack access to a validation set identically distributed to the test data, model selection in DG is not as straightforward in supervised learning. Some works adopt heuristic strategies whose behavior is not well-studied, while others simply omit a description how to choose hyperparameters. This leaves open the possibility that hyperparameters were chosen using the test data, which is not
3

---

Published as a conference paper at ICLR 2021
methodologically sound. Differences in results arising from inconsistent tuning practices may be misattributed to the algorithms under study, complicating fair assessments.
We believe that much of the confusion surrounding model selection in DG arises from treating as merely& a question of experimental design. To the contrary, model selection requires making theoretical assumptions about how the test data relates to the training data. Different DG algorithms make different assumptions, and is not clear a priori which ones are correct, or how they influence the model selection criterion. Indeed, choosing reasonable assumptions is the heart of DG research. Therefore, DG algorithm without a strategy to choose its hyperparameters should be regarded as incomplete.
Recommendation 1 DG algorithm should be responsible for specifying a model selection method.
While algorithms without well-justified model selection methods are incomplete, they may be useful stepping-stoner in research agenda. In this case, instead of using an ad-hoc model selection method, we can evaluate incomplete algorithms by considering an oracle model selection method, where we select hyperparameters using some data from the test domain. Of course, it is important to avoid invalid comparisons between oracle results and baselines tuned without an oracle method. Also, unless we restrict access to the test domain data somehow, we risk obtaining meaningless results (we could just train on such test domain data using supervised learning).
Recommendation 2 Researchers should disclaim any oracle-selection results as such and specify policies limit access to the test domain.
3.1 THREE MODEL SELECTION METHODS FOR DG
Having made broad recommendations, we review and justify three model selection criteria for DG. Appendix B.3 illustrates these with an specific example.
Training-domail validation We split each training domain into training and validation subsets. We train models using the training subsets, and choose the model maximizing the accuracy on the union of validation subsets. This strategy assumes that the training and test examples follow similar distributions. For example, Ben-David et al. (2010) bound the test error of a classifier with the divergence between training and test domains.
Leave-one-domain-outt validation Given dir training domains, we train dur models with equal hyperparameters, each holding one of the training domains out. We evaluate each model its held-out domain, and average the accuracies of these du models over their held-out domains. Finally, we choose the model maximizing this average accuracy, retrained on all di domains. This strategy assumes that training and test domains follow meta-distribution over domains, and that our goal is maximize the expected performance under this meta-distribution. Note that leaving k > domains out would increase greatly the number of experiments, and introduces a hyperparameter k.
Test-domain validation (oracle) We choose the model maximizing the accuracy on a validation set that follows the distribution of the test domain. Following carlier recommendation to limit test domain access, we allow one query (the last checkpoint) per choice of hyperparameters, disallowing early stopping. Recall that this is not valid benchmarking methodology. Oracle-based results be either optimistic, because we select models using the test distribution, or pessimistic, because the query limit reduces the number of considered hyperparameters. We also tried limiting the size of the oracle test set instead of the number of queries, but this led to unacceptably high variance.
3.2
CONSIDERATIONS FROM THE LITERATURE
Some references in prior work discuss additional strategies to choose hyperparameters in DG. For instance, Krueger et al. (2020, Appendix B. suggest choosing hyperparameters to maximize the performance across all domains an external dataset. This "leave-one-dee dat-sssteett out" is akin to the second strategy outlined above. Albuquerque et al. (2019, Section 5.3.2) suggest performing model selection based on the loss function (which often incorporates an sggorthm-sppciffe regularizer). and D' Innocente and Caputo (2018, Section 3) derive strategy specific their algorithm. Finally, tools
4

---

Published as a conference paper at ICLR 2021
from differential privacy enable multiple reuses of a validation set (Dwork et al., 2015), which could be a tool to control the power of test-domain validation (oracle).
DOMAINBED: A PYTORCH TESTBED FOR DOMAIN GENERALIZATION
At the heart of our large scale experimentation is DOMAINBED, PyTorch (Paszke et al., 2019)
testbed streamline reproducible and rigorous research DG:
httpps///gchabb..cmm/cccbbbbbbbe
The initial release comprises fourteen algorithms, seven datasets, and three model selection methods (those described in Section 3), as well as the infrastructure to run all the experiments and generate all the ETEX tables below with a single command. The first version of DOMAINBED focuses on image classification, leaving for future work other types of tasks. DOMAINBED is a living project: together with pull requests from collaborators, we continuously update the above repository with new algorithms, datasets, and result tables. As illustrated in Appendix B.5, adding new algorithm or dataset DOMAINBED a matter of a few lines of code.
Algorithms DOMAINBED currently includes fourteen algorithms chosen based on their impact over the years, their published performance, and a desire include varied DG strategies. These are Empirical Risk Minimization (ERM, Vapnik (1998)), Group Distributionally Robust Optimization (GroupDRO, Sagawa al. (2019)), Inter-domain Mixup (Mixup. Xu et (2019); Yan al. (2020); Wang al. (2020b)), Meta-Learning for Domain Generalization (MLDG, Li al. (201 2018a)). Domain- Adversarial Neural Networks (DANN, Ganin et al. (2016)), Class-conditional DANN (C-DANN, Li al. (2018d)), Deep CORrelation ALigmment (CORAL, Sun and Saenko (2016)), Maximum Mean Discrepancy (MMD, Li et al. (2018b)), Invariant Risk Minimization (IRM Arjovsky et al. (2019)), Adaptive Risk Minimization (ARM, Zhang et al. (2020)), Marginal Transfer Learning (MTL, Blanchard et al. (2011; 1; 2017)). Style-Agnostic Networks (SagNet, Nam et al. (2019)), and Represen- tation Self Challenging (RSC, Huang et al. (2020)). Appendix B.1 describes these algorithms, and Appendix B. B.4 lists their network architectures and hyperparameter search distributions.
Datasets DOMAINBED currently includes downloaders and loaders for seven standard DG image classification benchmarks. These are Colored MNIST (Arjovsky et al., 2019), Rotated MNIST (Ghifary et al., 2015). PACS (Li et al., 2017). VLCS (Fang et al., 2013), OfficeHome (Venkateswara et al., 2017) 2017), Terra Incognita (Beery et al., 201 8), and DomainNet (Peng et al., 2019). The datasets based MNIST rotations). The rest of the datasets are "real" since domains vary in unknown ways. Appendix B.2 describes these datasets.
"synthetic" since changes across domains are well understood (colors and
Implementation choices We highlight three implementation choices made towards a consistent and realistic evaluation setting. First, whereas prior work is inconsistent in its choice of network architecture, we finetune ResNet-50 models (He et al., 2016) pretrained on ImageNet for all non- MNIST experiments. We note that recent state-of-the-art results (Balaji al., 2018; Nam al., 2019; Huang et al., 2020) also use ResNet-50 models. Second, for all non-MNIST datasets, we augment training data using the following protocol: crops of random size and aspect ratio, resizing to 224 x 224 pixels, random horizontal flips, random color jitter, grayscaling the image with 10% probability, and normalization using the ImageNet channel statistics. This augmentation protocol is increasingly standard in state-of-the-e
DG work (Nam et al., 2019; Huang et al., 2020; Krueger et al., 2020; Carlucci et al., 2019a: Zhou et al., 2020; Dou et al., 2019; Hendrycks et al., 2020; Wang al., 2020a; Seo et al., 2020; Chattopadhyay et al., 2020). We use no augmentation for MNIST-based datasets. Third, and for RotatedMNISTT we divide all the digits evenly among domains, instead of replicating the same 1000 digits to construct all domains. We deviate from standard practice for two reasons: using the same digits across training and test domains leaks test data, and reducing the amount of training data complicates the task in an unrealistic way.
5

---

Published as a conference paper at ICLR 2021
5 EXPERIMENTS
We rum experiments for all algorithms, datasets, and model selection criteria shipped in DOMAINBED. We consider all configurations of dataset where we hide domain for testing, resulting in the training of 58,000 models. To generate the following results, we simply run sweep Py at commit 0x7df6f00 from DOMAINBED's repository.
Hyperparameter search For each algorithm and test domain, we conduct random search (Bergstra and Bengio, 2012) of 20 trials over joint distribution of all hyperparameters (Appendix B.4). Appendix C.4 shows that running more than 20 trials does not improve our results significantly. We each model selection criterion to select amongst the 20 models from the random search. We split the data from each domain into 80% and 20% splits. We use the larger splits for training and final evaluation, and the smaller splits select hyperparameters (for illustration, Appendix B.3). All hyperparameters are optimized anew for each algorithm and test domain, including hyperparameters like learning rates which are common to multiple algorithms.
Standard error bars While some DG literature reports error bars across seeds, randomness arising from model selection is often ignored. This is acceptable if the goal is best-versus- best comparison, but prohibits analyses concerning the model selection process itself. Instead, we repeat our entire study three times, making every random choice dataset splits. Every number we report is a mean (and its standard error) over these repetitions.
hyperparameters, weight initializations, and
5.1 RESULTS
Table 3 summarizes the results of our experiments. Appendix C contains the full results per dataset and domain. As anticipated in introduction, draw three conclusions from our results.
Claim 1: Carefully tuned ERM outperforms the previously published state-of-the-art Table I (full version in Appendix A.5) shows this result, when we provide ERM with -ommin validation set for hyperparameter selection. Such state-of-the-arr average performance of our ERM baseline holds even when we select the best competitor available in the literature separately for each benchmark. One reason for ERM's strong performance that we use ResNet-50, whereas some prior work uses smaller ResNet- 18 models. As recently shown in the literature (Hendrycks et al., 2020), this suggests that better in-distribution generalization a dominant factor behind better nut-of-distributtin generalization. Our result does not refute prior work: it is possible that with stronger implementations, some competing methods may improve upon ERM. Rather, we provide a strong, realistic, and reproducible baseline for future work to build upon.
Claim 2: When evaluated in a consistent setting, no algorithm outperforms ERM in more than one point We observe this result in Table 3, obtained by running from scratch every combination of dataset, algorithm, and model selection criterion in DOMAINBED. Given any model selection criterion, no method improves the average performance of ERM in more than point. At the number of trials performed, no improvement over ERM is statistically significant according to t-test at significance level 0.05. While new algorithms could improve upon ERM (an exciting premise!), getting substantial DG improvements in a rigorous way proved challenging. Most of our baselines can achieve ERM-like performance because there have hyperparameter configurations under which they behave like ERM (e.g. regularization coefficients that can be set to zero). Our advice to DG practitioners is use ERM (which is safe contender) or CORAL (Sun and Saenko, 2016) (which achieved the highest average score).
Claim 3: Model selection methods matter We observe that model selection with training domain validation set outperforms love--one-dmain out cross-validation across multiple datasets and algorithms. This does not mean that using training domain validation set is the right way to tune hyperparameters. In fact, the stronger performance of oracle-selection (+2.3 points for ERM) suggests headroom to develop improved DG model selection criteria.

---

Published as a conference paper at ICLR 2021
Table 3: DG accuracy for all algorithms, datasets and model selection criteria in DOMAINBED, These experiments compare fourteen popular DG algorithms across seven benchmarks in the exact same conditions, showing the competitive performance of ERM.
Algorithm
CMNIST
RMNIST
VLCS
PACS
OfficeHomme
Terraine
DomainNet
Average
0.1
98.0 0.0
=
85.5
0.2
66.5 0.3
46.1 1.8
40.9
IRM
5.1
9.11
7.5
83.5
64.3 2.2
47.6 0.8
33.9
GrouDROO
+.0
98.0
76.7-0.5
84.4
0.8
66.0 0.7
43.2 1.1
33.3 + 0.2
64.8
Mixup
5.21
9..0
7744 0.6
84.6 0.6
68.1 0.3
4 0.8
39.2 0.1
66.7
MLDG
5..5
9..9
84.9 772=00
66.8 0.6
0.9
4.1
66.7
CORAL
5..5
98.0
7..8=066
0.3
68.7
47.6
4..5
MMD
5.2
9.0
0.9
0.5
66.3
1.6
2.5
DANN
0.3
97.8
7..6=0.4
83.6
65.9 0.6
4.5
3.1
66.1
CDANN
51.7
0.1
=.1
0.9
65.8
1.6
3.3
65.6
MTL
5..4
9..9
7
84.6
66.4 0.5
45.6 1.2
4..6
66.2
SagNet
5..7
9..0
0.5
86.3=0.2
68.1 0.1
48.6
4.1
67.2
ARM
5.2
9.12
77.6
0.3
0.4
64.8 0.3
45.5 0.3
+
VREx
5.1
0.1
78.3 0.2
84.9
0.6
66.4 0.6
46.4
0.6
33.6 2.9
65.6
RSC
5.27
97.6 77.60.11
77.1 0.5
85.2
0.9
65.5 0.9
46.6
1.0
38.9 0.5
66.1
Model selection: training-dovnn domannn validation set
Algorithm
CMNIST
RMNIST
VLCS
PACS
OfficeHome
TerraIne
DomainNet
Average
ERM
0.1
97.7 +:
77.2 0.4
83.0
0.7
65.7 0.5
41.4 1.4
40.6
IRM
4.2
9.
0.6
81.5
0.8
64.3 1.5
3.6
33.5
3.0
62.0
GroupDRO
0.1
77.9 9.1
83,5 0.2
65.2 0.2
4..9
33.0 0.3
62.7
Mixup
3.74
97.8
7.6
832 0.4
67.0
0.4
38.5 0.3
63.8
MIDG
0.2
9..6
7.9
82.9
6.5
4.92
41.0
CORAL
3.87
9.8
78.7
82.6 0.5
68.5 0.2
46.3 1.7
MMD
3.1
9..
0.5
83.2
0.2
60.2 5.2
16.55.155
23.4 9.5
60.7
DANN
40.7 2.3
9.
76.9 0.4
810 1.1
64.9 1.2
44.4
18.21.22
63.4
CDANN
3.44
9.2
97.5
7.2
78.8
64.3 1.7
39.9 3.2
3.10
62.2
35.0
0.1
7..6=0.5
0.
0.5
4.29
40.6
SagNet
0.1
9.00
0.3
0.1
0.3
4.9
0.2
ARM
3.0
0.11
76.6
81.
6.24
4.76
0.1
62.2
VREx
3.3
93.4
76.7 1.0
8133 0.9
6.3
3.0
3..4
60.6
RSC
36.5
97.6 0..
77.5 = 0.5
82.6 0.7
65.8 = 0.7
40.0 = 0.8
38.9 0.5
62.7
Model selection: Mane--one-domiine cross-validation
Algorithm
CMNIST
RMNIST
VLCS
PACS
OfficeHome
TerraIne
DomainNet
Average
ERM
57.8 0.2
97.8 0.1
77.6 0.3
86.7
0.3
66.4 0.5
53.0 0.3
41.3 0.1
68.7
IRM
67.7
97.5 0.2
=6.9
84.5
1.1
63.0
50.5 0.7
28.0 5.1
66.9
GroupDRO
6..1
9..9
7.5
87.1
0.1
66.2 0.6
5244 0.1
33.4 1.4+0. 0.3
67.9
Mixup
5..4+002
9..0
78.1-0.3
86.8
0.3
68.0
54.4 54.4 0.3
39.6
69.0
MLDG
0.4
77.5 77.5-0
86.8
0.3
1..0
CORAL
5..6
98.0
77.7=022 0.2
87.1 0.5
68.4 0.2
5.2
4.18
69.2
MMD
1.3
9..0
=
87.2
66.2
5.4
2..5
66.9
DANN
$7.0
97.9
0.5
85.2
0.2
65.3 0.8
50.6
3.1
67.7
CDANN
59.5
9..9
0.2
85.8
0.8
65.3 0.5
50.8
38.5
0.3
0.1
86.7
0.2
66.5 0.4
522+
4.18
SagNel
0.3
97.9
1 0.0
77.6 0.1
86.4
0.4
67.5 0.2
52.5 0.4
40.8
10.8 0.2
68.7
ARM
6.72
98.1 0.1
77.8
0.3
85.8
0.2
64.8 0.4
512
36.0 0.2
68.1
VREX
67.0
07.90.1
78.1 0.2
87.2
0.6
65.7 0.3
514
3..1
68.2
58.5
97.6 0.1
77.8 0.6
86.2
0.5
66.5 0.6
52.1 0.2
38.9
68.2
Model selection: test-domain validation set (oracle)

---

Published as a conference paper at ICLR 2021
Table 4: Ablation study on ERM showing the impact of (i) using raw images versus data augmentation, and (ii) using ResNet-18 versus ResNet-50 models. Model selection: Mominng-domain validation set.
Algorithm
VLCS
PACS
OfficeHome
TerraInc
Domain
Avg
ERM (raw, 18)
75.8 0.3
79.6 0.3
61.0
35.0 1.3
35.8 0.2
62.4
ERM (aug. 18)
75.8 + 0.1
79.1 + 0.8
60.0 0.6
40.0 0.6
35.3
62.8
ERM (raw, 50)
78.6 + 0.1
83.2 + 0.6
67.7 0.2
41.5 4
41.4 +.1
66.0
ERM (aug. 50)
77.5
1 0.4
85.5 1 0.2
66.5 0.3
46.1 1.8
40.9 0.1
66.6
5.2 ABLATION STUDY ON ERM
To better understand our ERM performance, we perform an ablation study on the neural network architecture and the data augmentation protocol. Table 5.2 shows that using ResNet-50 neural network architecture, instead of smaller 18, improves DG test accuracy by 3.7 points. Using data augmentation improves DG test accuracy by 0.5 points. However, these ResNet models were pretrained on ImageNet using data augmentation, so the benefits of augmentation are partly absorbed by the model. In fact, we hypothesize that among models pretrained on ImageNet, domain generalization performance is mainly influenced by the model's original test accuracy on ImageNet.
DISCUSSIONS
We provide several discussions help the reader interpret our results and motivate future work.
Our negative claims are fundamentally limited Broad negative claims (e.g. "algorithm X does not outperform ERM") do not specify an exact experimental setting and are therefore impossible to rigorously prove. In order to be verifiable, such claims must be restricted to specific setting. This limitation is fundamental to all negative result claims, and ours (Claim 2) is no exception. We have shown that many algorithms don't substantially improve on ERM in our setting, but the relevance of that setting is a subjective matter ultimately left for the reader.
In making this judgement, the reader should consider whether they agree with our methodological and implementation choices, which we have explained and motivated throughout the paper. We also note that our implementation can outperform previous results (Table 1). Finally, DomainBed is not a black box: our implementation is open-source and actively maintained, and we invite the research community to improve on our results.
Is this as good as it gets? We question whether DG is possible in some of the considered datasets. Why do we assume that neural network should be able to classify cartoons, given only photorealistic training data? In the case of Rotated MNIST, do truly mmation-nnvarian features discriminative of the digit class exist? Are those features expressible by a neural network? Even in the presence of correct model selection, is the outto--distribution performance of modern ERM implementations as good as it gets? Or is it simply as poor as every other alternative? How far are we from the achievable DG performance? Is this upper-bound simply the test error in-domain?
Are these the right datasets? Most datasets considered the DG literature not reflect realistic situations. If one wanted to classify cartoons, the easiest option would be to collect small labeled dataset of cartoons. Should we consider more realistic, impactful tasks for better research in DG? Some alternatives are medical imaging in different hospitals and self self-driving cars in different cities.
Generalization requires untestable assumptions Every time we use ERM, we assume that train- ing and testing examples follow the same distribution. This an untestable assumption in every single instance. The same applies for DG: each algorithm assumes different (untestable) type of invariance across domains. Therefore, the performance of a DG algorithm depends on the problem at hand, and only time tell if we have made a good choice. This is akin to the generalization of a scientific theory such as Newton's 's gravitation, which cannot be proven, but rather only resist falsification.
8

---

Published as a conference paper at ICLR 2021
7 CONCLUSION
Our extensive empirical evaluation of DG algorithms leads to three conclusions. First, a carefully tuned ERM baseline outperforms the previously published state-of-the-art results in terms of average performance (Claim 1). Second, when compared to thirteen popular DG alternatives on the exact same experimental conditions, we find out that no competitor able to outperform ERM by more than one point (Claim 2). Third, model selection is non- con-trivial for DG, and it should be an integral part of any proposed method (Claim 3). Going forward, we hope that our results and DOMAINBRR promote realistic and rigorous evaluation and enable advances in domain generalization.
REFERENCES
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. arXiv, 2020.
Kei Akuzawa, Yusuke Iwasawa, and Yutaka Matsuo. Adversarial invariant feature learning with accuracy constraint for domain generalization. arXiv, 2019.
Ehab A AlBadawy, Ashirbani Saha, and Maciej A Mazurowski. Deep learning for segmentation of brain tumors: Impact of cross- inssiiinsitttionaal training and testing. Medical physics, 2018.
Isabela Albuquerque, João Monteiro, Tiago H Falk, and Ioannis Mitliagkas. Adversarial target-
invariant representation learning for domain generalization. arXiv, 2019.
Isabela Albuquerque, Nikhil Naik, Junnan Li, Nitish Keskar, and Richard Socher. Improving out-of-distribution generalization via multi-task self-supervised pretraining. arXivv 2020.
Michael A Alcom, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. CVPR, 2019.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint 1907v 2019.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. MetaReg: Towards domain general- ization using meta-regullrivatton. NIPS, 2018.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. ECCV, 2018.
Shai BendDavid, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 2010.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. JMLR, 2012.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. NIPS, 2011.
Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. arXiv, 2017.
Victor Bouvier, Philippe Very, Céline Hudelot, and Clément Chastagnol. Hidden covariate shift: A minimal assumption for domain adaptation. arXiv, 2019.
Fabio M Carlucci, Antonio D' D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. CVPR, 2019a.
Fabio Maria Carlucci, Paolo Russo, Tatiana Tommasi, and Barbara Caputo. Hallucinating agnostic images to generalize across domains. ICCVW, 2019b.
Daniel C Castro, Ian Walker, and Ben Glocker. Causality matters medical imaging. arXiv, 2019.
Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman. Learning to balance specificity and invariance for in and out of domain generalization, 2020.

---

Published as a conference paper at ICLR 2021
Dengxin Dai and Lue Van Gool. Dark model adaptation: Semantic image segmentation from daytime to nighttime. ITSC, 2018.
Aniket Anand Deshmukh, Yunwen Lei, Srinagesh Sharma, Urun Dogan, James W Cutler, and Clayton Scott. generalization error bound for multi-class domain generalization. arXiv, 2019.
Zhengming Ding and Yun Fu. Deep domain generalization with structured low-rank constraint. /EEE Trunsactions on Image Processing, 2017.
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via agnes-ag learning of semantic features. NIPS, 2019.
Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. The reusable holdout: Preserving validity in adaptive data analysis. Science, 2015.
Antonio D' D'Innocente and Barbara Caputo. Domain generalization with domain-specific aggregation modules. German Conference on Pattern Recognition, 2018.
Chen Fang. Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. ICCV, 2013.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ICML, 2017.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR, 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv, 2018.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. ICCV, 2015.
Muhammad Ghifary, David Balduzzi, W Bastiaan Kleijn, and Mengjie Zhang. Scatter component analysis: A unified framework for domain adaptation and domain generalization. IEEE TPAMI, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, 2014.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. JMLR, 2012.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Will Heaven. Google's medical Al was super accurate in a lab. real life was a different story. MIT Technology Review, 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis of -uitt--diittibution generalization, 2020.
Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multidomain discriminant analysis. UAI, 2019.
Weihua Hu, Gang NIu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? arXiv, 2016.
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. arXiv preprint 2XXi:2007.022544 2020.
10

---

Published as a conference paper at ICLR 2021
Maximilian llse, Jakub M Tomezak, Christos Louizos, and Max Welling. DIVA: Domain invariant variational autoencoders. arXiv preprint arXiv: 1XX1::9905..0427,, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv, 2015.
Fredrik D Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domain- invariant representations. arXiv, 2019.
Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei Efros, and Antonio Torralba. Undoing the damage of dataset bias. ECCV, 2012.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
David Krueger, Ethan Caballero, Joem-HHenrk Jacobsen, Amy Zhang, Jonathan Binas, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (REx). arXiv, 2020.
Yann LeCun. The mnist database of handwritten digits. http://yan.. lecun. com/exdb/mmist/, 1998.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. AAAI, 2018a.
Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. ICCV, 2019a.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Sequential learning for domain generalization. arXiv, 2020.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. CVPR. 2018b.
Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. AAAI, 2018c.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao.
Deep domain generalization via conditional invariant adversarial networks. ECCV, 2018d.
Yiying Li. Yongxin Yang, Wei Zhou, and Timothy M Hospedales. Feature-critic networks for heterogeneous domain generalization. arXiv, 2019b.
Massimiliano Mancini, Samuel Rota Bulò, Barbara Caputo, and Elisa Ricci. Best sources forward: domain generalization through source-specific nets. ICIP, 2018a.
Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Robust place categoriza- tion with deep domain generalization. IEEE Robotics and Automation Letters, 2018b.
Toshihiko Matsuura and Tatsuya Harada. Domain generalization using mixture of multiple latent domains. arXiv, 2019.
Saeid Motiian, Marco Piccirilli, Donald Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. ICCV, 2017.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. ICML, 2013.
Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, Bernhard Schölkopf, et al. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends in Machine Learning, 2017.
Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap via style-agnostic networks. arXiv, 2019.
11

---

Published as a conference paper at ICLR 2021
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE TKDE, 2009.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. NIPS, 2019.
Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Visual domain adaptation: survey of recent advances. IEEE Signal Processing, 2015.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. ICCV, 2019.
Christian S Perone, Pedro Ballester, Rodrigo C Barros, and Julien Cohen-Adad. Unsupervised domain adaptation for medical imaging segmentation with seff-ensembling. Neurolmage, 2019.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2016.
Mohammad Mahfujur Rahman, Clinton Fookes, Mahsa Baktashmotlagh, and Sridha Sridharan. Correlation- aware adversarial domain adaptation and generalization. Pattern Recognition, 2019a.
Mohammad Mahfujur Rahman, Clinton Fookes, Mahsa Baktashmotlagh, and Sridha Sridharan. Multi-component image translation for deep domain generalization. WACV, 2019b.
Mateo Rojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. JMLR, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNET large scale visual recognition challenge. 1JCV, 2015.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv, 2019.
Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization, 2020.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross grasiett training. arXiv, 2018
Pierre Stock and Moustapha Cisse. Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases. ECCV, 2018.
Baochen Sun and Kate Saenko. Deep CORAL: Correlation alignment for deep domain adaptation. ECCV, 2016.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. AAAI, 2016.
Damien Teney, Ehsan Abbasnejad, and Anton van den Hengel. Unshuffling data for improved generalization. arxiv, 2020.
Antonio Torralba and Alexei Efros. Unbiased look at dataset bias. CVPR, 2011.
Vladimir Vapnik. Statistical learning theory wiley. New York, 1998.
Hemanth Venkateswara, Jose Fusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. CVPR, 2017.
Georg Volk, Stefan Müller, Alexander von Bernuth, Dennis Hospach, and Oliver Bringmann. Towards robust enn-based object detection through augmentation with synthetic rain variations. /TSC, 2019.
12

---

Published as a conference paper at ICLR 2021
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio
Savarese. Generalizing to unseen domains via adversarial data augmentation. NIPS, 2018.
Haohan Wang, Zexue He, Zachary C Lipton, and Eric P Xing. Learning robust representations by projecting superficial statistics out. arXiv,, 2019.
Shujun Wang, Lequan Yu, Caizi Li, Chi- Thi-Wing Fu, and Pheng-Ann Heng. Learning from extrinsic and intrinsic supervisions for domain generalization, 2020a.
Yufei Wang. Haoliang Li, and Alex C Kot. Heterogeneous domain generalization via domain mixup. ICASSP, 2020b.
Garrett Wilson and Diane J Cook. A survey of unsupervised deep domain adaptation. arXiv, 2018.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. arXiv, 2019.
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv,, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Paz. mixup: Beyond empirical risk minimization. ICLR, 2018.
Ling Zhang, Xiaosong Wang, Dong Yang. Thomas Sanford, Stephanie Harmon, Baris Turkbey, Holger Roth, Andriy Myronenko, Daguang Xu, and Ziyue Xu. When unseen domain generalization unnecessary? rethinking data augmentation. arXiv, 2019.
Marvin Zhang, Henrik Marklund, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: A meta-learning approach for tackling group shift. arXiv, 2020.
Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representation for domain adaptation. arXiv, 2019.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep domain-adversarte image generation for domain generalisation. arXiv preprint arXiv:2003.0605: 06055, 2020.
13

---

Published as a conference paper at ICLR 2021
A A DECADE OF LITERATURE ON DOMAIN GENERALIZATION
In this section, we provide an exhaustive literature review on a decade of domain generalization research. The following classifies domain generalization algorithms according into four strategies to learn invariant predictors: learning invariant features, sharing parameters, meta-learning, or performing data augmentation.
A.I LEARNING INVARIANT FEATURES
Muandet et al. (2013) use kernel methods to find feature transformation that (i) minimizes the distance between transformed feature distributions across domains, and (ii) does not destroy any of the information between the original features and the targets. In their pioneering work, Ganin et al. (2016) propose Domain Adversarial Neural Networks (DANN), domain adaptation technique which uses generative adversarial networks (GANs, Goodfellow et al. (2014)), to learn a feature representation that matches across training domains. Akuzawa et al. (2019) extend DANN by considering cases where there exists an statistical dependence between the domain and the class label variables. Albuquerque et al. (2019) extend DANN by considering one-versus-all adversaries that try to predict to which training domain does each of the examples belong to. Li et al. (2018b) employ GANs and the maximum mean discrepancy criteria (Gretton al., 2012) to align feature distributions across domains. Matsuura and Harada (2019) leverages clustering techniques to learn domain- invariant features even when the separation between training domains is not given. Li al. (2018c;d) learns a feature transformation & such that the conditional distributions ((o(X4) | y = y) match for all training domains and label values y. Shankar et al. (2018) use a domain classifier to construct adversarial examples for a label classifier, and use label classifier to construct adversarial examples for the domain classifier. This results in a label classifier with better domain generalization. Li et al. (2019a) train robust feature extractor and classifier. The robustness comes from (i) asking the feature extractor to produce features such that a classifier trained on domain d can classify instances for domain d' f d, and (ii) asking the classifier to predict labels on domain d using features produced by a feature extractor trained on domain d' * d. Li et al. (2020) adopt a lifelong learning strategy to attack the problem of domain generalization. Motiian et al. (2017) learn a feature representation such that (i) examples from different domains but the same class are close, (ii) examples from different domains and classes are far, and (iii) training examples can be correctly classified. Ilse et al. (2019) train a variational autoencoder (Kingma and Welling, 2014) where the bottleneck representation factorizes knowledge about domain, class label, and residual variations in the input space. Fang et al. (2013) learn structural SVM metric such that the neighborhood of each example contains examples from the same category and training domains. The algorithms of Sun and Saenko (2016); Sun et al. (2016); Rahman al. (2019a) match the feature covariance (second order statistics) across training domains at some level of representation. The algorithms of Ghifary et al. (2016); Hu et al. (2019) use kernel-based multivariate component analysis minimize the mismatch between training domains while maximizing class separability.
Although popular, learning domain-invariann features has received some criticism (Zhao et al., 2019; Johansson et al., 2019). Some alternatives exist, as we review next. Peters et al. (2016); Rojas-Carulla al. (2018) considered that one should search for features that lead the same optimal classifier across training domains. In their pioneering work, Peters et al. (2016) linked this type of invariance to the causal structure of data, and provided a basic algorithm to learn invariant linear models, based on feature selection. Arjovsky et al. (2019) extend the previous to general gradient-based models, including neural networks, in their Invariant Risk Minimization (IRM) principle. Teney et al. (2020) build on IRM to learn a feature transformation that minimizes the relative variance of classifier weights across training datasets. The authors apply their method to reduce the learning of spurious correlations in Visual Question Answering (VQA) tasks. Ahuja et al. (2020) analyze IRM under a game-theorett perspective to develop an alternative algorithm. Krueger et al. (2020) propose an approximation the IRM problem consisting in reducing the variance of error averages across domains. Bouvier et al. (2019) attack the same problem as IRM by re- re-weighting data samples.
A.2 SHARING PARAMETERS
Blanchard et al. (2011) build classifiers f(r", "aal where jrl is a kernel mean embedding (Muandet et al., 2017) that summarizes the dataset associated to the example ad Since the distributional
14

---

Published as a conference paper at ICLR 2021
identity of test instances is unknown, these embeddings are estimated using single test examples at test time. See Blanchard et al. (2017); Deshmukh et al. (2019) for theoretical results on this family of algorithms (only applicable when using RKHS-based learners). Zhang et al. (2020) is extension of Blanchard et al. (2011) 1) where a separate CNN computes the domain embedding, appended to the input image as additional channels. Khosla al. (2012) learn one max-margin linear classifier wd = w + A per domain d. from which they distill their final, invariant predictor wc Ghifary et al. (2015) use multitask autoencoder learn invariances across domains. To achieve this, the authors assume that each training dataset contains the same examples; for instance, photographs about the same objects under different views. Mancini al. (2018b) train a deep neural network with one set of dedicated moach-oormllization layers (loffe and Szegedy, 2015) per training dataset. Then, a softmax domain classifier predicts how to linearly-combine the batch-normalization layers test time. Seo et al. (2020) combines instance normalization with batch-normalization to learn normalization module per domain, enhancing diito--dittiiuttion generalization. Similarly, Mancini et al. (2018a) learn softmax domain classifier used to linearly-combing domain-specine predictors at test time. D' Innocente and Caputo (2018) explore more sophisticated ways of aggregating domain-specific predictors. Li et al. (2017) extends Khosla et al. (2012) to deep neural networks by extending each of their parameter tensors with one additional dimension, indexed by the training domains, and set to a neutral value predict domain-agnostic test examples. Ding and Fu (2017) implement parameter- tying and low-rank reconstruction losses to learn predictor that relies on common knowledge across training domains. Hu al. (2016); Sagawa et al. (2019) weight the importance of the minibatches of the training distributions proportional to their error. Chattopadhyay et al. (2020) overlays multiple weight masks over a single network learn domain-invariant and domain-specific features.
A.3 META-LEARNING
Li et al. (2018a) employ Model-Agnostee Meta-Learning or MAML (Finn et al., 2017), to build predictor that learns how to adapt fast between training domains. Dou et al. (2019) use a similar MAML strategy, together with two regularizers that encourage features from different domains to respect inter-class relationships, and be compactly clustered by class labels. Li et al. (2019b) extend the MAML meta-learning strategy to instances of domain generalization where the categories vary from domain to domain. Balaji et al. (2018) use MAML to meta-learn a regularizer encouraging the model trained on one domain to perform well on another domain.
A.4 AUGMENTING DATA
Data augmentation is an effective strategy to address domain generalization (Zhang et al., 2019). Unfortunately, how to design efficient data augmentation routines depends on the type of data at hand, and demands significant amount of work from human experts. Xu et al. (2019); Yan et al. (2020); Wang et al. (2020b) use mixup (Zhang et al., 2018) to blend examples from the different training distributions. Carlucci et al. (2019a) constructs auxiliary classification task aimed at solving jigsaw puzzles of image patches. The authors show that this self-supervised learning task learns features that improve domain generalization. Similarly, Wang et al. (2020a) use metric learning and self sef-ssupeeiied learning to augment the out-of -iitr--iistiitution performance of an image classifier. Albuquerque et al. (2020) introduce the self-supervised task of predicting responses to Gabor filter banks, in order to learn more transferrable features. Wang et al. (2019) remove textural information from images to improve domain generalization. Volpi et al. (2018) show that training with adversarial data augmentation on single domain is sufficient to improve domain generalization. Nam et al. (2019) promote representations of data that ignore image style and focus on content. Rahman et al. (2019b); Zhou et al. (2020); Carlucci et al. (2019a) are three alternatives that use GANs to augment the data available during training time. Representation Self-Challenging (Huang et al., 2020) learns robust neural networks by iteratively dropping-out important features. Hendrycks et al. (2020) show that, together with larger models and data, data augmentation improves oft-----dst performance.
15

---

Published as a conference paper at ICLR 2021
A.5 PREVIOUS
NUMBERS
Table 5 compiles the best out-of-distribution test accuracies reported across a decade of domain generalization research.
Table 5: Previous ofate-of-the-ar in the literature of domain generalization.
Benchmark
Accuracy (by domain)
Algorithm
15
75
Average
82.50
96.30
93.40
78.60
94,20
80.50
87.58
D-MTAE (Ghifary et al., 2015)
94.60
(Motiaan al., 2017)
83.70
96.90
95.70
85.20
95.90
81.20
89.80
MMD-AAE (Li al., 2018b)
85.60
95.00
95.60
95.50
95.90
84.30
92.00
BestSources (Mancini et al., 2018a)
Rotated
MNIST
88.80
60
97.50
97.80
97.60
91.90
95.20
ADAGE (Carlucci
2019h)
88,30
98.00 60
98.00
97.70
97.70
91.40
95.28
CrossGrad (Shankar ot al., 2018)
90.00
HEX (Wang al., 2019)
89.23
99. 68
99.20
99.24
99.53
91.44
96.39
FeatureCritic (L. et al., 2019b)
93.50
99.30
DIVA et 2019)
95,90
98.90
98.80
98.90
98.90
96.40
98.00
Our ERM
C
L
S
Average
59.20
(Ghifary al., 2016)
92.30
62. 10
59. 10
67.10
65.00
CCSA (Motiian al., 2017)
89.15
64.99
58.88
62.59
67.67
MTSSL (Albuquerque al., 2020)
89.05
60.13
61.33
63,90
68.60
D-MTAE (Ghifary et al., 2015)
91.12
60.43
60.85
65.65
69.41
CIDG (Li 2018c)
88.83
63.06
62, 10
64.38
69.59
CIDDG cL al., 20188)
92.64
61.78
59, 60
66.86
70.22
MDA (Hu et al., 2019)
62.34
(Ding and 2017)
93.63
63.49
61.32
69.99
72.11
DBADG (Li el al., 2017)
VLCCS
62.60
72.30
MMD-AAF
2018h)
94,10
64.30
65,90
67.10
72.90
Ep-
al., 2019p)
96.93
60,90
64.30
70.62
73.19
JiGen (Carlncci et al., 2019s)
96.72
60,40
63 68
70.49
73.30
VREx (Kreeger al., 2020)
96.40
64,80
64.00
68.70
73.50
S-MLDG (Li ct al., 2020)
58.77
(Matsaura
Harada, 2019)
94.78
64,90
67. 64
69.14
74.11
MASF (Dou et al., 2019)
74.67
EISNet (Wang al., 2020n)
97.61
61.86
68.32
73.93
75.43
RSC (Huang al., 2020)
95.52
67.63
69.37
71.14
75.92
G2DM (Albuquerque et al., 2019)
97.70
64.30
73.40
74.60
77.50
Our ERM
C
P
Average
62.86
66.97
89.50
57.51
69.21
DBADG al.. 2017)
61.67
67.41
84.31
63,91
69.32
MISSL (Albuquerque al., 2020)
62.70
69.73
78.65
64.45
69.40
CIDDG (1i al., 2018d)
62.64
65.98
90.44
58.76
69.45
JAN-COMBO (Rahman al., 2019h)
66.23
66.88
88.00
$8.96
70.01 MLDG (Li ct al., 2018a)
66.80
69.70
HEX (Wang al., 2019)
64.10
66.80
90.20
60.
70.30
BestSources (Mancini et al., 201
FeatureCritic
2019h)
67.04
67.97
89.74
59.81
71.14
VREx (Kraeger al., 2020)
65.52
69.90
89. 16
63.37
71.98
CAADG (Rahman et 2019a)
64,70
72.30
86,
65.00
72.00
Epi-FCR
2019a)
66.60
73,36
88. 12
66.19
73.55
G2DM (Alboquerque et al., 2019)
PACS
72.46
90.68
(Dou 20,,
79.42
75.25
96.03
71.35
80.51
JiGen (Carlucci al., 201
80.50
94.80
72,80
S-MLDG
al., 2020)
79.48
13
94.30
75.30
81.55
D-SAM-A (D Innocente and Caputo, 2018)
81.28
77.16
96.09
72.29
81.83
MMLD (Matsuura and Harada, 2019)
84,20
78.
9:5.30
74.70
83.10
DDAIG (Zhou al., 2020)
83.58
77. .6
95.47
76.30
83.25
SagNets (Nam et al., 2019)
82.57
78.11
94.49
78.32
83.37
DMG (Chattopadhyay al., 2020)
87.20
79.20
97.
70.30
83.60
MetaReg (Balaji et al., 2018)
86.64
81.53
97. 11
78.07
85.84
FISNet (Wang et al., 2020s)
87.04
80.62
95.99
82.90
et al., 2020)
87.89
82.16
97.92
83.35
87.83
RSC (Huang al., 2020)
A
C
P
Average
66.52
JAN-COMBO (Rahman et al., 2019h)
53.04
47.51
71.47
72.79
61.20
JiGen (Carlucci et al., 2019a)
54.53
49.04
71.57
71.90
61.76
D-SAM-A (D' Innocente and Caputo, 2018)
Office
60.20
45.38
70.42
73.38
62.34
SagNets (Nam et al., 2019)
Home
59.37
45.70
71.84
74.68
62.90
DSON (Sco et al., 2020)
(Huang 2020) 2020)
71.63
59.20
$2.30
74.60
76.00
65.50
DDAIG (7hou et al., 2020)
61.30
52.40
75.80
76.60
ERM
16

---

Published as a conference paper at ICLR 2021
B MORE ABOUT DOMAINBED
B.1
ALGORITHMS
1. Empirical Risk Minimization (ERM, Vapnik (1998)) minimizes the errors across domains.
2. Group Distributionally Robust Optimization (DRO, Sagawa et al. (2019)) performs ERM while increasing the importance of domains with larger errors.
3. Inter-domain Mixup (Mixup. Xu et al. (2019); Yan et al. (2020); Wang et al. (2020b)) performs ERM on linear interpolations of examples from random pairs of domains and their labels.
4. Meta-Learning for Domain Generalization (MLDG, Li et al. (2018a)) leverages MAML (Finn et al., 2017) to meta-learn how to generalize across domains,
5. Domain-Adversnn Adversarial Neural Networks (DANN, Ganin et al. (2016)) employ adversarial net- work to match feature distributions across environments.
6. Class-conditional DANN (C-DANN, Li et al. (2018d)) is variant of DANN matching the conditional distributions P()(X)) = y) across domains, for all labels y.
7. CORAL (Sun and Saenko, 2016) matches the mean and covariance of feature distributions.
8. MMD (Li et al., 2018b) matches the MMD (Gretton et al., 2012) of feature distributions.
9. Invariant Risk Minimization (IRM Arjovsky et al. (2019)) learns a feature representation ((X")
such that the optimal linear classifier on top of that representation matches across domains.
10. Risk Extrapolation (VREx, Krueger et al. (2020)) approximates IRM with a variance penalty.
11. Marginal Transfer Learning (MTL, Blanchard et al. (2011; 2017)) estimates mean embedding per domain, passed as a second argument to the classifier.
12. Adaptive Risk Min. (ARM, Zhang al. (2020)) extends MTL with separate embedding CNN.
13. Style-Agnostic Networks (SagNets, Nam et al. (2019)) learns neural networks by keeping image content and randomizing style.
14. Representation Self Seff-Challenging (RSC, Huang et al. (2020)) learns robust neural networks by iteratively discarding (challenging) the most activated features.
B.2 DATASETS
DOMAINBED includes downloaders and loaders for seven multi-domain image classification tasks:
1. Colored MNIST (Arjovsky al., 2019) is variant of the MNIST handwritten digit classification dataset (LeCun, 1998). Domain d E {0.1, 0.3, 0.9} contains a disjoint set of digits colored either red or blue. The label is a noisy function of the digit and color, such that color bears correlation d with the label and the digit bears correlation 0.75 with the label. This dataset contains 70, 000 examples of dimension (2, 28, 28) and 2 classes.
2. Rotated MNIST (Ghifary et al., 2015) is variant of MNIST where domain d E { 0. 15, 30, 45, 60, 75 contains digits rotated by degrees. Our dataset contains 70, 000 examples of dimension (1, 28, 28) and 10 classes.
3. PACS (Li et al., 2017) comprises four domains d E { art, cartoons, photos, sketches ). This dataset contains 991 examples of dimension (3, 224, 224) and 7 classes.
4. VLCS (Fang et al., 2013) comprises photographic domains d Caltech LabelMe, SUN09, VOC2007 }. This dataset contains 10, 729 examples of dimension (3, 224, 224) and classes.
5. OfficeHome (Venkateswara et al., 2017) includes domains d art, clipart, product, real }. This dataset contains 15, 588 examples of dimension (3, 224, 224) and 65 classes.
6. Terra Incognita (Beery et al., 2018) contains photographs of wild animals taken by camera traps at locations d E [00 L38, L43, L46). Our version of this dataset contains 24, 788 examples of dimension (3, 224, 224) and 10 classes.
7. DomainNet (Peng et al., 2019) has six domains d E { clipart, infograph, painting, quickdraw, real, sketch This dataset contains 586, 575 examples of size (3, 224, 224) and 345 classes.
For all datasets, we first pool the raw training, validation, and testing images together. For each random seed, then instantiate random training, validation, and testing splits.
17

---

Published as a conference paper at ICLR 2021
B.3 MODEL SELECTION CRITERIA, ILLUSTRATED
Consider Figure 1, and let T; - {A,, B;, Cij for i & {1, 2). Training-domain validation trains each hyperparameter configuration
Ti and chooses the configuration with the highest performance in Tg. Leave-one-oud validation trains one clone Fz of each hyperparameter configuration on 7i Z, for Z E Ti: then, it chooses the configuration with highest Σ ZET, Performance (Fz, Z). Test-domain validation trains each hyperparameter configuration on I`i and chooses the configuration with the highest performance on D2, only looking at its final epoch. Finally, result tables show the performance of selected models on D1.
training domains
test domain
big split
A
B
C
D
small split
A2
B2
C2
D2
domain A
domain B
domain
domain D
Figure 1: Data configuration for benchmark with four domains A, B, C, D, where the test domain is D. We shuffle and divide the data from each domain into a big split and a small split.
B.4 ARCHITECTURES AND HYPERPARAMETERS
Neural network architectures used for each dataset:
Dataset
Architecture
Colored MNIST
MNIST ConvNet
Rotated MNIST
PACS
VLCS
OfficeHome
ResNet-50
TerraIncognita
Neural network architecture for MNIST experiments:
Layer
Conv2D (in=d, out=64)
ReLU
3
GroupNorm (groups=8)
4
Conv2D (in=64, out=128, stride=2)
5
ReLU
GroupNorm (8 groups)
7
Conv2D (in=128, out=128)
ReLU
9
GroupNorm (8 groups)
10
Couv2D (in=128, out=
11
ReLU
12
GroupNorm (8 groups)
13
Global average- average-pooling
For "ResNet-50". we replace the final (softmax) layer of ResNet50 pretrained ImageNet and fine-tune the entire network. Since minibatches from different domains follow different distributions, batch normalization degrades domain generalization algorithms (Seo al., 2020). Therefore, freeze all batch normalization layers before fine-tuning. We insert dropout layer before the final ResNet-50 linear layer.
18

---

Published as a conference paper at ICLR 2021
Table lists all algorithm hyperparameters, their default values, and their sweep random search distribution. We optimize all models using Adam (Kingma and Ba, 2015).
Table 6: Hyperparameters, their default values and distributions for random search.
Condition
Parameter
Default value
Random distribution
learning rate
0.00005
10Unfom((5,-335) Uniform(3.6.6)
batch size batch size (if ARM)
32
ResNet
ResNet dropout
0
RandomChoice (0, 0.1, 0.5))
generator learning rate
0.00005
Uniform! -5, -5.33.5)
discriminator learning rate
0.00005
10 Uniiomm (5..3.55)
learning rate batch size generator learning rate
0.001
10 Uniformm! -4.3,
64
2niform((.9)
not ResNet
0.001
10 1
-4.3,
discriminator learning rate
0.001
10 Uniform( -4.5, 2.5)
weight decay
0
MNIST
generator weight decay
weight decay
0
Umiioomm (.00iaammm6661)))
not MNIST
generator weight decay
0
10 -6)
lambda
1.0
10 -2.2)
discriminator weight decay
[
discriminator steps
2niforr((03)
DANN, C-DANN
discriminator width
256
int (nt(2uittem((6110)
discriminator depth discriminator dropout
3
RandomChoice [3, 5])
RandomChoice 0.1, 0.5))
discriminator grad penalty Adam 31
10 -211)
0.5
RandomChoice 0.5])
DRO
0.01
Unifommm(-1 1.1)
lambda warmup iterations
100
10 1n -1,5)
IRM
500
10 (0,)
Mixup
alpha
0.2
Unifomm((0
MLDG
beta
10 10Unifom((11.1
MMD
gamma
10 1,11
MTL
ema
0.99
RandomChoice((.5, 99..9999 1])
feature drop percentage batch drop percentage
1/3
Uniform( (0, 0.5)
RSC
1/3
Uniform 0.5)
SagNet
adversary weight
0.1
Uninoomm( 221)
lambda
10
10 Unitomm (- 1,5)
VREx
warmup iterations
500
10 (0
19

---

Published as a conference paper at ICLR 2021
B.5 EXTENDING DOMAINBED
Algorithms are classes that implement two methods: uudate (mi mniinibatchen and -predict (x). The update method receives a list of minibatches, one minibatch per training domain, and each minibatch containing one input and one output tensor. For example, to implement group DRO (Sagawa al., 2019, Algorithm 1), we simply write the following in algori thms, py:
class GroupDRO (ERM)
def init_ (self, imput_saape num_classes, num domains, hparams):
super (0__nnnttt.... wwwptttntttaaaeeeeee shape, nam clases,, num_domains, hparama) rolf.rogister__bufee ("q", torch. Tensor
updato (self, minibatches):
device "euda" if minibatches [0].ss__uud iss_cda elae "epu"
(sellfgg))
self.g torch, (len (minibatchess) to (device)
losses torch. (len (minibatches) (device)
for m in range (T (eiiniittthhss))))
x, y minibstches
losses [m] - crossssentrrpy (self predict (x),
self.g(m] - sself..parame ["dro_eta"] losses [m] deeta) .exp
self.q /= felf.q.suml)
lass torch. dot flosses, self.g) len ininibbtchhes)
self.onting ser. zerooragg)d lose bbackward(1 self.optimi ste.step()
return ("loss': 1oss. beas.item))
ALGORITHMS. apppodd GroupDRO')
By inheriting from ERM, the new GroupDRO class has access to a default classifier .ntwwrk,, op- timizer opt imi zer, and prediction method .predict (x) Finally, we tell DOMAINBED about the default values and hyperparameter search distributions of the hyperparameters of this new algo-
rithm. We do so by adding following to the function .hparams in wwwaarassreeistteeeeeeesess
hparama (' ['d__eta") (1e-2, 2, 10*+randem_statt uni conffommt (-3, -1)>
To add a new image classification dataset to DOMAINBED, arrange your image files as / /root//Myl heypDaaaset/domaaae http://caasssmmmaaaa jpg. Then, append to datasets. py:
class MyDataset (Mult wow.::vwiaammmmttnaaemaeeeeee Iowgeenninngeeeeeeerrr.. ENVIRONMENTS - (' snvi', 'Eav2', 'Env3']
def init (self, root, teat_envs, augmeat-Tzue):
self.dir as pa.pa poi (root, "MyDataaet/")
super () _inn._nn_____ iniii_i__ss (sel f.dir, test_envs, augment)
TErASETS.appenne "yyataset'')
We are now ready to train our new algorithm on our new dataset, using the second domain as test:
python train.py --model DRO --dataset MyDataset --data_dir /root --test_enva
--hparams "["dro_eta":_D.
Finally, we can a fully automated sweep all datasets, algorithms, test domains, and model selection criteria by simply invoking python sweep Py, after extending the file conmand.launcher py to your computing infrastructure. When the sweep finishes, the script collect results. py automatically generates all the result tables shown in this manuscript.
Extension to UDA One can use DOMAINBED to perform experimentation on unsupervised domain adaptation by extending the updatee (minibatches) methods to accept unlabeled examples from the test domain.
20

---

Published as a conference paper at ICLR 2021
C FULL DOMAINBED RESULTS
C.I MODEL SELECTION: TRAINING-DOMAIIN VALIDATION SET
C.I.1 COLOREDMNIST
Algorithm
+90%
+80%
-90%
Avg
ERM
71.7 + 0.1
72.9 + 0.2
10.0
0.1
51.5
IRM
72.5 + 0.1
0.5
10.2 0.3
52.0
GroupDRO
73.1 0.3
10.0 0.2
52.1
Mixup
+.4
0.1
1011
52.1
MLDG
0.2
7.2
9.8
51.5
CORAL
71.6
7.1
9.
51.5
MMD
0.3
7..1
9.9
51.5
DANN
0.9
0.1
10.0 0.0
51.5
CDANN
0.2
0.2
102
51.7
MTL
70,9
0.3
10.5 0.1
51.4
SagNet
0.2
7.20
1.0
51.7
ARM
0.5
0.3
10.2
56.2
VREx
1..4
72.9
+ 0.4
10.2
0.0
51.8
RSC
71.9 1 0.3
73.1 + 0.2
0.2
10.0
0.2
51.7
C.1.2 ROTATEDMNIST
Algorithm
0
15
30
45
60
75
Avg
ERM
95.9 + 0.1
98.9 + 0.0
98.8 + 0.0
98.9 + 0.0
98.9 + 0.0
96.4 0.0
98.0
IRM
95.5 0.1
98.8
98.7 + 0.1
98.6
9.0
9..
97.7
GroupDRO
95.6 0.1
0.1
98.9
99.0
98.9
96.5
98.0
Mixup
95.8 0.3
9.0
0.0
0.0
9..
9..
98.0
MILG
95.8 0.1
9.1
9..0
9..
99.0
0.3
97.9
CORAL
95.8 0.3
98.8
0.0
99.0 0.0
9.1
9..4
98.0
MMD
95.6 0.1
98.9
99.0+0.0
99.0 + 0.0
98.9
96.0 0.2
97.9
DANN
95.0 0.5
9..
9.0
99.0 0.1
0.0
9.2
97.8
CDANN
95.7 0.2
98.8
9..9
98.9 0.1
9.1
96.1 0.3
97.9
MTL
95.6 0.1
9..0
9.0
98.9 0.1
99.0
95.8
97.9
SagNet
95.9 0.3
9.1
9.1
99.1 0.0
99.0
9.
98.0
ARM
96.7 - 0.2
99.1
9.0
99,0
9..1
9.5
98.2
VREx
95.9 0.2
99.0
98.9
98.9 = 0.1
98.7
9..2
97.9
RSC
94.8 0.5
98.7 0.1
98.8 0.1 0.1
98.8 + 0.0
9.1
95.9 0.2
97.6
C.1.3 VLCS
Algorithm
C
L
S
Avg
ERM
97.7 + 0.4
64.3
0.5
74.6 + 1.3
77.5
IRM
98.6 0.1
6.9
7.64+
77.3 0.9
78.5
GroupDRO
9..3
6.9
6.55
7.7
76.7
Mixup
9.6
1.0
7.5
7.8
77.4
MLDG
9.24
6.72
71.0
7.
77.2
CORAL
0.1
6..1
73.4003
77.5
78.8
MMD
97.7+
64.0
72.8
7.3
77.5
DANN
9.3
1.4
7.31+
77.2
78.6
CDANN
0.3
65.1 15.
70.7
77.1
77.5
MTL
0.4
0.3
71.5
7.7
77.2
SagNet
97.9
0.5
7.
0.5
77.8
ARM
0.2
63.6
71.3
7..7
77.6
VREx
9.
1.44
74.1
7.
78.3
RSC
9.19
62.5 + 0.7
72.3
15.6..8.
77.1
21

---

Published as a conference paper at ICLR 2021
C.1.4 PACS
Algorithm
P
Avg
ERM
84.7 + 0.4
80.8 0.6
9.
79.3 + 1.0
85.5
IRM
84.8 1.3
1.14
96.7
76.1 1.0
83.5
GroupDRO
83.5 0.9
0.6
96.7
7 2.0
84.4
Mixup
86.1 06.1
0.8
97.6
75.8
84.6
MLDG
1.4
8.7
97.4
7.
84.9
CORAL
8
80.0
9.3
1.
86.2
MMD
16.11.4
0,9
9..
7.55
84.6
DANN
8.84
0.8
9..3
2.3
83.6
CDANN
84.6 1.8
0.9
9.
0.6
82.6
MTL
0.8
77.1 + 0.5
96.4-.88
+.8
84.6
SagNet
1.0
80.7 0.6
9..1
80.0 0.4
86.3
ARM
86.8 + 0.6
76.8 + 0.5
9.
79.3 1.2
85.1
VREx
86.0
79.1 = 0.6
96.9
77.7
1.7
84.9
RSC
85.4 0.8
79.7 = 1.8
9.
78.2 1 1.2
85.2
C.1.5
OFFICEHOME
Algorithm
A
C
P
Avg
ERM
61.3 + 0.7
5.34
7.1
76.6 + 0.3
66.5
IRM
58.9
1.6
7.91
2.5
64.3
GroupDRO
0.7
1.0
7.70
76.0
66.0
Mixup
+.84
0.6
76.9
0.2
68.1
MLDG
0.9
0.6
1.20
7.4
66.8
CORAL
6.4
0.5
0.1
7.5
68.7
MMD
0.2
0.3
7.13
7..4
66.3
DANN
59.9
=3.
7..6
0.5
65.9
CDANN
1.4
2.4
7.
7.86
65.8
MTL
0.7
52.4
7.
76.8 0.4
66.4
SagNet
6.2
0.4
7..8
78.3 0.3
68.1
ARM
5.89
0.5
7.11
75.2 0.3
64.8
VREx
60.7
0.9
75.3
76.6
66.4
RSC
60.7
51.4 0.3
74.8
75.1 + 1.3
65.5
C.1.6 IERRAINCOGNITA
Algorithm
L100
L38
L43
L46
Avg
ERM
49.8 + 4.4
42.1 + 1.4
56.9
35.7 + 3.9
46.1
IRM
54.6 1.3
39.8 1.9
56.2
3.8
47.6
GroupDRO
41.2
5..7 2.1
2.1
43.2
Mixup
2.0
1.4
5.89
33.9
47.9
MLDG
3.0
4.1
0.3
36.9
47.7
CORAL
2.4
4.0
1.0
39.8
47.6
MMD
41.9
34.8
1..0
35.2+11 1.8
42.2
DANN
+1.1
0.6
0.54
37.7 1.8
46.7
CDANN
4.90
4.8
1.79
39.8 +
45.8
MTL
1.2
6.3
1.1
37.8 18 0.8
45.6
SagNet
53.0
2.5
5.69
48.6
ARM
0.7
38.3 2.4
5..8
38.7 1.3
45.5
VREx
4..
41.7 - 1.3
56.8
38.7
46.4
RSC
50.2 2 2.2
39.2 + 1.4
36.3
40.8 + 0.6
46.6
22

---

Published as a conference paper at ICLR 2021
C.1.7 DOMAINNET
Algorithm
clip
info
paint
quick
sketch
Avg
ERM
58.1 + 0.3
18.8 + 0.3
46.7 0.3
12.2 1 0.4
59.6
49.8 0.4
40.9
IRM
48.5 2.8
15.0
38.3
10.9 0.5
48.2
42.3 3.1
33.9
GroupDRO
47.2
17.
0.5
0.3
0.
40.1
33.3
Mixup
55.7 0.3
18.5
0.5
0.4
0.3
39.2
MLDG
59.1
1.3
0.7
0.34
0.
50.2
41.2
CORAL
59.2
192
4.3
1..4
0.2
5.6
41.5
MMD
32.1 13.3
11.0
2
8.7
32.7
= 11.9
23.4
DANN
$3.1-002
1.1
4.7
11.8
5.4
46.8
38.3
CDANN
54.6 0.4
1..
4.9
1..1
5..2
4
38.3
MTL.
57.9 0.5
18.5
4.1
1..5
0.3
49.2
40.6
SagNet
57.7
19.0
4.3
1.5
5.5
4.2
40.3
ARM
0.3
1..3
1.19
9.1
5..4004
43.5
35.5
VREx
1.
3566
0.3
49.6
4,.0
33.6
RSC
55.0 + 1.2
0..
4.6
12.2
5.77
47.8
38.9
C.1.8 AVERAGES
Algorithm
ColoredMNIST
RotatedMNIST
OfficeHome
Terralncognita
DomainNet
Avg
ERM
51,5 11.5000 0.1
98.0 : 0.0
77.5 1 0.4
85.5 :: 0.2
66.5 = 0,3
46.1 1.8
40.9 i 0.1
66.6
IRM
52.0 1 0.1
97.7 0.1
78.5 1 0.5
83.5 1
64.3
2.2
47.6 0.8
33.9 2.8
65.4
GroupDRO
52.1 0.0
98.0
76.7 + 0.6
84.4
66.0 0.7
43.2
33.3 0.2
64.8
Mixup
52.1 =.2
98.0 0.1
77.4
84.6 0.6
68.1 0.3
4.8
39.2 0.1
66.7
MEDG
0,1
9
77.2 0.4
849
66.8
47.7 0.9
41.2
66.7
CORAL
5..
98.0 0.1
78.8 0.6
06.2.000 0.3
0,3
47.6
41.5
67.5
MMD
84.6
4.
63.3
DANN
$1.5 1.33
97.8 0.1
78.6 0.4
83.6 0.4
6..9
46.7 0.5
38.3
66.1
MTL
1..4
9.0
77.2 1 0.4
84.6
6.5
1.2
40.6 1.01
66.2
SagNet
ARM
56,2
98.2
77.6
0.3
85
6,3
43.5
35.5
66.1
VREX
5..8
9..
78.3 1 0.2
84
6.4
46.4
33.6
65.6
RSC
51.7
9..
77.1 0.5
85.2 0.9
05.5
1.0
38.9
66.1
C.2 MODEL SELECTION: LEAVE-ONE-DOMAINNOLL CROSS- VALIDAVA
C.2.1 COLOREDMNIST
Algorithm
+90%
+80%
-90%
Avg
ERM
50.0 + 0.2
50.1 0.2
10.0 + 0.0
36.7
IRM
46.7 2.4
0.3
23.1
10.7
40.3
GroupDRO
50.1 0.5
0.5
1.1
36.8
Mixup
36.6 10.9
5.
0.1
33.4
MLDG
50.1
5.3
1..0
36.7
CORAL
0.0
592
1..
39.7
MMD
5.2
5..0
9.9
36.8
DANN
4.1
6..1
10.0
40.7
CDANN
63.2
44.4
9.9
39.1
MTL
4.9
0.0
1..1
35.0
SagNet
4.49
0.3
1..0
36.5
ARM
0.3
0.3
10.2
36.8
VREx
5..2
5.55
10.1
36.9
RSC
0.3
49.7 0.4
10.1
36.5
23

---

Published as a conference paper at ICLR 2021
C.2.2 ROTATEDMNIST
Algorithm
0
15
30
45
60
75
Avg
ERM
95.3 + 0.2
98.9 + 0.1
98.9 0.1
98.8 + 0.1
98.5 + 0.1
96.2 + 0.2
97.7
IRM
0.5
0.2
98.7
96.6
98.4
9.1
97.0
GroupDRO
95.7
9.1
98.9
98.61022
9.26
9.9
97.6
Mixup
9..8
9.0
98.
99.0
9.0
9.3
97.8
MLDG
9.4
9.1
99.0
98.8
9.1
96.0
97.6
CORAL
0.5
9.2
9.2
98.6
9.1
96.3
97.8
MMD
95.8
0.1
99.0 0.0
9.1
0.1
9.2
97.8
DANN
0.5
9.5
98.5 0.1
9.
9.1
96.1
97.6
CDANN
94.3
0.3
98.9 0.1
0.17
9.1
95.7
97.5
MTL
95.5
9.3
98.8
9.1
9.1
95.6
97.8
SagNct
94.0
9.2
98.
99.1
9.1
74.2
94.0
ARM
0.1
0.1
9.0
98.
0.1
9.3
98.1
VREx
0.2
9.07
9.1
9..
74.0
20.1
0.5
93.6
RSC
94.6 0.0
9.24
9.1
9.0
98.8 0.1
0.1
95.9
35.9042
0.4
97.6
C.2.3 VLCS
Algorithm
L
S
Avg
ERM
98.0 0.4
62.6
1.9
77.5 - 1.9
77.2
IRM
98.6 0.3
1.1
6.
71.5
76.3
GroupDRO
98.1 0.3
6.9
71.0
1.41
77.9
Mixup
98.4
0.74
72.9
7..1
77.7
MLDG
98.5
6.2
7.86
7.80
77.2
CORAL
96.9
65.7
7.7
7.87
78.7
MMD
9.1
65.6
6.07
7.9
77.3
DANN
1.3
63.7
1.
7.72
76.9
CDANN
0.6
6.84
1.4
0.56
77.5
MTL
9.66
1.3
71.0
7.7
76.6
SagNet
0.4
0.8
7.
0.46
77.5
ARM
1.5 0.5
7.66
7.9
76.6
VREx
96.9
64.8 2.0
6.77
75.5
76.7
RSC
97.5 0.6
63.1 + 1.2
73.0
76.2 0.5
77.5
C.2.4 PACS
Algorithm
A
C
P
Avg
ERM
83.2 + 1.3
76.8 + 1.7
9..
74.8 + 1.3
83.0
IRM
81.7 2.4
1.3
96.3
71.1
2.2
81.5
GroupDRO
0.7
77.3
0.8
96.8
7.4
83.5
Mixup
1.9
77.0
96.8
1.6
83.2
MLDG
3.6
2.3
96.2
7.11
82.9
CORAL
8.8
0.4
96.8
7..6
82.6
MMD
8.79
7
9..1+
76.5
83.2
DANN
8.8
72.4 2.8
9..5+0.8
1.3
81.0
CDANN
7.83
73.8 + 1.6
9.5
66.8
78.8
MTL
78.9 1.5
9.3
73.1
83.7
SagNet
1.9
75.4 1.3
9.
0.6
82.3
ARM
0.3
73.3 - 1.9
72.1 9.
81.7
VREx
8..6440
74.1 -.1 0.3
96.9
72.8 2.1
81.3
RSC
83.7
82.9 + 1.1
95.6
68.1 +
82.6
24

---

Published as a conference paper at ICLR 2021
C.2.5 OFFICEHOME
Algorithm
A
P
R
Avg
ERM
61.1 + 0.9
50.7 0.6
74.6 0.3
76.4 + 0.6
65.7
IRM
58.2 0.4
1.2
23.3
74.1 1.7
64.3
GroupDRO
51.0
033
7.2
65.2
Mixup
0.5
0.3
7.28
0.3
67.0
MLDG
6..5
5.2
7..4
7..6
66.1
CORAL
6.
5.2
7..6
0.2
68.5
MMD
6.7
0.5
59.2
0.70
60.2
DANN
1.3
0.9
7..
0.6
64.9
CDANN
58.7
2.1
73.6
7.10
64.3
MTL
0..1
52.1
74.7
0.60
65.7
SagNet
6.80
0.3
7..6
0.4
67.6
ARM
58.7
49.8
7.5
75.9
0.1
64.4
VREX
57.6
1.3
0.2
75.8
1 0.7
64.9
RSC
61.6
51.1 0.8
74.8
75.7 + 0.9
65.8
C.2.6
TERRAINCOGNITA
Algorithm
L100
L38
L43
L46
Avg
ERM
34.4 +: 5.6
38.1
4.0
55.7 + 1.0
37.4 + 1.1
41.4
IRM
46.7
40.9 2.1
3.3
24.9
10.0
41.2
GroupDRO
45.2
2.0
1.4
3..3
44.9
Mixup
1.5
2.1
5.89
1.5
48.7
MLDG
1.9
0.2
1..2
3..3
46.2
CORAL
7.2
3.5
5.91
3.21
46.3
MMD
2.2
1.6
1.9
39.5
46.5
DANN
4.9
3.1
5..5
36.2
44.4
CDANN
7.3
4.4
41.0
4.8
39.9
MTL
4.6
1.1
5.8
1.9
44.9
SagNet
48.1 5.3
47.1
1.
39.1
47.2
ARM
3.5
5..9
35.4
42.6
VREx
46.4 1.4
25.5 5.8
39.6
37.8 3.6
37.3
RSC
40.0 + 1.3
32.1 2.5
0.5
34.2
1 0.2
40.0
C.2.7 DOMAINNET
Algorithm
clip
info
paint
quick
sketch
Avg
ERM
58.1
17.8
0.3
47,0
0.3
12.2 0.4
59.2 + 0.7
49.5 + 0.6
40.6
IRM
4.
37.3
10.9 0.5
48.0
42.3
3.1
33.5
GroupDRO
47.2
0.6
0.5
9.2
51.6
39.2 1.2
33.0
Mixup
1.6
18.0
0.5
11.5
1.18
46.9 0.2
38.5
MLDG
58.3
19.3
4.7
133
5.24
49.8
41.0
CORAL
0.1
1.3
0.1
1.
0.5
4.8
41.1
MMD
13.3
11.0
21.
8.72211
13.8
28.9 11.9
23.4
DANN
5.7
18.0
4.72
11.8
0.4
46.8
38.2
CDANN
173
4.
11.6
06.2+
4..9
38.0
MTL.
0.3
1..
4..7+
1.
0.2
4.1
40.6
SagNet
56.2+0.3
18.9
4.5
12.6
58.2+0.6
49.1
40.2
ARM
49.0
+5.8
401
9.4
53.0
43.4
35.2
VREX
4..5
1.
4.66
1.3
49,6149
42.0
33.4
RSC
$5.01..
1.
4.6
1.22
5..7
47.8
38.9
25

---

Published as a conference paper at ICLR 2021
C.2.8 AVERAGES
Algorithm
ColoredMNIST
RotatedMNIST
VLCS
PACS
OfficHHome
Terralncognita
DomainNet
Avg
36.7
77.2
0..0
41.4
40.6
63.2
IRM
40.3 4.2
97.0 :: 0.2
76.3 4: 0.6
81.5 0.8
64.3 1 1,5
41.2 3.6
31.5 : 3.0
62.0
GroupDRO
36.8
62.7
Mixup
33.4 4.7
97.8 0.0
77.7 +: 0.6
83.2 0.4
67.0 0.2
48.7 0.4
38.5 + 0.3
63.8
MLDG
36.7 0.2
77
829
66.1 0.5
46.2
64.0
CORAL
39.7
9.1
78.7
82.6 0.5
0,2
4.
4111
65.0
MMD
3688
97.8 1. 0.1
77.3 77.3033
83.2
60.2
46.5
234
60.7
DANN
40.7 00.7233
0.2
76.9
81.
6.29
44.4
0.2
63.4
CDANN
39.1 19.1444
97.5
17.5 0.2
78.8 2.2
64.3 1.7
19.9
38.0
62.2
MTL
35.0
9.1
76.6 1 0.5
83.7
65.7 0.5
1.2
40.6 00610.11
63.5
SagNet
36.5 0.1
94.0 3.0
77.5 0.3
0.3
823
0.1
67.6
47.2
402
63.6
ARM
36.8 1.801000 0.0
98.1 0.1
76.6
0.5
81.7 0.2
64.4 64.4
42.6
36.2
62.2
VRES
36.9 16.9-0.. 0.3
93.6
76.7 + 1.0
+03+009
64.9 1.3
37.3
334
60.6
RSC
3.2
9..
07.5
0.7
0.7
0.80
38.9
62.7
C.3 MODEL SELECTION: TEST-DOMAIN VALIDATION SET (ORACLE)
C.3.1
COLOREDMNIST
Algorithm
+90%
+80%
-90%
Avg
ERM
71.8 + 0.4
7
28.7 + 0.5
57.8
IRM
72.0 + 0.1
0.3
5
67.7
GroupDRO
73.5 + 0.3
0.3
36.8 2.8
61.1
Mixup
72.5 + 0.2
28.6 0.4
58.4
MLDG
+.
0.2
29.1 + 0.9
58.2
CORAL
0.2
0.2
1.6
58.6
MMD
2.3
5.2 1.
63.3
DANN
0.5
0.5
2.7
57.0
CDANN
71.8
0.1
33.8 6.4
59.5
MTL
7.2
0.2
28.0 0.6
57.6
SagNet
72.1
0.3
2..4
58.2
ARM
0.9
76.8
2.1
63.2
VREx
0.3
0.3
5.22
67.0
RSC
0.1
0.1
30.2
58.5
C.3.2
ROTATEDMNIST
Algorithm
0
15
30
45
60
75
Avg
ERM
95.3 : 0.2
98.7 + 0.1
98.9 + 0.1
98.7 1 0.2
98.9 0.0
0.0
96.2 0.2
97.8
IRM
94.9 0.6
9.2
9.1
98.6
0.1
0.3
97.5
GroupDRO
95.9 0.1
9.1
9.1
9..
98.6
9..3
97.9
Mixup
95.8 0.3
0.07
9..0
9.1
98.8
9..6
98.0
MLDG
95.7
98.9
9.8
98.9
98.
9..8
97.8
CORAL
96.2 0.2
98.8
9.1
98.8 0.1
98.9
9..4
98.0
MMD
96.1 0.2
9.09
9.00
98.8 = 0.0
9.09
96.4
98.0
DANN
95.9 0.1
9.19
9.2
98.7 + 0.1
0009
0.3
97.9
CDANN
95.9 0.2
9.08
9..
98.9 0.1
9.1
96.1
97.9
MTL
96.1 0.2
98.9
9.0
98.7 0.1
9.0
0.3
97.9
SagNet
95.9 0.1
99.0
9.1
0.1
98.8
9.1
97.9
ARM
95.9 0.4
99.0
0.1
0.1
9..1
9..7+
98.1
VREx
95.5
99.0
98
98.8
9.
9.44
97.9
RSC
95.4 0.1
9..6
98.6 + 0.1
98.9 +.0 0.0
9..8
9..4
97.6
26

---

Published as a conference paper at ICLR 2021
C.3.3 VLCS
Algorithm
L
S
Avg
ERM
97.6 0.3
67.9
7.
74.0 + 0.6
77.6
IRM
97.3
6..7
71.0
7.4
76.9
GroupDRO
9.2
6.2
72.8
1.3
77.4
Mixup
9.4
6.4
71.5
7..7
78.1
MLDG
97.1
0.5
71.5
7.9
77.5
CORAL
0.2
67.5
7.66
7..5
77.7
MMD
98.8
66.4
0.5
7..6
77.9
DANN
9..
66.3
7.
80.1
79.7
CDANN
0.1
0.5
7.6
0.5
79.9
MTL
9.7
+6.1
72.0+
1.1
77.7
SagNet
0.3
66.4 0.4
7..6
7.80
77.6
ARM
0.6
66.5 0.3
7..7
7.74
77.8
VREx
9.2
7. 0.7
1.
78.1
RSC
98.0 0.4
67.2 0.3
1.
75.6
77.8
C.3.4 PACS
Algorithm
P
Avg
ERM
86.5 + 1.0
81.3 0.6
9.
82.7 + 1.1
86.7
IRM
04.2
9..9 1.5
78.3
84.5
GroupDRO
87.5 0.5
97.1 0.6
81.1
87.1
Mixup
87.5 0.4
0.7
97.4+
80.8
86.8
MIDG
1.2
22.5000
9..7
81.2 0.6
86.8
CORAL
8..6
0.9
82.7 9.5
87.1
MMD
88.1
$2.6
9.5
81.2 1.2
87.2
DANN
0.4
80.3
7.19 9.3
85.2
CDANN
0.6
80.7
9.
77.6 1.5
85.8
MTL
87.0 0.2
82.7 0.8
9.7
80.5 + 0,8
86.7
SagNet
87.4 0.5
4
96.3
80.7
86.4
ARM
85.0 + 1.2
81.4 + 0.2
9..9
40.9 0.5
85.8
VREx
87.8 - 1.2
81.8 0.7
97.4
82.1
87.2
RSC
86.0 0.7
81.8 - 0.9
96.8
30.4
86.2
C.3.5
OFFICEHOME
Algorithm
A
C
P
R
Avg
ERM
61.7 + 0.7
53.4
74.1 0.4
76.2 + 0.6
66.4
IRM
3.2
51.
71.7
72.7
63.0
GroupDRO
1.6
5.3
0.3
75.9
66.2
Mixup
0.2
0.4
7.3
7.70
68.0
MLDG
6.75
0.5
7..0
7.5
66.6
CORAL
0.8
0.9
7..5
7..2
68.4
MMD
1.0
5.54
7..9+
7..1
66.2
DANN
1.4
51.8
7.
75.5
65.3
CDANN
5.2
52.1
7..9
76.2 0.2
65.3
MTL
6..7
53.5
7.62
76.6 0.6
66.5
SagNet
0.5
0.5
7.
77.8
67.5
ARM
5.5
51.8 0.7
7..0
74.4
64.8
VREx
59.6
53.3
7.
76.6
65.7
RSC
0.87
53.0 + 0.9
74.8 + 0.8
76.3 + 0.5
66.5
27

---

Published as a conference paper at ICLR 2021
C.3.6 TERRAINCOGNITA
Algorithm
L100
1.38
L43
I.46
Avg
ERM
59.4
0.6
60.1
0.5
53.0
IRM
56.5
49.8
2.2
38.6
50.5
GroupDRO
1.5
48.3
58.6
4.22
52.4
Mixup
67.6
1.3
59.0
4.1
54.4
MLDG
59.2 0.1
0.9
58.4
1.0
52.0
CORAL
60.4 0.9
0.5
59.3
4..4
52.8
MMD
6.1
0.3
0.5
4.2
52.0
DANN
1.92
0.7
5.9
4.9
50.6
CDANN
56.3
0.9
1.
4..4
50.8
MTL
28.
48.4 4
58.9
1.3
52.2
SagNet
1.9
50.5 2.3
5.5
0.6
52.5
ARM
1.51
48.3 1
5.6
4.19
51.2
VREx
1.7
46.5 = 0.5
58
43.8
51.4
RSC
1.4
46.7 = 0.4
5.58
4.6
52.1
C.3.7 DOMAINNET
Algorithm
clip
info
paint
quick
real
sketch
Avg
ERM
58.6 0.3
19.2 + 0.2
47.0 0.3
13.2 0.2
59.9 + 0.3
49.8 + 0.4
41.3
IRM
40.4 6.6
2.7
5.77
9.8 9.8
1.2
1.2
9.0
36.7
28.0
GroupDRO
0.5
17.5
0.3
0.2
0.5
4..
33.4
Mixup
5.1
1..7
4.5
0.3
0.5
48.2
39.6
MLDG
0.1
19.6
0.2
0.2
6..1
50.4
41.6
CORAL
1.1
19.9
4.2
1..0
59.8
50.4 0.4
41.8
MMD
32.2 13.3
11.2
11.
8.2
$2.7
29.0 1 11.8
23.5
DANN
0.2
18.3
0.7
0.1
5.4
46.8 1 0.6
38.3
CDANN
54.6
17.
4.72
0.2
5..2
45.9
38.5
MTL
58.0
1922
0.1
0.2
0.11
4.0
40.8
SagNet
0.3
19.1
0.5
0.45
58.9
4.2
40.8
ARM
0.4
16.5
4.8
0.1
0.3
4..
36.0
VREx
4.5
1..1+188
3.05
9.1
4.6
3.57
30.1
RSC
1.2
1.5
0.6
12.5
0.77
47.8 0.9
38.9
C.3.8 AVERAGES
Algorithm
ColoredMNIST
RotatedMNIST
VLCS
PACS
OfficHHome
TerraIncoguita
DomainNet
Avg
ERM
57.8 0.2
97.8 + 0.1
77.6
86.7
+ 0.3
66.4 = 0.5
53.0
41.3
68.7
IRM
67.7
84.5
63.0 2.7
5..5
66.9
GrouDRRO
61.1 0.9
97.9 1 0.1
77.4
87.1 0.1
66.2
0.6
12.1
33.4
0.3
67.9
Mixup
58.4
68.0
1..4
39.6
69.0
MLDG
18.2..0.4
97.80..1
77,5
86.8 0.4
66.6 0.3
52.0
41.6
68.7
77.7
MMD
63.3 1.3
9.1
0.1
0.1
6,22
52.0
23,5
66.9
DANN
$7.0
9.9
79.7 0.5
85.2
6..3
50.6
38.
67.7
CDANN
2.0
9..
19.9
85.8
0.8
0,5
50.8
0.2
68.2
MIL
0.3
9..
77.7 0.5
86.7
66.5-.4
$2.20.4
40.8
68.5
SagNet
58.2
97
77.6 1 0.1
1.4
6.25
$2.5104
40.8
68.7
ARM
63.2 63.2 0.7
9..1
77.8
0.3
85.8
64.8 448404
$1.2
36.0
0.2
68.1
VREs
67.0 11
9..
78.1
1
0.2
87.2
65.7 0.3
11.4
30.1
68.2
RSC
58.5 0.5
97.6
77.8
0.6
86.2
66.5 0.6
$2.1 02.
38.9 0.6
68.2
28

---

Published as a conference paper at ICLR 2021
C.4
RESULTS OF LARGER PACS SWEEP WITH 100 HYPERPARAMETER TRIALS
ERM, model selection:
A
C
P
Avg
training-domain
86.6 0.8
79.7 = 0.6
96.6 = 0.4
77.8 0.8
85.2
leave-one-out-domain
86.4
78.2 + 1.0
96.8 0.2
76.0 2.1
84.4
test-domain (oracle)
89.3 0.3
82.2 + 0.5
97.6 + 0.2
82.7 1.1
88.0
29


Reviews:
[{"review_id": "xOh9mnNxlZF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThe paper investigates the success of domain generalization (DG) approaches developed in recent years. In doing so, the authors evaluate a large variety of the most successful recent variants under principled model selection schemes (training-domain and leave-out-out validation), and find that standard empirical risk minimization outperforms – or is at least comparable – to all recently proposed state-of-the-art competitors. In bringing together a large set of heterogeneous methods the work makes an interesting contribution to the current DG literature. At the same time, in its current form I found the manuscript to be lacking any sufficiently substantial recommendations on how to improve the currently available zoo of methods.\n\n### Strengths\n\nThe manuscript contains a very rigorous and impressively detailed background section (moved to the appendix) as well as experimental evaluations of the most important recent methods in domain generalization. The methodological introduction is very well written and principled, and contrasts domain generalization against other learning setups (from generative learning all the way to continual learning, domain adaptation etc.).\n\nThe experimental section does a convincing job of comparing empirical risk minimization against the various DG competitors that have been developed in recent years (DIVA, RSC, DDAIG, etc.), and in particular makes important recommendations that should find their way into practical research around the theme of DG. The proposed DomainBed environment (code in supplementary materials) appears of high quality, although it is difficult to gauge whether its ease-of-use in terms of extending it to new approaches will convince future researcher to adopt it and incorporate it with their propositions.\n\nAll in all, I found the manuscript to be a compelling read that contains an interesting alternative viewpoint on the role and limitations of DG. That being said, I think more needs to be done to make this paper more inspiring and useful for the wider community, in particular with regards to its concreteness (see below).\n\n### Weaknesses\n\nAs highlighted above the paper is extremely well-written. However, it often leans in directions that I find too implicit, and – in my opinion – unnecessarily so. For example: \"selecting hyperparameters is a learning problem at least as hard as fitting the model (inasmuch as we may interpret any model parameter as a hyperparameter)\" (page 3, final paragraph). While this is a stimulating sentence, as a reader I am left wondering how this is crucial to motivating the central elements of the paper. In addition, what is the average DG researcher going to extract from this? Left untouched, I would suspect such a statement might invite some additional criticism from the viewpoint of previous DG research: how can one possibly guarantee that the tuning as part of DomainBed is optimal for the large heterogeneous variety of DG approaches out there, and who is to say that this can't be improved upon?\n\nWhile page 8 introduces a number of interesting and compelling open questions, I would encourage the authors to provide more guidance in this context, potentially in the form of some experimentation, or analysis of the large set of results (e.g. which methods have strengths where, etc.). Some (again, too implicit) comments are there – e.g. \"Therefore, we believe there is great promise in DG algorithms able to self-evaluate and self-adapt during test time.\" – more concise propositions as to which ideas the DG community can consider would be extremely helpful here.\n\nI think the final paragraph of the paper nicely summarizes what I find to be the central weakness of this manuscript: in asking what a good benchmark is, instead of proposing some sort of alternative or making a recommendation, the authors opt for an intellectually compelling quote by Proust. Sure it's lovely to read, but I would again caution that is going to be of limited practical usefulness.\n\nIn summary, I therefore remain somewhat unconvinced that the manuscript, at least in its current form that mainly resorts to criticism of other DG approaches (while undoubtedly warranted (!)), but otherwise steers clear of any concise recommendations for future improvements, is an optimal use of content for an 8 pages conference paper.\n\n### Minor points\n\n\"Although challenging, DG is the best approximation to real prediction problems, where unforeseen distributional discrepancies between training and testing data are surely expected.\" (page 3, third paragraph). DG is an interesting problem, but I'd be a bit more cautious in elevating its role for ML predictions problems in the real-world. At least in its current form DG approximates real-world problems from a (constrained) direction, and the above statement doesn't yet synergize well will the paragraph that rightfully asks whether DG benchmarks \"are [..] the right datasets?\" (page 8). Some clarification around the restrictions of DG early on would be helpful here.\n\nWhile an interesting analysis of the differences between learning scenarios and domain generalization has been demarcated clearly from other types of learning problems, a very brief mentioning of the difference between multi-task and multi-domain learning would be beneficial given there is some confusion around these terms in the current literature.", "extra_scores": {}, "extra_text": {}}, {"review_id": "p0hi5VRMaq4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper critically re-examines research in domain generalisation (DG), ie building models that robustly generalise to out-of-distribution data. It observes that existing methods are hard to compare, in particular due to unclear hyper-parameter and model selection criteria. It introduces a common benchmark suite including a well designed model selection procedure, and re-evaluates existing methods on this suite. The results show that under such controlled evaluation, the benefit of existing DG methods over vanilla empirical risk minimisation (ERM) largely disappear. This raises the concern that existing DG methods might be over-tuned and hard to replicate. By releasing the controlled benchmark suite, future research progress can be more reliably measured. \n\nStrength: \n+ The paper highlights an important point. Comparing existing methods is indeed tricky and complicated by model selection and hyper-parameter selection issues, etc.  It makes good recommendations for practice, such as requiring that any DG method also specifies its model selection method.  (We knew this already, but it’s good to remind people explicitly!) Helpfully it specifies a few reasonable options for model selection criteria which future papers could refer to rather than inventing ad-hoc approaches. \n+ A common benchmark with a pre-specified strategy for hyper-parameter/early-stopping could be very helpful for more reliably comparing and measuring research progress in future. \n+ A significant amount of effort was expended running a large and properly comparable evaluation across all several existing methods.\n\nWeakness: \n1. Strength of claim & validation. This paper is in part making a very strong negative result claim that a wide range of existing methods don’t work when implemented “properly”. This might be true, but then there is onus on the paper to make sure that all of the evaluation details are completely watertight.  \n- Are 20 trials enough for random search on all these models? It would be good to show some evidence that performance has saturated at this point (e.g., that performance doesn’t improve further with an order of magnitude more search). It would be good to also show some specific hyper-parameters found by the random search, so experts in the specific algorithms shown can assess if something sensible was found. Do the discovered hyper-parameters correspond to ERM for those methods where ERM is a special case?\n- Are the hyper-parameter choices and ranges (Tab 6) satisfactory? For example some algorithms I have expertise on include hyper-parameters not documented in Tab 6, and its not clear how these are set. As another example, the DG research in my group has used SGD with momentum, not Adam, due to better stability in our experience. It’s not clear how this change affects things. \n- Re Tab 6. It’s not clear: When optimising the methods do you jointly optimize the Resnet hyper parameters (such as learning rate, batch size) with the DG-specific parameters (bottom half of table) for each individual DG method. Or do you optimize ResNet hyperparameters first for ERM and then fix those before optimising the DG hyper-parameters?\n- Overall, while the benchmark should be a useful contribution anyway, to believe the specific numerical conclusions, we have to trust the authors did a really good job re-implementing everything. If this research project had been setup instead as a competition where DG developers submit algorithms according to the constraints imposed by the benchmark for independent evaluation, and still reached the sae conclusion, then it would be more believable. \n\n2. Insight. If we accept the headline conclusion that ERM indeed outperforms all the existing methods, it would be really nice to have some insight into what went wrong along the way. Some of these were alluded to, but not properly analysed.  For example:\n- Do the prior methods have visible benefit over ERM on the more commonly evaluated networks like ResNet-18, and that benefit just fails to transfer to ResNet-50? We don’t know, because the comparison is only made on ResNet-50. \n- Do the prior methods have visible benefit over ERM in the absence of data augmentation, and not otherwise? \n- How much of the negative result is due to “proper tuning” improving ERM compared to previous poorly tuned implementations of the baselines; versus worsening the previously over-tuned DG methods?\n- What is the primary source of over-tuning in existing DG methods? Which dominates among model selection, hyper-parameter selection, dataset split?\n- Insights such as these would make the paper more satisfying, as well as believability -- by identifying more precisely the factors behind the negative result. Without these, it feels a bit empty. \n\n3. Minor: \n- I don’t understand the final comment in “untestable assumptions” on Pg 8. “DG algorithms that self-evaluate and self-adapt at test time” seems wrong. Adapting at test-time per-se, is against the problem spec of DG and blurs into domain adaptation.\n\n============ UPDATE =============\n\nThanks to the authors for their feedback. I appreciate the efforts on clarification and loose-end tying.\n\nOne outstanding thing to clarify to help us understand whether Claim 1 can be fully supported: \n- AFAIK the Table 1 / Table 5 that underpin this claim are comparing numbers copied from previous papers with numbers generated from Domain Bed benchmark? However I suspect the splits are not the same. For example, some previous benchmarks have a fixed split by default, while I understood Domain Bed use multiple random splits? If so the numbers are not directly comparable, and it still may not be fair to make a strong claim that tuned ERM outperforms prior work.", "extra_scores": {}, "extra_text": {}}, {"review_id": "kz4PdGANHJd", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\n\nIn this work, authors implement and tune 14 DG algorithms and compare them across 6 datasets with 3 model selection criteria, and they find (i) a careful implementation of ERM outperforms the SoTAs. (ii) no competitor can outperform ERM by more than one point (iii) model selection matters for DG.\n\n##########################################################################\n\nReasons for score:\n\n\nI believe \"the well-tuned ERM outperforms many SoTA DG methods\" may not surprise many researchers in this area, but I would very much like to see a paper delivering this message clearly. Thus, I recommend an acceptance. \n\n##########################################################################\n\nPros:\n\n\n1. For quite a while, ERM (when carefully implemented and tuned) being the SoTA in DG is \"elephant in the room\". At least, we should admit that, quite often, the \"improvements\" claimed by those DA methods are gone when switching from a weaker backbone (e.g., ResNet-18) to a stonger one (e.g., ResNet-50). A high-standard testing protocol is a very important contribution in DG research.\n\n2. This work brings an open-sourced software for replicating the existing methods, and comparing the newly proposed ones in a consistent and realistic setting.\n\n##########################################################################\n\nCons:\n\n\n1. Domain-Net, as a much larger scale and more challenging dataset, could be considered. \n\n2. Can you elaborate the last sentence of Claim 2, i.e.,  \"our advice to DG practitioners is to use CORALwith a hyperparameter search distribution that allows ERM-like behavior\".\n\n##########################################################################\n\nA typo: (Page 6) \"Table 5.2\" shows that using a ResNet-50 neural network architecture. I think it should be \"Table 4\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "BC8jzrPf4Xh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors implement a test bed to evaluate domain generalization methods in a unified way. The works is important because current methods use different model selection approaches, which may not reflect the inherent properties of the DG algorithms. \n\nModel selection is a fundamentally difficult problem in the presence of distribution shift. However, it was significantly ignored in previous works. It is nice to see that the authors provide three kinds of model selection methods. From the results, it seems that existing DG methods do not have a clear advantage over ERM even when test-domain validation test is used. Does this mean existing methods themselves are not good? Or the dataset might not be appropriate for DG? It seems hard even for human to generalize to new domains when given a small number of domains with many changing factors. \n\nI have some questions regarding the test bed details.\n1)\tDid the authors implement the existing methods or use the source codes provided by the authors?\n2)\tThe authors carefully implemented and tuned ERM, did the authors also tuned the other methods carefully? This may require a significant amount of work, because different methods may need different engineering tricks.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides an interesting analysis on the research on Domain Generalization with main principles and limitations. The authors provide a strong rebuttal to address some comments pointed by reviewers. All the reviews are very positive.
Hence, I recommend acceptance.
Decision: Accept (Poster)