In Search of Lost Domain Generalization

1 INTRODUCTION . Machine learning systems often fail to generalize out-of-distribution , crashing in spectacular ways when tested outside the domain of training examples ( Torralba and Efros , 2011 ) . The overreliance of learning systems on the training distribution manifests widely . For instance , self-driving car systems struggle to perform under conditions different to those of training , including variations in light ( Dai and Van Gool , 2018 ) , weather ( Volk et al. , 2019 ) , and object poses ( Alcorn et al. , 2019 ) . As another example , systems trained on medical data collected in one hospital do not generalize to other health centers ( Castro et al. , 2019 ; AlBadawy et al. , 2018 ; Perone et al. , 2019 ; Heaven , 2020 ) . Arjovsky et al . ( 2019 ) suggest that failing to generalize out-of-distribution is failing to capture the causal factors of variation in data , clinging instead to easier-to-fit spurious correlations prone to change across domains . Examples of spurious correlations commonly absorbed by learning machines include racial biases ( Stock and Cisse , 2018 ) , texture statistics ( Geirhos et al. , 2018 ) , and object backgrounds ( Beery et al. , 2018 ) . Alas , the capricious behaviour of machine learning systems out-of-distribution is a roadblock to their deployment in critical applications . Aware of this problem , the research community has spent significant efforts during the last decade to develop algorithms able to generalize out-of-distribution . In particular , the literature in Domain Generalization ( DG ) assumes access to multiple datasets during training , each of them containing examples about the same task , but collected under a different domain or experimental condition ( Blanchard et al. , 2011 ; Muandet et al. , 2013 ) . The goal of DG algorithms is to incorporate the invariances across these training domains into a classifier , in hopes that such invariances will also hold in novel test domains . Different DG solutions assume different types of invariances , and propose algorithms to estimate them from data . Despite the enormous importance of DG , the literature is scattered : a plethora of different algorithms appear yearly , each of them evaluated under different datasets , neural network architectures , and model selection criteria . Borrowing from the success of standardized computer vision benchmarks ∗Alphabetical order , equal contribution . Work done while IG was at Facebook AI Research . This paper is a living benchmark , always refer to the latest version available at https : //arxiv.org/abs/2007.01434 such as ImageNet ( Russakovsky et al. , 2015 ) , the purpose of this work is to perform a rigorous comparison of DG algorithms , as well as to open-source our software for anyone to replicate and extend our analyses . This manuscript investigates the question : How useful are different DG algorithms when evaluated in a consistent and realistic setting ? To answer this question , we implement and tune fourteen DG algorithms carefully , to compare them across seven benchmark datasets and three model selection criteria . There are three major takeaways from our investigations : • Claim 1 : A careful implementation of ERM outperforms the state-of-the-art in terms of average performance across common benchmarks ( Table 1 , full list in Appendix A.5 ) . • Claim 2 : When implementing fourteen DG algorithms in a consistent and realistic setting , no competitor outperforms ERM by more than one point ( Table 3 ) . • Claim 3 : Model selection is non-trivial for DG , yet affects results ( Table 3 ) . As such , we argue that DG algorithms should specify their own model selection criteria . As a result of our research , we release DOMAINBED , a framework to streamline rigorous and reproducible experimentation in DG . Using DOMAINBED , adding a new algorithm or dataset is a matter of a few lines of code . A single command runs all experiments , performs all model selections , and auto-generates all the result tables included in this work . DOMAINBED is a living project : we welcome pull requests from fellow researchers to update the available algorithms , datasets , model selection criteria , and result tables . Section 2 kicks off our exposition with a review of the DG setup . Section 3 discusses the difficulties of model selection in DG and makes recommendations for a path forward . Section 4 introduces DOMAINBED , describing the features included in the initial release . Section 5 discusses the experimental results of running the entire DOMAINBED suite , illustrating the competitive performance of ERM and the importance of model selection criteria . Finally , Section 6 offers our view on future research directions in DG . Appendix A reviews one hundred articles spanning a decade of research in DG , summarizing the experimental performance of over thirty algorithms . 2 THE PROBLEM OF DOMAIN GENERALIZATION . The goal of supervised learning is to predict values y ∈ Y of a target random variable Y , given values x ∈ X of an input random variable X . Predictions ŷ = f ( x ) about x originate from a predictor f : X → Y . We often decompose predictors as f = w ◦ φ , where we call φ : X → H the featurizer , and w : H → Y the classifier . To solve the prediction task we collect the training dataset D = { ( xi , yi ) } ni=1 , which contains identically and independently distributed ( i.i.d . ) examples from the joint probability distribution P ( X , Y ) . Given a loss function ` : Y × Y → [ 0 , ∞ ) measuring prediction error , supervised learning seeks the predictor minimizing the risk E ( x , y ) ∼P [ ` ( f ( x ) , y ) ] . Since we only have access to the data distribution P ( X , Y ) via the dataset D , we instead search a predictor minimizing the empirical risk 1n ∑n i=1 ` ( f ( xi ) , yi ) ( Vapnik , 1998 ) . The rest of this paper studies the problem of Domain Generalization ( DG ) , an extension of supervised learning where training datasets from multiple domains ( or environments ) are available to train our predictor ( Blanchard et al. , 2011 ) . Each domain d produces a dataset Dd = { ( xdi , ydi ) } nd d=1 containing i.i.d . examples from some probability distribution P ( Xd , Y d ) , for all training domains d ∈ { 1 , . . . , dtr } . The goal of DG is out-of-distribution generalization : learning a predictor able to perform well at some unseen test domain dtr + 1 . Since no data about the test domain is available during training , we must assume the existence of statistical invariances across training and testing domains , and incorporate such invariances ( but nothing else ) into our predictor . The type of invariance assumed , as well as how to estimate it from the training datasets , varies between DG algorithms . We review a hundred articles in DG spanning a decade of research and thirty algorithms in Appendix A.5 . DG differs from unsupervised domain adaptation . In the latter , unlabeled data from the test domain is available during training ( Pan and Yang , 2009 ; Patel et al. , 2015 ; Wilson and Cook , 2018 ) . Table 2 compares different machine learning setups to highlight the nature of DG problems . The causality literature refers to DG as learning from multiple environments ( Peters et al. , 2016 ; Arjovsky et al. , 2019 ) . Although challenging , the DG framework can capture some of the difficulty of real prediction problems , where unforeseen distributional discrepancies between training and testing data are surely expected . At the same time , the framework can be limiting : in many real world scenarios there may be external variables informing about task relatedness ( space , time , annotations ) that the DG framework ignores . 3 MODEL SELECTION AS PART OF THE LEARNING PROBLEM . Here we discuss issues surrounding model selection ( choosing hyperparameters , training checkpoints , architecture variants ) in DG and make specific recommendations for a path forward . Because we lack access to a validation set identically distributed to the test data , model selection in DG is not as straightforward as in supervised learning . Some works adopt heuristic strategies whose behavior is not well-studied , while others simply omit a description of how to choose hyperparameters . This leaves open the possibility that hyperparameters were chosen using the test data , which is not methodologically sound . Differences in results arising from inconsistent tuning practices may be misattributed to the algorithms under study , complicating fair assessments . We believe that much of the confusion surrounding model selection in DG arises from treating it as merely a question of experimental design . To the contrary , model selection requires making theoretical assumptions about how the test data relates to the training data . Different DG algorithms make different assumptions , and it is not clear a priori which ones are correct , or how they influence the model selection criterion . Indeed , choosing reasonable assumptions is at the heart of DG research . Therefore , a DG algorithm without a strategy to choose its hyperparameters should be regarded as incomplete . Recommendation 1 A DG algorithm should be responsible for specifying a model selection method . While algorithms without well-justified model selection methods are incomplete , they may be useful stepping-stones in a research agenda . In this case , instead of using an ad-hoc model selection method , we can evaluate incomplete algorithms by considering an oracle model selection method , where we select hyperparameters using some data from the test domain . Of course , it is important to avoid invalid comparisons between oracle results and baselines tuned without an oracle method . Also , unless we restrict access to the test domain data somehow , we risk obtaining meaningless results ( we could just train on such test domain data using supervised learning ) . Recommendation 2 Researchers should disclaim any oracle-selection results as such and specify policies to limit access to the test domain . 3.1 THREE MODEL SELECTION METHODS FOR DG . Having made broad recommendations , we review and justify three model selection criteria for DG . Appendix B.3 illustrates these with an specific example . Training-domain validation We split each training domain into training and validation subsets . We train models using the training subsets , and choose the model maximizing the accuracy on the union of validation subsets . This strategy assumes that the training and test examples follow similar distributions . For example , Ben-David et al . ( 2010 ) bound the test error of a classifier with the divergence between training and test domains . Leave-one-domain-out validation Given dtr training domains , we train dtr models with equal hyperparameters , each holding one of the training domains out . We evaluate each model on its held-out domain , and average the accuracies of these dtr models over their held-out domains . Finally , we choose the model maximizing this average accuracy , retrained on all dtr domains . This strategy assumes that training and test domains follow a meta-distribution over domains , and that our goal is to maximize the expected performance under this meta-distribution . Note that leaving k > 1 domains out would increase greatly the number of experiments , and introduces a hyperparameter k. Test-domain validation ( oracle ) We choose the model maximizing the accuracy on a validation set that follows the distribution of the test domain . Following our earlier recommendation to limit test domain access , we allow one query ( the last checkpoint ) per choice of hyperparameters , disallowing early stopping . Recall that this is not a valid benchmarking methodology . Oracle-based results can be either optimistic , because we select models using the test distribution , or pessimistic , because the query limit reduces the number of considered hyperparameters . We also tried limiting the size of the oracle test set instead of the number of queries , but this led to unacceptably high variance .

Summary: This paper critically re-examines research in domain generalisation (DG), ie building models that robustly generalise to out-of-distribution data. It observes that existing methods are hard to compare, in particular due to unclear hyper-parameter and model selection criteria. It introduces a common benchmark suite including a well designed model selection procedure, and re-evaluates existing methods on this suite. The results show that under such controlled evaluation, the benefit of existing DG methods over vanilla empirical risk minimisation (ERM) largely disappear. This raises the concern that existing DG methods might be over-tuned and hard to replicate. By releasing the controlled benchmark suite, future research progress can be more reliably measured.