An Empirical Investigation of Domain Generalization with Empirical Risk Minimizers

1 Introduction . Conventional wisdom in domain generalization was recently upturned by the work of Gulrajani & Lopez-Paz ( 2020 ) on the DomainBed benchmark . In their work , the authors use careful model selection and evaluation to show that models trained using empirical risk minimization ( Vapnik , 1999 ) are able to achieve near state-of-the-art performance on a variety of popular benchmarks , surpassing the performance of most algorithms specialized for domain generalization ( Arjovsky et al. , 2019 ; Yan et al. , 2020 ; Li et al. , 2017a ) . In a parallel research effort , there has been an increased interest in understanding when deep neural networks trained using empirical risk minimization generalize well in-domain ( Jiang et al. , 2019 ; Neyshabur et al. , 2017 ; Dziugaite & Roy , 2017 ; Bartlett et al. , 2017 ; Arora et al. , 2018 ; Daniely & Granot , 2019 ) . However , understanding when empirical risk minimizers are able to generalize out-of-distribution remains poorly understood . Answering this question would help us characterize the failure modes of domain generalization algorithms , and to develop specialized baselines that would allow us to break through the dominance of empirical risk minimization . We argue that the out-of-distribution generalization performance of a classifier depends mainly on two factors . On the one hand , we expect good out-of-distribution generalization only for classifiers able to achieve good in-distribution generalization ( also called source-domain validation accuracy ) . On the other hand , we expect out-of-distribution generalization to degrade as the discrepancy between the distributions of training and testing data increases , relative to the function family of interest . This reveals an interplay between the properties of the neural network , the used training data , and the ∗ Both D.L.P and D.J.S . contributed to the work equally 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . differences between training and testing regimes ( Figure 1 ) . Given all of these distinct ingredients , how does one build a coherent theory of domain generalization with strong predictive power for modern , over-parametrized neural networks ? The seminal work of Ben-David et al . ( 2007 , 2010 ) lays down a theoretical foundation for the related setup of unsupervised domain adaptation . In particular , the theory in Ben-David et al . ( 2007 , 2010 ) upper-bounds the out-of-distribution performance of an ERM based on a fixed feature space as the sum of two terms : ( i ) the source-domain validation accuracy , and ( ii ) the distance between the input distribution of source-domain and target-domain examples under the fixed feature space . This theory has found success in practice , improving both linear models ( Ben-David et al. , 2007 ) and deep neural networks ( Ganin et al. , 2015 ) in the setup of unsupervised domain adaptation . Algorithmically speaking , one can implement this theory by finding a feature representation under which ( i ) one can perform well on source-domain inputs , and ( ii ) it is not possible to distinguish source-domain inputs from target-domain inputs . While the theory above deals with the setup of unsupervised domain adaptation ( where algorithms have access to unlabeled target-domain data during training ) , here we are interested in the related but distinct domain generalization scenario where algorithms only have access to source-domain data during training . Therefore , to remain in the domain generalization setup , we will put the theory of Ben-David et al . ( 2007 ) to use from a different angle . Instead of training representations in such a way that encourages source-domain and target-domain inputs to be indistinguishable , our methodology consists in training a very large number of ERM models only using source-domain data , to later investigate if the models exhibiting good out-of-distribution generalization indeed learned a feature space under which it is difficult to tell apart source-domain and target-domain inputs . Training over 12,000 deep neural network image classifiers using empirical risk minimization , using the standard DomainBed pipeline , we reveal that the theory of Ben-David et al . ( 2007 ) has limited power to predict when a classifier will generalize out-of-domain . We investigate this further by unpacking whether assumptions in the theory are violated or if the bounds are slack . Our findings suggest that in practice , while the assumptions from the theory are largely not violated in the learned feature spaces of ERMs , one can obtain good generalization despite being able to tell apart the source and target data , limiting the practical applicability of the theory to explain domain generalization . We investigate and provide more intuitions for this phenomenon in section 4 . In search of other correlates of domain generalization that might point the way to new theoretical approaches , we set out to build a catalog of measures that could predict the out-of-distribution generalization performance of our suite of ERM trained neural networks . In building such a catalog of generalization measures , we take inspiration from recent literature undertaking a similar effort to explain variance of in-domain generalization of deep neural networks ( Jiang et al. , 2019 ; Neyshabur et al. , 2017 ; Dziugaite et al. , 2020 ) . After an exhaustive empirical evaluation , we find that several of the proposed measures are relatively strong predictors of out-of-distribution performance . These include measures related to predictive entropy , the Mixup criterion ( Zhang et al. , 2017 ) , the norm of the Jacobian ( Novak et al. , 2018 ) , and the Fisher information matrix ( Amari , 1998 ) . To summarize , our work makes the following contributions : • We perform a large scale empirical study testing the theory from Ben-David et al . ( 2007 , 2010 ) on deep neural networks trained on the DomainBed ( Gulrajani & Lopez-Paz , 2020 ) domain generalization benchmark . • We find that the theory has limited ability to predict which ERMs will generalize out-ofdistribution . • We perform an empirical evaluation of several other candidate measures for explaining domain generalization of neural networks , finding measures that outperform theoretically motivated quantities . 2 Background on Relevant Theory . There is a rich line of work on the theory of domain adaptation , which focuses on learning and adapting when presented with test data drawn from a different input distribution from that seen during training ( Ben-David et al. , 2007 ; Mohri & Medina , 2012 ; Cortes et al. , 2019 ) . There are also a number of algorithms for domain adaptation which make use of unlabeled data to learn more invariant predictors , both for classical kernel machines ( Pan et al. , 2011 ; Gretton et al. , 2009 ) as well as deep neural networks ( Ganin et al. , 2015 ) . The results typically provide bounds on the test error in a target domain in terms of the test error of the model on a training domain , and a measure of the difference between the train and target input distributions in a feature space . Here we are interested in whether the theory can accurately predict out-of-distribution performance of an ERM . In particular , we first focus on a classic result from Ben-David et al . ( 2007 ) , which , given a fixed feature representation ( R ) , derives generalization bounds for the test performance in terms of how well the model can distinguish source from target samples . This work focuses on the single-source ( SS ) setting , where a single source-domain is used for training . A central result from this work is a bound on the test error on the target domain ( T ( h ) ) , consisting of three distinct terms : the sourcedomain validation error ( S ( h ) ) , the so-called H-divergence between the source and target domains , and the joint source-target validation error when optimizing a single model for both environments , i.e . λ = minh∈H { T ( h ) + S ( h ) } . In fact , in the theory , λ-closeness is assumed , and we will test this assumption later . We consider the predictive value of each of these three terms separately as well as their sum in our analysis . The H-divergence can be understood as related to and approximated by the best performance of a classifier in the function family H trained to distinguish samples from the source and target domains . Subsequent work generalized this theory to multi-source ( MS ) training in Ben-David et al . ( 2010 ) . Assuming an equal weighting of all training domains , the gap between the test errors in the target and source domains can be bounded above by the sum of two terms : the average λj , where λj = minh∈H { T ( h ) + j ( h ) } , and the H∆H-divergence , similar to the H-divergence but for the symmetric difference hypothesis class . A full description of these terms can be found in the supplementary material . 3 Experiments . In this section we describe our methodology and experimental setup for addressing the problem of implicit domain generalization of ERMs . We first start with a brief overview of the DomainBed benchmark and then describe the collection of models we train in an effort to measure generalization . 3.1 DomainBed Benchmark and Training Setup . DomainBed ( Gulrajani & Lopez-Paz , 2020 ) 1 is a recently released toolbox for domain generalization which provides implementations of various domain generalization algorithms such as Invariant Risk Minimization ( Arjovsky et al. , 2019 ) , Variance Risk Extrapolation ( Krueger et al. , 2020 ) , and metalearning based domain generalization ( Li et al. , 2017a ) in addition to unsupervised domain adaptation approaches ( Sun & Saenko , 2016 ; Ganin et al. , 2015 ; Yan et al. , 2020 ) . It also provides various datasets such as RotatedMNIST ( Ghifary et al. , 2015 ) , VLCS ( Fang et al. , 2013 ) , and PACS ( Li et al. , 2017b ) . These datasets are provided in the form of multiple environments e = { ( Xie , yie ) } Ni=1 , each 1https : //github.com/facebookresearch/DomainBed ( MIT License ) with a different training distribution pe ( X ) and ( generally ) the same conditional label distribution p ( y∣X ) . Domain generalization algorithms are then trained on a subset of source environments Esource = { ejsource } Mj=1 and tested on target environments Etarget = { e j target } Kj=1 . An intriguing result from the DomainBed benchmark is that standard ERM training can match or outperform more specialized domain generalization algorithms . We would like to understand this phenomenon better by characterizing properties that distinguish ERMs ( Empirical Risk Minimizers ) which perform well from those which do not . We follow Gulrajani & Lopez-Paz ( 2020 ) in combining all the training environments into one source dataset S = ⋃e∈Etrain e used to train our ERMs . Testing is performed independently on each target environment . We use the same network architectures as those used in the DomainBed paper ( see supplemental information for more details ) . Overall our networks span shallow convolutional neural networks with 386K parameters ( on the MNIST datasets ) to residual networks with 25 million parameters ( on VLCS and PACS ) . We prioritize a careful , exhaustive hyperparameter sweep with the above architectures , to obtain models with different levels of performance ( Gulrajani & Lopez-Paz , 2020 ) , over studying different architectures which is arguably a second order effect on this problem . Following Gulrajani & Lopez-Paz ( 2020 ) , for each combination of dataset and training environments , we pick 100 random hyperparameter settings of batch size , learning rate , weight decay , and dropout ( for resnet models ) . For both source S and target T , we hold out 50 % of the data for validation . Since no training happens on T this means that we can use 50 % of the data in T for computing generalization measures and the other 50 % for assessing the out-of-distribution error of an an empirical risk minimizer ĥ , which we term as T ( ĥ ) . Analogously , we denote the validation error of the model on the source domain as S ( ĥ ) .

Summary: The paper provides a large scale study of out-of-distribution generalization measures. It first provides a brief introduction of the core theoretical foundations of the field. Then the paper trains 12000 models on DomainBed Benchmarks to examine the predictive power of theory-based measures. The paper argues that theory based measure fail to accurately capture the OOD generalization behavior. Following the methodology of Jiang 2019 et al, the paper examine how various empirical measures (sharpness, entropy, Fisher, ...) predict OOD generalization behavior. In the end, the paper lists the predictive measures and examines their performance details.