Finding lost DG: Explaining domain generalization via model complexity

1 INTRODUCTION . Machine learning systems have shown exceptional performance on numerous tasks in computer vision and beyond . However performance drops rapidly when the standard assumption of i.i.d . training and testing data is violated . This domain-shift phenomenon occurs widely in many applications of machine learning ( Csurka , 2017 ; Zhou et al. , 2021 ; Koh et al. , 2021 ) , and often leads to disappointing results in practical machine learning deployments , since data ‘ in the wild ’ is almost inevitably different from reference training sets . Given the practical significance of this issue , a large number of methods have been proposed that aim to improve models ’ robustness to deployment under a different distribution than used for training ( Zhou et al. , 2021 ) , a problem setting known as domain generalisation ( DG ) . These methods span diverse approaches such as specialised neural architectures , data augmentation strategies , and regularisers . Nevertheless , the DG problem setting is difficult to model formally for principled derivation and theoretical analysis of algorithms , the target domain of interest is unobservable during training , and can not be directly approximated by the training domains due to unknown distribution shift . Therefore the majority of these existing approaches are based on poorly understood empirical heuristics . To make matters worse , a recent study by Gulrajani & Lopez-Paz ( 2021 ) assessed the state of DG research with a carefully conducted comparative evaluation of algorithms on a large benchmark suite under a common platform . They found that published methods were not as effective as claimed , and in particular reported that ‘ no existing method reliably beats a well tuned empirical risk minimization ( ERM ) baseline ’ . We argue that this negative result highlights the need for better theory in this area in order to understand why existing algorithms have such erratic performance , and to guide the development of principled algorithms that are more effective and reliable . To this end , our first contribution is to present an intuitive learning-theoretic bound for DG performance . Intuitively , while the held-out domain of interest is indeed unobservable during training , we can bound its performance using learning theoretic tools similar to the standard ones used to bound the performance on ( unobserved ) testing data given ( observed ) training data . In particular we show that the performance on a held out target domain is bounded by the performance on known source domains , plus two additional model complexity terms , that describe how much a model can possibly have overfitted to the training domains . This theoretical contribution leads to several insights . Firstly , our theory suggests that DG performance is governed by a trade-off between empirical risk and model complexity that is analogous to the corresponding and widely understood trade-off that explains generalisation in standard i.i.d . learning as an overfitting-underfitting trade-off ( Geman et al. , 1992 ) . Based on this , we hypothesise that performance variability is determined by implicit or explicit regularisation . That is , the plethora of different strategies available ( Zhou et al. , 2021 ) – from data-augmentation to specialised optimisers – actually affect DG performance by explicitly or implicitly choosing different fit-complexity trade-offs . We corroborate this hypothesis by evaluating a number of models in the DomainBed suite in terms of complexity , and showing that their apparently erratic performance in Gulrajani & Lopez-Paz ( 2021 ) is actually consistent with an explanation in terms of implied complexity . Practically , our analysis suggests that the model selection strategy ( Hastie et al. , 2009 ) is a factor in DG performance that is at least as important as the actual mechanism of model complexity control ( i.e. , Tuning of regularisation strength vs specific parametric design of regulariser ) . In particular , regularisation should be stronger when optimizing for future DG performance than when optimizing for performance on seen domains . Unfortunately , model complexity is hard to carefully control in deep learning due to the large number of relevant factors ( architecture , regularisers , implicit regularisation from optimiser , etc ) . Gulrajani & Lopez-Paz ( 2021 ) attempted to address this by hyper-parameter search in the DomainBed benchmark , but are hampered by the computational infeasibility of accurate hyper-parameter search . In this paper , we use linear models and off-the-shelf self-supervised features to demonstrate much more clearly how cross-domain performance depends on complexity . Specifically , our theoretical and empirical results show that , contrary to the conclusion of Gulrajani & Lopez-Paz ( 2021 ) , simple domain-wise cross-validation is a better objective to drive DG model selection . In summary , based on our new generalisation bound , and associated empirical analysis , our takehome messages are : ( i ) Model fit vs complexity trade-off is a key determinant of DG performance , that explains existing DG algorithm performance variability . ( ii ) The complexity control strategy used to determine bias-variance trade-off is crucial in practice , with peak DG performance achieved when optimizing model complexity based on domain-wise validation . ( iii ) Regularisation required for optimal DG is greater than for conventional optimization for within-domain performance . 2 RELATED WORK . Theoretical Analysis of the DG Setting and Algorithms The DG problem setting was first analysed in Blanchard et al . ( 2011 ) . Since then there have been some attempts to analyse DG algorithms from a generalisation bound perspective ( Muandet et al. , 2013 ; Blanchard et al. , 2021 ; Hu et al. , 2020 ; Albuquerque et al. , 2020 ; Rosenfeld et al. , 2021 ) . However these studies have theoretical results that are either restricted to specific model classes , such as kernel machines , or make strong assumptions about how the domains seen during training will resemble those seen at test time—e.g. , that all domains are convex combinations of a finite pre-determined set of prototypical domains . In contrast , our Rademacher complexity approach can be applied to a broad range of model classes ( including neural networks ) , and makes comparatively milder assumptions about the relationship between domains—i.e. , they are i.i.d . samples from another arbitrary distribution over domains . The majority of the existing work investigating the theoretical foundations of DG follow the initial formalisation of the domain generalisation problem put forth by Blanchard et al . ( 2011 ) , where the goal is to minimise the expected error over unseen domains . However , several recent works have also explored the idea of bounding the error on a single unseen domain with the most pathological distribution shift ( Janzing , 2019 ) . This type of analysis is typically rooted in methods from causal inference , rather than statistical learning theory . As a consequence , they are able to make stronger claims for the problems they address , but the scope of their analysis is necessarily limited to the scenarios where their assumptions about the underlying causal structures are valid . For example , Janzing ( 2019 ) provides bounds that assume problems conform to a specific class of structural equation models , and the analysis is performed under the assumption that infinite training data is available within each of the observed training domains . Throughout the work we address the stan- dard DG formalisation given by Blanchard et al . ( 2011 ) , where one is concerned with the expected performance of a model on domains sampled from some distribution over domains . Others rely on trying to link between domain adaptation objectives ( where target domains are observable for alignment to source domains ) and domain generalisation ( where target domains are not observable and thus can not correctly be used in a learning objective ) . Albuquerque et al . ( 2020 ) proceed by making assumptions on the structure of the distribution over possible domains ( i.e. , that it has support determined by the convex hull of a finite set of prototypical domains ) , which allows them to upper bound the domain alignment metric . Ye et al . ( 2021 ) provide a bound that depends on an unobservable domain distance quantity , which they then approximate in experiments using kernel density estimates . Rosenfeld et al . ( 2021 ) is another piece of work that theoretically investigates the generalisation of ERM in a DG setting . They deal with online DG , where each time-step corresponds to observing a new domain , and the learner must produce a new model capable of generalising to novel domains . Another point of difference between their work and the standard DG problem setting of Blanchard et al . ( 2011 ) is that the domain at each time-step is chosen by an adversary . They analyse this game for a finite number of time-steps , but they assume each domain has an infinite amount of data . They also put some limitations on the adversary : e.g. , it must choose a domain that is a convex combination of a finite number of pre-determined domains . In contrast , our theoretical analysis is in the more realistic setting where one has a finite amount of data per domain , and the domains we consider are not limited to convex combinations of a set of prototypical domains . Possibly the most similar work to our theoretical contributions is due to Ahuja et al . ( 2021 ) , who also provide learningtheoretic generalisation bounds for DG . However , their analysis only applies to finite hypothesis classes ( which does not include , e.g. , linear models or neural networks ) , whereas ours can be applied to any class amenable to analysis with Rademacher complexity . Empirical Analysis The main existing empirical analysis on DG is Gulrajani & Lopez-Paz ( 2021 ) , who compared several state of the art DG methods under a common evaluation and hyper-parameter tuning protocol called DomainBed . They ultimately defend Empirical Risk Minimization ( ERM ) over more sophisticated alternatives on the grounds that no competitor consistently beats it across the benchmark suite . We also broadly defend ERM , and build on the same benchmark , but differently we provide a much deeper analysis into when and why ERM works . More specifically : ( i ) We provide a new theoretical analysis of ERM ’ s generalisation quality unlike the prior purely empirical evaluation , ( ii ) We re-use the DomainBed benchmark to directly corroborate this theory under controlled conditions using linear models where model complexity can be tractably and accurately tuned . ( iii ) We use our complexity-based analysis to explain the previously erratic results of prior DomainBed competitors in terms of model complexity . ( iv ) We identify , and empirically validate , the preferred model selection criterion for DG , a point which was inconclusive in Gulrajani & Lopez-Paz ( 2021 ) . 3 BOUNDING RISK FOR DOMAIN GENERALIZATION . I.i.d . learning is concerned with learning a mapping from some input space X , to a label space Y , given data drawn from a distribution on X × Y . One aims to find a model , f∗ ∈ F , that minimises the expected loss ( also called risk ) on unseen data , f∗ = arg min f∈F Lp ( f ) , Lp ( f ) = E ( ~x , y ) ∼p [ ` ( f ( ~x ) , y ) ] , ( 1 ) where p is the data distribution and ` ( · , · ) is the loss function . In practice , we only have access to a finite set of data , S = { ( ~xi , yi ) } mi=1 , sampled i.i.d . from this distribution , so must minimise an empirical risk estimate , f̂ = arg min f∈F L̂p ( f ) , L̂p ( f ) = 1 m m∑ i=1 ` ( f ( ~xi ) , yi ) . ( 2 ) wherem is the number of training examples . One of the central focuses of statistical learning theory is to bound the difference between these two types of risk . For example , in the standard single domain setting this can be done via Lp ( f ) ≤ L̂p ( f ) + 2Rm ( F ) +O ( √ ln ( 1/δ ) m ) , ( 3 ) which holds with probability at least 1− δ , andR ( F ) is known as the empirical Rademacher complexity of the hypothesis class , F . This complexity term is defined as Rm ( F ) = E~σ [ sup f∈F 1 m m∑ i=1 σi ` ( f ( ~xi ) , yi ) ] . ( 4 ) For a hypothesis class consisting of norm-constrained linear classifiers , one can achieve the following upper bound on the empirical Rademacher complexity ( Shalev-Shwartz & Ben-David , 2014 ) , Rm ( F ) ≤ XB√ m , F = { ~x 7→ ~x · ~w : ‖~w‖2 ≤ B } , where we assume that each input , ~x , has a Euclidean norm of at mostX . There are a variety of ways to define hypothesis classes for neural networks , but most recent approaches take the view of fixing a particular architecture and specifying constraints on the norms of weights or distances they can move from their initialisations ( Bartlett et al. , 2017 ; Neyshabur et al. , 2019 ; Gouk et al. , 2021 ) . Domain generalisation . While standard i.i.d . learning assumes all data come from the same distribution , the DG problem setting assumes the existence of an environment E , of distributions p ( Blanchard et al. , 2011 ) . Note that we do not restrict what types of differences one could see between different domains sampled from E : for any two distributions in p , qsupp ( E ) it could be the case that either p ( ~x ) 6= q ( ~x ) , or p ( y|~x ) 6= q ( y|~x ) , or even that both types of distribution shift have occurred . The conceptually simplest—and often implicitly assumed—goal of DG methods is to minimise the expected risk across different distributions that could be sampled from the environment , LE ( f ) = Ep∼E [ Lp ( f ) ] . ( 5 ) This object is also the most commonly analysed idealised objective in the learning theory literature ( Blanchard et al. , 2011 ; Muandet et al. , 2013 ; Blanchard et al. , 2021 ; Rosenfeld et al. , 2021 ) , but other formulations also exist ( Arjovsky et al. , 2019 ) . As with the single domain learning problem , we only have access to an empirical estimate of the risk , L̂E ( f ) = 1 n n∑ j=1 L̂pj ( f ) , ( 6 ) where we assume for ease of exposition that all n domains have the same number of examples .

Summary: This paper considers the problem of domain generalization (DG), wherein predictors are trained on a related set of training domains and evaluated on an unseen test domain.  The authors first present a learning-theoretic bound on the performance of an average-case formulation for DG, and then present a set of experiments that consider the trade-off between complexity and out-of-distribution (OOD) performance.  The experiments indicate that such trade-offs exist for linear models, and there is also evidence that the trade-off persists for shallow neural networks.