Finding lost DG: Explaining domain generalization via model complexity

1 INTRODUCTION . Machine learning systems have shown exceptional performance on numerous tasks in computer vision and beyond . However performance drops rapidly when the standard assumption of i.i.d . training and testing data is violated . This domain-shift phenomenon occurs widely in many applications of machine learning ( Csurka , 2017 ; Zhou et al. , 2021 ; Koh et al. , 2021 ) , and often leads to disappointing results in practical machine learning deployments , since data ‘ in the wild ’ is almost inevitably different from reference training sets . Given the practical significance of this issue , a large number of methods have been proposed that aim to improve models ’ robustness to deployment under a different distribution than used for training ( Zhou et al. , 2021 ) , a problem setting known as domain generalisation ( DG ) . These methods span diverse approaches such as specialised neural architectures , data augmentation strategies , and regularisers . Nevertheless , the DG problem setting is difficult to model formally for principled derivation and theoretical analysis of algorithms , the target domain of interest is unobservable during training , and can not be directly approximated by the training domains due to unknown distribution shift . Therefore the majority of these existing approaches are based on poorly understood empirical heuristics . To make matters worse , a recent study by Gulrajani & Lopez-Paz ( 2021 ) assessed the state of DG research with a carefully conducted comparative evaluation of algorithms on a large benchmark suite under a common platform . They found that published methods were not as effective as claimed , and in particular reported that ‘ no existing method reliably beats a well tuned empirical risk minimization ( ERM ) baseline ’ . We argue that this negative result highlights the need for better theory in this area in order to understand why existing algorithms have such erratic performance , and to guide the development of principled algorithms that are more effective and reliable . To this end , our first contribution is to present an intuitive learning-theoretic bound for DG performance . Intuitively , while the held-out domain of interest is indeed unobservable during training , we can bound its performance using learning theoretic tools similar to the standard ones used to bound the performance on ( unobserved ) testing data given ( observed ) training data . In particular we show that the performance on a held out target domain is bounded by the performance on known source domains , plus two additional model complexity terms , that describe how much a model can possibly have overfitted to the training domains . This theoretical contribution leads to several insights . Firstly , our theory suggests that DG performance is governed by a trade-off between empirical risk and model complexity that is analogous to the corresponding and widely understood trade-off that explains generalisation in standard i.i.d . learning as an overfitting-underfitting trade-off ( Geman et al. , 1992 ) . Based on this , we hypothesise that performance variability is determined by implicit or explicit regularisation . That is , the plethora of different strategies available ( Zhou et al. , 2021 ) – from data-augmentation to specialised optimisers – actually affect DG performance by explicitly or implicitly choosing different fit-complexity trade-offs . We corroborate this hypothesis by evaluating a number of models in the DomainBed suite in terms of complexity , and showing that their apparently erratic performance in Gulrajani & Lopez-Paz ( 2021 ) is actually consistent with an explanation in terms of implied complexity . Practically , our analysis suggests that the model selection strategy ( Hastie et al. , 2009 ) is a factor in DG performance that is at least as important as the actual mechanism of model complexity control ( i.e. , Tuning of regularisation strength vs specific parametric design of regulariser ) . In particular , regularisation should be stronger when optimizing for future DG performance than when optimizing for performance on seen domains . Unfortunately , model complexity is hard to carefully control in deep learning due to the large number of relevant factors ( architecture , regularisers , implicit regularisation from optimiser , etc ) . Gulrajani & Lopez-Paz ( 2021 ) attempted to address this by hyper-parameter search in the DomainBed benchmark , but are hampered by the computational infeasibility of accurate hyper-parameter search . In this paper , we use linear models and off-the-shelf self-supervised features to demonstrate much more clearly how cross-domain performance depends on complexity . Specifically , our theoretical and empirical results show that , contrary to the conclusion of Gulrajani & Lopez-Paz ( 2021 ) , simple domain-wise cross-validation is a better objective to drive DG model selection . In summary , based on our new generalisation bound , and associated empirical analysis , our takehome messages are : ( i ) Model fit vs complexity trade-off is a key determinant of DG performance , that explains existing DG algorithm performance variability . ( ii ) The complexity control strategy used to determine bias-variance trade-off is crucial in practice , with peak DG performance achieved when optimizing model complexity based on domain-wise validation . ( iii ) Regularisation required for optimal DG is greater than for conventional optimization for within-domain performance . 2 RELATED WORK . Theoretical Analysis of the DG Setting and Algorithms The DG problem setting was first analysed in Blanchard et al . ( 2011 ) . Since then there have been some attempts to analyse DG algorithms from a generalisation bound perspective ( Muandet et al. , 2013 ; Blanchard et al. , 2021 ; Hu et al. , 2020 ; Albuquerque et al. , 2020 ; Rosenfeld et al. , 2021 ) . However these studies have theoretical results that are either restricted to specific model classes , such as kernel machines , or make strong assumptions about how the domains seen during training will resemble those seen at test time—e.g. , that all domains are convex combinations of a finite pre-determined set of prototypical domains . In contrast , our Rademacher complexity approach can be applied to a broad range of model classes ( including neural networks ) , and makes comparatively milder assumptions about the relationship between domains—i.e. , they are i.i.d . samples from another arbitrary distribution over domains . The majority of the existing work investigating the theoretical foundations of DG follow the initial formalisation of the domain generalisation problem put forth by Blanchard et al . ( 2011 ) , where the goal is to minimise the expected error over unseen domains . However , several recent works have also explored the idea of bounding the error on a single unseen domain with the most pathological distribution shift ( Janzing , 2019 ) . This type of analysis is typically rooted in methods from causal inference , rather than statistical learning theory . As a consequence , they are able to make stronger claims for the problems they address , but the scope of their analysis is necessarily limited to the scenarios where their assumptions about the underlying causal structures are valid . For example , Janzing ( 2019 ) provides bounds that assume problems conform to a specific class of structural equation models , and the analysis is performed under the assumption that infinite training data is available within each of the observed training domains . Throughout the work we address the stan- dard DG formalisation given by Blanchard et al . ( 2011 ) , where one is concerned with the expected performance of a model on domains sampled from some distribution over domains . Others rely on trying to link between domain adaptation objectives ( where target domains are observable for alignment to source domains ) and domain generalisation ( where target domains are not observable and thus can not correctly be used in a learning objective ) . Albuquerque et al . ( 2020 ) proceed by making assumptions on the structure of the distribution over possible domains ( i.e. , that it has support determined by the convex hull of a finite set of prototypical domains ) , which allows them to upper bound the domain alignment metric . Ye et al . ( 2021 ) provide a bound that depends on an unobservable domain distance quantity , which they then approximate in experiments using kernel density estimates . Rosenfeld et al . ( 2021 ) is another piece of work that theoretically investigates the generalisation of ERM in a DG setting . They deal with online DG , where each time-step corresponds to observing a new domain , and the learner must produce a new model capable of generalising to novel domains . Another point of difference between their work and the standard DG problem setting of Blanchard et al . ( 2011 ) is that the domain at each time-step is chosen by an adversary . They analyse this game for a finite number of time-steps , but they assume each domain has an infinite amount of data . They also put some limitations on the adversary : e.g. , it must choose a domain that is a convex combination of a finite number of pre-determined domains . In contrast , our theoretical analysis is in the more realistic setting where one has a finite amount of data per domain , and the domains we consider are not limited to convex combinations of a set of prototypical domains . Possibly the most similar work to our theoretical contributions is due to Ahuja et al . ( 2021 ) , who also provide learningtheoretic generalisation bounds for DG . However , their analysis only applies to finite hypothesis classes ( which does not include , e.g. , linear models or neural networks ) , whereas ours can be applied to any class amenable to analysis with Rademacher complexity . Empirical Analysis The main existing empirical analysis on DG is Gulrajani & Lopez-Paz ( 2021 ) , who compared several state of the art DG methods under a common evaluation and hyper-parameter tuning protocol called DomainBed . They ultimately defend Empirical Risk Minimization ( ERM ) over more sophisticated alternatives on the grounds that no competitor consistently beats it across the benchmark suite . We also broadly defend ERM , and build on the same benchmark , but differently we provide a much deeper analysis into when and why ERM works . More specifically : ( i ) We provide a new theoretical analysis of ERM ’ s generalisation quality unlike the prior purely empirical evaluation , ( ii ) We re-use the DomainBed benchmark to directly corroborate this theory under controlled conditions using linear models where model complexity can be tractably and accurately tuned . ( iii ) We use our complexity-based analysis to explain the previously erratic results of prior DomainBed competitors in terms of model complexity . ( iv ) We identify , and empirically validate , the preferred model selection criterion for DG , a point which was inconclusive in Gulrajani & Lopez-Paz ( 2021 ) . 3 BOUNDING RISK FOR DOMAIN GENERALIZATION . I.i.d . learning is concerned with learning a mapping from some input space X , to a label space Y , given data drawn from a distribution on X × Y . One aims to find a model , f∗ ∈ F , that minimises the expected loss ( also called risk ) on unseen data , f∗ = arg min f∈F Lp ( f ) , Lp ( f ) = E ( ~x , y ) ∼p [ ` ( f ( ~x ) , y ) ] , ( 1 ) where p is the data distribution and ` ( · , · ) is the loss function . In practice , we only have access to a finite set of data , S = { ( ~xi , yi ) } mi=1 , sampled i.i.d . from this distribution , so must minimise an empirical risk estimate , f̂ = arg min f∈F L̂p ( f ) , L̂p ( f ) = 1 m m∑ i=1 ` ( f ( ~xi ) , yi ) . ( 2 ) wherem is the number of training examples . One of the central focuses of statistical learning theory is to bound the difference between these two types of risk . For example , in the standard single domain setting this can be done via Lp ( f ) ≤ L̂p ( f ) + 2Rm ( F ) +O ( √ ln ( 1/δ ) m ) , ( 3 ) which holds with probability at least 1− δ , andR ( F ) is known as the empirical Rademacher complexity of the hypothesis class , F . This complexity term is defined as Rm ( F ) = E~σ [ sup f∈F 1 m m∑ i=1 σi ` ( f ( ~xi ) , yi ) ] . ( 4 ) For a hypothesis class consisting of norm-constrained linear classifiers , one can achieve the following upper bound on the empirical Rademacher complexity ( Shalev-Shwartz & Ben-David , 2014 ) , Rm ( F ) ≤ XB√ m , F = { ~x 7→ ~x · ~w : ‖~w‖2 ≤ B } , where we assume that each input , ~x , has a Euclidean norm of at mostX . There are a variety of ways to define hypothesis classes for neural networks , but most recent approaches take the view of fixing a particular architecture and specifying constraints on the norms of weights or distances they can move from their initialisations ( Bartlett et al. , 2017 ; Neyshabur et al. , 2019 ; Gouk et al. , 2021 ) . Domain generalisation . While standard i.i.d . learning assumes all data come from the same distribution , the DG problem setting assumes the existence of an environment E , of distributions p ( Blanchard et al. , 2011 ) . Note that we do not restrict what types of differences one could see between different domains sampled from E : for any two distributions in p , qsupp ( E ) it could be the case that either p ( ~x ) 6= q ( ~x ) , or p ( y|~x ) 6= q ( y|~x ) , or even that both types of distribution shift have occurred . The conceptually simplest—and often implicitly assumed—goal of DG methods is to minimise the expected risk across different distributions that could be sampled from the environment , LE ( f ) = Ep∼E [ Lp ( f ) ] . ( 5 ) This object is also the most commonly analysed idealised objective in the learning theory literature ( Blanchard et al. , 2011 ; Muandet et al. , 2013 ; Blanchard et al. , 2021 ; Rosenfeld et al. , 2021 ) , but other formulations also exist ( Arjovsky et al. , 2019 ) . As with the single domain learning problem , we only have access to an empirical estimate of the risk , L̂E ( f ) = 1 n n∑ j=1 L̂pj ( f ) , ( 6 ) where we assume for ease of exposition that all n domains have the same number of examples .

Summary: This paper presents a theory of domain generalization based on statistical learning theory (Rademacher complexity) and demonstrates a trade-off between training loss and model complexity. They show that existing methods are actually controlling the model complexity. Based on the analysis, the authors argue that proper model selection is critical for complexity control. Instead of hyper-parameter search, they propose to use domain-wise cross-validation as the model selection strategy. Experiments on the DomainBed benchmark show the effectiveness of the proposed method.