Quantifying and Improving Transferability in Domain Generalization

1 Introduction . One of the cornerstone assumptions underlying the recent success of deep learning models is that the test data should share the same distribution as the training data . However , faced with ubiquitous distribution shifts in various real-world applications , such assumption hardly holds in practice . For example , a self-driving recognition system trained using data collected in the daytime may continually degrade its performance during nightfall . The system may also encounter weather or traffic conditions in a new city that never appear in the training set . In light of these potentially unseen scenarios , it is 1Code available at https : //github.com/Gordon-Guojun-Zhang/Transferability-NeurIPS2021 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of paramount importance that the trained model can generalize Out-Of-Distribution ( OOD ) : even if the target domain is not exactly the same as the source domain ( s ) , the learned model should hopefully behave robustly under slight distribution shift . To this end , one line of works focuses on learning the so-called invariant representations [ 2 , 16 , 57 , 58 ] . At a colloquial level , the goal here is to learn feature embeddings that lead to indistinguishable feature distributions from different domains . In practice , both the feature embeddings and the domain discriminators are often parametrized by neural networks , leading to an adversarial game between these two . Furthermore , in order to avoid degenerate solutions , the learned features are required to be informative about the output variable as well . This is enforced by placing a predictor over the features and minimize the corresponding supervised loss simultaneously [ 17 , 32 , 48 , 49 ] . Another line of recent works aims to learn features that can induce invariant predictors , first termed as the invariant risk minimization ( IRM ) [ 3 , 39 ] paradigm . Roughly speaking , the goal of IRM is to discover a feature embedding , upon which the optimal predictors , i.e. , the Bayes predictor , are invariant across the training domains . Again , at the same time , the features should be informative about the output variable as well . However , the optimization problem of IRM is rather difficult , and several follow-up works have proposed different relaxations to the original formulation [ 1 , 26 ] . Despite being extensively studied , both theoretical [ 41 , 59 ] and empirical [ 20 , 23 ] works have shown the insufficiency of existing algorithms for domain generalization ( DG ) . Methods based on invariant features ignore the potential shift in the marginal label distributions across domains [ 59 ] and the methods based on invariant predictors are not robust to covariate shift [ 26 ] . Perhaps surprisingly , empirical works have shown that with proper data augmentation and careful model tuning , the very basic algorithm of empirical risk minimization ( ERM ) demonstrates superior performance on domain generalization over existing methods on benchmark image datasets [ 20 , 23 ] . This sharp gap between theory and practice calls for a fundamental understanding of the following question : What kind of invariance should we look for , in order to ensure that a good model on source domains also achieves decent accuracy on a related target domain ? In this work we attempt to answer the above question by proposing a criterion for models to look at , dubbed as transferability , which asks for an invariance of the excess risks of a predictor across domains . Different from existing proposals of invariant features and invariant predictors , which seek to find feature embeddings that respectively induce invariant marginal and conditional distributions , our notion of transferability depends on the excess risk , hence it directly takes into account the joint distribution over both the features and the labels . We show how it can be used to naturally derive a new upper bound for the target error , and then we discuss how to estimate the transferability empirically with enough samples . Our definition also inspires a method that aims to find more transferable features via representation learning using adversarial training . Empirically , we perform experiments to measure the transferability of several existing algorithms , on both small and large scale datasets . We show that many algorithms , including ERM , are not quite transferable under the definition ( Fig . 1 , see more details in §5 ) : when we go away from the optimal classifier ( with distance δ in the parameter space ) , it could happen that the source accuracy remains high but the target accuracy drops significantly . This implies that during the training process , an existing algorithm may find a good source classifier with low target accuracy , hence violating the requirement for invariance of excess risks . In contrast , our algorithm is more transferable , and achieves consistent improvement over existing state-of-the-art algorithms , corroborating our findings . 2 What is Transferability ? . In this section we present our definition of transferability in the classification setting . The setup of domain generalization is the following : Settings and Notation Given n labeled source domains S1 , . . . , Sn , the problem of domain generalization is to learn a model from these source domains , in the hope that it performs well on an unseen target domain T that is “ similar ” to the source domains . Throughout the paper , we assume that both the source domains and the unseen target domain share the same input and output spaces , denoted as X and Y , respectively . For multi-class classification , the output space Y = [ K ] is a set of labels for multi-class classification . For binary classification , we consider Y = { −1 , +1 } . DenoteH as the hypothesis class . We define the classification error of a classifier h ∈ H on a domain D ( or S for source domains , or T for target domains ) as:2 D ( h ) = E ( x , y ) ∼D [ ` ( h ( x ) , y ) ] . ( 1 ) For ` ( h ( x ) , y ) = 1 ( h ( x ) 6= y ) , where 1 ( · ) is the usual indicator function , we use 0−1D ( h ) to denote it is the 0-1 loss . In domain generalization , we often have several source domains . For the ease of presentation , we only consider a single source domain in this section , and later extend to the general case in Section 5 . Given two domains , the source domain S and the target domain T , the task of domain generalization is to transfer a classifier h that performs well on S to T . We ask : how much of the success of h on S can be transferred to T ? Note that in order to evaluate the transferability from S to T , we need information from the target domain , similar to the test phase in traditional supervised learning . We believe a good criterion of transferability should satisfy the following properties : 1 . Quantifiable : the notion should be quantifiable and can be computed in practice ; 2 . Any near-optimal source classifier should be near-optimal on the target domain . 3 . If the two domains are similar , as measured by e.g. , total variation , then they are transferable to each other , but the converse may not be true . At first glance the second criterion above might seem too strong and restrictive . However , we argue that in the task of domain generalization , we only have labeled source data and there is no clue to distinguish a classifier from another if both of them perform equally well on the source domain . Based on the second property , we first propose the following definition of transferability : Definition 1 ( transferability ) . S is ( δS , δT ) H-transferable to T if for δS > 0 , there exists δT > 0 such that argmin ( S , δS ) H ⊆ argmin ( T , δT ) H , where : argmin ( D , δD ) H : = { h ∈ H : D ( h ) ≤ inf h∈H D ( h ) + δD } . In the literature the set argmin ( D , δD ) H is also known as a δD-minimal set [ 24 ] of D , which represents the near-optimal set of classifiers . Note that the δ-minimal set depends on the hypothesis classH . Throughout the paper , we omit the subscriptH in the definition when there is no confusion . Def . 1 says that near-optimal source classifiers are also near-optimal target classifiers . Furthermore , it is easy to verify that our transferability is transitive : if S is ( δS , δP ) -transferable to P , and P is ( δP , δT ) -transferable to T , then S is ( δS , δT ) -transferable to T . Next we define transfer measures , which we will show to be equivalent with Def . 1 in Prop . 5 . Definition 2 ( quantifiable transfer measures ) . Given some Γ ⊆ H , ∗S : = infh∈Γ S ( h ) and ∗T : = infh∈Γ T ( h ) we define the one-sided transfer measure , symmetric transfer measure and the realizable transfer measure respectively as : TΓ ( S‖T ) : = sup h∈Γ T ( h ) − ∗T − ( S ( h ) − ∗S ) , ( 2 ) TΓ ( S , T ) : = max { TΓ ( S‖T ) , TΓ ( T ‖S ) } = sup h∈Γ | S ( h ) − ∗S − ( T ( h ) − ∗T ) | , ( 3 ) TrΓ ( S , T ) : = sup h∈Γ | S ( h ) − T ( h ) | . ( 4 ) The distinction between Γ andH will become apparent in Prop . 5 . Note that the one-sided transfer measure is not symmetric . If we want the two domains S and T to be mutually transferable to each other , we can use the symmetric transfer measure . We call both quantities as transfer measures . Furthermore , the symmetric transfer measure reduces to ( 4 ) in the realizable case when ∗S = ∗ T = 0 . In statistical learning theory , D ( h ) − ∗D is often known as an excess risk [ 24 ] , which is the relative error compared to the optimal classifier . The transfer measures can thus be represented with the difference of excess risks . With Def . 2 , we can immediately obtain the following result that upper bounds the target error : 2Throughout the paper , we will use the terms domain and distribution interchangeably . Proposition 3 ( target error bound ) . Given Γ ⊆ H , for any h ∈ Γ , the target error is bounded by : T ( h ) ≤ S ( h ) + ∗T − ∗S + TΓ ( S‖T ) ≤ S ( h ) + ∗T − ∗S + TΓ ( S , T ) . ( 5 ) The first error bound of such type for a target domain uses H-divergence [ 8 , 9 , 12 ] for binary classification ( or more rigorously , the H∆H-divergence ) . The main difference between ours and H-divergence is that H-divergence only concerns about the marginal input distributions , whereas the transfer measures depend on the joint distributions over both the inputs and the labels . We note that Proposition 3 is general and works in the multi-class case as well . Moreover , even in the binary classification case we can prove that our Proposition 3 is tighter thanH-divergence ( see Proposition 27 in the appendix ) . In practice we may not know the optimal errors . In this case , we can use the realizable transfer measure to upper bound the symmetric transfer measure ( note that ∗S or ∗ T may not be zero ) : Proposition 4 . For Γ ⊆ H and domains S , T we have : TΓ ( S , T ) ≤ 2TrΓ ( S , T ) . Since Def . 1 essentially asks that the excess risks of approximately optimal classifiers on the source domain are comparable between the source and target domains , we can show that Def . 1 and Def . 2 are equivalent if Γ is a δ-minimal set : Proposition 5 ( equivalence between transferability and transfer measures ) . Let δS > 0 and Γ = argmin ( S , δS ) and suppose infh∈Γ T ( h ) = infh∈H T ( h ) . If TΓ ( S‖T ) ≤ δ or TΓ ( S , T ) ≤ δ , then S is ( δS , δ + δS ) -transferable to T . Furthermore , if S is ( δS , δT ) -transferable to T , then TΓ ( S‖T ) ≤ δT and TΓ ( S , T ) ≤ max { δS , δT } . In Prop . 5 , we do not require Γ = H since it is unnecessary to impose that all classifiers inH have similar excess risks on source and target domains . Instead , we only constrain Γ to be a δ-minimal set , i.e. , Γ includes approximately optimal classifiers of S . See also Example 8 . An additional assumption is that Γ also includes the optimal classifier of T which can be ensured by controlling δS .

Summary: This paper aims to provide a formulation of transferability. The main intuition is that any near-optimal source classifier should be also near-optimal on the target domain. So the authors quantify transferability by measuring the relation between near-optimal classifiers in source and target domains and derive genralization bounds based on this. In addition, new algorithm is proposed based on the intuition.