Quantifying and Improving Transferability in Domain Generalization

1 Introduction . One of the cornerstone assumptions underlying the recent success of deep learning models is that the test data should share the same distribution as the training data . However , faced with ubiquitous distribution shifts in various real-world applications , such assumption hardly holds in practice . For example , a self-driving recognition system trained using data collected in the daytime may continually degrade its performance during nightfall . The system may also encounter weather or traffic conditions in a new city that never appear in the training set . In light of these potentially unseen scenarios , it is 1Code available at https : //github.com/Gordon-Guojun-Zhang/Transferability-NeurIPS2021 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of paramount importance that the trained model can generalize Out-Of-Distribution ( OOD ) : even if the target domain is not exactly the same as the source domain ( s ) , the learned model should hopefully behave robustly under slight distribution shift . To this end , one line of works focuses on learning the so-called invariant representations [ 2 , 16 , 57 , 58 ] . At a colloquial level , the goal here is to learn feature embeddings that lead to indistinguishable feature distributions from different domains . In practice , both the feature embeddings and the domain discriminators are often parametrized by neural networks , leading to an adversarial game between these two . Furthermore , in order to avoid degenerate solutions , the learned features are required to be informative about the output variable as well . This is enforced by placing a predictor over the features and minimize the corresponding supervised loss simultaneously [ 17 , 32 , 48 , 49 ] . Another line of recent works aims to learn features that can induce invariant predictors , first termed as the invariant risk minimization ( IRM ) [ 3 , 39 ] paradigm . Roughly speaking , the goal of IRM is to discover a feature embedding , upon which the optimal predictors , i.e. , the Bayes predictor , are invariant across the training domains . Again , at the same time , the features should be informative about the output variable as well . However , the optimization problem of IRM is rather difficult , and several follow-up works have proposed different relaxations to the original formulation [ 1 , 26 ] . Despite being extensively studied , both theoretical [ 41 , 59 ] and empirical [ 20 , 23 ] works have shown the insufficiency of existing algorithms for domain generalization ( DG ) . Methods based on invariant features ignore the potential shift in the marginal label distributions across domains [ 59 ] and the methods based on invariant predictors are not robust to covariate shift [ 26 ] . Perhaps surprisingly , empirical works have shown that with proper data augmentation and careful model tuning , the very basic algorithm of empirical risk minimization ( ERM ) demonstrates superior performance on domain generalization over existing methods on benchmark image datasets [ 20 , 23 ] . This sharp gap between theory and practice calls for a fundamental understanding of the following question : What kind of invariance should we look for , in order to ensure that a good model on source domains also achieves decent accuracy on a related target domain ? In this work we attempt to answer the above question by proposing a criterion for models to look at , dubbed as transferability , which asks for an invariance of the excess risks of a predictor across domains . Different from existing proposals of invariant features and invariant predictors , which seek to find feature embeddings that respectively induce invariant marginal and conditional distributions , our notion of transferability depends on the excess risk , hence it directly takes into account the joint distribution over both the features and the labels . We show how it can be used to naturally derive a new upper bound for the target error , and then we discuss how to estimate the transferability empirically with enough samples . Our definition also inspires a method that aims to find more transferable features via representation learning using adversarial training . Empirically , we perform experiments to measure the transferability of several existing algorithms , on both small and large scale datasets . We show that many algorithms , including ERM , are not quite transferable under the definition ( Fig . 1 , see more details in §5 ) : when we go away from the optimal classifier ( with distance δ in the parameter space ) , it could happen that the source accuracy remains high but the target accuracy drops significantly . This implies that during the training process , an existing algorithm may find a good source classifier with low target accuracy , hence violating the requirement for invariance of excess risks . In contrast , our algorithm is more transferable , and achieves consistent improvement over existing state-of-the-art algorithms , corroborating our findings . 2 What is Transferability ? . In this section we present our definition of transferability in the classification setting . The setup of domain generalization is the following : Settings and Notation Given n labeled source domains S1 , . . . , Sn , the problem of domain generalization is to learn a model from these source domains , in the hope that it performs well on an unseen target domain T that is “ similar ” to the source domains . Throughout the paper , we assume that both the source domains and the unseen target domain share the same input and output spaces , denoted as X and Y , respectively . For multi-class classification , the output space Y = [ K ] is a set of labels for multi-class classification . For binary classification , we consider Y = { −1 , +1 } . DenoteH as the hypothesis class . We define the classification error of a classifier h ∈ H on a domain D ( or S for source domains , or T for target domains ) as:2 D ( h ) = E ( x , y ) ∼D [ ` ( h ( x ) , y ) ] . ( 1 ) For ` ( h ( x ) , y ) = 1 ( h ( x ) 6= y ) , where 1 ( · ) is the usual indicator function , we use 0−1D ( h ) to denote it is the 0-1 loss . In domain generalization , we often have several source domains . For the ease of presentation , we only consider a single source domain in this section , and later extend to the general case in Section 5 . Given two domains , the source domain S and the target domain T , the task of domain generalization is to transfer a classifier h that performs well on S to T . We ask : how much of the success of h on S can be transferred to T ? Note that in order to evaluate the transferability from S to T , we need information from the target domain , similar to the test phase in traditional supervised learning . We believe a good criterion of transferability should satisfy the following properties : 1 . Quantifiable : the notion should be quantifiable and can be computed in practice ; 2 . Any near-optimal source classifier should be near-optimal on the target domain . 3 . If the two domains are similar , as measured by e.g. , total variation , then they are transferable to each other , but the converse may not be true . At first glance the second criterion above might seem too strong and restrictive . However , we argue that in the task of domain generalization , we only have labeled source data and there is no clue to distinguish a classifier from another if both of them perform equally well on the source domain . Based on the second property , we first propose the following definition of transferability : Definition 1 ( transferability ) . S is ( δS , δT ) H-transferable to T if for δS > 0 , there exists δT > 0 such that argmin ( S , δS ) H ⊆ argmin ( T , δT ) H , where : argmin ( D , δD ) H : = { h ∈ H : D ( h ) ≤ inf h∈H D ( h ) + δD } . In the literature the set argmin ( D , δD ) H is also known as a δD-minimal set [ 24 ] of D , which represents the near-optimal set of classifiers . Note that the δ-minimal set depends on the hypothesis classH . Throughout the paper , we omit the subscriptH in the definition when there is no confusion . Def . 1 says that near-optimal source classifiers are also near-optimal target classifiers . Furthermore , it is easy to verify that our transferability is transitive : if S is ( δS , δP ) -transferable to P , and P is ( δP , δT ) -transferable to T , then S is ( δS , δT ) -transferable to T . Next we define transfer measures , which we will show to be equivalent with Def . 1 in Prop . 5 . Definition 2 ( quantifiable transfer measures ) . Given some Γ ⊆ H , ∗S : = infh∈Γ S ( h ) and ∗T : = infh∈Γ T ( h ) we define the one-sided transfer measure , symmetric transfer measure and the realizable transfer measure respectively as : TΓ ( S‖T ) : = sup h∈Γ T ( h ) − ∗T − ( S ( h ) − ∗S ) , ( 2 ) TΓ ( S , T ) : = max { TΓ ( S‖T ) , TΓ ( T ‖S ) } = sup h∈Γ | S ( h ) − ∗S − ( T ( h ) − ∗T ) | , ( 3 ) TrΓ ( S , T ) : = sup h∈Γ | S ( h ) − T ( h ) | . ( 4 ) The distinction between Γ andH will become apparent in Prop . 5 . Note that the one-sided transfer measure is not symmetric . If we want the two domains S and T to be mutually transferable to each other , we can use the symmetric transfer measure . We call both quantities as transfer measures . Furthermore , the symmetric transfer measure reduces to ( 4 ) in the realizable case when ∗S = ∗ T = 0 . In statistical learning theory , D ( h ) − ∗D is often known as an excess risk [ 24 ] , which is the relative error compared to the optimal classifier . The transfer measures can thus be represented with the difference of excess risks . With Def . 2 , we can immediately obtain the following result that upper bounds the target error : 2Throughout the paper , we will use the terms domain and distribution interchangeably . Proposition 3 ( target error bound ) . Given Γ ⊆ H , for any h ∈ Γ , the target error is bounded by : T ( h ) ≤ S ( h ) + ∗T − ∗S + TΓ ( S‖T ) ≤ S ( h ) + ∗T − ∗S + TΓ ( S , T ) . ( 5 ) The first error bound of such type for a target domain uses H-divergence [ 8 , 9 , 12 ] for binary classification ( or more rigorously , the H∆H-divergence ) . The main difference between ours and H-divergence is that H-divergence only concerns about the marginal input distributions , whereas the transfer measures depend on the joint distributions over both the inputs and the labels . We note that Proposition 3 is general and works in the multi-class case as well . Moreover , even in the binary classification case we can prove that our Proposition 3 is tighter thanH-divergence ( see Proposition 27 in the appendix ) . In practice we may not know the optimal errors . In this case , we can use the realizable transfer measure to upper bound the symmetric transfer measure ( note that ∗S or ∗ T may not be zero ) : Proposition 4 . For Γ ⊆ H and domains S , T we have : TΓ ( S , T ) ≤ 2TrΓ ( S , T ) . Since Def . 1 essentially asks that the excess risks of approximately optimal classifiers on the source domain are comparable between the source and target domains , we can show that Def . 1 and Def . 2 are equivalent if Γ is a δ-minimal set : Proposition 5 ( equivalence between transferability and transfer measures ) . Let δS > 0 and Γ = argmin ( S , δS ) and suppose infh∈Γ T ( h ) = infh∈H T ( h ) . If TΓ ( S‖T ) ≤ δ or TΓ ( S , T ) ≤ δ , then S is ( δS , δ + δS ) -transferable to T . Furthermore , if S is ( δS , δT ) -transferable to T , then TΓ ( S‖T ) ≤ δT and TΓ ( S , T ) ≤ max { δS , δT } . In Prop . 5 , we do not require Γ = H since it is unnecessary to impose that all classifiers inH have similar excess risks on source and target domains . Instead , we only constrain Γ to be a δ-minimal set , i.e. , Γ includes approximately optimal classifiers of S . See also Example 8 . An additional assumption is that Γ also includes the optimal classifier of T which can be ensured by controlling δS .

Summary: Authors introduced the notion of transferability between pairs of domains (i.e. joint distributions over input x output spaces) along with a discrepancy measure, which can be used to assess how dissimilar are two data sources relative to a given class of predictors. Such a quantity may be useful in either assessing how well is a trained model likely to perform on new data, or as a training signal so that more preference is given to predictors able to generalize out of the training data distribution. The main strength of this work in my opinion is the fact that the proposed transferability measure accounts for the joint distributions over data and labels. However, while I appreciate the depth of the discussion and results in terms of the properties of the proposal, I believe a key missing discussion is the effect that commonly used sets of assumptions have in the proposed transferability measure. Moreover, the evaluation lacks in evidencing the benefits of the proposal relative to current strategies. Please refer to the following for further details.