The Frechet Distance of training and test distribution predicts the generalization gap

1 INTRODUCTION . Imagine there are two students who are studying for an exam . Student A studies by diligently learning the class material by heart . Student B studies by learning the underlying reasons for why things are the way they are . Come test day , student A is only able to answer test questions that are very similar to the class material while student B has no trouble answering different looking questions that follow the same reasoning . Distilled from this example , we note there is a trade-off between how “ well ” a student studied , i.e. , how indifferent the student is to receiving exercise or test questions , and how close the test questions are to the exercise questions . While most machine learning work studies the generalization error , i.e. , the error when testing on different samples from the same distribution , we do not take the match of train and test distribution as given . In fact , it appears that the distance between train and test distribution may be critical for successful “ generalization ” . Following a similar line of thought , Uguroglu & Carbonell ( 2011 ) devised a distribution measurement to select only features that do not vary from one domain to another . In contrast , we are interested in linking performance directly to the distance between train and test distribution . Invariance to distribution shifts : We say that a function is invariant to a given input perturbation when the corresponding output does not change with the perturbation . This is desirable when trying to achieve robustness to irrelevant data variations which are called nuisances ( Achille & Soatto , 2018 ) . As outlined by Achille & Soatto ( 2018 ) ; Censi & Murray ( 2011 ) , the “ optimal ” learned function from input to output is maximally invariant to all data variations that do not contain information about the output . To the extent to which a learner reacts to such nuisance variations , which carry no information about the output , it will incur a performance change in expectation . The difficulty lies in knowing what can be ignored and what can not . Similarity between training and test distribution : Another strategy would be to ensure that the training and test distribution match which has been investigated in a number of diverse settings ( Tzeng et al. , 2017 ; Arjovsky et al. , 2017 ) . Variations of this theme were encountered by Zhang et al . ( 2016 ) , where they show that networks are able to fit random labels perfectly , yet understandably fail to generalize to the test set of the correct label distribution . Contribution : We frame the learning problem of training on a training set then shifting to test on a test set as a transfer learning problem that goes beyond the usual generalization which assumes the same distribution for training and testing . Based on our analysis , we aim to explicitly express the trade-off of the distance between both data set distributions given by the Wasserstein distance , which measures how close training and testing distribution are , and how invariant the learned function is to the training to test distribution shift . Joined together this is expressed via the Wasserstein distance between training and test samples embedded in the feature space of the learned function . Our experiments show a strong negative linear correlation between the distribution distances and network performance . This corroborates the notion that as long as the true function class is not found , it is best to ensure that training and test distribution are close . While this conclusion seems intuitive , it may often be overlooked in the hunt for more data . 2 RELATED WORK . The trade-off and effect on performance between invariance to distribution shifts and the magnitude of such distribution shifts has been an active field of research , albeit mostly implicitly . Given the correct invariance , changes in distribution do not have a strong effect and learning “ generalizes ” . What has not been investigated so far , to the best of our knowledge , is a more systematic treatment of measuring distribution distances and invariances of learned functions . These assumptions currently remain mainly implicit in the literature . Invariance : Invariance is a key component of Deep Learning ( Goodfellow et al. , 2009 ; Achille & Soatto , 2018 ) with many works focusing on increasing invariance by incorporating structure into their models ( LeCun et al. , 1998 ; Bruna & Mallat , 2012 ; Cohen et al. , 2018 ; Kumar et al. , 2017 ; van der Wilk et al. , 2018 ) . Other works apply GANs ( Goodfellow et al. , 2014 ) , e.g. , to domain adaptation problems ( Tzeng et al. , 2017 ; Hoffman et al. , 2017 ) , to map source and target distributions to the same space while being invariant to occurring distribution shifts . As stand in for many similar works , Tzeng et al . ( 2015 ) employ invariance to improve visual adaptation to domain changes . In this work , rather than trying to beat a benchmark , we aim to establish the quantitative and qualitative trade-offs between invariance and distribution shift . In contrast to adapting the structure of a model , many works achieve invariance through data augmentation ( Krizhevsky et al. , 2017b ; Shorten & Khoshgoftaar , 2019 ; Hernández-Garcı́a et al. , 2019 ) . While most augmentations are handpicked based on domain knowledge , automated methods based on GANs have been successfully applied to data augmentation ( Antoniou et al. , 2017 ; Kim et al. , 2017 ) . Any of these transformations can be considered a canonization ( Censi & Murray , 2011 ) , i.e. , a generalization of data normalization to group actions , by removing unwanted nuisances to increase robustness . Many other techniques such as regularization Leen ( 1995 ) , and dimension reduction can also be considered part of the umbrella of invariance learning ( Jolliffe & Springer-Verlag , 2002 ; Hadsell et al. , 2006 ; Saul et al. , 2006 ) . In this work , we simply take the inherent invariance of learned embeddings to approximate the invariance of our function approximator and focus on the distributional shift from train to test distribution . Distribution shifts : Shifts in data set distribution ( Storkey , 2009 ; Quionero-Candela et al. , 2009 ) can take a number of forms where either the input distribution p ( x ) , the target distribution p ( y ) or a combination thereof changes . Likewise imbalanced datasets ( Japkowicz & Stephen , 2002 ) , methods of over and undersampling ( Bowyer et al. , 2011 ) , and domain shifts ( Zhang et al. , 2013 ) can be described in this framework . By knowing how the distribution changes from one setting to another , Zanardi et al . ( 2019 ) were able to define an operating domain for different vision sensors and employ this for a semi-supervised learning setup . Part of evaluating distribution shifts is measuring these probability distribution distances . A popular use for this is the Maximum Mean Discrepancy ( MMD ) distance ( Borgwardt et al. , 2006 ) that has found application in anomaly detection ( Zou et al. , 2017 ) and two-sample testing ( Gretton et al. , 2012 ) . Other distances such as the Wasserstein distance have made an impact in computational graphics modeling ( Solomon et al. , 2015 ) and in generating realistic-looking synthetic images using GANs ( Arjovsky et al. , 2017 ) . In this work , we apply the Wasserstein distance as a powerful tool to investigate the performance drop when moving from train to test samples . 3 INSIGHTS FROM SMALL-SCALE EXPERIMENTS . Before jumping ahead , we gain an intuitive understanding of the behavior of a neural network in the case of mismatched train and test distributions . In particular we are interested in the effect of distribution shift from training to test distribution and the effect of invariance of the learned function to such changes . Looking at this through a different lens , we also investigate what impact the type of function we are approximating , and the type of function we are approximating with can have . First , we run experiments detailing changes in the input distribution as illustrated in the top row of Fig . 1 . From left to right , we increase input values x by adding a scalar α . As can be readily observed , the quality of the function approximation deteriorates under such a distribution shift . If on the other hand , we normalize the input data by subtracting the mean of the distribution , the function approximation remains unchanged as seen in the second row of Fig . 1 . This added invariance through normalization made the function approximation robust to the given distribution shift . Secondly , we run function approximation experiments in which we have a training set in one interval and a test set in another . We focus on what happens when the true function class is the same as the one used for approximation and what happens when this is not the case . We observe in the third row of Fig . 1 that when the correct function is found , as in the second and third image , the approximated function works even in a domain in which it was not trained . On the other hand , the first plot of the third row and the plots in the fourth row show that if the true function is not found exactly , then outside of the training domain errors accumulate . Further , we conclude from the bottom row in Fig . 1 that a oversized training domain ( bottom right plot ) can hurt performance on the test domain , when compared to the middle plot with overlapping train and test domains . Our takeaway is that invariance indeed influences how distribution shifts affect changes in training and testing performance . We however also note that this topic is more nuanced . The difficulty lies in the fact that finding the “ correct ” invariances also means finding the true function as visualized in the third row of Fig . 1 . These smaller experiments make us expect that it may be unlikely to find the true invariances of an underlying functional relationship . In such a case , ensuring that train and test distribution match thus becomes a viable option to enforce a small change from train to test performance . 4 THE TRADE-OFF BETWEEN INVARIANCE AND DISTRIBUTION SHIFT We begin our investigation by stating that learning is a process that maps a training dataset Dtrain with distribution ptrain ( x , y ) to a learned function approximation f ( ptrain , x ) ≈ y ∀ ( x , y ) ∈ Dtrain , inducing a target distribution f ( ptrain ) as depicted in Fig . 2 . In deep learning this is usually achieved by minimizing a loss function on the training set using stochastic gradient descent ( Robbins & Monro , 1951 ) . Of interest however is the performance of the learned function on a testing dataset Dtest with possibly different distribution ptest ( x , y ) . By modeling the learned relationship f ( ptrain , x ) ∈ Y as a function on the data distribution ptrain ( x , y ) , we are then able to obtain a relationship between changes in the data distribution and changes in the learned function and its performance . Theorem 1 . We are given two dataset distributions p1 ( x , y ) and p2 ( x , y ) of input data x ∈ X and output y ∈ Y situated in the probability space with Wasserstein metric W2 ( · , · ) and assume that f ( p , x ) : P × X → Y is the function of a function family F continuous in the probability space P that minimizes the norm loss L ( f ( p1 , x ) , y ) = ||f ( p1 , x ) − y||2 when the data is distributed according to p1 ( x , y ) . Then the expected absolute difference in loss for distribution p2 has the following lower bound . Ep2 |L ( f ( p2 , x ) , y ) − L ( f ( p1 , x ) , y ) | ≥ Ep2 ∣∣∣∣∣∣∂f ( p , x ) ∂p ∣∣∣∣∣∣ ·W2 ( p1 , p2 ) ( 1 ) where ∂f ( p , x ) ∂p denotes the average value of ∂f ( p , x ) ∂p along the integral as part of the intermediate value theorem . Proof . First we note that we can write the learned function for distribution p2 as the function learned for distribution p1 plus the line integral of the change of the function along the Wasserstein geodesic between p1 and p2 . ( i ) f ( p2 , x ) = f ( p1 , x ) + ∫ p2 p1 ∂f ( p , x ) ∂p ∂p , ∂p : = ∇pW2 ( p , p1 ) ( ii ) ∫ p2 p1 ∂p = ∫ p2 p1 ∇p ( W2 ( p , p1 ) ) dp = W2 ( p2 , p1 ) −W2 ( p1 , p1 ) = W2 ( p2 , p1 ) Then the change in loss is lower bounded as follows : Ep2∆L = Ep2 |L ( f ( p2 , x ) , y ) − L ( f ( p1 , x ) , y ) | = Ep2 ∣∣∣ ||f ( p2 , x ) − y|| − ||f ( p1 , x ) − y|| ∣∣∣ ≥ Ep2 ||f ( p2 , x ) − f ( p1 , x ) || ( i ) = Ep2 ∣∣∣∣∣∣ ∫ p2 p1 ∂f ( p , x ) ∂p ∂p ∣∣∣∣∣∣ Intermediate value= Ep2 ∣∣∣∣∣∣∂f ( p , x ) ∂p ∫ p2 p1 ∂p ( x ) ∣∣∣∣∣∣ ( ii ) = Ep2 ∣∣∣∣∣∣∂f ( p , x ) ∂p W2 ( p1 , p2 ) ∣∣∣∣∣∣ = Ep2 ∣∣∣∣∣∣∂f ( p , x ) ∂p ∣∣∣∣∣∣ ·W2 ( p1 , p2 ) From the above lower bound in Ineq . 1 , we can deduce two ways of achieving a minimal change in loss when moving from training distribution p1 to testing distribution p2 . ( 1 ) Either the difference in distribution from training set to test set is small W2 ( p1 , p2 ) ≈ 0 . In this case , by assuming the learned function fits the training distribution perfectly and is not ill-conditioned , we can deduce that also the test distribution is fit perfectly , or ( 2 ) The function f ( p ) does not change when the dataset distribution changes from p1 to p2 by being invariant to such changes s.t . Ep2 ∣∣∣∣∂f ( p , x ) ∂p ∣∣∣∣ ≈ 0 . Clearly , if the learned function can not distinguish between , i.e . is invariant to , changes in training and test set , then even a shift of distribution to the test set can not alter the quality of the performance . If on the other hand , as may be more common , the function class is not invariant , then the difference in distribution from training to test set becomes decisive . Note that since we regard changes in performance , it is vital to already have a good performance on the training distribution . From the above analysis , we believe that it may be essential to employ training sets that are as close as possible in distribution to the test set . If this is the case one does not need to worry about finding the right function class with the right invariances to approximate the true function . In the case where the learned function has “ too much invariance ” , the function is not noticing some of the relevant variations in the data , e.g . in the extreme a constant would not change from training to testing distribution . While its performance does not change from training to test distribution , its training performance already suffers from unnecessary invariance . If on the other hand , the function is not invariant enough it might fit the training set well yet lose performance when switching to the test distribution . Instead of either calculating the distribution distance on the raw data ( W2 ( p1 , p2 ) ) , we compute the distance on an embedding of the raw data that aims to mirror the distances the function f ( · ) would perceive . This has the advantage of avoiding to compute demanding high dimensional distribution distances from raw data on the one hand , and having to train networks for both training and test distribution on the other hand . Additionally , it may be hard to properly compare the distribution of a function trained on the training distribution vs. one trained on the test distribution since training and test set oftentimes are of very different size . To measure distribution distances , we employ the Fréchet distance ( FD ) ( Fréchet , 1957 ) , also called the 2-Wasserstein distance ( Vaserstein , 1969 ) , by assuming all probabilities to be normally distributed . The FD is a distance measure between two normal distributions , also adopted in the Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) to evaluate the quality difference of real and synthetic images . Instead of comparing raw data , we use the metric by exploiting domain-relevant features using an embedding network such as the inception network ( Szegedy et al. , 2015 ) . The FD of two normal distributions N ( µ1 , Σ1 ) and N ( µ2 , Σ2 ) is given by : FD = d2 ( ( µ1 , Σ1 ) , ( µ2 , Σ2 ) ) = ||µ1 − µ2||22 + Tr ( Σ1 + Σ2 − 2 ( Σ1Σ2 ) 1 2 ) While Heusel et al . ( 2017 ) show that the distance correlates well with human judgement in the vision domain , it can also be applied to other domains if a domain relevant model replaces the inception network . We apply the FD to measure the mismatch of two sets in the following sections .

Summary: The authors consider the relation between Frechet distance of training and test distribution and the generalization gap. The authors derive the lower bound for the difference of loss function w.r.t. training and test set by the Wasserstein distance between embedding training and test set distribution. Empirically, the authors illustrate a strong correlation between test performance and the distance in distributions between training and test set.