Distributional Generalization: Structure Beyond Test Error

1 INTRODUCTION . In learning theory , when we study how well a classifier “ generalizes ” , we usually consider a single metric – its test error ( Shalev-Shwartz & Ben-David , 2014 ) . However , there could be many different classifiers with the same test error that differ substantially in , say , the subgroups of inputs on which they make errors . Reducing classifiers to a single number misses these rich aspects of their behavior . In this work , we propose formally studying the entire joint distribution of classifier inputs and outputs . That is , the distribution ( x , f ( x ) ) for samples from the distribution x ∼ D for a classifier f ( x ) . This distribution reveals many structural properties of the classifier beyond test error ( such as where the errors occur ) . In fact , we discover new behaviors of modern classifiers that can only be understood in this framework . As an example , consider the following experiment ( Figure 1 ) . Experiment 1 . Consider a binary classification version of CIFAR-10 , where CIFAR-10 images x have binary labels Animal/Object . Take 50K samples from this distribution as a train set , but apply the following label noise : flip the label of cats to Object with probability 30 % . Now train a WideResNet f to 0 train error on this train set . How does the trained classifier behave on test samples ? Options below : ( 1 ) The test error is low across all classes , since there is only 3 % overall label noise in the train set . ( 2 ) Test error is “ spread ” across the animal class . After all , the classifier is not explicitly told what a cat or a dog is , just that they are all animals . ( 3 ) The classifier misclassifies roughly 30 % of test cats as “ objects ” , but all other animals are largely unaffected . The reality is closest to option ( 3 ) as shown in Figure 1 . The left panel shows the joint density of train inputs x with train labels Object/Animal . The right panel shows the classifier predictions f ( x ) on test inputs x . There are several notable things about this experiment . First , the error is localized to cats in the test set as it was in the train set , even though no explicit cat labels were provided . The interpolating model is thus sensitive to subgroup-structures in the distribution . Second , the amount of error on the cat class is close to the noise applied on the train set . Thus , the behavior of the classifier on the train set generalizes to the test set in a stronger sense than just average error . Specifically , when conditioned on a subgroup ( cat ) , the distribution of the true labels is close to that of the classifier outputs . Third , this is not the behavior of the Bayes-optimal classifier , which would always output the maximum-likelihood label instead of reproducing the noise in the distribution . The network is thus behaving poorly from the perspective of Bayes-optimality , but behaving well in a certain distributional sense ( which we will formalize soon ) . Now , consider a seemingly unrelated experimental observation . Take an AlexNet trained on ImageNet , a 1000-way classification problem with 116 varieties of dogs . AlexNet only achieves 56.5 % test accuracy on ImageNet . However , it at least classifies most dogs as some variety of dog ( with 98.4 % accuracy ) , though it may mistake the exact breed . In this work , we show that both of these experiments are examples of the same underlying phenomenon . We empirically show that for an interpolating classifier , its classification outputs are close in distribution to the true labels — even when conditioned on many subsets of the domain . For example , in Figure 1 , the distribution of p ( f ( x ) |x = cat ) is close to the true label distribution of p ( y|x = cat ) . We propose a formal conjecture ( Feature Calibration ) , that predicts which subgroups of the domain can be conditioned on for the above distributional closeness to hold . These experimental behaviors could not have been captured solely by looking at average test error , as is done in the classical theory of generalization . In fact , they are special cases of a new kind of generalization , which we call “ Distributional Generalization ” . 1.1 DISTRIBUTIONAL GENERALIZATION . Informally , Distributional Generalization states that the outputs of classifiers f on their train sets and test sets are close as distributions ( as opposed to close in just error ) . That is , the following joint distributions1 are close : ( x , f ( x ) ) x∼TestSet ≈ ( x , f ( x ) ) x∼TrainSet ( 1 ) The remainder of this paper is devoted to making the above statement precise , and empirically checking its validity on real-world tasks . Specifically , we want to formally define the notion of approximation ( ≈ ) , and understand how it depends on the problem parameters ( the type of classifier , number of train samples , etc ) . We focus primarily on interpolating methods , where we formalize Equation ( 1 ) through our Feature Calibration Conjecture . 1.2 OUR CONTRIBUTIONS AND ORGANIZATION . We discover new empirical behaviors of interpolating classifiers , and we propose quantitative conjectures to characterize these behaviors . • In Section 3 , we introduce a “ Feature Calibration ” conjecture , which unifies our experimental observations . Roughly , Feature Calibration says that the outputs of classifiers match the statistics of their training distribution when conditioned on certain subgroups . • In Section 4 , we experimentally stress test our Feature Calibration conjecture across various settings in machine learning , including neural networks , kernel machines , and decision trees . This highlights the universality of our results across machine learning . 1These distributions also include the randomness in sampling the train and test sets , and in training the classifier , as we define more precisely in Section 3 . • In Section 5 , we relate our results to classical generalization , by defining a new notion of Distributional Generalization which extends and subsumes the classical notion . • Finally , in Section 5.2 we informally discuss how Distributional Generalization can be applied even for non-interpolating methods . Our results extend our scientific understanding of of interpolating methods , and introduce a new type of generalization exhibited across many methods in machine learning . 1.3 RELATED WORK AND SIGNIFICANCE . Our work has connections to , and implications for many existing research programs in deep learning . Implicit Bias and Overparameterization . There has been a long line of recent work towards understanding overparameterized and interpolating methods , since these pose challenges for classical theories of generalization ( e.g . Zhang et al . ( 2016 ) ; Belkin et al . ( 2018a ; b ; 2019 ) ; Liang & Rakhlin ( 2018 ) ; Nakkiran et al . ( 2020 ) ; Schapire et al . ( 1998 ) ; Breiman ( 1995 ) ; Soudry et al . ( 2018 ) ; Gunasekar et al . ( 2018 ) ) . The “ implicit bias ” program here aims to answer : Among all models with 0 train error , which model is actually produced by SGD ? Most existing work seeks to characterize the exact implicit bias of models under certain ( sometimes strong ) assumptions on the model , training method or the data distribution . In contrast , our conjecture applies across many different interpolating models ( from neural nets to decision trees ) as they would be used in practice , and thus form a sort of “ universal implicit bias ” of these methods . Moreover , our results place constraints on potential future theories of implicit bias , and guide us towards theories that better capture practice . Benign Overfitting . Most prior works on interpolating classifiers attempt to explain why training to interpolation “ does not harm ” the the model . This has been dubbed “ benign overfitting ” ( Bartlett et al. , 2020 ) and “ harmless interpolation ” ( Muthukumar et al. , 2020 ) , reflecting the widely-held belief that interpolation does not harm the decision boundary of classifiers . In contrast , we find that interpolation actually does “ harm ” classifiers , in predictable ways : fitting the label noise on the train set causes similar noise to be reproduced at test time . Our results thus indicate that interpolation can significantly affect the decision boundary of classifiers , and should not be considered a purely “ benign ” effect . Classical Generalization and Scaling Limits . Our framework of Distributional Generalization is insightful even to study classical generalization , since it reveals much more about models than just their test error . For example , statistical learning theory attempts to understand if and when models will asymptotically converge to Bayes optimal classifiers , in the limit of large data ( “ asymptotic consistency ” Shalev-Shwartz & Ben-David ( 2014 ) ; Wasserman ( 2013 ) ) . In deep learning , there are at least two distinct ways to scale model and data to infinity together : the underparameterized scaling limit , where data-size model-size always , and the overparameterized scaling limit , where data-size model-size always . The underparameterized scaling limit is well-understood : when data is essentially infinite , neural networks will converge to the Bayes-optimal classifier ( provided the model-size is large enough , and the optimization is run for long enough , with enough noise to escape local minima ) . On the other hand , our work suggests that in the overparameterized scaling limit , models will not converge to the Bayes-optimal classifier . Specifically , our Feature Calibration Conjecture implies that in the limit of large data , interpolating models will approach a sampler from the distribution . That is , the limiting model f will be such that the output f ( x ) is a sample from p ( y|x ) , as opposed to the Bayes-optimal f∗ ( x ) = argmaxy p ( y|x ) . This claim— that overparameterized models do not converge to Bayes-optimal classifiers— is unique to our work as far as we know , and highlights the broad implications of our results . Locality and Manifold Learning . Our intuition for the behaviors in this work is that they arise due to some form of “ locality ” of the trained classifiers , in an appropriate embedding space . For example , the behavior observed in Experiment 1 would be consistent with that of a 1-Nearest-Neighbor classifier in a embedding that separates the CIFAR-10 classes well . This intuition that classifiers learn good embeddings is present in various forms in the literature , for example : the so-called called “ manifold hypothesis , ” that natural data lie on a low-dimensional manifold ( Narayanan & Mitter , 2010 ; Sharma & Kaplan , 2020 ) , as well as works on local stiffness of the loss landscape ( Fort et al. , 2019 ) , and works showing that overparameterized neural networks can learn hidden low-dimensional structure in high-dimensional settings ( Gerace et al. , 2020 ; Bach , 2017 ; Chizat & Bach , 2020 ) . It is open to more formally understand connections between our work and the above . Uncertainty Estimation . Since the appearance of the current work on arXiv , it has been directly built on by other authors . The work of Jiang et al . ( 2021 ) investigates our conjectures further , and extends them to develop a method for out-of-distribution uncertainty estimation . This highlights the fundamental nature and importance of our results , since they have already been used in a practical application . A full discussion of related works is in Appendix A .

Summary: The paper reports an interesting phenomenon in deep over-parameterized networks which train to zero train error (known as interpolating networks). Not only does the accuracy on test mimic the accuracy on train, but the accuracy measured on certain subsets of the train set also matches that on the corresponding subsets in test set. The paper coins the term distributional generalization for this phenomenon. The paper then proceeds to perform a number of experiments across a bunch of image datasets to exhibit the generality of this phenomenon.