Distributional Generalization: Structure Beyond Test Error

1 INTRODUCTION . In learning theory , when we study how well a classifier “ generalizes ” , we usually consider a single metric – its test error ( Shalev-Shwartz & Ben-David , 2014 ) . However , there could be many different classifiers with the same test error that differ substantially in , say , the subgroups of inputs on which they make errors . Reducing classifiers to a single number misses these rich aspects of their behavior . In this work , we propose formally studying the entire joint distribution of classifier inputs and outputs . That is , the distribution ( x , f ( x ) ) for samples from the distribution x ∼ D for a classifier f ( x ) . This distribution reveals many structural properties of the classifier beyond test error ( such as where the errors occur ) . In fact , we discover new behaviors of modern classifiers that can only be understood in this framework . As an example , consider the following experiment ( Figure 1 ) . Experiment 1 . Consider a binary classification version of CIFAR-10 , where CIFAR-10 images x have binary labels Animal/Object . Take 50K samples from this distribution as a train set , but apply the following label noise : flip the label of cats to Object with probability 30 % . Now train a WideResNet f to 0 train error on this train set . How does the trained classifier behave on test samples ? Options below : ( 1 ) The test error is low across all classes , since there is only 3 % overall label noise in the train set . ( 2 ) Test error is “ spread ” across the animal class . After all , the classifier is not explicitly told what a cat or a dog is , just that they are all animals . ( 3 ) The classifier misclassifies roughly 30 % of test cats as “ objects ” , but all other animals are largely unaffected . The reality is closest to option ( 3 ) as shown in Figure 1 . The left panel shows the joint density of train inputs x with train labels Object/Animal . The right panel shows the classifier predictions f ( x ) on test inputs x . There are several notable things about this experiment . First , the error is localized to cats in the test set as it was in the train set , even though no explicit cat labels were provided . The interpolating model is thus sensitive to subgroup-structures in the distribution . Second , the amount of error on the cat class is close to the noise applied on the train set . Thus , the behavior of the classifier on the train set generalizes to the test set in a stronger sense than just average error . Specifically , when conditioned on a subgroup ( cat ) , the distribution of the true labels is close to that of the classifier outputs . Third , this is not the behavior of the Bayes-optimal classifier , which would always output the maximum-likelihood label instead of reproducing the noise in the distribution . The network is thus behaving poorly from the perspective of Bayes-optimality , but behaving well in a certain distributional sense ( which we will formalize soon ) . Now , consider a seemingly unrelated experimental observation . Take an AlexNet trained on ImageNet , a 1000-way classification problem with 116 varieties of dogs . AlexNet only achieves 56.5 % test accuracy on ImageNet . However , it at least classifies most dogs as some variety of dog ( with 98.4 % accuracy ) , though it may mistake the exact breed . In this work , we show that both of these experiments are examples of the same underlying phenomenon . We empirically show that for an interpolating classifier , its classification outputs are close in distribution to the true labels — even when conditioned on many subsets of the domain . For example , in Figure 1 , the distribution of p ( f ( x ) |x = cat ) is close to the true label distribution of p ( y|x = cat ) . We propose a formal conjecture ( Feature Calibration ) , that predicts which subgroups of the domain can be conditioned on for the above distributional closeness to hold . These experimental behaviors could not have been captured solely by looking at average test error , as is done in the classical theory of generalization . In fact , they are special cases of a new kind of generalization , which we call “ Distributional Generalization ” . 1.1 DISTRIBUTIONAL GENERALIZATION . Informally , Distributional Generalization states that the outputs of classifiers f on their train sets and test sets are close as distributions ( as opposed to close in just error ) . That is , the following joint distributions1 are close : ( x , f ( x ) ) x∼TestSet ≈ ( x , f ( x ) ) x∼TrainSet ( 1 ) The remainder of this paper is devoted to making the above statement precise , and empirically checking its validity on real-world tasks . Specifically , we want to formally define the notion of approximation ( ≈ ) , and understand how it depends on the problem parameters ( the type of classifier , number of train samples , etc ) . We focus primarily on interpolating methods , where we formalize Equation ( 1 ) through our Feature Calibration Conjecture . 1.2 OUR CONTRIBUTIONS AND ORGANIZATION . We discover new empirical behaviors of interpolating classifiers , and we propose quantitative conjectures to characterize these behaviors . • In Section 3 , we introduce a “ Feature Calibration ” conjecture , which unifies our experimental observations . Roughly , Feature Calibration says that the outputs of classifiers match the statistics of their training distribution when conditioned on certain subgroups . • In Section 4 , we experimentally stress test our Feature Calibration conjecture across various settings in machine learning , including neural networks , kernel machines , and decision trees . This highlights the universality of our results across machine learning . 1These distributions also include the randomness in sampling the train and test sets , and in training the classifier , as we define more precisely in Section 3 . • In Section 5 , we relate our results to classical generalization , by defining a new notion of Distributional Generalization which extends and subsumes the classical notion . • Finally , in Section 5.2 we informally discuss how Distributional Generalization can be applied even for non-interpolating methods . Our results extend our scientific understanding of of interpolating methods , and introduce a new type of generalization exhibited across many methods in machine learning . 1.3 RELATED WORK AND SIGNIFICANCE . Our work has connections to , and implications for many existing research programs in deep learning . Implicit Bias and Overparameterization . There has been a long line of recent work towards understanding overparameterized and interpolating methods , since these pose challenges for classical theories of generalization ( e.g . Zhang et al . ( 2016 ) ; Belkin et al . ( 2018a ; b ; 2019 ) ; Liang & Rakhlin ( 2018 ) ; Nakkiran et al . ( 2020 ) ; Schapire et al . ( 1998 ) ; Breiman ( 1995 ) ; Soudry et al . ( 2018 ) ; Gunasekar et al . ( 2018 ) ) . The “ implicit bias ” program here aims to answer : Among all models with 0 train error , which model is actually produced by SGD ? Most existing work seeks to characterize the exact implicit bias of models under certain ( sometimes strong ) assumptions on the model , training method or the data distribution . In contrast , our conjecture applies across many different interpolating models ( from neural nets to decision trees ) as they would be used in practice , and thus form a sort of “ universal implicit bias ” of these methods . Moreover , our results place constraints on potential future theories of implicit bias , and guide us towards theories that better capture practice . Benign Overfitting . Most prior works on interpolating classifiers attempt to explain why training to interpolation “ does not harm ” the the model . This has been dubbed “ benign overfitting ” ( Bartlett et al. , 2020 ) and “ harmless interpolation ” ( Muthukumar et al. , 2020 ) , reflecting the widely-held belief that interpolation does not harm the decision boundary of classifiers . In contrast , we find that interpolation actually does “ harm ” classifiers , in predictable ways : fitting the label noise on the train set causes similar noise to be reproduced at test time . Our results thus indicate that interpolation can significantly affect the decision boundary of classifiers , and should not be considered a purely “ benign ” effect . Classical Generalization and Scaling Limits . Our framework of Distributional Generalization is insightful even to study classical generalization , since it reveals much more about models than just their test error . For example , statistical learning theory attempts to understand if and when models will asymptotically converge to Bayes optimal classifiers , in the limit of large data ( “ asymptotic consistency ” Shalev-Shwartz & Ben-David ( 2014 ) ; Wasserman ( 2013 ) ) . In deep learning , there are at least two distinct ways to scale model and data to infinity together : the underparameterized scaling limit , where data-size model-size always , and the overparameterized scaling limit , where data-size model-size always . The underparameterized scaling limit is well-understood : when data is essentially infinite , neural networks will converge to the Bayes-optimal classifier ( provided the model-size is large enough , and the optimization is run for long enough , with enough noise to escape local minima ) . On the other hand , our work suggests that in the overparameterized scaling limit , models will not converge to the Bayes-optimal classifier . Specifically , our Feature Calibration Conjecture implies that in the limit of large data , interpolating models will approach a sampler from the distribution . That is , the limiting model f will be such that the output f ( x ) is a sample from p ( y|x ) , as opposed to the Bayes-optimal f∗ ( x ) = argmaxy p ( y|x ) . This claim— that overparameterized models do not converge to Bayes-optimal classifiers— is unique to our work as far as we know , and highlights the broad implications of our results . Locality and Manifold Learning . Our intuition for the behaviors in this work is that they arise due to some form of “ locality ” of the trained classifiers , in an appropriate embedding space . For example , the behavior observed in Experiment 1 would be consistent with that of a 1-Nearest-Neighbor classifier in a embedding that separates the CIFAR-10 classes well . This intuition that classifiers learn good embeddings is present in various forms in the literature , for example : the so-called called “ manifold hypothesis , ” that natural data lie on a low-dimensional manifold ( Narayanan & Mitter , 2010 ; Sharma & Kaplan , 2020 ) , as well as works on local stiffness of the loss landscape ( Fort et al. , 2019 ) , and works showing that overparameterized neural networks can learn hidden low-dimensional structure in high-dimensional settings ( Gerace et al. , 2020 ; Bach , 2017 ; Chizat & Bach , 2020 ) . It is open to more formally understand connections between our work and the above . Uncertainty Estimation . Since the appearance of the current work on arXiv , it has been directly built on by other authors . The work of Jiang et al . ( 2021 ) investigates our conjectures further , and extends them to develop a method for out-of-distribution uncertainty estimation . This highlights the fundamental nature and importance of our results , since they have already been used in a practical application . A full discussion of related works is in Appendix A .

Summary: The paper proposes a new notion of ``distributional generalization". It formalizes this through a conjecture (feature calibration conjecture) which says that the output distribution of an interpolating classifier matches the distributions of the labels on a certain class of sub-groups of the data. The paper evaluates this conjecture empirically, and also proves it for certain nearest neighbor models.