On the geometry of generalization and memorization in deep neural networks

1 INTRODUCTION . Deep neural networks have many more learnable parameters than training examples , and could simply memorize the data instead of converging to a generalizable solution ( Novak et al. , 2018 ) . Moreover , standard regularization methods are insufficient to eliminate memorization of random labels , and network complexity measures fail to account for the generalizability of large neural networks ( Zhang et al. , 2016 ; Neyshabur et al. , 2014 ) . Yet , even though memorizing solutions exist , they are rarely learned in practice by neural networks ( Rolnick et al. , 2017 ) . Recent work have shown that a combination of architecture and stochastic gradient descent implicitly bias the training dynamics towards generalizable solutions ( Hardt et al. , 2016 ; Soudry et al. , 2018 ; Brutzkus et al. , 2017 ; Li and Liang , 2018 ; Saxe et al. , 2013 ; Lampinen and Ganguli , 2018 ) . However , these claims study either linear networks or two-layer non-linear networks . For deep neural networks , open questions remain on the structure of memorization , such as where and when in the layers of the network is memorization occurring ( e.g . evenly across all layers , gradually increasing with depth , or concentrated in early or late layers ) , and what are the drivers of this structure . Analytical tools for linear networks such as eigenvalue decomposition can not be directly applied to non-linear networks , so here we employ a recently developed geometric probe ( Chung et al. , 2018 ; Stephenson et al. , 2019 ) , based on replica mean field theory from statistical physics , to analyze the training dynamics and resulting structure of memorization . The probe measures not just the layer capacity , but also geometric properties of the object manifolds , explicitly linked by the theory . We find that deep neural networks ignore randomly labeled data in the early layers and epochs , instead learning generalizing features . Memorization occurs abruptly with depth in the final layers , caused by * : Equal contribution , + : Correspondence . decreasing manifold radius and dimension , whereas early layers are minimally affected . Notably , this structure does not arise due to gradients vanishing with depth . Instead , analytical analysis show that near initialization , the gradients from noise examples contribute minimally to the total gradient , and that networks are able to ignore noise due to the existence of ’ shared features ’ consisting of linear features shared by objects of the same class . Of practical consequence , generalization can then be re-gained by rolling back the parameters of the final layers of the network to an earlier epoch before the structural signatures of memorization occur . Moreover , the ’ double descent ’ phenomenon , where a model ’ s generalization performance initially decreases with model size before increasing , is linked to the non-monotonic dimensionality expansion of the object manifolds , as measured by the geometric probe . The manifold dimensionality also undergoes double descent , whereas other geometric measures , such as radius and center correlation are monotonic with model size . Our analysis reveals the structure of memorization in deep networks , and demonstrate the importance of measuring manifold geometric properties in tracing the effect of learning on neural networks . 2 RELATED WORK . By demonstrating that deep neural networks can easily fit random labels with standard training procedures , Zhang et al . ( 2016 ) showed that standard regularization strategies are not enough to prevent memorization . Since then , several works have explored the problem experimentally . Notably Arpit et al . ( 2017 ) examined the behavior of the network as a single unit when trained on random data , and observed that the training dynamics were qualitatively different when the network was trained on real data vs. noise . They observed experimentally that networks learn from ’ simple ’ patterns in the data first when trained with gradient descent , but do not give a rigorous explanation of this effect . In the case of deep linear models trained with mean squared error , more is known about the interplay between dynamics and generalization ( Saxe et al. , 2013 ) . While these models are linear , the training dynamics are not , and interestingly , these networks preferentially learn large singular values of the input-output correlation matrix first . During training , the dynamics act like a singular value detection wave ( Lampinen and Ganguli , 2018 ) , and so memorization of noisy modes happens late in training . Experimental works have explored the training dynamics using variants of Canonical Correlation Analysis ( Raghu et al. , 2017 ; Morcos et al. , 2018 ) , revealing different rates of learning across layers , and differences between networks that memorize or generalize ( Morcos et al. , 2018 ) . Using Centered Kernel Alignment , Kornblith et al . ( 2019 ) find that networks differ from each other the most in later layers . However , as these metrics measure similarity , experiments are limited to comparing different networks , offering limited insight into specific network instances . As noted in ( Wang et al. , 2018 ) , the similarity of networks trained on the same data with different initialization may be surprisingly low . Our work builds on this line of research by using a direct , rather than comparative , theory-based measure of the representation geometry ( Chung et al. , 2016 ; Cohen et al. , 2019 ) to probe the layerwise dynamics as learning gives way to memorization . With the direct theory-based measure , we can probe individual networks over the course of training , rather than comparing families of networks . 3 EXPERIMENTAL SETUP . As described in ( Arpit et al. , 2017 ) , we adopt the view of memorization as `` the behavior exhibited by DNNs trained on noise . '' We induce memorization by randomly permuting the labels for a fraction of the dataset . To fit these examples , a DNN must use spurious signals in the input to memorize the ‘ correct ’ random label . We train our models for either 1 , 000 epochs , or until they achieve > 99 % accuracy on the training set , implying that randomly labeled examples have been memorized . We do not use weight decay or any other regularization . Once a model has been trained with partially randomized labels , we apply the recently developed replica-based mean-field theoretic manifold analysis technique ( MFTMA hereafter ) ( Chung et al. , 2018 ) to analyze the hidden representations learned by the model at different layers and epochs of training . This quantitative measure of the underlying manifold geometry learned by the DNN provides insight into how information about learned features is encoded in the network . This method has been applied to pre-trained models ( Cohen et al. , 2019 ) , whereas here we study the training dynamics in the memorization regime . 3.1 DATASETS AND MODELS . We carried out experiments on a variety of neural network architectures and datasets . For CIFAR-100 ( Krizhevsky et al. , 2009 ) , we trained AlexNet ( Krizhevsky et al. , 2014 ) , VGG-16 ( Simonyan and Zisserman , 2014 ) and ResNet-18 ( He et al. , 2016 ) models , each slightly modified for use with the dataset . We also used Tiny ImageNet dataset1 , a reduced version of ImageNet ( Deng et al. , 2009 ) with ResNet-18 . Training methods similar to those used for the CIFAR-100 dataset did not result in complete memorization on Tiny ImageNet , owing to the larger and more complex dataset , and so we show results on partial memorization after training for 1,000 epochs . Due to space constraints , we present results for VGG-16 trained on CIFAR-100 . Other architectures and datasets have similar findings , as detailed in the Appendix . Our networks are trained on a dataset with some fraction of its labels randomly permuted . To analyze the behavior of the trained models , we define four subsets of examples as follows : Unpermuted examples : The set of labeled examples from the training set which did not have their labels permuted . That is , the networks were trained on these examples using the correct label . In the case of = 100 % permuted data , this set is empty , as we do not count examples with labels randomly assigned to the correct class . Permuted examples : The set of labeled examples from the training set which had their labels randomly permuted . The networks were trained to classify these examples according to their random labels . By design , there is some chance that the random assignment results in the label remaining unchanged , however these examples are still in the set of the permuted examples . The motivation for this choice can be found in the Appendix . Restored examples : These examples are the same as the permuted examples , except when carrying out analysis we restored the correct , unpermuted labels . For example , for a permuted example with 1This dataset is from the Tiny ImageNet Challenge : https : //tiny-imagenet.herokuapp.com/ an image of class ’ cat ’ that was trained with label ’ dog ’ , the corresponding restored example would be the same image with the label ’ cat ’ restored . A sampling of unpermuted , permuted , and restored examples is shown in Fig . 1A , with the denoted labels used for training and analysis . Finally , we define test examples : Test examples : The set of labeled examples from a holdout set not seen during training . All analyses on test examples are done using the correct label . 3.2 REPLICA-BASED MEAN FIELD THEORY MANIFOLD ANALYSIS . We apply the replica MFTMA analysis ( Chung et al. , 2018 ; Cohen et al. , 2019 ; Stephenson et al. , 2019 ) which formally connects object category manifolds ’ linear separability with the underlying geometric properties , such as manifold dimension , radius , and correlations . A key feature in object manifold capacity theory is that it generalises the notion of perceptron shattering capacity from discrete points to class manifolds . Given P object manifolds defined by the clouds of feature vectors in N feature dimension , where each manifold is assigned random binary labels where points within the same manifold have an identical label , rendering a total of 2P manifold dichotomies . The manifold capacity , αc = P/N is defined by the critical number of object manifolds where most of the manifold dichotomies can be shattered ( separated ) by a linear hyperplane . This is closely related to Cover ’ s function counting theorem , except in this theory , the units for counting P for capacity are category manifolds , rather than discrete point patterns . The manifold capacity , therefore , measures the linear separability of categories in the representation . Surprisingly , it ’ s been reported that the object manifold capacity for general point clouds can be formulated in a similar manner to the closed-form manifold capacity of D-dimensional balls of radius R in random orientations ( Chung et al. , 2016 ) , providing the closed form expressions for effective dimension DM and effective radius RM for the convex hulls of general point clouds in random orientation . More recently , the theory of manifolds has been refined to incorporate correlation structure in the manifold locations , allowing for the estimation of manifold capacities in real data with correlations ( Cohen et al. , 2019 ) . This analysis , based on the series of theoretical developments mentioned above , formally links the linear separability of category class manifolds ( manifold capacity , αM ) with the class manifolds ’ geometric properties ( manifold dimension , DM , manifold radius , RM , and the correlations between the manifold centers , ρcenter ) . The descriptions of the manifold capacity and the geometrical properties are summarized below ( for more details , see Appendix ) . Manifold Capacity αM = P/N is a measure of the decodable class information in a given data , defined by critical number of object manifolds ( P ) that can be linearly classified with random dichotomy labels for each manifold , in feature dimension N. Large αM implies well-separated manifolds in feature space . Using replica mean field theory , this quantity can be predicted ( Chung et al. , 2018 ) by object manifolds ’ dimension , radius , and their center correlation structure provided below ( see illustrations in Fig . 1C and Appendix ) . Note that the manifold capacity can also be computed empirically ( Empirical capacity , hereafter ) , by doing a bisection search to find the critical number of features N such that the fraction of linearly separable random manifold dichotomies is close to 1/2 . We show that theoretical predictions match empirical capacity with our data in Fig . 1B . Manifold Dimension DM captures the dimensions of the set of anchor points , which are representative support vectors that determine the optimal hyperplane separating the binary dichotomy of manifolds . These anchor points are estimated from the guiding Gaussian vectors ( Fig . 1C and Appendix ) , and the position of the anchor point changes as orientations of other manifolds are varied . The average of the dot product between anchor points and the Gaussian vector defines the effective manifold dimension of given point clouds , and it reflects the dimensionality of embedded anchor points in the optimal margin hyperplane . A small dimension implies that the anchor points occupy a low dimensional subspace . Manifold Radius RM is an average norm of the anchor points in the manifold subspace , capturing the manifold size seen by the decision boundary . Small RM implies tightly grouped anchor points . Center Correlation ρcenter measures the average of absolute values of pairwise correlations between various centers . A small correlation indicates that , on average , manifolds lie along different directions in feature space . In the rest of the text , we will refer to the quantities αM , DM , RM , and ρM as the Manifold Geometry Metrics ( MGMs ) . Small values for manifold radius , dimension and center correlation have been shown to result in higher values for manifold capacity ( Chung et al. , 2018 ; Cohen et al. , 2019 ) , and so represent a more favorable geometry for classification ( Fig . 1C ) . For all experiments , we evaluate these four quantities using P = 100 classes , with M = 50 randomly drawn examples per class . We define unpermuted , permuted , restored , and test manifolds for each class at each layer of the network . Given a neural network N ( x ) , we extract Nl ( x ) , the activation vector at layer l generated on input x . We then define a manifold by the point cloud of activation vectors generated by inputs of the same class . This results in P manifolds , one for each class . The set of unpermuted manifolds is then obtained by using the corresponding manifolds with examples and labels that were unpermuted . The set of permuted , restored , and test manifolds are defined similarly . We run the MFTMA on each set of manifolds following the procedures in ( Chung et al. , 2018 ; Cohen et al. , 2019 ; Stephenson et al. , 2019 ) . All results are conducted with 10 random seeds , which govern the network initialization , label permutation , and MFMTA analysis initial conditions .

Summary: This paper analyses memorization in DNNs, from the lens of memorization = fitting random labels, and finds that it seems to happen in later layers. These results are obtained using the MFTMA framework, a manifold analysis tool, testing geometric properties of individual layers. The analysis also attempts to explain why such a phenomenon exists, and makes a few interesting observations.