On the geometry of generalization and memorization in deep neural networks

1 INTRODUCTION . Deep neural networks have many more learnable parameters than training examples , and could simply memorize the data instead of converging to a generalizable solution ( Novak et al. , 2018 ) . Moreover , standard regularization methods are insufficient to eliminate memorization of random labels , and network complexity measures fail to account for the generalizability of large neural networks ( Zhang et al. , 2016 ; Neyshabur et al. , 2014 ) . Yet , even though memorizing solutions exist , they are rarely learned in practice by neural networks ( Rolnick et al. , 2017 ) . Recent work have shown that a combination of architecture and stochastic gradient descent implicitly bias the training dynamics towards generalizable solutions ( Hardt et al. , 2016 ; Soudry et al. , 2018 ; Brutzkus et al. , 2017 ; Li and Liang , 2018 ; Saxe et al. , 2013 ; Lampinen and Ganguli , 2018 ) . However , these claims study either linear networks or two-layer non-linear networks . For deep neural networks , open questions remain on the structure of memorization , such as where and when in the layers of the network is memorization occurring ( e.g . evenly across all layers , gradually increasing with depth , or concentrated in early or late layers ) , and what are the drivers of this structure . Analytical tools for linear networks such as eigenvalue decomposition can not be directly applied to non-linear networks , so here we employ a recently developed geometric probe ( Chung et al. , 2018 ; Stephenson et al. , 2019 ) , based on replica mean field theory from statistical physics , to analyze the training dynamics and resulting structure of memorization . The probe measures not just the layer capacity , but also geometric properties of the object manifolds , explicitly linked by the theory . We find that deep neural networks ignore randomly labeled data in the early layers and epochs , instead learning generalizing features . Memorization occurs abruptly with depth in the final layers , caused by * : Equal contribution , + : Correspondence . decreasing manifold radius and dimension , whereas early layers are minimally affected . Notably , this structure does not arise due to gradients vanishing with depth . Instead , analytical analysis show that near initialization , the gradients from noise examples contribute minimally to the total gradient , and that networks are able to ignore noise due to the existence of ’ shared features ’ consisting of linear features shared by objects of the same class . Of practical consequence , generalization can then be re-gained by rolling back the parameters of the final layers of the network to an earlier epoch before the structural signatures of memorization occur . Moreover , the ’ double descent ’ phenomenon , where a model ’ s generalization performance initially decreases with model size before increasing , is linked to the non-monotonic dimensionality expansion of the object manifolds , as measured by the geometric probe . The manifold dimensionality also undergoes double descent , whereas other geometric measures , such as radius and center correlation are monotonic with model size . Our analysis reveals the structure of memorization in deep networks , and demonstrate the importance of measuring manifold geometric properties in tracing the effect of learning on neural networks . 2 RELATED WORK . By demonstrating that deep neural networks can easily fit random labels with standard training procedures , Zhang et al . ( 2016 ) showed that standard regularization strategies are not enough to prevent memorization . Since then , several works have explored the problem experimentally . Notably Arpit et al . ( 2017 ) examined the behavior of the network as a single unit when trained on random data , and observed that the training dynamics were qualitatively different when the network was trained on real data vs. noise . They observed experimentally that networks learn from ’ simple ’ patterns in the data first when trained with gradient descent , but do not give a rigorous explanation of this effect . In the case of deep linear models trained with mean squared error , more is known about the interplay between dynamics and generalization ( Saxe et al. , 2013 ) . While these models are linear , the training dynamics are not , and interestingly , these networks preferentially learn large singular values of the input-output correlation matrix first . During training , the dynamics act like a singular value detection wave ( Lampinen and Ganguli , 2018 ) , and so memorization of noisy modes happens late in training . Experimental works have explored the training dynamics using variants of Canonical Correlation Analysis ( Raghu et al. , 2017 ; Morcos et al. , 2018 ) , revealing different rates of learning across layers , and differences between networks that memorize or generalize ( Morcos et al. , 2018 ) . Using Centered Kernel Alignment , Kornblith et al . ( 2019 ) find that networks differ from each other the most in later layers . However , as these metrics measure similarity , experiments are limited to comparing different networks , offering limited insight into specific network instances . As noted in ( Wang et al. , 2018 ) , the similarity of networks trained on the same data with different initialization may be surprisingly low . Our work builds on this line of research by using a direct , rather than comparative , theory-based measure of the representation geometry ( Chung et al. , 2016 ; Cohen et al. , 2019 ) to probe the layerwise dynamics as learning gives way to memorization . With the direct theory-based measure , we can probe individual networks over the course of training , rather than comparing families of networks . 3 EXPERIMENTAL SETUP . As described in ( Arpit et al. , 2017 ) , we adopt the view of memorization as `` the behavior exhibited by DNNs trained on noise . '' We induce memorization by randomly permuting the labels for a fraction of the dataset . To fit these examples , a DNN must use spurious signals in the input to memorize the ‘ correct ’ random label . We train our models for either 1 , 000 epochs , or until they achieve > 99 % accuracy on the training set , implying that randomly labeled examples have been memorized . We do not use weight decay or any other regularization . Once a model has been trained with partially randomized labels , we apply the recently developed replica-based mean-field theoretic manifold analysis technique ( MFTMA hereafter ) ( Chung et al. , 2018 ) to analyze the hidden representations learned by the model at different layers and epochs of training . This quantitative measure of the underlying manifold geometry learned by the DNN provides insight into how information about learned features is encoded in the network . This method has been applied to pre-trained models ( Cohen et al. , 2019 ) , whereas here we study the training dynamics in the memorization regime . 3.1 DATASETS AND MODELS . We carried out experiments on a variety of neural network architectures and datasets . For CIFAR-100 ( Krizhevsky et al. , 2009 ) , we trained AlexNet ( Krizhevsky et al. , 2014 ) , VGG-16 ( Simonyan and Zisserman , 2014 ) and ResNet-18 ( He et al. , 2016 ) models , each slightly modified for use with the dataset . We also used Tiny ImageNet dataset1 , a reduced version of ImageNet ( Deng et al. , 2009 ) with ResNet-18 . Training methods similar to those used for the CIFAR-100 dataset did not result in complete memorization on Tiny ImageNet , owing to the larger and more complex dataset , and so we show results on partial memorization after training for 1,000 epochs . Due to space constraints , we present results for VGG-16 trained on CIFAR-100 . Other architectures and datasets have similar findings , as detailed in the Appendix . Our networks are trained on a dataset with some fraction of its labels randomly permuted . To analyze the behavior of the trained models , we define four subsets of examples as follows : Unpermuted examples : The set of labeled examples from the training set which did not have their labels permuted . That is , the networks were trained on these examples using the correct label . In the case of = 100 % permuted data , this set is empty , as we do not count examples with labels randomly assigned to the correct class . Permuted examples : The set of labeled examples from the training set which had their labels randomly permuted . The networks were trained to classify these examples according to their random labels . By design , there is some chance that the random assignment results in the label remaining unchanged , however these examples are still in the set of the permuted examples . The motivation for this choice can be found in the Appendix . Restored examples : These examples are the same as the permuted examples , except when carrying out analysis we restored the correct , unpermuted labels . For example , for a permuted example with 1This dataset is from the Tiny ImageNet Challenge : https : //tiny-imagenet.herokuapp.com/ an image of class ’ cat ’ that was trained with label ’ dog ’ , the corresponding restored example would be the same image with the label ’ cat ’ restored . A sampling of unpermuted , permuted , and restored examples is shown in Fig . 1A , with the denoted labels used for training and analysis . Finally , we define test examples : Test examples : The set of labeled examples from a holdout set not seen during training . All analyses on test examples are done using the correct label . 3.2 REPLICA-BASED MEAN FIELD THEORY MANIFOLD ANALYSIS . We apply the replica MFTMA analysis ( Chung et al. , 2018 ; Cohen et al. , 2019 ; Stephenson et al. , 2019 ) which formally connects object category manifolds ’ linear separability with the underlying geometric properties , such as manifold dimension , radius , and correlations . A key feature in object manifold capacity theory is that it generalises the notion of perceptron shattering capacity from discrete points to class manifolds . Given P object manifolds defined by the clouds of feature vectors in N feature dimension , where each manifold is assigned random binary labels where points within the same manifold have an identical label , rendering a total of 2P manifold dichotomies . The manifold capacity , αc = P/N is defined by the critical number of object manifolds where most of the manifold dichotomies can be shattered ( separated ) by a linear hyperplane . This is closely related to Cover ’ s function counting theorem , except in this theory , the units for counting P for capacity are category manifolds , rather than discrete point patterns . The manifold capacity , therefore , measures the linear separability of categories in the representation . Surprisingly , it ’ s been reported that the object manifold capacity for general point clouds can be formulated in a similar manner to the closed-form manifold capacity of D-dimensional balls of radius R in random orientations ( Chung et al. , 2016 ) , providing the closed form expressions for effective dimension DM and effective radius RM for the convex hulls of general point clouds in random orientation . More recently , the theory of manifolds has been refined to incorporate correlation structure in the manifold locations , allowing for the estimation of manifold capacities in real data with correlations ( Cohen et al. , 2019 ) . This analysis , based on the series of theoretical developments mentioned above , formally links the linear separability of category class manifolds ( manifold capacity , αM ) with the class manifolds ’ geometric properties ( manifold dimension , DM , manifold radius , RM , and the correlations between the manifold centers , ρcenter ) . The descriptions of the manifold capacity and the geometrical properties are summarized below ( for more details , see Appendix ) . Manifold Capacity αM = P/N is a measure of the decodable class information in a given data , defined by critical number of object manifolds ( P ) that can be linearly classified with random dichotomy labels for each manifold , in feature dimension N. Large αM implies well-separated manifolds in feature space . Using replica mean field theory , this quantity can be predicted ( Chung et al. , 2018 ) by object manifolds ’ dimension , radius , and their center correlation structure provided below ( see illustrations in Fig . 1C and Appendix ) . Note that the manifold capacity can also be computed empirically ( Empirical capacity , hereafter ) , by doing a bisection search to find the critical number of features N such that the fraction of linearly separable random manifold dichotomies is close to 1/2 . We show that theoretical predictions match empirical capacity with our data in Fig . 1B . Manifold Dimension DM captures the dimensions of the set of anchor points , which are representative support vectors that determine the optimal hyperplane separating the binary dichotomy of manifolds . These anchor points are estimated from the guiding Gaussian vectors ( Fig . 1C and Appendix ) , and the position of the anchor point changes as orientations of other manifolds are varied . The average of the dot product between anchor points and the Gaussian vector defines the effective manifold dimension of given point clouds , and it reflects the dimensionality of embedded anchor points in the optimal margin hyperplane . A small dimension implies that the anchor points occupy a low dimensional subspace . Manifold Radius RM is an average norm of the anchor points in the manifold subspace , capturing the manifold size seen by the decision boundary . Small RM implies tightly grouped anchor points . Center Correlation ρcenter measures the average of absolute values of pairwise correlations between various centers . A small correlation indicates that , on average , manifolds lie along different directions in feature space . In the rest of the text , we will refer to the quantities αM , DM , RM , and ρM as the Manifold Geometry Metrics ( MGMs ) . Small values for manifold radius , dimension and center correlation have been shown to result in higher values for manifold capacity ( Chung et al. , 2018 ; Cohen et al. , 2019 ) , and so represent a more favorable geometry for classification ( Fig . 1C ) . For all experiments , we evaluate these four quantities using P = 100 classes , with M = 50 randomly drawn examples per class . We define unpermuted , permuted , restored , and test manifolds for each class at each layer of the network . Given a neural network N ( x ) , we extract Nl ( x ) , the activation vector at layer l generated on input x . We then define a manifold by the point cloud of activation vectors generated by inputs of the same class . This results in P manifolds , one for each class . The set of unpermuted manifolds is then obtained by using the corresponding manifolds with examples and labels that were unpermuted . The set of permuted , restored , and test manifolds are defined similarly . We run the MFTMA on each set of manifolds following the procedures in ( Chung et al. , 2018 ; Cohen et al. , 2019 ; Stephenson et al. , 2019 ) . All results are conducted with 10 random seeds , which govern the network initialization , label permutation , and MFMTA analysis initial conditions .

Summary: This paper investigates memorization in deep neural networks (DNNs).  Authors leverage mean field theoretic geometric analysis method (MFTMA) to analyze when and where memorization occurs in a DNN. Through empirical analysis, they show that i) generalizing feature are learned initially and that memorization happen later in training mostly in the top layers ii) we can mitigate memorization by rewinding the top layers parameters to earlier values. They also show that their MFTMA metrics can highlight the phenomena of double decent. Finally, they demonstrate that gradient descent initially ignores noisy example and focus on correctly labeled examples.