Identity Crisis: Memorization and Generalization Under Extreme Overparameterization

1 INTRODUCTION . The remarkable empirical success of deep neural networks is often attributed to the availability of large data sets for training . However , sample size does not provide a comprehensive rationale since complex models often outperform simple ones on a given data set , even when the model size exceeds the number of training examples . What form of inductive bias leads to better generalization performance from highly overparameterized models ? Numerous theoretical and empirical studies of inductive bias in deep learning have been conducted in recent years ( Dziugaite & Roy , 2016 ; Kawaguchi et al. , 2017 ; Bartlett et al. , 2017 ; Neyshabur et al. , 2017 ; Liang et al. , 2017 ; Neyshabur et al. , 2018 ; Arora et al. , 2018 ; Zhou et al. , 2019 ) but these postmortem analyses do not identify the root source of the bias . One cult belief among researchers is that gradient-based optimization methods provide an implicit bias toward simple solutions ( Neyshabur et al. , 2014 ; Soudry et al. , 2018 ; Shah et al. , 2018 ; Arora et al. , 2019 ) . However , when a network is sufficiently large ( e.g. , the number of hidden units in each layer is polynomial in the input dimension and the number of training examples ) , then under some mild assumptions , gradient methods are guaranteed to fit the training set perfectly ( Allen-Zhu et al. , 2018b ; Du et al. , 2018a ; b ; Zou et al. , 2018 ) . These results do not distinguish a model trained on a data distribution with strong statistical regularities from one trained on the same inputs but with randomly shuffled labels . Although the former model might achieve good generalization , the latter can only memorize the training labels . Consequently , these analyses do not tell the whole story on the question of inductive bias . Another line of research characterizes sufficient conditions on the input and label distribution that guarantee generalization from a trained network . These conditions range from linear separability ( Brutzkus et al. , 2018 ) to compact structures ( Li & Liang , 2018 ) . While very promising , this direction has thus far identified only structures that can be solved by linear or nearest neighbor classifiers over the original input space . The fact that in many applications deep neural networks significantly outperform these simpler models reveals a gap in our understanding of deep neural networks . As a formal understanding of inductive bias in deep networks has been elusive , we conduct a novel exploration in a highly restrictive setting that admits visualization and quantification of inductive bias , allowing us to compare variations in architecture , optimization procedure , initialization scheme , and hyperparameters . The particular task we investigate is learning an identity mapping in a regression setting . The identity mapping is interesting for four reasons . First , it imposes a structural regularity between the input and output , the type of regularity that could in principle lead to systematic generalization ( He et al. , 2016 ; Hardt & Ma , 2017 ) . Second , it requires that every input feature is transmitted to the output and thus provides a sensitive indicator of whether a model succeeds in passing activations ( and gradients ) between inputs and outputs . Third , conditional image generation is a popular task in the literature ( e.g. , Mirza & Osindero , 2014 ; Ledig et al. , 2017 ) ; an identity mapping is the simplest form of such a generative process . Fourth , and perhaps most importantly , it admits detailed analysis and visualization of model behaviors and hidden representations . Consider networks trained on the identity task with 60k MNIST digits . Although only digit images are presented during the training , one might expect the strong regularity of the task to lead to good generalization to images other than digits . Figure 1 compares three different architectures . The top row shows various input patterns , and the next three rows are outputs from a 20-layer convolutional net ( CNN ) , a 10-layer fully connected net ( FCN ) with rectified-linear unit ( ReLU ) activation functions , and a 1-layer FCN . The 1-layer FCN amounts to a convex optimization problem with infinitely many solutions , however gradient decent converges to a unique closed-form solution . All nets perform well on the training set ( first three columns ) and transfer well to novel digits and digit blends ( columns 4–6 ) . Yet , outside of the hull of hand-printed digits , only the CNN discovers a reasonably good approximation to the identity function . Figure 1 reflects architecture-specific inductive bias that persists even with 60k training examples . Despite this persistence , a model ’ s intrinsic bias is more likely to be revealed with a smaller training set . In this paper , we push this argument to the limit by studying learning with a single training example . Although models are free to reveal their natural proclivities in this maximally overparameterized regime , our initial intuition was that a single example would be uninteresting as models would be algebraically equivalent to the constant function ( e.g. , via biases on output units ) . Further , it seemed inconceivable that inductive biases would be sufficiently strong to learn a mapping close to the identity . Unexpectedly , our experiments show that model behavior is subtle and architecture dependent . In a broad set of experiments , we highlight model characteristics—including depth , initialization , and hyperparameters—that determine where a model lands on the continuum between memorization ( learning a constant function ) and generalization ( learning the identity function ) . The simplicity of the training scenario permits rich characterization of inductive biases . 2 RELATED WORK . The consequences of overparameterized models in deep learning have been extensively studied in recently years , on the optimization landscape and convergence of SGD ( Allen-Zhu et al. , 2018b ; Du et al. , 2018a ; b ; Bassily et al. , 2018 ; Zou et al. , 2018 ; Oymak & Soltanolkotabi , 2018 ) , as well as the generalization guarantees under stronger structural assumptions of the data ( Li & Liang , 2018 ; Brutzkus et al. , 2018 ; Allen-Zhu et al. , 2018a ) . Another line of related work is the study of the implicit regularization effects of SGD on training overparameterized models ( Neyshabur et al. , 2014 ; Zhang et al. , 2017 ; Soudry et al. , 2018 ; Shah et al. , 2018 ; Arora et al. , 2019 ) . The traits of memorization in learning are also explicitly studied from various perspectives such as prioritizing learning of simple patterns ( Arpit et al. , 2017 ) or perfect interpolation of the training set ( Belkin et al. , 2018 ; Feldman , 2019 ) . More recently , coincidentally with the writing of this paper , Radhakrishnan et al . ( 2018 ) reported on the effects of the downsampling operator in convolutional auto-encoders on image memorization . Their empirical framework is similar to ours , fitting CNNs to the autoregression problem with few training examples . We focus on investigating the general inductive bias in the extreme overparameterization case , and study a broader range of network types without enforcing a bottleneck in the architectures . 3 EXPERIMENTS . We explore a progression of models : linear convex models , linear non-convex models ( with multiple linear layers ) , fully-connected multilayered architectures with nonlinearities , and finally the case of greatest practical importance , fully convolutional networks . In all architectures we study , we ensure that there is a simple realization of the identity function ( see Appendix B ) . We train networks by minimizing the mean squared error using standard gradient descent . 3.1 FULLY CONNECTED NETWORKS . Figure 2 shows examples of predictions from multi-layer fully connected networks . Infinitely many solutions exist for all models under this extreme over-parameterization , and the figure shows that all the models fit the training example perfectly . However , on new test examples , contrasting behaviors are observed between shallow and deep networks . In particular , deeper models bias toward predicting a constant output , whereas shallower networks tend to predict random white noises on unseen inputs . The random predictions can be characterized as follows ( proof in Appendix C ) . Theorem 1 . A one-layer fully connected network , when trained with gradient descent on a single training example x̂ , converges to a solution that makes the following prediction on a test example x : f ( x ) = Π‖ ( x ) + RΠ⊥ ( x ) , ( 1 ) where x = Π‖ ( x ) +Π⊥ ( x ) decomposes x into orthogonal components that are parallel and perpendicular to the training example x̂ , respective . R is a random matrix from the network initialization , independent of the training data . For test examples similar to the training example—i.e. , where Π‖ ( x ) dominates Π⊥ ( x ) —the outputs resemble the training output ; on the other hand , for test examples that are not highly correlated to the training example , Π⊥ ( x ) dominates and the outputs looks like white noise due to the random projection by R. The behavior can be empirically verified from the second row in Figure 2 . Specifically , the first test example is a mixture of the training and an unseen test example , and the corresponding output is a mixture of white noise and the training output . For the remaining test examples , the outputs appear random . Although Theorem 1 characterizes only the 1-layer linear case , the empirical results in Figure 2 suggest that shallow ( 2 layer ) networks tend to have this inductive bias . However , this inductive bias does not miraculously obtain good generalization : the trained model fails to learn either the identity or the constant function . Specifically , it predicts well in the vicinity ( measured by correlations ) of the training example x̂ , but further away its predictions are random . In particular , when the test example x is orthogonal to x̂ , the prediction is completely random . In deeper ( 6 layer ) nets , the deviations from the identity function take on a quite different characteristic form . Interestingly , deeper linear networks behave more like deeper ReLU networks , with a strong bias towards a constant function that maps any input to the single training output . A multilayer linear network with no hidden-layer bottleneck has essentially the same representational power as a 1-layer linear network , but gradient descent produces different learning dynamics that alter the inductive biases . See Appendix E for more results and analysis on FCNs . 3.2 CONVOLUTIONAL NETWORKS . We next study the inductive bias of convolutional neural networks with ReLU activation functions . Figure 3 shows predictions on various test patterns obtained by training CNNs of varying depths . Compared to FCNs , CNNs have strong structural constraints that limit the receptive field of each neuron to a spatially local neighborhood and the weights are tied and being used across the spatial array . These two constraints match the structure of the identity target function . ( See Appendix B.3 for an example of constructing the identity function with CNNs . ) Similar to the fully connected case , for one-layer CNN , we can bound the error as follows ( proof in Appendix D ) . Theorem 2 . A one-layer convolutional neural network can learn the identity map from a single training example with the mean squared error over all output pixels bounded as MSE ≤ Õ ( m ( m/C − r ) C ) ( 2 ) where m is the number of network parameters , C is the number of channels in the image , and r ≤ m/C is the rank of the subspace formed by the span of the local input patches . The error grows with m , the number of parameters in the network . For example , learning CNNs with larger receptive field sizes will be harder . Even though the bound seems to decrease with more ( input and output ) channels in the image , note that the number of channels C also contributes to the number of parameters ( m = KHKWC2 ) , so there is a trade-off . Unlike typical generalization bound that decays with number of i.i.d . training examples , we have only one training example here , and the key quantity that reduces the bound is the rank r of the subspace formed by the local image patches . The size of the training image implicitly affects bounds as larger image generates more image patches . Note the rank r also heavily depends on the contents of the training image . For example , simply padding the image with zeros on all boundaries will not reduce the error bound . With enough linearly independent image patches , the subspace becomes full rank r = m/C , and learning of the global identity map is guaranteed . The theorem guarantees only the one-layer case . Empirically—as shown in Figure 3—CNNs with depth up-to-5 layers learn a fairly accurate approximation to the identity function , with the exception of a few artifacts at the boundaries . For a quantitative evaluation , we measure the performance by calculating the correlation ( See Appendix J for the results in MSE . ) to two reference functions : the identity function and the constant function that maps every input to the training point x̂ . To examine how a model ’ s response varies with similarity to the training image , we generate test images having correlation ρ ∈ [ 0 , 1 ] to the training image by : ( 1 ) sampling an image with random pixels , ( 2 ) adding αx̂ to the image , picking α such that the correlation with x̂ is ρ ; ( 3 ) renormalizing the image to be of the same norm as x̂ . For ρ = 0 , the test images are orthogonal to x̂ , whereas for ρ = 1 , the test images equal x̂ . The results for CNNs of different depths are shown in Figure 4 . The quantitative findings are consistent with the visualizations : shallow CNNs are able to learn the identity function from only one training example ; very deep CNNs bias towards the constant function ; and CNNs of intermediate depth correlate well with neither the identity nor the constant function . However , 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Layer index 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Er ro r r at e layers 1 3 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Figure 5 : Illustration of the collapse of predictive power as function of layer ’ s depth . Error rate is measured by using the representations computed at each layer to a simple averaging based classifier on the MNIST test set . The error rate at each layer is plotted for a number of trained CNNs of different depth . The thick red line shows the curve of an untrained 20-layer CNN for reference . unlike FCNs that produce white-noise-like predictions , from Figure 3 CNNs of intermediate depth behave like edge detectors . To evaluate how much information is lost in the intermediate layers , we use the following simple criterion to assess the representation in each layer . We feed each image in the MNIST dataset through a network trained on our single example . We collect the representations at a given layer and perform a simple similarity-weighted classification . For each example xi from the MNIST test set , we predict its class as a weighted average of the ( one hot ) label vector of each example xj from the MNIST training set , where the weight is the inner-product of xi and xj . This metric does not quantify how much information is preserved as the image representation propagates through the layers , because the representation could still be maintained yet not captured by the simple correlation-weighted classifier . It nonetheless provides a simple metric for exploring the ( identity ) mapping : using the input image as the baseline , if a layer represents the identity function , then the representation at that layer would obtain a similar error rate when using the input representation ; on the other hand , if a layer degenerates into the constant function , then the corresponding representation would have error rate close to random guessing of the label . The results are plotted in Figure 5 . The error curve for a randomly initialized 20-layer CNN is shown as reference : at random initialization , the smoothing effect renders the representations beyond the sixth layer unuseful for the averaging-based classifier . After training , the concave nonmonotonicity in the curves indicates loss and then recovery of the information present in the input . Trained networks try to recover the washed out intermediate layer representations as means to link the input and the output layer . However , if the depth is too large , the network tries to infer input-output relations using partial information , resulting in models that behave like edge detectors . Finally , for the case of 20 layers , the curve shows that the bottom few layers do get small improvements in error rate comparing to random initialization , but the big gap between the input and output layer drives the network to learn the constant function instead . On a first sight , this deems to underscore a vanishing gradient problem , but Figure 1 reveals that given a sufficient number of training examples , a 20-layer CNN can still learn the identity map . See also Appendix G for further discussions on vanishing gradients . Since CNNs preserve spatial structure , we can also visualize information loss in the intermediate layers . The visualization results , described in Appendix F , are consistent with the aforementioned observations .

Summary: This paper studies the inductive bias of neural nets by considering the toy example of learning an identity map through a single data point (and hence the NNs are always overparametrized). The authors compare CNNs versus FCNs, and find that CNNs tend to “generalize” in terms of actually learning the concept of an identity, whereas FCNs are prone to memorization. The authors also present results under various different settings such as changing the filter size or the number of hidden channels of CNNs. The conclusion is that the simpler the network architecture is, the better it generalizes. Another observation is that deep CNNs exhibit extreme memorization.