Identity Crisis: Memorization and Generalization Under Extreme Overparameterization

1 INTRODUCTION . The remarkable empirical success of deep neural networks is often attributed to the availability of large data sets for training . However , sample size does not provide a comprehensive rationale since complex models often outperform simple ones on a given data set , even when the model size exceeds the number of training examples . What form of inductive bias leads to better generalization performance from highly overparameterized models ? Numerous theoretical and empirical studies of inductive bias in deep learning have been conducted in recent years ( Dziugaite & Roy , 2016 ; Kawaguchi et al. , 2017 ; Bartlett et al. , 2017 ; Neyshabur et al. , 2017 ; Liang et al. , 2017 ; Neyshabur et al. , 2018 ; Arora et al. , 2018 ; Zhou et al. , 2019 ) but these postmortem analyses do not identify the root source of the bias . One cult belief among researchers is that gradient-based optimization methods provide an implicit bias toward simple solutions ( Neyshabur et al. , 2014 ; Soudry et al. , 2018 ; Shah et al. , 2018 ; Arora et al. , 2019 ) . However , when a network is sufficiently large ( e.g. , the number of hidden units in each layer is polynomial in the input dimension and the number of training examples ) , then under some mild assumptions , gradient methods are guaranteed to fit the training set perfectly ( Allen-Zhu et al. , 2018b ; Du et al. , 2018a ; b ; Zou et al. , 2018 ) . These results do not distinguish a model trained on a data distribution with strong statistical regularities from one trained on the same inputs but with randomly shuffled labels . Although the former model might achieve good generalization , the latter can only memorize the training labels . Consequently , these analyses do not tell the whole story on the question of inductive bias . Another line of research characterizes sufficient conditions on the input and label distribution that guarantee generalization from a trained network . These conditions range from linear separability ( Brutzkus et al. , 2018 ) to compact structures ( Li & Liang , 2018 ) . While very promising , this direction has thus far identified only structures that can be solved by linear or nearest neighbor classifiers over the original input space . The fact that in many applications deep neural networks significantly outperform these simpler models reveals a gap in our understanding of deep neural networks . As a formal understanding of inductive bias in deep networks has been elusive , we conduct a novel exploration in a highly restrictive setting that admits visualization and quantification of inductive bias , allowing us to compare variations in architecture , optimization procedure , initialization scheme , and hyperparameters . The particular task we investigate is learning an identity mapping in a regression setting . The identity mapping is interesting for four reasons . First , it imposes a structural regularity between the input and output , the type of regularity that could in principle lead to systematic generalization ( He et al. , 2016 ; Hardt & Ma , 2017 ) . Second , it requires that every input feature is transmitted to the output and thus provides a sensitive indicator of whether a model succeeds in passing activations ( and gradients ) between inputs and outputs . Third , conditional image generation is a popular task in the literature ( e.g. , Mirza & Osindero , 2014 ; Ledig et al. , 2017 ) ; an identity mapping is the simplest form of such a generative process . Fourth , and perhaps most importantly , it admits detailed analysis and visualization of model behaviors and hidden representations . Consider networks trained on the identity task with 60k MNIST digits . Although only digit images are presented during the training , one might expect the strong regularity of the task to lead to good generalization to images other than digits . Figure 1 compares three different architectures . The top row shows various input patterns , and the next three rows are outputs from a 20-layer convolutional net ( CNN ) , a 10-layer fully connected net ( FCN ) with rectified-linear unit ( ReLU ) activation functions , and a 1-layer FCN . The 1-layer FCN amounts to a convex optimization problem with infinitely many solutions , however gradient decent converges to a unique closed-form solution . All nets perform well on the training set ( first three columns ) and transfer well to novel digits and digit blends ( columns 4–6 ) . Yet , outside of the hull of hand-printed digits , only the CNN discovers a reasonably good approximation to the identity function . Figure 1 reflects architecture-specific inductive bias that persists even with 60k training examples . Despite this persistence , a model ’ s intrinsic bias is more likely to be revealed with a smaller training set . In this paper , we push this argument to the limit by studying learning with a single training example . Although models are free to reveal their natural proclivities in this maximally overparameterized regime , our initial intuition was that a single example would be uninteresting as models would be algebraically equivalent to the constant function ( e.g. , via biases on output units ) . Further , it seemed inconceivable that inductive biases would be sufficiently strong to learn a mapping close to the identity . Unexpectedly , our experiments show that model behavior is subtle and architecture dependent . In a broad set of experiments , we highlight model characteristics—including depth , initialization , and hyperparameters—that determine where a model lands on the continuum between memorization ( learning a constant function ) and generalization ( learning the identity function ) . The simplicity of the training scenario permits rich characterization of inductive biases . 2 RELATED WORK . The consequences of overparameterized models in deep learning have been extensively studied in recently years , on the optimization landscape and convergence of SGD ( Allen-Zhu et al. , 2018b ; Du et al. , 2018a ; b ; Bassily et al. , 2018 ; Zou et al. , 2018 ; Oymak & Soltanolkotabi , 2018 ) , as well as the generalization guarantees under stronger structural assumptions of the data ( Li & Liang , 2018 ; Brutzkus et al. , 2018 ; Allen-Zhu et al. , 2018a ) . Another line of related work is the study of the implicit regularization effects of SGD on training overparameterized models ( Neyshabur et al. , 2014 ; Zhang et al. , 2017 ; Soudry et al. , 2018 ; Shah et al. , 2018 ; Arora et al. , 2019 ) . The traits of memorization in learning are also explicitly studied from various perspectives such as prioritizing learning of simple patterns ( Arpit et al. , 2017 ) or perfect interpolation of the training set ( Belkin et al. , 2018 ; Feldman , 2019 ) . More recently , coincidentally with the writing of this paper , Radhakrishnan et al . ( 2018 ) reported on the effects of the downsampling operator in convolutional auto-encoders on image memorization . Their empirical framework is similar to ours , fitting CNNs to the autoregression problem with few training examples . We focus on investigating the general inductive bias in the extreme overparameterization case , and study a broader range of network types without enforcing a bottleneck in the architectures . 3 EXPERIMENTS . We explore a progression of models : linear convex models , linear non-convex models ( with multiple linear layers ) , fully-connected multilayered architectures with nonlinearities , and finally the case of greatest practical importance , fully convolutional networks . In all architectures we study , we ensure that there is a simple realization of the identity function ( see Appendix B ) . We train networks by minimizing the mean squared error using standard gradient descent . 3.1 FULLY CONNECTED NETWORKS . Figure 2 shows examples of predictions from multi-layer fully connected networks . Infinitely many solutions exist for all models under this extreme over-parameterization , and the figure shows that all the models fit the training example perfectly . However , on new test examples , contrasting behaviors are observed between shallow and deep networks . In particular , deeper models bias toward predicting a constant output , whereas shallower networks tend to predict random white noises on unseen inputs . The random predictions can be characterized as follows ( proof in Appendix C ) . Theorem 1 . A one-layer fully connected network , when trained with gradient descent on a single training example x̂ , converges to a solution that makes the following prediction on a test example x : f ( x ) = Π‖ ( x ) + RΠ⊥ ( x ) , ( 1 ) where x = Π‖ ( x ) +Π⊥ ( x ) decomposes x into orthogonal components that are parallel and perpendicular to the training example x̂ , respective . R is a random matrix from the network initialization , independent of the training data . For test examples similar to the training example—i.e. , where Π‖ ( x ) dominates Π⊥ ( x ) —the outputs resemble the training output ; on the other hand , for test examples that are not highly correlated to the training example , Π⊥ ( x ) dominates and the outputs looks like white noise due to the random projection by R. The behavior can be empirically verified from the second row in Figure 2 . Specifically , the first test example is a mixture of the training and an unseen test example , and the corresponding output is a mixture of white noise and the training output . For the remaining test examples , the outputs appear random . Although Theorem 1 characterizes only the 1-layer linear case , the empirical results in Figure 2 suggest that shallow ( 2 layer ) networks tend to have this inductive bias . However , this inductive bias does not miraculously obtain good generalization : the trained model fails to learn either the identity or the constant function . Specifically , it predicts well in the vicinity ( measured by correlations ) of the training example x̂ , but further away its predictions are random . In particular , when the test example x is orthogonal to x̂ , the prediction is completely random . In deeper ( 6 layer ) nets , the deviations from the identity function take on a quite different characteristic form . Interestingly , deeper linear networks behave more like deeper ReLU networks , with a strong bias towards a constant function that maps any input to the single training output . A multilayer linear network with no hidden-layer bottleneck has essentially the same representational power as a 1-layer linear network , but gradient descent produces different learning dynamics that alter the inductive biases . See Appendix E for more results and analysis on FCNs . 3.2 CONVOLUTIONAL NETWORKS . We next study the inductive bias of convolutional neural networks with ReLU activation functions . Figure 3 shows predictions on various test patterns obtained by training CNNs of varying depths . Compared to FCNs , CNNs have strong structural constraints that limit the receptive field of each neuron to a spatially local neighborhood and the weights are tied and being used across the spatial array . These two constraints match the structure of the identity target function . ( See Appendix B.3 for an example of constructing the identity function with CNNs . ) Similar to the fully connected case , for one-layer CNN , we can bound the error as follows ( proof in Appendix D ) . Theorem 2 . A one-layer convolutional neural network can learn the identity map from a single training example with the mean squared error over all output pixels bounded as MSE ≤ Õ ( m ( m/C − r ) C ) ( 2 ) where m is the number of network parameters , C is the number of channels in the image , and r ≤ m/C is the rank of the subspace formed by the span of the local input patches . The error grows with m , the number of parameters in the network . For example , learning CNNs with larger receptive field sizes will be harder . Even though the bound seems to decrease with more ( input and output ) channels in the image , note that the number of channels C also contributes to the number of parameters ( m = KHKWC2 ) , so there is a trade-off . Unlike typical generalization bound that decays with number of i.i.d . training examples , we have only one training example here , and the key quantity that reduces the bound is the rank r of the subspace formed by the local image patches . The size of the training image implicitly affects bounds as larger image generates more image patches . Note the rank r also heavily depends on the contents of the training image . For example , simply padding the image with zeros on all boundaries will not reduce the error bound . With enough linearly independent image patches , the subspace becomes full rank r = m/C , and learning of the global identity map is guaranteed . The theorem guarantees only the one-layer case . Empirically—as shown in Figure 3—CNNs with depth up-to-5 layers learn a fairly accurate approximation to the identity function , with the exception of a few artifacts at the boundaries . For a quantitative evaluation , we measure the performance by calculating the correlation ( See Appendix J for the results in MSE . ) to two reference functions : the identity function and the constant function that maps every input to the training point x̂ . To examine how a model ’ s response varies with similarity to the training image , we generate test images having correlation ρ ∈ [ 0 , 1 ] to the training image by : ( 1 ) sampling an image with random pixels , ( 2 ) adding αx̂ to the image , picking α such that the correlation with x̂ is ρ ; ( 3 ) renormalizing the image to be of the same norm as x̂ . For ρ = 0 , the test images are orthogonal to x̂ , whereas for ρ = 1 , the test images equal x̂ . The results for CNNs of different depths are shown in Figure 4 . The quantitative findings are consistent with the visualizations : shallow CNNs are able to learn the identity function from only one training example ; very deep CNNs bias towards the constant function ; and CNNs of intermediate depth correlate well with neither the identity nor the constant function . However , 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Layer index 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Er ro r r at e layers 1 3 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Figure 5 : Illustration of the collapse of predictive power as function of layer ’ s depth . Error rate is measured by using the representations computed at each layer to a simple averaging based classifier on the MNIST test set . The error rate at each layer is plotted for a number of trained CNNs of different depth . The thick red line shows the curve of an untrained 20-layer CNN for reference . unlike FCNs that produce white-noise-like predictions , from Figure 3 CNNs of intermediate depth behave like edge detectors . To evaluate how much information is lost in the intermediate layers , we use the following simple criterion to assess the representation in each layer . We feed each image in the MNIST dataset through a network trained on our single example . We collect the representations at a given layer and perform a simple similarity-weighted classification . For each example xi from the MNIST test set , we predict its class as a weighted average of the ( one hot ) label vector of each example xj from the MNIST training set , where the weight is the inner-product of xi and xj . This metric does not quantify how much information is preserved as the image representation propagates through the layers , because the representation could still be maintained yet not captured by the simple correlation-weighted classifier . It nonetheless provides a simple metric for exploring the ( identity ) mapping : using the input image as the baseline , if a layer represents the identity function , then the representation at that layer would obtain a similar error rate when using the input representation ; on the other hand , if a layer degenerates into the constant function , then the corresponding representation would have error rate close to random guessing of the label . The results are plotted in Figure 5 . The error curve for a randomly initialized 20-layer CNN is shown as reference : at random initialization , the smoothing effect renders the representations beyond the sixth layer unuseful for the averaging-based classifier . After training , the concave nonmonotonicity in the curves indicates loss and then recovery of the information present in the input . Trained networks try to recover the washed out intermediate layer representations as means to link the input and the output layer . However , if the depth is too large , the network tries to infer input-output relations using partial information , resulting in models that behave like edge detectors . Finally , for the case of 20 layers , the curve shows that the bottom few layers do get small improvements in error rate comparing to random initialization , but the big gap between the input and output layer drives the network to learn the constant function instead . On a first sight , this deems to underscore a vanishing gradient problem , but Figure 1 reveals that given a sufficient number of training examples , a 20-layer CNN can still learn the identity map . See also Appendix G for further discussions on vanishing gradients . Since CNNs preserve spatial structure , we can also visualize information loss in the intermediate layers . The visualization results , described in Appendix F , are consistent with the aforementioned observations .

Summary: This paper studies the inductive bias in deep neural networks. The authors train several FF image recognition networks and many CNN variants on a single image, and observe that most networks fall into one of two categories: either memorizing the output of the single training sample, or learning the identity function and generalizing to new, unseen images. The paper is clearly written, present a broad set of experiments, and provides interesting insights, that are somewhat surprising.