Representation mitosis in wide neural networks

Deep neural networks ( DNNs ) defy the classical bias-variance trade-off : adding parameters to a DNN that interpolates its training data will typically improve its generalization performance . Explaining the mechanism behind this “ benign overfitting ” in deep networks remains an outstanding challenge . Here , we study the last hidden layer representations of various state-of-the-art convolutional neural networks and find evidence for an underlying mechanism that we call representation mitosis : if the last hidden representation is wide enough , its neurons tend to split into groups which carry identical information , and differ from each other only by a statistically independent noise . Like in a mitosis process , the number of such groups , or “ clones ” , increases linearly with the width of the layer , but only if the width is above a critical value . We show that a key ingredient to activate mitosis is continuing the training process until the training error is zero . 1 INTRODUCTION . Deep neural networks ( DNN ) routinely have enough parameters to achieve zero training error , even with random labels ( Zhang et al. , 2017 ; Arpit et al. , 2017 ) . In defiance of the classical bias-variance trade-off , the performance of these interpolating classifiers continuously improves as the number of parameters increases well beyond the number of training samples ( Geman et al. , 1992 ; Neyshabur et al. , 2015 ; Spigler et al. , 2019 ; Nakkiran et al. , 2020 ) . Despite recent progress in describing the implicit bias of stochastic gradient descent towards “ good ” minima ( Gunasekar et al. , 2018a ; b ; Soudry et al. , 2018 ; Ji & Telgarsky , 2019 ; Arora et al. , 2019 ; Chizat & Bach , 2020 ) , and the detailed analysis of solvable models of learning ( Advani et al. , 2020 ; Neal et al. , 2018 ; Mei & Montanari , 2019 ; Belkin et al. , 2019 ; Hastie et al. , 2019 ; d ’ Ascoli et al. , 2020 ; Adlam & Pennington , 2020 ; Lin & Dobriban , 2020 ; Geiger et al. , 2020 ) , the mechanisms underlying this “ benign overfitting ” ( Bartlett et al. , 2020 ) in deep NNs remain unclear , especially since “ bad ” local minima exist in their optimisation landscape and SGD can reach them ( Liu et al. , 2020 ) . In this paper , we describe a phenomenon in wide , deep neural networks that we call representation mitosis and which offers a possible mechanism for benign overfitting . We illustrate this mechanism in Fig . 1 for a family of increasingly wide DenseNet40s ( Huang et al. , 2017 ) on CIFAR10 ( Krizhevsky et al. , 2009 ) . The blue line in Fig . 1 shows how the average classification error ( error ) approaches the performance of a large ensemble of networks ( error∞ ) as the width of the network increases ( Geiger et al. , 2020 ) . Consistently with Zagoruyko & Komodakis ( 2016 ) , we find that their performance improves continuously with width . For simplicity we will refer to the width W of last hidden representation as the width the network . When W is greater than 350 a network becomes wide enough to reach zero training error ( see Fig 9-c in Sec A.2 ) and the error decays approximately as W−1/2 . We make our key observation by performing the following experiment : we randomly select a number wc of neurons from the last hidden layer of the widest DenseNet40 , and remove all the other neurons from that layer as well as their connections . We then evaluate the performance of this “ chunk ” of wc neurons , without retraining the network . The performance of chunks of varying sizes is shown in the same figure in orange . There are clearly two regimes : for small chunks , the error decays faster than w −1/2 c , while beyond a critical chunk size w∗c ( shaded area ) , the error of a chunk of wc neurons is roughly the same as the one of a full network with wc neurons . Furthermore , the error of the chunks decays with the same power-law w −1/2 c beyond this critical chunk size . This observation suggests that the final hidden representation of an input in a trained , wide DNN is highly redundant beyond the critical width w∗c . The decay rate of −1/2 in particular implies that in this regime chunks of wc neurons can be thought as statistically independent estimators of the same features of the data , differing only by a small , uncorrelated noise . This suggests a possible mechanism for benign overfitting : as the network becomes wider , additional neurons are first used to learn new features of the data . Beyond the critical width w∗c , additional neurons in the final layer don ’ t fit to new features in the data , and hence over-fit ; instead , they make a copy , or a clone , of a feature that is already part of the final representation . The last layer thus splits into more and more clones as the networks grows wider in a process akin to mitosis in cell biology ( Alberts et al. , 2015 ) , as we illustrate in the bottom of Fig . 1 . The accuracy of these wide networks then improves with their width because the network implicitly averages over an increasing number of clones in its representations to make its prediction . We thus call this effect representation mitosis . This paper provides a quantitative analysis of representation mitosis in non-trivial datasets and architectures . Our main findings can be summarised as follows : 1 . A chunk of wc random neurons of the last hidden representation of a wide neural network predicts the output with an accuracy which scales with w−1/2c if the layer is wide enough and wc is large enough . In this regime we call the chunk a “ clone ” ; 2 . Clones can be linearly mapped one to another , or to the full representation , with an error which can be described as uncorrelated random noise . 3 . Clones are created by training a well regularised model until the model reaches zero training error . If training is stopped too early ( e.g . when the training accuracy is similar to the test accuracy ) , or if the training is performed without sufficient regularization , 1. and 2. do not take place , even if the last representation is very wide . 2 METHODS . 2.1 NEURAL NETWORK ARCHITECTURES . We report experimental results obtained with various architectures ( fully connected networks , WideResNet-28 , DenseNet40 , ResNet50 ) and various data sets ( CIFAR10/100 Krizhevsky et al . ( 2009 ) , ImageNet Deng et al . ( 2009 ) ) . We trained all the networks with SGD with momentum and weight decay following the standard practice to set the relevant hyperparameters . We give detailed information on our training setups in A.1 . All our experiments are run on Volta V100 GPUs . In the following , we discuss how we vary the architecture of the models . Fully-connected networks on MNIST We train a fully-connected network to classify the parity of the MNIST digits ( LeCun & Cortes , 1998 ) ( pMNIST ) following the protocol of Geiger et al . ( 2020 ) . MNIST digits are projected on the first ten principal components , which are then used as inputs of a five layer fully-connected network ( FC5 ) . The four hidden representations have the same width W and the output is a real number whose sign is the predictor of the parity of the input digit . Wide-ResNet-28 and DenseNet40 on CIFAR10/100 We train CIFAR10 and CIFAR100 on family of Wide-ResNet-28 ( Zagoruyko & Komodakis , 2016 ) ( WR28 ) . The number of last hidden neurons in a WR28_n is 64 · n , obtained after average pooling the last 64 · n channels of the network . In our experiments we also analyze two narrow versions of the standard WR28_1 which are not typically used in the literature . We name them WR28_0.25 and WR28_0.5 since they have 1/4 and 1/2 of the number of channels of WR28_1 . Our implementation of DenseNet40 follows the DeseNet40-BC variant ( Huang et al. , 2017 ) . We vary the number of input channels c in { 8 , 16 , 32 , 64 , 128 , 256 } . The number of last hidden features for this architecture is 5.5 · c. ResNet50 on ImageNet We modify the ResNet50 architecture ( He et al. , 2016 ) multiplying by a constant factor c ∈ { 0.25 , 0.5 , 1 , 2 , 4 } the number of channels of all the layers after the input stem . When c = 2 our networks differ from the standard Wide-ResNet50_2 since Zagoruyko & Komodakis ( 2016 ) only double the number of channels of the bottleneck of each ResNet block . As a consequence in our implementation the number of features w after the last pooling layer is w = 2048 · c while in Zagoruyko & Komodakis ( 2016 ) w is fixed to 2048 . 2.2 ANALYSIS METHODS . Reconstructing the wide representation from a smaller chunk To assess the predictive power of the chunk representations we search for the best linear map A , of dimensions W × w , able to minimise the squared difference ( x ( W ) − x̂ ( W ) ) 2 between the W activations of the full layer representation ( x ( W ) ) and the activations predicted from a chunk of size w , x̂ ( W ) = Ax ( w ) . ( 1 ) This least squares problem is solved with ridge regression ( Hastie et al. , 2001 ) with regularization set to 10−8 , and we use the R2 coefficient of the fit to measure the predictive power of a given chunk size . The multi-output R2 value is computed as an average of the W single-output R2 values corresponding to the different coordinates , weighted by the variance of each coordinate . We further compute the covariance matrix Cij ( of dimensions W ×W ) of the residuals of this fit , and then obtain the correlation matrix as ρij = Cij√ CiiCjj + 10−8 , ( 2 ) with a small regularisation in the denominator to avoid instabilities when the standard deviation of the residuals falls below machine precision . To quantify the independence of the chunk representations we take the average of the absolute values of the non-diagonal entries of the correlation matrix ρij . For short we refer to this quantity as a ‘ mean correlation ’ . Reproducibility We provide code to reproduce our experiments and our analysis online at https : //anonymous.4open.science/r/representation_mitosis-EB80 . 3 RESULTS The test error of chunks of wc neurons of the final representation asymptotically scales as w−1/2c The mechanism of representation mitosis is inspired by the following experiment : we compute the accuracy of models obtained by selecting a random subset of wc neurons from the final hidden representation of a wide neural network . We consider three different data sets ( pMNIST , CIFAR10 and CIFAR100 ) and trained networks of width W = 512 for pMNIST and CIFAR10 , and W = 1024 for CIFAR100 . In all these cases , W is large enough to be firmly in the regime where the accuracy of the networks scales ( approximately ) as W−1/2 ( see Fig . 2 ) . We select wc neurons at random and we compute the test accuracy of a network in which we set to zero the activation of all the other w −wc neurons . Importantly , we do not fine-tune the weights after selecting the wc neurons : all the parameters are left unchanged , except that the activations of the “ killed ” neurons are not used to compute the output . We take 500 random samples of neurons for each chunk width wc . In Fig . 3 we plot the test error of the chunked models as a function of wc ( orange lines ) . In all the three networks the behaviour is similar . When wc becomes larger than a critical value w∗c , which depends on the dataset and architecture used , the test error decays as w −1/2 c with the chunk size , the same law observed for full networks of the same width ( Fig . 2 ) . This implies that a model obtained by selecting a random chunk of wc > w∗c neurons from a wide final representation behaves similarly to a full network of width W = wc . Furthermore , a decay with rate −1/2 suggests that the final representation of the wide networks can be thought of as a collection of statistically independent estimates of a finite set of data features relevant for classification . Adding additional neurons to the chunk hence reduces their prediction error in the same way an additional measurement reduces the measurement uncertainty , leading to the −1/2 decay . At smaller wc < w∗c instead , the test error of the chunked models decays faster than w −1/2 c in all the cases we considered , including the DenseNet architecture trained on CIFAR10 shown in Fig . 1 . In this regime , adding neurons to the the final representation , improves the quality of the model significantly quicker than it would in independently trained models of the same width ( see Fig . 1 for a pictorial representation of this process ) . We call chunks of neurons of size wc ≥ w∗c clones . In a wide network , clones can exist only if the width W of the last representation is larger than w∗c , and the maximum number of clones is W/w∗c . In the following we characterize more precisely the properties of the clones . Clones have the same expressive power of the full representation A well trained deep network often represents the salient features of the data set well enough to achieve ( close to ) zero classification error on the training data . In the top panels of Fig . 4 , we show that wide networks are able to interpolate their training set also using just a subset of wc > w∗c random neurons : the dark orange profiles show that when the size of a chunk is greater than ∼ 50 for pMNINST , 100 for CIFAR10 and regions indicate regions where the error of the chunks with wc neurons decays as w −1/2 c . 200 for CIFAR100 , the predictive accuracy on the training set remains almost 100 % . Beyond w∗c , the neurons of the final representation therefore become redundant , since the training error remains ( close to ) zero even after removing neurons from it . We call a chunk of neurons a clone if it fully captures the relevant features of the data , up to some uncorrelated random noise . Clones reconstruct almost perfectly the full representation From a geometrical perspective , the important features of the final representation correspond to directions in which the data landscape show large variations ( Bengio et al. , 2013 ) . A clone can be seen as a chunk that is wide enough to encode almost exactly these directions , but using much less neurons than the full final representation . We analyze this aspect by fitting all the W activations starting from a random chunk of wc activations with ridge regression with a small regularization penalty according to Eq . ( 1 ) . The blue profiles in Fig . 4 , bottom panels , show the R2 coefficient of fit as a function of the chunk size wc for pMNIST ( left ) , CIFAR10 ( center ) , CIFAR100 ( right ) . When wc is really small , say below 6 for pMNIST , 20 for CIFAR10 and 60 for CIFAR100 , the R2 coefficient grows almost linearly with wc . In this regime , adding a randomly chosen activation from the full representation to the chunk increases substantially R2 . When wc becomes larger R2 reaches almost one and the representation enters what we call a mitosis phase . This transition happens when wc is still much smaller than W and correspond approximately to the regime in which the test error starts scaling with the inverse square root of wc ( see Fig . 3 ) . The almost perfect reconstruction of the original data landscape with few neurons can be seen as a consequence of the low intrinsic dimension of the representation ( Ansuini et al. , 2019 ) . The ID of the widest representations gives a lower bound on the number of coordinates required to describe the data manifold , and hence on the neurons that a chunk needs in order to have the same classification accuracy as the whole representation . The ID of the last hidden representation is 2 in pMNIST , 12 in CIFAR10 , 14 in CIFAR100 , numbers which are much lower that the width at which a chunk can be considered a clone . Clones differ from each other by uncorrelated random noise In the mitosis regime , the small residual difference between the representation chunks and the full representation can be approximately described as statistically independent random noise . The green profile of Fig . 4 , bottom panels , show the mean absolute non-diagonal correlation of the residuals of the linear fit , a measure which indicates the level of correlation of the chunk representations and the full representation ( see Methods ) . Before the mitosis width w∗c , the residuals are not only large , but also significantly correlated , but as their width increases above w∗c the correlation drops basically to zero . Therefore , in network which are wider than w∗c any two chunks of equal size wc > w ∗ c can be effectively considered as equivalent copies , or clones , of the same representation , differing only by a small and non-correlated noise , consistently with the scaling law of the error shown in Fig . 3 . The dynamics of mitosis In the previous paragraphs we set forth evidence in support of the hypothesis that large chunks of the final representation of wide DNNs behave approximately like an ensemble of independent measures of the full feature space . This allowed us to interpret the decay of the test error of the full networks with the network width observed empirically in Fig . 2 . The three conditions that a chunked model satisfies in the regime in which its test error decays as w −1/2 c are represented in Fig . 4 : ( i ) the training error of the chunked model is close to zero ; ( ii ) the chunked model can be used to reconstruct the full final representation with an R2 ∼ 1 and ( iii ) the residuals of this reconstruction can be modeled as independent random noise . These three conditions are all observed at the end of the training . We now analyze the dynamics of mitosis . We will see that to enter the mitosis regime , models not only need to be wide enough , but also , crucially , they need to be trained to maximise their performance . Clones are formed in two stages , which occur at different times during training . The first phase begins as soon as training starts : the network gradually adjusts the chunk representations in order to produce independent copies of the data manifold . This can be clearly observed in the left panel of Fig . 5 , which depicts the mean correlation between the residuals of the linear fit from the chunked to the full final representations of the network , the same quantity that we analyze in Fig . 4 , but now as a function of the training epoch . Both Figs . 4 and 5 analyse the WR28-8 on CIFAR10 . As training proceeds , the correlations between residuals diminish gradually until epoch 160 , and becomes particularly low for chunks greater than 64 . After epoch 160 further training does not bring any sizeable reduction in their correlation . At epoch 160 the full network achieves zero error on the training set , as shown in orange in the middle and right panels of Fig . 5 . This event marks the end of the first phase , and the beginning of the second phase of the mitosis process where the training error of the clones keeps decreasing while the full representation ( blue ) has already reached zero training error . For example , chunks of size 64 at epoch 150 have training errors comparable to the test error ( dashed line of the middle panel ) . In the subsequent ∼ 20 epochs the training error of clones of size 128 and 256 reaches exactly zero , and the training error of chunks of size 64 reaches a plateau . Importantly , both phases of mitosis improve the generalization properties of the network . This can be seen in the right panel of Fig . 5 , which reports training and test error of the network , with the two mitosis phases highlighted . The figure shows that both mitosis phases lead to a reduction in the test error , although the first phase leads by far to the greatest reduction , consistently with the fact that the greatest improvements in accuracy typically arise during the first epochs of training . The mitosis process can be considered finished around epoch 180 , when all the clones have reached almost zero error on the training set . After epoch 180 we also observe that the test error stops improving . In Sec . A.2 we report the same analysis done on CIFAR100 ( see Fig . 7 ) and CIFAR10 trained on a DenseNet40 ( see Fig . 9- ( d-e-f ) ) .

Summary: The paper shows empirically that under certain conditions (overtraining, large width in all layers, data augmentation and regularization) neural networks (NNs) tend to learn redundant representations in the last layer. Precisely, a large enough random subset of the penultimate layer activations of size $w_c$ can almost perfectly linearly predict the whole activation of width $W$.  Further, the training error when pruning the network to these random subsets also goes down (lagging behind the training error of the full network) to 0 monotonically as training progresses.  Test error of these pruned networks appears to go down as $w_c^{-1/2}$, similarly to the test error of regular networks going down as $W^{-1/2}$.  The authors put forward some preliminary interpretation of the effect and suggest this redundancy effect could be causally responsible for the phenomenon of generalization of wide networks improving with width.