Representation mitosis in wide neural networks
Authors: Diego Doimo
Aldo Glielmo
Sebastian Goldt
Alessandro Laio
Venue: ICLR 2022 Submitted (2022)

Abstract:   Deep neural networks (DNNs) defy the classical bias-variance trade-off: adding parameters to a DNN that interpolates its training data will typically improve its generalization performance. Explaining the mechanism behind this ``benign overfitting'' in deep networks remains an outstanding challenge. Here, we study the last hidden layer representations of various state-of-the-art convolutional neural networks and find evidence for an underlying mechanism that we call "representation mitosis": if the last hidden representation is wide enough, its neurons tend to split into groups which carry identical information, and differ from each other only by a statistically independent noise. Like in a mitosis process, the number of such groups, or "clones'', increases linearly with the width of the layer, but only if the width is above a critical value. We show that a key ingredient to activate mitosis is continuing the training process until the training error is zero 

Under review as a conference paper at ICLR 2022
REPRESENTATION MITOSIS IN WIDE NEURAL NET- WORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks (DNNs) defy the classical bias-variance trade-off: adding parameters to a DNN that interpolates its training data will typically improve its generalization performance. Explaining the mechanism behind this "benign overfitting" in deep networks remains an outstanding challenge. Here, we study the last hidden layer representations of various state-of-the-ai convolutional neural networks and find evidence for an underlying mechanism that we call representation mitosis: if the last hidden representation is wide enough, its neurons tend to split into groups which carry identical information, and differ from each other only by a statistically independent noise. Like in mitosis process, the number of such groups, or "clones", increases linearly with the width of the layer, but only if the width is above critical value. We show that key ingredient to activate mitosis is continuing the training process until the training error is zero
INTRODUCTION
Deep neural networks (DNN) routinely have enough parameters to achieve zero training error, even with random labels (Zhang et al., 2017; Arpit et al., 2017). In defiance of the classical bias-variance trade-off, the performance of these interpolating classifiers continuously improves the number of parameters increases well beyond the number of training samples (Geman et al., 1992; Neyshabur al., 2015; Spigler al., 2019; Nakkiran et al., 2020). Despite recent progress in describing the implicit bias of stochastic gradient descent towards "good" minima (Gunasekar et al., 201 8a;b; Soudry al., 2018: Ji & Telgarsky, 2019; Arora al., 2019; Chizat & Bach, 2020), and the detailed analysis of solvable models of learning (Advani et al., 2020; Neal et al., 2018; Mei & Montanari, 2019; Belkin et al., 2019; Hastie et al., 2019; d' Ascolii et al., 2020; Adlam & Pennington, 2020; Lin & Dobriban, 2020; Geiger et al., 2020), the mechanisms underlying this "benign overfitting" (Bartlett et al., 2020) in deep NNs remain unclear, especially since "bad" local minima exist in their optimisation landscape and SGD can reach them (Liu et al., 2020).
In this paper, we describe a phenomenon in wide, deep neural networks that we call representation mitosis and which offers possible mechanism for benign overfitting. We illustrate this mechanism in Fig. 1 for a family of increasingly wide DenseNet40s (Huang et al., 2017) on CIFARIO (Krizhevsky al., 2009). The blue line in Fig. shows how the average classification error (error) approaches the performance of a large ensemble of networks (crror as the width of the network increases (Geiger et al., 2020). Consistently with Zagoruyko & Komodakis (2016), we find that their performance improves continuously with width. For simplicity we will refer to the width W of last hidden representation as the width the network. When W is greater than 350 a network becomes wide enough to reach zero training error (see Fig 9-c in Sec A.2) and the error decays approximately as W- We make our key observation by performing the following experiment: we randomly select number we of neurons from the last hidden layer of the widest DenseNet40, and remove all the other neurons from that layer as well as their connections. We then evaluate the performance of this "chunk" of we neurons, without retraining the network. The performance of chunks of varying sizes is shown in the same figure in orange. There are clearly two regimes: for small chunks, the error decays faster than we -1/2 while beyond critical chunk size w (shaded area), the error of chunk of w. neurons is roughly the same as the one of a full network with w. neurons. Furthermore, the error of the chunks decays with the same pawer-law we 1/2 beyond this critical chunk size.

---

Under review as a conference paper at ICLR 2022
representation
b
Input
full network
Block
0.3
0.89
Home
rehwork wr
ofro orror-
0.1
lest error
chunk
0.00
0.00
Input
chunked network
Dense Block
Cot
Dog
68
170
202
704
1400
full netchunk size
add new feature
clone existing features
Relevant Data Features
A5
network width
Figure 1: The mechanism of representation mitosis. (a) We analyse the final representations of deep neural networks (DNN), namely the activities of the last hidden layer of neurons (light blue) We focus on the performance and the statistical properties of randomly chosen subsets of we neurons which we call "chunks", In the chunked network shown here, w. = out neurons are held, and used to predict the output. (b) As we increase the size of the chunk w, in state-of-the-art DNN, here a DenseNet40, the test error of the chunked network (orange line) becomes similar to the test error of full network of width W we (blue line). In this regime, which is reached when w. is larger than a threshold - (shaded area) the error approaches its asymptotic value error x as power-law wa 1/2 (dashed line). (c) Illustration of three final representations for networks of increasing width. As the width of the network increases, additional neurons fit to new features of the data (red neuron). As the network width goes beyond critical width W", additional neurons instead copy features already learnt from data, and form what we call clones. This mechanism, which we call representation mitosis, is suggested by the w. 1/2 decay of the chunk error, and by the statistical analysis present in this paper.
This observation suggests that the final hidden representation of an input in a trained, wide DNN is highly redundant beyond the critical width w The decay rate of -1/2 in particular implies that in this regime chunks of W. neurons can be thought as statistically independent estimators of the same features of the data, differing only by small, uncorrelated noise. This suggests possible mechanism for benign overfitting: as the network becomes wider, additional neurons are first used to learn new features of the data. Beyond the critical width w additional neurons the final layer don't fit to new features in the data, and hence over-fit; instead, they make a copy, or clone, of a feature that already part of the final representation. The last layer thus splits into more and more clones as the networks grows wider in process akin to mitosis in cell biology (Alberts et al., 2015), as we illustrate in the bottom of Fig. 1. The accuracy of these wide networks then improves with their width because the network implicitly averages over an increasing number of clones in its representations to make its prediction. We thus call this effect representation mitosis.
This paper provides a quantitative analysis of representation mitosis in non-trivial datasets and architectures. Our main findings can be summarised as follows:
1. A chunk of w. random neurons of the last hidden representation of wide neural network predicts the output with an accuracy which scales with w. 1/2 if the layer wide enough and W. is large enough. In this regime we call the chunk "clone";
2

---

Under review as a conference paper at ICLR 2022
2. Clones can be linearly mapped one another, to the full representation, with error which can be described as uncorrelated random noise.
3. Clones are created by training a well regularised model until the model reaches zero training error. If training is stopped too early (e.g. when the training accuracy is similar to the test accuracy), or if the training is performed without sufficient regularization, 1. and 2. do not take place, even if the last representation is very wide.
2 METHODS
2.1 NEURAL NETWORK ARCIITECTURES
We report experimental results obtained with various architectures (fully connected networks, Wide- ResNet-28, DenseNet40, ResNet50) and various data sets (CIFAR 10/100 Krizhevsky et al. (2009), ImageNet Deng et al. (2009)). We trained all the networks with SGD with momentum and weight de- cay following the standard practice to set the relevant hyperparameters. We give detailed information on our training setups in A.I. All our experiments are run on Volta V100 GPUs. In the following, we discuss how we vary the architecture of the models.
Fully-connected networks MNIST We train fully-connected network to classify the parity of the MNIST digits (LeCun & Cortes, 1998) (pMNIST) following the protocol of Geiger et al. (2020). MNIST digits are projected on the first ten principal components, which are then used as inputs of five layer fully-connected network (FC5). The four hidden representations have the same width W and the output is real number whose sign is the predictor of the parity of the input digit.
Wide-ResNett-28 and DenseNet40 on CIFAR10/100 We train CIFARIO and CIFAR on family of Wide-ResNet-288 (Zagoruyko & Komodakis, 2016) (WR28). The number of last hidden neurons in WR28 n is 64 n, obtained after average pooling the last 64 n channels of the network. In our experiments we also analyze two narrow versions of the standard WR28_I which are not typically used in the literature. We name them WR28 0.25 and WR28 0.5 since they have 1/4 and 1/2 of the number of channels of WR28 1. Our implementation of DenseNet40 follows the DeseNet40- BC variant (Huang et al., 2017). We vary the number of input channels e {8, 16, 32, 64, 128, 256). The number of last hidden features for this architecture is 5.5 c.
ResNet50 on ImageNet We modify the ResNet50 architecture (He et al., 2016) multiplying by constant factor E (0.25, 0.5, 1, 2, 4] the number of channels of all the layers after the input stem. When = 2 our networks differ from the standard Wide-ResNet500 (2016) only double the number channels of the bottleneck each ResNet block. As a consequence in our implementation the number of features after the last pooling layer is 2048 while in Zagoruyko& & Komodakis (2016) w is fixed to 2048.
since Zagoruyko & Komodakis
2.2 ANALYSIS METHODS
Reconstructing the wide representation from a smaller chunk To assess the predictive power of the chunk representations we search for the best linear map A, of dimensions W w, able to minimise the squared difference (x(W) - between the W activations of the full layer representation (x (x(W) and the activations predicted from a chunk of size w,
(W) (')
(1)
This least squares problem is solved with ridge regression (Hastie al., 2001) with regularization set to 10-8, and we use the R. coefficient of the fit to measure the predictive power of given chunk size. The multi-output R² value is computed as an average of the W single-output R² values corresponding to the different coordinates, weighted by the variance of each coordinate. We further compute the covariance matrix Ci (of dimensions W x W) of the residuals of this fit, and then obtain the correlation matrix
Cj
Pij VC + 10-8
3

---

Under review as a conference paper at ICLR 2022
with small regularisation in the denominator to avoid instabilities when the standard deviation of the residuals falls below machine precision. To quantify the independence of the chunk representations we take the average of the absolute values of the non-diagonal entries of the correlation matrix Pij- For short we refer to this quantity as a 'mean correlation'.
Reproducibility We provide code to reproduce our experiments and our analysis online at https: mitosis-EBBB
//tp:p/pwos..eeoo......aae////e/ www.ooonscceeeeee//eeeeeeeeeeeee // a/aaanonmous.
RESULTS
The test error of chunks of we neurons of the final rep- resentation asymptotically scales as we 1/2 The mech- anism of representation mitosis is inspired by the following experiment: we compute the accuracy of models obtained by selecting a random subset of we neurons from the fi- nal hidden representation of a wide neural network. We consider three different data sets (pMNIST,, CIFARIO and
10a
JANNIST-FOS
error error
10-05
CIFARIDO 10//R/000
(IFAR11 DerooNatto
10-1
10-1.
CIFAR100 and trained networks of width W 512 for PMNIST and CIFAR 10, and W = 1024 for CIFARI00, In all these cases, W is large enough to be firmly in the regime where the accuracy of the networks scales (approx- imately) as --1/2 (see Fig. 2). We select we neurons at random and we compute the test accuracy of a network in which we set to zero the activation of all the other w we neurons. Importantly, we do not fine-tune the weights after selecting the we neurons: all the parameters are left un- changed, except that the activations of the "killed" neurons are not used to compute the output. We take 500 random samples of neurons for each chunk width we-
10-2
10-25
16
32
64
128
256 512 1024
network size
Scaling of the test error with width for various DNN The av- erage test error of neural networks with various architectures approaches the test error of an ensemble of such networks the network width increases. The network size shown here is the width of the final representation. For large width, we find power-law behaviour error error x W-1/2 across data and architectures. Full experimental details in Sec. 2.1
Figure 2:
In Fig. 3 we plot the test error of the chunked models as function of we (orange lines). In all the three networks the behaviour is similar. When we becomes larger than critical value w which depends on the dataset and architecture used, the test error decays as we -12 with the chunk size, the same law observed for full networks of the same width (Fig. 2). This implies that model obtained by selecting a random chunk of we > w neurons from a wide final representation behaves similarly to full network of width W we- Furthermore, decay with rate -1/2 suggests that the final representation of the wide networks can be thought of as collection of statistically independent estimates of a finite set data features relevant for classification. Adding additional neurons to the chunk hence reduces their prediction error in the same way an additional measurement reduces the measurement uncertainty, leading to the -1/2 decay.
At smaller we < " instead, the test error of the chunked models decays faster than we w- in all the cases we considered, including the DenseNet architecture trained on CIFAR 100 shown in Fig. 1. In this regime, adding neurons to the the final representation, improves the quality of the model significantly quicker than it would in independently trained models of the same width (see Fig. I for a pictorial representation of this process). We call chunks of neurons size We Z w clones In wide network, clones can exist only if the width W of the last representation is larger than w and the maximum number of clones is W/w* In the following we characterize more precisely the properties of the clones.
Clones have the same expressive power of the full representation A well trained deep network often represents the salient features of the data set well enough to achieve (close to) zero classification eтror the training data. In the top panels of Fig. 4, show that wide networks are able to interpolate their training set also using just a subset of w, > w random neurons: the dark orange profiles show that when the size of chunk is greater than 50 for PMNINST, 100 for CIFAR10 and
4

---

Under review as a conference paper at ICLR 2022
PMNIST
CIFAR10
CIFAR100
10
Churko
10-25
chunks
0.3G
Chuntz
10-15
10
0.18
10-17
0.05
Test error
10
0.08
10 11
0.25
13
0.08
10-2
0.04
10-11
10-25
10-25
0.08
10-13
0.17
15
32
04
128
255
512
16
32
64
128
250
512
16
32
04
125
250
512 1024
chunk size
chunk size
chunk size
Figure 3: Scaling of the test error of chunks of neurons extracted from the final representation of wide NNs We plot how the test error of chunked networks approaches the error of an ensemble of chunked networks the chunk size we increases. Chunks are formed by selection a number of we neurons at random from the final representation of the widest networks: a FC5 on PMNIST (width
W = 512), and Wide-ResNet-22 for CIFARIO (W 512) and CIFARION 100 (W 1024). The shaded regions indicate regions where the error of the chunks with We neurons decays as we 112
200 for CIFAR 100, the predictive accuracy on the training set remains almost 100%. Beyond w the neurons of the final representation therefore become redundant, since the training error remains (close to) zero even after removing neurons from it. We call a chunk of neurons a clone if it fully captures the relevant features of the data, up to some uncorrelated random noise.
Clones reconstruct almost perfectly the full representation From geometrical perspective, the important features of the final representation correspond to directions in which the data landscape show large variations (Bengio al., 2013). A clone can be seen as a chunk that is wide enough to encode almost exactly these directions, but using much less neurons than the full final representation. We analyze this aspect by fitting all the W activations starting from a random chunk of ut activations with ridge regression with a small regularization penalty according to Eq. (1). The blue profiles in Fig. 4, bottom panels, show the R2 coefficient of fit as a function the chunk size w. for PMNIST (left), CIFAR10 (center), CIFAR 100 (right). When W. is really small, say below 6 for PMNIST, 20 for CIFAR1O and 60 for CIFARI0O, the R² coefficient grows almost linearly with wc. In this regime, adding randomly chosen activation from the full representation to the chunk increases substantially R2. When W. becomes larger R2 reaches almost one and the representation enters what we call mitosis phase. This transition happens when We is still much smaller than W and correspond approximately to the regime in which the test error starts scaling with the inverse square root of w. (see Fig. 3). The almost perfect reconstruction of the original data landscape with few neurons can be seen as a consequence of the low intrinsic dimension of the representation (Ansuini et al., 2019). The ID of the widest representations gives a lower bound on the number of coordinates required to describe the data manifold, and hence on the neurons that a chunk needs in order to have the same in
classification accuracy as the whole representation. The ID of the last hidden representation PMNIST, 12 in CIFARIO, 10, 14 in CIFAR 100, numbers which are much lower that the width at which a chunk can be considered a clone.
Clones differ from each other by uncorrelated random noise In the mitosis regime, the small residual difference between the representation chunks and the full representation can be approximately described as statistically independent random noise. The green profile of Fig. 4, bottom panels, show the mean absolute non-diagonal correlation of the residuals of the linear fit, measure which indicates the level of correlation of the chunk representations and the full representation (see Methods), Before the mitosis width w the residuals are not only large, but also significantly correlated, but as their width increases above w the correlation drops basically to zero. Therefore, in network which are wider than w any two chunks of equal size w. > w can be effectively considered equivalent copies, or clones, of the same representation, differing only by small and non-correlated noise, consistently with the scaling law of the error shown in Fig. 3.
5

---

Under review as a conference paper at ICLR 2022
PMNIST
b
CIFAR10
C
CIFAR100
training error
10%
10%
10%
Sharks WObed
Surks
videdt
0.1%
0.1%
0.1%6
32
125
512
32
125
512
04
258
1024
full widllrchunk size
full width/chunk size
full widllvchunk size
0.3
0.00
0.75
0.75
0.08
maan COT:
R corralation residuals
0.04
0.50
0.50
0.50
0.25
0.25
0.00
0.0
0.00
0.00
0.00
0.00
2
8
32
125
512
2
32
125
512
4
16
64
206
1024
chunk size
chunk size
chunk size
Figure 4: The three signatures of representation mitosis (i) The training errors of the full networks (blue) and of the chunks taken from the widest network (orange) approach zero beyond critical width / chunk size, resp. (panels a-c). (ii) The final representation of the widest network can be reconstructed from a chunk using linear regression (I) with explained variance Ri2 close (blue lines in panels d-f). (iii) The residuals of the linear map can be modeled as independent noise: we show this by plotting the mean correlation of these residuals (green line, panels d-f), averaged over 100 reconstructions starting from different chunks. A low correlation at high R² indicates that the chunk contains the information of the full representation with some statistically independent noise. Experimental setup: FC5 on PMNIST, Wide ResNet-28 on CIFAR 10/100. Full details in Methods section 2.1
The dynamics of mitosis In the previous paragraphs we set forth evidence in support of the hypothesis that large chunks of the final representation of wide DNNs behave approximately like an ensemble of independent measures of the full feature space. This allowed us interpret the decay of the test error of the full networks with the network width observed empirically in Fig. 2. The three conditions that chunked model satisfies in the regime in which its test error decays as w. -////2 are represented Fig. 4: (i) the training error of the chunked model is close to zero; (ii) the chunked model can be used reconstruct the full final representation with an R 1 and (iii) the residuals of this reconstruction can be modeled as independent random noise. These three conditions are all observed the end of the training. We analyze the dynamics of mitosis. We will see that to enter the mitosis regime, models not only need to be wide enough, but also, crucially, they need to be trained maximise their performance.
Clones are formed in two stages, which occur at different times during training. The first phase begins as soon as training starts: the network gradually adjusts the chunk representations order to produce independent copies of the data manifold. This can be clearly observed in the left panel of Fig. 5, which depicts the mean correlation between the residuals of the linear fit from the chunked the full final representations of the network, the same quantity that we analyze in Fig. 4, but now as function of the training epoch. Both Figs. 4 and 5 analyse the WR28-8 on CIFAR 10. As training proceeds, the correlations between residuals diminish gradually until epoch 160, and becomes particularly low for chunks greater than 64. After epoch 160 further training does not bring any sizeable reduction in their correlation. At epoch 160 the full network achieves zero eтror on the training set, as shown in orange in the middle and right panels of Fig. 5. This event marks the end of the first phase, and the beginning of the second phase of the mitosis process where the training error of the clones keeps decreasing while the full representation (blue) has already reached zero training error. For example, chunks of size 64 at epoch 150 have training errors comparable to the test error (dashed line of the

---

Under review as a conference paper at ICLR 2022
churk size
franning ener
0.200
0.10
test eeror phassl
mean corr.
32
0.08
churt AIPA
0.00
phase?
0.10
120
training
0.04
0.04
120
0.03
206
l0st 61101
0.00
100
190170 250
100
190
170
200
190
160
170
180
190
200
opoch
opoch
opoch
Figure 5: The onset of mitosis during training. As in Fig. 4, we show the mean correlation of the residuals of the linear reconstruction the final representation from chunks, but this time as function of training epochs. A small correlation indicates that the reconstruction error in going from chunks to final representation can be modelled as independent noise. Data obtained from the same WR28 8 trained on CIFAR 10 in Fig. 4. b: Training error during training for chunks of different sizes. After the network has reached zero training error at ~ 160 epochs, continuing to train improves the training accuracy of the chunks. Test and training error during training for the full network. Between epoch 160 and 180 the clones of the full network progressively achieve zero training error. In the same epochs, one observes a small improvement in the test error.
middle panel). In the subsequent ~ 20 epochs the training error of clones of size 128 and 256 reaches exactly zero, and the training error of chunks of size 64 reaches plateau.
Importantly, both phases of mitosis improve the generalization properties of the network. This can be seen in the right panel of Fig. 5, which reports training and test error of the network, with the two mitosis phases highlighted. The figure shows that both mitosis phases lead to reduction in the test eπror, although the first phase leads by far to the greatest reduction, consistently with the fact that the greatest improvements in accuracy typically arise during the first epochs of training. The mitosis process can be considered finished around epoch 180, when all the clones have reached almost error on the training set. After epoch 180 we also observe that the test error stops improving. In Sec. A.2 we report the same analysis done on CIFARION (see Fig. 7) and CIFARIO trained DenseNet40 (see Fig. 9-(d-e-f))
a
0.99
b
0.25
-B-
regularized well regularized Wy-05
chunk size
error er
0.4
32
mean corr.
0.15
-8
256
0.10
0:13
0.05
0.11
0.00
128
200
chunk size
epoch
Figure 6: A network trained without regularization on CIFAR1O. a: the test error of chunks of Rese2288 trained without data augmentation and weight decay (blue) and for well- regularized network (orange, taken from Figure 3-b). b: Mean correlation between residuals of the linear reconstruction of the full representation from chunks of different sizes for two networks: one trained without data augmentation and weight decay (thick lines), and one using state-of of-the-art techniques (thin lines, same data as in Fig. 5-a).
7

---

Under review as a conference paper at ICLR 2022
3.1 LIMITATIONS
Clones appear only in well-trained regularized networks So far in this work we have shown only examples of networks and datasets which representation mitosis takes place. However, the onset of this mechanism is related to nontrivial details in the learning algorithm. If the network is not trained using state-of-the-ar procedures, none of the signatures described above appear even if the width of the final representation is much larger than W Figure 6 shows the case of the Wide-ResNee1288 analyzed in Fig. 5 trained on exactly the same dataset (CIFARIO 10) but without data augmentation and weight decay. As shown in panel a, in the network trained without sufficient regularization (blue line) the error does not scale as wte 11/2 not even for large we- This, as we have seen, indicates that the last representation cannot be split in chunks. Conrespondingly, the chunks cannot be mapped to the full representation by a linear transformation: the mean correlation of the residuals of the linear map of the chunks to the full representation remains approximately constant during training, and is always much higher then what we observed for the same architecture and dataset, when training is performed with weight decay and data augmentation (panel b). We performed a similar analysis on the DenseNer40 (see Fig. 10), observing an analogous trend.
Reaching mitosis on ImageNet A second scenario where we didn't observe the hallmarks of mitosis is ImageNet. We trained a family of ResNet50 where we multiply all the channels of the layers after the input stem by constant factor E f0.25, 0.5, 1. 2, 43. In this manner the widest final representation we consider consists of 8192 neurons, which is four times wider than both the standard ResNet50 (He et al., 2016 6) and its wider version (Zagoruyko & Komodakis, 2016) (see Sec. 2. I). We trained all the networks following the standard protocols and achieved test errors comparable or slightly lower than those reported in the literature (see Sec. A.1).
In this setting none of the elements associated with a development of independent clones can be observed. The scaling of the test error of the chunks is steeper than we 1/2 (see Fig. 8-a) suggesting that chunks remain significantly correlated to each other. Figure 8-b shows that the mean correlation of the residuals does not decrease during training, as it happens for the networks we trained on CIFARIO 10 and CIFARIOO. We conclude that a representation with 8192 neurons seems too narrow to encode all the relevant features redundantly on ImageNet, which has 1000 classes. Indeed, the top-1 classification error of the largest ResNet50 on the training set is ~ 92%, well below the interpolation threshold and a chunk large as 4096 activations is not able to reconstruct all the relevant variations of the data as it does in the cases analyzed in Sec. 3 (see Fig. 8-c).
4 DISCUSSION
This work is an attempt explain the apparently paradoxical observation that over preeeaaamttriiioioonn boosts the performance of DNNs. This "paradox" is actually not a peculiarity of DNNs: if one trains a prediction model with n parameters using the same training set, but starting from independent initial weights and receiving samples in an independent way, one can obtain, say, models which, in suitable conditions, provide predictions of the same quantity with independent noise due to initialization, SGD schedule, etc. If one estimates the target quantity by an ensemble average, the statistical error will (ideally) scale with m 1/2, and therefore with N 1/2 where N = n m is the total number of parameters of the combined model. This will happen even if N much larger than the number of data.
What is less trivial is that a DNN is able to accomplish this scaling within a single model, in which all the parameters are optimised collectively via minimization of single loss function. Our work describes a possible mechanism at the basis of this phenomenon in the special case of a neural networks in which the last layer is very wide. We observe that if the layer is wide enough, random subsets of its neurons can be viewed as approximately independent representations of the same data manifold (or clones). This implies a scaling of the error with the width of the layer as W- which is qualitatively consistent with our observations.
The impact of network architecture. The capability of a network to produce statistically inde- pendent clones in its last layer is architecture -eepeurrrddeeeeeett, since we find that the network width W required enter the mitosis regime needs to be large, in Fig. 3. The total number of parameters
8

---

Under review as a conference paper at ICLR 2022
in DNN scales with W in manner that, in general, depends on the architecture. If the mitosis mechanism we propose is correct, the scaling of the test error with N will thus be, in general, architecture-deeeedeett At the same time, we also verified that increasing the width of only the final representation is not sufficient to enter the mitosis regime. We give an example of this effect in Fig. 11 in Sec. A.2, where we show that the test error of Wide-ResNet28 CIFAR 10 does not decrease if only the width of the final representation is increased, while the rest of the architecture is kept at constant width.
The impact of training. Even for wide enough architectures, clones appear only if the training schedule is appropriately chosen. In our examples, by stopping the training too early, for example when the training error is similar to the test error, the chunks of the last representation would not become entirely independent from one another, and therefore they could not be considered clones. In fact, we have seen that the separation of the clones is completed only when the test error on a model restricted each clone becomes very small.
Neural scaling laws Capturing the asymptotic performance of neural network via scaling laws is another active research area. Hestness et al. (2017) gave an experimental analysis of scaling laws w.r.t the training data set size in a variety of domains. Rosenfeld et al. (2020); Kaplan et al. (2020) experimentally explored the scaling of the generalisation error of deep networks with the number of parameters/data points across architectures and application domains for supervised learning, while Henighan et al. (2020) identified empirical scaling laws in generative models. Geiger et al. (2020) found that the generalisation error of fully-connected networks trained on the PMNIST task we also consider scales with the number of parameters. Bahri al. (2021) showed the existence of four scaling regimes and described them theoretically in the setting of random features, which is an instance of lazy learning (Chizat et al., 2019). Sharma & Kaplan (2020) relate the exponent of scaling laws with respect to the number of training samples to the dimension of the data manifold. Our analysis supports the hypothesis that the test error in wide neural networks scales as the inverse square root of the width, if the width is large enough, and if the classification task is not too complex.
Relation to theoretical results in the mean-field regime. Our empirical results also agree with recent thcoretical results that were obtained for two-layer neural networks (Mei et al., 2018; Rotskoff & Vijndnn 2018; Chizat & Bach, 2018; Sirignano & Spiliopoulos, 2019; Goldt et al., 2019; Refinetti et al., 2021). These works characterise the optimal solutions of two-layer networks trained on synthetic datasets with some controlled features. In the limit of infinite training data, these optimal solutions correspond to networks where neurons in the hidden layer duplicate the key features of the data. These "denoising solutions" "distributional fixed points" were found for networks with wide hidden layer (Mei et al., 2018; Rotskoff & Vanden-Fijnded Spiliopoulos, 2019) and wide input dimension (Goldt et al., 2019; Refinetti et al., 2021). Another point of connection with the thcoretical literature is the concept of dropout stability. A network is said to be c-dropout stable it its training loss changes by less than c when half the neurons are removed at random from each of its layers Kuditipudi et al. (2019). Dropout stability has been rigorously linked to several phenomena in neural networks, such as the connectedness of the minima of their training landscape Shevchenko & Mondelli (2020); Nguyen et al. (2021).
2018; Chizat & Bach, 2018; Sirignano &
Implicit ensembling in deep learning The success of various deep learning architectures and techniques has been linked to some form of ensembling. Veit et al. (2016) proposed that the performance of ResNets (He et al., 2016) stems from effective ensembling of shallower networks due to the residual connections. The successful dropout regularisation technique (Hinton et al., 2012; Srivastava al., 2014) samples from an exponential number of "thinned" networks during training to prevent co-adaptation of hidden units. While this can be seen as a form of (implicit) ensembling, here we make the observation that co-adaptation of hidden units in the form of clones occurs without dropout, and is crucial for their improving performance with width. Recent theoretical work on random features (d' Ascoli et al., 2020; Adlam & Pennington, 2020; Lin & Dobriban, 2020) also suggests that ensembling and over-parameterisation are two sides of the same coin, at least in the lazy regime of neural networks (Chizat et al., 2019), where the weights of the networks' hidden layers don't 't move much during training. In this paper, instead we focus networks in the feature learning regime where weights move significantly. We hope that our results motivate theoretical studies along those lines in the feature learning regime.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
B. Adlam and J. Pennington. Understanding double descent requires fine-grained bias-variance decomposition. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 11022-11033 Curran As- sociates, Inc., 2020. URL httLtttta///prreeediing neurips ctteiis.cccpappee wttp:/cppppppppeeeeeeeeeeee /2t/p//rre/////////// file/ Ta22ee223333221111e0001111123332
M.S. Advani, A.M. Saxe, and H. Sompolinsky, High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428 446, 2020.
B. Alberts, D. Bray, K. Hopkin, A.D. Johnson, J. Lewis, M. Raff, K. Roberts, and P. Walter. Essential cell biology. Garland Science, 2015.
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. In Advances in Neural Information Processing Systems, pp. 6109-6119 19, 2019.
Arora, N. Cohen, Hu, and Y. Luo. Implicit regularization deep matrix factoriza- tion. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL pestps://proceee c0c783b5 d80 (o8.fffdd7d88111cc1o f1.811d1446 4aw.ee00011111ee6eeeee (ew.:e4444ee2200----oe
ngs neurips pdf.
weeerips.ccppapeeeee wccppapprr/00// 9/ file/
D. Arpit, S. Jastrzebskk, M.S. Kanwal, T. Maharaj, A. Fischer, A. Courville, and Y. Bengio. A Closer Look at Memorization in Deep Networks. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma. Explaining neural scaling laws. arXiv preprint 10X.v:2102.06701, 2021.
P.L.. Bartlett, P.M. Long, G. Lugosi, and A. Tsigler. Benign overfitting in linear regression. Proceed- ings of the National Academy of Sciences, 117(48):30063-3000 2020.
M. Belkin, D. Hsu, Ma, and Mandal. Reconciling modern machine-learinng practice and the classical biri-variance trade-off. Proceedings of the National Academy of Sciences, 116(32): 15849-158544 2019.
Bello, W. Fedus, X. Du, E.D. Cubuk, A. Srinivas, T-Y. Lin, Shlens, and B. Zoph. Revisiting resnets: Improved training and scaling strategies. arXiv preprint arXiv:2103.07579. 2021.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1822 2013.
L. Chizat and Bach. On the global convergence of gradient descent for over pver-parameterized models using optimal transport. In Advances in Neural Information Processing Systems 31, pp. 3040-3050, 2018.
L. Chizat and F. Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference Learning Theory, pp. 1305-1338. PMLR, 2020.
L. Chizat, E. Oyallon, and F. Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems, pp. 2937-2947, 2019.
S. d' Ascoli, M. Refinetti, G. Biroli, and F. Krzakala. Double trouble in double descent: Bias and variance(s) in the lazy regime. In ICML,, 2020.
Deng, Dong, R. Socher, L.-J. Li, Li, and L.. Fei-Fei. ImageNet: Large-Scale Hierarchical Image Database. In CVPR09, 2009.
M. Geiger, A. Jacot, S. Spigler, F. Gabriel, L. Sagun, S. d' d'Ascoli, G. Biroli, C. Hongler, and M. Wyart. Scaling description generalization with number of parameters in deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2020())0034011, 2020.
2020.
10

---

Under review as a conference paper at ICLR 2022
S. Geman, E. Bienenstock,, and R. Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4-1):1-58, 1992.
Goldt, M.S. Advani, A.M. Saxe, Krzakala, and L. Zdeborová. Dynamics of stochastic gradi- ent descent for two-layer neural networks in the teacher-student setup. In Advances in Neural Information Processing Systems 32, 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 706.02677, 2017.
S. Gunasekar, Lee, D. Soudry, and N. Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning, pp. 1832-1841. PMLR, 201 8a.
Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. 31, 2018b. URL Pap 2ew.eff6eeeeeeeee2222222222
neur ips. cc/
paper/ 2018
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements Statistical Learning. Springer Series Statistics. Springer New York Inc., New York, NY, USA, 2001.
Trevor Hastic, Andrea Montanari, Saharon Rosset, and Ryan Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint (000:/9933.08666 2019.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Henighan, Kaplan, M. Katz, M. Chen, C. Hesse, Jackson, H. Jun, T.B. Brown, P. Dhari- wal, S. Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. Patwary, M. Ali, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv: 2017.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580. 2012. URL http: / rxiv org/ abs 1207. 0580.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Z. Ji and M. Telgarsky. The implicit bias of gradient descent nonseparable data. In Conference Learning Theory, pp. 1772-1798. PMLR, 2019.
J. Kaplan, S. McCandlish, T. Henighan, T.B. Brown, B. Chess, R. Child, S. Gray, A. Radford, Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001. 08361, 2020.
A. Krizhevsky, G. Hinton, et al. Leanning multiple layers of features from tiny images. 2009. URL https: www. cs..wwwc........................ cs. torontoroooooooooo.. wdw//krii// wet.tii//eeaeannneeeeeee feaae--fauuurseee 2200000TRR... pdÉ.
Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Sanjeev Arora, and Rong Ge. Explaining landscape connectivity of low-cost solutions for multilayer nets. arXiv preprint 1906.0900.0622 2019.
Y. LeCun and C. Cortes. The MNIST database of handwritten digits, 1998.
Licong Lin and Edgar Dobriban. What causes the test error? going beyond bias-variance via anova. arXiv preprint arXhv:2010.05170, 2020.
S. Liu, D. Papailiopoulos, and D. Achlioptas. Bad global minima exist and sgd can reach them. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 8543 8552. Curran Asso- ciates, Inc., 2020. URL https: wttp:/ceeee..nneee.e..eee.eee.ee www.ipe.....c 61 84 91 a 9b68 6. 7e 1 50:.9911112999949999999 -aaeer..d... .dd..
11

---

Under review as a conference paper at ICLR 2022
S. Mei, A. Montanari, and P. Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):77655-E7)) E7671, 2018.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 2019.
P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu. Mixed precision training. In International Confer- ence on Learning Representations, 2018. URL https: cettps://opeerevttii net wwtp://eeeeeooummeeeeemm for r1gs9JgR2.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. International Conference on Learning Representations, 2020. URL https: openreview. net/ wtt/m//rummmm/
B. Neal, S. Mittal, A. Baratin, Tantia, M. Scicluna, S. Lacoste-Julien, and Mitliagkas. A modern take on the bias-variance tradeoff in neural networks. arXiv preprint 1100v1181. 2018.
B. Neyshabur, R. Tomioka, and N. Srebro. In search of the real inductive bias: On the role of implicit regularization deep learning. In ICLR, 2015.
Quynh Nguyen, Pierre Brechet, and Marco Mondelli. On connectivity of solutions in deep learn- ing: The role of over-paramettrriation and feature quality. Advances Neural Information Processing Systems, volume 34. Curran Associates, Inc., 2021.
M. Refinetti, S. Goldt, F. Krzakala, and L. Zdeborová. Classifying high-dimensional gaussian mixtures: Where kernel methods fail and neural networks succeed. In ICML, 2021.
J.S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit. A constructive prediction of the generaliza- tion error across scales. In International Conference on Learning Representations, 2020. URL https: /pperr/ieeeeeeeeeeeeeeee net forum? forumt?1==ryenvppK03
G.M. Rotskoff and E. Vanden-Eijnden. Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks. In Advances in Neural Information Processing Systems 3/, pp. 7146-7155, 2018.
U. Sharma and J. Kaplan. A neural scaling law from the dimension of the data manifold. arXiv preprint arXiv:2004. 10802, 2020.
Alexander Shevehenko and Marco Mondelli. Landscape connectivity and dropout stability of sgd solutions for pvrr-parameterized neural networks. In International Conference Machine Learning, pp. 8773-8787 8784. PMLR, 2020.
J. Sirignano and K. Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 2019.
D. Soudry, E. Hoffer, and N. Srebro. The implicit bias gradient descent on separable data. In International Conference Learning Representations, 2018.
Spigler, M Geiger, S d' Ascoli, Sagun, Biroli, and M Wyart. jamming transition from under-to pvereparametrizzttion affects generalization in deep learning. Journal of Physics A: Mathematical and Theoretical, 52(47):474001, 2019.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()): 1929-1958, 2014.
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Asso- ciates, Inc., 2016. URL httpa://proceedingg/////////// neurips....uu...... http://rrrrrppppccaa//////////// pdf.
3
12

---

Under review as a conference paper at ICLR 2022
S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint
2016. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 HYPERPARAMETERS USED AND TRAINING PROCEDURES
conneccd networks on MNIST We train the fully-connected networks for 5000 epochs with stochastic gradient descent using the following hyperparameters: batch size 256, momentum 0.9, learning rate 10-3, weight decay 10-2. We decrease the initial learning rate with cosine schedule.
Wide-RessNet-28 and DenseNet40-BC on CIFAR10/100 All the models are trained for 200 epochs with stochastic gradient descent with batch size 128, momentum 0.9, and cosine annealing scheduler starting with a learning rate of 0.1. The training set is augmented with horizontal flips with 50% probability and random cropping the images padded with four pixels each side. On CIFARIO 10 trained on WR28 we select a weight decay equal to 5 10-4 and label smoothing magnitudes equal 0.1 for WR28 {0.25, 0.5. 2). On CIFARIO trained on Densenet40-BB we set select weight decay equal to 5 10 and label smoothing magnitudes equal to 0,05 for all the networks On CIFAR100 trained on WR28 we set weight decays equal to {10, 5, 5. 5--10 -4 and label smoothing magnitudes equal to (0.1. 0.07, 0.05, 0, 0) for WR28_ 2, 4, 8, 16) respectively. All the hyperparameters were selected with small grid search.
ResNet50 on ImageNet We train all the ResNet50 with mixed precision (Micikevicius et al., 2018) for 120 epochs with a weight decay of 4 10-5 and label smoothing rate of 0.1 (Bello et al., 2021). The input size is 224 224 and the training set is augmented with random crops and horizontal flips with 50% probability. The per-GPU batch size is set to 128 and is halved for the widest networks fit the GPU memory. The networks are trained on 16 Volta V100 GPUs as keep the batch size B equal to 1024. The learning rate is increased linearly from 0 to 0.1-63 (Goyal et al., 2017) for the first five epochs and then annealed zero with a cosine schedule.
Table 1: Test accuracy (average over four runs)
CIFARIO
CIFARI00
ImageNet (topl)
network
accuracy
network
accuracy
network
accuracy
Wide-RN28 0.25
84.1
Wide-RN28
70.4
RN50_0.25
67.0
Wide-RN28 0.5
90.3
Wide-RN28 2
75.7
RN50 0..
74.1
Wide-RN28
93.4
Wide-RN28 4
79.6
RN50
77.6
Wide-RN28_2
95.2
Wide-RN28_8
80.8
RN50
79.1
Wide-RN28_4 4
95.9
Wide-RN28_I6
81.9
RN50_4
79.5
Wide-RN28 8
96.1
DenseNet40-BC (k=8)
91.6
DenseNer4O-BC (k=16)
93.9
DenseNer40-BC. (k=32)
95.1
DenseNer40-BCC (k=64)
95.7
DenseNet40-BC (k=128)
96.0
14

---

Under review as a conference paper at ICLR 2022
A.2
ADDITIONAL EXPERIMENTS
chank
0120
1004
0.70
mban cort.
125
training
0.18
120
0.16
Banning
prosee
0.073
012
256
0.10
0.10
phoso
0.000
0.025
0.003
100
130-550
200
100
150
200
140
150
150
200
epoch
epoch
epoch
Figure 7: Dynamics of mitosis on CIFARIO0 a:As Fig. 4, we show the mean correlation residuals of the linear reconstruction of the final representation of Wide- Resnet28 8 from chunks, but this time as a function training epochs, A small correlation indicates that the reconstruction error in going from chunks final representation error of chunks of a Wide-Resnet28 8 and its full layer representation. From epoch 150 to epoch 185 the training error of the chunks with size 128/256 decreases below 0.5%, while for smaller chunks sizes it remains above 5%. Random chunks with size larger than 128/256 can fit the training set, thus having the same representational power as the whole network on the training data. For W 128/256 the test accuracy is decaying approximately with the same law that of independent networks with the same width (see Fig. 3). This picture suggests that for CIFAR100 the size of clone 128/256, slightly larger than the size of the clones in CIFARIA 10. c: Training and test error dynamics for the same Wide-Rest ResNer28 8. After epoch 150 the training error of the full network remains consistently smaller than 0.1 (orange profile) while the test error continues decrease until epoch 185 from 0. 199 to 0. 1765 (blue protile). In the same range of epochs (150-185) the training error of smaller
be modelled independent noise. b: Training
chunks decreases sensibly (see panel b).
b
020
0.025
null not chunks
chunk dize
efror o
test error
mean COTT.
0.020
2048
512
1024
0.010
4060
0.03
0.1
0.00
8.00
512
1024
2045
4085
5182
100 120
2015 6192
width/
size
opoch
Figure 8: No representation mitosis in ResNet50 on ImageNet a: Decay of the test eтror as function of the network width (blue) and for chunks of the widest ResNet50 (orange) to the error of an ensemble of ResNet50_ 4. The ensemble consists of four networks. b: Mean correlation (see Sec. 2.2) of the residuals of the linear map of a chunk of the last hidden representation to the full representation. The network examined is ResNet50 4. c: R² coefficient of the ridge regression fit of a chunk of the last hidden representation of ResNet50 4 to full layer representation.
15

---

Under review as a conference paper at ICLR 2022
a
0.3
0.39
0.1
0.13
0.78
10"
0.00
Test
rasiduals
10"
001
0.00
60
10
uadees not
notwork
1400
1408
22
262
1405
0,150
0.06
0.129
0.30
M
and of phone
0,100
704
and of plass
120
1040
0.000
240
0.22
0.000
0.00
0.00
100 50 801000 1585C 190000
130
142
150
189
110
200
180
180
1771
189
130)
200
Figure 9: The representation mitosis for DenseNet40 architecture. Decay of the test of independent networks (blue) and chunks of the widest network (orange) to the error of an ensemble average of ten of the widest networks (DenseNet4O-BCC k=128) b: Blue profile: Rt² coefficient of the ridge regression of chunk of we neurons (æ-axis) to the full layer representation. Green profile: mean correlation of the residuals of the mapping as described various DenseNet40 of increasing width (blue) and of chunks of the widest architecture (orange). d: The mean correlation of the residuals from the linear reconstruction of the final representation from chunks of a given size for DenseNer40-BC (k=128) during training. e: Training error dynamics of chunks a DenseNet40-BC (k=128). f: Training and test error dynamics for DenseNet4O-BCC (k=128).
Sec. 2.2. c: Training error of
10a
chunk size
Widest net
0,150
error erro
10-95
mean corr.
0.125
0.100
178
10
0.075
701
0.060
10-1.
0.025
0,000
10-2
chunk size
epoch
Figure 10: Densenet40 without mitosis A DenseNet40-BB (k=128) trained on CIFARIO without weight decay and data augmentation. This experiment reproduces DenseNet the analysis shown on a Wide-ResNet288 in Sec. 3.1. It shows that a: also in a DenseNet architecture not well regularized error -error decays faster than w. 1/2 and b: the mean correlation of the residuals do not decrease during training. The thin profiles of panel b are the same as those shown in Fig. 9-d.
16

---

Under review as a conference paper at ICLR 2022
CIFAR10 WR28 (T-shaped)
0.347
0.13
WR28800.5
test error
1(16)
WR28 ((112)
0.062
WF228
0.04
standlrrd network (av 20 runs, Fig. 1)
0.034
16
32
64
128
256
512
n" activations
Figure 11: Wide-RN28 with a wide output layer trained on CIFAR10 We tested whether is enough increase the width of the final representation the width of the full network. We trained a ResNet28 1 increasing only the number channels in the last layer. We modified the number of output channels of the last block of conv4 and analysed the representation after average pooling, as we did in the other experiments. The network was trained for 200 epochs using the same hyperparameters and protocol described in Sec. 2. The figure shows that the test еттоr of the modified ResNet28 for is approximately constant (blue profile). On the contrary when we increase the width of the whole network the test error decays to the asymptotic test error with an approximate scaling of 1/vu (orange profile). Therefore mitosis occurs when the width of the whole architecture increased, rather than just the width of the final layer.
mitosis, or if instead one has to increase
17


Reviews:
[{"review_id": "ar837Z5-l5q", "reviewer": "Reviewer_YoAs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is mostly empirical in nature, and I think the authors performed a very nice set of carefully designed experiments to explore their novel \"cloning\" effect. I appreciate the variety of architectures and vision datasets considered, and I further appreciate the candor in Section 3.1 on Limitations. I also think that the analysis of reconstructing the wide representation from the smaller subset is a nice way of making the point about features being cloned and the information being present in the smaller subset.\n\nI think two primary weaknesses involve the aforementioned limitations.\n\nThe first weakness is the necessity of using heavy regularization and complicated training procedures. For typical discussions of overparameterized networks -- e.g. ones that discuss the kernel learning regime -- regularization isn't necessary to see the double descent phenomenon highlighted in the Belkin et al. paper cited by the authors. In fact, gradient descent is supposed to automatically pick out the minimal norm solution, and such models overparameterized trained this way do not overfit even if they are not explicitly regularized. Thus, I conclude that the \"cloning\" effect observed by these authors cannot be the main explanation for the way DNNs appear to defy the classical bias-variance trade-off. Further, I wonder whether \"cloning\" is somehow an artifact of the augmentation+regularization used. Perhaps the authors could explore this more and/or come up with ways to disentangle these effects or explore cloning/mitosis in other overparameterized models so that its origin can be better understood?\n\nNext, I think it's problematic that the cloning effect was not observed for ImageNet, though this is definitely a lesser weakness: it could potentially be explained by the fact that the authors were unable to get a wide enough final hidden-layer representation in comparison to the number of classes in ImageNet. As further evidence, the authors highlight that they were not able to train their widened ResNet50 to convergence, which was a prerequisite to see \"cloning\" in the other instances. However, I am worried -- since there isn't any strong theoretical explanation or intuition provided -- that the cloning observed is potentially an artifact of modeling simpler datasets -- e.g. ones where PCA would give reasonable representations -- and thus is not a very general phenomenon in deep neural networks.\n\nSome additional comments: \n- I would enjoy further discussion about what determines $w_c$ for a particular set of architecture, learning algorithm, and dataset. Does varying the depth of the network have any effect on this threshold? (In recent analyses of wide networks, the network's depth to width ratio plays an important role in determining when a network becomes \"wide,\" so this seems like an appropriate scale to compare the width to.)\n- The authors claim that the residual after reconstruction is simply white noise. Is it possible that higher order correlations than the covariance are present and not captured by the subset chunk?\n- In the final paragraph, the authors say that they focus on networks in the feature learning regime, but I thought that to observe their mitosis effect that networks need to be sufficiently wide. Thus, I find their claim a little hard to believe. Do they think that there's another transition for even wider networks that are lazy learners, in which there is no cloning? (By the way, I think that in many ways, the infinite-width limit provides a natural starting point to explain mitosis, since the hidden representations in such networks are all independent at initialization and throughout training.)\n", "summary_of_the_paper": "This paper investigates the overparameterization problem in deep learning through a set of experiments that study the final-layer hidden representations of wide and deep networks. The authors find that -- for wide enough networks and after training long enough -- that they can achieve similar test error from a model made by sampling only a subset of the final-layer hidden representations and then deleting the other connections in that layer. Furthermore, in such situations they can find a linear map from the subset that predicts the full set of final-layer representations, suggesting that the information contained in the deleted features is \"cloned\" and present in remaining subset. Overall, this effect and its dynamics are explored empirically across a number of models, datasets, and learning scenarios. Curiously, the cloning only appears in networks where data augmentation and weight decay is used, and the authors were unable to exhibit this effect on ImageNet.", "summary_of_the_review": "I think this is a nice paper that demonstrates an interesting empirical effect and proposes a promising connection to the behavior of overparameterized deep neural networks. However, in its current form, I think the submission is marginally below the acceptance threshold. In particular, I am bothered by the fact that the effect appears only after training \"using state-of-the-art procedures\" and is absent for complicated datasets (the example in the paper being ImageNet).\n\nOverall, I think this paper makes a nice set of observations and performs careful scientific experiments. With a little more clarity in terms of how general this effect is, I would be happy to accept it to ICLR.\n\n### After Author Responses\nI appreciate the comments of the authors. I still am concerned about how general the mitosis effect is, since it doesn't occur in simpler models. However, with the additional clarity, I have increased my score to be at the acceptance level.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XjvPVQ0p4M", "reviewer": "Reviewer_vhrZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n- Introduce a novel, interesting phenomenon to understand feature learning in properly trained wide neural networks\n- Sufficient empirical evidences to support the claims\n- Overall a well-written paper\n\nWeaknesses: \n- The conditions to achieve representation mitosis are not pinned down precisely. \n- There are some heuristic claims saying that clones appear only in *well-trained* regularized networks and do not appear for the experiments on ImageNet. The more interesting question is, given (some assumptions/conditions for) a dataset, a model architecture and a training algorithm, when and how precisely can the mitosis be reached? This could be useful for choosing models and training methods to achieve the mitosis even before training the model.\n\nMinor comments/questions:\n- How does the depth and connectivity (say, sparse vs. dense networks) of the neurons in the earlier layers affect the mechanism? \n- Will achieving the mitosis generally lead to better generalization? There seems to be evidences of this on page 7.\n- When choosing the chunk of neurons randomly, what guides the choice for the number (or fraction) of neurons? What if we do not choose it randomly but choose it according to some deterministic procedure instead?\n ", "summary_of_the_paper": "This paper provides a novel explanation of benign overfitting in wide neural networks by introducing and studying a mechanism called representation mitosis. The key idea is that if the readout layer of the properly trained network is wide enough, then its neurons could split into groups (clones) that carry identical information, and differ from each other by a statistically independent noise (this mimics what happen in mitosis, thus the name for the mechanism). Moreover, above certain threshold the number of such groups increases linearly with the width, shedding light on when features are learnt. Empirical results are provided to demonstrate the mechanism. ", "summary_of_the_review": "The mechanism of representation mitosis seems to be interesting and is worth exploring. The paper provides satisfactory explanations and ample demonstrations to help understanding the mechanism. However, some of the explanations are at best heuristic in nature and would be much stronger if they could be supported by rigorous theory. I am inclined to go with a weak reject for now.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FowRtwr4fl3", "reviewer": "Reviewer_7hrx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**: this paper identifies an interesting phenomenon in deep neural networks and studies from various aspects.\n\n**Weakness**: the empirical evidences are a bit misaligned with the claims, and it is unclear how this observations could be used. Please see below for details:\n\n1. The mitosis phenomenon is described as formation of \"clones\" of neurons. However, the evidences only show that a (large enough) random subset of neurons could retain the prediction performance or even reconstruct the original outputs. From this observation, it is unclear if there are one-to-one correspondence of copies of neurons, or it's simply the same set of information are encoded in the subspace distributedly, i.e. a (linear) combination of a subset of neurons corresponds to another subset of neurons, but there is no neuron level correspondence. It would be great if the paper could empirically identify the neurons and their \"clones\" to more directly support the claim of representation mitosis. Otherwise, the current observation might not be very different from the phenomenon of parameterization redundancy of large neural networks that are reported in many previous papers that a subset of neurons or even whole layers could be removed without affecting the network performance.\n\n2. It is unclear what is special about the last hidden layer. \n\n3. I find it very interesting that the representation mitosis requires using state-of-the-art training setup and continued training after training error reaches zero. If the paper could dig deeper into those situations and identify what happens in different scenarios and the underlying reason for the discrepancy, then that would be a much more interesting paper. \n\n4. The mitosis phenomenon was not reproduced on ImageNet. The suspected reason is that the network is not big enough in this case. It is understandable that the computation cost might be high to train very large models on ImageNet. However, nowadays there are many pre-trained ImageNet models available to download online, some of which (e.g. some vision transformer models) are potentially very large. Since mitosis analysis can be applied to those pre-trained models without re-training. It would be interesting to see if they are still absent in those larger models.\n\n5. It would be great if some applications of the observations could be demonstrated. Can we use this to improve training? Or improve architecture design? Or guide neural network pruning? \n\n-----\n**After rebuttals**: Thanks the authors for posting the rebuttals, which addresses some of my concerns. I realized some of the experiments (ImageNet) cannot be relatively cheaply done and added to the paper after reading the rebuttal. However, I'm still concerned about how general this phenomenon occurs (in simpler architecture or in more complicated dataset). Furthermore, I still find the presentation (including text description and figure illustrations) of the phenomenon as \"clones\" of features very confusing, which the authors acknowledged are not neuron-to-neuron one-to-one mappings. As a result, I'm keeping my current rating.\n", "summary_of_the_paper": "This paper identify and studies a phenomenon called \"representation mitosis\": as the neural network width increases above a critical threshold, the neurons in the last layer becomes \"redundant\" in the sense that they start to form groups of neurons where they carry identical information, and differ from each other only by a statistically independent noise. ", "summary_of_the_review": "This paper identifies a potential interesting phenomenon in deep neural networks. However, I think the empirical evidences need to be strengthened to support the claim well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Bxw6oO9rFeZ", "reviewer": "Reviewer_KFV6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths:\n\n1. The presented mitosis effect is interesting and is clearly demonstrated in certain settings.\n\n1. Authors also show settings where the effect is clearly absent.\n\n1. Understanding how overparameterization improves generalization remains an open research question and this finding could stimulate fruitful future research.\n\n1. Overall the paper is well-written and figures are of good quality.\n\n## Weaknesses:\n\n1. IMO the main weakness of the paper is clarity,  and I hope this can be improved during rebuttal. I have found the writing to be very concise / missing details / references / mathematical derivations to be able to fully appreciate and evaluate the claims made in the paper. Specifically:\n\n\t1. What exactly is $\\textrm{error}_{\\infty}$? Is it an ensemble of the full network of the largest $\\max W$ considered? Or is it an ensemble of networks of width $W$, where it changes along the $x$-axis of the plot? When using chunks, is an ensemble of chunks, or of full networks of width $W$, or $\\max W$? Without knowing the exact definition, it's very hard to interpret a lot of plots in the paper. Further, specific ensemble sizes should be specified in the appendix.\n    1. For all plots showing the $\\textrm{error} - \\textrm{error}_{\\infty}$, I would appreciate to also see respective plots of simply error, and loss in the appendix, especially if \"error infinity\" depends on width W. In this case it's not clear from the plots that the error is even going down with width, since \"error infinity\" could increase. If \"error infinity\" is a single constant for all points on the plot, this is less important.\n    1. Start of page 2: *\"The decay rate of −1/2 in particular implies that in this regime chunks of $w_c$ neurons can be thought as statistically independent estimators of the same features of the data, differing only by a small, uncorrelated noise\"*. Could you please expand on this statement, perhaps provide some mathematical details in the appendix / references? I don't understand it currently, i.e. why, and what exactly does the $-1/2$ scaling implies.\n    1. End of same paragraph: *\"The accuracy of these wide networks then improves with their width because the network implicitly averages over an increasing number of clones in its representations to make its prediction\"*. Similarly, could you please explain more formally what is implicit averaging and why does it improve accuracy?\n    1. Middle of page 4: *\"This implies that a model obtained by selecting a random chunk of $w_c > w_c^∗$ neurons from a wide final representation behaves similarly to a full network of width $W = w_c$\"*. per my point above, due to the ambiguous nature of \"error infinity\", it is not clear to me if they actually behave similarly, i.e. have comparable test error, or only comparable $\\textrm{error} -\\textrm{error}_{\\infty}$. Again, providing plots of only test error and loss would be much appreciated, especially if \"error infinity\" is not a constant but changes with width $W$ or even chunk width $w_c$.\n    1. End of same paragraph: *\"Furthermore, a decay with rate −1/2 suggests that the final representation of the wide networks can be thought of as a collection of statistically independent estimates of a finite set of data features relevant for classification. Adding additional neurons to the chunk hence reduces their prediction error in the same way an additional measurement reduces the measurement uncertainty, leading to the −1/2 decay.\"* Similarly to a comment above, I would appreciate it if this statement was made mathematically explicit in the appendix or a specific reference provided.\n    1. Page 5, top: *\"We call a chunk of neurons a clone if it fully captures the relevant features of the data, up to some uncorrelated random noise.\"* Please provide a more formal definition.\n    1. Page 5 below: *\"The ID of the widest representations gives a lower bound on the number of coordinates required to\ndescribe the data manifold, and hence on the neurons that a chunk needs in order to have the same\nclassification accuracy as the whole representation. The ID of the last hidden representation is 2 in\npMNIST, 12 in CIFAR10, 14 in CIFAR100, numbers which are much lower that the width at which a\nchunk can be considered a clone.\"* Could you elaborate more on the dependence between ID and $w_c$? I can see why $w_c$ can't be smaller than ID, but curious if there's anything more that could be said, e.g. perhaps could be a more specific argument on how $w_c$ depends on $W$ and ID in some toy setting. It's OK if you don't have an answer, but I am curious if there are ways to ballpark estimate $w_c$ to gain better intuition into it.\n\t1. Top of page 6: *\"In the previous paragraphs we set forth evidence in support of the\nhypothesis that large chunks of the final representation of wide DNNs behave approximately like an\nensemble of independent measures of the full feature space. This allowed us to interpret the decay of\nthe test error of the full networks with the network width observed empirically in Fig. 2\"*. As above, I don't know what an \"ensemble of independent measures on the full feature space\" means, or why it explains the scaling, so would appreciate much more mathematical details here.\n\t1. Discussion, section 4: again, I don't understand what \"statistically independent learning schedule\" is, and it's hard to follow the argument in the first paragraph. Would appreciate mathematical details in the appendix or a specific reference.\n    1. Discussion, bottom: *\"The number of clones grows linearly\"* - why does it grow linearly, and not, for example, as ${W \\choose w_c}$? Per above, this will also depend on what you define to be a clone exactly.\n    1. Equation (1): what exactly are $\\bf{x}$s? Training set activations, or validation set, or test set, or all together? Do the findings of the paper hold for all settings (train/test/train+test) above?\n    \n1. Another issue I find with the paper are the claims that mitosis explains why test error falls with width. For example, on page 9 the authors claim *\"we make the observation that co-adaptation of hidden units in the form of clones occurs without\ndropout, and is crucial for their improving performance with width\"*; in Figure 11 the authors conclude that since the T-shaped network generalizes poorly, therefore it also can't have mitosis, which I find unjustified, and think this Figure should at least show the standard mitosis plot like in Figure 1. Overall I did not find any evidence in the paper that mitosis is responsible for improved generalization with width, if anything, Figure 8 shows that it's not. To make claims about mitosis being responsible for improved generalization with width you would need to run many more experiments, comparing generalization scaling of networks with and without mitosis. But I'm pretty sure if mitosis does not arise without data augmentation and/or weight decay as the paper claims, then it can't be responsible for improved generalization with width, since non-regularized networks still improve their generalization with width even without regularization (see e.g. https://arxiv.org/pdf/1412.6614.pdf).\n\n1. I appreciate that you have identified that mitosis requires data augmentation and/or weight decay. It would be very interesting to ablate this further, and identify which is responsible for it exactly, especially since these are two very different techniques. In general, I wish there was more insight into why exactly the conditions you listed in the paper are necessary for mitosis.\n\n## Minor:\n\n1. Page 1, extra space after Sec. A.2\n1. Figure 1/2 - inconsistent y-axis name w/ and w/o \"log\".\n1. Figure 1 - no y-axis ticks.\n1. Page 4, extra comma after \"significantly\"?\n1. Figure 2: why stop so narrow for FC5, isn't this the cheapest setting, where width could be taken to be much wider than other more expensive models?\n1. Page 6 bottom - \"(orange)\" -> \"(blue)\"?\n1. Page 7\" \"done CIFAR100\" -> \"done on\".\n1. Figure 5 - would prefer matching chunk sizes and colors in (a) and (b), and plotting all panels from initialization, with shared $x$-axis.\n1. Figure 6 (b) - does $x$-axis start at 0? If so, how come thick and thin lines don't start at the same $y$-values?\n1. Page 8: \"much higher than what observed\" - \"we have observed\"?\n1. Page 9: \"found that that\".\n1. Page 9: \"along those line\" - plural \"lines\"?\n1. Figure 9: \"blu\" -> \"blue\".\n", "summary_of_the_paper": "The paper shows empirically that under certain conditions (overtraining, large width in all layers, data augmentation and regularization) neural networks (NNs) tend to learn redundant representations in the last layer. Precisely, a large enough random subset of the penultimate layer activations of size $w_c$ can almost perfectly linearly predict the whole activation of width $W$.\n\nFurther, the training error when pruning the network to these random subsets also goes down (lagging behind the training error of the full network) to 0 monotonically as training progresses.\n\nTest error of these pruned networks appears to go down as $w_c^{-1/2}$, similarly to the test error of regular networks going down as $W^{-1/2}$.\n\nThe authors put forward some preliminary interpretation of the effect and suggest this redundancy effect could be causally responsible for the phenomenon of generalization of wide networks improving with width.\n", "summary_of_the_review": "# Post-rebuttal update\n\nThe authors have clarified most of my questions, and I am raising my score to weak accept. My original conclusion remains similar, but improved clarity makes it a stronger submission. Recap:\n\nWhy accept: \n* The empirical phenomenon of mitosis is interesting and shown to be robustly present in some cases, but absent in others. \n* The paper is well-written and clear (given recent updates and replies to my questions, and a few more clarifications that I expect to see in the final revision).\n\nWhy weak: \n* Exact conditions for mitosis are still not well understood. In the rebuttal the authors have updated the condition set to 100% training set fit + \"optimal\" [what is optimal?] weight decay), which is good, but I still don't think this is precise enough, and most notably the paper does not contain enough systematic experimental evidence to confirm it (e.g. mitosis measurements in wide FC networks, sweeping the weight decay from 0 to X, and observing that the rate of mitosis increases respectively).\n* Exact implications of mitosis are also not well understood / demonstrated (e.g. what is the relationship between mitosis and generalization? Can a network demonstrate improved generalization with increasing width without mitosis? Can a network without improving generalization with width exhibit mitosis? etc).\n\n# Original Review\n\n\nThe paper presents an interesting empirical finding, but in my opinion does not explain very well why mitosis occurs, or what are the implications of it, which is why I am leaning to reject it at this time. However, many of my concerns could be addressed in the rebuttal, so I might change my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper undertakes an empirical investigation of overparameterized neural networks, studying the last hidden representation and identifying "representation mitosis," a cloning effect whereby neurons split into groups that carry the same information. The effect is observed for a variety of architectural configurations/datasets, and a detailed set of experiments are performed to investigate the behavior.

The reviewers had split opinions about this paper, with most reviewers appreciating the novelty and salience of the observations, but with some reviewers expressing skepticism about the generality of the effect. While the experiments are thorough and revealing, the practical importance of representation mitosis remains somewhat unconvincing.

A primary motivating factor for the analysis is the search for an explanation of the unexpectedly good generalization behavior of oveparameterized networks and the origin of "benign overfitting." As highlighted in the reviews, the sensitivity of the mitosis effect to (1) training to zero loss and (2) optimal regularization suggests that it cannot be the sole explanation for benign overfitting, since the latter can and does occur without these conditions. The authors acknowledge this situation, and respond that their focus is on state-of-the-art models used by the community, rather than on toy settings. For this to be a persuasive response, more compelling results in these state-of-the-art situations should be evidenced -- in particular, as several reviewers pointed out, the negative results on ImageNet undermine this point to some extent.

Overall, representation mitosis does seem like an interesting and potentially important phenomenon, but further work is needed to develop persuasive evidence in support of the interpretations and implications. While this is a borderline submission, I believe it falls just short of the mark, and cannot recommend acceptance.
Decision: Reject