Redundant representations help generalization in wide neural networks
Authors: Diego Doimo
Aldo Glielmo
Sebastian Goldt
Alessandro Laio
Venue: NeurIPS 2022 Accept (2022)
Keywords: generalization
wide neural networks

Abstract: Deep neural networks (DNNs) defy the classical bias-variance trade-off: adding parameters to a DNN that interpolates its training data will typically improve its generalization performance. Explaining the mechanism behind this ``benign overfitting'' in deep networks remains an outstanding challenge. Here, we study the last hidden layer representations of various state-of-the-art convolutional neural networks and find that  if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information and differ from each other only by statistically independent noise. The number of such groups increases linearly with the width of the layer, but only if the width is above a critical value. We show that redundant neurons appear only when the training is regularized and the training error is zero.

Redundant representations help generalization in wide neural networks
Diego Doimo*
Aldo Glielmo
International School for Advanced Studies
International School for Advanced Studies
Bank of Italy!
Sebastian Goldt
Alessandro Laio
International School for Advanced Studies
International School for Advanced Studies
Abstract
Deep neural networks (DNNs) defy the classical bias-variance trade-off: adding parameters a DNN that interpolates its training data will typically improve its generalization performance. Explaining the mechanism behind this "benign overfitting" in deep networks remains an outstanding challenge. Here, we study the last hidden layer representations of various statc-of-the-ee convolutional neural networks and find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information and differ from each other only by statistically independent noise. The number of such groups increases linearly with the width of the layer, but only if the width is above critical value. We show that redundant neurons appear only when the training is regularized and the training error is zero.
Introduction
Deep neural networks (DNN) have enough parameters to achieve zero training error, with random labels [1, 2]. In defiance the classical bias-variance trade-off, the performance of these interpolating classifiers improves the number of parameters increases well beyond the number of training samples [3-6]. Despite recent progress in describing the implicit bias of stochastic gradient descent towards "good" minima [7-12], and the detailed analysis of solvable models of learning [13-21], the mechanisms underlying this "benign overfitting" [22] in deep neural networks remain unclear, especially since their loss landscape contains "bad" local minima and SGD can reach them [23].
In this paper, describe phenomenon in wide DNNs that could be possible mechanism for benign overfitting when the networks are trained with regularization. We illustrate this mechanism in Fig. 1 for a family of increasingly wide DenseNet40s [24] trained CIFAR 10 [25] following common practice, in particular using weight decay (see Sec. 2.1). For simplicity, we refer to the width W of the last hidden representation the width of the network. The blue line in Fig. 1-b shows that the average classification error (error) approaches the performance of a large ensemble of networks (error [21] as increase the network width W. In agreement with [26], find that the performance of these DenseNets improves continuously with width. For widths greater than 350, the networks are wide enough to reach zero training error (see Appendix, Sec. Fig S2-c) and,
ddoimo @ sissa.it
'The views and opinions expressed in this paper those of the authors and do necessarily reflect the official policy or position Bank Italy.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

representation
Input
chunk network
full
Dense
error error-
0.5
network
Home Dog
0.13
test error
chunk
0.03
0.06
Input
chunked
Got
0.01
network
176
352
704
1400
full net/chunk size
clone existing features
add new feature
Relevant Data Features
Final representations
fd
W
network width W
Figure 1: The redundancy of representations in wide neural networks. a: We analyze the final representations of deep neural networks (DNN), namely the activities of the last hidden layer of neurons (light blue) We focus on the performance and the statistical properties of randomly chosen subsets of neurons which we call "chunks". In the chunked network shown here, We 5 out of 9 neurons are kept and used to predict the output. As we increase the size of the chunk w. that we keep in state-of-the-ar DNN, here DenseNet40, the test error of the chunk (orange line) becomes similar to the test error of a full network of width = We (blue line). In this regime, which is reached when W. is larger than threshold w (shaded area) the eπror approaches its asymptotic
value error as a power-law w. (dashed line). error is the error of an ensemble average of 20 networks of the widest size. Illustration of three final representations for networks of increasing width. In small networks, an additional neuron fits new features of the data (red neuron). As the network width goes beyond critical width W", additional neurons instead copy features already learned from data, instead of over-fitting to features that are not relevant to the task. This mechanism is suggested by the w. 1/2 decay of the chunk error, and by the statistical analysis, we present in this paper.
interestingly, their test error decays approximately as W- Our goal to understand how the error of the network can keep decaying beyond the interpolation threshold, and why it decays as W 1/2
We make our key observation by performing the following experiment: we randomly select a number W. of neurons from the last hidden layer of the widest DenseNet40 and remove all the other neurons from that layer as well as their connections (Fig. 1-a). We then evaluate the performance of this chunk of we neurons, without retraining the network. The orange profile of Fig. 1-b shows the test error of chunks of varying sizes. There are two regimes: for small chunks, the error decays faster
than we 1/2 while beyond a critical chunk size w. (shaded area), the error of chunk of w. neurons is roughly the same as the one of a full network with we neurons. Furthermore, the error the chunks decays with the same power-law w. 1/2 beyond this critical chunk size.
The decay rate of 1/22 suggests that in this regime chunks of w. neurons can be thought of statist- ically independent estimators of the same features of the data, differing only by small, uncorrelated noise. In other words, beyond the critical width w. the final hidden representation of an input in a trained, wide DNN becomes highly redundant. This motivates a possible mechanism for benign overfitting, schematically portrayed in Fig. I-c: as the network becomes wider, additional neurons are first used to learn new features of the data. Beyond the critical width w additional neurons in the final layer don't fit new features in the data, and hence over-fit; instead, they make a copy, or clone, of feature that is already part of the final representation. The last layer thus splits into more

---

and more clones as the network grows wider as we illustrate at the bottom of Fig. 1. The accuracy of these wide networks then improves with their width because the network implicitly averages over an increasing number of clones in its representations to make its prediction.
This paper provides a quantitative analysis of this phenomenon various data sets and architectures. Our main findings can be summarized as follows:
1. A chunk of we random neurons of the last hidden representation a wide neural network predicts the output with error that decays as wie -1/2 if the layer is wide enough and w, is large enough. In this regime, we call the chunk a "clone":
2. Clones fit the training set with zero error and be linearly mapped to another, to the full representation, with an emror that can be described as uncorrelated random noise.
3. Clones appear the model is trained with weight decay and the training set is fitted with zero enror. If training is stopped too early if the training is performed without regularization,
1. and 2. do not take place, even if the last representation is very wide.
2 Methods
2.1 Neural network architectures
We report experimental results obtained with several architectures (fully connected networks, Wide- ResNet-28, DenseNet40, ResNet50) and data sets (CIFARI0/100) [25], ImageNet [27]). We train all the networks using SGD with momentum and, importantly, weight decay. The amount of weight decay is found with small grid search, while the other relevant hyperparameters are set following standard practice. We give detailed information on our training setups in Sec. A of the Appendix. All our experiments are run on Volta V100 GPUs. In the following paragraphs, we describe how we vary the width W of the models.
Fully-connected networks MNIST. We train a fully-connected network classify the parity of the MNIST digits [28] (pMNIST) following the protocol of Geiger et al. [21]. MNIST digits are projected on the first ten principal components, which are then used as inputs of five-layer fully-connected network (FC5). The four hidden representations have the same width W and the output a real number whose sign is the predictor of the parity of the input digit.
Wide-ResNet-288 and Netso on CIFAR10/100. We train CIFAR10 10 and CIFAR100 on family of Widee-ResNet-28 [26] (WR28). The number W of the last hidden neurons in a WR28-n 64 n, obtained after average pooling the last 64 n channels of the network. In our experiments, we also analyze two narrow versions of the standard WR28-1 which are not typically used the literature. We name them WR28-0.25 and WR28-0.5 since they have 1/4 and 1/2 of the number of channels of WR28-1. Our implementation of DenseNet40 follows the DenseNet40-BH variant [24]. We vary the number of input channels in {16, 32, 64, 128, 256), which is twice the growth rates k of the networks [24]. The number W of the last hidden features of this architecture is 5.5 e.
ResNet50 on ImageNet. We modify the ResNet50 architecture [29] by multiplying by a constant factor e E (0.25, 0.5, 1, 1,2, 4) the number of channels of all the layers after the input stem. When = 2 our networks differ from the standard Wide-ResN Nes550-2 [26] since we double the channels of all the layers and not just those of the bottleneck of the ResNet blocks. As a consequence in our implementation, the number of features after the last pooling layer is W 2048 while in [26] W is fixed to 2048.
2.2 Analytical methods
Reconstructing the wide representation from a smaller chunk. To determine how well a subset of w neurons can reconstruct the full representation of size W we search for the W w linear map A, able to minimize the squared difference (x(W) - xW))² between the W activations of the full layer representation, x(W), and the activations predicted from the chunk of size w. (W):
x (W)
Ax
(1)
3

---

This least-squares problem is solved with ridge regression [30] with regularization set to 10 10- and we use the R² coefficient of the fit measure the predictive power of given chunk size. The R value is computed as an average of the single-activations R values corresponding to the W output coordinates of the fit, weighted by the variance of each coordinate. We further compute the W x W covariance matrix Ciş of the residuals of this fit, and from Cip we obtain the correlation matrix as:
Cij
Pij CS + 10-8'
(2)
with small regularization in the denominator avoid instabilities when the standard deviation of the residuals falls below machine precision. To quantify how much the errors of the fit are correlated, we average the absolute values of the non-diagonal entries of the correlation matrix Pi-- For short, refer to this quantity as a 'mean correlation'.
Reproducibility.
We provide code to reproduce our experiments and our analysis online at https: wwtp:///waeegeooooeeeeeeaaaeeeaa
3 Results
The test error of chunks of we neurons of the final representation asymptotically scales as w. w-- The mechanism we propose is inspired by the following exper- iment: we compute the test accuracy of models obtained by selecting a random subset of We neurons from the final hidden representation of a wide neural network. We select we neurons at random and we compute the test accuracy of network in which we set to zero the activation of all the other w We neurons of the final layer. Importantly, we do not fine-tune the weights after selecting the w. neurons: all the remaining parameters of the previous layers are left unchanged and only the the activations of the "killed" neurons of the last hidden representation are not used to compute the logits. We take 500 random samples of neur- ons for each chunk width we- We consider three different data sets: PMNIST trained on fully connected network, CIFARIO and CIFAR 100 trained on convolutional net- works. The width W of the network is 512 for PMNIST and CIFAR10, and W 1024 for CIFAR 100 (see Sec. 2..). all these cases, W is large enough to firmly in the regime where the accuracy of the networks scales (approximately) as 1/2 (see Fig. 2).
100
AMINIET-FOS
CIFAR10 W/225
10-0.
CIFAR 100/WR22
error error
CIFAR10. DDl.Deiooi
10-2
10-15
10-2
10-25
16
32
64
128
256
512 1024
network size
Scaling of the test error with width for various DNN. The average test error of neural networks with vari- ous architectures approaches the test er- ror of an ensemble of such networks as the network width increases. The network size shown here is the width of the final representation. For large width, we find power-law behavior error error W-1/2 across data sets and architectures. Full experimental details in Sec. 2.1
Figure 2:
In Fig. we plot the test error of the "chunked models" a function of w. (orange lines). The behavior is similar in
all three networks: the test error decays as w. -1/ for chunks that are larger than a critical value w which depends on the data set and architecture used. This decay follows the same law observed for full networks of the same width (Fig. 2). This implies that model obtained by selecting a random chunk of w. > W. neurons from a wide final representation behaves similarly to a full network of width W We. Furthermore, decay with rate -1/2 suggests that the final representation of the wide networks can be thought of as a collection of statistically independent estimates of finite set of data features relevant for classification. Adding neurons to the chunk hence reduces their prediction error in the same way an additional measurement reduces the measurement uncertainty, leading to the -1/2 decay.
At we smaller than w instead, the test error of the chunked models decays faster than w. cases we considered, including the DenseNet architecture trained on CIFARI0 10 shown in Fig. 1-b. In this regime, adding neurons to the final representation improves the quality of the model significantly quicker than it would in independently trained models of the same width (see Fig. I-c for a pictorial representation of this process). We call chunks of neurons of size w. 2 w clones. In the following, we characterize more precisely the properties of the clones.
1/2 in all the

---

PMNIST
b
CIFAR10
C
CIFAR100
10
chunks
10-15
chunks
0.36
10:93
0.65
error errorr
10-15
3.05
0.13
10-17
0.35
1est error
10-15
0.06
10 11
0.23
10-2
2.08
10
0.04
10-15
10-25
10
0.00
10-11
0.1%
16
32
64
128
256
512
8
16
39
64
198 256
519
16
32 64
198 256
512 1204
chunk size
chunk
chunk
Figure 3: Scaling of the test error of chunks of neurons extracted from the final hidden representation of wide NNs. We plot how the test error of chunked networks approaches error the eπror of an ensemble of 20 networks of the widest size (c.g. W = 1024 for CIFARI00) as the chunk size we increases. Chunks are formed by selecting a number of we neurons at random from the final hidden representation of the widest networks: a FC5 on PMNIST (width W = 512). and for CIFARIO (W 512) and CIFARI00 (W~ 1024). The shaded regions indicate regions where the error of the chunks with we neurons decays as w. -1/2
Clones interpolate the training data. A trained deep network often represents the salient features of the data set well enough achieve (close to) zero classification error the training data. In the top panels of Fig. 4, we show that wide networks can interpolate their training set also using just a subset of w, w² random neurons: the dark orange profiles show that when the size of a chunk is greater than w ~ 50 for PMNIST, 100 for CIFAR1O and 200 for CIFAR 100, the predictive accuracy on the training set remains almost 100%. The minimal size of a clone w can be identified with the minimal number of neurons required to interpolate the training set. Beyond w², the neurons of the final representation become redundant since the training error remains (close to) zero even after removing neurons from it. The number of distinct clones in network of width W is n W/w". If distinct clones provide independent measures of the same salient features of the data, the test error decays approximately as - equivalently W- In the following, we will indeed see that distinct clones differ from each other by uncorrelated random noise.
Clones reconstruct the full representation almost perfectly. From a geometrical perspective, the important features of the final representation correspond to directions in which the data landscape shows large variations [31]. clone a chunk that is wide enough to encode almost exactly these directions (since its training error is almost zero), but using much fewer neurons than the full final representation. We analyze this aspect by performing a linear reconstruction of the W activations of the last hidden representation of the widest network starting from random subset of we activations using ridge regression with a small regularization penalty according to Eq. (1). The blue profiles in Fig. f-(d.e.), show the R² coefficient of fit as function of the chunk size We for PMNIST (left), CIFAR1O 10 (center), CIFAR 100 (right). When We is very small, say below 6 for PMNIST, 20 for CIFAR10 and 60 for CIFAR the R coefficient grows almost linearly with W. In this regime, adding a randomly chosen activation from the full representation to the chunk increases substantially R2 When w. becomes larger I22 reaches almost one. This transition happens when w. is still much smaller than W and corresponds approximately to the regime in which the test error starts scaling with the inverse square root of W. (see Fig. 3). The almost perfect reconstruction of the original data landscape with few neurons is a consequence of the low intrinsic dimension (ID) of the representation [32]. The ID of the widest representations gives lower bound on the number of coordinates required to describe the data manifold, and hence on the neurons that a chunk needs in order to have the same classification accuracy as the whole representation. The ID of the last hidden representation is 2 in PMNIST, 12 in CIFAR 10, 14 in CIFAR 100, numbers which are much lower than w., the width at which a chunk can be considered clone.
Clones differ from each other by uncorrelated random noise. When We w the small residual difference between the chunked representation and the full representation can be approximately
The linear trend can not be clearly seen in Fig. 4 as plot the axix with logarithmic scale.
5

---

PMNIST
b
CIFAR10
CIFAR100
described statistically independent random noise. The green profiles of the bottom panels of Fig. 4 show the mean correlation of the residuals of the linear fit (see Sec. 2.2). Below w the residuals not only large but also significantly correlated, since they related to relevant features of the data that are not covered by the neurons of the chunk. As the chunk width increases above w the correlation between residuals drops basically zero. Therefore, in networks wider than w any two chunks of equal size W. > w can be effectively considered as equivalent copies, or clones, of the same representation (that of the full layer), differing only by a small and non-correlated noise, 0 DIRACC
100%
100%
100%
traiming error
chunce
10%
12%
10%
196
195
115
dhrrks
Shuras
0.1%
0.1%
0.1%
full widltrchunk size
full width/chunk size
full widllichunk size
1.00
1.00
0.06
0.75
0.00
mean COT:
R²
0.04
0.50
correlation residuals
0.50
0.04
0.1
0.02
0.25
0.25
0.25
0.00
0.00
0.00
2
32
125
512
2
32
128
512
4
258
1024
chunk Size
chunk
chunk size
Figure 4: The three signatures representation redundancy. (i) The training errors of the full networks (blue) and of the chunks taken from the widest network (orange) approach zero beyond critical width/chunk size, resp. (panels a-c). (ii) The final representation the widest network can be reconstructed from chunk using linear regression (1) with explained variance R2 close to (blue lines in panels d-f). (iii) The residuals of the linear map can be modeled as independent noise: we show this by plotting the mean correlation of these residuals (green line, panels d-f), averaged over 100 reconstructions starting from different chunks. A low correlation at high R² indicates that the chunk contains the information of the full representation with some statistically independent noise. Experimental setup: FC5 on PMNIST, Wide ResNet-28 on CIFARIO/100 Full details in Methods section 2.1
consistently with the scaling law of the eтror shown in Fig. 3.
The dynamics training. the previous paragraphs, we set forth evidence support of the hypothesis that large chunks of the final representation of wide DNNs behave approximately like an ensemble independent measures of the full feature space. This allowed us to interpret the decay of the test error of the full networks with the network width observed empirically in Fig. 2. The three conditions that chunked model satisfies in the regime in which its test error decays as w. are represented in Fig. 4: (i) the training error of the chunked model is close to zero; (ii) the chunked model can be used to reconstruct the full final representation with an R² of this reconstruction can be modeled independent random noise. These three conditions observed at the end of the training. We now analyze the training dynamics. We will see that for the clones to arise, models not only need to be wide enough but also, crucially, they need to be trained to maximize their performance.
and (iii) the residuals all
Clones are formed in two stages, which occur at different times during training. The first phase begins as soon as training starts: the network gradually adjusts the chunk representations in order to produce independent copies of the data manifold. This can be clearly observed in Fig. 5-a, which depicts the mean correlation between the residuals of the linear fit from the chunked to the full final representations of the network, the same quantity that analyze in Fig. 4-(d-e-f, green profiles),

---

chunk size
raining orror
0.10
test
mean corr.
0.15
64
0.00
chunk 620
0.06
end of phaso2
phose1
128
training
0.06
32
emrr
0.04
256
0.04
128
0.05
0.02
512
258
lesi eroor
0.00
20
150170
100
170
150
170
180
150
200
epoch
epoch
epoch
Figure 5: The onset of clones during training, a: As in Fig. 4, we show the mean correlation of the residuals of the linear reconstruction of the final representation from chunks, but this time as a function of training epochs. A small correlation indicates that the reconstruction eπror in going from chunks to final representation can be modeled as independent noise. Data obtained from the same WR28-8 trained on CIFARIO 10 as in Fig. 4. b: Training error during training for chunks of different sizes. After the network has reached zero training error at 160 epochs, continuing to train improves the training accuracy of the chunks. c: Test and training error during training for the full network. Between epochs 160 and 180, the clones of the full network progressively achieve zero training error. In the same epochs, one observes small improvement in the test error.
but now as a function of the training epoch. Both Figs. 4 and 5 analyze the WR28-8 on CIFARIO. As training proceeds, the correlations between residuals diminish gradually until epoch 160 and become particularly low for chunks greater than 64. After epoch 160 further training does not bring any sizeable reduction in their correlation. At epoch 160 the full network also achieves zero error on the training set, as shown in Fig. 5-b (brown) and Fig. 5-c (blue). This event marks the end of the first phase and the beginning of the second phase where the training error of the clones keeps decreasing while the full representation (blue) has already reached zero training error. For example, chunks of size 64 at epoch 150 have training errors comparable to the test error (dashed line of the middle panel). In the subsequent 20 epochs the training error of clones of size 128 and 256 reaches exactly zero, and the training error of chunks of size 64 reaches plateau.
Importantly, both phases improve the generalization properties of the network. This can be seen in Fig. 5-c, which reports the training and test eттоr of the network, with the two phases highlighted. The figure shows that both phases lead reduction in the test error, although the first phase leads by far to the greatest reduction, consistent with the fact that the greatest improvements in accuracy typically arise during the first epochs of training. The formation of clones can be considered finished around epoch 180 when all the clones have reached almost zero error on the training set. After epoch 1 180 we also observe that the test error stops improving. the Appendix (Sec. B) we report the same analysis done on CIFAR 100 (see Fig. S1) and CIFARIO trained on a DenseNet40 (see Fig. S2-(d-e-f)).
Clones appear only in regularized networks. So far in this work, have shown only examples of regularized networks and data sets in which representations are redundant. However, if the network not regularized, some of the signatures described above don't appear even if the width of the final representation is much larger than W (the minimum interpolating width). Figure 6 shows the case of the Wide-RResNet2-S analyzed in Fig. 5 trained on exactly the same data set (CIFARIO) 11) but without weight decay. As shown in Fig.6-a in the network trained without regularization (blue line) the error
does not scale we 1/2 This, as we have seen, indicates that the last hidden representation cannot be split in clones equivalent to the full layer. Indeed, the mean correlation of the residuals of the linear map of the chunks to the full representation remains approximately constant during training (Fig. 6-b), and is always much higher than what we observed for the same architecture and data set when training is performed with weight decay. We performed similar analysis on the NenseNet40 (see Fig. S3), observing an analogous trend.
Clones appear only if network interpolates the training set: the case of ImageNet. We saw that a chunk of neurons can be considered clone if it fully captures the relevant features of the data, achieving almost zero training error (see Fig. 4). This condition is not satisfied for most of the
7

---

b
we weight decay weightt decay
error error
0.4
A--0.2
0.20
16
mean CorT.
0.15
0.2
0:10
0.05
0.00
64 128 256
512
50
100
150
200
chunk size
epoch
Figure 6: A network trained without weight decay on CIFAR1O. a: the test error of chunks of Wide-ResNet28-8 trained without weight decay (blue) and with weight decay (orange, taken from Figure 3-b). b: Mean correlation between residuals of the linear reconstruction of the full representation from chunks of different sizes for two networks: one trained without weight decay (thick lines), and one using weight decay (thin lines, same data as in Fig. 5-a).
networks trained on ImageNet [16], therefore we do not expect to see redundant representations in this important case. We verified this hypothesis by training family of ResNet50s where multiply all the channels the layers after the imput stem by a constant factor c E {0.25, 0.5, 1, 2, 4]. this manner the widest final representation we consider consists of 8192 neurons, which is four times wider than both the standard ResNet50 [29] and its wider version [26] (see Sec. 2.1). We trained all the networks following the standard protocols and achieved test errors comparable to slightly lower than those reported the literature (see Appendix, Sec. A). We find that even in the case the
largest ResNet50, the top-1 error on the training is 8% (see Fig. 7-a) and the network does achieve interpolation, as discussed also in [16].
In this setting, none of the elements associated with the development of independent clones can be observed. The scaling of the test error of the chunks is steeper than w. (see Fig. 7-b) suggesting that chunks remain significantly correlated to each other. Figure 7-c shows that the mean correlation of the residuals does not decrease during training, as it happens for the networks we trained on CIFAR 10 and CIFAR 100. We conclude that in ResNet50, a representation with 8192 neurons is too narrow to encode all the relevant features redundantly on ImageNet, and a chunk as large as 4096 activations is not able to reconstruct all the relevant variations of the data it does in the cases analyzed in Sec. 3.
training error
error -rrrr
0.3
wisest-ret
0.49
test error
mean CDTT.
0.025
- 1024
- 512
10%
0.020
2018
4006
0.035
0.005
0.1%
0.01
0.2 0.000
512 1024 2048 4096 8192
512 1024 2048 4096 8192 full width/ / chunk size
20
50
100
120
network width
epoch
Figure 7: ResNet50 trained on ImageNet a: ImageNet training error as a function of the ResNet50 width. b: Decay of the test error as a function of the network width (blue) and for chunks of the widest ResNet50 (orange) to the error of an ensemble of ResNet50-4. The ensemble consists of four networks. e: Mean correlation (see Sec. 2.2) of the residuals of the linear map of chunk of the last hidden representation to the full representation. The network analyzed is ResNet50-4.
8

---

4 Discussion
This work is an attempt to explain the paradoxical observation that over-parameteriiation boosts the performance of DNNs. This "paradox" is not a peculiarity of DNNs: if one trains prediction model with n parameters using the same training set, but starting from independent initial weights and receiving samples in an independent way, one can obtain, say, models which, in suitable conditions, provide predictions of the same quantity with independent noise due to initialization, SGD schedule, etc. If one estimates the target quantity by ensemble average, the statistical error will (ideally) scale with m 1/2, and therefore with N 1/2, where N = n m.m. is the total number of parameters of the combined model. This will happen even if N is much larger than the number of data.
What is less trivial is that DNN can accomplish this scaling within single model, in which all the parameters are optimized collectively via the minimization of a single loss function. Our work describes a possible mechanism at the basis of this phenomenon in the special case of neural networks in which the last layer is very wide and the model is regularized. We observe that if the layer is wide enough, random subsets of its neurons can be viewed as approximately independent representations of the same data manifold (or clones). This implies a scaling of the error with the width of the layer as W 1/2,, which is qualitatively consistent with our observations.
The impact of network architecture. The capability of network to produce statistically inde- pendent clones is genuine effect of the over-parametrization of the whole network as we find that redundancies appear even if the last layer width is kept constant and the width of all intermediate layers is increased (see Appendix, Sec. B, Fig. S4-a, ). At the same time, we also verified that if the network is too narrow to interpolate the training set, increasing the width of only the final representation is not sufficient to make the last layer redundant. We give an example of this effect in Fig. S4-b, where we show that the test error of a WR28-1 on CIFAR 10 does not decrease if only the width of the final representation is increased, while the rest of the architecture is kept a constant width.
The impact of training. The mechanism we described is robust to different training objectives since we trained the convolutional networks with cross-entropy loss and the fully connected networks with hinge loss. However, even for wide enough architectures, clones appear only if the training is continued until the training error reaches zero. In our examples, by stopping the training too early, for example when the training error similar the test error, the chunks of the last representation would not become entirely independent from one another, and therefore they could not be considered clones.
Neural scaling laws. Capturing the asymptotic performance of neural networks via scaling laws is
an active research area. Hestness el al. (33) gave an experimental analysis of scaling laws w.r.t. the training data set size in variety of domains. Rosenfeld el al. and Kaplan el al. [34, 35] experimentally explored the scaling of the generalization error of deep networks with the number of parameters/data
points across architectures and application domains for supervised learning, while Henighan et al. [36] identified empirical scaling laws in generative models. Bahri et al. [37] showed the existence of four scaling regimes and described them theoretically in the NTK or lazy regime [38-40], where the network weights stay close to their initial values throughout training. None of these works propose a mechanism that would explain these scalings with properties of the representation. Geiger et al. found that the generalization error can be related to the fluctuations of the output induced by initialization and showed that scales as W- in networks trained without weight decay both in the NTK [21] and in the mean field [41] regimes. We instead consider the feature learning regime and train our networks with weight decay which is unavoidable obtain models with state-of-the-ar performance. This might explain the difference in the scaling law that we observe empirically. Previous theoretical work did not study the impact of weight decay on scaling laws, so we hope that our results can spark further studies on the role of this essential regularizer.
Relation to theoretical results in the mean-field regime. Our empirical results also agree with recent theoretical results that were obtained for two-layer neural networks [42-47]. These works characterize the optimal solutions of two-layer networks trained on synthetic data sets with some controlled features. In the limit of infinite training data, these optimal solutions correspond to

---

networks where neurons in the hidden layer duplicate the key features of the data. These "denoising solutions" or "distributional fixed points" were found for networks with wide hidden layers [42-45] and wide input dimension [46, 47]. Another point of connection with the theoretical literature is the concept of dropout stability. A network is said to be c-dropout stable its training loss changes by less than when half the neurons are removed at random from each of its layers [48]. Dropout stability has been rigorously linked to several phenomena in neural networks, such as the connectedness of the minima of their training landscape [49, 50].
Bias-variance trade-off and implicit ensembling The success of various deep learning archi- tectures and techniques has been linked to some form of ensembling. The successful dropout regularisation technique [51, 52] samples from an exponential number of "thinned" networks during training to prevent co-adaptation of hidden units. While this can be seen as form of (implicit) ensembling, here we observe that co-adaptation of hidden units in the form of clones occurs without dropout, and is crucial for their improving performance with width. Recent theoretical work on random features showed that ensembling and peer-parameterization are two sides of the same coin and that both mitigate the increase in the variance of the network that classically leads to worse performance with paramcterization due to the bias-variane vaarianee trade-off [18-20]. The plots of bias and variance in Fig. S5 for the architectures trained on the CIFARI0 and CIFAR 100 data sets show that the clone size in these cases is slightly above the peak of the variance and almost coincides with the interpolation width of the full networks of the same size.
Impact for applications. The framework introduced in this work allows verifying if a neural network sufficiently expressive to encode multiple statistically independent representations of the same ground truth, which, we believe, is fair proxy of model quality and robustness. In particular, we find that reaching interpolation the training set is not necessarily detrimental for generalization, and is instead a necessary condition for developing redundancies which, in turn, reduces the test error.
References
[1] C. Zhang, S. Bengio, M. Hardr, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
[2] D. Arpit. S. Jastrzebski, M.S. Kanwal, T. Maharaj. A. Fischer, A. Courville, and Y. Bengio. Closer Look at Memorization in Deep Networks. In Proceedings of the 34th International Conference Machine Learning. 2017.
[3] S. Geman, E. Bienenstock. and R. Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58, 1992.
[4] B. Neyshabur, R. Tomioka, and N. Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. In ICLR, 2015.
[5] S Spigler, M Geiger, S d' Ascoli, L Sagun, G Biroli, and M Wyart. A jamming transition from under-to over-parametrization affects generalization in deep learning. Journal of Physics A: Mathematical and Theoretical, 52(47):474001, 2019.
[6] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference Learning Representations, 2020.
[7] S. Gunasekar, J. Lee, D. Soudry, and N. Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning, pages 1832-1841. PMLR, 2018.
[8] S. Gunasekar, J. D. Lee, D. Soudry, and N. Srebro. Implicit bias of gradient descent on linear convolutional networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- -isanBianchhi and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[9] D. Soudry, F. Hoffer, and N. Srebro. The implicit bias of gradient descent on separable data. In International Conference on Learning Representations, 2018.
10

---

[10] Z. Ji and M. Telgarsky. The implicit bias of gradient descent on nonseparable data. In Conference on Learning Theory, pages 1772-1798. PMLR, 2019.
[11] S. Arora, N. Cohen, W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[12] L.. Chizat and Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory, pages 1305-1338. PMLR, 2020.
[13] M.S. Advani, A.M. Saxe, and H. Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428 446, 2020.
[14] B. Neal, S. Mittal, A. Baratin, V. Tantia, M. Scicluna, S. Lacoste-Julien, and I. Mitliagkas. A modern take on the bias-variance tradeoff in neural networks. arXiv preprint arXiv:1810.0859/ 2018.
[15] S. Mei and A. Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 2019.
[16] M. Belkin, D. Hsu, S. Ma, and S. Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 1 116632):1884--1584 2019.
[17] T. Hastie, A. Montanari, S. Rosset, and R.J. Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949-988 2022.
[18] S. d' Ascoli, M. Refinetti, Biroli, and F. Krzakala. Double trouble double descent Bias and variance(s) in the lazy regime. In ICML,, 2020.
[19] B. Adlam and J. Pennington. Understanding double descent requires fine tineigrained variance decomposition. In H. Larochelle, M. Ranzato, R. Hadsell, M. F, Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1 11022-11032. Curran Associates, Inc., 2020.
[20] L. Lin and E. Dobriban. What causes the test error? going beyond bias-variance via anova. J. Mach. Learn, Res., 22:155-1, 2021.
[21] M. Geiger, A. Jacot, S. Spigler, F. Gabriel, I.. Sagun, S. d' Ascoli, G. Biroli, C. Hongler, and
M. Wyart. Scaling description of generalization with number of parameters in deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2020(2):023401, 2020.
[22] P.L. Bartlett, P.M. Long, G. Lugosi, and A. Tsigler. Benign overfitting in linear regression. Proceedings of the National Acudemy of Sciences, 11744))30063-33000, 2020.
[23] S. Liu, D. Papailiopoulos, and D. Achlioptas. Bad global minima exist and sgd can reach them. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 8543-8552. Curran Associates, Inc., 2020.
[24] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
[25] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. ht- -eatammninn-catuus-2200000
hts://wwwwcs.toronto.ccuu
2009.
[26] S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint 1651::1605.07146. 2016.
[27] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
[28] Y. LeCun and C. Cortes. The MNIST database of handwritten digits, 1998.

---

[29] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[30] T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.
[31] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1888, 2013.
[32] A. Ansuini, A. Laio, J. H. Macke, and D. Zoccolan. Intrinsic dimension of data representations in deep neural networks. In Advances in Neural Information Processing Systems, pages 6109- 6119, 2019.
[33] J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. Patwary, M. Ali, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv:1712.004400 2017.
[34] J.S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit. A constructive prediction of the generalization error across scales. In International Conference on Learning Representations, 2020.
[35] J. Kaplan, S. McCandlish, T. Henighan, T.B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001. 08361, 2020.
[36] T. Henighan, Kaplan, M. Katz, M. Chen, C. Hesse, Jackson, H. Jun, T.B. Brown, P. Dhari- wal, S. Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14700,
[37] Y. Bahri, E. Dyer. J. Kaplan, J. Lee, and U. Sharma. Explaining neural scaling laws, arXiv preprint arXiv:2/02.06701, 2021.
[38] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances Neural Information Processing Systems 32, pages 8571-8580, 2018.
[39] S. Du, J. Lee, Y. Tian, A. Singh, and B. Poczos. Gradient descent learns one-hidden-layer CNN: Don't be afraid of spurious local minima. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 1339-1344 2018.
[40] L. Chizat, E. Oyallon, and F. Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems, pages 2937-2947 2019.
[41] M. Geiger, S. Spigler, A. Jacot, and M. Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment 2020(1 11:1)))331 13301, 2020.
[42] S. Mei, A. Montanari, and P. Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 116(33))77665--7777 2018
[43] G.M. Rotskoff and E. Vanden-Eijndent Parameters as interacting particles: long time conver- gence and asymptotic error scaling of neural networks. In Advances in Neural Information Processing Systems 31, pages 7146-7155, 2018
[44] L. Chizat and F. Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. In Advances in Neural Information Processing Systems 31, pages 3040-300000 3050, 2018
[45] J. Sirignano and K. Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 2019.
[46] S. Goldt, M.S. Advani, A.M. Saxe, F. Krzakala, and I.. Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. In Advances in Neural Information Processing Systems 32, 2019.
12

---

[47] M. Refinetti, S. Goldt, F. Krzakala, and L. Zdeborova. Classifying high- dimensional gaussian mixtures: Where kemnel methods fail and neural networks succeed. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 8936-8947, PMLR, 18 Jul 2021.
[48] R. Kuditipudi, X. Wang, H. Lee, Y. Zhang, Z. Li, W. Hu, R. Ge, and S. Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. In H. Wallach, H. Larochelle, A. Beygelzimer, d' d'Alché-Buu, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[49] A. Shevchenko and M. Mondelli. Landscape connectivity and dropout stability of sgd solutions for over-parameterized neural networks. In International Conference on Machine Learning, pages 8773-8784. PMLR, 2020.
[50] Q. Nguyen, Brechet, and M. Mondelli. On connectivity of solutions in deep learning: The role of aneetearamettizzationn and feature quality. In Advances in Neural Information Processing Systems, volume 34. Curran Associates, Inc., 2021.
[51] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. 1207.0580 2012.
[52] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()):1929-1958, 2014.
[53] Micikevicius, Narang. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston. O. Kuchaiev, G. Venkatesh, and H. Wu, Mixed precision training, In International Conference on Learning Representations, 2018.
[54] I. Bello, W. Fedus, X. Du, E. D. Cubuk, A. Srinivas, T.-Y. Lin, J. Shlens, and B. Zoph. Revisiting resmets: Improved training and scaling strategies. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 22614-22621
Curran Associates, Inc., 2021.
[55] Goyal, P. Dollár, R. Girshick, Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint arXiv:1706. 02677, 2017.
[56] Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization of neural networks. In Hal Daumé III and Aarti Singh, editors, Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10767-10777. PMLR, 13-18 Jul 2020.
5 NeurIPS checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We provide code to reproduce all our experiments and figures at https: //t////annnn //anonymoouu..../ .ww.oouus............... 4open. ccciencerrrreeuudttt.reeeeereeee
E4FD.
13

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Methods section.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs. internal cluster, or cloud provider)? [Yes] We ran all our experiments on V100 GPUs
4. If you using existing assets (c.g., code, data, models) curating/releasing
assets... (a) If your work uses existing assets, did you cite the creators? [Yes] We used the standard MNIST [28], CIFARTO-CIFAR data sets [25] and ImageNet [27]. See Methods section
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "upWp7MJDIDP", "reviewer": "Reviewer_Ef6D", "summary": "This paper studies the representation of the last hidden layer of wide neural networks. The authors observe that these representations for networks that achieve zero training error contain redundant \"clones\". It is observed that by randomly dropping neurons of the last hidden layer, the performance is similar to training the network in the reduced size. This could serve as an explanation for the benign overfitting of over-parameterized neural networks.", "questions": "1- If redundant representations help generalization for wide enough neural networks, then we should observe that for the full network in Figure 1 (b), the error is lower than the chunk network. However, for large widths, we are observing the opposite. Then, either the explanation made in the introduction is not clear or the conclusion simply doesn't hold.\n\n2- Figure 1: Why the reference for error is error_inf? Why not report the error itself? Also, what is the classification error computed on? \n\n3- In finding number 1, what do you mean by \"w_c predicts the output with an accuracy that scales with w_c^(-0.5)\"? Why is the slope of the figure of error versus net size referred to as \"accuracy\"? \n\n4- Why is w_c selected randomly? Why not select them from the correlation values to ensure the correlated neurons are not selected together and only the uncorrelated ones are selected together?  \n\n5- The main motivation behind the work is its explainability of the bias-variance mystery. However, the exact connection has not been made. How would the bias-variance curve look in your experiments? \n\n6- Does the point regarding regularization only apply to weight decay or any form of regularization? If the latter, then the scope of the finding would be very limited. Also, what is the explanation behind this?\n", "limitations": "Yes, the authors mention the limitation of the observation in finding number 3.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1- The observation is novel and interesting. \n\n2- Some ablation studies including studying the intermediate layers have been reported.\n\nWeaknesses:\n\n1- A missing discussion from the paper is the connection between the presented observation and pruning neural networks. If it's the case that neurons are redundant, how can we select the redundant ones and drop them to have pruned networks? This could be a useful pruning strategy that is not covered in the paper. Overall, the importance of the outcome of the study is not clear and no application is shown.\n\n2- The significance of the result is limited: The scope of the observation is limited to having zero training error and using weight decay. If the network is stopped early or there is no regularization the observations won't hold. Also, the last layer of the neural network should be very wide.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bexVuc3ivWW", "reviewer": "Reviewer_aMM9", "summary": "The paper provided an insight into the “benign overfitting” of convolutional neural networks, by inspecting the last hidden layer representations of the networks. They observed and proposed an explanation of the phenomenon that the neurons tend to split into groups carrying duplicated information when the network is wide enough and the training error is zero.  ", "questions": "1. Limitations on architectures: the research is focused on convolutional neural networks. \n    - Does it generalize to Transformers or other networks with attention maps? \n    - How much does it depend on the existence of the convolution units, which have limited span? A related question: by changing the filter size of the convolutional layers from small to very large, will the following claim hold “... neurons tend to split into groups that carry identical information”?\n\n2. In several places of the work, e.g. Figure 5, the author(s) mentioned that the training error reached zero, and they continued the training to get the claimed results. A naive question: if the training error stays zero, where does the gradient come from? \n    - Is the training error actually very small? If so, it is important to show the training curve or relevant statistics during the stage where the neurons are trying to duplicate themselves, in order to understand whether there is any phase transition or dynamical changes. \n    - If the training error is actually zero, the updates of parameters may solely be due to weight decay. If so, the weights are simply shrinking by a factor of (1-learning_rate * weight_decay_factor) per step after the training error reaches 0, and this effect alone introduces the phenomenon of duplicated neurons. Then this would be a very useful training recipe, which could be validated and applied on broader sets of architectures.\n\n3. It is claimed that “clones appear if the training error is zero and the model is trained with weight decay”. I understand that weight decay is a way of regularization. But is there any explanation why standard regularizations, e.g. imposing L2 norm or Gaussian priors, don’t work? Or if other regularizations also work, how do they compare with using weight decay?\n\n4. The main claim of “distinct clones provide independent measures of the same salient features of the data” seems to rely on the fact that the error decays approximately as W^{-1/2}, when the chunk width W goes beyond critical chunk size w_c. However, in Figure 2, there are only 2 data points per model beyond w_c=256; in Figure 3, the W^{-1/2} law seems not so well for CIFAR 10 and CIFAR 100; in Figure 8, it seems far away from W^{-1/2} for ImageNet. \n    - Based on these, apparently, the empirical claim of “the error of the chunks with w neurons decays as w^{-½}” is a weak claim, especially for more complex datasets CIFAR 100 and ImageNet. Are there any other experiments or insights to support the main claim?\n\n5. Is there direct proof of the claim in line 152 “In the following we will indeed see that distinct clones differ from each other by uncorrelated random noise.” Would it be possible to verify this claim by creating statistics of the activations of neurons in the last layer?\n\n6. Other minor issues:\n    - When mentioning NTK in ln 290, I assume you are talking about neural tangent kernels. Please add proper references for that.\n    - For any figures/tables in appendix, please refer then with “Appendix”, e.g. ln 245, Figure 8-c should be Appendix Figure 8-c  \n", "limitations": "Yes. Limitations are well stated and discussed.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Strengths\n\n  - Originality: the paper brought a novel insight into an old problem of deep neural networks: why overfitting does not kill the generalization capability.  \n  - Quality: the author(s) did comprehensive experiments and ablation studies on four datasets, pMNIST, CIFAR10, CIFAR100 and ImageNets with different variations of Wide-ResNet-28, ResNet50 and DenseNet. \n  - Clarity: The paper was well written with clear explanations of background, related works, methods, experimentations and limitations.\n  - Significance: The problem that the author(s) were trying to attack is very important to both the understanding of the neural network training dynamics and the design principles of the wide and deep neural network architectures in practice. \n\n* Weaknesses:\n  - Please refer to the Questions sections.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "X4ENOHy2iDc", "reviewer": "Reviewer_NAgd", "summary": "This paper studied the statistical properties of randomly selected chunks of neurons in the last layer representations of wide deep neural networks. \n\nSpecifically, the authors observed that when the chunk size is larger than a threshold, the test error is similar to that of a full network of the same width under a decay rate of $-1/2$. This implies that sufficiently large chunks can be viewed as statistically independent estimators of the same features of the data (referred to as \"clone\"), and the authors further characterize the various properties of \"clones\".", "questions": "Do the authors consider conducting their experiments on ImageNet with a model with higher redundancy, e.g., increase the width of the intermediate layers in addition to the final layer? It is interesting to see if clones exist and their properties.\n\n", "limitations": "No limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The observation that sufficiently large chunks of neurons can be viewed as independent representations of the same data manifold is interesting. It may provide new insights for the connection between redundancy and generalization, model pruning and ensemble learning. \n2. The hypothesis is well-motivated by observations on multiple architectures and datasets. \n3. Extensive quantitative analysis has been conducted to verify the hypothesis and characterize the properties of \"clones\".\n\nWeakness:\n\n1. The study is only based on small scale datasets and networks. It seems that clones appear only when the model has sufficient capacity to interpolate the training data. As the properties of clones remain unknown in larger-scale settings, further implications and applications can be limited.\n\nMinor: \n\n1. Missing y-axis in Figure 1.b. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept