Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper shows empirically that under certain conditions (overtraining, large width in all layers, data augmentation and regularization) neural networks (NNs) tend to learn redundant representations in the last layer. Precisely, a large enough random subset of the penultimate layer activations of size $w_c$ can almost perfectly linearly predict the whole activation of width $W$.

Further, the training error when pruning the network to these random subsets also goes down (lagging behind the training error of the full network) to 0 monotonically as training progresses.

Test error of these pruned networks appears to go down as $w_c^{-1/2}$, similarly to the test error of regular networks going down as $W^{-1/2}$.

The authors put forward some preliminary interpretation of the effect and suggest this redundancy effect could be causally responsible for the phenomenon of generalization of wide networks improving with width.

main_review: ## Strengths:

1. The presented mitosis effect is interesting and is clearly demonstrated in certain settings.

1. Authors also show settings where the effect is clearly absent.

1. Understanding how overparameterization improves generalization remains an open research question and this finding could stimulate fruitful future research.

1. Overall the paper is well-written and figures are of good quality.

## Weaknesses:

1. IMO the main weakness of the paper is clarity,  and I hope this can be improved during rebuttal. I have found the writing to be very concise / missing details / references / mathematical derivations to be able to fully appreciate and evaluate the claims made in the paper. Specifically:

	1. What exactly is $\textrm{error}_{\infty}$? Is it an ensemble of the full network of the largest $\max W$ considered? Or is it an ensemble of networks of width $W$, where it changes along the $x$-axis of the plot? When using chunks, is an ensemble of chunks, or of full networks of width $W$, or $\max W$? Without knowing the exact definition, it's very hard to interpret a lot of plots in the paper. Further, specific ensemble sizes should be specified in the appendix.
    1. For all plots showing the $\textrm{error} - \textrm{error}_{\infty}$, I would appreciate to also see respective plots of simply error, and loss in the appendix, especially if "error infinity" depends on width W. In this case it's not clear from the plots that the error is even going down with width, since "error infinity" could increase. If "error infinity" is a single constant for all points on the plot, this is less important.
    1. Start of page 2: *"The decay rate of −1/2 in particular implies that in this regime chunks of $w_c$ neurons can be thought as statistically independent estimators of the same features of the data, differing only by a small, uncorrelated noise"*. Could you please expand on this statement, perhaps provide some mathematical details in the appendix / references? I don't understand it currently, i.e. why, and what exactly does the $-1/2$ scaling implies.
    1. End of same paragraph: *"The accuracy of these wide networks then improves with their width because the network implicitly averages over an increasing number of clones in its representations to make its prediction"*. Similarly, could you please explain more formally what is implicit averaging and why does it improve accuracy?
    1. Middle of page 4: *"This implies that a model obtained by selecting a random chunk of $w_c > w_c^∗$ neurons from a wide final representation behaves similarly to a full network of width $W = w_c$"*. per my point above, due to the ambiguous nature of "error infinity", it is not clear to me if they actually behave similarly, i.e. have comparable test error, or only comparable $\textrm{error} -\textrm{error}_{\infty}$. Again, providing plots of only test error and loss would be much appreciated, especially if "error infinity" is not a constant but changes with width $W$ or even chunk width $w_c$.
    1. End of same paragraph: *"Furthermore, a decay with rate −1/2 suggests that the final representation of the wide networks can be thought of as a collection of statistically independent estimates of a finite set of data features relevant for classification. Adding additional neurons to the chunk hence reduces their prediction error in the same way an additional measurement reduces the measurement uncertainty, leading to the −1/2 decay."* Similarly to a comment above, I would appreciate it if this statement was made mathematically explicit in the appendix or a specific reference provided.
    1. Page 5, top: *"We call a chunk of neurons a clone if it fully captures the relevant features of the data, up to some uncorrelated random noise."* Please provide a more formal definition.
    1. Page 5 below: *"The ID of the widest representations gives a lower bound on the number of coordinates required to
describe the data manifold, and hence on the neurons that a chunk needs in order to have the same
classification accuracy as the whole representation. The ID of the last hidden representation is 2 in
pMNIST, 12 in CIFAR10, 14 in CIFAR100, numbers which are much lower that the width at which a
chunk can be considered a clone."* Could you elaborate more on the dependence between ID and $w_c$? I can see why $w_c$ can't be smaller than ID, but curious if there's anything more that could be said, e.g. perhaps could be a more specific argument on how $w_c$ depends on $W$ and ID in some toy setting. It's OK if you don't have an answer, but I am curious if there are ways to ballpark estimate $w_c$ to gain better intuition into it.
	1. Top of page 6: *"In the previous paragraphs we set forth evidence in support of the
hypothesis that large chunks of the final representation of wide DNNs behave approximately like an
ensemble of independent measures of the full feature space. This allowed us to interpret the decay of
the test error of the full networks with the network width observed empirically in Fig. 2"*. As above, I don't know what an "ensemble of independent measures on the full feature space" means, or why it explains the scaling, so would appreciate much more mathematical details here.
	1. Discussion, section 4: again, I don't understand what "statistically independent learning schedule" is, and it's hard to follow the argument in the first paragraph. Would appreciate mathematical details in the appendix or a specific reference.
    1. Discussion, bottom: *"The number of clones grows linearly"* - why does it grow linearly, and not, for example, as ${W \choose w_c}$? Per above, this will also depend on what you define to be a clone exactly.
    1. Equation (1): what exactly are $\bf{x}$s? Training set activations, or validation set, or test set, or all together? Do the findings of the paper hold for all settings (train/test/train+test) above?
    
1. Another issue I find with the paper are the claims that mitosis explains why test error falls with width. For example, on page 9 the authors claim *"we make the observation that co-adaptation of hidden units in the form of clones occurs without
dropout, and is crucial for their improving performance with width"*; in Figure 11 the authors conclude that since the T-shaped network generalizes poorly, therefore it also can't have mitosis, which I find unjustified, and think this Figure should at least show the standard mitosis plot like in Figure 1. Overall I did not find any evidence in the paper that mitosis is responsible for improved generalization with width, if anything, Figure 8 shows that it's not. To make claims about mitosis being responsible for improved generalization with width you would need to run many more experiments, comparing generalization scaling of networks with and without mitosis. But I'm pretty sure if mitosis does not arise without data augmentation and/or weight decay as the paper claims, then it can't be responsible for improved generalization with width, since non-regularized networks still improve their generalization with width even without regularization (see e.g. https://arxiv.org/pdf/1412.6614.pdf).

1. I appreciate that you have identified that mitosis requires data augmentation and/or weight decay. It would be very interesting to ablate this further, and identify which is responsible for it exactly, especially since these are two very different techniques. In general, I wish there was more insight into why exactly the conditions you listed in the paper are necessary for mitosis.

## Minor:

1. Page 1, extra space after Sec. A.2
1. Figure 1/2 - inconsistent y-axis name w/ and w/o "log".
1. Figure 1 - no y-axis ticks.
1. Page 4, extra comma after "significantly"?
1. Figure 2: why stop so narrow for FC5, isn't this the cheapest setting, where width could be taken to be much wider than other more expensive models?
1. Page 6 bottom - "(orange)" -> "(blue)"?
1. Page 7" "done CIFAR100" -> "done on".
1. Figure 5 - would prefer matching chunk sizes and colors in (a) and (b), and plotting all panels from initialization, with shared $x$-axis.
1. Figure 6 (b) - does $x$-axis start at 0? If so, how come thick and thin lines don't start at the same $y$-values?
1. Page 8: "much higher than what observed" - "we have observed"?
1. Page 9: "found that that".
1. Page 9: "along those line" - plural "lines"?
1. Figure 9: "blu" -> "blue".

summary_of_the_review: # Post-rebuttal update

The authors have clarified most of my questions, and I am raising my score to weak accept. My original conclusion remains similar, but improved clarity makes it a stronger submission. Recap:

Why accept: 
* The empirical phenomenon of mitosis is interesting and shown to be robustly present in some cases, but absent in others. 
* The paper is well-written and clear (given recent updates and replies to my questions, and a few more clarifications that I expect to see in the final revision).

Why weak: 
* Exact conditions for mitosis are still not well understood. In the rebuttal the authors have updated the condition set to 100% training set fit + "optimal" [what is optimal?] weight decay), which is good, but I still don't think this is precise enough, and most notably the paper does not contain enough systematic experimental evidence to confirm it (e.g. mitosis measurements in wide FC networks, sweeping the weight decay from 0 to X, and observing that the rate of mitosis increases respectively).
* Exact implications of mitosis are also not well understood / demonstrated (e.g. what is the relationship between mitosis and generalization? Can a network demonstrate improved generalization with increasing width without mitosis? Can a network without improving generalization with width exhibit mitosis? etc).

# Original Review


The paper presents an interesting empirical finding, but in my opinion does not explain very well why mitosis occurs, or what are the implications of it, which is why I am leaning to reject it at this time. However, many of my concerns could be addressed in the rebuttal, so I might change my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provides a novel explanation of benign overfitting in wide neural networks by introducing and studying a mechanism called representation mitosis. The key idea is that if the readout layer of the properly trained network is wide enough, then its neurons could split into groups (clones) that carry identical information, and differ from each other by a statistically independent noise (this mimics what happen in mitosis, thus the name for the mechanism). Moreover, above certain threshold the number of such groups increases linearly with the width, shedding light on when features are learnt. Empirical results are provided to demonstrate the mechanism. 
main_review: Strengths: 
- Introduce a novel, interesting phenomenon to understand feature learning in properly trained wide neural networks
- Sufficient empirical evidences to support the claims
- Overall a well-written paper

Weaknesses: 
- The conditions to achieve representation mitosis are not pinned down precisely. 
- There are some heuristic claims saying that clones appear only in *well-trained* regularized networks and do not appear for the experiments on ImageNet. The more interesting question is, given (some assumptions/conditions for) a dataset, a model architecture and a training algorithm, when and how precisely can the mitosis be reached? This could be useful for choosing models and training methods to achieve the mitosis even before training the model.

Minor comments/questions:
- How does the depth and connectivity (say, sparse vs. dense networks) of the neurons in the earlier layers affect the mechanism? 
- Will achieving the mitosis generally lead to better generalization? There seems to be evidences of this on page 7.
- When choosing the chunk of neurons randomly, what guides the choice for the number (or fraction) of neurons? What if we do not choose it randomly but choose it according to some deterministic procedure instead?
 
summary_of_the_review: The mechanism of representation mitosis seems to be interesting and is worth exploring. The paper provides satisfactory explanations and ample demonstrations to help understanding the mechanism. However, some of the explanations are at best heuristic in nature and would be much stronger if they could be supported by rigorous theory. I am inclined to go with a weak reject for now.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper identify and studies a phenomenon called "representation mitosis": as the neural network width increases above a critical threshold, the neurons in the last layer becomes "redundant" in the sense that they start to form groups of neurons where they carry identical information, and differ from each other only by a statistically independent noise. 
main_review: **Strength**: this paper identifies an interesting phenomenon in deep neural networks and studies from various aspects.

**Weakness**: the empirical evidences are a bit misaligned with the claims, and it is unclear how this observations could be used. Please see below for details:

1. The mitosis phenomenon is described as formation of "clones" of neurons. However, the evidences only show that a (large enough) random subset of neurons could retain the prediction performance or even reconstruct the original outputs. From this observation, it is unclear if there are one-to-one correspondence of copies of neurons, or it's simply the same set of information are encoded in the subspace distributedly, i.e. a (linear) combination of a subset of neurons corresponds to another subset of neurons, but there is no neuron level correspondence. It would be great if the paper could empirically identify the neurons and their "clones" to more directly support the claim of representation mitosis. Otherwise, the current observation might not be very different from the phenomenon of parameterization redundancy of large neural networks that are reported in many previous papers that a subset of neurons or even whole layers could be removed without affecting the network performance.

2. It is unclear what is special about the last hidden layer. 

3. I find it very interesting that the representation mitosis requires using state-of-the-art training setup and continued training after training error reaches zero. If the paper could dig deeper into those situations and identify what happens in different scenarios and the underlying reason for the discrepancy, then that would be a much more interesting paper. 

4. The mitosis phenomenon was not reproduced on ImageNet. The suspected reason is that the network is not big enough in this case. It is understandable that the computation cost might be high to train very large models on ImageNet. However, nowadays there are many pre-trained ImageNet models available to download online, some of which (e.g. some vision transformer models) are potentially very large. Since mitosis analysis can be applied to those pre-trained models without re-training. It would be interesting to see if they are still absent in those larger models.

5. It would be great if some applications of the observations could be demonstrated. Can we use this to improve training? Or improve architecture design? Or guide neural network pruning? 

-----
**After rebuttals**: Thanks the authors for posting the rebuttals, which addresses some of my concerns. I realized some of the experiments (ImageNet) cannot be relatively cheaply done and added to the paper after reading the rebuttal. However, I'm still concerned about how general this phenomenon occurs (in simpler architecture or in more complicated dataset). Furthermore, I still find the presentation (including text description and figure illustrations) of the phenomenon as "clones" of features very confusing, which the authors acknowledged are not neuron-to-neuron one-to-one mappings. As a result, I'm keeping my current rating.

summary_of_the_review: This paper identifies a potential interesting phenomenon in deep neural networks. However, I think the empirical evidences need to be strengthened to support the claim well.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the overparameterization problem in deep learning through a set of experiments that study the final-layer hidden representations of wide and deep networks. The authors find that -- for wide enough networks and after training long enough -- that they can achieve similar test error from a model made by sampling only a subset of the final-layer hidden representations and then deleting the other connections in that layer. Furthermore, in such situations they can find a linear map from the subset that predicts the full set of final-layer representations, suggesting that the information contained in the deleted features is "cloned" and present in remaining subset. Overall, this effect and its dynamics are explored empirically across a number of models, datasets, and learning scenarios. Curiously, the cloning only appears in networks where data augmentation and weight decay is used, and the authors were unable to exhibit this effect on ImageNet.
main_review: The paper is mostly empirical in nature, and I think the authors performed a very nice set of carefully designed experiments to explore their novel "cloning" effect. I appreciate the variety of architectures and vision datasets considered, and I further appreciate the candor in Section 3.1 on Limitations. I also think that the analysis of reconstructing the wide representation from the smaller subset is a nice way of making the point about features being cloned and the information being present in the smaller subset.

I think two primary weaknesses involve the aforementioned limitations.

The first weakness is the necessity of using heavy regularization and complicated training procedures. For typical discussions of overparameterized networks -- e.g. ones that discuss the kernel learning regime -- regularization isn't necessary to see the double descent phenomenon highlighted in the Belkin et al. paper cited by the authors. In fact, gradient descent is supposed to automatically pick out the minimal norm solution, and such models overparameterized trained this way do not overfit even if they are not explicitly regularized. Thus, I conclude that the "cloning" effect observed by these authors cannot be the main explanation for the way DNNs appear to defy the classical bias-variance trade-off. Further, I wonder whether "cloning" is somehow an artifact of the augmentation+regularization used. Perhaps the authors could explore this more and/or come up with ways to disentangle these effects or explore cloning/mitosis in other overparameterized models so that its origin can be better understood?

Next, I think it's problematic that the cloning effect was not observed for ImageNet, though this is definitely a lesser weakness: it could potentially be explained by the fact that the authors were unable to get a wide enough final hidden-layer representation in comparison to the number of classes in ImageNet. As further evidence, the authors highlight that they were not able to train their widened ResNet50 to convergence, which was a prerequisite to see "cloning" in the other instances. However, I am worried -- since there isn't any strong theoretical explanation or intuition provided -- that the cloning observed is potentially an artifact of modeling simpler datasets -- e.g. ones where PCA would give reasonable representations -- and thus is not a very general phenomenon in deep neural networks.

Some additional comments: 
- I would enjoy further discussion about what determines $w_c$ for a particular set of architecture, learning algorithm, and dataset. Does varying the depth of the network have any effect on this threshold? (In recent analyses of wide networks, the network's depth to width ratio plays an important role in determining when a network becomes "wide," so this seems like an appropriate scale to compare the width to.)
- The authors claim that the residual after reconstruction is simply white noise. Is it possible that higher order correlations than the covariance are present and not captured by the subset chunk?
- In the final paragraph, the authors say that they focus on networks in the feature learning regime, but I thought that to observe their mitosis effect that networks need to be sufficiently wide. Thus, I find their claim a little hard to believe. Do they think that there's another transition for even wider networks that are lazy learners, in which there is no cloning? (By the way, I think that in many ways, the infinite-width limit provides a natural starting point to explain mitosis, since the hidden representations in such networks are all independent at initialization and throughout training.)

summary_of_the_review: I think this is a nice paper that demonstrates an interesting empirical effect and proposes a promising connection to the behavior of overparameterized deep neural networks. However, in its current form, I think the submission is marginally below the acceptance threshold. In particular, I am bothered by the fact that the effect appears only after training "using state-of-the-art procedures" and is absent for complicated datasets (the example in the paper being ImageNet).

Overall, I think this paper makes a nice set of observations and performs careful scientific experiments. With a little more clarity in terms of how general this effect is, I would be happy to accept it to ICLR.

### After Author Responses
I appreciate the comments of the authors. I still am concerned about how general the mitosis effect is, since it doesn't occur in simpler models. However, with the additional clarity, I have increased my score to be at the acceptance level.
review_title: null

Meta-review: This paper undertakes an empirical investigation of overparameterized neural networks, studying the last hidden representation and identifying "representation mitosis," a cloning effect whereby neurons split into groups that carry the same information. The effect is observed for a variety of architectural configurations/datasets, and a detailed set of experiments are performed to investigate the behavior.

The reviewers had split opinions about this paper, with most reviewers appreciating the novelty and salience of the observations, but with some reviewers expressing skepticism about the generality of the effect. While the experiments are thorough and revealing, the practical importance of representation mitosis remains somewhat unconvincing.

A primary motivating factor for the analysis is the search for an explanation of the unexpectedly good generalization behavior of oveparameterized networks and the origin of "benign overfitting." As highlighted in the reviews, the sensitivity of the mitosis effect to (1) training to zero loss and (2) optimal regularization suggests that it cannot be the sole explanation for benign overfitting, since the latter can and does occur without these conditions. The authors acknowledge this situation, and respond that their focus is on state-of-the-art models used by the community, rather than on toy settings. For this to be a persuasive response, more compelling results in these state-of-the-art situations should be evidenced -- in particular, as several reviewers pointed out, the negative results on ImageNet undermine this point to some extent.

Overall, representation mitosis does seem like an interesting and potentially important phenomenon, but further work is needed to develop persuasive evidence in support of the interpretations and implications. While this is a borderline submission, I believe it falls just short of the mark, and cannot recommend acceptance.
Decision: Reject