On the geometry of generalization and memorization in deep neural networks
Authors: Cory Stephenson
suchismita padhy
Abhinav Ganesh
Yue Hui
Hanlin Tang
SueYeon Chung
Venue: ICLR 2021 Poster (2021)
Keywords: deep learning theory
representation learning
statistical physics methods
double descent

Abstract: Understanding how large neural networks avoid memorizing training data is key to explaining their high generalization performance. To examine the structure of when and where memorization occurs in a deep network, we use a recently developed replica-based mean field theoretic geometric analysis method. We find that all layers preferentially learn from examples which share features, and link this behavior to generalization performance. Memorization predominately occurs in the deeper layers, due to decreasing object manifolds’ radius and dimension, whereas early layers are minimally affected. This predicts that generalization can be restored by reverting the final few layer weights to earlier epochs before significant memorization occurred, which is confirmed by the experiments. Additionally, by studying generalization under different model sizes, we reveal the connection between the double descent phenomenon and the underlying model geometry. Finally, analytical analysis shows that networks avoid memorization early in training because close to initialization, the gradient contribution from permuted examples are small. These findings provide quantitative evidence for the structure of memorization across layers of a deep neural network, the drivers for such structure, and its connection to manifold geometric properties.


Published as a conference paper at ICLR 2021
ON THE GEOMETRY OF GENERALIZATION AND MEMO- RIZATION IN DEEP NEURAL NETWORKS
Cory Stephenson" 11 Hanlin Tang' and Sue Yeon Chung*
Suchismita Padhy" "1 Abhinav Ganesh', Yue Hui?,
Intel Labs, Stanford University. Columbia University,
/cory.stephenson.suck seaarppeennoosuccssiitte pachy,, abhinavv ganesh. J intel come yueh stanford. hanlin. lang @ intel. sueyeon. sweyenn.chunge @ columbia.edu
ABSTRACT
Understanding how large neural networks avoid memorizing training data is key to explaining their high generalization performance. To examine the structure of when and where memorization occurs in a deep network, use recently developed replica-bace basied mean field theoretic geometric analysis method. We find that all layers preferentially learn from examples which share features, and link this behavior to generalization performance. Memorization predominately occurs in the deeper layers, due to decreasing object manifolds' radius and dimension, whereas early layers are minimally affected. This predicts that generalization can be restored by reverting the final few layer weights to earlier epochs before significant memorization occurred, which is confirmed by the experiments. Ad- ditionally, by studying generalization under different model sizes, we reveal the connection between the double descent phenomenon and the underlying model geometry. Finally, analytical analysis shows that networks avoid memorization early in training because close to initialization, the gradient contribution from permuted examples are small. These findings provide quantitative evidence for the structure of memorization across layers of deep neural network, the drivers for such structure, and its connection to manifold geometric properties.
INTRODUCTION
Deep neural networks have many more learnable parameters than training examples, and could simply memorize the data instead of converging to generalizable solution (Novak et al.. 2018). Moreover, standard regularization methods are insufficient to eliminate memorization of random labels, and network complexity measures fail to account for the generalizability of large neural networks (Zhang et al. 2016;; Neyshabur et al.) 2014). Yet, even though memorizing solutions exist, they are rarely learned in practice by neural networks (Rolnick et al. 2017).
Recent work have shown that combination of architecture and stochastic gradient descent implicitly bias the training dynamics towards generalizable solutions (Hardt et al.] 2016) Soudry et al. 2018 Brutzkus et al. 2017 Li and Liang] 2018 Saxe al., 2013] [ampinen and Ganguli 2018). However, these claims study either linear networks or two-layer non-linear networks. For deep neural networks, open questions remain on the structure of memorization, such as where and when in the layers of the network is memorization occurring (e.g. evenly across all layers, gradually increasing with depth, or concentrated in early or late layers), and what are the drivers of this structure.
Analytical tools for linear networks such as eigenvalue decomposition cannot be directly applied to non-linear networks, so here we employ a recently developed geometric probe (Chung al. 2018) Stephenson et al.) 2019). based on replica mean field theory from statistical physics, to analyze the training dynamics and resulting structure of memorization. The probe measures not just the layer capacity, but also geometric properties of the object manifolds, explicitly linked by the theory.
We find that deep neural networks ignore randomly labeled data in the early layers and epochs, instead learning generalizing features. Memorization occurs abruptly with depth in the final layers, caused by
*: Equal contribution, +: Correspondence.

---

Published as a conference paper at ICLR 2021
decreasing manifold radius and dimension, whereas early layers are minimally affected. Notably, this structure does not arise due to gradients vanishing with depth. Instead, analytical analysis show that near initialization, the gradients from noise examples contribute minimally to the total gradient, and that networks are able to ignore noise due to the existence of "shared features' consisting of linear features shared by objects of the same class. Of practical consequence, generalization can then be re- re-gained by rolling back the parameters of the final layers of the network to an earlier epoch before the structural signatures of memorization occur.
Moreover, the 'double descent' phenomenon, where a model's generalization performance initially decreases with model size before increasing, is linked to the non-monotonic dimensionality expansion of the object manifolds, as measured by the geometric probe. The manifold dimensionality also undergoes double descent, whereas other geometric measures, such as radius and center correlation are monotonic with model size.
Our analysis reveals the structure of memorization in deep networks, and demonstrate the importance of measuring manifold geometric properties in tracing the effect of learning on neural networks.
RELATED WORK
By demonstrating that deep neural networks can easily fit random labels with standard training procedures, Zhang et al. (2016) showed that standard regularization strategies are not enough to prevent memorization. Since then, several works have explored the problem experimentally. Notably Arpit et al. (0117) examined the behavior of the network as single unit when trained on random data, and observed that the training dynamics were qualitatively different when the network was trained on real data vs. noise. They observed experimentally that networks learn from 'simple" patterns in the data first when trained with gradient descent, but do not give a rigorous explanation of this effect.
In the case of deep linear models trained with mean squared error. more known about the interplay between dynamics and generalization (Saxe et al. 2013). While these models are linear, the training dynamics are not, and interestingly, these networks preferentially learn large singular values of the input-output correlation matrix first. During training, the dynamics act like singular value detection wave (Lampinen and Ganguli 2018), and so memorization of noisy modes happens late training.
Experimental works have explored the training dynamics using variants of Canonical Correlation Analysis (Raghu et al. 2017 Morcos et al., 2018), revealing different rates of learning across layers, and differences between networks that memorize generalize (Morcos et al. 2018).. Using Centered Kernel Alignment, Kornblith et al. 2019) find that networks differ from each other the most in later layers. However, as these metrics measure similarity, experiments are limited to comparing different networks, offering limited insight into specific network instances. As noted in (Wang et al. 2018). the similarity of networks trained on the same data with different initialization may be surprisingly low.
Our work builds on this line of research by using a direct, rather than comparative, theory- measure of the representation geometry (Chung et al., 2016) Cohen et al. 2019) to probe the layerwise dynamics as learning gives way to memorization. With the direct theory-based measure, we can probe individual networks over the course of training, rather than comparing families of networks.
3 EXPERIMENTAL SETUP
As described in (Arpit et al. 2017). we adopt the view of memorization as "the behavior exhibited by DNNs trained on noise." We induce memorization by randomly permuting the labels for a fraction of the dataset. To fit these examples, a DNN must use spurious signals in the input to memorize the 'correct' random label. We train our models for either 1, 000 epochs, or until they achieve > 99% accuracy on the training set, implying that randomly labeled examples have been memorized. We do not use weight decay or any other regularization.
Once a model has been trained with partially randomized labels, we apply the recently developed replica- replica-based mean-field theoretic manifold analysis technique (MFTMA hereafter) (Chung et al. 2018) to analyze the hidden representations learned by the model at different layers and epochs of training. This quantitative measure of the underlying manifold geometry learned by the DNN provides insight into how information about learned features is encoded in the network. This method
2

---

Published as a conference paper at ICLR 2021
(A)
Chre
anchor pointă
manifold
Manifold Capacity
s.t. linerryy separable
maximum
1
Manifold Dimension
Du (f-Rtt
1
Traiing Boato
oner
Permattel
to
T, to
Manifold Radius
Ry VPPe
Gaussian vector
Center correlations
Ponteer (14 Gylies
P manifolds
N amblent dimension
Araysi Cadte
List nower
Examples
outline training labels
Before Training (Highly Tangled)
Best epoch (Generalizing)
Last (Memorizinn)
(8)
soft fill color / ground truth labels
Rermutted Restered
Manifolds
Empincaal
Solid outline / training manifolds
Manifold Capacity
Low
High
Highest
2
Low
High
Intermediate
c0
test manifolds
Theoreical capaeity (MFTMA))
Figure 1: Description of Manifold Analysis. (A) Random samples of unpermuted, permuted, and restored examples showing the label used for training and our geometric analysis. Note that restored and permuted examples only differ the label used for analysis. (B) Empirical manifold capacity matches theory. (C) (Top) MFTMA framework formally links manifold capacity, a measure object manifolds' linear separabilityy, to geometric properties. Reduced dimension, radius, center correlations result in larger manifold capacity and vice versa. (Bottom) Hypothesis of how network training affects geometry and separability of object manifolds in the presence of label noise.
has been applied to pre-trained models (Cohen et al. 2019), whereas here we study the training dynamics in the memorization regime.
3.1 DATASETS AND MODELS
We carried out experiments on a variety of neural network architectures and datasets. For CIFAR 100 (Krizhevsky al. 2009), trained AlexNet (Krizhevsky et al.) (014), VGG-16 (Simonyan and Zisserman 2014) and 18 (He et al. 2016) models, each slightly modified for use with the dataset. We also used Tiny ImageNet datase(" reduced version of ImageNet (Deng et al. 2009 with ResNet-18. Training methods similar to those used for the CIFAR- 100 dataset did not result in complete memorization on Tiny ImageNet, owing the larger and more complex dataset, and so we show results on partial memorization after training for 1,000 epochs. Due to space constraints, we present results for VGG-16 trained on CIFAR-I00. Other architectures and datasets have similar findings, as detailed in the Appendix.
Our networks are trained on dataset with some fraction of its labels randomly permuted. To analyze the behavior of the trained models, we define four subsets of examples as follows:
Unpermuted examples: The set of labeled examples from the training set which did not have their labels permuted. That is, the networks were trained on these examples using the correct label. In the case of 100% permuted data, this set is empty, as we not count examples with labels randomly assigned to the correct class.
Permuted examples:
The set of labeled examples from the training set which had their labels randomly permuted. The networks were trained to classify these examples according to their random labels. By design, there some chance that the random assignment results in the label remaining unchanged, however these examples are still in the set of the permuted examples. The motivation for this choice can be found in the Appendix.
Restored examples: These examples the same the permuted examples, except when carrying out analysis we restored the correct, unpermuted labels. For example, for a permuted example with
This datasct is from the Tiny ImageNet Challenge: ittps:///tiyy-imgeeeee..tet

---

Published as a conference paper at ICLR 2021
(A)
AlaeNet- CITAR-30C
Text
Perrittdd Mestured
0.25
Eooch
0.00
CIFAR-10D
VGG16 CIFA-100
0.25
0.00
ResNe128 -FAR-110
ResNeti8 CIAA 100
1.00
0.25
0.00
Epoch
1.00
ResMe118 Thyy Imagehe:
Reshetts Try Inagelet
0.75
101
107
12
167
10
Figure 2: Generalization and memorization Training and test accuracy curves for all models and datasets (e = 50%). A (right): Accuracy on unpermuted, permuted, and restored training examples. B: UMAP visualization of the final layer features in VGG16 with 50% at initialization, best epoch, and final epoch. At the best epoch, unpermuted, restored, and test manifolds show similar structure, implying that the network has learned the features shared between the two datasets. First 10 classes of CIFAR 100 are included.
by accuracy and visualization. A (left):
an image class "cat' that was trained with label 'dog', the corresponding restored example would be the same image with the label "cat" restored.
sampling unpermuted, permuted, and restored examples is shown in Fig. TA with the denoted labels used for training and amalysis. Finally, we define test examples:
Test examples: The set of labeled examples from a holdout set not seen during training. All analyses on test examples are done using the correct label.
3.2 REPLICA BASED MEAN FIELD THEORY MANIFOLD ANALYSIS
We apply the replica МFTМА analysis (Chung al. 2018 Cohen et al., 2019, Stephenson et al.. 2019) which formally connects object category manifolds' linear separability with the underlying geometric properties, such as manifold dimension, radius, and correlations.
A key feature in object manifold capacity theory is that it generalises the notion of perceptron shattering capacity from discrete points to class manifolds. Given object manifolds defined by the clouds of feature vectors in N feature dimension, where each manifold is assigned random binary labels where points within the same manifold have an identical label, rendering a total 2" manifold dichotomies. The manifold capacity, ae P/N is defined by the critical number of object manifolds where most of the manifold dichotomies can be shattered (separated) by a linear hyperplane. This is closely related Cover's function counting theorem, except in this theory, the units for counting P for capacity are category manifolds, rather than discrete point patterns. The manifold capacity, therefore, measures the linear separability of categories in the representation. Surprisingly, it's been reported that the object manifold capacity for general point clouds can be formulated in a similar manner to the fored-form manifold capacity of D-dimensional balls of radius R in random orientations (Chung et al. 2016). providing the closed form expressions for effective dimension DM and effective radius RM for the convex hulls of general point clouds in random orientation. More recently, the theory of manifolds has been refined to incorporate correlation structure in the manifold locations, allowing for the estimation of manifold capacities in real data with correlations (Cohen et al. 2019). This analysis, based on the series of theoretical developments mentioned above, formally links the linear separability of category class manifolds (manifold capacity, (rr) with the class manifolds' geometric properties (manifold dimension, Da. manifold radius, RM, and the correlations between the manifold centers, Prenter)) The descriptions of the manifold capacity and the geometrical properties are summarized below (for more details, see Appendix).
4

---

Published as a conference paper at ICLR 2021
Manifold Capacity aus P/N is measure of the decodable class information in given data, defined by critical number of object manifolds (P) that can be linearly classified with random dichotomy labels for each manifold, in feature dimension N. Large aM implies well-separated manifolds in feature space. Using replica mean field theory. this quantity can be predicted (Chung et al. 2018) by object manifolds' dimension, radius, and their center correlation structure provided below (see illustrations in Fig. IC and Appendix). Note that the manifold capacity can also be computed empirically (Empirical capacity, hereafter), by doing a bisection search to find the critical number of features N such that the fraction of linearly separable random manifold dichotomies is close to 1/2. We show that theoretical predictions match empirical capacity with our data in Fig. [B.
Manifold Dimension DM captures the dimensions of the set of anchor points, which are repre- sentative support vectors that determine the optimal hyperplane separating the binary dichotomy of manifolds. These anchor points are estimated from the guiding Gaussian vectors (Fig. [ and Appendix), and the position of the anchor point changes as orientations of other manifolds are varied. The average of the dot product between anchor points and the Gaussian vector defines the effective manifold dimension of given point clouds, and it reflects the dimensionality of embedded anchor points in the optimal margin hyperplane. A small dimension implies that the anchor points occupy low dimensional subspace.
Manifold Radius Rat is an average norm of the anchor points in the manifold subspace, capturing the manifold size seen by the decision boundary. Small Ras implies tightly grouped anchor points.
Center Correlation Peenter measures the average of absolute values of pairwise correlations between various centers. A small correlation indicates that, on average, manifolds lie along different directions in feature space.
In the rest of the text, we will refer to the quantities o.M DM. RM, and Pa as the Manifold Geometry Metries (MGMs). Small values for manifold radius, dimension and center correlation have been shown to result in higher values for manifold capacity (Chung et al. 2018 Cohen et al. 2019). and so represent more favorable geometry for classification (Fig. (C). For all experiments, we evaluate these four quantities using P = 100 classes, with M = 50 randomly drawn examples per class.
We define unpermuted. permuted, restored, and test manifolds for each class at each layer of the network. Given a neural network N(z), we extract (z), the activation vector at layer generated on input x. We then define a manifold by the point cloud of activation vectors generated by inputs of the same class. This results in P manifolds, one for each class. The set of unpermuted manifolds is then obtained by using the corresponding manifolds with examples and labels that were unpermuted. The set of permuted, restored, and test manifolds are defined similarly. We run the MFTMA on each set of manifolds following the procedures in (Chung et al., [0188] Cohen et al., 2019; Stephenson et al. 2019). All results are conducted with 10 random seeds, which govern the network initialization, label permutation, and MFMTA analysis initial conditions.
4 RESULTS
The networks we tested demonstrate memorization of the training dataset, with the eventual decrease of the test accuracy (A left orange curves) and near 100% training accuracy (blue curves). Training with the original dataset also demonstrates this behavior, albeit less exaggerated. More epochs are required fit the permuted examples (Fig. 2 right, orange curves), consistent with (Zhang et al., 2016). However, these examples quickly converge once they begin, reminiscent of the singular value detection wave observed in deep linear networks (Saxe et al. 2013 Lampinen and Ganguli 2018). Importantly, before the permuted examples are learned, many are actually classified as their correct label, as shown by the accuracy increase the restored examples (right, green curves). This is despite the model being trained to emit the wrong randomized label for the restored inputs. Intuitively, restored examples are classified correctly because the networks have learned features that generalize beyond the set of unpermuted (correctly labeled) training examples.
We visualize this effect with a UMAP (McInnes et al. 2018) of the final layer features (Fig. 2B) At the best generalizing epoch, the unpermuted, restored, and test examples are separable while permuted examples are mixed. By the last epoch, however, separability is high for unpermuted and permuted examples, indicating memorization, while restored examples are no longer separable.
5

---

Published as a conference paper at ICLR 2021
(A) Best Epoch
Unpermuted
1.6
0.09
Permuted
155
0.08
Restored
1.4
0.07
Test
De
0.15
0.05
11
0.04
15-
0.10
(B) Last Epoch
10
15
10
16
0
16
18
0.35
0.32
0.30
0.30
1.4
Dcontor
0.25
Rx
Dee
25-
0.06
20
0.04
10
10
Layer
Layer
Layer
Layer
Figure 3: MFT Geometry of generalization and memorization on VGG16. A: Best Epoch. Geometry of restored and test manifolds show significant differences, and permuted manifolds show no evidence of learning. Error bars represent 99% CI of the mean. B: Final epoch, after significant memorization has occurred. Restored and test manifolds show similar trends, and only differ from unpermuted manifolds after layer 14. Faded lines show the results at best epoch for ease of comparison.
(A)
[B)
(C)
(D)
EE)
Best Epoch
Finel Erpoctt
Scwidd tpoch
Rowin
Figure 4: (A-B) Rewinding training: Accuracy on the train set (A), and the test set (B) for VGG-16 trained for 1000 epochs with individual layer's parameters rolled back to an carlier epoch (rewind epoch). The final three linear layers of VGG-16 are unchanged throughout. Horizontal line indicates the peak generalization performance from early stopping. (C-E) Double descent phenomenon, with increasing model parameters, at the best epoch (blue) and final epoch (orange). The region linked increased manifold dimensionality (E), with unchanged radius and center correlation.
Generalizing features are learned in early layers and epochs: MFTMA provides fine-grained view into how each layer's features encode class information by measuring the manifold geometry metrics (MGMs). Figure 3 shows the MGMs computed on VGG-16 trained on CIFAR- 100. Trends are similar across other architectures, permutation fractions and datasets, as shown the Appendix.
At peak generalization (the best epoch, with highest test set accuracy Fig. AA), the early layers exhibit no difference between the manifolds (excluding the permuted manifold). For deeper layers, the manifolds increase in capacity aus indicating that their features contain more information about the correct class label as depth increases. Importantly, this even holds for the restored manifolds (green curve), despite the model being explicitly trained to produce random label. Similarly, the manifold radius Ras and dimension DM decreases for the restored manifolds. This shows quantitatively that in the early epochs of training (up to the point of early stopping) the features learned by all layers, but especially the early layers, of the model generalize across many examples of a class. In contrast, the manifold capacity for permuted manifolds remains near the theoretical lower bound as the network has not yet memorized the randomly selected labels.
"The thcoretical lower bound in this a.M 0.04 from Cover's theorem Cover 1965

---

Published as a conference paper at ICLR 2021
Memorization occurs abruptly in later layers and epochs: At the end of training, after memo- rization. a different set of trends emerges, as shown in Fig. [3. The unpermuted manifolds (blue curve) behavior similarly to the best epoch (blue, shaded). The permuted manifolds (orange) however show a dramatically different trend; the manifold capacity is increased significantly in later layers (Conv16 and onwards), the network has learned to group these examples by their random labels. Similarly, the manifold radius and dimension for permuted examples decrease significantly. In these same later layers, however, the restored manifolds (green) now have decreased capacity compared to the best epoch. While the restored manifolds were initially untangled by the early layers, the later layers have learned to tangle them again as the permuted examples are memorized.
The dramatic trend reversals above are not observed in the early layers. Even after the permuted examples have been memorized, the feature geometry has not changed significantly, implying that the learned features in early layers are still fairly general, with examples grouped according to their true labels even if their training label was randomly assigned.
Undoing memorization by rewinding individual layers: If memorization occurs mostly in later layers, the effects of memorization should be undone by selectively modifying only the weights of the final layers. We carried out rewinding experiment inspired by /Zhang et al. (2019). Using the final model trained for 1,000 epochs, we selectively rewind the parameters of a single layer to their values an earlier epoch (the rewind epoch E). Whereas Zhang et al. (2019) only examine E we rewind to various E throughout training to probe the dynamics. Indeed, as shown in Fig. 4B, simply rewinding the parameters of the final convolutional layer (yellow curve) to an earlier epoch reverses the memorization effect, yielding model that is >90% as accurate as the best early-stopped model. Note that the final three linear layers are not being rewound. This is similar to the freeze training method (Raghu et al.) 2017), which selectively freezes early layers, but here we do not modify the training procedure, and find rather that later layers must be rewound. Furthermore, note that early layers (blue curves) monotonically approach their final performance, whereas later layers (green-yellow) initially learn weights that generalize before reaching the memorizing state.
Prior work (Raghu et al. 2017; Morcos et al. 2018) show that early representations stabilize first, which is consistent with our MFTMA analysis of representations (Fig. 33. However, the behavior of the network weights exhibit the opposite trend; early layer weights (blue curves) take longer to stabilize compared to deeper layer weights (green-yellow curves) (Fig. 4A).
Double descent phenomenon in manifold dimensions: The double descent phenomenon is where the generalization performance gets worse before getting better as the model size gets larger. To check the representational correlates underlying this phenomenon, we varied the parameter count of ResNet- 18 network by scaling the layer width by a common factor, and trained on Cifar 100 with 10% label noise (similar to (Nakkiran et al. 2019).. As expected, the model accuracy exhibits double descent (Fig. 4C). What are neural representations underlying this phenomenon? Our geometric analysis on test data shows that the object manifolds' dimensionality at the final epoch also first decreases, then increases, and decreases again with larger model size, whereas other geometric properties have monotonic relationships similar to the best epoch (Fig. 4D-E).
Manifolds dimensions are linked to the dimensions embedded to the linear classifying hyperplanes (Chung et al. 22018), and higher object manifold dimension renders unfavorable geometry for linear classification, explaining the lower test accuracy. Larger models with higher feature dimensions allow a higher upper bound for object manifolds' dimension. However, large feature dimensions also allow more room for object manifolds without needing to be packed in the classification hyperplane, which can result in lower manifold dimensions. These two factors compete for object manifolds' dimensionality. a property closely linked to their separability. Our geometric analysis reveals that the dimensionality of data manifolds associated with each object class also shows double descent phenomenon, recasting the problem into that of representations. Investigation of the underlying mechanisms are important open problem for future work.
Gradient descent dynamics initially ignores permuted examples: A tempting explanation for the lack of memorization and slower training dynamics observed in the early layers is vanishing gradients (Glorot and Bengio 2010) in early compared to later layers. However, as shown in Fig. 5 (A), the gradients do not vanish in the early layers, and not small until well after memorization.
7

---

Published as a conference paper at ICLR 2021
Both (B)
Unpermuted
Permuled
(D)
Ratio of Credlents
Parmued (P 1/10
Dependent mee
logi [94]
17c 1.
Layer Wenretrr
Epochs
Epochs
Epochs
Epoch
Figure 5: Gradient analysis. (A-C): e norm of the gradient of the loss over the different layers and epochs of training for VGG16, computed on (A) all samples, (B) unpermuted samples only, and (C) permuted samples only. (D): Ratio of gradients. For the label dependent part (dep) of the gradient, the gradients averaged from the unpermuted samples much larger than the gradient from permuted samples (green line). For the independent part, the two components are roughly equal (Red).
Additionally, the unpermuted and permuted examples contribute similarly to the total gradient during the last phase of training shown in Fig. (B, C).
While the gradient magnitudes cannot explain our observations, more careful analysis the gradient descent dynamics reveals their origin; near initialization, the gradient from the permuted examples are minimal, with learning dominated by gradients from the unpermuted examples. For tractability, analyze gradient descent instead of SGD, consistent with other work (Lampinen and Ganguli 2018; Advani and Saxe 2017). For the large batch sizes and small learning rates in our experiments, our experimental findings
reflected in the much simpler case of gradient descent. Consider neural network N Nxx, 0) with input æ and parameters 0. We obtain the model PMr by applying a softmax, c.g. (M|cx, 0) Z Z=ze.N.(x.0) with Z - (z,00) for classes E 21,2.... P). When training the network to minimize the cross entropy cost function L with respect to the label distribution Pa(c|ær f), the following claims hold true:
Claim 1. The gradient of the cross entropy can be written as the sum of label dependent and label independent part: VoL VoLdep VoLind-
We write the cross entropy loss and substitute the softmax model PM above to obtain the following decomposition, where (-) denotes the average over samples of the entire dataset,
( Pr(ckT)NN(())
C C= Σ Pi(czz)) (o (PM(c) = Σ
log log +< (I)
+
The first operand above depends on the labels Pt. and we denote as Leep. For the second operand, the labels Pt were marginalized away, leaving only the partition function of the softmax, which is label independent (Cind). See the Appendix for more details.
For a ford-ror network, the label dependent and label independent components of the gradient can be understood in terms of shared features.
Claim 2.
For the final layer L the network, we can write the gradients of both components as the weighted average of the activations, centered by the average activation (see Appendix for full derivation):
dLdep Ow
(LL(a( (0/ (xx- (im)))))-1)) )b-11)
dLind
= (PM(al)s)) (x,) -5/- )/-1))
Owh
Here, a denotes the class and - (ar)) is the activation of unit 3 for the penultimate layer given input z;. By inspection of Eq. 2 we see that to contribute to training, individual examples must generate activation dy that differs significantly from the average activation 0₂ (dx)
Furthermore, the label dependent part of the gradient is enhanced when examples of the same class cause similar patterns of activations relative to the global mean, and therefore add constructively
8

---

Published as a conference paper at ICLR 2021
to the gradient during the ( operation. Intuitively, this means many examples of a class must share the same attribute that @ codes. Conversely, if different examples of a class activate / o/11 differently, the gradients add destructively. The situation for other layers is similar shown in the Appendix, with an added minor complexity due to the jacobian of subsequent layers.
Claim 3. Near initialization, the label dependent gradient vanishes for permuted examples, and the label independent term is small for both umpermated and permuted examples.
In standard initialization schemes (Glorot and Bengio 2010), the network output distribution Pas is near uniform at the start of training. implying 3ind = O as the two terms cancel in Eq. 2
The label dependent part deep has a qualitatively distinct behavior in the early epochs. as the training labels Pt. are not uniformly distributed. However, for the permuted examples additional simplification is possible. As Claim 2 shows, the label dependent gradient depends on the cross correlation between the centered features (2,) - /3 / and the labels (c(zz) Since permuted examples are constructed by random labeling, this cross correlation vanishes for large datasets. Thus, the dominant contribution to the learning comes from the correctly labeled examples. At random initialization, correctly labeled examples may still have shared features while the permuted examples do not. Previous work (Jacot al.) 2018 [ee et al., 2019) suggests that very wide networks remain near initialization where our analysis in Claim 3 holds, so we hypothesize that wider networks rely more on shared features, though this may not always be optimal for accuracy (Tippaa et al. 2019).
These claims are supported experimentally, as shown in Fig. 5 D, which compares the relative sizes of the gradients throughout training. For the unpermuted examples (Fig. 5)) 3Lind small compared to depp initially, and equilibriate only later. For permuted examples (Fig. [5B) due to the vanishing cross-correlation mentioned above, instead dCepp is comparatively smaller to dLind, indicating that these examples do not share features at initialization. Consistent with the claims above, for the label dependent part of the gradient (Fig. 5F), the contribution from the unpermuted examples are initially significantly larger than from the permuted examples.
DISCUSSION
We provide quantitative evidence, based on representational geometry. of when and where memo- rization occurs throughout deep neural networks, and the drivers for such structure. The findings go beyond the comparative results (Raghu al. 2017: Morcos et al., 20188] Komblith et al. (009) with direct probing of network instances, and provide empirical evidence and theoretical explanation to the qualitative findings of (Arpit et al., 2017). In addition, our observation linking double descent to increasing object manifold dimensionality highlights the value of geometric probes, recasting the phenomenon into that of feature representations, and opening a new direction for future study.
With standard initialization techniques, the approximate linearity of modern deep networks at the beginning of training (Lee et al. 2019) implies that the networks are sensitive to features that are approximately linear projections of the data. Since randomly labeled examples have few if any shared features, they make a minimal contribution to the gradient at the start of training. Therefore, the network learns primarily from the correct labels until the network leaves the linear regime. Our experimental results support this hypothesis; representations early in training are not impacted by permuted data, and such data have smaller gradient contribution than from the correctly labeled data. Our conclusions are consistent with prior reports on Coherent Gradients (Chatterjee] 2020).
We additionally observed several surprising phenomenon that warrant further investigation. The abrupt instead of gradual transition in memorization with layer depth (Fig. 3 is inconsistent with previous expectations (Morcos et al.) 2018). The lack of gradient magnitude differences between carly and late layers, even though they behave qualitatively different. The effect of double descent phenomenon on the measured manifold geometries and their competing underlying structural causes. The different generalization properties of early and later layers deep into training. These observations and future work are enabled by our more precise and rigorous investigation of the structure across layers of memorization in deep neural networks.

---

Published as a conference paper at ICLR 2021
REFERENCES
Madhu S Advani and Andrew M Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv preprint arXiv:/710.0360 2017.
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning Vollume 70, pages 242. JMLR. org, 2017.
Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz. Sgd learns over-parameterrized networks that provably generalize on linearly separable data. arXiv preprint arXiv:1710.10174, 2017.
Satrajit Chatterjee. Coherent gradients: An approach to understanding generalization in gradient descent-based optimization. ICLR, 2020.
Yoon Chung, Daniel D Lee, and Haim Sompolinsky, Linear readout of object manifolds. Physical Review 93(6):060301, 2016.
Yeo Chung, Daniel D Lee. and Haim Sompolinsky. Classification and geometry of general perceptual manifolds. Physical Review X. 803)031003,, 2018.
Uri Cohen, YueYeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. bioRxiv, page 644658, 2019.
Thomas M Cover. Geometrical and statistical properties systems of lincar inequalities with applications in pattern recognition. IEEE transactions on electronic computers, (3):326-334, 1965.
Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Li, Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pages 248 248-255. leee, 2009.
Elizabeth Gardner. The space of interactions in neural network models. Journal of physics A: Mathematical and general, 21(1):257, 1988.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 249-256, 2010.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference Machine Learning, pages 1225-1234, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pages 770-778, 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnl:: Convergence and generalization in
neural networks. In Advances in neural information processing syxtems, pages 8571-8580, 2018.
Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. arXiv preprint wwXiv:1955.000 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Sutskever, and Hinton. Imagenet classification with deep convolutional neural. Neural Information Processing Systems, pages 1-9, 2014.
Andrew K Lampinen and Surya Ganguli. An analytic theory generalization dynamics and transfer learning in deep linear networks. arXiv preprint arXiv: 10374, 2018.
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickse Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in neural information processing systems, pages 8570-8500088 8581, 2019.
Yuanzhi Li and Yingyu Liang. Learning overparameterriee neural networks via stochastic gradient descent on structured data. In Advances in Neural Information Processing Systems, pages 8157-8166, 2018.
Jonathan Mamou, Hang Le, Miguel A Del Rio, Cory Stephenson, Hanlin Tang, Yoon Kim, and YeoY Chung. Emergence of separable manifolds in deep language representations. arXiv preprint 2xxx:2006.00099 2020.
10

---

Published as a conference paper at ICLR 2021
Leland McInnes, John Heall. Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29), 2018.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems, pages 5727-5736, 2018.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. arXiv preprint urXiv:1912.0229., 2019.
Behnam Neyshabur, Ryota Tomioka, and Nathan Stebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614 2014.
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensi- tivity and generalization in neural networks: an empirical study. In International Conference on Learning Representations, 2018. URLhttps::/ /ppnrreveeeeeeeeeeeeeeee net forum? di-rmmm222-C
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, pages 6076 000-6085,22 2017.
David Rolnick, Andreas Veit, Serge Belongic, and Nir Shavit, Deep learning is robust to massive label noisc. arXiv preprint arXiv:1705 10694, 2017.
Andrew M Saxe, James I. McClelland, and Surya Ganguli. Exact solutions the nonlinear dynamies of learning in deep lincar neural networks. arXiv preprint 1322::33/. 2013.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint urXii:1149..1. 1556, 2014.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):8822 2878, 2018.
Cory Stephenson, Jenelle Feather, Suchismita Padhy, Oguz Elibol, Hanlin Tang, Josh MeDermott, and Sue Yoon Chung. Untangling in invariant speech recognition. In Advances in Neural Information Processing Systems, pages 14368-14378, 2019.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may at odds with accuracy. In International Conference on Learning Representations, number 2019, 2019.
Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun Hc, and John Hopcroft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. In Advances Neural Information Processing Systems, pages 3584-9598 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 161iv:161 (tfi::661..35333 1.05300, 2016.
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? arXiv preprint 1902x:190..0199 2019.
A SUPPLEMENTAL INFORMATION
A.1 MEAN FIELD THEORETIC MANIFOLD ANALYSIS (MFTMA)
Here provide summary of replica- based Feeld Theoretic Manifold Analysis (MFTMA) used in this paper. First introduced in (Chung al. (2018) 3 this framework has been used to analyze internal representations of deep networks for visual (Cohen et al. 2019). speech (Stephenson et al.) 2019) and natural language tasks (Mamou et al. 2020). MFTMA framework is a generalization of statistical mechanical theory of perceptron capacity for discrete points Garddner 1988) to manifolds. Traditional point perceptron capacity measures the maximum number of points in general position that can lincarly separated given the feature dimension and the ensemble of random dichotomy labels for points. Whereas the MFTMA framework measures the "manifold" capacity, defined as the maximum number of classes that can be linearly separated given the ensemble of random dichotomy labels for classes. MFTMA framework allows for the characterization of geometric properties of object manifolds connection their linear separability, it has been shown that the object manifolds' properties can be used to predict the manifold classification capacity (Chung et al. 2018 Cohen et al. 2019 (Stephenson et al. [0019] Mamou et al. 2020). As the framework formally connects the representational geometric properties and the manifold classification capacity, the measures from this framework are particularly useful in understanding how information content about object categories are embedded in the structure of the internal representations from the end-to-end trained systems (such models from our paper).
11

---

Published as a conference paper at ICLR 2021
A.1.1 MANIFOLD CAPACITY
In system where P object manifolds are embedded in N-dimensional ambient feature space, load is defined P/N. Given a fixed feature dimension .N. large/small load implies that many/few object manifolds embedded in the feature dimension. Consider linear classification problem where positive and negative labels are assigned randomly to /' object manifolds (note that all the points within the same object class share the same label), and the problem is to find a lincarly classifying hyperplane for these random manifold dichotomies. As each of the P manifolds are allowed positiveenegative labels, there are 2" manifold dichotomies. Manifold capacity is defined as the critical load a. P/N such that above most of the dichotomies have lincarly separating solution, and below it, most the dichotomies do not have linearly separating solution. Intuitively, a system with a large manifold capacity has object manifolds that are well separated in the feature space, and a system with small manifold capacity has object manifolds that are tangled in the feature space.
Interpretation Manifold capacity has multiple useful interpretations. First, the manifold capacity is defined as the critical load for linear classification task, it captures the lincar separability of object manifolds. Second, the manifold capacity is defined as the maximun number of object manifolds that can be packed in the feature space such that they are linearly separable, it has a meaning of how many object manifolds can be "stored" in a given representation such that they are linearly separable. Third, as linear classifier is often thought of as linear decoder, the manifold capacity captures the amount of linearly decodable object information per feature dimension embedded in the distributed representation.
Mean-Field-TThee (MFT) Manifold Capacity Manifold capacity, 0.M = P/N, can be estimated using the replica mean field formalism with the framework introduced by (Chung et al. [2018] Cohen et al.)
2019). As mentioned in the main text, ar is estimated as FMMT,, for MFT manifold capacity. from the statistics of anchor points, representative point for the points within object manifold that contributes linear classification solution.
The general form of the MFT manifold capacity, exact in the thermodynamic limit, has been shown (Chung ct al. 2018 Cohen et al. 2019) to be:
to +
1 1
||($|]]
T,to
where (.. a mean over random D- and dimensional Gaussian vectors f, to whose components are i.i.d. normally distributed f; (),
This framework introduces the notion of anchor points, 5, uniquely given by each ū, to. representing the variability introduced by all other object manifolds, in their arbitrary orientations. å represents weighted sum of support vectors contributing to the linearly separating hyperplane in KKT (Karush-Kuhn, Marush-Kuhh-TTucker Tuccer) interpretation.
A.1.2 MANIFOLD GEOMETRIC MEASURES
The statistics of these anchor points play a key role in estimating the manifold's geometric properties, as they are defined as: Rai ((T)) and DM = <(F where = //||i]| is a unit vector in the direction
of 5, and T = (f, to), which is a combined coordinate for manifold's embedded space, E, and manifold's 's center direction to-
The manifold dimension measures the dimensionality of the projection of oo its unique anchor point s, capturing the dimensionality of the regions the manifolds playing the role of support vectors. other words, the manifold dimension is the dimensionality of the object manifolds realized by the linearly separating hyperplane.
If the object manifold centers are in random locations and orientations, the geometric properties predict the MFT manifold capacity (Chung al. 2018). by CMPT RS craall (RM, Dx) where, epdl D)
RRVD Dto (RVDDtt)) as defined (Chung et al. 2016). x1
In real data, the manifolds have various correlations, hence the above formalism has been applied to the data projected into the null spaces of manifold centers, similar to the method proposed by (Cohen al. 2019)
The validity of this method shown various literature CChen et al. 2019 Stephenson al. 2019 Mamou jet al. 2020), where the good match between the MFT manifold capacity and a ground truth (empirical) capacity has been demonstrated (computed using bisection search on the critical number of feature dimensions required in order to reach roughly 50% chance of having linearly separable solutions given fixed number of manifolds and their geometries). Hence, the manifold capacity am refers the mean-field approximation of the manifold
12

---

Published as a conference paper at ICLR 2021
capacity, aMFF.. For details the theoretical derivations and interpretations for the mean-field theoretic algorithm, see (Cohen et al. <ChnnC et al. 2018
A.2 DERIVATION OF CLAIMS
this section, we provide detailed derivation of three claims made the main text. deep neural network with a P-class softmax applicd to the output has an output distribution of the form
(clz) = N No(2)
(3)
With = Ei. Me(t)) Here, (c((r) and Ne(a) is the et component of the network's output given the input æ. Given training labels Pr (c(aaa) (frequently one-hot, but not necessarily so) for each input o, training set D, the network is trained to minimize the cross entropy loss
interpreted as the prohability the model assigns to class e given
P Σ Pr(c)x) log (Phr(cle))
C C
(4)
Here, () + Ei is the uniform average the samples in the dataset.
A.2.1 DERIVATION OF CLAIM
Due to the normalized form Par (c|2r), the gradient the cross entropy decouples into two parts:
F Σ Epaldloraac)) + log (2)
(1)
C ==
(5)
Note that because Z independent the class label, the labels Pr. (clwr sum away second term. Taking the gradient with respect to the parameters 0 gives
VoL-
Σ PL(c2)VoN.(())
+ (vo
(6)
Noting that the first term on the right hand side depends on the labels PL (c((r) while the second term docs not, wc can write
V&L VoLdep VoLind
Denoting that gradient of cross entropy respect the model parameters can written as sum of a label dependent part Vo Caep that depends on Pt (c/r), and a label independent part VoLind that is completely independent of (clx). The specific form of these two terms is dependent upon the parameterization of the model.
A.2.2 DERIVATION OF CLAIM
Continuing from section A.2.I we make use of convenient identity that follows from the form of the softmax defined in Eq. 3
Vo log (3)==" =2
Voe Nala)
VoNo(x)
(8)
Am
Using this, the gradient of the cross entropy with respect to the network parameters 0 can then be written as
VoL Σ P.(ck..VoNN..)))
Σ
Parlcle:)
After relabling summation index the network takes on a feedforward structure (omitting biases for clarity) we have
Ne(x) 1 62-1 (WL-1... (W*x))
(10)
13

---

Published as a conference paper at ICLR 2021
Where Wt and o' is the weight matrix and nonlinearity respectively at layer I. Via straightforward application of the chain rule, the partial derivative of Ne(x) w.r.t the weight wass an element of the weight matrix at layer I, can be written as
@Nc(z) (2) Ow
( (x) gcos(2)
(11)
Here, J'h (1) is the Jacobian of the network from layer to the final layer, and is a function of x due to nonlinearity. Jea (a) can interpreted as sensitivity Ne(x) due changes in the pre-activation feature at layer 1. The quantity & (xx) is element of the activation vector at layer I - 1, and can he interpreted the 8th feature learned by the network at layer - 1. The combination Peas (2)) encodes how sensitive the network output at is small change in was given the input
The partial derivative of the loss then
Oc
P Σ (c(ce (ole)r)) +
Σ Paelelm))
(12)
aw
Further insight can gained by inserting a factor of Doap Leus with
Deng Σ (z)) (x)
(13)
Which is the average partial derivative Nx(2)) over all classes and all datapoints in the dataset. Further using the normalization of the label and model output distributions, we can write Jea. = = Pc(ccx))Deaa and Feap Dem Par Par (c(a,))) Then write
Oc
aLdep
aLind
(14)
Ow
Ow-.p
Ow-ap
0Cacr and the label independent part of the Where we introduced the label dependent part the gradient ow
gradient 0eind as introduced in section
A.2.1
Using deaps these can he written as
öLdez
Pe(cuus) (Sowa(x) - geme)
Oway
(15)
DLind
P
(ac(eass) (dens(x) Feass)
Ow
An important property for these two terms is that in this form, we have the special case of 0Lasz = if the The ag
training label distribution (c(c)r is uniform, and similarly BLinu = 0 if the network's 's output distribution Buly
(cl(r) is uniform. This follows from the definition of Hears as the average over points in the dataset with uniform measure over classes.
For the special case of layer L, the final layer of the network, we have Jl (x) = 6.a corresponding to linear activation function at the final layer. The two components of the gradient then simplify to
dLdep
(2((a)))) (0% (z.) - 0/ 1)
Owhy
(16)
0Lind
(Par(alas) (x.) -1:))-111)) 1)=
Ow!e
With 2/
A.2.3 DERIVATION OF CLAIM 3
For small weight initialization, the network can be approximately linearized around zero weight Using the fact that the label independent part vanishes when the network outputs uniform distribution as shown section
By is the Kronecker delta Forr ReL.U networks, this is slightly more complex the network is differentiable with zero weights.
14

---

Published as a conference paper at ICLR 2021
A.2.2 that this term the gradient scales with the size of the weights the (1)) term in the expansion about zero weights is zero.
iLind Ow
O (Iwl)
(17)
sinct
Here. the notation lenaa denotes that the gradient is evaluated with the initial weights. Hence, near initialization, the label independent part of the gradient is small regardless of if it computed from unpermuted permuted examples.
To gain insight into how the label dependent part behaves at initialization, we can write the gradient to ([ in the size of the weights. In this case, the ((1) term is also zero as Denne const Fon at zero weights for all c, and hence V VoLdep for zero weights. The leading order is then the O(hel) term:
@Ne(a) Er ap
J'n Σ M + O(lul*)
(18)
Here, the Jacobian J'a no longer function up to layer I 1 has been linearized to M, 2y. The label dependent part of the gradient then has simpler form
the dimensionality the input data æ. and network
SLdep Ow
P Σ
Pu(clas)
Scs Σ M y=1
Heas
(19)
where now, Lens
Den y=== S'a M (or) By linearity of the expectations, this is
dLdep Owtp
ΣΣ
(ea (Pu(elas))n) $eas)
(20)
cm1
Without loss of generality, for simplicity will assume centered dataset, which has (2-)) = 0. This does not affect the following arguments, but allows for simpler notation. With this assumption, we have
Shoep Ow o
Jea M (Pr(c))))))
(21)
For many datasets, this is nonzero, as the data has some correlation with labels, and so ( (elzzs))))))) > For datasets consisting of a mix of unpermuted and permuted examples, the cross correlation can be split into contributions from each part:
[Dul (P)())s))a)) [D]
+ Del (P((c))))))) [
(22)
Here, [D] is the size of the dataset, Du. Dr denotes the set of unpermuted, permuted examples, and [Da], Dpp denotes the number of unpermuted, permuted examples. Note have [D.] + ID,1 [DD..
For permuted examples, the label is assigned uniformly randomly and independent the data. Assuming the labels are one-hot as our experiment, the cross correlation (Pr. (clwx))))=) is simply the average of wn over examples given specific random label. By the central limit theorem, we have ( (clzz)))
R
O (v P (prl) for P classes. With a fraction of permuted examples, we have [Ppl = P
<|2] For the label
dependent part of the gradient, this gives
dLdep
Sea Mayy' (11 +o
eP D
(23)
Juh G3
P c=1
As the size of the datasct grows, |D] + oo, the contribution from permuted examples vanishes.
Further splitting the label dependent gradient into contributions from unpermuted and permuted examples
dLdep Ow
(24)
Ow
Provided < 1, we obtain an ordering relationship for the relative size of label dependent part the gradient computed on unpermuted and permuted examples for large datascts:
DCany copperm
ac
2
(25)
Owh.
Jw
D
1nitl12
15

---

Published as a conference paper at ICLR 2021
log( ( de
log (1
10
Epochs
0.0
101
0.0
10
-2.5
Con
conv20
curv20
conue
carvzo
conv20
co
corv20
conv2 3
conzo canvzo 11
Linear 12
Liner
Cond2o
tonuto
carv20
tonuto
carvzo
con220 c
comvz
Convzo 10-
Con 11
Lincor 12
13
Unwer 14
in log(
(2lomll)) Ila))
[IVe
Epachs
2.5
0.0
10
0.0
10a
-2.
ConvED
Conv
Con Gowii
Con
ConvD
CoweD
ConeD
CownD
conc2o
convzo 11
Liner
Unarr
Liver 14
Comeoo
Gown
Convi
Con 10
Con-20 1t
Unvari 12
Liver 13
Unver 14
Figure A.I: Top: e norm of the label dependent gradient calculated for unpermuted examples (IV Cumperm (12)) and permuted examples ([) dep ||2- Note that the label independent gradient dep
has a very small norm in the early epochs for permuted examples. Bottom: e² norm of the label independent gradient calculated for unpermuted examples (IV Lnn [[ (|2) and permuted examples V Ctnd""] ||2. The trends are overall similar between permuted and unpermuted examples, though somewhat higher magnitude for unpermuted examples in the later epochs.
Where the degree by which the left hand side is greater than the right hand side is determined by the cross covariance matrix (c(aa))) )
In summary, we see in Eq. 17 that the label independent part of the gradient is small early in training given small weight initialization. This part the gradient behaves similarly for permuted and unpermuted examples. In Eq.
25 we see that the contribution to the label dependent part of the gradient from permuted examples vanishes for large datasets, while the contribution from unpermuted examples does not provided the cross correlation between input features and labels is nonzero. This suggests that with small weight initialization, the gradient descent dynamics initially ignores the labels of permuted examples.
Figure A. shows breakdown how the two components the gradient computed on both unpermuted and permuted examples evolve over the course of training for the different layers of the VGG16 model trained on CIFAR-100. We see that the label dependent part hehaves qualitatively differently for the unpermuted examples than for the permuted examples, the permuted examples give close to zero contribution early in training in agreement with Eq. 23 The label independent part of the gradient shows similar trends between unpermuted and permuted examples, though the final epochs, the unpermuted examples have a slightly larger label independent gradient indicating slightly greater model confidence on these examples. As the label dependent and label independent parts of the gradient have differing signs, they compete with cach other and cancel when the loss is minimized, but are not independently zero and in fact grow during training. The slightly larger label independent gradient for unpermuted examples balanced by a corresponding slightly larger label dependent gradient at the end of training.
A.3 A SHORT COMMENT ON THE RANDOMIZATION OF LABELS
In our experiments, we define the permuted examples as the set of all examples which had their labels randomly permuted. Due to random chance, some fraction of the lahel permutations will result in the randomly assigned lahel being identical to the training (or 'correct') label. This choice winds up being important, the inclusion of this possibility implies that on average, the input data contains no information about the correct label, so they must be memorized during training. If the set of permuted examples were taken to be the set of examples that have labels differing from their training labels. this would no longer be true. For example, given a dataset with classes o, b, if sceing the permuted label for datapoint x, implics that x, definitely does not actually belong to class a, features useful for predicting the permuted label a from 2, may also be useful for predicting that x; belongs to or and vice versa. In this case, memorization is not the only way to solve the problem.
A.4 UMAP VISUALIZATION FOR EARLIER LAYERS
In the main text, we show an example of UMAP visualization for the final layer features in VGG16 trained on the CIFAR-100 dataset. This type of visualization is also possible in the carly layers, although the lower separability results noisier visualization. Nevertheless, as shown in Fig. SI for layer 15 of VGG16, some
16

---

Published as a conference paper at ICLR 2021
Unpermuted
Restored
Test
Epach 0
Unpermuted
Permated
Restored
Test
Epach best
Unpermuted
Permuted
Restored
Epach last
Figure A.2: UMAP visualization of the layer 15 features in VGG16 with f = 50% initialization. best epoch, and final epoch. At the best and last epoch, unpermuted, restored. and test manifolds show similar structure, implying that the network has learned the features shared between the two datasets.
information can be gleaned using this method. We see that in this intermediate layer, unpermuted, restored, and test examples appear behave similarly even after the final epoch of training. The memorization observed for the permuted examples the final layers the network does not appear here, and instead restored examples show some separability. This is consistent with the quantitative trends shown in the MGMs the main text, where we found that the early and intermediate layers do not show evidence of memorization and instead learn features that group examples according to their true label.
A.S EXPERIMENT WITH SYNTHETIC DATA
As an early experiment, we created a synthetic manifold dataset test our understanding and also examine non-CNN architectures such multi-layer perceptrons. This dataset consists of spherical manifolds, allowing finer degree of control over the manifold geometry and initial separability of the data. We constrained the centers the spheres to diametrically opposed pairs which are orthogonal all other pairs, ensuring that this data lincarly separable if the sphere radius is less than r 1. Otherwise, the spheres overlap and lincar separation is impossible. For the results presented in the main text, we used values of r 5, d = 30, and D = 1024, well into the linearly inseparable regime better mimic complex datasets.
An example of a pair of classes used in the experiments shown in the main Note that the constraint that the sphere centers to be diametrically opposed pairs implics that only the shared direction paraliel to the centers contains lincarly decodable information about the class ID, as the data from each class is mean zero along all other directions. Figure A.3) demonstrates this for classes 0, 1. showing partial separability along feature dimension O. and i.i.d. data for other features. The remaining features similar, and other class pairs have a similar structure.
be seen in Fig. A.3 (Left).
We trained a 5 layer feed- forward network on this datasct with 1024-dimensional linear layers interspersed with ReLU nonlincarities. This network has zo 5.2 10 parameters, over two orders of magnitude larger than the number of training points, such that complete memorization is easy. We see similar trends to those shown in the main text in train and test accuracy, as well as the accuracy unpermuted, permuted and restored examples as shown in Fig. A.3 (Right).
Figure A.4 (Top) shows the best epoch results for the MGMs on the feedforward model trained on synthetic data. The trends here are similar, in that we see evidence for increasing separability of unpermuted, restored, and test manifolds across the layers of the network, but very little evidence of untangling of permuted manifolds this point. Again, early layer features for unpermuted, restored, and test manifolds are all similar, with differences
17

---

Published as a conference paper at ICLR 2021
Figure A.3: Left: Classes and from the synthetic dataset. Note only subset of features useful for separation, the rest of the features contain no information about class 0 vs. 1. Right: Accuracy trends for five layer feedforward model trained this dataset with 0.5.
Unperrured
25
2
31
1.5
3.20
23-
3.35
3n
23
0
Figure A.4: A: Manifold analysis on the feedforward model trained with synthetic data at best epoch. B: Manifold analysis on the feedforward model trained with synthetic data at the last epoch.
becoming more pronounced in the later layers. Note that due to how the centers are initialized for the data, the center correlation Peenter is
at the input.
Figure A.4 (Bottom) shows the results the MGMs the feedforward model trained on synthetic data the final epoch of training. In contrast to Fig. A.4 we now see rising separability of permuted manifolds after layer 2. and restored manifolds initially hecome more separable, hefore the effect saturates in the final three layers. Test manifolds now show very little separability, indicating the memorization of the permuted examples has significantly impacted the generalization performance of the network.
A.6 МFTмА RESULTS FOR ADDITIONAL MODELS AND DATASETS
0.5
Here, we include the results the experiments described the main text on the remaining models evaluated.
Figure A.5JA) A) shows the results for the MGMs on AlexNet trained on CIFAR- 1-0 at the best epoch of training. Again, unpermuted, restored. and test manifolds behave nearly identically and become increasingly separable deeper into the network, with the exception of the final two layers 7 and 8. This decrease the final layers perhaps part of reason why this model performs worse than VGG16 and ResNet18 on CIFAR-100 partially explained by the large label dependent part of the gradient on permuted examples shown Fig. A. in the final two layers, which indicates little preference for learning correctly labeled examples in these last two layers.
Figure A.S shows the results for the MGMs AlexNet trained CIFAR-100 the final epach of training. Now, permuted manifolds begin to become separable the final three layers the model, and 7. Though, perhaps owing to the smaller model capacity of AlexNet compared to VGG16, this effect is smaller than we observed in other models. This is also reflected in the smaller difference between unpermuted and restored manifolds in the final layer. We note that this model also took many more epochs to memorize the data than VGG16 ResNet18 shown in the training
in the main text.
18

---

Published as a conference paper at ICLR 2021
Best Epoch
Unpermated
3
2
639
54
(8) Last Epoch
W
124
9/5
4.07
1.00
6.0
289
315
Figure A.5: A: Manifold analysis on AlexNet trained on CIFAR-100 at best epoch. B: Manifold analysis on AlexNet trained on CIFAR-100 100 the last epoch. Error bars indicate 95% confidence intervals computed over 10 training runs with different random seeds.
(A) Best Epoch
Inperrated Nemutse
0.48
0.00
0.99
0.50
Epoch
Figure A.6: A: Manifold analysis on ResNet18 trained on CIFAR- 100 at best epoch. B: Manifold analysis ResNet18 trained on CIFAR-100 at the last epoch. Error bars indicate 95% confidence intervals computed over 10 training runs with different random seeds.
Figure A.6 A) shows results for the MGMs on ResNet18 trained on CIFAR 100. Here, we only show the metrics computed on the 2D Conv. + ReLU layer in the residual block and omit the analysis on the skip connections for fair comparison with other models. Again, we see that at the best epoch of training, unpermuted. restored and test manifolds all become increasingly separable deeper into the network, with restored and test manifolds behaving similarly. Permuted manifolds exhibit low separability at every layer, indicating that these examples have yet to be leamned at any layer.
FigureA..6 B) shows the MGMs computed for ResNet18 trained on CIFAR- 100 at the final epoch of training. Again, permuted manifolds show increasing separability in the final two layers (labeled 7 and 8 here) of the model, while the manifold capacity remains at the lower bound all earlier layers. Unpermuted, restored, and test manifolds behave identically up to layer numerically very similar to the best epoch behavior, indicating little memorization in the early layers. In the final two layers, unpermuted manifolds show a more favorable geometry than restored or test manifolds.
Figure A.7 A) shows the MGMs computed on ResNet18 trained on the Tiny ImageNet dataset at the best epoch of training. Interestingly, we note that test manifolds exhibit more favorable geometry for classification than unpermuted, restored, or permuted manifolds many layers of this model. This is likely due to test set consisting of easier examples than the training set on average. Aside from this, the observed trends are similar to the ones observed in other models and datasets in that unpermuted, restored, and test manifolds show numerically similar trends, and increase in manifold capacity (and decrease in manifold radius and dimension) across the layers of the network, while the permuted manifolds remain the manifold capacity lower hound.
19

---

Published as a conference paper at ICLR 2021
Best Epoch
Jipertuled
3.25
3.00
6-1M
1.2-
6:00
3.20
(B) Last Epoch
0,10
1.0
100
140
2
120
Destr Ge033
Figure A.7: A: Manifold analysis on R trained on CIFAR-100 at best epoch. B: Manifold analysis on ResNet18 trained on CIFAR-100 at the last epoch. Error bars indicate 95% confidence intervals for the MGMs.
Finally, Fig. AA.7 B) shows the MGMs computed ResNet18 trained on the Tiny ImageNet dataset the final epoch training Again see that permuted manifolds show increased separability in the final two layers the network (7 and 8) while restored and test manifolds show an initially increasing manifold capacity with layers, followed by saturation and decrease in layers 7 and 8 coinciding with the rise in permuted manifold capacity. Up to layer 6. the trends for unpermuted. restored, and test manifolds are numerically very similar to those seen at the best epoch of training. indicating little memorization has occurred in these layers. This can also be seen from the permuted manifold capacity which remains near the lower bound these layers.
A.7 MANIFOLD CAPACITY FOR CIFAR- 100 MODELS VARYING
The trends presented in the main text hold for wide range of permutation fractions Figure A.8 shows summary the manifold capacity results for three models trained CIFAR- 100 for between and 1. Figure|A.8g A) shows the manifold capacity measured at the first layer for the three models. As discussed in the main text, in the early layers of the models we see no significant difference hetween unpermuted, restored, and test manifolds, while the separability of permuted manifolds remains near the lower bound. These trends stable across permutation fractions, though we see slight increase in manifold capacity with e, but this trend is within error bars for most models. Continuing to the middle layers shown Figure B). there still no difference between unpermuted, restored, and test manifolds, and the separability of permuted manifolds remains near the lower bound. We do see now slight decrease in manifold capacity for larger values of c.
In the last layers, the situation is quite different A.8 C) shows for the final layer in each model. Here, unpermuted manifolds show higher separability than restored ResNet18, permuted manifolds show significant separability indicating memorization. manifold capacity measured on test examples decreases significantly with increasing permutation fraction as the memorization increasingly degrades the model performance.
and for the two larger models, VGG16 and note also that
A.8 ADDITIONAL LAYER REWINDING PLOTS
We also carried out the layer rewinding experiment on AlexNet and ResNet18 trained on CIFAR- 100, and found similar trends as seen on VGG16 shown in the main text. Figures A.9 A. A.10 the results averaged over several training runs. For the case of ResNet18, we show every third convolutional layer, but include the final two layers to show the effect. We see that rewinding the last layer in the case of AlexNet corrects significant fraction the decrease test accuracy, while in the case of deeper ResNet18, final two layers show an effect as in VGG16. In the case of ResNet18, the final layer obtains its peak value at epoch 0, a result corresponding to the behavior observed in (Zhang et al. 2019).
Finally, include Fig. A.11 as updated version of the plot in the main text for VGG16 including additional nuns with different random seeds. We find the trends in the main text robust across di different seeds.
20

---

Published as a conference paper at ICLR 2021
(B) First layer
AlexNet
VGG16
ResNer18
504
0.22
9.0
0.42
A60
0642
001
608
C4
04
(B) Middle layer
AlexNet
VGG16
ResNet11
1.00
0888
6.6
6308
2300
Permutation fraction
Permutation fraction
Permutation fraction
Last layer
VGG16
ResNet11
612
038
0078
0644
0644
636
038
05
Permutation fraction (c)
Permutation fraction (c)
Permutation fraction le)
Figure A.8: A Manifold capacity for the first convolutional layer of AlexNet, VGG16, and ResNet18 computed for varying c. B: Manifold capacity for the middle convolutional layer. C: Manifold capacity for the final layer. Error bars indicate 95% confidence intervals computed over 10 training runs with different seeds.
5200
430
d
Rewind Epoch
Rewind Epoch
Figure A.9: A: Accuracy on the train set for AlexNet trained for 1000 epochs with an individual layer's parameters rolled back to earlier epoch (rewind epoch). B: Corresponding results measured on the test set. Horizontal line indicates the peak generalization performance obtained by early stopping. Shaded region indicates the 95% confidence interval obtained over 10 training runs with different random seeds.
A. ADDITIONAL GRADIENT PLOTS
Here, we show the results of the gradient analysis on AlexNet trained on CIFAR-100 100, ResNet18 trained on CIFAR-100, and ResNet18 trained on Tiny ImageNet. We find that these results are consistent with the results shown in the main VGG16 trained on CIFAR-100.
21

---

Published as a conference paper at ICLR 2021
(B)
G1100 13
Train
44
O1r82 21
Ored 9
10
15
Rew Ind Epoch
Rewind Epoch
Figure A. 10: A: Accuracy on the train set for ResNet18 trained for 1000 epochs with an individual layer's parameters rolled back to an earlier epoch (rewind epoch). Corresponding results measured on the test set. Horizontal line indicates the peak generalization performance obtained by early stopping. Shaded region indicates the 95% confidence interval obtained over training runs with different random seeds.
1.-
Rew Ind Epach
Rewind Epoch
Figure A.11: A: Accuracy on the train set for VGG16 trained for 1000 epochs with an individual layer's parameters rolled back an earlier epoch (rewind epoch). B: Corresponding results measured on the test set. Horizontal line indicates the peak generalization performance obtained by early stopping. Shaded region indicates the 95% confidence interval obtained over 3 training runs with different random seeds.
Figure A.12 shows norm of the total gradient for AlexNet trained on CIFAR-100 computed on examples as well as the unpermuted and permuted examples independently. No clear differences sizes between these gradients is observed, and we do not see vanishing gradients in the early layers in the later epochs of training.
Figure A.12 shows the relative size (log ratios) of the label dependent and label independent components of the discussed the main
gradients on unpermuted and permuted examples for AlexNet trained CIFAR-100 text and Sec. [.22] the early epochs, the label dependent contribution from unpermuted examples dominates the label dependent part from the permuted examples. However, in this case we do note that in the final two layers (4 and here), the permuted examples do contribute large label dependent part, which is perhaps related the somewhat decreased manifold capacity in this model shown in Fig. A.5 and Fig. A.S
Figure A.13 shows the norm of the total gradient for ResNet18 trained CIFAR-I00 computed on examples as well as the unpermuted and permuted examples independently. Again, we do not see evidence for vanishing gradients in the carly layers or late epochs. We do note that the gradient on unpermuted examples is slightly larger than the gradient from permuted examples, although the difference is relatively small.
Figure A.13 shows the relative size (log ratios) of the label dependent and label independent components of the gradients on unpermuted permuted examples for ResNet1 trained on CIFAR- The label dependent part of the gradient for unpermuted examples again dominates the label dependent part from permuted samples in the early epochs (up to epoch 150 after which they are of comparable size). This is true for all layers of the network.
Figure A.14 shows the norm of the total gradient for ResNet18 trained on Tiny ImageNet computed on all examples well the unpermuted and permuted examples independently. The gradient trends here are somewhat different than observed the CIFAR- 100 trained networks, although qualitatively similar. Specifically, we do not see evidence for vanishing gradients in the early layers or late epochs. Again, the gradient on unpermuted examples is slightly larger than the gradient from permuted examples, although the difference is relatively small. The largest difference from the CIFAR-100 trained networks is that the gradient in the first. layer's is much larger than the later layers.
22

---

Published as a conference paper at ICLR 2021
logl/
logi
1a/ 1/114 ls)
log [| 1/ 1
10"
Epoche
10a
0
10*
Epochs
-2
-1
-1
10P
Egochs
0
(F)
(G)
10*
log( [0 12/1//
log! |V4
/// /
-2
10a
10"
[og][V[stmla [s)
10P
Epochs
10-
10A
Epochs
10a
10a
10a
Figure A.12: (A-C): l norm of the gradient of the loss over the different layers and epochs of training for AlexNet, computed on (A) all samples, (B) unpermuted samples only, and (C) permuted samples only. (D-G): Log ratio of gradients. D: For unpermuted samples, green indicates that the label dependent (dep) part is larger than the label independent part (ind). E: For permuted examples, purple indicates that dep < ind. F: For the dependent part, those computed on the unpermuted examples dominate that of permuted examples. G: For the independent part, the two components are roughly equal.
(A)
(D)
loo
|V / b/
log [IV [opp
b/
10a
Epachs
10a
10a
Egechs
10-
104
1
-2
0
(B)
10a
Epochs
10a
log( [V2eroenn
log( [Vcc
la/ caem
-2
102
109
(C)
| (a]
Epochs
10a
Epochs
10a
-1 10a
10a
Figure A.13: (A-C): l norm of the gradient of the loss over the different layers and epochs of training for ResNet18 trained on CIFAR-100, computed on (A) all samples, (B) unpermuted samples only, and (C) permuted samples only. (D-G): Log ratio of gradients. D: For unpermuted samples, green indicates that the label dependent (dep) part is larger than the label independent part (ind). For permuted examples, purple indicates that dep < ind. F: For the dependent part. those computed on the unpermuted examples dominate that permuted examples. G: For the independent part, the two components roughly equal.
shows shows relative size (log ratios) of the label dependent and label independent components of the gradients on unpermuted and permuted examples for ResNet18 trained on Tiny ImageNet. Again, the label dependent term from the unpermuted examples is much larger in magnitude than the label dependent term from the permuted examples in the early epochs. As in the case of AlexNet, this effect is smaller in the later layers, as the permuted examples produce non-negligible label dependent gradient in these layers.
Figure A.14
23

---

Published as a conference paper at ICLR 2021
[
11/144 ls)
log [| 1/ 1
10"
Epoche
10a
0
Epocts
10
101
-2
10a
11
-1
10P
Epochs
(F)
(G)
10a
logí [0
log! 1V b// b
-2
10"
10"
107
Epochs
10-
Epochs
107
10a
10a
10a
Figure 1414: (A-C): l norm of the gradient of the loss over the different layers and epochs of training for ResNet18 trained Tiny ImageNet, computed (A) all samples, (B) unpermuted samples only, and (C) permuted samples only. (D-G): Log ratio of gradients. D: For unpermuted samples, green indicates that the label dependent (dep) part is larger than the label independent part (ind). E: For permuted examples, purple indicates that dep < ind. F: For the dependent part, those computed on the unpermuted examples dominate that of permuted examples. G: For the independent part, the two components are roughly equal.
A.10 MODEL ARCHITECTURES AND TRAINING DETAILS
All networks were trained using the Adam optimizer with learning rate of x 10 and hatch size of 1, 0244 This choice of hyperparameters networks, but rather to provide consistent training methodology across all network architectures and datasets. We note that at high fractions of label permutation (not shown in this work) typical training hyperparameters may not work well for training, and we found that this combination of decreased learning rate and increased batch size works across all fractions of label permutation the cost of decreasing generalization performance somewhat. Here, we give complete specification for the network architectures used. Note that some cases (particularly for the ResNets where we omit analysis of skip connections) the layer numbers given here may differ from those in the previously shown results. Additionally, when evaluating the MGMs, we evaluate using the representations following nonlinearity.
made
optimize for the generalization performance of any of the
24

---

Published as a conference paper at ICLR 2021
Table 2: Modified Alex
Table 1: FeedForward network
Layer
Type
Size
Layer
Type
Size
0
Input
3 32 x 32
0
Input
1024 features
2D Conv, + ReLU
64 11 filters
1
Linear ReLU 1024 x 1024
2
2D MaxPool
2 2 stride 2
2
Linear ReLU
1024 x 1024
3
2D Cony, + ReLU 192
filters
4
2D MaxPool
2 2 stride
3
Linear + ReLU 1024 x 1024
5
2D Cony. + ReLU
384 3 x 3 filters
4
Linear ReLU 1024 x 1024
6
2D Conv. ReLU
256
filters
5
Linear ReLU 1024 x 1024
7
2D Cony, + ReI.U
1024 3 x 3 filters
6
Linear
1024 x 100
8
2D MaxPool
x stride
9
Lincar
1024 100
Table 4: ResNet18 for CIFAR- 100
Table 3: VGG16 for CIFAR-100
Layer
Type
Size
Layer
Type
Size
2D MaxPool
2D Conv. RotchNorm Rel II
2D Coay. BatchNomm ReLU
2D Conv. BatchNomm ReLU
2D MaxPool
Inpur
2D Comv. BotchNomm ReLU
BotchNomm ReL.U
BonchNonm RtL.U
RonchNorm Rel 1::
3x 52 32
66.3 3 filtors
256
2 2 stride 2
6 3 litters
2 scids
128 3 filters
128 3 3 filters
4
2
3
0 Isput
1
2D Cook
20 Com
2D Coox
2D Com
20 Com BaichNorm RRLUU
Com BaichNorm
Com BaicNorm Ral.U
Com BaichNorm RRLU
Com BaichNorm ReLU
BatdiNorm + ReLU
BatchNomm + Rel.U)
BatchNorm + ReLU
BatchNorm + ReLU
64 x 3 filtes
64 61.3 filter.
64
64
3 12 x 32
128 x filters
filters
fileers
filters
10
MaxPool
smide
Com BauchNorm
Com BatchNorm ReLU
21
20
17
18
12
13
13
14
16
19
Linear + ReLU + Dropout
2D MaxPool
Linaar Ral Dropout
2D Carw BanchNorm ReL.U
2D Conv.
2D Conv.
Linear
ManPool
BBtchNomm
ButchNorm ReLU
BatchNorm ReLU
BotchNorn
RonchNorm
2 stide
5123 x filters
5123 filters
40566 x 100
512 x 1006
612 3 x liltes
41166 x 4/96
x 2 scide 2
filess
22
20
21
17
19
18
16
Libear
20 Com. BatchNorm
2D Com,
20 Com
2D Adaptive Averae Pool
2D Com BatchNorm ReL.U
2D Cam BatchNorm + Rel.1
2D Cam
Coos. BatchNorm
Cam BatchNorm Rel.U
Cam BatchNorm Rel.0
BatchNorm + ReLU
BatchNorm + ReL.U
BatchNorm - ReLU
5123 fitter
512.3 x filters
512 x 100
5123 x filters
512 x liteer
256.3 x Riters
512 x fitees
1x
filter
lilters
filee
Table 5: ResNet18 for TinyImageNet
Layer
Type
Size
Input
x 64 64
2D Cony.
BoochhNom R.L.U
6677% 7 litters
2D MaxPool
3xx 3 stidd 2
2D Conv.
RotchNorm ReL.U
06:3 filter
ButchNomm ReLU
lilters
2D Comy.
ButchNomm ReLU
66 3 filters
BascNNon RaL.U
lilters
BaachNom ReLU
RonchNomm
BatchNomm
10
ButchNorm ReLU
BaachNonm
liless
12
BaxchNomm
lilles
13
Corv
RoschNorn
filers
14
Cony.
BatchNorm ReLU
256 filters
15
2D Cony.
BoschNorm ReLU
256 filter
16
2D Cory
BaschNorm ReLU
256-1 x filtes
17
2D Coms
RonchNorm Rel IT
512.3 x fiters
18
2D Cams
RonchiNrm
612.3 x filles
19
2D Cony.
ButchNomm ReLU
512 1 filtees
20
2D Cony.
BotchNorm ReLUU
512-3 3 filter
21
2D Coay.
BotchNorm ReLU
612 3 fittess
12
2D Adipiive Avarge Pool
1x1
23
Lancar
612 200
25


Reviews:
[{"review_id": "tNw-vUDSOPh", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper empirically studies the reason for the phenomenon that deep neural networks can memorize the data labels, even the labels are randomly generated. New geometric measures by replica mean-field theory are applied in the analysis. \n\nThe findings of the paper are interesting. It shows the heterogeneity in layers and training stage of the neural net:\n\ni) Memorization occurs in deeper layers; rewinding the final layer to the early weights mitigates memorization.\n\nii) When memorization happens, the early layer still learn representations that can generalize.\n\niii) In the training, early activations stabilize first, and deeper layers weights stabilize first. \n\niv) Near initialization, the gradient is dominated by unpermuted examples.\n\nI have the following questions/comments:\n\n- It is better to further explain the intuition of the Manifold Geometry Metrics.  The current Figure 1(B) is not very clear.\n\n- In Manifold Capacity, what do P and N exactly mean? Is this P the number of classes as used elsewhere?\n\n- The paper explains that by training on permuted examples, the network can learn generalizable representations at the initial training stage because the gradient ignores permuted examples. But why in the later training stage, the early layers and later layers show different generalization properties?\n\nIn general, this paper carries well-organized experiments. One shortcoming is that the paper does not provide a methodology to solve the generalization problem or further theoretical analysis of the observations.  But the empirical discoveries are novel and can be beneficial to the deep learning community.  \n\n###########\n\nUpdates: Thanks for the authors' response. The modified version improves clarity. I think this paper provides nice observations and initial analysis to the community and can be beneficial to future work, so I recommend this paper to be accepted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "hXWN4UqX-yh", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyses memorization in DNNs, from the lens of memorization = fitting random labels, and finds that it seems to happen in later layers. These results are obtained using the MFTMA framework, a manifold analysis tool, testing geometric properties of individual layers. The analysis also attempts to explain why such a phenomenon exists, and makes a few interesting observations.\nThis paper does not propose any new algorithm, but instead settles some important questions by infirming or affirming past speculation on layer behaviour found in the literature.\n\nI find three particularly interesting results in this paper:\n- later layers seem to be responsible for memorization, while early layers seem to converge last but consistently learn \"generalizing\" features (although this may not be true for other architectures)\n- increasing the dimensionality of the network to induce double descent _decreases_ the manifold dimensionality of the last layer. This is consistent with overparameterization making everything smoother/flatter and more easily disentangleable in the last layer.\n- for examples with the wrong class, gradients initially vanish (due to destructive interference), which seems to be a driving force for the initial good generalization performance.\n\nDownsides of the paper:\n- The setting explored here is somewhat artificial, (1) the requirement on a high enough epsilon (random label proportion) may not represent real use of DNNs (I write this having seen Fig A.8; this is also a common criticism of double-descent results) (2) the models trained here don't seem to exceed 40% testing accuracy, again not necessarily representing real use of DNNs (this is a bit surprising considering even models from back in 2013 had above 60% accuracy on CIFAR100).\n- Although the results of the paper do not hinge entirely on it, the reliance on MFTMA limits the interpretation somewhat: while an interesting tool, it's not clear to me that it allows us to make strong statements about the geometry of neural networks. In particular for the early layers, MFTMA may not be able to capture the geometry of features which might still be somewhat entangled yet possess a lot of richness.\n- I have some issues with the presentation of the paper\n- This paper does not really introduce a novel lens on generalization or significantly new ideas (although I'd argue it formalizes existing ideas and properly tests them empirically).\n\nOn the value of the contribution:\n- I think having empirical evidence of the studied phenomena is valuable, more so than previous speculation on them.\n- The empirical results presented here do open the door for new questions to be answered and may help focus the ongoing investigation of memorization and generalization in DNNs\n\n\nAdditional comments:\n- Something seems wrong with Figure 2B-middle two columns. Aren't permuted and restored examples the same inputs X but with the corresponding Y changed? If this is the case, then their UMAP should be the same, the only difference between the second column and the third column should be the coloring of the points. I presume that the figure shows a different minibatch of Xs for these two columns; I would highly recommend not doing so and using the exact same inputs. It would be consistent with the text, and the presentation, e.g. Fig 1A.\n- All Figures: the label fonts should be bigger. From the ICLR formatting guidelines: \"use 10 point type [for text]\", and \"all artwork must be neat, clean, and legible.\" Having to zoom in and out to be able to read figures properly hurts accessibility and legibility, which detracts from the quality of the paper. Packing text, results, and figures in an 8-page document can be hard, but synthesizing information, including visual information contained in figures, is an essential skill in conveying knowledge.\n-- Here are a few suggestions for this particular paper: Figure 1A seems unnecessary, the text conveys these 3 concepts clearly; Figure 1B is important and should take the entire width of the page, with legible fonts; Figure 2A's subplots all share the same X and Y axis, making their naming redundant and taking up space; Figure 2B's column labels are also repeated needlessly, taking up vertical space; Figure 3's X axis doesn't need individual layer name labels, and could be replaced with a single \"Layer depth\" label -- 3A and 3B also share this axis, leading to wasted vertical space (space that could be used to make fonts larger); idem for Figure 4A, individual layers do not need to be named, but rather the concept of layer depth can be conveyed with a properly labelled colorbar gradient -- 4CDE could be less wide and leave more horizontal space to make fonts larger.\n- In Figure 5A, it's not immediately clear that the X axis are individual layers, the log(nabla) label should be on the colorbar rather than on top of the figure. I'd also suggest flipping the X and Y axis, as the X axis is typically used for time; this would allow there to have the three subplots side by side with a shared labelled colorbar on the right (matplotlib seems to be used here, see matplotlib.pyplot.subplots's sharex/sharey arguments for examples).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "djl0JLE9qT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### \nSummary:\nThis paper investigates memorization in deep neural networks (DNNs).  Authors leverage mean field theoretic geometric analysis method (MFTMA) to analyze when and where memorization occurs in a DNN. Through empirical analysis, they show that i) generalizing feature are learned initially and that memorization happen later in training mostly in the top layers ii) we can mitigate memorization by rewinding the top layers parameters to earlier values. They also show that their MFTMA metrics can highlight the phenomena of double decent. Finally, they demonstrate that gradient descent initially ignores noisy example and focus on correctly labeled examples.\n\n###\nReasons for score: \nI lean toward acceptance. This paper makes interesting observation regarding memorization of deep network, it performs a good empirical study which provide enough evidences for the different claims.  Although, MFTMA could be a better explained in the main paper. \n \n###\n Pros:\n- As stated above, the paper makes interesting observation regarding memorization of deep network.\n-  It performs a thorough empirical study.  \n\n###\nCons: \n- I found it hard to understand MFTMA without referring to the appendix A. It would be nice to expand the explanation of MFTMA in the main paper. In addition, it would be good to further explain Fig 1. B which contains a lot of information. \n- Does the observation scale to larger dataset such as ImageNet ? \n- Experiments are run for only one seed.\n \n\n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "V151H2sR6O2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors apply MFTMA to DNNs trained on CIFAR with label noise to analyze their behaviors between generalization and memorization. Based on experimental results, they claim that what is involved in memorization are not lower layers but higher layers. This claim is convincing. Another claim that this is not caused by a vanishing gradient effect is plausible, too. I'm sure these results give some insights into understanding generalization and memorization by DNNs.\n\nQuestions. \nWhy do the authors consider only convolutional layers, not fully-connected layers, for the analyses? In the experiment of rewinding individual layers, the three FC layers are left untouched. Why?\n\nIs MFTMA the only method that can examine/verify the above finding?\n\nComments. \nAt the first reading, I didn't understand what \"restored examples\" means, and it took me a while to understand it. The caption for Fig. A.7 has an error; CIFAR100 should be Tiny ImageNet. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper offers novel insights about memorization, the process by which deep neural networks are able to learn examples with incorrect labels. The core insight is that late layers are responsible for memorization. The paper presents a thorough examination of this claim from different angles. The experiments involving rewinding late layers are especially innovative.

The reviewers found the insights valuable and voted unanimously for accepting the paper. The sentiment is well summarized by R2: "The findings of the paper are interesting. It shows the heterogeneity in layers and training stage of the neural net".

I would like to bring to your attention the Coherent Gradients paper (see also R1 comment). This and other related papers already discusses the effect of label permutation on the gradient norm. Please make sure you discuss this related work. As a minor comment, please improve the resolution of all figures in the paper. 

In summary, it is my pleasure to recommend the acceptance of the paper. Thank you for submitting your work to ICLR, and please make sure you address all remarks of the reviewers in the camera-ready version. 
Decision: Accept (Poster)