Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path
Authors: X.Y. Han
Vardan Papyan
David L. Donoho
Venue: ICLR 2022 Oral (2022)
Keywords: neural collapse
deep learning theory
deep learning
inductive bias
equiangular tight frame
ETF
nearest class center
mean squared error loss
MSE loss
invariance
renormalization
gradient flow
dynamics
adversarial robustness

Abstract: The recently discovered Neural Collapse (NC) phenomenon occurs pervasively in today's deep net training paradigm of driving cross-entropy (CE) loss towards zero. During NC, last-layer features collapse to their class-means, both classifiers and class-means collapse to the same Simplex Equiangular Tight Frame, and classifier behavior collapses to the nearest-class-mean decision rule. Recent works demonstrated that deep nets trained with mean squared error (MSE) loss perform comparably to those trained with CE. As a preliminary, we empirically establish that NC emerges in such MSE-trained deep nets as well through experiments on three canonical networks and five benchmark datasets. We provide, in a Google Colab notebook, PyTorch code for reproducing MSE-NC and CE-NC: https://colab.research.google.com/github/neuralcollapse/neuralcollapse/blob/main/neuralcollapse.ipynb. The analytically-tractable MSE loss offers more mathematical opportunities than the hard-to-analyze CE loss, inspiring us to leverage MSE loss towards the theoretical investigation of NC. We develop three main contributions: (I) We show a new decomposition of the MSE loss into (A) terms directly interpretable through the lens of NC and which assume the last-layer classifier is exactly the least-squares classifier; and (B) a term capturing the deviation from this least-squares classifier. (II) We exhibit experiments on canonical datasets and networks demonstrating that term-(B) is negligible during training. This motivates us to introduce a new theoretical construct: the central path, where the linear classifier stays MSE-optimal for feature activations throughout the dynamics. (III) By studying renormalized gradient flow along the central path, we derive exact dynamics that predict NC.

Published as a conference paper at ICLR 2022
NEURAL COLLAPSE UNDER MSE Loss: PROXIMITY TO AND DYNAMICS ON THE CENTRAL PATH
X.Y. Han" Cornell University xh3322ccrnel1 edu
Vardan Papyan"
David L. Donoho Stanford University
University of Toronto va rdan -aardddaappan wwt.-ppayyyyttrooto.c.o ca
Banoho@stanfordee
ABSTRACT
The recently discovered Neural Collapse (NC) phenomenon occurs pervasively in today's ' deep net training paradigm of driving cross-entropy (CE) loss towards zero. During NC, last-layer features collapse to their and class-means collapse to the same Simplex Equiangular Tight Frame, and clas- sifier behavior collapses the nearest-class-mean decision rule. Recent works demonstrated that deep nets trained with mean squared error (MSE) loss perform comparably those trained with CE. As preliminary, we empirically establish that NC emerges in such MSE-trained deep nets as well through experiments on three canonical networks and five benchmark datasets. We provide, in Google Colab notebook, Py PyToch code for reproducing MSE-NC and CE-NC: here The mmalytically-trataale MSE loss offers more mathematical opportunities than the hard-to-ande tonaloo-n CE loss, inspiring us to leverage MSE loss towards the theoretical investigation of NC. We develop three main contributions: (I) We show a new decomposition of the MSE loss into (A) terms directly interpretable through the lens NC and which assume the last-layer classifier is exactly the least-squares classifier; and (B) term capturing the deviation from this least-squares classifier. (II) We exhibit experiments on canonical datasets and networks demonstrating that term-(B) is negligible during training. This motivates to introduce a new theo- retical construct: the central path, where the linear classifier stays MSE-optimal for feature activations throughout the dynamics. (III) By studying renormalized gradient flow along the central path, we derive exact dynamics that predict NC.
both classifiers
INTRODUCTION
Modern deep learning includes paradigmatic procedures that are commonly adopted, but not entirely understood. Some examples are multi-layered architectures, stochastic gradient descent, batch normalization, cross-entropy (CE) loss, and training past zero error towards zero loss. Analyzing the properties of these practices is an important research task.
In this work, we theoretically investigate behavior of last-layer features in classification deep nets. In particular, we consider the training of deep neural networks on datasets containing images from C' different classes with N examples each class. After passing the i-th example in the e-th class through all layers except the last-layer of the network, the network outputs some last-layer features hi E I.. The last-layer of the network-whiee for each class c, possesses a classifier we E R and bias b. E R- then predicts label for the example using the rule arg maxe ((wer h4,c) + be).
The network's performance is evaluated by calculating the error defined by
Error = Ave 1fc * arg max ((war, h4.0) + +br)),
d!
while the weights, biases, and other parameters of the network (that determine the behavior of the layers before the last layer) in the network are updated by minimizing the CE loss defined by
exp (w., h.,o) ) + bab
CE = - Ave i,e
Dc expl(wer hi,e) + bcr}
"Equal contribution. Listed alphabetically.

---

Published as a conference paper at ICLR 2022
where Ave is the operator that averages over its subscript indices.
Prior works such as Zhang et al. (2016); Belkin et al. (2019) have shown that overparameterized classifiers (such deep nets) "memorize" their training set without harming performance unseen test data. Moreover, works such as Soudry et al. 2018) have further shown that continuing to train networks past memorization can still lead performance improvement Papyan, Han and Donoho (2020) recently examined this setting, referring to the phase during which one trains past zero-error towards zero-CE-loss the Terminal Phase of Training (TPT). During TPT, they exposed phenomenon called Neural Collapse (NC).
1.1
NEURAL COLLAPSE
NC is defined relative the feature global mean,
HG Ave hi.c:
3,0
the feature class-means,
Pe Ave c=1,...,C;
the feature within-elass covariance,
diss MaRE
DO singlec SIT Vertes
Ew Ave(hi, (t Me)
i,c
and the feature between-class covariance,
Σn Ave(pe (G) - pG)
is characterized by the following four limiting behaviors where limits take place with increasing training epoch t:
(NC1) Within-class variability collapse
Ew
where denotes the Moore-Penrose pseudoinverse. (NC2) Convergence to Simplex ETF:
l - |G||2||M4 He Holl2 {
(p pG. pe HG)
c=d T- c#d
Figure 1: Portrait of Neural Collapse. Top figure depicts the last-layer features, class- means, and classifiers with which NC is defined-as well as the Simplex ETF to which they all converge with training. Bottom figure shows the deviations of features from their corresponding class-means. Reproduced and modified from Figure I Papyan, Han, and Donoho 2020
- 1 - 1l - mall2 +0 ve
(NC3) Convergence self-duality:
We
p. M4
|well2
[r Hallz
(NC4): Simplification to nearest class center:
arg max (we, ht) ber arg mim ||h
He' lla
A similar phenomenon had been previously observed boosting Bartlett et al. 1998).
AA alternative line of work on "carly stopping" (Prechelt 1998 Li ct al. 2020 Rice et al.
2020) advocates
for terminating the training process carly and shows that it could be beneficial when training on noisy data small datasets. Such datasets out of the scope of this paper.
*Our characterization of (NC1) is more precise than that given by Papyan, Han, and Donoho (200)) which
only states ~0 for rhetorical simplicity. The convergence of the trace-norm of Σ Ew to zero is actual quantity measured by Figure of Papyan, Han, and Donoho 2020) when demonstrating (NC1). ΣS Ew
2

---

Published as a conference paper at ICLR 2022
The (NC2) property captures convergence to simple geometric structure called an Equiangular Tight Frame (ETF). An ETF is a collection of vectors { {Vele- C separated pair-wise angles. In classification deep nets, last-layer features are of higher dimension than the number of classes i.e. P>C. In this setting, the maximal angles are given by
having equal lengths and equal, maximally
(Ue, v2)
1,
for c d -- for oo f
|ell2 12
and the ETF is called a Simplex ETT Observe that as C' increeses, the ETF approaches a (partial) orthogonal matrix. Thus, when C' is large, this translates the intuitive notion that classifiers and class- means tend with training to (near) orthogonality.
1.2 DEEP NET CLASSIFICATION WITH MSE LOSS
While classification deep nets are typically trained with CE loss, Demirkaya et al. (2020) and Hui & Belkin (2020) recently reported deep nets trained with squared error (MSE) loss,
1
C(W, b, H) = Ave W Wh + b - Vi.ol12 1 +
(1) W
1| + [661))
(2)
2
2
X
WH bI ON
Y
W 7 + 1(612))
2CN
achieve comparable test performance as those trained with CE loss. Above, H E R PXCN and Y RCCCN are matrices resulting from stacking together the feature vectors h and one-hot vectors Vi., their respective columns; W RCXP is the matrix resulting from the stacking of classifiers w. rows; RC is the vector resulting from concatenating the scalars {be {be] and 1cN is the length-CIN vector of ones. Table T in Appendix[A] shows supplementary measurements al. (2020), i.c. the measurements affirm that MSE-trained networks indeed achieve accuracies on testing data comparable to those of CE-trained networks (cf. the analogous Table 1 in Papyan, Han, and Donoho (2020)). The analytically- tractable -tractaaell MSE loss offers more mathematical opportunities than the hard-to-analyze CE loss and inspires this paper's main theoretical contributions explicitly characterizing MSE-NC.
affirming the findings of Hui & Belkin (2020); Demirkaya
1.3 CONTRIBUTIONS
Our main contributions are follows:
We propose a new decomposition the the MSE loss: L = Lxci + LNC23 + Lts- The terms LxCi and LNC23 possess interpretations attributable NC phenomena and assume the classifier W is exactly the least-squares optimal classifier Wis (relative to the given H); and Lis captures the deviation of W from WLS. (Section
We provide empirical measurements of decomposition (Figure 2 on realistic dataset- network combinations showing that Lts becomes negligible during training-leading us to define the central path where = (Section
We reveal a key invariance property on central path. The invariance motivates the examination of a representative set of features, X-E H. that we call renormalized
features. (Sections 3.22
We study the gradient flow those renormalized features along the central path and derive exact, closed-form dynamics that imply NC. The dynamics explicit in terms of the
captures the subtlety that the size of the within-class "noise" (captured by Ew) should be viewed relative to the size of the class-means (captured by Eg).
"Traditional research on ETFs (cf. [Strohmer & Heath Jr 2003 Fickus & Mixon 2015) ) examines the PCC setting where C' ETF- span their ambient R." space. However, in the PSC setting of classification deep nets, the vectors not span I Following the precedent of Papyan. Han, and Donoho (2020). interpret ETFs as cqual-length and maximally-equianguu
vectors that are not necessarily spanning.
AAssume the stacking is performed in i-then-c order: the first column is (i,c) (1, 1). the second is (i, = (2, 1).... the N-th is (i, (N, 1), the (N + 1)-st is (i, c) = (1,2), and so on. This matters for formalizing FimuationggFquaiion iijn Corollary
3

---

Published as a conference paper at ICLR 2022
singular value decomposition of the renormalized feature class-meant at initialization.
(Section 3.3
Additionally, we complement this paper with new, extensive measurements on five benchmark datasets in particular, the MNIST, FaShionMNIST, CIFARIO. SVHN, and STL10 dataset
o (Dngg
. 2009 Krizhevsky Hinton, 2009) LeCun et al. 2010, Xiao al. (017) and three canonical
deep nets particular, the VGG, ResNet, and DenseNet networks (He et al. 2016; Huang et al.
2017 Simonyan & Zisserman 2014) verify the empirical reality of MSE-NC i.e. they show
(NC4) indeed occur for networks trained with MSE loss. These experiments establish that theoretical modeling of MSE-NC is empirically well mell-motivated. They are seachy-together spanning collectively defer them to Appendix A
four pages with seven figures and a table
DECOMPOSITION OF MSE LOSS
Inspired by the community's interest in the role of the MSE loss in deep net training (Demirkaya et al. 2020; Hui & Belkin 2020; Mixon et al. 2020; Poggio & Liao 2020a b). we derive new decomposition of the MSE loss that gives insights into the NC phenomenon. First, absorb the bias vector into the weight matrix by defining the extended weight matrix W [W, b] E CP+++))
so that Equation 2 can be rewritten and the extended feature vector h. [ha,ci 1] E p
A
L(W H)
Ave Whe yr.oll2 1.0
W W 2
Using [hi.c} define the entities Pee Pa. H. and Ew analogously to those in Section 1.1 We further define the extended total covariance and extended class-means matrices, respectively, as
ΣT Ave (hi ū (R++))((P+))
1.
M P Ticl E (P+1)xC
Next, we reformulate, with weight decay incorporated, classic result of Webb Lowe 1990)): Proposition (Webb & Lowe (1990) with Weight Decay). For fixed extended features H, the optimal classifier minimizing the MSE loss L(W, H) is
WiS C. M "M -E + XI)-1
where I is the identity matrix. Note that Wis depends on H only.
Note that we can interpret Wis as a function of H, leading us to identify the following decomposition of L where one term depends on H only.
Theorem 1. (Decomposition of MSE Loss; Proof in Appendix B The MSE loss, L(W H), can be
decomposed into two terms, CWW H) = Lis( (H) + Lts W. H) where
Lis H = Ave Wishi, vr + Wisll
2 be
and
Cts W. H
1
th ( Wis) (Dr + 21) (w Wis) }
2
In the above, Lis( H)) is independent of W. Intuitively, it is the MSE-performance of the optimal classifiers Wis (rather than the "real classifiers" W) on input H.
"CIFAR100 and ImageNet are omitted because Demirkaya et al. 2020 Hui & Belkin 2020) showed that they require an additional scaling heuristic on top of the traditional MSE loss. Rigorous investigation into this scaling-heuristic would have demanded mathematical analysis outside the scope of this paper as well as experimentation that would have consumed expensive amounts of computational resources.
4

---

Published as a conference paper at ICLR 2022
The component Lis (W, HH) is non-negative and is zero only when W WLs. Therefore, L (H) quantifies the distance of W from W.s. In short, the least-squares component, LLs (H), captures the behavior of the network when the classifier possesses optimal, least squares behavior. Meanwhile, the deviation component. Lis (W, H). captures the divergence from that behavior.
We can further decompose Lis H) into two terms, one capturing activation collapse (NC1) and the other capturing convergence to Simplex ETF of both features and classifiers ((NC2) and (NC3)).
Theorem 2. (Decomposition of Least-Squares Component; Proof in Appendix C The least-square component, LLs( H), of the MSE decomposition in Theorem can be further decomposed into Lis(H) H) Lxci H) + LNCuA( H), where
1
LNCI H
tr WLS
Ew + AI W
2
LNC2S
H)
2C Wis T1
Inspection of these terms is revealing. First, observe that LxC23 is a function of the class-means and MSE-optimal classifiers. Minimizing Lxc2s will push the (unextended) class-means and classifiers towards the same Simpex ETF matrix"i.e. (NC2)-(NC3). Next, note that the within-class variation independent of the means. Thus, despite the fact that classifiers are converging towards some (potentially large) ETF matrix, we can always reduce LxCI by pushing Ew towards zero, which corresponds to (NC1).
Figure 2 measures the empirical values of the above decomposition terms on five canonical datasets and three prototypical networks. It shows that Cis( W H) becomes negligible compared to LLS (H) when training canonical deep nets on benchmark datasets. other words, L(W, H)sLL H) starting from early epoch in training and persists into TPT.
This motivates us to formulate the central path:
p (Wis(H) H) H E R((+1)xCN]
where the notation (-s(-) makes explicit the fact that Wis is function of H only. Intuitively, for a classificr-feature pair to lie on the central path. i.c. (W, H) E P. means that the "real classifier" W will exactly equal WL optimal classifier that would result from fixing H and minimizing L w.r.t. just the classifier. Combined with Theorem only if L(W, - (H). Figure 2 shows that classifier-feature pairs lie approximately on the central path during TPT, allowing us to shift focus from L to LLs-
(W, H) P if and
3 EXACT CLOSED-FORM ANALYSIS ON CENTRAL PATH
Consider in this section. For the subsequent theory, we adopt the unconstruined features (Mixon et al. 2020) or layer-peeled (Fang et al., 2021) modeling perspective") We consider continuous training dynamics with gradient flow where time-of- ofriinni- is denoted by the variable t20. Within this model, we analyze the dynamics of the renormalized features X X= H on the central path
d x --==Illxx (Lx)is(X)),
dt
"The middle term is the of pesitive-semidefnite matrices-one of which is positive-definite.
"Appendices C.1 C.2 elaborates upon this interpretation of Lxc2a in more detail.
"Both the unconstrained features model |Mixon al. 2020) and the (1-)layer-peeled model (Fang al. 2021) advocate for the mathematical analysis of deep nets through variants gradient flow directly on last- last-layer features and classifiers. The layer-peeled model is so- named because it proposes to analyze deep nets by "peeling away" one layer at a time. The unconstrained features model is so-name because features are not constrained to be the output of deep net forward pass but rather free variables that be optimized directly: This captures the nearly unlimited flexibility afforded feature engincering transformations in modern deep nets by the many nonlinear, overparameterized layers.
Intuitiivlly one should interpret t=0 the time-of-training when feature-classifie pairs effectively enter the central path.
5

---

Published as a conference paper at ICLR 2022
MSD
Eyec
Cix
Breer
NINIST
SVII
CEPARIO
STLJD
1o-00
10-23
20-05
14-22
2o-04
190 209 2001 .000 2011
100 18U 200
200 2011
350
1433 380 2007
2001 2001 250
30 10A1 190
2001 350
2001 2001
201 3101
300
300
Epech
Epech
Kpoch
Rgech
Spond
Figure 2: Decomposition of MSE loss: Each array column shows a benchmark image classification dataset while each row shows canonical deep net architecture trained with MSE loss. The red vertical line indicates the epoch at which zero training error was achieved. In each array cell, we plot terms of
the MSE loss decomposition L(W, H) LxC1 (H) LNC2I (H) Cts (W, .H) from Section 2 Starting from an early epoch in training, Liss (W, H) becomes negligible compared to the dominant
term, LxCI H),, implying Cis W, H)&Liss Lis Lxci (H) LNeas H), i.e. the features and classifiers are effectively on the central path during TPT. Note that LNC2/3 H) diminishes the fastest among all the terms: Intuitively, this shows that the network primarily focuses on distributing the feature class-means into "uniform" Simplex ETF configuration (NCI)-(NC2) early on and, from there, compresses the activations towards their experimental details are in Appendix A Outlier behavior is discussed in Appendix A.7
i.e. (NC1). much as possible. Further
where the operator xx projects the gradient onto the tangent space of the manifold X, of all identity-covariance features. For Σ 3 H be well-defined, assume that Ew remains full-rank during training. We call Equation the continually renormalized gradient flow and will motivate it in more detail in later subsections. We now state main theorem:
Theorem 3. (NC Under Continually Renormalized Gradient Flow) Consider the continually renormalized gradient flow (Equation 5) in which the dynamics are restricted to the central path, and the features are renormalized to identity within-class covariance. Assume the features are initialized with zero global mean. Then, Neural Collapse emerges on the renormalized features with explicit dynamics of the flow given Proposition 2 as well as Corollaries 7 and later this paper.
Three assumptions standout which we discuss below:
(A1) Restriction to central path: Figure shows that practice, during TPT Lis (W, H) is
observed to be negligible compared to the dominant term, LLs( H). This indicates (W. H) ) is close the central path [Equation 4p where L(W, H) Lis(H) One the central path, one is effectively training with Lis as the loss.
Proximity the central path implies that the last-layer biases become close to bLs. Having biases b = bus is effectively performing classification with no bias term (i.c. using only a linear classifier W) on zero-mean data. To see this, define the globally-centered features,
(A2) Zero global mean:
11 Y is a variant of the well-known generalized Stiefel manifold Absil et al. 2009 Chapter 3.4).
12 "'Gvven the definition Equation 1] the positive-definitnnee because C.N >> P ie. the number of examples is larger than the width of the last-layer. Note this does not contradict (NC1). (NCI) is characterized by Ew approaching the zero vero-matrix with training (relative to EB). In practice (for example, in supplementary Appendix experiments, observed that the zero-matrix is never actually achieved during the training dynamics and En indeed remains full-rank.
Ew unsurprising in deep net contexts

---

Published as a conference paper at ICLR 2022
globally-centerd means, and total-covariance matrix in unextended coordinates:
- uo M = [ PG..... MC Ma];
ΣT
HH
CN
Proposition T then reduces to the following forms for the unextended weights and biases:
Wis Wiss huS) [ M ET',
(6)
For an arbitrary activation-targe target pair (h, y). the prediction error obeys
Wish bLS- - y = C-MMM ΣT h + C- C-
M
C M MM ET PG y
Wis(h pG) - (y (y
The last line exhibits two terms in parentheses, both traceable to the action of the bias bLS. The term h M&G demonstrates that the bias induces global-mean subtraction on h, while the term y C-11 demonstrates that the bias also induces a global-ment subtraction on the one-hot targets.
(A3) Renormalization to identity covariance: Renormalization is ubiquitous in the empirical deep
learning literature through works such batch normalization and its variants (loffe & Szegedy 2015
Salimans & Kingma 2016 Wu & He. 2018 Ulyanov al. 2016 [a al., 2016 Krizhevsky et al.
2012). It is also ideologically precedented in the theoretical ML literature through works such as
Douglas al. (1998); Banburski al. (2019); Poggio & Liao (2020a b). Within this paper. the is inspired by an invariance property of the loss as well as the intuitive
concept of the signal-to-noise ratio from mathematical statistics (discussed later in Sections 3.1 3.2).
In the subsequent sections, we motivate and introduce the theoretical constructs necessary for proving Theorem 3 and for presenting the explicit dynamics of the flow.
3.1 INVARIANCE PROPERTY
By Assumption (A2), classification the central path is equivalent classification globally- centered features using only the linear classifier WLs. Then, using Equation we can equivalently represent the central path (Equation 4) as follows:
Definition (Zero Global Mean Central Path).
p P= P {Wis. H)
Wis= Σ
An invariance property holds the central path. Let A denote symmetric, full-rank matrix. Then, the explicit form in Equation 6 for Wis implics
Wis( = C- (AM)
[AH) (AH)
T AH
=C-IM A A -1 HH A
A- AH
-C-IM (H (H H M -
(H)
where the notation (.s(.)) makes explicit that Wis is function of set of input features. Thus, the actual predictions made by the least squares classifier are invariant to choice of the coordinates in which we express H, i.e. all features AH equivalently performing. Among those coordinate systems, we will prefer the one in which the "noise" is "whitened" or "sphered." Recall that we assume Ew Where the notation (-(() expresses the within-class covariance function of the
features-is positive-definit. Consider the coordinate transformation Н-AH with A = E (H). In these coordinates, the features are "renormalized" to spherical covariance,
Σw (AH) AEW A I,
so we will call AH the renormalized features.. The class-means of AH are
(AH AM = E (H)
where the notation (-)) expresses that the means are function of features. Thus, Equation describes continuous training dynamics where we continually "renormalize" features to their equivalently- representer.
7

---

Published as a conference paper at ICLR 2022
3.2 SIGNAL-TO-NOISE RATIO MATRIX ON THE CENTRAL PATH
The En M term evokes the canonical idea of a signal-to-noise ratio from statistics. To see this, note that the classifier in Equation 6 can be rewritten as
Wis = M (C-MMM + Ew)"
The combined presence of the terms M and Ew in the denominator, along with M in the numerator, signals to us the presence of the informative signal-to-noise (SNR) ratio matrix:
SNR = E M.
(8)
Why this terminology? The globally-centered class-means represent the overall "signal" indicating that the class-means are separated from each other: if they were non-separated-e-g e-g. all equal M would be zero. However, a classifier decision must keep in view noise- captured by which might confuse the classifier. So, one wishes the signal be large compared the noise. may help to consider the mental model-which is not at all necessary to the correctness of our analysis-unde which the activations are normally distributed with h.,c ~ NNpc Ew). Under that model, linear classifier will indeed get confused if the norm l Mcll22 is "small compared" to Ew. Replacing this somewhat vague statement by a discussion quantitative properties of the SNR matrix is well understood by mathematical statisticians to be decisive for understanding classification performance
in the normal case. Additionally, observe that SNR (EW H). Thus, connecting back to Section 3.1 the SNR matrix is simply the meass matrix of the renormalized features.
The SNR can be further understood through its singular value decomposition (SVD).
Definition (SVD of SNR Matrix). Denote the SVD of the SNR matrix (Equation 8) as follows:
SNR = UNVV Σ wyu,
j=1
Here, (wele=t are the non-zero singular values of SNR; S = diag (fwe) E RCC, and the left and right singular--vctors, U E PRPxC and V E RCxC are partial orthogonal and orthogonal matrices, respectively, with columns {u,} and {- The SNR matrix rank C 1 since Ew is assumed full-rank and M is rank C 1 due to global-mean subtraction.
The non-zero singular values, [ are decisive for understanding the separation performance of the least-squares linear classifier. Good performance demands that the singular values be large. Works such as Fukunaga [192]] Chapter 10) and Hastie Tibshirani (1996) show that the magnitude of each singular value of the SNR matrix is the size of the class separation in the direction of the corresponding singular vector. In this sense, the smallest singular value captures the smallest separation between classes. Consequently, driving- larger the smallest non-zero singular value makes the task of linear separation more immune to (spherical) noise and to Euclidcan- wawpaaan-ooo--onttarannn adversarial noise. As will see in the next section, examining the dynamics of these singular-values when the features undergo gradient flow in renormalized coordinates leads to precise characterizations of Neural Collapse 13
3.3 DYNAMICS OF SNR
For any matrix Z, we use Z, in this section to denote the state of that matrix at time t. Similarly, we denote with (wo (t){t)] C-1 the state of the non-zero SNR singular values at t. We now present Proposition 2 and Corollaries 2 that provide the explicit dynamics referenced by Theorem 3 Proposition (Dynamics of Singular Values of SNR Matrix; Proof in Appendix D.44). Continually renormalized gradient flow on the central path Equation 5) induces the following closed-form dynamics on the SNR singular values (Definition)
ei log(w;(t)) + cow) + ctwu(() = a,++ +t, 120, for all j=
(9)
Cb. C22 and ca are positive constants independent of j. and aj is constant depending (0)
For additional, supplementary intuitions about invariance, renormalization, and continually renormalized gradient flow, see the exposition in Appendices D.I D3
8

---

Published as a conference paper at ICLR 2022
Corollary (Properties of SNR Singular Values; Proof in Appendix ).5). SNR singular values (Definition 2 following the Equation Adynamics satisfy the following limiting behaviors:
1. lim w,(t) = oo
and lim
2,00 = 1. Vijea
for all j = 1...., , -1.
max, (1) 2. lim miny wy (!)
The first fact shows that all non-zero singular values of the SNR matrix, Eny M, grow to infinity at an asymptotic rate of 4/t/33. Intuitively, this shows that the "signal" becomes infinitely large compared to the "noise", i... (NC1). The second fact shows that the singular values of E & M (which has zero-mean columns) approach equality-impllyee implying that the limiting matrix is a Simplex ETF (Lemma 7 Appendix [D66 i.e. (NC2). Finally, (Payyan, Han, and Donoho (2020) The- orem 1) show that (NC1-2) imply (NC3-4) on the central path. Corollary Matllaryyfffmme formalizes these intuitions.
Corollary 2 (Neural Collapse Under MSE Loss; Proof in Appendix D.6. Under continually renormalized gradient flow (Equation 5). the SNR matrix (Equation 8) converges
lim 8-+00 (m)x(()
SNR, = V.V.
(10)
where Uo E PCx1() and Vo E (CX(C-1) are the left and right singular vectors of the SNR the non-zero singular values; and Wmax (t) is the
matrix (Definition 2) at t=0 corresponding largest singular value at time t. Furthermore, Corollary 7 Timplies the occurrence of (NCI)-(NC4) ie. renormalized gradient flow on the central path leads to Neural Collapse.
Moreover, denoting the Kronecker product with &, the renormalized features matrix converges to
lim Wmax (t)
1
E Hw (O)Vo') @ V.
(11)
RELATED WORKS
Since the publication of Papyan, Han, and Donoho (2020), several works MMixn et al.) 2020; Lu &
Steinerberger 2020 E & Wojtowytsch| (200)) Poggio & Liao 2020a Fang et al. 2021 Ergen &
Pilanci 2020
Zhu et al. 2021) proposed mathematical frameworks ratifying Neural Collapse.
Among them. Mixon et al. (2020) and
and Poggio & Liao (2020a b) also examine MSE-NC. Our contri-
butions distinct from the MSE-NC analyses of Mixon
(2020) and PPgggio & Liao (2020a
b):
The work of Mixon et al. (2020)
relies on a linearization of the unconstrained features model ODE
while(Poggio & Liao (2020a b) in special section co-authored with Andrzej cenbburski-examinne
homogeneous, weight-
networks. In contrast, this paper considers the dynamics of the
renormalized gradient flow on the central path (motivated by the discussions Sections 2 3 Neither
Mixon et al. 2020 nor Poggio & Liao (2020a]b) provide exact, closed-form dynamics as we do.
Outside the MSE setting, Lu & Steinerberger (2020); E & Wojtowytsch (2020); Ergen & Pilanci (2020) Fang et al. (2021): Zhu et al.] (2021) examine the emergence of the NC properties under
variants of the unconstrained features/layer-Pee peeled model trained with CE loss. These works focus on
characterizing the loss landscape or the global minima without describing the dynamics.
We provide detailed survey of all above-mentioned papers in Appendix E
CONCLUSION
In this paper, after verifying that NC occurs when prototypical classification deep nets are trained with MSE loss on canonical datasets, we then derive and measure a novel decomposition of the MSE loss. We observed that the last-layer classifier tends to the least-squares classifier- motivating us to define the central path on which the classifier behaves exactly as optimal least-squares classifier. On the central path, we showed invariance properties that inspired to examine the renormalized features and their corresponding continually renormalized gradient flow. This flow induces closed-form dynamics that imply the occurrence of Neural Collapse.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
This paper is reproducible. Experimental details about all empirical results described in this paper are provided in Appendix A Additionally, provide PyTorch (Paszke al. 2019) code for reproducing NC under both MSE and CE as well as the Figure 2 decomposition in the following Google Colaboratory notebook: here Experimental measurements used to generated the plots in this paper have been deposited in the Stanford Digital Repository, here All datasets and networks used in experiments originate from the PyTorch Model Zoo with prepproesssing details described in Appendix A Formal statements and proof' of all our theoretical results are provided in Appendices BD
ACKNOWLEDGEMENTS
Some of the computing for this project was performed on the Sherlock cluster. We would like to thank Stanford University and the Stanford Research Computing Center for providing computational resources and support that contributed to these research results.
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), [funding reference number 512236 and 512265]. This research was funded by Connaught New Researcher Award at the University Toronto. This research was enabled in part by support pro- vided by Compute Ontario (www.compure, www.computeentario.coo and Compute Canada www.computecanaaa.comm ca). This work was also partially supported by NSF Division of Mathematical Sciences Grants 1407813, 113, 1418362, and 181 1614 and by private donors.
REFERENCES
P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. Princcton University Press, 2009.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pp. 322-332. PMLR, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.06650 2016.
Andrzej Banburski, Qianli Liao, Brando Miranda, Lorenzo Rosasco, Fernanda De La Torre, Jack Hidary, and Tomaso Poggio Theory ii:: Dynamics and generalization in deep networks. arXiv preprint arXiv:1903.0499) 2019.
Andrzej Banburski, Fernanda De La Torre, Nishka Plant, Ishana Shastri, and Tomaso Poggio. Cross- validation stability of deep networks. Technical report, Center for Brains, Minds and Machines (CBMM), 2021.
Peter Bartlett, Yoav Freund, Wee Sun Lee, and Robert E Schapire. Boosting the margin: A new explanation for the effectiveness of voting methods. The annals of statistics, 2665):1651-1686, 1998.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern muchine-learning practice and the classical bias- variance trade-off. Proceedings of the National Academy of Sciences, 116632):15849-15585 2019.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over- wer-parameterized models using optimal transport. Advances in neural information processing systems, 31 31:3036-3046, 2018.
Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. arXiv preprint arXiv:2002. 04486, 2020.
Ahmet Demirkaya, Jiasi Chen, and Samet Oymak. Exploring the role of loss functions in multiclass classification. In 2020 54th Annual Conference on Information Sciences and Systems (CISS), pp. 1-5. IEFE, 2020.
10

---

Published as a conference paper at ICLR 2022
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248-255. Icee, 2009.
Scott C Douglas, SY Kung, and Shun-ichi Amari. A self-stabilizze minor subspace rule. IEEE Signal Processing Letters, 5( 5(12):328-330, 1998.
Weinan and Stephan Wojtowytsch. On the emergence of tetrahedral symmetry in the final and penultimate layers of neural network classifiers. arXiv preprint arXiv:2012..55420, 2020.
Tolga Ergen and Mert Pilanci. Revealing the structure of deep neural networks via convex duality. arXiv preprint arXiv:2002.09773, 2020.
Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Layer-peeled model: Toward understanding well-trained deep neural networks. arXiv preprint arXiv:2101.12699 2021.
Matthew Fickus and Dustin G Mixon. Tables of the existence of equiangular tight frames. arXiv preprint arXiv:/504.0000 2015.
Keinosuke Fukunaga. Introduction to Statistical Pattern Recognition. Elsevier, 1972.
Trevor Hastie and Robert Tibshirani. Discriminant analysis by gaussian mixtures. Journal of the Royal Statistical Society: Series (Methodological), 58(1):155-176, 1996.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross- entropy in classification tasks. arXiv preprint arXiv:2006.0733 2020.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097- 2012.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. AT&T Labs [Online]. Available: hup://yunn. lecun. comfexdb/mnistt, 2, 2010.
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparamettrrize neural networks. In International conference on artificial intelligence and statistics, pp. 4313-4324. PMLR, 2020.
Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss. arXiv preprint arXiv:2012.08-45. 2020.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665-BB) 2018.
11

---

Published as a conference paper at ICLR 2022
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.11619, 2020.
Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of Neural Collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):
24652-24663, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, pp. 8026-8037, 2019.
Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint arXiv:2101.00072, 2020a.
Tomaso Poggio and Qianli Liao. Implicit dynamic regularization in deep networks. Technical report, Center for Brains, Minds and Machines (CBMM), 2020b.
Lutz Prechelt. Early stopping-bur when? In Neural Networks: Tricks of the trade, pp. 55-69. Springer, 1998.
Leslie Rice, Eric Wong. and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference Machine Learning, pp. 8093 -1044 PMLR, 2020.
Grant M Rotskoff and Eric Vanden-Eijndens Parameters as interacting particles: long time conver- gence and asymptotic error scaling of neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7146-7155, 2018.
Tim Salimans and Durk Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29:901-909, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2014.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1): 2822-2878, 2018
Thomas Strohmer and Robert W Heath Grassmannian frames with applications to coding and communication. Applied and computational harmonic analysis, (4(3):257-275.2000 2003.
James Townsend. Differentiating the singular value decomposition. Technical report, Technical Report 2016, httpss//j-towns. github. io/papers/svd-deriivtiie
2016.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint
2016.
Andrew R Webb and David Lowe. The optimised internal representation of multilayer classifier networks performs nonlinear discriminant analysis. Neural Networks, 3(4):367-375, 1990.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp- 3-19, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset for bench- marking machine learning algorithms, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 2016.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. arXiv preprint arXiv:2105.02335, 2021.
12

---

Published as a conference paper at ICLR 2022
APPENDIX
MSE Neural Collapse experiments
14
A.1
Datasets
14
A.2
Networks
14
A.3
Optimization methodology
................................
14
Computational resources
14
A.5
Formatting
15
A.6
Experimental results
15
A.7
Exceptions: STLIO-ResNee and STLIO-DenseNer
21
A.8
Open Questions: Weight-decay, batch normalization, SGD, generalization, test data
T
Theorem
22
T Thoorem
22
Intuitions for Theorem 2 unextended coordinates
23
Additional intuitions for Theorem terms
23
Theorem
23
D.I
Implications of invariance
23
................................
D.2
Aligned SNR coordinates
25
D.3
Continually renormalized gradient flow
27
D.4
Proof of Proposition
(gradient flow in aligned SNR coordinates)
28
D.5
Proof of Corollary
31
D.6
Proof of Corollary
E
Related works examining Neural Collapse
33
Mixon, Parshall, and Pi (020))
34
E.2
Lu and Steinerberger (2020)
34
and Wojtowytsch (0020)
34
E,4
Poggio & |iao (2020a b) (with Banburski)
35
Ergen Pilanci [2020]
35
E.6
Fang, He, Long, and Su (2021)
35
Zhu, Ding, Zhou, Li, You, Sulam, and Qu (022))
36
13

---

Published as a conference paper at ICLR 2022
A MSE NEURAL COLLAPSE EXPERIMENTS
The experiments in this section examine the properties of Neural Collapse (NC) on deep nets trained using MSE loss. The direct MSE-analogues the cross-entropy (CE) loss table and figures in Papyan Han, and Donoho (2020) are in Table I and Figures 3 here. Furthermore, Figures 10 compares the MSE-NC experiments observed this paper with the CE-NC behaviors in Papyan, Han, and
Donoho (202)). Experimental descriptions and discussions are within the captions. Subsections A. A.5 describe formatting and experimental details. Then, Subsection A cooolcilyy presents all experimental figures for this section. Subsection a.dresssss outlier behaviors observed in the plots. Finally, Subsection A.8 discusses open questions inspired by NC.
A. DATASETS
We consider the MNIST, FashionMNIST, CIFAR10, SVHN, and STL.10 datasets with pre-processing the same as in Papyan, Han, and Donoho (2020).
A.1.1 CHOICE OF DATASETS
Demirkaya et al. 2020) and Hui & Belkin (020) showed multi-class classification deep nets trained with MSE sometimes with a heeurisiccll-saling--- demmonngddemoottraaeee comparable test-error performance to those trained with CE loss. Even without additional scaling, i.e. with vanilla MSE loss, we found this true for ten class classification datasets such as MNIST, FashionMNIST, SVHN, CIFARIO, and STL10- such as CIFAR 100 and ImageNet), both Demirkaya et al. (2020) Hui & Belkin (2020) showed that the additional ccaling-heuristic does need to be applied to the loss before comparable test-performance can be achieved. In informal, exploratory experiments not reported here, we were able to reproduce their results on datasets with more than classes. But, we feel these scaling-heuristie merit further scientific investigation of their own and are beyond the scope of this article 14 so we do not include datasets that require sealing-heurisee tuning.
focus these five datasets. For more than ten classes (i.c. datasets
A.2 NETWORKS
We train VGG, ResNet, and DenseNet models with the same depth-selection procedures and architeaure-specffcattio choices as in Papyan, Han, and Donoho 2020). In the MSE setting, the final chosen depths were as follows:
DATASET
VGG
RESNET
DENSENET
MNIST
VGGII
RESNETI8 DENSENET40
FASHIONMIIST
VGGI1
RESNET18 DENSENET250
SVHN
VGG13
RESNET34 DENSENET40
CIFARIO
VGGII
RESNET5O DENSENET 100
STLIO
VGGIT
RESNETI8 DENSENET201
A.3
OPTIMIZATION METHODOLOGY
The optimization algorithm, parameters, and hyperparameter tuning are the same as in Papyan, Han and Donoho (2020).
A.4 COMPUTATIONAL RESOURCES
The experiments were run on Stanford University's 's Sherlock computing cluster. Each dataset-network combination was trained single GPU attached CPU with at least 32GB of RAM- specific types of the CPUs/GPUs vary according to whichever was first assigned to us by the HPC cluster scheduler.
"TToorughh experimentation with this dalling-heuriste would also consume prohibitively expensive amounts of computational resources.
14

---

Published as a conference paper at ICLR 2022
A.5
FORMATTING
The coloring and formatting of the plots are the same as in Papyan, Han, and Donoho (2020).
A.6
EXPERIMENTAL RESULTS
Table 1: Table comparing test-aceuracy at moment O-error is achieved vs. at the end of training. Analogous to Table of Papyan, Han, and Donoho (2020). The median improvement is 0.962 percentage points; the mean 1.833 percentage points.
DATASET
NET
Acc. O-ERROR
Acc. FINAL
VGG
99.23
99.59
MNIST
RESNET
99.16
99.70
DENSENET
99.62
99.70
VGG
92.76
92.95
FASHION
RESNET
93.55
93.76
DENSENET
90.56
92.95
VGG
89.35
93.91
SVHN
RESNET
85.18
92.65
DENSENET
95.61
95.23
VGG
83.13
84.54
CIFARIO
RESNET
75.43
76.39
DENSENET
91.77
91.78
VGG
60.43
67.24
STLIO
RESNET
59.35
60.56
DENSENET
58.74
60.41
Maun
Chuocifers
FoshOoMMNIST
VGG
0.4
ReaN64
6-2
Doun&No
0.9
0.8
Epech
Epech
Eyech
Eposh
Figure 3: Plots analogous to Figure in Papyan et al. 2020), but on networks trained with MSE Loss. Results demonstrate that last-layer features and classifiers approach equinormness.
15

---

Published as a conference paper at ICLR 2022
Mewn Articaaions
Cheeifeers
MNNST
FusblonMNIST
SVEEN
CIPAR10
SIL.10
VCG
+0 Brroe
RooNot
DonoNet
1.00
200
300
1.00
201
200
100
200
2000
100
200
300
1.0
202
350
Figure 4: Plots analogous to Figure in Papyan al. (020)) but networks trained with MSE Loss. Results demonstrate that last-layer features and classifiers approach equiangularity.
Claodffers
MNIST
SVHN
CIFARIO
STLI1
0.6
0.6
VGG
1.0
0.6
HonNNe
0.0
0..0
1E Stifted
300
Epoch
Epech
Epech
Epech
Epech
Figure 5: Plots analogous to Figure in Papyan al. 2020), but networks trained with MSE Loss. Results demonstrate that last-layer features and classifiers approach maximal-equiangglarn
16

---

Published as a conference paper at ICLR 2022
SVHN
CIFAR10
VOG
1.2
RosNet
DousNNot
200
200
202
200
203
Epoads
Epends
Epords
Epects
Epects
Figure 6: Plots analogous Figure in Papyan al. (020), but networks trained with MSE Loss. Results demonstrate that last-layer features and classifiers approach self-duality.
1.0+001
T1/ 900/C
1.0e000
VCG
1.0-01
143
14c 00
1.0e4011
1.0e-00
Tri
1.0-0L
1.0--2
1.00
200
200
1.50
200
110
500
2001
560
2501
2001
140
500
200
Figure 7: Plots analogous to Figure 6 in Papyan al. 2020), but networks trained with MSE Loss. Results demonstrate that last-layer features undergo variability collapse.
17

---

Published as a conference paper at ICLR 2022
MNIST
FashioMMNNS
SVHN
CIPARII
5T7.10
0.8
VG0
Irogetie
0.5
RecVel
Propertes
0.8
DoneNNe
0.6
Proprttes
Rpsal
Rysch
Fpuads
Rpaads
Baosss
Figure Plots analogous Figure in Papyan al. (2020), but on networks trained with MSE Loss. Results demonstrate that classifier decisions converge to those of the nearest class center decision rule.
2e-00
M
2--11
10:01
VCG
2::
8--11
1m--00
0 Error
36-03
Se-02
$c-01
1x:1
3+-22
9o-02
111
36-03
2--01
1e-000
1o-01
20-00
20-03
Iod1
Te-31
So
20-01
2e+00
1m411
31-03
Bobrartee
Se-02
2e-00
20422
1c:00
1o402
Sw-01
Fewoch
Kjeeh
Figure 9: Plots analogous to Figure in Papyan et al. (2020), but on networks trained with MSE Loss. Results demonstrate that networks become more robust when trained beyond 0-error. The median improvement in the robustness measure in the last epoch over the first epoch achieving zero training error 0. 1662; the mean improvement 0.9278.
18

---

Published as a conference paper at ICLR 2022
Croen@atropplooo
2.8e-403
VGG
J.8e: 400
1.h-42
1.hodd
1.0e-433
Reehet
14b022
1.8ei00
1.8e.-0
DomeeNet
1.0e-011
1.--03
Spach
Epech
Epods
Xpoch
Kpech
Figure 10: Activation collapse under MSE loss vs. CE loss: Comparison of activation collapse observed in this paper for networks trained under MSE loss (Figure 7 with that observed in Paapyam et al. (2020) under CE loss. Networks trained with MSE loss tend to achieve faster activation collapse than those trained with CE.
Enttopptt Lowe
M8RLas
2e-008
Do:
2c:03
Se-31
De 00
3e-02
1c-90
2e-01
Joy
2c-22
2--01
Se-400
So-01
10-40
Se-32
1n.01
2e-12
1e-02
Sm-t0
Se--00
9+-01
Sc-91
2e-00
To-ttt
Ze-31
So-122
1w-01
26-01
1e-01
2e-00
De-01
Sw-12
3c42
20-00
Im01
2e-22
Be-05
1o-2
1000
300
310
150
3001
200
300
300
Kpoeih
Fyech
Spoods
Figure 11: Adversarial robustness under MSE loss vs. CE loss: Comparison of adversarial robustness observed in this paper for networks trained under MSE loss (Figure 9 with that observed in Papyan et al. (2020) under CE loss. Robustness tends to be sometimes several magnitudes better when the networks trained with MSE loss than with CE loss.
19

---

Published as a conference paper at ICLR 2022
Crw.EEntrogo.ww
MSKLOM
MINSST
MINIS
SVEN
CIFARIO
SILI0
1x-43
Je 91
26-43
Sed1
Se-22
tn
200
2003 1101 2401 200 1101 2401 200 200
Figure 12: Activation collapse on test data for both losses: Activation collapse observed on test data for models trained with MSE loss (from this current paper) and CE loss (posted by Paayyan, Han, and Donoho (2020) on Stanford Data Repository). On the test data, activation collapse still visibly occurs on multiple dataset-network combinations: Albeit the rate of collapse is much slower on the test data compared that on the train data, and the plotted measure (described in Figure 6 of Papyan, Han and Donoho (2020)) the last epoch is larger than that the train data. Also interesting is that the value at the last epoch is roughly monotonic with the difficulty the dataset. See discussion in Section A.8
20

---

Published as a conference paper at ICLR 2022
A.7 EXCEPTIONS: STLIO-RESNET AND STLIO- DENSENET
The ST-eo-ResNet and STL.eo-DenseNer dataset-network pairs stand out among the experiments described in this cectio-ssometime displaying outlier behavior either by converging more slowly to Neural Collapse or by exhibiting trends inconsistent with those found in other dataset-network pairs.
STL10 stands apart from the other canonical datasets for multiple reasons and is "less-typical" benchmark compared to more "compulsory" datasets like CIFARIO. In this particular case, we hypothesize that its outlier behavior might be caused by the size of the STL10 images (96 x 96 compared to the 32 x 32, after padding, of the other datasets)-leadinnn problem, which, in turn, induces a harder non-convex optimization problem, which might make SGD less likely to converge to a useful optimum or might make the convergence much slower and harder to observe under fixed computational budget.
to a higher- digeer-di
It was previously noted in Hui & Belkin (2020) and Demirkaya et al. (2020) that more challenging classification problems- in those projects, problems with more classes to be labeled- -myy require modifications to the MSE loss. We decided that such modifications are unnecessary in the ten class problems examined in this paper's experiments. But perhaps the outlier nature of the TTL10-ResNer and STLIO-DenseNG combinations signal that the MSE loss modifications proposed by Hui & Belkin (2020) and Demirkkya et al. (2020) ought to have been used.
A.8 OPEN QUESTIONS: WEIGITT-DECAY, BATCII NORMALIZATION SGD, GENERALIZATION, TEST DATA
The experiments in this paper as well as those in Papyan, Han, and Donoho (2020) were conducted under the canonical setting where deep nets were trained using SGD with weight-decay and batch normalization-lladlee
one to wonder about the roles that these particular ingredients play on the NC phenomena. Additionally, most observables focus on the train data-raising the question of how the NC phenomena behave on test data as well as their relationship to generalization. We find these questions intriguing, but feel each deserves careful experimentation outside the scope of this paper.
Nonetheless, we note here that several existing papers have already begun insightful investiga- tions in these directions. For example, weight-decay, weight-normalization (as a proxy for batch- normalization), and SGD play key roles in the analyses of Banburski al, (2019) : Poggio Liao (2020a b) that, along other things, lead to the prediction of NC in homogeneous deep nets. [Banburski ct al. (222)) also explores the connection between NC and margin distributions with generalization.
Another example is Zhu et al. 2021) in which the authors conduct several experiments on ResNets trained on CIFARIÓ 10. On their models, the authors examine NC-related properties relative to the train data, test data, and randomly labeled data; they also conducted ablation studies varying control parameters (weight-decay, width, etc.) and the optimization algorithm (SGD, ADAM, L-BFGS). Based their results, Zhu et al. (2021) propose
prooohii-rooiinggg. conjectures on the role of each of these components (see Section E.7 for a brief survey of their findings).
As preliminary exploration, include Figure 12 showing the variability collapse (NC1) behavior on test data for the networks trained with MSE loss (used in this paper) as well as those trained with CE loss (6 by Papyan, Han, and Donoho (2020). From Figure 12 we that variability collapse occurs much slower on the test data than on the train data. Although not shown here, the other NC phenomena behave similarly.
In this Appendix, we show Figure 12 since it is concise, within reach, and may be of interest to readers who share our curiosity. We refrain from presenting the entire series of multi-figure NC measurements on test data because it is outside the scope of this paper and would significantly lengthen this appendix. We also refrain from speculating on any underlying explanations until more meticulous experiments are conducted.
For example, one sense in which STL-100 is clearly different from the other datasets is the comparatively small number of laheled training examples per class.
16The CE test data results were released by Papyan, Han, and Donoho 2020) on the paper's corresponding Stanford Data Repository entry.
21

---

Published as a conference paper at ICLR 2022
B THEOREM 1
Theorem 1. (Decomposition of MSE Loss) The MSE loss, L(W, H), can be decomposed into two
terms, L(W, H) = Lis(H) + Lts(W, (W, H), where
1 Ave Wishi, yi.el + X W :.
Lis (
and
(is(W W. H = tr W Wis) (Er + Mato + 21) (W- Wis) }.
2
Proof. Recall the MSE loss:
X W 1
C(W, H) Ave Whec yr.ell2 1.0
(12)
Consider specific extended activations matrix H; the least-squares solution W minimizing L(W, H) with H kept fixed must obey the following first-order optimality condition:
Ave Ave(Wist Wishi,c yi + +WWs= 0.
(13)
From Proposition
T in the main manuscript, the solution to the above is given by
Wis C-1 (Er + PaAo + AI)-
The loss can be rewritten
1 Ave Wishie - Vi.o (W- Wis) W
2
Combining the above with Equation 13 gives,, after rearranging:
1 Ave [issc 3i +
Ave I(W Wis)hi.ol22 ter Wis(W Wis) + A W W
2
i.c
which is equivalent to
1 Ave Wishie v1.el12 +
Ave (
Wssh 11 Wisl? + 2 Wis 1
2
2 ie
Using the above, the loss can indeed be decomposed as
L(W, H) LLs(H) (H) Lts (W, H)),
where
A Wishiie vrael Wisl
Lis H
Ave 3,c
(14)
and
L W. H
Ave IW Wis he.l2 + W W ie
2
1
ir I
( Wis) (T + Moji + 21) (w Wis)
2
This completes the proof.
C THEOREM 2
Theorem 2. (Decomposition of Least-Squares Component) The least-squares component, LLs( H), of the MSE decomposition in Theorem can be further decomposed into Lis H) Lxxi H Lycz H), where
1 tr WTs Ew + AI W
LuCi H
2
H) =
WisM 1|
2C
22

---

Published as a conference paper at ICLR 2022
Proof. Under Euclidean distance, perturbations the extended activations matrix H which affect only the or global-mean -ean are orthogonal to perturbations which affect only the within-class covariance. Using this, LLs( H) can be further decomposed via the Pythagorean theorem:
H = Ave LLS
(hs, - m.)l + Wisli Ave Wis yr.ell..
The first and second terms above merge into LNC1 H ) while the third term becomes LNC2S HH):
CNcI(H) =
2 tr WLs Σ + AT) Wis]
(15)
1
Exc23 H) 2C
Wis M I|[F.
This completes the proof.
C.1 INTUITIONS FOR THEOREM 2 IN UNEXTENDED COORDINATES
For intuition, consider when X = 0, i.e. the no weight-decay case. Proposition | in the main text gives
-
(16)
where M E PxC the matrix with columns μec Mcg. Returning to the unextended coordinates, Lxci H) simplifies
LNCi (H) = to WisEw Wis).
(17)
where H R PCCN has columns hicc NG. The term LxC23 H) also simplifies instructively to
Excua
20 W.s $ $
(18)
where + RCCC is the standard Simplex ETF:
$ = I I1T
C.2 ADDITIONAL INTUITIONS FOR THEOREM TERMS
The expressions in Equation 13 and Equation 17 further evoke the intuition that
Lxci H is variance term that goes to only under activation collapse (NC1)
More specifically, we see that, with the class-means held constant, the only way to have Lxci (H) 0 is for Σw O. Similarly, by examining Equation 18 we see that
Lxcza H) quantifies deviations of WisM-ia M-iie. the matrix of class-mean predictions- from the standard Simplex ETF.
Under (NC2) and (NC3). both WLs and M tend to jointly aligned ETF's, possibly in an alternate pose; so Wis + $UT and M >> U for partial orthogonal matrix U satisfying UTU Ic- Since $2 4. (NC2) and (NC3) together demand WisM + d. Thus, Lxcra (H) can be interpreted as semi-metric reflecting the distance from achieving (NC2) and (NC3).
D THEOREM 3
D. IMPLICATIONS OF INVARIANCE
In this section, will diseuss in more detail how the invariance observed Section 3.1 leads the the continually renormalized gradient flow (Equation 5). In particular, we have seen in Section 3f of the main text that, on the central path, both the predictions Wis (H) H and the MSE loss
23

---

Published as a conference paper at ICLR 2022
H M XeM
Figure 13: Fiber bundle. Any full-rank features matrix, H. has a representative element, X, on X. For any X X, a fiber is the X i.e. {AX: A S} (the Minkowski set product), where S is the set of symmetric positive-definit definite matrices. Features on the same fiber generate the same class predictions and the same MSE loss. The optimization described in Sections D.1 D.3 moves fiber-tu-fiber.
L(Wis(H), MH) invariant under the transformation H AH when A is invertible matrix. Since interest mainly lies with A = W will restrict the intuitive discussion in this section to positive-definite A.
Now, consider the set H of features H with with non-singular within-class covariances. We view H as a fiber bundle [7] where-on each fiber-live (apparently) different activations, H, that (in fact) generate the exact same class predictions and the exact same MSE loss (see Figure [3}. The base space-we we will denote it Xan be taken to be the collection of X E H where E (X) I. Every
H E H. is representable as H AX where A = Σw H and En (X) = I. The activation matrices in the base space might be variously called whitennite "nomalized.."ttaa"i"ii""""" or "sphered." We shall call X' a normalized features manifold 18
Invoking invariance, can make the following observations about our optimization task:
1. If, specific H, happens be normalized set of activations (i.e. H, X' is currently located in the base space X), there is no performance benefit to leaving the base space. There is also no performance benefit to moving along fiber; only by moving from fiber-to-fiber can we improve performance.
2. In some sense, waste time except when jumping from tibertt-fiiber; might prefer to stay in the base space all the time by forcing the dynamics to jump from fiber-to-fiber.
On the other hand, we started with the viewpoint of studying gradient flow of the original H. So, we consider the following natural model for the application of gradient descent in X:
1. For a given initial feature activations, Ho E H. we renormalize- a starting point Xo X: here X,- = E (Ho)Ho Ho- Class predictions and MSE loss performance for the MSE-optimal classifier do not change from this renormalization.
2. We compute the usual gradient of MSE loss, at Xo. obtaining a step AX, = V Lis where is step size.
An larger space could be considered, where Ew Ew(H) ) necessarily full rank, and then other components of this space having covariances rank C, 1,..., could be considered. We just discuss the full-rank case here.
Thiss is sometimes called the Generalized Stiefel Manifold and will be defined formally Section D.2
24

---

Published as a conference paper at ICLR 2022
3. We take step, going
= + AXu.
4. Hi will necessarily be normalized. We map back along whatever fiber have landed upon, obtaining a corresponding point in the base space X', call this X1. Here,
X = Mw H1
Class predictions and MSE loss performance do not change from this renormalization.
5. Repeat steps 2, 3, 4 at X,, obtaining thereby X2; and so on.
This process might be deseribed as gradient descent with continual renormalization:
1. Renormalize the initial features activation matrix;
2. Compute the ordinary gradient of MSE loss and take the gradient descent step;
3. Renormalize again after each such step; and
4. Repeat steps 2 and 3.
While the continual renormalization process differs from usual gradient flow, it is both intuitively understandable and sensible. See discussion of assumption (A3) in Section
D.2 ALIGNED SNR COORDINATES
Analysis of continually renormalized gradient flow can be simplified, without loss of generality, by applying change of basis in both row and column space:
Definition 3 (Transformation into Aligned SNR Coordinates). Consider the SVD decomposition of the SNR from Definition 2 ootputting left singular vectors U e IK PxC and right singular vectors V CCCC For any matrix Z IR" pxx define the transformation Z + U ZV as the transformation into aligned-SNR coordinates.
The aligned-SNR coordinates so-named because it diagonalizes i.e. aligns the SNR matrix: Definition 4 (Aligned SNR Matrix). Consider the SVD decomposition of the SNR from Definition 2 Combined with Equation 8 observe that
diag (1w)) = = MV
We call Ω the aligned SNR matrix.
The following facts about the aligned SNR matrix becomes useful later in our derivations: Observation 1 (SVD of Aligned SNR Matrix), The SVD of the aligned SNR matrix itself is simply
C-1
Ω
Σ w
j=1
with the canonical basis CC its singular vectors.
As discussed in Section 3.2, the [we]e=] CC-1 that comprise St are decisive for understanding separation performance. In fact, when A=0, the MSE loss can be entirely characterized by the SNR singular values on the central path:
Lemma (Spectral Representation of MSE Loss). When X 0, MSE loss obeys
C(Wis( H) Lis Lis(Wis H
H
Σ
j=1 w C L (1W1)
2
on the central path, and its decomposition obeys
1
LNCi H
Lxci (w]))T]))
2
CC W3
w w
- ))
LNCIa
-
-
LNeza (4w))5=))
25

---

Published as a conference paper at ICLR 2022
Proof. Using Equation 16
Equation 17
and Equation 18
the loss on the central path equals
L(Wis; M)
||c
M E
M +
(19)
2C
Observe M ΣT M and d are simultaneously diagonalizable since they share the same , null space and & has C equal eigenvalues. Also, observe that
Ex Ew C MM
Let 52 be submatrix of the aligned-SNR matrix Q corresponding to the non-zero singular values i.e. Ω diag ((w))==²)) After taking sinultaneous-diagnnlizztii and canceling partial orthogonal matrices, Equation 19 can be written solely in terms Ω:
ā T (Ω +CIc-1) -1 n
Ic-1 0
( - 20
+ ir Ω
(Ω + +CIc-1)~20
2
F
This can be further simplitied into
( -1)
w/
2
w/
[((w)}(=))
Ic
+
j=l
2 Σ w/+C
(20)
This completes the proof for Lemma 1
Thus, the aligned SNR matrix, U Σ " MV. maximally simplifies analysis by reducing the vanilla
SNR matrix to a diagonal matrix (Definition 4 whose entries determine the MSE loss (Lemma I
We introduce the alignment of features into their own notion of aligned SNR coordinates:
Definition (Renormalization to aligned SNR Coordinates). Consider E PXCN with corresponding SNR left and right singular vectors U and V (Definition 2. Then, define the SNR- aligned renormalization of the features as
X == T (Ew)- H(V In)= UT (HCHT) H(V @ IN)
where & is the Kronecker product and C' is a class-centering matrix defined as
(IeN AYT).
C= CN
(21)
As discussed in Section 3.1, the transformation H Ew² H does not change the predictions the least squares classifier. Hence, it does not change the MSE loss. It is casy to check that this still holds for X i.e. invariance is preserved by the change into SNR-aligned coordinates.
It is also easy to check that the sphericity of the within-class covariance is preserved as well: Observation 2 (Sphericity of Within- Sithin-CClass Class Covariance of SNR Coordinates).
XCXT Ic
Finally, routine applications Definitions and canceling out of (partial) orthogonal matrix multi- plications in aligned SNR coordinates lead to the following simplified representation for Z and its differential:
Observation 3 (Relation Between SNR Coordinates and SNR Matrix). The aligned SNR matrix. Ω (Definition 4 can be expressed as function of the features in aligned SNR-coordinates X (Defiittio as simply
1
2(X) =
T
N
26

---

Published as a conference paper at ICLR 2022
Moreover, let dla, (:) TxX +R be the differential 1-form 1 associated with the ij-th entry of
S2. Define di (+) TxX + RxxC such that, for some Z E Txx, each entry matrix of the matrix di (Z) d (Z). Then, it follows that
Uu (Z) N TY E
From geometric perspective, the transformation in Definition maps the features, H, X belonging to a Normalized Features Manifold: Definition (Normalized Features Manifold).
X { E RCXON XCX = Ic}
To understand the dynamics of the singular values of Ω2-wiich is connected to the dynamics of the features through Lemma D- we will analyze the gradient flow on this manifold. To this end, first identify its tangent space at a particular X as well as the projection operator onto that tangent space. Proposition 3 (Tangent Space of Normalized Features Manifold).
Tx (Z E RCCN XCZ + ZCX = 0}
Proposition 4 (Projection Onto Tangent Space of Normalized Features Manifold).
I x (Z) Z 2 [XCZ + +ZCX X
D.3 CONTINUALLY RENORMALIZED GRADIENT FLOW
In fact, gradient flow on the normalized features manifold (Definition (6) is the continuous analogue of the intuitive, discrete algorithm in Subsection D.T. In particular, the intuitive algorithm consists of gradient step, H Xo + A Xo followed by mapping to the base space of the fiber bundle, X E(H, - H. For small A these two steps are in fact equivalent (up to negligible term) to taking X' X-projecced gradient step on the manifold X' as proven below:
Lemma 2. Assuming Xo X, renormalized gradient step.
x= =WE (H,)H = =E (Xu + AXo) (Xo + AXa),
equivalent, up to an O (1AXx11²) term, to a T X-projected gradient step, iee.
X1 =xo +Ixxx +0 (14x,11²).
Proof. Notice that
X = (H,
= =(Xo + AX,) + AX (Xo +X.+AX)
=II+AX,CX; X.CAX, +AXxCXXX) (Xo + AXo),
where in the last step we used our assumption that Xo E X, i.e, XOCX, By Taylor's Theorem,
1 A A 0(1AA13)
(I A1+
2
Therefore, we get
X; I (AX,XX. + XXCAX AXXCAX)) + +O (1AXXI1)) (Xo AXx)
Xx + AX CA - X,CAX,) Xo + +0 +0(11XXX1)) (14X,11²)
=X, + ( + + (IAX.I1)
where the last step follows from Proposition
4
27

---

Published as a conference paper at ICLR 2022
As transition from discrete gradient descent continuous gradient flow, the step size - 0, and the residual O (IAX0ll*) in the above lemma becomes negligible, since AXo - Vis (o)). This motivates us to study, in the next section, the continually renormalized gradient flow (as defined in Equation of the main text) of X on X':
X = -xxxx (Vxhis(X)),
dt
D.4
PROOF OF PROPOSITION 2 (GRADIENT FLOW IN ALIGNED SNR COORDINATES)
To prove Proposition 2 we first set up some auxiliary notation and lemmas. Denote the j-th row (transposed) X and Y. respectively, by
I =
= X ej
(22)
and
Y ej.
Recall that-when differentiating SVDs the derivative of the j-th singular value, wj, only depends on the j-th singular subspace (see Equation of Townsend (2016)). As consequence, we will see in the following lemma that, for any differential dX, the corresponding duj only depends on dæy,
Lemma 3 (Derivative of Singular Values With Respect to SNR Coordinates). Given the differen- tial I-form dwy (-) x - IR, the following holds for 19
19
dx E RCXCN
duy (II [xxxx(dx)) (~e v -w dæy,
Vj = 1, lj.....,C C.
where (consistent with Desinition! 2 we adopt the convention that wc=0 is the O-th singular value of the SNR matrix.
Proof. Without loss of generality, assume that X is represented in aligned SNR coordinates (Section D.2. Using Observation and the differential of the singular value decomposition (see Equation 17 of Townsend (016)):
dwy (Ixxa(dX)) e_ di (IIxxx(dX)) ej,
(23)
where dl (-) is defined as in Observation
Next, Observation 3 implies
dn (11 N lxxxx(dX) YT.
(24)
N
Using the projection onto the tangent space, given in Proposition above,
11 x(xx+ dx 2 ( C + XCdX )
(25)
Combining Equation 23
Equation 24 and Equation 25 we obtain:
dwj (Ixx = Uu (II x (dX)) ej
(26)
c e Il x(xx(dX) TTe,
1 - [X (xx- (X + xCaxT) x rTe,
Notatio:: For real-valued functions f. we use (.f()) Xx + R to denote the associated differential I-form i.c. the function outputting the directional derivative of in the direction of the argument; For matrices succeeding parenthesis-to denote the matrix differential (ef. Townsend (2016). These notions are equivalent: df (.) can be represented a vector df the dual-space of Ax When Z is a function of some vector v, dZ can be represented as a collection of 1-forms by defining d dZ.g (-)
Z mm
dZ
aZ.g .) for cach entry Z.j.
28

---

Published as a conference paper at ICLR 2022
Moreover, using Observation
and
simplify these expressions into
N @exdxTTe e, -N y dx,
1
e dxxx XoCCxXY@@ ej = dx dx e, wya C dæ
N
1 XCdx XY ey =uee XC ej w Cdæy.
Ne,
Finally, substituting the above three expressions into Equation 26 we get:
duy (IT (ITxxx(X))) = ( N v w c dæy,
which proves the claim.
Using Lemma 3 we can now obtain the dynamics of æ; under gradient flow: Lemma 4. Under continually renormalized gradient flow (Equation 5).
d n (C+ (ND) -W 1,..,...,C C,
wi
dt
where (consistent with Definition 2 we adopt the convention that wc=0 is the C-th singular value of the SNR matrix.
Proof. Recall æ E CCN is the j-th row of X E RCXON which we assume to be in aligned SNR coordinates without loss of generality (Section D.2p. Applying the flow definition in Equation 5 to the i-th element of zy gives
d dt F/
dL dx
B,
Txxx
e,
-(IITxX
dLLs
1lTxx
eejey
dx
dLLss
[TTxx(e;e]
dx
where dGss E RCXON is the standard derivative within the ambient RCXCN -space, e; E RCN and dx ej E R are the canonical basis vectors, and (.,)) is the Frobenius matrix inner-product. Next, R satisfies the following:
observe that the differential I-form dLLs (-): Tx -
dLLS dx
dLis (2)
2
VZ E TxX.
Taking = II x ( then leads to
d
dt 2yz = dLis (Ixx
C-1 dLis dwk (TTxx(eeee))
dw
k=1
dLis dw, (ITxxx(e,ee))
dwy
where the second step follows by the chain rule, and the last step follows from Lemma 3 in that dwv (IIxx((e,e,) 0ff * j. Moreover, Lemma 3 also gives
dwy (Trxa(eeeF)) = f w.n
29

---

Published as a conference paper at ICLR 2022
It then follows that N - wy Cx,
d dt
dLis
2j
dwj
Wj (C (C+
No w w,CCx,))
where the last step follows from differentiating the expression in Lemma
The gradient flows of æ; induce the dynamics of wj. which are described below.
Lemma 5 (Induced Dynamics on SVD of SNR). Under continually renormalized gradient flow
Equuaiion 5). the dynamics for the j-th non-zero SNR singular value (Definition 2
d 1 W
dt W/ +C
Vj 1j=L...,C-1.
(27)
Proof. Without loss of generality, assume X is represented in aligned SNR coordinates (Section D.2 Observation [ the differential of the singular value decomposition (see Equation 17 of Townsend (2016)), and Observation collectively imply that for any
20 dx E TxX,
1 N
duy (dx) = e, di (dX)
dx e,
dx
Ny dæy.
(28)
Recall that the differential of w/ only depends on the differential of æ; (see Lemma Then, can apply the chain rule:
dwj dæy day di N (C Wj
1 N wa w,Cxx;
di W/
(29)
where the last equality results substituting-in in Lemma and Equation 28 Since YYT NI.
N2 UJ y, N
Applying the same relation and Equation 21
2/1 Cy, e, XCY e, =
= CN x. (I I
1
(..Lrr)) rTe, =0.
1 INN
Combining all the above equations, we obtain
d गो W/ N (C w
W
The closed- form given Proposition now directly follows.
Proposition 2 (Dynamics of Singular Values of SNR Matrix). Continually renormalized gradient flow on the central path (Equation 5) induces the following closed-form dynamics on the SNR singular values (Definition) 2):
C1 lolgow (f) + (t) + c1w -
82 0, for all
j j= =l,,.,
(9)
C1, and are positive constants independent of j. and aj is constant depending on w,)
Proof. Follows from symbolically solving the ODE in Lemma 5 with routine methods. The constants C2N, 02 CNN, and c3 N
For our goal of applying the chain rule in Equation 29 we restrict our attention to just dX Tx
The definition of renormalized gradient flow Equation 5) ensures that will always be in Xx
30

---

Published as a conference paper at ICLR 2022
D.5 PROOF OF COROLLARY
Corollary I (Properties of SNR Singular Values). SNR singular values (Definition) 2 following the Equation 9 Odynamics satisfy the following limiting behaviors:
1. lim w,(t) oc
and lim
2010 =1 Vi/es
for all j I,....C.. 1.
max; wi (1) 2. lime+vo miny (3)
1.
Proof. As tends to infinity, the right-hand side of Eqquatin 9 diverges to infinity and therefore so does the left-hand side (LHS), As the LHS approaches infinity, the logarithmic terms become negligible compared to the dominant quartic term, implying (f( + co. Since wi((t are sin- gular values, they must be non non-negative- neggaiie--immlyynng (t) + oc. Based on the same argument,
for all j. Since the constant C3 is independent of j, it follows that observe that lim w,(
lim max, (1( 1.
miny (t)
D.6 PROOF OF COROLLARY
Lemma 6. Under continually renormalized gradient flow (Equation 5), the left and right singular vectors of the SNR matrix remain constant i.e. they are independent of
Proof. Without loss generality, assume X in represented aligned SNR coordinates (Section D.2p. Recall that, in this coordinate system, the corresponding SNR matrix Ω - XY is diagonal, and the left and right singular vectors are simply partial identity matrices.
To show that the singular values of the SNR remain constant, then suffices to show that dn dx Y
dt di
a diagonal matrix as wel
Lemma gives
1 D,Y -0y-D. -DXC x N
dt
where Dj and D2 are CxC diagonal matrices with the diagonal entries {
w/ (Ctugy
and
respectively. Then,
(C+ug)
d Ω = dt - - - -DxXYY = dt
where, in the last equality, used the easy-to-check identities that YY T and CY=0. In more detail, the first identity follows from the fact that is the one-hot label vectors stacked as columns; the second identity follows from the facts that C is the class class-centering matrix, and the one-hot label is the same for examples from the same class. Thus, we have shown di which concludes our proof.
ar is diagonal,
Lemma 7. A matrix E E PxC is Simplex ETF if and only if
1. E has exactly C-1 non-zero singular values, which are all equal.
2. has rank-1 nullspace spanned by the ones vector, i.e., ELc-0. In other words, E has zero-mean columns.
Proof. First, recall from Papyan, Han, and Donoho (2020 Definition 1) that a PxC matrix E is called Simplex ETF if it satisfies
E - EEE-a
C 1 (-
- 1o1))
--11o1)
21 Intuuiiivll changing the diagonal SNR matrix by diagonal matrix increment implies the updated SNR matrix is still diagonal. This, in turn, implies the left and right singular vectors of the updated SNR matrix still the left and right partial identity matrices, respectively.
31

---

Published as a conference paper at ICLR 2022
for some scaling a>00. We now prove the equivalence.
Simplex ETF implies 1-2: Consider the SVD decomposition E-UESEVÉ where UE is Px(C-1) partial orthogonal matrix satisfying U SE is the (C-1)x(C-1) diagonal matrix of singular values, and VE is a (C-1)xC partial orthogonal matrix satisfying V Vg=I. Since E is a Simplex ETF, by definition,
C (
E =a
and according the SVD decomposition
E VESSUgUBSEVi VES VE.
Therefore,
VEST VE -
C
Notice that the right-hand-side has C equal singular values. This implies SE thus, as well- has C- equal singular values. In other words, Sø is a (C--1x((C-1) diagonal matrix equal to diag(s..... s) for some scalar s>0.
Next, notice that on the one hand
(
- 1c =0=
C 1
E Elc Elc== -0
0-1 0-1 1cle
On the other hand,
Elc -
Combining the above, we get VE VEVILC-O 1c-0. Since VEVE is a CxC matrix of rank C-1, we deduce that the rank-1 nullspace VE spanned by the C-dimensional ones vector, i.e. V c==0. Consequently, E has zero-mean columns since Elc=Ug.S&Vi l==0.
1-2 implies Simplex ETF: Assume E has exactly C-1 non-zero, equal singular values as well as rank-1 1 nullspace spanned by the ones-vector, i.e., = Then, B- where
Ug is P x (C-1) partial orthogonal matrix satisfying U
SE diag( s, .... is (C-1)x( (C-1)x(C-1) diagonal matrix for scalar s>0;; and
E is (C-1)xC partial orthogonal matrix satisfying V VE=I and also satisfying E 1c
Therefore,
E E VeSeU VESV VEVV..
Using our assumptions on VE,
VEVE + -
where we divide each of the ones-vectors by VC create a unit vector. Thus, we conclude
EE=S 52 (1-
( 1-
C ( (
where a = 22 C-1. Hence, by definition, E is Simplex ETF.
Corollary (Neural Collapse Under MSE Loss). Under continually renormalized gradient flow Equation 5), the SNR matrix Equation 8 converges
lim 10000 (t)x(()
SNR, V.V.T.
(10)
where U, E 1x- and V. e (C are the left and right singular vectors of the SNR
matrix (Defiitton 2 1=0 corresponding the non-zero singular values; and Wmax (1) is the
32

---

Published as a conference paper at ICLR 2022
largest singular value at time t. Furthermore, Corollary I renormalized gradient flow on the central path leads to Neural Collapse
implies the occurrence of (NCI)-(NC4) i.e.
matrix converges to Moreover, denoting the Kronecker product with C, the renormalized features feature
lim (1000 Wmax (t)
1
Sw, = (.. @ 1
(11)
10--00 of SNR, diverge to infinity and that their ratio tends to one. By Lemma 6 the renormalized gradient flow will not change the singular vectors of the SNR matrix, and combine with the fact that the singular values converge to equality (second fact of Corollary [ - we get the limit in Enquatio 10
Proof. Derivation of Eqquttin 10: Corollary 1 proves the singular values j = -....,C- C
Derivation of (NC1): Let sy(+) denote the j-th singular value of its argument. Then, observe that
Ew.1) Ct
(M,,)) Ew.1)
(Def. of EB,)
Cr
Σ
(M,M,)
Eiv..)
(Cyelic property of trace)
C-1
= = Σ
85
Ei (M,M,)) :i::))
(Trace is sum of singular values)
1=1
C-1
C Σ
8
ب (M.M, Ew
(Def. of pseudoinverse)
1=1
C
Σ
83
M.
M.
C-1
00 0.
(Corollary 1)
By Horn & Johnson 2001
Theorem 1.3.22).
Xy E B,t Ew::) X E1 b,, 0
where (-) denotes the j-th eigenvalue its argument, and the last inequality follows from the positive-semideffiiteess of 0.55 E
00.5 B W.r- Since the trace the sum of eigenvalues, the only way for the trace of Σ E Ew: to tend to zero if all eigenvalues also tend to zero. All eigenvalues tending to zero implies the matrix itself tends to zero, i.e.
lim Σ B,r Ew.e = O, t-000
which is the definition of (NC1) in Section
Derivation of (NC2)-(NC4): Recall that the SNR matrix has zero-mean columns and rank C-1 (see Definition 2). This, combined with the fact that the singular values converge to equality (second fact of Corollary D imply, by Lemma 7 that the renormalized class-means converge to a Simplex ETF i.e. (NC2).
From Theorem of Papyan, Han, and Donoho 2020). we then know that (NC3) and (NC4) follow from (NC1) and (NC2) on the central path.
Derivation of Equation 11 Combining the limit in Equation 10 with (NC1) proves the limit in Equation
E RELATED WORKS EXAMINING NEURAL COLLAPSE
In this section, we discuss the contributions and limitations of seven recent works that propose and analyze theoretical abstractions of Neural Collapse. These works are only available in preprint, and may not yet be peer-reviewed. Thus, they might ultimately appear with very different claims or results. Additionally, works such as Poggio Liao (2020alb) b): Ergen & Pilanci (2020) also analyze behaviors other than NC; will only discuss the parts relevant to Neural Collapse here.
33

---

Published as a conference paper at ICLR 2022
E.1
MIXON, PARSHALL, AND Pi (2020)
Mixon et al. 2020)) considered the unconstrained features model in Equation 2) (without weight decay) where, under gradient flow, (W; H) evolve according to a nonlinear ordinary differential equation (ODE). They followed two-step strategy for studying Neural Collapse. First, they linearized the ODE-claiming nonlinear terms are negligible for models initialized near the origin and and proved the simplified ODE converges to subspace of (W, H) satisfying (NC1) and (NC3). Second, they proved that gradient flow, restricted to that subspace, converges to (NC2).
The assumption of small weights and classifiers leading to the linearized ODE is not aligned with today's s paradigm. Specifically, the most commonly used He initialization (He et al. 2013) is designed: (i) create weights with non-negligible magnitude; and (ii) preserve the magnitude of features, as they propagate throughout the layers of the network, exactly so that last-layer features would have non-negligible magnitude. Moreover, the analysis of Mixon et al. essentially assumes that (NC1) and (NC3) occur much sooner than (NC2).. However, from the experiments in both Papyan, Han, and Donoho 2020) and this paper, there is no empirical evidence that (NC2) happens slower than (NC1) and (NC3) in practice.
E.2
LU AND STEINERBERGER (0200)
While the MSE loss provides a mathematically natural setting for analysis, the modern paradigm in multi-class classification with deep learning involves training with CE loss, which is more challenging to analyze than MSE.
Lu & Steinerberger (220)) studied the (one-example-per-class) unconstrained features model with CE loss:
min CE(W, MM) W.M
s.t. ||wal|2 = | 1.
Since under linear separability the CE loss can be driven arbitrarily close zero. just by re-scaling the norms of W and M, the authors further imposed norm constraint on w, and f4.. Lu & Steinerberger observe that the global minimizer this optimization problem only achieved once W and M are the same Simplex ETF. This derivation is suggestive, but it does not identify closed-form dynamics which would get gradient flow to such a global minimizer, nor does it address the rate of convergence to Neural Collapse. Additionally, the constraint Me possesses to standard deep net training-whar where procedures often control the norm of the weights W, but not features H nor class-means M.
immediate direct analogy
E.3 E AND WOrTOWYTSCH (020)
E & Wojtowytsch 2020 also consider the unconstrained features mode
22 with CE loss,
min
CrossEntropy (W H)
s.t. ||]]| S 1, hs,ell2 1,
where they adopt a more technical, spectral norm constraint on W specify their model.
Building on the results Chizat & Bach 2018 8 2020). E & Wojtowytsch also construct a simple counter-example showing that Neural Collapse need not occur in two-layer, infinite-width networks which have been the focus of intense recent study in the theoretical deep learning community (Mei
al. 2018 Rotskoff & Vanden- -annen-Eijndenj 2018; Arora al., 2019). Thus, [E & Wojtowytschi counterexample suggests the alternative perspective that, despite the expressiveness of wnddnite-width, two-layer networks, such abstractions do not capture key aspects of trained deep nets.
As with Lu & Steinerberger (2020), standard deep net training does not possess any direct analogies for constraining the norm features (as opposed to weights)- noor are there any paradigmatic regularizations that correspond to controlling the spectral norm on W. Moreover, the work does not characterize any closed-form dynamies or the rate of convergence to Neural Collapse.
22hh model is unconstrained in the sense that the features are allowed to move directly with gradient flow and are not constrained to be the output of forward pass-non not in the sense that there are no constraints on the optimization problem.
34

---

Published as a conference paper at ICLR 2022
E.4
POGGIO & LIAO [2020 B
(WITH BANBURSKI)
Distinguished from the simplified unconstrained features models in the previously mentioned works the theoretical analysis of Poggio & Liao (2020a b (in special section, co-authored with Andrzej Banburski).
The authors study deep homogeneous classification networks, with weight normalization layers, trained with stochastic gradient descent and weight decay. This is much closer to today's training paradigm, but the setting still differs from the one in which Neural Collapse has been empirically observed in Papyan. Han, and Donoho (2020) and in Section A of this paper. In particular, they replace batch normalization with weight normalization and consider deep homogeneous networks; homogeneous networks can not have bias vectors nor skip connections, which are present both in ResNet and DenseNet. Moreover, the work gives explicit deseriptions of neither the dynamics nor the rate of convergence to Neural Collapse.
E.5 ERGEN & PILANCI 2020
While the above-described works tend to focus on either the used-in-practice CE loss or the theoretically-insige
MSE loss. [Ergen & Pilanci] (2020) observed that these are both instances of the general class of convex loss functions and, thus, one could derive insights from the classical convex analysis literature. Moreover, compared to Mixon et al. (2020); Lu & Steinerberger (2020); E & Wojtowytsch| (2020), this work studies the optimization starting from the second-to-last layer features rather than the last-layer features. In particular, the authors to show that NC emerges in the optimal solution of an equivalent proxy-model to the following optimization:
strong-dualitg argument
WE (BNy,a (WL-IHL-1)) x) (1H11 + |all + 1/
min
where L(-) is general convex loss, H are the second-to-bee layer activations, WL-1 are the second-to-last layer weights, Wr. are the network classifiers, Y are the training targets, weight-decay parameter, BN, (-) is batch-norm operator parameterized by and y, and (+)+ is ReL.U. The incorporation of batch-normalization and weight-decay ensures the existence of bounded, well-defined optimal solutions-sec serving a similar role to that of weight-normalization and weight decay in Poggio & Liao 2000a b) as well as the norm constraints in the other aforementioned related works.
a
Since -roong-duality only characterizes properties of the converged optimal solution of an optimization model, Ergen & Pilanci| (2020) does not provide insights into the dynamics with which that solution is achieved which training.
E.6 FANG, HE, LONG, AND SU 2021
In Fang al. (2021) the authors introduce the (N- Mayer-ee-deee model in which one considers only the direct optimization of the N-th-to-last layer features of a deep net along with the weights that come after the N-th-to-last layer. The motivating philosophy is that, after raw inputs are passed through some initial number of layers, the overparameterization of those layers would allow us to effectively model the N-th-to-last layer features as freely-moving in some subset of Euclidean space. In this terminology, the concurrent works of Mixon et al. (2020); Lu & Steinerberger (2020); E & Wojtowytsch (2000) on the unconstrained (last-layer) Teatures model could be considered instances of a saaee--peeeed model; while the model of Ergen & Pilanci (220)) could be considered as a 2-layer-peeled model. This perspective is attractive because it gives a name and organization to a common modeling philosophy behind the above-described body of independent works. For comparison, the work of Poggio Liao 2000aa b) non-example of layer-peeled modeling as considers optimization on the weights of homogeneous deep nets and not the input features.
Fang et al. (202I) then analyzes a convex relaxation of the 1-layer-peeled model-with norm constraints on the weights and features-into a semidefinite program. Not only do the authors show that this model exhibits Neural Collapse in the canonical setting of balanced examples ceample--per-cieee per-class, but they also analyze the behavior of this model under imbalanced classes. While, in the imbalanced case, one would intuitively expect the Simplex ETF to "skew" to have bigger angles around over- represented classes and smaller angles around under-represennte ones: Fang al. (222)) identifies the
35

---

Published as a conference paper at ICLR 2022
surprising phenomenon named minority collapse- in their model where, when the imbalances pass a certain threshold, the last-layer featuree and classifiers of the reper-represented classes collapse to be exactly the same. However, their work does not provide closed-form dynamics rates which collapse-in in either the balanced or imbalance acceus
E.7
ZHU. DING, Zhou, LI, YOu, SULAM, AND QU (2021)
In Zhu et (02I), the authors examine the following unconstrained features model:
Aw 2 [ λH
Ab 2
min CrossEntropy (WH - b, b)
H
|
where (W, b) are the classifier weights and biases, H are the last layer features, and (Aw AH. Ab) are weight-decay parameters. On this model, the authors not only prove that all minima exhibit Neural Collapse but also that all local minima are global minima. In comparison to our current paper. Zhu et al. (202T) focus on characterizing the landscape of the loss and, thus, do not explore the dynamics and rate at which such minimizers of the loss achieved.
Zhu et al. (202I) also make notable empirical contributions by conducting a series of experiments on the MNIST and CIFARIO 10 datasets trained MLPs, ResNet18, and ResNet50. Their measurements give evidence for the following novel NC-related phenomena in deep classification networks:
1. NC is algorithm independent: NC emerges in realistic classification deep net training regardless of whether the algorithm is SGD, ADAM, or L-BFGS.
2. NC occurs on random labels: NC emerges even when the one-hot target vectors are completely shuffled.
3. Width improves NC: Increasing network width expedites NC when training with random labels.
The authors of Zhu et al., (2021) moreover conducted ablation experiments suggesting that the following substitutions can be made to deep neural net architectures without affecting performance:
1. Weight-decay substitution: Replacing (A) weight-decay on the norm of all network parameters with (B) weight-decay just the norm of the last-layer features and classifiers.
2. Classifier substitution: Replacing (A) the last-layer classifiers that trained with SGD with (B) Simplex ETF classifiers that are fixed throughout training.
While the authors only demonstrated these new behaviors on limited network-dataset combinations, these experiments indeed inspire interesting conjectures about the generalization behavior of deep nets as well as potential architecture design improvements; Zhu et al. (202)) discuss many of these conjectures and related open-questions in detail.
36


Reviews:
[{"review_id": "oO3Tn1yCN5x", "reviewer": "Reviewer_Jnw3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the paper:\n- The phenomenon of Neural Collapse is certainly an interesting and thought-provoking and analyzing it further to understand its premises is certainly worthwhile of publication. In that sense, the contributions of this work are very relevant and welcoming.\n- The experiments seem to corroborate that the intuitions/derivations of the paper are on the right track. I particularly like the idea of decomposing the loss in two terms, one that is the 'optimal loss' and a secondary loss that accumulates the remaining errors.\n- I find the connection with the dynamics of learning a fascinating direction, and I think this is where research should focus on. It would be very interesting if indeed the dynamics of the learning, perhaps by examining the eigenvalues of the signal-to-noise ratio matrix, lead to Neural Collapse. However, I am not sure about this, because in all honesty it was hard to parse the respective subsection.\n- I would spend more time on the central path, which I find an interesting concept. What does it really mean? Are the features H supposed to be 'fixed' or at least 'fixed within infinitesimal time steps'? I believe this is the largest and most interesting contribution, and a good proof why using the MSE is relevant in this context.\n\nThe weaknesses of the paper:\n- A major weakness of the work is writing and structure. While very interesting ideas are in it, and it is clear that there is a worthwhile message, it is very hard to understand in precise detail the claims, so that a 'third' reader can derive their own insights. As one example, the intro is very technical, with lots of forward references, and extends till p 4. It creates a feeling of repetition and unclarity at the same time, eg, proposition 1 is basically defined twice. Lots of different concepts, terminologies, and ideas are mixed and the text often jumps from one place to another, even referencing later parts of the text that have not been read, assuming someone does read a paper sequentially. Another example is that not notation is not always clearly explained. For instance, what is the time t in equation 5? I suppose time steps during training. For the lack of writing clarity alone, I am not sure if the paper should be accepted, it would be a pity for the work itself.\n- It is not clear (at all) what are the 'assumptions' that are made for the sake of the analysis.\n-- For instance, equation 5 assumes that the features H (and thus X) are converged, that is the manifold of the identity-covariance features is fixed? Or, is it assumed that in infinitesimal training steps the features H (and the manifold H) remain roughly equal? In practice, what does it mean to have the loss L_{LS} for given features H, since the features H change during the training? How is this computed in the plots of figure 2?\n-- In the sentence 'As LNC1 decreases, individual activations would need to tend to their corresponding class-mean', how is this conclusion derived? Do you assume W_{LS} to be fixed? Otherwise, W_{LS} can also reduce the loss, no (in fact, that is the point of learning)?\n-- Although that is a point for the original NC paper, I think NC4 is self-evident.\n- Focusing on the figure 2, it shows that NC2/3 is much much smaller. Does this mean that the model basically distributes features 'uniformly' early on (thus NC2/3 goes to zero fast) and from thereon, it tries to group/cluster each class features as much as possible? What about overfitting? How well does the models generalize if keeping the training till zero loss? Doesn't this contradict standard practices, like 'early stopping'? To put otherwise, what would happen if having small training sets, say 50 or 100 examples per class.\n- Isn't the zero-global mean after bias b_LS an obvious conclusion, in that after subtracting the bias, the average is zero-mean? That is the point of the bias, no?\n- How do you obtain the A^{-1} in equation 8? Is this part of some definition, or Linear Algebra? More generally, is equation (8) suppose to derive a result or to state it? How do you go from W(AH) to W(H) A^-1? I think you mean to say that the W operator is linear (matrix multiplication), so the multiplier A can go out, but how do you derive the A^-1?\n- What is the intution behind the signal-to-noise ratio matrix in the off-diagonal elements? Class confusion so to speak?\n- Section 3.3 is very involved and I am not sure I understand how the NC1-4 are derived. It is stated what we are to conclude from it, but no guidance is provided on why this is the case. I am not asking for the proof, but for the interpretation of the results. For instance, in equation (11) we have the \\omega_max in the denominator, while #1 in Corollary 1 tells that the eigevalues go to infinity. Does this imply that the SNR divided by inifinity goes to 0? Generally, I am not sure what am I to take from this section.\n\n", "summary_of_the_paper": "This paper extends the recent work on Neural Collapse, using Mean Squared Error (MSE) instead of CE, as MSE is easier for analysis. With this, the paper shows that the least square loss can be decomposed into one that corresponds to a so called 'central' path (namely a set of optimal tuples (W, b, H) given H for which there is an optimal loss), and a perpendicular loss. The paper shows that the perpendicular loss is much smaller than the optimal least square loss and thus Neural Collapse appears due to the fact that the optimizer focuses on the central path. Empirical studies confirm the findings.", "summary_of_the_review": "All in all, the paper had very nice ideas, but the writing and presentation is suboptimal. This means that it is not ready yet for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SrHhXVjfxY", "reviewer": "Reviewer_9pr9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n--The authors provided extensive experiments to show that NC occurs in DNNs with MSE loss.\n\n--The authors provided motivation and explained the required conditions of the theorems and the limitations of their results. Overall, the conditions of the theorems do not look too restrictive and unrealistic.\n\n--The appendix has an extensive overview of related work.\n\n--In the paper, a new theoretical construct, central path, is proposed and analysed.\n\nCons:\n\n--Overall, the paper is well-written, but there are some places that are not easy to follow, for example, Section 1.1. I would recommend the authors to state the problem and all the necessary notions first and only after this to introduce what NC is. The first sentence of the introduction also does not sound good.\n\n--The authors investigate the NC phenomenon from the point of view of the last classifier layer only and do not consider how DNNs generate the features that are passed as input to this classifier. It is still not clear how the NC phenomena are connected to generalization. While MSE delivers similar performance to cross-entropy loss (CE), it is less used in classification settings than CE.\n\n\nAdditional comments:\n\n--In Section 1.3 I would recommend adding references to datasets and architectures.\n\n--I would recommend adding an explanation into Section 1.2 about how classification performed when the model is trained with MSE loss.\n\n--Could you please clarify for me whether NC4 is an independent condition or it follows from (NC1-NC3)?\n\n--Could you please clarify how condition (A2) is related to empirical observations?", "summary_of_the_paper": "The authors of this paper proposed a theoretical explanation of Neural Collapse (NC) for the DNNs with MSE loss. In particular, the authors showed that MSE loss can be decomposed into a sum of terms that corresponds to NC conditions and proposed the theoretical model (central path) that explains why NC emerges in DNNs with MSE under some mild assumptions.", "summary_of_the_review": "I would recommend accepting the paper. The paper further extends our theoretical understanding of deep learning. The result is new and supported by experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "RerSUaD-4AK", "reviewer": "Reviewer_pHXk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper empirically shows that NC also occurs when training deep networks for classification using the MSE loss on five canonical datasets and three backbone networks.\n- The theoretical part of the paper to justify the NC phenomenon for the unconstrained features model seems rigorous. The decomposition of the loss function and how it is helpful in understanding NC is interesting.\n\nWeaknesses:\n- I think the paper contains too much information to be wised packed in a single 9-page conference paper. Specifically, for one of its main contribution, namely the empirical study of the NC with MSE loss, almost all the supportive experiments are deferred to the Appendix, while the main body of the paper only focuses on explaining the theoretical part. I have doubts on such practice (claim the contribution are two-fold while only mainly presenting one of them in the main paper). Note that the authors also admit in Section 1.3 that the empirical study is too long to be included in the paper. \n\n- Some statements need more clarification. In the legend of Figure 2, the term \"Lperp\" should be referred to as \"L^\\perp\" in the caption. Also, in the caption it says \"early in the training, L^\\perp becomes negligible compared to the dominant term LNC1\"; however, I don't see this from the figure: for many of them, the L^\\perp curve is on the above of LNC1 during the early phase of training. How to explain this?", "summary_of_the_paper": "This paper studies the phenomenon of Neural Collapse (NC) and empirically shows that it occurs during the training of deep networks with the MSE loss. Then it theoretically analyzes NC with MSE loss by decomposing it and introducing the notion of central path. It shows a closed-form dynamics predicts NC in this setting.", "summary_of_the_review": "The strength and weakness of the paper are very clear, as described above. I would give an overall score of marginally above the acceptance threshold based on its theoretical nature and serious study of the phenomenon.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0_Ri6Pcfjlu", "reviewer": "Reviewer_pKnQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pro**\n\n\\+ The submission is well motivated and coherently structured. (Empirical observation -> Splitting the loss function in summands -> Examining the gradient flow from the predominant summand)\n\n\\+ The theory is able to explain, some aspects, of the empirical observations by Papyan, Han and Donoho (2020) . It separates itself from the prior work, as it contains closed form solutions of the class means (of renormalized features).\n\n\\+ The submission is relevant, as understanding why and how optimization of current neural networks works is crucial for making informed improvements in future algorithms.\n\n**Contra**\n\n\\- The main result (Section 3.3) is formalized in terms of the singular values of the SNR matrix. However, the interpretation is not obvious. I would advice to expand on the footnote 9 and move it to the main text as a brief discussion of the findings.\n\n\\- There is an additional assumption which is required for Theorem 3, which is not mentioned in the statement and the subsequent discussion, but somewhere later: The within class covariance $\\Sigma_W$ is full rank. While one expects the assumption to be fulfilled, this assumption should still be stated more prominently.\n\n\\- There is a correctness issue, due to which I currently cannot give a higher score. The conclusions might not necessarily be wrong, but I am a bit wary. I hope, this can be resolved with the author response. \n\n1. It is claimed that Corollary 1 and Eq. 11 imply neural collapse. So in particular, they imply (NC1): $\\Sigma_W \\to 0$. However, these corollaries are concluded from the dynamics specified in Equation 5, where the representations are always such that $\\Sigma = I$ is constant.\n\n2. The first limit in Corollary 2, i.e. the statement that the singular vectors  of SNR stay constant with respect to $t$, is reasoned for by the fact that $L(w(t))$ only depends on the singular values (and not the singular vectors) and so its gradient does not depend on the singular vectors. However, the dynamics in Eq 5 do not only depend on the gradient, but also on an projection operator, which might change the singular vectors.  \nIntuitively, the dynamics in Eq 5 correspond to gradient updates followed by a renormalization step, i.e. matrix multiplication such that $X$ has identity covariance. This multiplication might possibly chance the singular vectors.\n\n\n*Questions and suggestions unrelated to score:*\n\nIt might be a good idea to reduce the notation in section 2, by considering only the case of zero bias and move the general case to the appendix. In Section 3, the setting is reduced to the case of no bias term anyway.\n\nIs symmetry really required for equation 8 to hold, or does invertibility suffice?\n\nCould you explain the terminology central path (Equation 4)? I guess, central refers to the least-squares optimality, but why is it a path? It does not appear to be one dimensional.\n\nIn appendix D, there is a rather informal discussion on the feature space as a fiber bundle. Can the renormalized gradient flow be formalized as a connection on this bundle?\n\nThere is a typo in Corollary 1. The second limit should depend on $c_3$ instead of $c_4$.", "summary_of_the_paper": "Recently, Papyan, Han and Donoho (2020) have observed that training neural networks beyond zero training error leads to simplex arrangements of the features. The submission studies this phenomenon, called neural collapse, from a theoretical perspective, when training is done via minimizing the square loss.\n\nIn particular, it is shown that the square loss can be split, such that one summand corresponds to the quality of the features (quantified by the loss of the MSE classifier on them) and that the other summand corresponds to the quality of the classification layer (quantified by the its deviationfrom the least squares classifier). Furthermore, the first summand quantifies the closeness of the features to neural collapse.\nIn a second step, the authors investigate the gradient flow induced from the first summand and derive closed form solutions.", "summary_of_the_review": "I am quite positive on the submission, however, there are two issues related to the proofs of the theorems. These issues might only be due to a misunderstanding on my part, but I am not sure. I hope the authors can clarify on this. Until then, I rate the submission as \"5: marginally below the acceptance threshold\".\n\n**Post author response update**\n\nThe authors adressed the correctness issues.\n\nRegarding 1. The authors confirmed, that Corollary 1, does *not* imply that NC1 ($\\lim_{t\\to\\infty} \\Sigma_{W,t} = 0$) and presented the remedy of replacing NC1 by a scale sensitive version, corresponding to the ratio of inner class variations to the between-class variations.  While this is a weaker result, it is still a wortwhile contribution and strongly connected to the observations by Papyan, Han and Donoho (2020).  \nIn fact, the updated definition of NC1, is precisely the quantity measured in the experiments by Papyan, Han and Donoho.\n\nRegarding 2. The authors added the missing proof to the appendix, that the singular vectors of SNR indeed stay constant when subject to the dynamics from Eq. 5.\n\nWith these two major issues out of the way I will adapt my recommendation to **8, accept**.\n\nLast but not least, I want to thank the authors for their detailed and clear responses. I really appreciate, that you took the time and effort to already updating the manuscript with proofs for my initial and follow-up questions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends the Neural Collapse (NC) phenomenon discovered by Papyan, Han and Donoho (2020) on deep learning image classifications with Cross Entropy (CE) loss, to the scenario with Mean Squared Error (MSE), that achieves similar performance to CE and favors deeper analysis. In particular, the paper shows that the least square loss can be decomposed orthogonally into a 'central' path as the optimal least square loss, and its perpendicular loss. Moreover, the paper shows by experiments that after the zero training error (Terminal Phase of Training, or TPT) the perpendicular loss is typically much smaller than the optimal least square loss, and the optimal least square loss is further decomposed into the NC1 loss which is the dominance and NC2/3 loss (even smaller than the perpendicular loss). Such a discovery with loss decomposition is very thought provoking to understand the training dynamics of deep neural networks.

Reviewers unanimously accept the paper, so is the final recommendation.
Decision: Accept (Oral)