An Unconstrained Layer-Peeled Perspective on Neural Collapse
Authors: Wenlong Ji
Yiping Lu
Yiliang Zhang
Zhun Deng
Weijie J Su
Venue: ICLR 2022 Poster (2022)
Keywords: neural collapse
uncostrained model
implicit regularization

Abstract: Neural collapse is a highly symmetric geometry of neural networks that emerges during the terminal phase of training, with profound implications on the generalization performance and robustness of the trained networks. To understand how the last-layer features and classifiers exhibit this recently discovered implicit bias, in this paper, we introduce a surrogate model called the unconstrained layer-peeled model (ULPM). We prove that gradient flow on this model converges to critical points of a minimum-norm separation problem exhibiting neural collapse in its global minimizer. Moreover, we show that the ULPM with the cross-entropy loss has a benign global landscape for its loss function, which allows us to prove that all the critical points are strict saddle points except the global minimizers that exhibit the neural collapse phenomenon. Empirically, we show that our results also hold during the training of neural networks in real-world tasks when explicit regularization or weight decay is not used.

Published as a conference paper at ICLR 2022
AN UNCONSTRAINED LAYER-PEELED PERSPECTIVE ON NEURAL COLLAPSE
Wenlong Ji Peking University
Yiping Lu Stanford University
Yiliang Zhang University of Pennsylvania
Zhun Deng
Harvard University
Weijie J. Su University of Pennsylvania
ABSTRACT
Neural collapse is a highly symmetric geometry of neural networks that emerges during the terminal phase training, with profound implications on the general- ization performance and robustness of the trained networks. To understand how the last-layer features and classifiers exhibit this recently discovered implicit bias, in this paper, we introduce surrogate model called the unconstrained layer-peeled model (ULPM). We prove that gradient flow on this model converges to critical points of minimum-nore separation problem exhibiting neural collapse in its global minimizer. Moreover, we show that the ULPM with the cross-entropy loss has a benign global landscape for its loss function, which allows us to prove that all the critical points are strict saddle points except the global minimizers that exhibit the neural collapse phenomenon. Empirically, we show that our results also hold during the training of neural networks in real-world tasks when explicit regularization or weight decay is not used.
INTRODUCTION
Deep learning has achieved faae--of-the-at performance in various applications (LeCun et al., 2015), such as computer vision (Krizhevsky et al., 2012), natural language processing (Brown et al., 2020), and scientific discovery (Long et al., 2018; Zhang et al., 2018). Despite the empirical success of deep learning, how gradient descent or its variants lead deep neural networks to be biased towards solutions with good generalization performance on the test set is still a major open question. To develop a theoretical foundation for deep learning, many studies have investigated the implicit bias of gradient descent in different settings (Li et al., 2018; Amari al., 2020; Vaswani et al., 2020; Soudry et al., 2018; Lyu & Li, 2019; Arora et al., 2019).
It is well acknowledged that well-trained end-to-end deep architectures can effectively extract features relevant to a given label. Although theoretical analysis of deep learning has been successful in recent years (Arora et al.: Goldblum et al., 2019), most of the studies that aim to analyze the properties of the final output function fail to understand the features learned by neural networks. Recently, in Papyan et al. (2020), the authors observed that the features the same class will collapse to their mean and the mean will converge to an equiangular tight frame (ETF) during the terminal phase of training, that is, the stage after achieving zero training error. This phenomenon, namely, neural collapse (Papyan et al., 2020), provides clear view of how the last-layer features in the neural network evolve after interpolation and enables us to understand the benefit of training after achieving zero training error to achieve better performance in terms of generalization and robustness. To theoretically analyze the neural collapse phenomenon, Fang et al. (2021) proposed the layer-peeled model (LPM) as a simple surrogate for neural networks, where the last-layer features modeled as free optimization variables. In particular, in a balanced K K-class classification problem using a neural network with d neurons in the last hidden layer, the LPM takes the following form:
min W H n
Σ L LWW,, v)), s.t. I W /
1 |H S Cz, C.
(1.1)

---

Published as a conference paper at ICLR 2022
where C1, C2 positive constants. Here, W [W0, W2..... _uk] final linear classifier. H = [/1, h22... hn] E Rd is the feature the last layer and yi is the corresponding label. The intuition behind the LPM is that modern deep networks over-parameterized, with the capacity to learn any representations of the input data. It has been shown that equiangular tight frame (ETF), i.e., feature with neural collapse, is the only global optimum of the LPM objective (1.1) (Fang et al., 2021; Lu & Steinerberger, 2020; Wojtowytsch & E, 2020; Zhu al., 2021).
E R is the weight of the
often highly
However, feature constraints in LPMs are not equivalent to weight decay used in practice. In this study, we directly deal with the unconstrained model and show that gradient flow can find those neural collapse solutions without the help of explicit constraints and regularization. To do this, we build a connection between the neural collapse and recent theories on max-margin implicit regularization (Lyu & Li, 2019; Wei et al., 201 8), and use it to provide a convergence result to the first-order stationary point of a minimum- entropy loss enjoys benign global landscape where all the critical points are strict saddles in the tangent space, except for the only global minimizers that exhibit the neural collapse phenomenon. Finally, verify our insights via empirical experiments.
separation problem. Furthermore, we illustrate that the cross-
In contrast to previous theoretical works on neural collapse, our analysis does not incorporate any explicit regularization or constraint on features. A comparison with other results can be found in Table and we defer a detailed discussion to Section 5.2. The reasons we investigate the unregularized objective are summarized follows:
Feature regularization or constrain is still not equivalent to weight decay used in practice. However, previous studies have justified that neural networks continue to perform well without any regularization or constraint (Zhang et al., 2021). Moreover, is proved that SGD with exponential learning rate unconstrained objective is equivalent to SGD with weight decay. (Li & Arora, 2019).
2. As shown in this study, neural collapse exists even under unconstrained setting, which implies the emergence of neural collapse should be attributed to gradient descent and cross-entropy loss rather than explicit regularization.
3. Regularization or constraint feature constraint can be barriers for existing theories of neural networks (Jacot et al., 2018 8; Lyu & Li, 2019). By allowing features be totally free, we hope our results can inspire further analysis to plug in realistic neural network.
Feature Norm
Feature Norm Regularization
Loss Function
Reference
Contribution
Constraint
(Papyan et al., 2020)
Empirical Results
X
CE Loss
(Wojtowytsch & E, 2020)
(Lu & Steinerberger, 2020)
Global Optimum
X
CE Loss
(Fang et al., 2021)
(Mixon et al., 2020)
Modified Training Dynamics
(Poggio & Liao, 2020)
X
& Loss
(Han et al., 2021)
(Zhu et al., 2021)
Landscape Analysis
X
CE Loss
concurrent
Training Dynamics+
This paper
X
X
CE Loss
Landscape Analysis
Table 1: Comparison of recent analysis for neural collapse. We provide theoretical results with the minimum modification of the training objective function. Here we use CE loss refer the cross-entropy loss.
1.1 CONTRIBUTIONS
The contributions of the present study can be summarized as follows.
2

---

Published as a conference paper at ICLR 2022
We build relationship between the max-margin analysis (Soudry et al., 2018; Nacson et al., 2019b; Lyu & Li, 2019) and the neural collapse and provide the implicit bias analysis to the feature rather than the output function. Although both parameters and features diverge to infinity, we prove that the convergent direction is along the direction of the minimum-norm separation problem.
Previous works (Lyu & Li, 2019; Ji et al., 2020) only prove that gradient flow on homoge- neous neural networks will converge to the KKT point of the corresponding minimum- norm separation problem. However, the minimum-norm separation problem remains highly non-convex problem and local KKT point may not the neural collapse solution. In this study, we perform more detailed characterization of the convergence direction via landscape analysis.
Previous analysis about neural collapse relies on the explicit regularization or constraint. In this study, we show that the implicit regularization effect of gradient flow is sufficient to lead to neural collapse solution. The emergence of neural collapse should be attributed to gradient descent and loss function, rather than explicit regularization or constraint. We put detailed discussion in Section 5.2.
1.2
RELATED WORKS
Implicit Bias of Gradient Descent: To understand how gradient descent or its variants helps deep learning find solutions with good generalization performance the test set, recent line of research have studied the implicit bias of gradient descent different settings. For example, gradient descent is biased toward solutions with smaller weights under &2 loss (Li al., 2018; Amari et al., 2020; Vaswani et al., 2020) and will converge to large margin solution while using logistic loss (Soudry et al., 2018; Nacson al., 2019b; Lyu & Li, 2019; Chizat & Bach, 2020; Ji al., 2020). For linear networks, Arora et al. (2019); Razin & Cohen (2020); Gidel et al. (2019) have shown that gradient descent determines a low-rank approximation.
Loss Landscape Analysis: Although the practical optimization problems encountered in machine learning are often nonconvex, recent works have shown the landscape can enjoy benign properties which allow further analysis. In particular, these landscapes do not exhibit spurious local minimizers or flat saddles and can be easily optimized via gradient- based methods (Ge al., 2015). Examples include phase retrieval (Sun et al., 2018), low-rank matrix recovery (Ge et al., 2016; 2015), dictionary learning (Sun et al., 2016; Qu et al., 2019; Laurent & Brecht, 2018) and blind deconvolution (Lau et al., 2019).
2 PRELIMINARIES AND PROBLEM SETUP
In this paper, || || denotes the Frobenius norm, || ||2 denotes the matrix spectral norm, || - denotes the nuclear norm, || denotes the vector la norm and tr(-) is the trace of matrices. We use [K] = (1, , to denote the set of indices up to K.
2.1 PRELIMINARIES
We consider a balanced dataset with K classes U ) A standard fully connected neural network can represented as:
f (æ; W Fult) = by + Wro (b;-1 + WI.10 (...o (b + Wiz)...))
(2.1)
Here Walll (W1, W2.... WL) denote the weight matrices in each layer, (b1, by. bz) denote the bias terms, and o--) denotes the nonlinear activation function, for example, ReLU or sigmoid. Let h& (b WL-10 (..-o (b, + W) E Rd denote the last layer feature for data TK.i and h4 _ Σn E hu denotes the feature mean within the k-th class. To provide formal definition of neural collapse, we first introduce the concept of a simplex equiangular tight frame (ETF):
Definition 2.1 (Simplex ETF), A symmetric matrix M E KRKx K is said to be simplex equiangular tight frame (ETF) if
K Q(Ix 1x)
M V K
(2.2)
3

---

Published as a conference paper at ICLR 2022
Where Q R K is matrix with orthogonal columns.
Let W E R Rxxd W,. = [wi, W22 ... wx] be the weight of the final layer classifier, the four criteria of neural collapse can be formulated precisely as:
(NC1) Variability collapse: As training progresses, the within-class variation of the activa- tion becomes negligible as these activation collapse to their class mean h2 = - h
hggi
hall = 0, 1 S < K.
The vectors of the class-meam (after centering by their global-meat converge to having equal length, forming equal-size angles between any given pair, and being the maximally pairwise-distance configuration constrained to the previous two properties.
(NC2) Convergence to Simplex ETF:
cos (e, h1)
hhell = ||rl|, 1 #
(NC3) Convergence to self-duality: The linear classifiers and class-means means will converge align with each other, up to appropriate rescaling, that is, there exist a universal constant
C >> 0 such that
Wy =Ch,, le E [K].
(NC4) Simplification to Nearest Class-Center For given deepnet activation:
h = a (bn-1 + WL-11a (...o) (b, + W( )) E R",
the network classifier converges to choose whichever class has the nearest train class-mean arg min (wx, h)) - arg
arg min || h&l1
k
In this paper, we say that a point W e Kxd E gdxak satisfies the neural collapse conditions o is neural collapse solution if the above four criteria are all satisfied for (W, H)).
2.2 PROBLEM SETUP
We mainly focus on the neural collapse phenomenon, which only related to the classifiers and features the last layer. Since the general analysis of the highly non-smooth and non-convex neural network is difficult, we peel down the last layer of the neural network and propose the following unconstrained layer-peeled model (ULPM) as simplification to capture the main characters related to neural collapse during training dynamics. A similar simplification is commonly used in previous theoretical works (Lu & Steinerberger, 2020; Fang et al., 2021; Wojtowytsch E, 2020; Zhu et al., 2021), 1), but ours does not have any constraint or regularization on features. It should be mentioned that although Mixon al. (2020); Han et al. (2021) also studied the unconstrained model, their analysis adopted approximation to real dynamics and was highly dependent the lg loss function, which is rarely used classification tasks. Compared with their works, we directly deal with the real training dynamics and cover the most popular cross-entropy loss in classification tasks.
Let W [wi W2. ... wx] E K and
[h1,1, ... hi, h2.1. .g,al R Kn be the matrices of classifiers and features in the last layer, where K is the number of classes and is the number data points in each class. The ULPM is defined as follows:
K
exp(w hr.)) ΣØ exp(wx hx.))
L(W H W.
min
min W.H
log
(2.3)
k=
Here, we do not have any constraint or regularization on features, which corresponds the absence of weight decay in deep learning training. The objective function (2.3) is generally non-convex on (W, H) and we aim to study the landscape of the objective function (2.3). Furthermore, we consider the gradient flow of the objective function:
(W())
OC(W()) (t), H()) dH OL(W()) H())
dt
aw
dt
0H
4

---

Published as a conference paper at ICLR 2022
3 MAIN RESULTS
In this section, we present our main results regarding the training dynamics and landseape analysis of (2.3). This section is organized as follows: First, in Section 3.1, we show the relationship between the margin and neural collapse in our surrogate model. Inspired by this relationship, we propose a minimum-norm separation problem (3.1) and show the connection between the convergence direction of the gradient flow and the KKT point of (3.1). In addition, we explicitly solve the global optimum of (3.1) and show that must satisfy the neural collapse conditions. However, owing to the non- convexity, we find an Example 3.1 in Section 3.2, which shows that there exist some bad KKT points such that simple gradient flow will get stuck in them and does not converge to the neural collapse solution, which is proved to be optimal in Theorem 3.3. Then, present our second-order analysis result in Theorem 3,4 to show that those bad points will exhibit decreasing directions in the tangent space thus gradient descent and its variants can avoid those bad directions and only converge the neural collapse solutions (Lee et al., 2016; Ge et al., 2015).
3.1 CONVERGENCE TO THE FIRST-ORDER STATIONARY POINT
Heuristically speaking, the simplex ETF (Definition 2.1) gives a set of vectors with the maximum average angle between them. As a result, neural collapse implies that the neural networks tend to maximize the angles between each class and the corresponding classifiers. At high level, such behavior is quite similar to margin maximization which is known to be implicit regularization effect of gradient descent and has been extensively studied in Soudry et al. (2018); Nacson et al. (2019b); Lyu & Li (2019); Ji et al. (2020). First, we illustrate the connection between the margin and neural collapse. Recall that the margin of a single data point z and the associated feature h is Qe.5 W; H) == w hxri - max ek w hx Then, the margin of the entire dataset can defined as
H) Qmin W. WHH)= :=
min IN.i WW H).
ke(K],ie[m]
The following theorem demonstrates that neural collapse yields the maximum margin solution of our ULPM model:
Theorem 3.1 (Neural collapse as max-margin solution). For the ULPM model (2.3), the margin of the entire dataset always satisfies
W 1 H
Qmin (W, H)
2(K 1)yn
and the equality holds if and only if (W, .)) satisfies the neural collapse conditions with [[[]] |||
Based on this finding, we present an analysis of the convergence of the gradient flow on the ULPM (2.3). Following Lyu & Li (2019), we link the gradient flow on cross-entropy loss to a minimum-norm separation problem.
Theorem 3.2. For problem (2.3), let (W (t), Ht)) be the path of gradient flow at time t, if there exist a time to such that C(W((t)) (to)) < log then any limit point of
H(t)
W(t) VII + (HOP
(1)()) W()) = / + HHIP
is along the direction (i.c., a constant multiple) of a Karush-Kuhn-Tune Tuccer (KKT) point of the following minimum-norm separation problem:
1 min W.H
H
2
2
(3.1)
s.f. s.f.un h un h 1 k kjje (K
Remark 3.1. Here we write (3.1) as a constraint problem. but the constraint is introduced by the implicit regularization effect of gradient flow our ULPM objective (2.3). Since the training dynamics will diverge to infinity, we hope to justify that the diverge direction is highly related to neural collapse and appropriate normalization is needed, which is why it appears to be
5

---

Published as a conference paper at ICLR 2022
constraint optimization form. Our goal is justify that the neural collapse phenomenon is caused by the properties of the loss function and training dynamics rather than an explicit regularization or constraint, which seems to be necessary for previous studies (Fang et al., 2021; Lu & Steinerberger, 2020; Wojtowytsch & E, 2020).
Remark 3.2. In Theorem 3.2, we assume that there exists a time to such that ((W(,,), H < log 2. Note that the loss function can be rewritten as:
K
L(W H =
log(1
+ Σ exp(ws,hs wggh,,))
(3.2)
JzL
the requirement C(W, .H) < log implies that wghg, w wy 2 0 for any k.j E [K] e [n]. which is equivalent to Gmin (,, H) > O; that is, every feature is separated perfectly by the classifier. This assumption is common in the study of implicit bias in the nonlinear setting (Nacson et al., 201 9a; Lyu & Li, 2019; Ji al., 2020) and its validity can be justified by the fact that neural collapse is found only in the terminal phase of training in the deep neural network, where the training accuracy has achieved 100%. It is also interesting direction to remove this assumption and study the early-stage dynamics of training, which is beyond the scope of this study and we leave to future exploration.
Theorem 3.2 indicates that the convergent direction of the gradient flow is restricted to the max-margin directions, which usually have good robustness and generalization performance. In general, the KKT conditions are not sufficient to obtain global optimality because the minimum- norm separation problem (3.1) is non-convex. On the other hand, we can precisely characterize its global optimum in the ULPM case based on Theorem 3.1:
Corollary 3.1. Every global optimum of the minimum norm separation problem (3.1) is also KKT point that satisfies the neural collapse conditions.
With Theorem 3.2 bridging dynamics with KKT points of (3.1) and Corollary 3.1 associating global optimum of (3.1) with neural collapse, the remaining work is to close the gap between the KKT point and the global optimum.
3.2
LANDSCAPE ANALYSIS
In the convex optimization problem, the KKT conditions are usually equivalent to global optimality, Unfortunately, owing to the non-convex nature of the objective (2.3), the KKT points can also be saddle points local optimum other than the global optimum. In this section, we aim show that this non-convex optimization problem is actually not scary via landscape analysis. To be more specific, we prove that except for the global optimum given by neural collapse, all the other KKT points are actually saddle points that can be avoided by gradient flow.
In contrast to previous landscape analysis of non-convex problems, where people aim to show that the objective has a negative directional curvature around any stationary point (Sun et al., 2015; Zhang al., 2020), our analysis slightly different. Note that since the model is unconstrained, once features can be perfectly separated, the ULPM objective (2.3) will always decrease along the direction of the current point and the optimum is attained only infinity. Although growing along all of those perfectly separating directions can let the loss function decrease to 0, the speed of decrease is quite different and there exists an optimal direction with the fastest decreasing speed, As shown in Section 3.1, first-order analysis of training dynamics fails to distinguish such an optimal direction from KKT points, and we need second-ordes order analysis to help us fully characterize the realistic training dynamics. First, we provide example to illustrate the motivation and necessity of second-order landscape analysis.
Example 3.1 (A Motivating Example), Consider the case where K = 4, n = 1, let (W, H)) be the following point:
1
0
0
1
0
W = H
0
0
-1
-1
One can easily verify that (W, H) enables our model to classify all features perfectly. Furthermore, can show that it is along the direction of KKT point of the minimum-nome separation problem

---

Published as a conference paper at ICLR 2022
(3.1) by constructing the Lagrangian multiplier A (Az)) follows:
0
1
0
0
A
1
0
2
To see this, simply write down the corresponding Lagrangian (note that aim to justify (W, H)) is along the direction of a KKT point of (3.1), to make it a true KKT point, one needs to multiple 1/v2 on W, H):
c(W H,A) = 1
H
ΣΣ
Ans W - ww, -1). -
j
Simply take derivatives for W, HH and A we find that it satisfies the KKT conditions. However, the gradient of (W, H) is:
0
2 2e
-1
1 0 0
= H) = 2 2c²
-1
We can that the directions of the gradient and the parameter align with each other (i.e., W is parallel to WWW,W, H), ; and H is parallel to VnL(W, H)), which implies that simple gradient descent may get stuck this direction and only grow the parameter norm. However, if construct:
1+0 a
-1+a
a
a
- -1+a
1+a a
W = H
1+ 2a
a
-a
1-a a -1 -1- -1a a 1 1
,
-1-a
By simple calculation, we find that f(aa) == L(W', H') satisfies ('(a) = 0, f'((())
0 (a) < 0. Since
|F = WWIF. H| || | and I| W ||' H
0, this result
implies that for any e > O, we can choose appropriate a such that:
W J II. || H 11:
|
W' W 11 + H H H <e, C(W", H') ( L(W,H)
In Example 3.1, it is shown that there are some KKT points of the minimum-norm separation problem (3.1) that are not globally optimal, but there exist some better points close it; thus, the gradient- based method can easily avoid them (see Lee et al. (2016): Ge et al. (2015) for detailed discussion). In the following theorem, we will show that the best directions are neural collapse solutions in the sense that the loss function is the lowest among all the growing directions.
Theorem 3.3. The optimal value of the loss function (2.3) on a sphere is obtained (i.c., L(W, H) S L(W', H') for any || I| + 2 collapse conditions and |[ = ||=|
|| + || (1: (+)) if only if (W, H)) satisfies neural
Remark 3.3. Note that the second condition is necessary because neural collapse conditions do specify the norm ratio of W and H. That is, if (W, satisfies the neural collapse conditions, then for any 8 8 E R, (aW, 3H) will also satisfy them, but only some certain o, B are optimal.
Now, we turn those points that are not globally optimal. To formalize our discussion in the motivating example 3.1, we first introduce the tangent space:
Definition 3.1 (tangent space). The tangent space of (W, H)) is defined as a set directions that orthogonal to (W. .H):
T(W, H) = {AW IR Kxd AH E Rdxnk) (r(W AW) + tr((H AH) 0}
Our next result justifies observation in Example 3.1 that for those non-optimal points, there exists a direction in the tangent space such that moving along this direction will lead to a lower objective value.
7

---

Published as a conference paper at ICLR 2022
Theorem 3.4. If (W, H) is not the optimal solution in Theorem 3.3 (i.e., (W, H) is not neural collapse solution or it is a neural collapse solution but [W]le * |H[)), then there exists a direction (AW, AH) T(W, H)) and constant M > 0 such that for any 0 << ò < M,
L(W + 6AW, H + 8AH) < C(W, .H). Further more, it implies that for any > 0 there exists point (W', [H') such that: H
(3.3)
W P +
H 1p |
H
(3.4)
W W
| - H HIP < C(W", H L(W,H)
Remark 3.4. The result in (3.3) gives us a decreasing direction orthogonal to the direction of (W.H). As shown in Example 1, the gradient on these non-optimal points might be parallel to (W, H); thus, the first-order analysis fails to explain the prevalence of neural collapse. Here the decreasing direction is obtained by analyzing the Riemannian Hessian matrix and finding its eigenvector corresponding to a negative eigenvalue, which further indicates that these points are first-order saddle points in the tangent space. That why we name it second-order landscape analysis. A formal statement and proof are presented in the appendix C. Previous works have shown that for a large family of gradient- based methods, they can avoid saddle points and only converge to minimizers (Lee et al., 2016; Ge et al., 2015; Panageas et al., 2019), thus our landscape analysis indicates that the gradient flow dynamics only find neural collapse directions.
EMPIRICAL RESULTS
Caaatirr ICFAR-S01
Latige feaure 10
Shanter 101A130
ICFAR-20)
05-
Elsodlor (MUN)))
Cheoderr (9666)
frotar (ron))
Lad bye featue (98657)
20-4
20-
Opech
Eooch
(a) Variation of the norms
(b) Within-class variation
(c) Cosines between pairs
(d) Self-duality
Figure 1: Experiments real datasets without weight decay. We trained ResNet18 both MNIST and CIFARIO datasets. The x a-axis the ligures are set to have log(log(!)) scales and y-axis in the figures are set to have log scales.
To evaluate our theory, we trained the ResNet18 (He al., 2016) on both MNIST (LeCun al., 1998) and CIFAR-10 (Krizhevsky et al., 2009) datasets without weight decay, and tracked how the last layer features and classifiers converge to neural collapse solutions. The results are plotted in Figure 1. Here we reported the following metrics to measure the level of neural collapse:
1. The variation of both feature norm (i.e., Std(I hl))/Ag(((hh: hll)) and classifier norm in the last layer (i.c.. (I (/)))//)////// Avg(( (x))))).
2. Within-class variation for last layer features (i.e., Avg( h& ll)/Av((( ( hll))
3. Average cosine between last layer features (i.e., Avg(1 cos (ug he)) + - and that of last layer classifiers (i.e., Avg( cos( (ūk (0x) + 1/(K 1)))).
4. Self-duality distance between features and classifiers corresponding to the same class in the last layer. (i.e., Avg( (ha h)/l has hll w*/l| [Tx Il))
Here smaller value of each metric indicates a closer state to neural collapse solution. Specifically, the within-class variation measures (NC1) ,the variation of norms measures and cosine between pairs measure (NC2), the self-duality measures (NC3), and (NC4) has been shown to be corollary of (NCI)-(NC3) (Papyan et al., 2020). More experiment results under various settings and training details can be found in Appendix D. As shown in the Figure 1, all of these metric decreases along the training epochs. These results reveal strong evidence for the emergence of neural collapse in the unconstrained setting and provide sound support for theory.
8

---

Published as a conference paper at ICLR 2022
5
CONCLUSION AND DISCUSSION
5.1 CONCLUSION
To understand the implicit bias of neural features from gradient descent training, we built a connection between max-margin implicit bias and the neural collapse phenomenon and studied the ULPM in this study. We proved that the gradient flow of the ULPM converges to the KKT point of a minimum- nommm separation problem, where the global optimum satisfies the neural collapse conditions. Although the ULPM is non-convex, we show that ULPM has benign landscape where all the stationary points strict saddle points in the tangent space, except for the global neural collapse solution. Our study helps demystify the neural collapse phenomenon, which sheds light on the generalization and robustness during the terminal phase of training deep networks in classification problems.
5.2 RELATIONSHIP WITH OTHER RESULTS ON NEURAL COLLAPSE
Theoretical analysis of neural collapse was first provided by Lu & Steinerberger (2020); Wojtowytsch & E (2020); Fang al. (2021), who showed that the neural collapse solution is the only global minimum of the simplified non-convex objective function. In particular, Wojtowytsch & E (2020); Lu & Steinerberger (2020) studied continuous integral form of the loss function and showed that the features learned should have a uniform distribution on the sphere. A more realistic discrete setting was studied in Fang et al. (2021), where the constraint is on the entire feature matrix rather than individual features. Our result utilizes the implicit bias of the cross- cross-entropy loss function to remove the feature norm constraint, which not practical real applications.
Although the global optimum can be fully characterized by neural collapse conditions, the ULPM objective is still highly non-convex. Regarding optimization, Mixon et al. (2020); Poggio & Liao (2020); Han et al. (2021) analyzed the unconstrained feature model with E loss and established convergence results for collapsed features for gradient descent. However, they fail to generalize to the more practical cross-entropy loss functions used in classification tasks. The analysis relies highly on the f2 loss to obtain a closed-form gradient flow and still requires some additional approximation to guarantee global convergence.
The most relevant study is a concurrent work (Zhu et al., 2021), which provides a landscape analysis of the regularized unconstrained feature model. Zhu et al. (2021) turns the feature norm constraint in Fang et al. (2021) into feature norm regularization and still preserves the neural collapse global optimum. At the same time, shows that the modified regularized objective shares a benign landscape, where all the critical points are strict saddles except for the global one. Although our study and Zhu et al. (2021) discover similar landscape results, we believe our characterization remains closer to the real algorithms since we do not introduce any constraints or regularization on the feature norm following the conventional setting realist training. The regularization of features introduced in Zhu et al. (2021) is still different from weight decay regularization (Krogh & Hertz, 1992). However, weight decay on homogeneous neural networks equivalent to gradient descent with scaling step size an unregularized objective (Li & Arora, 2019; Zhang et al., 2018). Moreover, neural networks are found to perform well the absence of weight decay, which highlights the importance of implicit regularization (Zhang al., 2021). As result, instead of explicit feature norm in this study, we consider implicit regularization brought by the gradient flow on cross-entron -entroy.. We show that implicit regularization is sufficient lead the dynamics to converge to the neural collapse solution without the explicit regularization.
constrain/reqularizzn
ACKNOWLEDGMENTS
We are grateful to Qing Qu and X.Y. Han for helpful discussions and feedback on an early version of the manuscript. Wenlong Ji is partially supported by the the School of Mathematical Sciences at Peking University. Yiping Lu is supported by the Stanford Interdisciplinary Graduate Fellowship (SIGF).
elite undergraduate training program of
9

---

Published as a conference paper at ICLR 2022
REFERENCES
Shun-ichi Amari, Jimmy Ba, Roger Grosse, Xuechen Li, Atsushi Nitanda, Taiji Suzuki, Denny Wu, and Ji Xu. When does preconditioning help or hurt generalization? arXiv preprint arXiv:2006. 10732, 2020.
Raman Arora, Sanjeev Arora, Joan Bruna, Nadav Cohen, Rong Ge, Suriya Gunasekar, Chi Jin, Jason Lee, Tengyu Ma, Behnam Neyshabua, and Zhao Song. Theory of deep learn- ing. https: /www. cs Lecnotes/bookdinn http:////www....rinntteteoeee//e
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. arXiv preprint arXiv: 2019.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint 1cXiv:2005.14166 2020.
Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference Learning Theory, pp. 1338. PMLR, 2020.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01771 2018
Dutta, K. Deb, Rupesh Tulshyan, and Ramnik Arora. Approximate kkt points and a proximity measure for termination. Journal of Global Optimization, 16:1463-1499 2013.
C. Fang, H. He, Q. Long, and W. Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences (in press), 2021.
Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle voints-online stochastic gradient for tensor decomposition. Conference on learning theory, pp. 797-842. PMLR, 2015.
Rong Ge, Jason D Lee, and Tengyu Ma. Matrix completion has no spurious local minimum. arXiv preprint 1xXiv:1605.07220 2016.
Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradient dynamics in linear neural networks. arXiv preprint 19t.19904.13262. 2019.
Micah Goldblum, Jonas Geiping, Avi Schwarzschild, Michael Moeller, and Tom Goldstein. Truth or backpropaganda? an empirical investigation of deep learning theory. arXiv preprint arXiv:1910.00359, 2019.
Benjamin D Haeffele and René Vidal. Global optimality in tensor factorization, deep learning, and beyond. arXiv preprint arXiv:1506.075: 2015.
X. Y. Han, Vardan Papyan, and David I.. Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path, 2021.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: convergence and generalization in neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 8580-8589, 2018.
Ziwei Ji, Miroslav Dudík, Robert E Schapire, and Matus Telgarsky. Gradient descent follows the regularization path for general losses. In Conference on Learning Theory, pp. 2109-2136. PMLR, 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
10

---

Published as a conference paper at ICLR 2022
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances neural information processing systems, 25:1097-1105, 2012.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. In Advances in neural information processing systems, pp. 950-957, 1992.
Yenson Lau, Qing Qu, Han-Wen Kuo, Pengcheng Zhou, Yuqian Zhang, and John Wright. Short-and- geomerrri approach. arXiv preprint 1Xiiv:108.1000 2019.
sparse deconvolution-a geometric
Thomas Laurent and James Brecht. Deep linear networks with arbitrary loss: All local minima are global. In International conference on machine learning, pp. 2902-2907. PMLR, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE,
1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 2015.
Jason D Lee, Max Simchowitz, Michael Jordan, and Benjamin Recht. Gradient descent only
converges to minimizers. Conference on learning theory, pp. 1246-1257, PMLR, 2016.
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations. In Conference On Learning Theory, pp. 2-47. PMLR, 2018.
Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. arXiv preprint 1rXiv:1910.07454, 2019.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. In International Conference on Machine Learning, pp. 3208-3216. PMLR, 2018.
Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss. arXiv preprint arXiv:2012.08465 2020.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint Xrv: 90690509,, 2019.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:200 2. 2020.
Mor Shpigel Nacson, Suriya Gunasekar, Jason Lee, Nathan Srebro, and Daniel Soudry. Lexicographic and depth-sensitive margins in homogeneous and non-homogeneous deep models. In International Conference on Machine Learning, pp. 4683-4692. PMLR, 2019a.
Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. The 22nd International Conference on Artificial Intelligence and Statistics, pp. 3420-3428. PMLR, 2019b.
Ioannis Panageas, Georgios Piliouras, and Xiao Wang. First-order methods almost always avoid saddle points: The case of vanishing step-sizes. arXiv preprint 1rX.v:190 2019.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint arXiv:2101.000 000022, 2020.
Qing Qu, Yuexiang Zhai, Xiao Li, Yuqian Zhang, and Zhihui Zhu. Analysis of the optimization landscapes for overcomplete representation learning. arXiv preprint arXiv:1912.02427, 2019.
Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by
norms, arXiv preprint arXiv:2005.06399 2020.
11

---

Published as a conference paper at ICLR 2022
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for scage-scale image recognition. arXiv preprint arXiv:1409.1556. 2014.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1): 2822-2878, 2018.
Ju Sun, Qing Qu, and John Wright. When are nonconvex problems not scary? arXiv preprint arXiv:1510.06096, 2015.
Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere i: Overview and the geometric picture. IEEE Transactions Information Theory, (3(2):853-888 2016.
Ju Sun, Qing Qu, and John Wright. A geometric analysis of phase retrieval. Foundations of Computational Mathematics, 18(5):113--1188, 2018.
Sharan Vaswani, Reza Babanezhad, Jose Gallego, Aaron Mishkin, Simon Lacoste-Julien, and Nicolas Roux. To each optimizer a norm, to each norm its generalization. arXiv preprint
arXiv:2006.06821, 2020.
G Alistair Watson. Characterization of the subdifferential of some matrix norms. Linear algebra and its applications, 170:33-45, 45, 1992.
Colin Wei, Jason Lee, Qiang Liu, and Tengyu Ma. On the margin theory of feedforward neural networks. 2018.
Stephan Wojtowytsch and Weinan E. On the emergence of tetrahedral symmetry in the final and penultimate layers of neural network classifiers. arXiv preprint arXiv:2012.05420, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnis. novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1tXiv:1708.07747 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Linfeng Zhang, Jiequn Han, Han Wang, Roberto Car, and Weinan E. Deep potential molecular dynamics: a scalable model with the accuracy quantum mechanics. Physical Review Letters, 120(14): 2018.
Yuqian Zhang, Qing Qu, and John Wright. From symmetry to geometry: Tractable nonconvex problems. arXiv preprint arXiv:2007.0677 2020.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geo- metric analysis of neural collapse with unconstrained features. arXiv preprint arXiv:2105.02375. 2021.
12

---

Published as a conference paper at ICLR 2022
A ELEMENTS OF OPTIMIZATION
In this section, we introduce some basic definitions and theory about optimization. In the following discussion, consider standard form inequality constrained optimization problem:
min (xx =
(A.1)
s.t.
[n].
In addition, we assume all of those functions f and gi are twice differentiable. A point R is said to be feasible if and only if it satisfies all of the constraints in (A.1), i.e., gi(x) S 0, i E [n]. And the Lagrangian of problem (A.1) defined as following:
A) f(x) +
A)
(x).
2=1
A.1 KARUSH KUHN TUCKER CONDITIONS
Now let's first introduce the definition of Karush Kuhn Tucker (KKT) point and approximate KKT point. Here we follows the definition of (c, 6.. 8KT point in Lyu & Li (2019).
Definition A.1 (Definition of KKT point). A feasible point is said to be KKT point of problem (A.1)
there exist X1, '2, An 2 0 such that the following Karush Kuhn Tucker (KKT) conditions hold:
(z) + E A (z)=
2. (z)(z) 0, i E [n]
Definition A.2 (Definition (c, 8)K point). For any 6, O, feasible point (A.1) is said to be (KKT)-KKT point of problem (A.1) if there exist Å1, A2, .. An 2 0 such that:
1.
(=)ll se
2. (x) 2 5, [n]
Generally speaking, KKT conditions might not necessary for global optimality. We need some additional regular conditions to make it necessary. For example, as shown in Dutta et al. (2013) we can require the problem to satisfy the following Mangasarian-Fromon Fromovivzz constraint qualification (MFCQ):
Definition A.3 (Mangasarian-Fromonn
constraint qualification (MFCQ) ). For feasible point æ of (A.1), problem (A.1) is said to satisfy (MFCQ) if there exist vector v E Rd such that:
(V (x), u) > O, i [n].
(A.2)
Moreover, when MFCQ holds can build connection between approximate KKT point and KKT point, see detailed proof Dutta et al. (2013);
Theorem A.1 (Relationship between Approximate KKT point and KKT point). Let {Ix E Rd k E N be sequence of feasible points of (A.1) fo > 0: E N) and (6x > 0: E N) be two sequences of real numbers such that &e is an (ex, Kk) point for every k; and eg 6g + If sy x as k +oo. If MFCQ (A.3) holds at w, then KKT point of (P).
B OMITTED PROOFS FROM SECTION 3.1
Recall our ULPMM problem:
K
n
exp(wr] her)) ΣK exp(w) hx.))
min W
L(W,
H)= = Σ log
k=
13

---

Published as a conference paper at ICLR 2022
Let's review some basic notations defined in the main body, Let Sk,i, W hkk w heg Ak [K]. E [m]]_] [K], the margin of single feature hg is defined to be g H))
minj Skii. wy heg maxjąk w hg. We define the margin of entire dataset as gmin qmin (W H) mis gz. W. H.). We first prove Theorem 3.1 in the mainbody.
Proof of Theorem 3.1. First we can find that the margin will not change if we minus a vector a for all wj, so if we denote the mean of classifier w. = w; - 1 Σi= w, and then we have K
w hR, maxj4t w hu,; u. hE, maxj W hu, 2 qmin ( H),, that g h4 - : hgg 2 qmnn (W, H), Vj + E [K] i [n]]
(B.1)
Note that Σj=1 2
h = then sum this inequality over j we have:
(K - 1) he
Σ w. Fur = K2 h4 2 (K 1mqmin (W, H), Ak E [K], i e [x]]
37t
By Cauchy inequality, we have:
K - 1 + vn 12,1111)) 20 husi 2
Qmin (W, H).
(B.2)
K
Sum (B.2) over k and we have:
V + H 2 n(K- - 1) ( (W,H. H).
(B.3)
On the other hand, we know that:
K
K
K
|w:
wi
lw 1w = W/E.
K
Then we can conclude that:
W 1
H P..
Qmin (W, H)
2(K
1)yn
as desired. When the equality holds, first we have 1| l|
|| which equivalent to 1 < Wi = 0, Wi = wy. Take it back into (B.3), then we must have all of the equality holds in K
(B.2) and 1), which give
W
+ |
wk Vnhaa. bo 11 n n2,12
2K
Take this into (B.1) we have:
I| 11 + || 12
h hag Mig wT w
2K(K - 1)
which implies neural collapse conditions.
Now let's turn to the training dynamics and prove Theorem 3.2. Before starting the proof, we need to introduce some additional notations. Since the W and H are optimization variables here, we denote 0 = vec(W W, .H) as the whole parameter for simplicity and all of previous function can be defined on 0 by matching the corresponding parameters. Denote p = | as the norm of 0 and 3 l
Now we can state our first lemma to show how training dynamics of gradient flow on ULPM objective (B) related to a KKT point of (3.1).
Lemma B.I. If there exist time to such that L(O(lo)) log then for any 1 4o Ø 0/ /unin 10) is an (e, approximate KKT point of the following minimum- separation problem. More precisely, we have
2(1 3(1)
K2(K - 1)m 20 (t) (t)
5
7t)
where:
d0
d0
8
l12)
1 di
di
is the angle between 0 and its corresponding gradient.
14

---

Published as a conference paper at ICLR 2022
Proof. The training dynamics is given by gradient flow:
d#
0C(0)
di
00
Then by the chain rule we have:
dC dt
0C d0 00 dt
d0
(B.4)
di
It indicates that the loss function L is monotonically decreasing. If L(O(to)) log 2, we have
C(((t)) log Vt to- On the other hand, note that
K ΣΣ
((O(L))
log(1 + -e-ohssl1)) log(1 exp( qmin (t)));
(B.5)
k=
which gives Qmin (t) Vt to-
Let db de note that rewrite the ULPM objective function (B) L(0) Σ k= E log(1 Σjtx e e By the chain rule and the gradient flow equation have
OL K 00
Eier e Se,e0 Ih...j
where Ik.s. is the gradient of sk.iff ie. Ikii (0)....( Now let ge.s. gk.s.j/ Amin (os and construct Ae.i.
1/2
Tola 1+ E1En "KisTt we only need to show:
1 1- B
110
(B.6)
7
jzt
a
( -1) K2(K 1)m 2eqmin
(B.7)
jæk
To prove (B.6), we only need to compute (Recall that Ô 0/ gmin (aniin())23) (0)2/2)
K
p
p² Qmin
||0
11
(2 - 2.3).
Jman
|181
l1
A=
1=1
Note that:
log(e (())
N
7
((0) Σ
log(1 + Σe Ans) > log(1 + exp(pp -min)))
(B.8)
02
$8
Then have the following inequality:
Jmin
3 <
(B.9)
p2
Take this back into (B.8) we have (B.6) as desired.
To prove (B.7), first by our construction:
K
K
( 1) Gmiillglle km (=1
Eler -n (sk
9msnn)
Aml i
(B.10)
15

---

Published as a conference paper at ICLR 2022
Note that ||9112 2 (9. |
e
) =
(9. 0) and (gh.s. 0) = 23 since sk.i.s = w. hen w ha
have:
||91|2
(g,
Eiye C (gk.i.j)))
p
P
17h
2
Skiid
ELpe
k=
K
2
Gminn
Skas
(since SR.iJ 2 Qmin > 0 and e ekaid S 1)
K
1= 1=
2
Gmnnn 9min
p K
(B.11)
Take this inequality back into the (B.10) we have:
ΣΣΣ (B,s(SH,
K
-1)
Kp²
n ΣΣΣ
K
@lmin Ski0
Eiehh e (sk.i. - 4min
29 ki im
jzc
jit
s
Kº²
ΣΣΣ
K n
Eaman - S0, 88, (8K,5d 9min)
2g
k=1
jæ
K
5hi (Sk Qmin.)
20mmin
K2(K 1)m
2
Where the last inequality is obtained from the fact De elementary calculus.
1 Vr > 0, which can be proved by some
Based on Lemma B.I, we have shown that the (W, HH) will be a (e, 8KTT point, if we can show (c, 6) converge to zero, then by Theorem A.1 we know the limit point will be along the direction of KKT point. Ignoring the constant term, we only have to show how (t). 3(t) and Jmin (t) evolve along time. Now we provide the following lemmas to illustrate their dynamics. The first lemma aims at proving that the norms of parameter p(t) and (4) are monotonically increasing.
Lemma B.2. If there exist to such that L(((to)) < llog 2, then for any t to have:
dp² di
0.
2 0.
(B.
dt
dt
Proof. We can disentangle the whole training dynamics into the following two parts:
the radial part: : ²
de
the tangent part: u (I - T) de
de
First analyze the radial part, by the chain rule: ||0||2 = T =1 (0, de we have the following equation:
111
do?
For
do dt
22 dt
dp²
d0
0,
(B.13)
dt
dt
Eier
jzk
16

---

Published as a conference paper at ICLR 2022
where the last equality holds by equation (B.11).Then when t > tu. have shown that gmin (t) 2 combine this with the fact Sk,i.j 2 qmin we obtain the first inequality in (B.12)
dp²
K
Sk.i.j
2 dt
1+
Eiyx e
k=
ifk
(B.14)
K
2
E1px e Imin 2 O.
1+
jz
Next, we aim to prove the monotonicity of ft),, compute the derivative of ((t we have:
- log(e_(0) - -)
log 9 dt [log(-log(e - -1)) - 2 log p):
(log(- log(e
(B.15)
dt
ec(0) dC(() dc(()
e
log( - log(e c()) 1)) log (()) (( 1) cl dt
qmin e((o) (B.16)
dt
dt
Recall that we have:
dp²
K
52,10 1F
Σ 6 Su.5 8k,ij
dt
k=
1=1
K
Σ josk Skiis
Skild
2
jpkk okis Qmin
k=
1=1
Ejen e 8kis
log(1 Σ (8,11)
log(1 + Ejpp e sesss)) 1 Ejイt sssss Umin (B.17)
j
K
log(1 Sk.))) el(0) -1
2
L(0) Umin
k= im1
jzk
(()
=2 el( Qmin-
Then second last line is because the definition of the loss function C()) = Σx Σi= log(1 Ejpee e Bk.A)) 2 log(1 + Ejpe e= and the monotonicity of s²-1 (in fact, de ze
c-
"(z-e++11 < 0, Vz 0).
As result, we notice that
1 log ū(t)
1 dp²
-1
dc
log
2 dt
2 dt
dt
dt
At the same time, we notice that ||u||2= chain rule:
1 dd²
2
dp
log P on the one hand, and by the
p²
2 dt
2 dt
de
- 1
d8
dp
1
d0
d8
u
0
0
di
7
dt
di
7
dt
dl
p
Combine this with the radial term:
dl
d0
1 dp²
d
d8
||a|
+
|/ul12
log
p
p²
dt
dt
2 dt
dt
dt
Dividing
de² de both sides, we have
dc
do²
-1
( d
dØ
dl
dl
= di
log
p
+
di
log
di
(B.18)
17

---

Published as a conference paper at ICLR 2022
dC(()
e
dt
log p +
dt
log
p
dt
-
dt
9min 2(e
- 1)
Now by equation (B.15) and (B.16) we obtain:
1 d
dC(0) 1 el(0)
d
d log
d0
log 7
log p
(B.19)
dl
dl
Qmin 2 (e((0)-1
di
dl
dl
By (B.14) know the p is monotonically increasing, have म log p and then get the second inequality in (B.12).
Lemma B.2 gives us the monotonicity of 5. note that since the loss function ((((t))) log we
have (() 2 (44) > O, then we can treat 7() in Lemma B. positive constant. The remaining
work is to show (1) grows to infinity and 8(1) + To show Qmin (1) + oc, it's equivalent to
show C()) + 0 and we have the following lemma:
Lemma B.3. If there exist to such that L(((t))) < log 2, then C(O(t)) - and gmnn (0(t)) + oo as 0o, moreover have the convergence rate L(((1)) O(1(l)
Proof. By (B.4) and (B.13), the evolution of loss function L(O) can be written as:
dC(0) d8 2
d8
K n
Sh.s.
(2
Σ jitk
ligh s Sk.s,)²
dt dt
dt
110122
1+
Σ
Amm
Combine it with (B.5),(B14) and (B.17) we have:
K
el( -1
-2 e((0) -1
1+ E SkAd 2 2 ef(0)
4man
el10)
log(e
- 1),
which indicates:
dL(((t))
< 5-4( (((())
((0(1 1))
(B.20)
cc((()))
Since 0 L(((t)) <(O(t))
log 2, note that:
d e
0.
lim
1,
dx
x
r-0
which implies:
1 cl(((1)) -1
(B.21)
L(((t))
log(2))
On the other hand, we can find that:
((0())((()) -1) 1) < ( 1) < 0, 1< ((00) <
(B.22)
Combine equation (B.21) and (B.22) together, one can conclude that there exist constant C such that for & > to:
dL(((()) -CC(((())))
di
It further implies that:
0 d a
dt c(((t)) )
then integral on both side we have:
C(t-to) < L(((t)) c(((to))'
and
((((1)) = O(1(t)-
Thus we must have C(O(t)) - O and combine this with Jmin 2 (l0((e((0()) 1) we know Umin (0(t)) + as desired.
18

---

Published as a conference paper at ICLR 2022
To bound ((),, first need useful lemma bound the changes of the direction of B.
Lemma B.4. If there exist to such that (()()))) < log 2, then for any t >> 60
dā
log p.
dt
(to) dt
Proof.
First we know that:
d0
d0
d0
[T 00
dt
dt
p
dt
d0 dt
K
K
1+ Σ1Ex toiee Ikin
Σdpr
jzk
k=
jæk
(B.23)
Recall our Ie.. other hand, Combine it with (B.13) and (B.9) we have:
Dskan 09 and 8k,6.j = wr hg.s m hx we can find that |1 ||
S 2p. On the
di
d0
dp2 p²
log p S log - 7(4)) dt log p
dt
dt
2mmin
dt
Imin dt
7 dt
as desired. Where the second inequality holds by multiple amin on the right hand of (B.23) and the
formula of do² in equation (B.13), and the last inequality holds since we have shown that 7()) is dr monotonically increasing in (B.2) and (44) > O since L(lo) < log 2.
Let's turn back to 3, though we can not show directly that it increase to one, we can find a sequence of time } for each limit point such that (tmm) + 1
Lemma B.5, If there exist to such that C(O(to)) < log 2, then for every limit point 0 of (θ(t)
t > 0}, there exists sequence of time { > 0 m N} such that tn oc, 0 (fm) 0, and
(tm)) +1
Proof. Recall that in equation (B.19) we have shown that:
d
-1
d0
log 2
log p)
(B.24)
dt
dt
dt
Since d. log
do
dp²
1
(0,
and:
2p
dt
pd
dt
d0
d
d0
T d0
d0
dt
dt
||0||
p (0
dt
p
00
dt
=
p
-(T
(T
00
dt
Plug them into (B.24) we have:
00
d9
2
d
log 2 2
dt
log p = 2(3-2-11 11)
d
log
dt
66.
d0
dt
dt
For any 12 > t > to. integrate both sides from time t2 to tı we have: log 5(t2) log (t:) >
2 2 (B(t) -2- 1 d log pat. By the continuity of we know there exist time t* such that:
/ (B(1)-2 -1) d log (1z) - log (tz)) 2 2 1,
dt log pdt
-2(((1))2-1) 1.a
nha d
(B.25)
log pdl
dt
= -((3(tt)-2 (1ogg p(t2) - log p(t:)).
19

---

Published as a conference paper at ICLR 2022
By (B.9) know that 3 I Smin and the right hand is bounded, and the & is monotonically increasing, then there exist Yoa such that (tt) 7so.
p²
Now we are ready to construct the sequence of im first take sequence of {cm 0, m E N) such
that Cm + 0. We construct fm by induction, suppose we have already find t1 < t2 < ... tm-1 satisfy our requirement, since 0 is limit point of {0(t) t>> 0}. then can find time 8m such that:
7000 cm
[10(sm) - S Ema log
7mm))
By the monotonicity and continuity of p we can find a time s'n such that log (() -lg p(sm) S cm. Take t2 s'm t 8m in (B.25), there exist time im such that:
log (t2) log (t1) log (tz2 log p(t))
(tm( - 1)
S c2
(B.26)
on the other hand, by Lemma B.4 have:
[l((m)) - 0|1 <110( [10)) -8l1 + []0(sm) 0(tm))l
1
(B.27)
Sem + 7(16)
(log p(lm) logp(sm) S
cm
7(t))
Note that (0, de) > O, then by definition we know B > 0. Combine equations (B.26) and (B.27) we ar have (tmm) + 1 and (tmm) + as desired.
Now are ready to prove Theorem 3.2:
Proof of Theorem 3.2. By Lemma B.I, we know that once t
to.
(t) (t))) qmin (t), (t)) is an (v 201 201-80 (K--1)a -approximate KKT point. We have shown that 7(t) > 7(to)
in Lemma B.2, Qwin + oo Lemma B.3 and from Lemma B.5 we know for any limit point (W.H) of (t), W(t) = W(e)
( }} there exists sequence of time {tm > 0: m N) (IIW()) B--1 such that tm + 0a, ((mm. 1 and (Hm,,, (t..)) + (W.H). Then (W, H)) is along the direction of a limit point of sequence of (e, 8) approxxxiete KKT point with e, - 0. On the other hand, we can verify that the problem (3.1) satisfies MCFQ (A.3) by simply setting u 0, then:
H(*)
(K 28k.i. 2 0.
Now by Theorem A.1 we know (W, H) along the direction of KKT point of problem (3.1) LI
Theorem 3.2 characterize the convergent behaviour of gradient flow, under separable conditions the limit point is along the direction of a KKT point of (3.1), next we show that the global minimum of (3.1) must satisfy neural collapse conditions by proving Corollary 3.1:
Proof of Corollary 3. Since we have shown that the problem (3.1) satisfy MCFQ, then the KKT conditions are necessary for global optimality, we only need show the global optimum satisfies neural collapse conditions. First the constraints in (3.1) 1) can be transformed to be single constraint by the definition of margin:
Vk #j E [K], E [n],
w. hE
w h4s 2 1. + Amin (W, W.P. H) 21
(B.28)
Note that the margin is homogeneous:
Quin (a (aW, W.HH) = a² qmin (W, H), Va
Then for any point (W,H) satisfies qmin (W, H) > 0. after an appropriate scaling a. (aW, HH), Va 2 1/gmin (W W. H) is feasible for (3.1). Take optimum among all scaling fac- tor a we know the minimum norm is attained if and only if a² = 1 /m/uu (W. H). And the optimum norm is:
aaa
+ 2
+ H (2))
2
2 (W, H
20

---

Published as a conference paper at ICLR 2022
Then by Theorem 3.1 we have:
W H 2 2(K - 1)vn.
2min W. H)
And the global optimum is attained only when (W, H) satisfies neural collapse conditions
C
OMITTED PROOFS FROM SECTION 3.2
To begin with let's finish the computation in the motivating example (Example 3.1)
Proof in Example 3.1. Consider the case where K
4,
1, let (W,H) be the following point:
1
0
0
1
W H
0
0
-1
0
-1
One easily verify that this (W, H) enables model classify all of the features perfectly. Further more, we can show it is along the direction of a KKT point of the minimum-nome separation problem (3.1) construct the Lagrangian multiplier A (Aiz) (At)) following:
0
2
0
A
1
1
11
0
2
0
2
To see this, just write down the corresponding Lagrangian (note that to make it to be a true KKT point of (3.1), one needs to multiple 1/v2 on W. H):
L(W, H, W
H
ΣΣ 1=
1 wwh, 1). Nesl w,h,
2
Simply take derivatives for W, H and A find it satisfies KKT conditions. On the other hand, the gradient of (W, H) is
1 0
VwC(W, H) = VHC(W, H) = 2 2e- + 2c
2 2
-1
1 0
0
-1
We find that the directions of gradient and the parameter align with each other (i.e., W is parallel to V V L(W, H), and H is parallel to CWC,, H)), which implies that simple gradient descent may gets stuck in this direction and only grows the parameters' norm. However, we construct:
1
+1+a
W
-1+a
I+a
a
1 2a
-Q
-0
1 - a
-1 -a
-a
-
-1 a
1-a a
1+a
-1+a
H'
-1
1+a
-Q
-Q
1 + 20
a
1-a a -1
-1 -1-a
(
--a a
1 --a
Note that | ||.
W |r, H'
= H |
and
W
W 2 +
H'
H
-
0 a
-
First we compute:
2+ 2+ 4a²
2
2
- 2
-4
1
4a 2
2 + 4a
4a²
-4
W'
1+ 2
- 4a²
-aa
2 4a
4a - 2
- 2
- 2a
4a² - 2
2 + 4a²
21

---

Published as a conference paper at ICLR 2022
and:
e²
L(W', ,H' log
e² + 1426
2 2n
+ 2e
2
21-202
Our aim is to show that for any 6 >> O, there exist a such that |a| < c and L(W", H') < L(W H).
22x2-1
-2 20²
By the formulation of L(W', H'), it's sufficient to show that f(a) e 1+2a2 Then take the derivative f()) we have:
+2e 1+2a
< f(0).
Sa
8a (2a - 1)
4a2
16a
8a
((( = e el+2a
+ 2c
142a
1+ 2a
(1 + 20
(1+ 202)²
1 2a²
2x?_2
80
8a (2n² -1)
2
4
16a
8a
2
(a) 1+2n
2c
1426
1 2a
(1 ++22222
(1 20²)
1 1 2a.
2
6
64 (2a² -
-1) -1)
8
8 82aa 1)
+ e 1422
(1 + 20
(1+ 20
1 2a
(1 + 202)2
2
80a
8
128a
+ 2e
2l
2a?)
+ 2x
(1 + 20
Now we find that f0( and f" 16( + 1) 0. Since the function f(a) is continuously
twice differentiable, we can conclude that for any > 0, we can choose appropriate a such that:
W W
H
1 H Irr.
W W 7 + H H <E < C(W' H < L(W, H).
Now prove Theorem 3.3 by some similar strategies used in proving Theorem 3.1.
Proof of Theorem 3.3. Again rewrite the ULPM objective by introducing 8k.i.j w. hgg
w hg
K n
L(W H
ΣΣ log(1 + Eexp(= sk,.,)))
ji
In addition, we can find that centralizing w; does not change the value of Sk,i.f Let w; w;
1 >E wk Vi [K], then 8k,6. w hu.i w. hu,s = w hu,s 20, hu and Σk w = 0.
K
First by the strict convexity of e and Jensen Inequality:
K
n
L(W, H
log(1 + (K - (1) cxpp
K - Σ Sk.25))
R=
jtk
(C.1)
K n
Kwn his
log(1 + (K 11 -xcpppp(
K-1
Where the last equality is obtained from:
Σ J
sk,ij Σ
w hu,i t h (K - 1)w0 h Σ in h
Kwo h
s#k
22

---

Published as a conference paper at ICLR 2022
Now again by the strict convexity of log(1 + (K - 1) 11exp(-zz) and Jensen inequality, have:
K K lok.)
C(W H
log(1 + (K 1) exp(
K
k=
i=1
K
n
nk log(1 + (K exp(
wk hg,s))
n(K -
k=1
(C.2)
K
1
2 nk log(1 + (K 1) exp(
|| |0x0 || + vnlhs
2n(K-1 -
z nk log(1 + (K exxe
+ |H
2 2/n(K-1) -
Where the last inequality holds since W Σ || 2 > |0 k > wxll =
Ek= | l|
When all of the above inequality reduce to equality, must have:
1. Ee ii=1 w. 0, w. ww (the last inequality in (C.2))
2. wy Vnhgaa E [rx] (the third inequality in (C.2))
3. ||wx|| = | ll. | = [ Vk, k E [K], E [n]] (the second inequality in (C.2))
4. $1,i. w. hege w h - " h Y [K]. [m] (the first inequality in (C.1))
These four conditions are exactly equivalent to neural collapse conditions and [W]| ||
||
The global optimality is not enough to illustrate how does gradient flow converges neural collapse since there may exist some had local minimum. We will provide the following second-order analysis eliminate spurious local minimum. First, define the cross-entropy loss matrix Z R K
K
L(Z)
log
k=l 21
where zk.i. denote the j-th row and [(A: 1)K + i1-th column elements of Z. Then we have L(W, H) = L(W H). Now compute the gradient of L(Z) to each element:
OL(
Zk.i,
-
0zk,i,k
ΣK gsk.i.i
OL(Z)
2k.5.l
Vk
+ j.
Ozp.i.i
Σ eat.l
E K satisfies T VL(Z)= = 0, denote up as the maximum element of u, then we have:
0 Up Dzp.i.
(L(
Σ
Uq @zpri.
(L(Z)
=up
Zp.i.p
Σ Uq
Zp.i.g
Σ @Fpiil
E
eFrii
(C.3)
Σ (up Ug)
Zp.i.g
S O.
ΣK
@Fr.i
Where the last inequality holds if and only if uy up 8g [K]. Which indicates that the rank of
VL(Z) is K and u VL(Z) = 0 == 1. Now are ready prove Theorem 3.4 in the main body.
Proof of Theorem 3.4. First compute the gradient of ULPM objective (B), by the chain rule have:
VwL(W, H) VI(HH) VnC(W, H) W (L(WH).
23

---

Published as a conference paper at ICLR 2022
If there exist a vector (AW, AH) E T(W.H) such that VVw C(W. H). AW)
[ H), A & O, moreover we can assume (V (V (WW(WW,),WW) (W) CV, H,, A < 0 since zero, then by Taylor expansion:
take the negative direction if the formula is greater than
L(W + SAW, H + 6AH) - C(W, +
+ CCW, H))
we know that (AW, AH) satisfies our requirement.
Now let's 's discuss the case when:
= O, VWW, CWCC(W H), AW) + (
AH) T(W, H))
by definition of T(W, H), contains all vectors that are orthogonal to (W, H), so WVw H), CW H)) is parallel to (W, HH), that is, there exist À such that:
VL(WH) AW, W WLWWH = AH.
(C.4)
If there does not exist (AW, AH) satisfying the requirement, we know that for any feasible curve
<(t) (W (t). H (t) with (0) = (W, H)) on the sphere S = [(W", H²): W +
H
P = 0 admits the local minimum of [()(t) and thus:
0 < di ((o())) = =
H)) + VL(W H0)"
(C.5)
On the other hand, since the curve lies on the sphere S, denote h(W, H) = || [/ + H| p. then h(o(t)) must stay constant, take twice derivative we have:
d2 0 de
o' H) VWWW WWWWH))"(00)
(C.6)
Then sum these two conditions together, we have:
d² 0 S dea (C()(t)) + A 2 =
| () H))( + V 2
(), H))
2 (C.7)
By equation (C.4) we know that V(C + (Al (W H) Note that d'(0) E T(W H) since the curve lies on S and for any (AW, AH) E T(W, H) we can construct curve @(t) such that &' s'(0) = (AW, AH). Then (C.7) indicates that ((A W, AH) E T(W, H) we have:
0 <AW AH) LWW (AW AH) + |
2 (AW, AH) Vhh(W, ()A,W, AH)
= =(WW, AH) (FC(WH AH) + IKIKAW 13 + |AH
(C.8)
When A O, by equation (C.3) we know that (VL(WH)) 12 0, which gives us |X|
<
H) When À + 0. combine the two equations in (C.4) we know:
WA W = W T VL(WH)H = AHH = w W
(C.9)
which further implies:
W||F = || || H 12-
Thus we also have (Note that when W = . = 0 we can take À to be zero):
VL(WH)H XW + W la ( H|
= | S (VL(WH))l
Now when |\] < (WVLWW)lll we can show that it will contradict with (C.8): We have shown that the rank of (Z. is K -1, so by (C.4) and (C.9) there exist vector such that Wa H a 0.
24

---

Published as a conference paper at ICLR 2022
let and are the left and right singular vectors corresponding the largest singular value of VL(WH) construct AW ua AH = -av", ,then (AW, AH) E T(W, H.WH. H) and:
(AW, АH) v'W(W, (AW,, AH) X(IIAW I + ||
-WAH AWH (W H) (WAH AWH) + (VL(W AWAH) + INIIAA |AH
<2||a|| (2 (IA) u (L(WH))) <
Then only remains to analyze the | = ( H)|| cases, construct another convex optimiza- tion problem:
min L(2) + | Z l.,
(C.10)
suppose Z has SVD Z = UEVT, as we know that the subgradient of Z can be written as (see Watson (1992) for proof):
O| Z = (UVT W.WERkKon
- 51}.
(C.11)
On the other hand, we know that:
H HH H H W WH VV
WW WW WHH W =UE
which indicates that H H VEVT and WW = TEUTUETTAOO -Combine them with (C.4) we have:
VL H XWH + VL(WHVVEV T
* VL(WH)V = XU
WW WWWWL(WH) = AWH + UEU VL H) = NUEV
= VLWH) AVT.
Note that |\| = WW 2, then by (C.11) know that WW H) E llll| W H||-. Then by the strict convexity of (C.10) we know W H is the global minimum of it. In addition, we have
||W 1| | 2 W || In addition, previous works (Haeffele & Vidal, 2015) have
shown that:
2 = min 1
(IW H
z WH
which is equivalent to:
WW 1. - 2
W H
Now for any (W', H'), we know that:
C(W, H) + 2 W + (1 = L(W + + L(W'H') + [X]]] H'Il
<(W' H)) + 1 H (.)
which indicates (W, H) must attain global minimum of the following optimization problem:
A (IW + (H))))
min L(W, H)
W.H
2
If (W, H) does not satisfy neural collapse conditions, by the optimality of neural collapse solution (Theorem 3.3) we know there exists another point (W', H') such that L(W", H') < L(W, ,, H) and
W 1 H 3 W
+ H' thus:
[I]
X (IW' H 12:) 2
C(W H) +
W 1 [HI
L(W', H')
2
which contradicts with the global optimality of (W, H), thus (W, H) must satisfy all of the neural collapse conditions and finish the proof.
25

---

Published as a conference paper at ICLR 2022
D ADDITIONAL EMPIRICAL RESULTS
Gradient Descent on the ULPM Objective. We conduct experiments on the ULPM objective (2.3) to support the results of convergence toward neural collapse in our theories. We set N = 10, K = and d = 20, and used gradient descent with a learning rate of 5 to run 10 epochs. We characterize the dynamics of the training procedure in Figure 2 based on four aspects: (1) Relative variation of the centered class mean feature norms (i.c., Std hag hll)/ Al)/Avv([lhs - h)l)) and the variation of the classifier's norms (i.e.. (Id (10 (vg/ [x (2) Within-class variation of the last layer features (i.e., Avg( h2s (I)/ Avg ( - D.))). (3) The cosines between pairs of last layer features (i.e., Avg(] cos((2: ) + 1 - L)() and that of the classifiers (i.e.. Avg(1 cos( wyy ) ) - 1)))) (4) The distance between the normalized centered classifier and the normalized last layer feature (i.e.. Arg( - / - hll /x. [l)). Empirically, we observe that all four quantities decrease
at approximately the rate 0(1 (1)))))()))
Dassitter feature
Clacitier
Last Feaure
aesovve detereeeeeeeeee
10-1
Arg ad diferenc
13-4
Aug coat
xx30
x x=0-=
30
3x10-
4x
10
10r
15:1010
36
10
16
10
307
30t
10
101000
Eacch
Ezorh
Eports
Escrh
(a) Variation of the norms
(b) Within-olass variation
(c) Cosines between pairs
(d) Self-duality
Figure 2: Training dynamics in ULPM. The x-axis in the figures is set to have log(log(f)) scales, and the y-axis the figures are set to have log scales. (a) The dynamics of the variation of the centered class-mean features' norms (shown in blue) and the variation of the classifier's norms (shown in red). We observe that the logarithm of both terms decreases at a rate Dynamics of within-class variation of the last layer features. The logarithm of the variation converges at approximately the rate O(1/ log(t))). (c) The dynamics of the cosines between pairs of last layer features (shown in blue) and those of the classifiers (shown in red). The logarithm of both terms converge approximately at rate O log(t))) (d) Dynamics of the distance between the normalized centered classifier and normalized last layer feature. The logarithm of the quantity converges at approximately the rate O(1/ log(!))) to the point of self-duality.
(b)
Details of Realistic Training. the real data experiments, we trained the VGG-13 (Simonyan & Zisserman, 2014) and ResNet18 (He et al., 2016) on MNIST (LeCun et al., 1998), KMNIST (Clanuwat et al., 2018), FashionMNIST (Xiao et al., 2017) and CIFAR-IO datasets (Krizhevsky et al., 2009) without weight decay, and with learning rate of 0.01, momentum of 0.3, and batch size of 128. The metrics are defined similarly to the ULPM case and the experiment results are reported in Figure 1, 3, 4, 5, 6, 7, and 8. All experiments were run in Python (version 3.6.9) Google Colab. It
32
L2-
Casater IOPAB-201
Last fnstrr
00-
8.6
0.82
Opech
Caech
Exech
Eooch
(a) Variation of the norms
(b) Within-class variation
(c) Cosines between pairs
(d) Self-duality
Figure 3: Experiments on real datascts without weight decay. We trained VGG13 on CIFARIO datasct. The a-axis the figures are set to have log(log(t) scales and the y y-axis the figures are set to have log scales.
26

---

Published as a conference paper at ICLR 2022
liver Paatur (WN
Dusl erVD7
dog
Lait liver fear (MN-T)
fectum ((ART)
130
202
203
100
20
306
500
100
101
200
Opoch
kooch
Variation of the norms
(b) Within-elass variation
(c) Cosines between pairs
(d) Self-duality
Figure 4: Experiments on real datasets without weight decay. We trained a VGG13 on the MNIST datasct. The x-axis in the figures are set to have log(log(t)) scales and the y-axis in the figures are set to have log scales.
Lart layer Nanara (OWNET))
Lest uyer fusture (008657)
5441 243
303
30
100
0.59
520
207
Opoch
opech
Souch
Szuch
(a) Variation the norms
(b) Within-class variation
(c) Cosines between pairs
(d) Self-duality
Figure 5: Experiments on real datascts without weight decay. We trained a VGG18 on the KMNIST datasct.
The æ-axis in the figures are set to have (og())(()) scales and the y-axis in the figures are set to have log scales.
15
lover futura (ONNETI
Lest fasture (MMNIST)
0.2
4:59
0.2
Opoch
Opech
Eoach
(a) Variation of the norms
(b) Within-class variation
(c) Cosines between pairs
(d) SelF-duality
Figure 6: Experiments on real datasets without weight decay. We trained a ResNet18 8 on the KMNIST dataset.
The æ-axis in the figures to have log(l)g(L) scales and the y-axis in the figures have log scales.
Cara (FNNET)
Last eyer Teatard (TANETT
Qaeiter (TANITI
Daa
10-1
00
Opech
Eoech
(a) Variation of the norms
(b) Within-class variation
(c) Cosines between pairs
(d) Self-duality
Figure 7: Experiments real datasets without weight decay. We trained VGGI3 the Dashion-MNIST
dataset. The z--xxis in the figures are have log()o((t)) scales and the y-axis in the figures set have log scales.
27

---

Published as a conference paper at ICLR 2022
Ltd oyal Notam (HISN))
Lad Lyyrefre 39867)
00
02
0.22
03
300
500
epoch
tpem
Raach
(a) Variation of the norms
(b) Within-class variation
(c) Cosines between pairs
(d) Self-duality
Figure 8: Experiments on real datasets without weight decay. We trained a ResNet18 on the Mushion-MNISY dataset. The 2-axis in the figures are set to have log(log(t)) scales and the y-axis in the figures set to have log scales.
should be mentioned that we can observe that the variation of classifier norm stays at a low value and do not decrease in some settings (e.g.. Figure 7a and Figure 8a), this phenomenon is also found in Figure 2 of (Papyan et al., 2020), which might attribute to the network architecture (e.g., batch normalization) and characteristics of occurs for various network architectures and datasets under an unconstrained setting, which provides sound support for our theory.
datasets. Overall, we can find that neural collapse
E LIMITATION AND FUTURE DIRECTIONS
Currently, our analysis is still limited the terminal phase of training and only studies the training behavior after data is perfectly separated. Such assumption is necessary for all implicit regularization analysis (Lyu & Li, 2019; Ji et al., 2020; Nacson et al., 2019a) under a nonlinear setting, and how to fully characterize the early time training dynamics is still an open problem. Moreover, have provided a loss landscape analysis in the paper showing that there is no spurious minimum in the tangent space. To guarantee the training dynamics do not get stuck in saddle points and converge to neural collapse solution, we need to introduce randomness in the training dynamics (e.g. use stochastic gradient descent (Ge et al., 2015)), otherwise simple gradient flow may get stuck in saddle points as we have shown in Example 3.1. How to characterize the training dynamics of stochastic gradient descent in the nonlinear setting also an open problem. It is an interesting direction to further build a complete characterization of convergence to neural collapse solutions by addressing the early time training dynamics and studying the stochastic gradient descent.
28


Reviews:
[{"review_id": "aSDBzepKKGp", "reviewer": "Reviewer_Au7e", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting contribution. The problem analyzed in this paper is important, and their technique based on the optimization landscape coupled with the connection between the min-norm classifier and neural collapse is insightful. In addition, the paper is well written and clearly organized. \n\nMy main concerns are the following:\n\n- The narrative of the paper seems to be directed towards conveying the message that their studied model (without norm constraints) is more realistic, and in several places they comment on how other models studied in related works are \"further from practical networks\". I find these comments at least exaggerated, but potentially also inaccurate: in the studied model, the norms of the classifier and features diverge (they only converge in direction), which does not happen in practical deep learning models displaying neural collapse. As a result, one could also argue that the presented model is further from practical settings.. To be clear: I'm not trying to argue this is the case. I'm simply suggesting that trying to make a point along these lines sounds counter-productive, and the contribution of this paper would be better served without these subjective distractions.\n\n- I am slightly confused about the claim of the authors that they present a global analysis of the optimization landscape: it seems like their result holds for a proper initialization that has a loss function less than log(2), which implies that the features and classifiers are already separable. I agree with the authors that this is still relevant and interesting, as it precisely captures the behavior observed in the final stages of training. However, if this is the case, then the term \"global analysis\" is inaccurate, and this holds only upon a correct initialization. Could the authors explain why they refer to this analysis as global?\n\n- The empirical results on Fig.1 are slightly confusing to me: there are two different colors dedicated to \"last layer features\", in different graphs. Why is this so? Also, as far as I can tell, the variance of the class-specific classifiers does not seem to converge in either case, which contradicts the presented results. It is true that \"they stay within small values\", but this is different from the convergence results that the authors present as their main result. Can the authors explain this?\n\nMinor comments:\n- Page 2: \"Finally, We verify.. \"\n- page 7: \"unoptimal points\" -> non-optimal\n- Theorem 3.4 \"is (or is not) neural collapse solution\" -> \"is (or is not) a neural collapse solution\".\n\n\n\n\n\n\n", "summary_of_the_paper": "This paper analyzes the phenomenon of neural collapse from the simplified perspective of an unconstrained features models, whereby the only two optimization variables in the model are the last layer classifier and features, which are fit to some labels by minimizing the cross entropy loss. The authors leverage their observation that maximizing the margin results in a model achieving neural collapse, and then study the corresponding problem of minimum-norm classifier, and show that gradient flow converges directionally to the neural collapse solution. The authors then show that the optimization landscape is benign, in that all critical points that are not global minimum have a decreasing direction in the tangent space. ", "summary_of_the_review": "I believe this is a valuable contribution, but have some reservations about the presentation of the results. I would be happy to increase my rating after reading the responses by the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Pw0AQL0eXpx", "reviewer": "Reviewer_PTW6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper writing is good, and the presentation is clear\n- The theoretical part is interesting, which builds connection between max-margin implicit bias and NC, and the theories are seriously developed\n\nWeaknesses:\n- I find the empirical result section, i.e. Section 4, is quite weak. It only contains a Figure 1, which itself is not well-presented since it is hard for me to gain much insights. This section really needs to be improved.\n- Some statements are debatable. In the first bullet point of the main contributions, it says \"feature regularization is unrealistic and never used in practice\", but the L-2 norm regularization, or weight decay, is widely used in practice. In fact, in the experiments of [Papyan et al. 2020], it uses such weight decay by setting the parameter to be 1e-4 or 5e-4. Therefore, in Table 1, I don't think the \"feature norm regularization\" spot for [Papyan et al. 2020] should have a cross mark there.\n- The \"Notations\" section in the bottom of page 4 can be moved earlier; otherwise many notations are used widely before without introduction, e.g. [K]\n\n\nTypos:\n- In the second line after Eq. (1.1), the \"y_i\" is unbolded while I think it should be bolded?\n- In the paragraph above Eq. (2.3), the notation \"N\" should be \"n\"? since we are using \"n\" in Section 2.1\n- In the first equation in Section 3.1, \"k \\in [1,K], i \\in [1,n]\" --> \"k \\in [K], i \\in [n]\"\n", "summary_of_the_paper": "The paper studies the recently discovered Neural Collapse (NC) phenomenon for deep neural network training using Cross Entropy (CE) loss. It theoretically analyzes the problem with a so-called unconstrained layer-peeled model (ULPM), and then it shows that the ULPM with CE loss has a benign landscape.", "summary_of_the_review": "Overall, despite its weakness as I described above, this paper does provide some interesting approach in understanding the NC with CE loss. Considering its theoretical nature and the serious study, I would say it is marginally above the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JV2CW4UDitJ", "reviewer": "Reviewer_ez35", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this is a good paper and the result is interesting. The paper first shows a connection between the limiting point of gradient flow on the cross-entropy loss and the KKT solution of a minimum-norm separation problem. It also shows that every global optimal point of the minimum-norm separation problem satisfies the neural collapse condition. To deal with the concern that a KKT solution is not necessarily a global optimal point of the minimum-norm separation problem, a second-order landscape analysis of the minimum-norm problem is provided to show that at all the other KKT points of the minimum-norm separation problem, there is a direction that leads to a lower objective (of the Lagrangian of the minimum-norm problem).\n\nI think a clarification is needed on whether the paper shows that gradient flow on ULPM is guaranteed to return a solution that is a global optimum point of the minimum-norm separation problem, i.e. is guaranteed to converging to a point that exhibits the neural collapse. \n\nTypo: On the second line of the proof of Theorem 3.1, there is a typo regarding the definition of $\\tilde{w}_i$: $\\frac{1}{n} \\rightarrow \\frac{1}{ K}$.\n", "summary_of_the_paper": "This paper shows that for the cross-entropy loss, when updating both the weight vector and the features by gradient flow, the final solution exhibits the neural collapse behavior, where the neural collapse refers to an observation pointed out by (Papyan 2020), which is that (NC1) the within-class features collapse to their class mean, (NC2) the average of the features of each class converges to having the equal length and all pairs of the class-means have equal-size angles, (NC3) the weight and the class-mean are aligned, (NC4) the weight converges to the nearest class-center of the features.\nThe optimization problems (or the simplified model) in  the previous works that theoretically show the neural collapse require some constraints or regularization, which are seldom used in practice. The optimization model (ULPM) in this paper does not require these, and so is much closer to the realistic (of training deep networks).\n", "summary_of_the_review": "(see above)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "I37_RxTRfq", "reviewer": "Reviewer_sXBT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\nThis paper studies neural collapse, a widely observed phenomenon in training deep neural networks. Existing work on the analysis of neural collapse is based on either constraints or regularizers on the features and classifiers. The authors analyze the gradient flow of the ULMP without any constraints or regularizers, and provide second-order analysis of the objective function, ensuring gradient descent converges to neural collapse solutions. \n\n## Weaknesses\n1. Weight decay is often employed in training neural networks. This paper studies the training problem without any regularizers, but it is unclear why this formulation is closer to the practice than the one with regularizers. \n\n2. Eq. (3.3) only shows that a non-optimal solution is not a local minimum. It may be a higher-order saddle. If the results show the (Riemannian) Hessian has a negative eigenvalue, then I think this is a much stronger result and should be described in Theorem 3.4.  \n\n2. As mentioned in the paper, the training dynamics will diverge to infinity where the cross-entropy loss archives its minimum. Thus, is possible that eq. (3.3) holds for any finite $(W,H)$ (if we allow any direction for $(\\Delta W,\\Delta H)$), i.e., any finite $(W,H)$ is not a local solution?\n\n3. Which manifold does the tangent space in Definition 3.1 refers to? In eq. (3.3), why highlight the direction on the tangent space? \nIn general, the tangent space is used in the manifold optimization where the variables are constrained on the manifolds, but here the training loss has no constraints. Can you clarify this? Also in Remark 3.4, it says Hessian matrix. Is it Riemannian Hessian? ", "summary_of_the_paper": "This paper provides new analyses for understanding the neural collapse phenomenon observed in training deep learning classifiers. In particular, to understand the last-layer features and classifiers, the authors study the gradient flow of the unconstrained layer-peeled model (ULPM) without any regularizers and show that gradient flow converges to neural collapse classifiers and features. Experiments are provided to demonstrate the neural collapse phenomenon when no weight decay is used.", "summary_of_the_review": "Overall, this paper provides new analyses for understanding the neural collapse phenomenon. The analyzed formulation removes the constraints and regularizers on the features and constraints, which is different from the existing approach.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper follows the recent line of work of theoretically analyzing the Neural Collapse phenomenon, by making certain simplifying assumptions on the problem setup. In this case, the authors use cross-entropy loss on an unconstrained model where second-to-last representations become free variables. Their main results characterise the NC as the only global minimiser. 
Reviewers were positive about this work, and concluded it presents a valuable addition to the growing analysis of NC. They also pointed out several clarity issues that should be addressed in the final revision, including a more objective comparison to prior work. Ultimately this work will be an interesting addition to the conference, and therefore the AC recommends acceptance.
Decision: Accept (Poster)