Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold
Authors: Can Yaras
Peng Wang
Zhihui Zhu
Laura Balzano
Qing Qu
Venue: NeurIPS 2022 Accept (2022)
Keywords: neural collapse
Riemannian manifold
feature normalization
nonconvex optimization

Abstract: When training overparameterized deep networks for classification tasks, it has been widely observed that the learned features exhibit a so-called "neural collapse'" phenomenon. More specifically, for the output features of the penultimate layer, for each class the within-class features converge to their means, and the means of different classes exhibit a certain tight frame structure, which is also aligned with the last layer's classifier. As feature normalization in the last layer becomes a common practice in modern representation learning, in this work we theoretically justify the neural collapse phenomenon under normalized features. Based on an unconstrained feature model, we simplify the empirical loss function in a multi-class classification task into a nonconvex optimization problem over the Riemannian manifold by constraining all features and classifiers over the sphere. In this context, we analyze the nonconvex landscape of the Riemannian optimization problem over the product of spheres, showing a benign global landscape in the sense that the only global minimizers are the neural collapse solutions while all other critical points are strict saddle points with negative curvature. Experimental results on practical deep networks corroborate our theory and demonstrate that better representations can be learned faster via feature normalization. Code for our experiments can be found at https://github.com/cjyaras/normalized-neural-collapse.

Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold
Can Yaras* University of Michigan ejyarasQumich edu
Peng Wang"
Zhihui Zhu Ohio State University zhu 34400osu. edu
University of Michigan pengwadumich. edu
Laura Balzano
Qing Qu
University of Michigan girasole@umicom ch. edu
University of Michigan q qingquBumich. edu
Abstract
When training overparameterized deep networks for classification tasks, has been widely observed that the learned features exhibit so-called "neural collapse" phenomenon. More specifically, for the output features of the penultimate layer, for each class the within-class features converge their means, and the means of different classes exhibit certain tight frame structure, which is also aligned with the last layer's classifier. As feature normalization in the last layer becomes a common practice in modern representation learning, in this work we theoretically justify the neural collapse phenomenon under normalized features. Based on an un- constrained feature model, we simplify the empirical loss function in multi-clas class classification task into a nonconvex optimization problem over the Riemannian manifold by constraining all features and classifiers over the sphere. In this context, we analyze the nonconvex landscape of the Riemannian optimization problem over the product of spheres, showing benign global landscape in the sense that the only global minimizers are the neural collapse solutions while all other critical points are strict saddle points with negative curvature. Experimental results practical deep networks corroborate our theory and demonstrate that better representations can be learned faster via feature normalization. Code for experiments be found at https: /gi tithu...... com/ eeayaras/normalil neural collapse.
Introduction
Despite the tremendous success of deep learning in engineering and scientific applications over the past decades, the underlying mechanism of deep neural networks (DNNs) still largely remains mysterious. Towards the goal of understanding the learned deep representations, a recent line of seminal works [1-5] presents intriguing phenomenon that persists across range of canonical classification problems during the terminal phase of training. Specifically, has been widely observed that last-layer features (i.c., the output of the penultimate layer) and last-layer linear classifiers of trained DNN exhibit simple but elegant mathematical structures, in the sense that
the individual features of each class concentrate to their class-mess means. (NC2) Convergence to Simplex ETF: the class-mean have the same length and are maximally distant. In other words, they form a Simplex Equiangular Tight Frame (ETF).
(NC1) Variability Collapse:
"Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a)
(b)
Feature
Feature Classifier
Classifier
Figure 1: Comparison of features found with and without normalization. K 100 classes with n samples per class. Features and classifiers are found through optimizing the cross-entropy loss with UFM, where features embedded in 2-dimensional space, i.c., d = 2. (a) No normalization of features or classifiers. (b) Features and classifiers are constrained to the unit sphere (features are scaled down for visualization purposes).
Table I: Average UFM feature loss and accuracy over 10 trials with and without normalization to sphere, with the same set-up as in Figure 1.
Average CE Loss
Average Accuracy
No Normalization Normalization
1.63 + 0.03
49.9% + 2.39%
3.84 = 0.00
100.0% + 0.00%
(NC3) Convergence to Self-Duality: the last-layer linear classifiers perfectly match their class- means.
Such phenomenon is referred to as Neural Collapse (NC) [1], which has been shown empirically
to persist across a broad range of canonical classification problems, on different loss functions
(e.g., cross-entropy (CE) [1,3,4] 3,4 mean- -quared error (MSE) [5, 6]. and supervised contrasive (SC)
losses [7]. etc.), different neural network architectures (e.g.. VGG [8]. ResNet [9], and DenseNet [10],
etc.), and on variety of standard datasets (e.g., MNIST [1 1J. CIFAR [12), ImageNet [13], etc.).
Recently, independent lines of research, many works are devoted learning maximally compact
and separated features; see, e.g., [14-22]. This has also been widely demonstrated in number of
recent works [23-29], including state-of-the-an natural language models (such as BERT, RoBERTa, and GPT) [24].
Motivations & contributions. In this work, we further demystify why ACC happens network training with feature normalization (i.e., normalizing the last-layer features the unit hypersphere), mainly motivated by the following reasons:
Feature normalization is a common practice in training deep networks. Recently, many existing results demonstrated that training with feature normalization often improves the quality of learned representation with better class separation [7, 16, 18-20,30-32], Such representation is closely related to the discriminative representation in literature; see, e.g., [16, 18, 19, 333. As illustrated in Fig. 1 and Table 1, experimental results visualized in low-dimensional space show that features learned with normalization are more uniformly distributed over the sphere and hence are more linearly separable than those learned without normalization. In particular, it has been shown that the learned representations with larger class separation usually lead improved test performances: see, e.g., [7.34]. Moreover, it has been demonstrated that discriminative representations can also improve robustness to mislabeled data [30, 31], and has become a common practice in recent advances (self-supervised) pre-trained models [32,35].

---

A common practice of theoretically studying NC with norm constraints. Due to these practical reasons, many existing theoretical studies on NC consider formulations with both the norms of features and classifiers constrained [3, 7.36-88.. Based upon assumptions of unconstrained feature models [3, 4, 39], these works show that the only global solutions satisfy NC properties for a variety of loss functions (e.g., MSE, CE, SC losses, etc). Nonetheless, they only focused on the global optimality conditions without looking into their nonconvex landscapes, and therefore fail to explain why global NCC solutions can be efficiently reached by classical training algorithms such as stochastic gradient descent (SGD).
In this work, we study the global optimization landscape of the training loss with norm constraints on the features and classifiers. We consider the commonly used CE loss and formulate the problem as a Riemannian optimization problem over products of unit spheres (i.e., the oblique manifold). Our study is also based upon the assumption of the so-called unconstrained feature model (UFM) [4,5,39] or layer-peeled model [40], where the last-layer features of the deep network are treated as free optimization variables to simplify the nonlinear interactions across layers. The underlying reasoning is that modern deep networks are often highly overparameterized with the capacity of learning any representations [41-43], so that the last-layer features can approximate, or interpolate, any point in the feature space.
Assuming the UFM, we show that the Riemannian optimization problem has a benign global land- scape, in the sense that the loss with respect to (w.r.t.) the features and classifiers is a strict saddle function [44,45 over the Riemannian manifold. More specifically, we prove that every local min- imizer is a global solution satisfying the NC properties, and all the other critical points exhibit directions with negative curvature. Our analysis for the manifold setting is based upon a nontriv- ial extension of recent studies for the NC with penalized formulations [2, 6-). which could be of independent interest. Our work brings new tools from Riemannian optimization for analyzing optimization landscapes of training deep networks with an increasingly common practice of feature normalization. At the same time, we empirically demonstrate the advantages of the constrained formulation on the manifold over its penalized counterpart for training deep networks - faster training and higher quality representations. Lastly, under the UFM we believe the benign landscape over the manifold could hold for many other popular training losses beyond CE, such as the (supervised) contrastive loss [34]. We leave this for future exploration.
Prior arts and related works NC. The empirical NCC phenomenon has inspired recent line of theoretical studies on understanding why it occurs [4-7.66, .9, 40]. Like ours, most of these works studied the problem under the UFM. In particular, despite the nonconvexity, recent works showed that the only global solutions are NCC solutions for a variety of nonconvex training losses (e.g.. CE [4, 40J, MSE [5.6]. SC losses (7]) and different problem formulations (c.g.. penalized, constrained, and unconstrained) [2, 12.4-6,36) 36]. Recently, this study has been extended to deeper models with the MSE training loss [6]. More surprisingly, it has been further shown that the nonconvex losses under the UFM have benign global optimization landscapes, in the sense that every local minimizer satisfies NC properties and the remaining critical points are strict saddles with negative curvature. Such results have been established for both CE and MSE losses [4,5], where they considered the unconstrained formulations with regularization on both features and classifiers. We should also mention that the benign global optimization landscapes of many other problems in neural networks have been widely found in the literature; see, e.g., [46-50].
Moreover, there is line of recent works investigating the benefits NC on generalization of deep networks. The work [51] showed that CCC also happens on test data drawn from the same distribution asymptotically, but with less collapse for finite samples [52]. Other works 53] demonstrated that the variability collapse of features is actually happening progressively from shallow to deep layers, and [54] showed that test performance can be improved when enforcing variability collapse on features of intermediate layers. The works 565 showed that fixing the classifier as a simplex ETF improves test performance on imbalanced training data and long-taila tailed classification problems. For more details on related works, we refer the readers to the Appendix A.
Notation.
Let IR" be the n-dimensional Euclidean space and || |la be the Euclidean norm. We write matrices in bold capital letters such as A, vectors in bold lower-case such as and scalars in plain letters such as a. Given a matrix E TR we denote its k-thh column by ag its i-th row by a', its (i. --th element by ay, and its spectral norm by || We use (Aa to denote vector
3

---

that consists of diagonal elements of A, and we use ddiag A) to denote diagonal matrix composed by only the diagonal entries of A. Given a positive integer n. we denote the set {I...., n3 by [ni]. We denote the unit hypersphere in IR by Sd := {o Rd ||]|| 1}.
2 Nonconvex Formulation with Spherical Constraints
In this section, we review basic concepts of deep neural networks and introduce notation that will be used throughout the paper. Based upon this, formally introduce the problem formulation over the Riemannian manifold under the assumption of the UFM.
2.1 Basics Deep Neural Networks
In this work, we focus on the multi-class (e.g., K class) classification problem. Given input data RR,, the goal of deep learning learn deep hierarchical representation (or feature) h(x)
Φ0(x)= E Rd of the input along with linear classifier' W I k such that the output the (x)
W hxx) of the network fits the input æ: to one-hot training label y R.K. More precisely, in vanilla form an L-layer fully connected deep neural network can be written as
vh (x)
WL
(WL-1" (Wix b1) DL-1) + PL.
(1)
linear classifier W W
feature h
<he (x))
where each layer is composed of an affine transformation, represented by some weight matrix Wi. and bias by- followed by a nonlinear activation (.), and @ = { (Wa. ba) and 0 = ( [Wi k. by 1-1
denote the weights for all the network parameters and those up the last layer, respectively. Given
training samples {< } C RD x KK drawn from the same data distribution D, we learn the network parameters θ via minimizing the empirical risk over these samples,
K Lce (Vo( yk), s.t. OEC,
min
where yı E R one-hot vector with only the kth entry being and the remaining ones being 0 for all l E [K], Ik,i E RD is the i-th sample in the k-th class, {ng { denotes the number of training samples in each class, and the set C denotes the constraint set of the network parameters θ that we will specify later.
Here, we study the most widely used CE loss of the form
exp(2k) Ek exp(zt)
LOE(z, yu) = - log
K
2.2 Riemannian Optimization over the Product of Spheres
For the K-class classification problem, let us consider simple case where the number of training samples in each class is balanced (i.e., n M1 = 72
nx) and N Kn. We assume that the bias of the final layer by is zero with the last activation function a--) before the output being linear. Analyzing deep networks the (z) is a tremendously difficult task mainly due to the nonlinear interactions across large number of layers. To simplify the analysis, we assume the so-called unconstrained feature model (UFM) following the previous works 7,3,,38,33 39]. More specifically, we simplify the nonlinear interactions across layers by treating the last-layer features hu.i = Φe ( E IRd as free optimization variables, where the underlying reasoning is that modern deep networks are often highly overparameterized to approximate any continuous function [41-43]. Concisely, we write all the features in matrix form as
H = H H2
HK] E d Hş
[
h
haal E Rdxn, V E [K],
and correspondingly write the classifier W as
W =
[ur]
W2
wx]
dx
wo E Rd. Ak e KK].
Wee write W W in the transposed form for the simplicity of analysis.

---

Based upon the discussion in Section 1, we assume that both the features H and the classifiers W are normalized, i.e. | and ||w = T for all k E [K] and all E [ra]. where is
temperature parameter. As result, we obtain a constrained formulation of the NC problem over Riemannian manifold
1 N
min W.H
LOE W hige We) s.t. || T, ||he,il|2 1, vie [m), 4A [
(3)
Since the temperature parameter T can be absorbed into the loss function, we replace wy by Tw, and change the original constraint into ||wall2 constraints forms an oblique manifold [57] embedded Euclidean space,
for all k E [K]. Moreover, the product of spherical
OB(d, K) = {z E R E A l E [K]]
Consequently, we can rewrite Problem (3) Riemannian optimization problem over the oblique manifold w.r.t. W and H, i.e.
K
min f(W, H = W
Σ
LCE (T
hg.i
N
k=
s.t. H O), W E ().
In Section 3, we will show that all global solutions of this problem satisfy NC properties, and its objective function is strict saddle function [58,59] of (W, H) over the oblique manifold so that the NCC solution can be efficiently achieved.
f(h)
M $2
Riemannian derivatives over the oblique manifold. In Section 3, we will use tools from Riemannian optimization to characterize the global optimality condition and the geometric properties of the optimization landscape of Problem (4). To proceed, let first briefly introduce some basic derivations of the Riemannian gradient and Hessian, defined on the tangent space of the oblique manifold. For more technical details, we refer the readers to Appendix B. According to [57, Chapter 3 & 5] and [60,61],
gradf(h)
T
calculate the Riemannian gradients and Hessian of Problem (4) as follows. Since those quantities defined
the tangent space, according [60, Section 3. I] and the illustration in Figure 2, we first have the tangent
space OB(d, K) at W as
Figure An illustration of the Rie- mannian gradient of f(h) on sim-
Tw K) = Z d diag (W 2) =3.
ple manifold OB(3, 1) = $2.
This indicates that the tangent space contains all Z such that 25 is orthogonal to for all k;; when K 1, it reduces the tangent space to the unit sphere $d-1
Analogously, we can derive the tangent space for H with a similar form. Let us define
M = TT H, 9(M) == f(W, H).
First, the Riemannian gradient of f(W, .H) of Problem (4) is basically the projection of the ordinary Euclidean gradient VS(W, H)) onto its tangent space, i.e., gradw J(W, H) = f(W, H)) and grad J(W, H) (V H)). More specif-
PTwoBBa (V (V
ically, we have
grad w S(W, H) (H) T W ddiag (W" HVg(W)
grad h f(W, H) =WWVg(M) H ddiag (H WVg(MM).
(6)
Second, for any A (Aw: AH) E Raak dd compute the Hessian bilinear form of J(W, H) along the direction A by
V H|A,
V (M) [ (WT AH + H) T (WT AH + H)
2T (M) AH)
(7)
II practice, it is common practice normalize the output feature h. by its norm, i.e., h = ha// 11/. that [/hll = 1.
5

---

We compute the Riemannian Hessian bilinear form of f(W, H)) along any direction A OB K) TB) by
Hess (WWH))AAA = S(W H)IA, A] (Aw ddiag (MVg(M))) Aw)
(AH ddiag M Vg(M)) Ah).
(8)
where the extra terms besides f(, HH),,,A,,, A compensate for the curvature on the oblique manifold. For derivations of (7) and (8), see Appendix B.2. In the following section, we will use the Riemannian gradient and Hessian to characterize the optimization landscape of Problem (4).
3 Main Theoretical Analysis
In this section, we first characterize the structure of the global solution set of Problem (4). Based upon this, we analyze the global landscape of Problem (4) via characterizing its Riemannian derivatives.
3.1
Global Optimality Condition
For the feature matrix H, let us denote the class mean for each class
H
h1
hg] E TRdx where hg
72 Σ
he.: 1 1 K.
Based upon this, we show any global solution of Problem (4) exhibits NC properties in the sense that it satisfies (NC1) variability collapse, (NC2) convergence to simplex ETF, and (NC3) convergence to self-duality.
Theorem (Global Optimality Condition). Suppose that the feature dimension is smaller than the number of classes, i.e. d K, and the training labels are balanced in each class, i.e., n n2 nK Then for the CE loss f(W, H) in Problem (4), it holds that
(K-1)
KT
f(W, H) 2 log
1 (K-1)exp {
(K-1))
for all W [wi,.... wxl E OB(d, K) and H -h..1...... hK,rl NB.d,N). In particular, equality holds if and only if
(NC1) Variability collapse: h hus V [n]:
(NC2) Convergence to Simplex ETF: { 1K form a
simplex ETF in the sense
that
H 1 K- (KIK - 1xlk), H OB(d,)..
(NC3) Convergence to Self-duality: un = h4g vk E [K].
Compared the unconstrained regularized problems in [4.5], it is worth noting that the regularization parameters there influence the structure global solutions, while the remperature parameter r only affects the optimization landscape but not the global solutions. On the other hand, the result of our problem (4) is closely related to [7, Theorem 1] (i.c., spherical constraints vs. ball constraints). In fact, our problem and that in [7] share the same global solution set. As such, the proof follows similar ideas of a line recent works [4-7,36 16, 40], and we refer the readers to Appendix for the proof. should be noted that do not claim originality of this result compared to previous works. Instead, our major contribution lies in the following global landscape analysis.
3.2 Global Landscape Analysis
Due to the nonconvex nature of Problem (4), the characterization of global optimality alone in Theorem is not sufficient for guarantceing efficient optimization those desired global solutions. Thus, we further study the global landscape of Problem (4) by characterizing all the Riemannian critical points (W, H) E (B,d, K) x OB(d, N) satisfying
grad и f(W, H 0, grad f(W, H) 0.
We now state major result below.

---

3.0
2.5
Theoretical Empirical
Theoretical
2.5
2.0
Empirical
Optimal Value
2.0-
Optimal Value
1.5
1.5-
1.0-
1.0
0.5
0.5
0.0
0.0
10
20 40 50 10-3
10
10-1
10a
10
10"
10
K
Figure 3: Global optimization of (4) under UFM with d= 100 and = 5. Theoretical line refers to lower bound (global minimum) from Theorem 1. Empirical values found using gradient descent with random initialization. Left: Lower bound against number of classes K while fixing = 1. Right: Lower bound against temperature r while fixing K 10. The same empirical values are achieved over many trials due to the benign global landscape.
Theorem 2 (Global Landscape Analysis). Assume that the number of training samples in each class balanced, i.e., n = n1 =1.=... =.... = nkk If the feature dimension is larger than the number of classes, i.e., d > K, and the temperature parameter satisfies < 2(d 2) (K mod 2)/K)-1 then the function f(W, H) is a strict saddle function that has no spurious local minimum, in the sense that
Any Riemannian critical point (W; H) of Problem (4) that is not local minimizer mannian strict saddle point with negative curvatures, in the sense that the Riemannian Hes- sian Hess (W, H) at the critical point (W. HH) is neg-degee, and there exists a direction A (Aw AT) OB(d, K) THOB(d, N) such that
Rie-
Hess f(W, H.JAA <.
In other words, Amin (Hess (W, H)) < 0 at the corresponding Riemannian critical point.
Any local minimizer of Problem (4) a global minimizer of the form shown in Theorem 1.
For the details of the proof, refer readers to Appendix D. The second bullet point naturally follows from Theorem and the first bullet point. The major challenge of our analysis is showing the first bullet, i.e., to find negative curvature direction A for Hess f(W, H). Our key observation is that the set of non-global critical points can be partitioned into two separate cases. In the first case, the last two terms of (8) vanish, and we show that the second term of (7) is negative and dominates the first term for an appropriate direction. We require to not be too large, since the first term is O(7²), whereas the second term O In the second case, using the assumption that d K can find rank-one direction that makes the first term of (7) vanishing. In this case, we similarly show that the second term of (7) is negative but instead dominates the last two terms of (8), In the following, we discuss the implications, relationship. and limitations of our results in Theorem 2.
Efficient global optinization to NC solutions. Our theorem implics that the NC solutions can be efficiently reached by Riemannian first-order methods (c.g.. Riemannian stochastic gradient descent) with random initialization [58, 62, 63]; see Figure for a demonstration. For training practical deep networks, this be efficiently implemented by normalizing last-layer features when running SGD.
Relation to existing works on NC. Most existing results have only studied the global minimizers under the UFM [6, 7, 70J, which has limited implications for optimization. On the other hand, our landscape analysis is based upon nontrivial extension of that with the unconstrained problem formulation [4, 5]. Compared to those works, Problem (4) is much more challenging for analysis, due to the fact that the set of critical points of our problem is essentially much larger than that of [4.5]. Moreover, we empirically demonstrate the advantages of the manifold formulation over its regularized counterpart, in terms of representation quality and training speed.
7

---

Assumptions on the feature dimension d and temperature parameter T. Our current result requires that d > K, which the same as that in [4. Furthermore, through numerical simulations we conjecture that the global landscape also holds even when d < K, while the global solutions are uniform over the sphere [36] rather than being simplex ETFs (see Figure 1). The analysis on d < K is left for future work. On the other hand, the required upper bound on is for the ease of analysis and holds generally in practice, 3 but we conjecture that the benign landscape holds without it.
Relation to other Riemannian nonconvex problems. Our result joins recent line of work on the study of global nonconvex landscapes over Riemannian manifolds, such as orthogonal tensor decomposition [44], dictionary learning [64-68], subspace clustering [69], and sparse blind de- convolution [70-73]. For all these problems constrained over Riemannian manifold, it can be shown that they exhibit "equivalently good" global minimizers due to symmetries and intrinsic dow-dimensional structures, and the loss functions are usually strict saddles [44, (44,45, 74]. As we can see, the global minimizers (i.e., simplex ETFs) of our problem here also exhibit a similar rotational symmetry, the sense that W H = (QW) T (QH) for any orthogonal matrix Q. Additionally, our result shows that tools from Riemmanian optimization can be powerful for the study of deep learning.
4 Experiments
In this section, we support our theoretical results previous sections and provide further motivation with experimental results on practical deep network training. In the first experiment, we validate the assumption of UFM introduced in Section for analyzing NC, by demonstrating that NCC occurs for increasingly overparameterized deep networks. In the second experiment, we further motivate feature normalization with empirical results, showing that feature normalization can lead to faster training and better collapse than the unconstrained counterpart with regularization. This occurs not only with the UFM but also with practical overparameterized networks. We detail the network architectures, datasets, training details, and metries used in these experiments, as well as additional experiments in Appendix E.
4.1 Validation of the UFM for training networks with feature normalization
In Section 2, our study of the Riemannian optimization problem (4) is based upon the UFM, where we assume H is free optimization variable. Here, we justify this assumption by showing that NC happens for training overparameterized networks even when the training labels are completely random. By using random labels, we disassociate the input from their class labels, by which we can characterize the approximation power of the features of overparameterized models. To show this, we train ResNet-18 with varying widths (i.e., the number of feature maps resulting from the first convolutional layer) on CIFARIO with random labels, with normalized features and classifiers.
As shown in Figure 4, we observe that increasing the width of the network allows for perfect classification on the training data even when the labels are random. Furthermore, increasing the network width also leads to better NC, measured by the decrease in cach.VC metric. This validates that (i) our assumption of UFM is reasonable given that NC seems to be independent of the input data, and (ii) NC happens under the constrained formulation (4) on practical networks.
4.2 Improved training speed and better representation quality with feature normalization
We now investigate the benefits of using feature normalization, more specifically for improved training speed and better representation quality. First, we consider the UFM formulation, where we optimize Problem (4) and compare to the regularized UFM in [4]. The results are shown in Figure 5. We can see that normalizing features over the sphere consistently results in reaching perfect classification and greater feature collapse (i.c., smaller NC1,) quicker than penalizing the features. To demonstrate that these behaviors are reflected in training practical deep networks, we train both ResNet-18 and ResNet-50 architectures on a reduced CIFAR 100 [12] dataset with N = 3000 total samples, comparing the training accuracy and metrics of NC with and without feature normalization. The results are shown in Figure 6.
'For instance, a standard ResNet- 18 [9] model trained on CIFAR- 10 [12] has d = 512 and K = 10. In the same setting, we assume r < 1020, which is far larger than any useful setting of the temperature parameter (see Appendix E.3).
8

---

Wath-m
Whtthmm
Wethhdd
Wishhmm
With-m
3.-
Vddthma
3.5
Wistheg
Wichh
Witth-16
Wdth-166
Wetth16
Trainin Accuracy
Wishh-s
Wieth-12
Withh
Withh644
Wiath-64
Wiath-64
1.0
3.0
0.00
Epach
Epach
Epoch
Epoch
Figure 4: Validation of UFM ResNet with varying network width. NCC metrics and training accuracy of ResNet-18 networks of various widths on CIFAR 10 with n 200 over 200 epochs.
256
d 1024
10
M
1.
training Accuracy
14.
Training Accurscy
10
LF
Re
Xx--
82
NormalinddFabbe
15a
Normwkzedd-ahhe
Narmeliced-Taon
10-1
Kormelized-Tads
Rormaliad- True
Normalleed Vras
hormalicndd True
Normalicad Trum
ngg
15
15
33
12
25
20
10-1
20
35
Iteration
Figure 5: Faster training/feature collapse of UFM with feature normalization. Average (deviation denoted by shaded area) training accuracy and NC1 of UFM over 10 trials of (Riemmanian) gradient descent with backtracking line search. We set = 100 classes, n 30 samples per class.
From Figure (left), we can see that for the ResNet- 18 network, we reach perfect classification of the training data about 10-20 epochs sooner by using feature normalization compared to that of the unconstrained formulation. From Figure (right), training the ResNet-50 network without feature nor- malization for 100 epochs shows slow convergence with poor training accuracy, whereas using feature normalization arrives at above 90% training accuracy in the same number of epochs. By keeping the size of the dataset the same and increasing the number of parameters, it is reasonable that optimizing the ResNet-50 network more challenging due to the higher degree of overparameterization, yet this effect is mitigated by using feature normalization.
At the same time, for both architectures, using feature normalization leads to greater feature collapse (i.e., smaller NC1) compared to that of the unconstrained counterpart. As shown in recent work [1,5, 51], better NCC often leads to better generalization performance. Experimental results demonstrating that feature normalization generalizes better than regularization are provided in Appendix E.2. Last not least, we conjecture that the benign landscape holds beyond the CE loss (e.g.. SC [7], see Appendix E.4); and we believe the benefits of feature normalization are not limited to the evidence that we showed here, as it could also lead to better robustness [30,31] 31] 31] that is worth further exploration.
Conclusion & Discussion
In this work, we study the prevalence of the NC phenomenon with normalized features. Based upon the assumption of UFM, formulate the problem Riemannian optimization problem over the product of spheres. We showed that the loss function is a strict saddle function over the manifold with respect to the last-layer features and classifiers, with no other spurious local minimizers. We demonstrate this on practical deep network training, and show practical benefits of feature normalization in terms of training speed and learned representation quality. As future work, we would like to expand the study to other popular loss functions such as contrastive loss and study the settings d & K, which could be of great importance for studying self-supervised learning.
Acknowledgement
Can Yaras and Qing Qu acknowledge support from U-M START & PODS grants, NSF CAREER CCF 2143904, NSF CCF 2212066. NSF CCF 2212326, and ONR N00014-22-1-2529 grants. Peng Wang and Laura Balzano acknowledge support from ARO YIP 1911NF1110022 AFOSR YIP

---

ResNet-18
ResNet-50
1.0
10
NormalzzedRitee
kormalzeeiaatrr hormataes-Tuue
Training Accuracy
10
Normalzsss Fue
Training Accurs
Normdicac-fiise
Normukadd-lsbe
Normaiied=-ive
64
15
180
11-11
25
50
150
23
55 Epoch
75
300
10-1
25
53
75
200
Epoch
Epach
Epoch
Figure 6: Faster training/teature collapse with ResNet on CIFAR100 with feature normalization. Training accuracy and NC1 of ResNet-18 18 and Re on CIFARI00 with n 30 over 100 epochs.
1A9550-19-1-0026 and NSF CAREER CCF-1845076. Zhihui Zhu acknowledges support from NSF grants CCF-2240708 and CCF-2241298. We would like thank Zhexin Wu (ETH), Yan Wen (Tsinghua), and Pengru Huang (UMich) for fruitful discussion through various stages of the work.
References
[1] Vardan Papyan, XY Han, and David I. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117/400)24652-24666 2020.
XY Han, Vardan Papyan, and David L. Donoho. Neural collapse under mse loss: Proximity to and dynamies on the central path. arXiv preprint arXiv:2166.02000 106.02073, 2021.
[3] Cong Fang, Hangfeng He, Qi Long, and Weijie Su. Exploring deep neural networks via layer-peeled model: Minority collapse imbalanced training. Proceedings of the National Academy of Sciences, 118(43), 2021.
[4] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34, 2021.
[5] Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization land- scape of neural collapse under mse loss: Global optimality with unconstrained features. arXiv preprint wwXiv:2203.01255 2022.
[6] Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. arXiv preprint (rXii:2202.00087 2022.
Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised constrastive learning. In International Conference on Machine Learning, pages 3821-3830. PMLR, 2021.
[8] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 14409.1556 2014.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[10] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[11] Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit datahase. AT&T labs, 2010.
[12] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[13] Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pages 248-255. leee, 2009.
[14] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European conference computer vision, pages 499-515. Springer, 2016.
[15] Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. arXiv preprint 161.02961..022255 2016.
[16] Rajeev Ranjan, Carlos D Castillo, and Rama Chellappa. L2-constrained softmax loss for discriminative face verification. arXiv preprint 1903 2017.
10

---

[17] Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international conference on Multimedia, pages 1041-1049, 2017.
[18] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj. and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 212- 220, 2017.
[19] Hao Wang. Yitong Wang. Zheng Zhou, Xing Ji. Dihong Gong, Jingchao Zhou, Zhifeng Li. and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5265-5274 2018.
[20] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafciriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pages 1690-4699 2019.
[21] Federico Pernici. Matteo Bruni, Claudio Baecchi. and Alberto Del Bimbo. Maximally compact and separated features with regular polytope networks. In CVPR Workshops, pages 46-53, 2019.
[22] Federico Pernici, Mattco Bruni, Claudio Baccehi, and Alberto Del Bimbo. Regular polytope networks. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[23] Uri Cohen, Yuon Chung, Daniel Lee, and Haim Sompolinsky. Separability and geometry object manifolds in deep neural networks. Nature communications, 11(1):1-13, 2020.
[24] Jonathan Mamou, Hang Le. Miguel Del Rio, Cory Stephenson, Hanlin Tang. Yoon Kim, and Sueyeon Chung. Emergence of separable manifolds in deep language representations. International Conference on Machine Learning, pages 6713-6723, PMLR, 2020.
[25] Diego Doimo, Aldo Gliclmo, Alessio Ansuini, and Alessandro Laio. Hicrarchical nucleation in deep neural networks. Advances in Neural Information Processing Systems, 33:7526 2732666776666 2020.
[26] Gregory Naitzat, Andrey Zhitnikov, and Lek- Lek-Heng Lim. Topology of deep neural networks. J. Mach. Learn. Res., 21(184):1 101 2020.
[27] Stefano Recanatesi, Matthew Farrell, Madhu Advani, Timothy Moore, Guillaume Lajoie. and Eric Shea-Brown Dimensionality compression and expansion in deep neural networks. arXiv preprint arXiv: 1906. 00440, 2019.
[28] Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton. Analyzing and improving representations with the soft nearest neighbor loss. In International conference on machine learning, pages 2012 2020. PMLR, 2019.
[29] Christoph Hofer, Florian Graf, Marc Niethammer, and Roland Kwitt. Topologically densified distributions. In International Conference on Machine Learning, pages 4304-4313 PMLR, 2020.
[30] Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Ma. Deep networks from the principle of rate reduction. arXiv preprint arXiv:2010. 14765, 2020.
[31] Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song. and Yi Ma. Learning diverse and discrimi- native representations via the principle of maximal coding rate reduction. arXiv preprint arXiv:2006. 085588 2020.
[32] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through ali alignment and uniformity on the hypersphere. pages 9929-99923 (e0e99999,2220 2020.
[33] En Yu, Zhuoling Li, and Shoudong Han. Towards discriminative representation: Multi-view trajectory contrastive learning for online multi-object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8834-8843, 2022.
[34] Prannay Khosla, Piotr Teterwak. Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661-18677,, 2020.
[35] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020.
[36] Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross cross-entropy loss. arXiv preprint axXiv200/2008663 2020.
[37] Stephan Wojtowytsch et al. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. arXiv preprint arXtv:2012.004220 2020.
[38] Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer-peeled perspective on neural collapse. arXiv preprint a0X.v:21 2021.
[39] Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:0011.1169, 2020.
11

---

[40] Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Layer-peeled model: Toward understanding well-trained deep neural networks. arXiv preprint arXiv:2101. 12699, 2021.
[41] Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: a view from the width. In Proceedings of the 31st Interaatonaal Conference on Neural Information Processing Systems, pages 6232-6240, 2017.
[42] Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2):251-
257, 1991.
[43] Uri Shaham, Alexander Cloninger, and Ronald R Coifman. Provable approximation properties for deep neural networks. Applied and Computational Harmonic Analysis, 44(3):537 -577, 2018.
[44] Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points-online stochastic gradient for tensor decomposition. In Proceedings of The 28th Conference Learning Theory, pages 797-842, 2015.
[45] Ju Sun, Qing Qu, and John Wright. When are nonconvex problems not scary? arXiv preprint arXiv:1510.06090 2015.
[46] Jiawei Zhang, Yushun Zhang, Mingyi Hong, Ruoyu Sun, and Zhi Zhi-Quan Luo. When expressivity meets trainability: Fewer than n neurons can work. Advances in Neural Information Processing Systems, 34:9167-9180, 2021.
[47] Thomas Laurent and James Brecht. Deep linear networks with arbitrary loss: All local minima global. In International conference on machine learning, pages 2902-2907. PMLR, 2018.
[48] Ruoyu Sun, Dawei Li, Shiyu Liang, Tian Ding, and Rayadurgam Srikant. The global landscape of neural networks: An overview. IEFE Signal Processing Magazine, 37(5)95- 2020.
[49] Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization over-parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 59:85-116. 2022.
[50] Mahdi Soltanolkotabi, Adel Javanmard, and Jason Lee. Theoretical insights into the optimization landscape of over-parameterized shallow neural networks. IEEE Transactions on Information Theory, 65(2):742-769, 2018.
[51] Tomer Galanti. András György, and Marcus Hutter. On the role of neural collapse in transfer learning. In International Conference on Learning Representations. 2022.
[52] Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint K1l::2202.0888. 2022.
[53] Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research, 21(25)):1664 2020.
[54] Ido Ben-Shaul and Shai Dekel. Nearest class-center simplification through intermediate layers. arXiv preprint arXiv:2201. 089244, 2022.
[55] Liang Xie, Yiho Yang, Deng Cai, Dacheng Tao, and Xiaofei Hc. Neural collapse inspired attraction- repulsion-balanced loss for imbalanced learning. arXiv preprint arXiv:2204.0873 2022.
[56] Yibo Yang. Liang Xie, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dacheng Tao. Do we really need a learnable classifier at the end of deep neural network? arXiv preprini 2xXi:22003.0908/ 2022.
[57] Nicolas Boumal. An introduction to optimization smooth manifolds. To appear with Cambridge University Press, Jan 2022.
[58] C Jin, MI Jordan, R Gc, P Netrapalli, and SM Kakade. How to escape saddle points efficiently. In International Conference on Machine Learning, pages 2727-2752, 2017.
[59] Simon S Du, Chi Jin, Jason D Lee, Michael Jordan, Aarti Singh, and Barnabas Poczos. Gradient descent can take exponential time to escape saddle points. In Advances in neural information processing systems, pages 1067-1077 2017.
[60] Jiang Hu, Xin Liu, Zai-Wen Wen, and Ya-Xiang Yuan. A brief introduction to manifold optimization. Journal of the Operations Research Society of China, 8(2):199-248. 2020.
[61] P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms matrix manifolds. Princeton University Press, 2009.
[62] Christopher Criscitiello and Nicolas Boumal. Efficiently escaping saddle points on manifolds. Advances in Neural Information Processing Systems, 32, 2019.
[63] Yue Sun, Nicolas Flammarion, and Maryam Fazel. Escaping from saddle points on riemannian manifolds. Advances in Neural Information Processing Systems, 32, 2019.
[64] Qing Qu, Ju Sun, and John Wright. Finding a sparse vector in subspace: Linear sparsity using alternating directions. In Advances in Neural Information Processing Systems, pages 3401-3409, 2014.
12

---

[65] Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere I: Overview and the geometric picture. IEEE Transactions on Information Theory, 63(2):853-884, 2016.
[66] Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery the sphere II: Recovery by Riemannian trust-region method. IEEE Transactions on Information Theory, 63(2):853-884, 2017.
[67] Qing Qu, Yuexiang Zhai, Xiao Li, Yuqian Zhang, and Zhihui Zhu. Geometric analysis of nonconvex opti-
mization landscapes overeomplete learning. International Conference on Learning Representations. 2020.
[68] Qing Qu, Zhihui Zhu, Xiao Li, Manolis C. Tsakiris, John Wright, and René Vidal. Finding the sparsest vectors in a subspace: Theory, algorithms, and applications. arXiv preprint arXiv:2001 066700 2020.
[69] Peng Wang, Huikang I. Liu, Anthony Man-Cho So, and Laura Balzano. Convergence and recovery guarantees of the k-subspaces method for subspace clustering. In International Conference on Machine Learning, pages 22884-2291 18. PMLR, 2022.
[70] Man-Wen Kuo, Yenson Lau, Yuqian Zhang, and John Wright, Geometry and symmetry in short-and-sparse dcconvolution. In International Conference Machine Learning, pages 3570-3580. PMLR, 2019.
[71] Qing Qu, Xiao Li, and Zhihui Zhu. Exact recovery of multichannel sparse blind deconvolution via gradient descent. SIAM Journal on Imaging Sciences, 13(3):1630-100 1652, 2020.
[72] Yenson Lau, Qing Qu, Han-Wes Kuo, Pengeheng Zhou, Yuqian Zhang, and John Wright. Short and sparse deconvolution geometric approach. In International Conference on Learning Reprexentations, 2020.
[73] Yuqian Zhang, Han- Wen Kuo, and John Wright. Structured local optima in sparse blind deconvolution. IEEE Transactions on Information Theory, 66(1):419-452, 2019.
[74] Yuqian Zhang, Qing Qu, and John Wright. From symmetry to geometry: Tractable nonconvex problems. arXiv preprint arXiv:2007.0675. 2020.
[75] Vignesh Kothapalli, Ebrahim Rasromani, and Vasudev Awatramani. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.04000 040411 2022.
[76] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 161 1.03530, 2016.
[77] Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transfcrable features? In NeurlPS, 2021.
[78] Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Yi Ma. Redunet: A white-box deep network from the principle of maximizing rate reduction. ArXiv, abs/2105. 10446, 2021.
[79] Akshay Rangamani, Mengjia Xu, Andrzej Banburski, Qianli Liao, and Tomaso Poggio. Dynamics and neural collapse in deep classifiers trained with the square loss. Technical report, Center for Brains, Minds and Machines (CBMM), 2021.
[80] Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Neural collapse in deep homogeneous classifiers and the role of weight decay. In International Conference Machine Learning, pages 3821 3830. PMLR, 2021.
[81] Tomaso Poggio and Qianli Liao. Implicit dynamic regularization in deep networks. Technical report, Center for Brains, Minds and Machines (CBMM), 2020.
[82] Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv prepeint arXiv:2101.00m0 000072, 2020.
[83] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, 2017.
[84] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna, Rethinking the inception architecture for computer vision, 2015.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We state in Section 2 that our analysis uses the unconstrained feature model, and Section we discuss the requirement of d > K and upper bound on for Theorem 2.
13

---

(c) Did you discuss any potential negative societal impacts of your work? [N/A] This paper mainly focuses on understanding the neural collapse phenomena observed in practical neural networks. We believe there is no potential negative societal impact in this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We explicitly mention the assumptions in Section 3.
(b) Did you include complete proofs of all theoretical results? [Yes] We include all the proofs in the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]] The CIFAR datasets used here are cited and easily accessible. The GitHub code repository will be made available after review.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.
(c) Did you report error bars (c.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] Figure does not show error bars since the attained value is always the same independent of initialization due to the benign global landscape. In Section 4, we used shaded area as error bars for Figure 5, but regardless we have observed that the results similar and reproducible independent of the random seed. (d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cite them in Section 4.
(b) Did you mention the license of the assets? [Yes] This is mentioned in Section 4.
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] This mentioned in Section 4.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "zZb2rQ_Kueb", "reviewer": "Reviewer_VCm4", "summary": "The paper provides theoretical justification of the neural collapse phenomenon for normalized features. The proof is based on the unconstrained feature model (UFM) and the resulting optimization problem is analyzed over the Riemannian manifold by constraining features and classifiers prototypes over the sphere. The paper basically shows that critical points for the optimization (i.e. potential minima) are saddles (i.e. negative curvature critical points) while all others are global minimizers for which the neural collapse properties hold. The paper concludes with an empirical evaluation confirming the proposed theory.\n", "questions": "This form is intentionally left blank as questions are included in the previous form.\n", "limitations": "No negative societal impacts. Limitations are included into the Strengths And Weaknesses form.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n\n1) Interesting formulation of neural collapse in the case in which features are normalized. Feature normalization is a quite classic strategy/practice that has been used in several deep learning contexts (i. e. face recognition) in which maximal separation between learned features is required and therefore it naturally matches with the concept of maximal separation which is inherent to the neural collapse phenomenon. The provided theoretical analysis is very useful. The fact that a standard practice has a convergent geometric discriminative structure is a clear strength of this paper. \n\n2) The oblique manifold formulation is interesting and it appears to be well suited to the problem.\n\nWEAKNESSES:\n\n1) The paper is well written, but many parts seem to require further work. The main result (the proof according to the oblique manifold formulation) of the paper is neither introduced nor motivated and no intuitions are given. Although very interesting, it is hard for the reviewer and, possibly for the reader, to fully grasp the real contribution of the paper. The paper refers to a book (specifically to some exercises) from which the reader should possibly get some intuition and motivations. This part should be “softened”.\n\n2) The related work section in its current shape is somewhat short and without specific comments. It should be improved by following a thread with respect to what is proposed. The reviewer agrees that feature normalization can improve separability, however the four cited works (i.e. [22,23,24,7]) are not discussed in this regard. In which way those four works achieve separability? What is the quality of the representation? Does it depend on the task addressed? Is there any trade-off in the achievement of separability? Important related works are also missing. Many papers, especially from the feature learning and visual search literature are completely missing [A,B,C,D,E,F]. In particular it seems that [A] is one the first works which studied classifier prototypes and their relationship with the learned features. The works [B,C] and [D] firstly introduced feature normalization and their feature separability (i.e., discriminative features) behavior. Finally, [E] and [F] seem to be the first papers to apply maximal separability in a simplex shaped structure (and other regular polytopes). As recent neural collapse literature used the term simplex ETF, what is the difference between the  simplex regular polytopes and a simple ETF? Are they equivalent?\n\n3) The introduction should be better focused on the related work. On line 37 a large number of works are cited in a single shot as related to NC. However no comments are explicitly given. Given the diversity of the cited papers a discussion should be given.\n\n4) What data is used in Fig.1 and Tab.1? It seems that it is not real data. How is the data generated?\n\n5) The most important part related to the proof, should be improved. Some motivations and intuitions about the oblique manifold should be given. The cited book is addressing a dictionary learning problem. Although the affinity is clear, the paper in its current shape does not provide an intuitive access to the oblique manifold formulation. A potential reader of this paper would be forced to read too many references. The reviewer's suggestion is to increase the level of self-containment of the paper.\n\nReferences\n\n[A] Liu, W., Wen, Y., Yu, Z., & Yang, M. (2016). Large-margin softmax loss for convolutional neural networks. arXiv preprint arXiv:1612.02295.\n\n[B] Ranjan, R., Castillo, C. D., & Chellappa, R. (2017). L2-constrained softmax loss for discriminative face verification. arXiv preprint arXiv:1703.09507.\n\n[C] Wang, F., Xiang, X., Cheng, J., & Yuille, A. L. (2017, October). Normface: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international conference on Multimedia (pp. 1041-1049).\n \n[D] Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., & Song, L. (2017). Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 212-220).\n\n[E] Pernici, F., Bruni, M., Baecchi, C., & Del Bimbo, A. (2019, January). Maximally Compact and Separated Features with Regular Polytope Networks. In CVPR Workshops (pp. 46-53).\n\n[F] Pernici, F., Bruni, M., Baecchi, C., & Del Bimbo, A. (2021). Regular polytope networks. IEEE Transactions on Neural Networks and Learning Systems.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DEzdu32vqYc", "reviewer": "Reviewer_V69c", "summary": "Feature normalization has been a common practice. In order to justify neural collapse for normalized features, this paper formulate an unconstrained feature model (UFM) with spherical constraints. Theoretical results on the neural collapse global optimality and the benign landscape are performed. Experiments are conducted to show that UFM with feature normalization has better training and collapse performances than feature regularization.", "questions": "For practical application, one is more interested in generalization ability. Regularization has been known as an effective scheme to improve generalization. It is not clear about this for feature normalization. So, I think the authors should compare validation/test accuracy in Figure 4 and 5. ", "limitations": "There are discussions about the limitations of this work. \n\nNo concern about negative social impact. \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-\tThe motivation to interpret feature normalization from the neural collapse perspective is interesting. \n\n-\tThe paper is well organized and presented. \n\n-\tTheoretical work is sound. \n\nWeaknesses:\n\n-\tLimited contribution. The first theoretical result, the neural collapse global optimality of Eq. (3), is a minor extension of Theorem 1 in [1], which also has a spherical constraint. Besides, as stated by the author, the problem with spherical constraint in Eq. (3) has the same global solution as the one with ball constraint in [2,3]. But the feasible region of ball constraint in [2,3] is much larger. So, the result in this paper is less significant. \n\n-\tIn order to verify the theoretical results (Theorem 1 and 2) in this paper, the authors show that UFM with feature normalization has faster training and better collapse than feature regularization. But, the same theoretical results, including the same global optimality of neural collapse conditions and the same benign landscape properties, are also applicable to UFM with regularization [4], and UFM without either constraint or regularization [5]. In this case, how to explain the advantage of normalization over regularization in the experiments?\n\n[1] Lu et al., Neural collapse with cross-entropy loss,\n\n[2] Graf et al., Dissecting supervised contrastive learning, ICML 2021,\n\n[3] Fang et al., Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training, PNAS,\n\n[4] Zhu et al., A geometric analysis of neural collapse with unconstrained features, NeurIPS 2021,\n\n[5] Ji et al., An unconstrained layer-peeled perspective on neural collapse, \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5-zdAW4xFcV", "reviewer": "Reviewer_vkF5", "summary": "The paper considers the landscape of the optimization problem when training neural networks with normalized features under the cross-entropy loss. Under the assumption of the unconstrained feature model, the authors reformulated the problem as a Riemannian optimization problem, and showed that (1) global solutions of the problem satisfy the neural collapse properties and (2) the optimization problem has a benign global landscape (i.e. local minimizers are global minimizers, and all other critical points are strict saddle points).", "questions": "This is probably a minor question regarding the presentation of Figure 2.\n\nIn the end of section 3, the \"Limitations on the assumptions of the feature dimension d\" part, the authors stated that \"We believe the bound can be improved (from d>N) to d>K+1, .... corroborated by our experimental results (See Figure 2)\". \n - I am expecting Figure 2 to show an experiment with K+1<d<N, which would empirically support the author's claims that we do not necessarily need d>N.\n - But instead, it looks like the authors chose d=100, n=5 for that Figure, which still guarantees d>n, so it does not really corroborate the claim above. Am I missing something, or should the graph be modified?", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Quality and Clarity__: The paper is written clearly, and I enjoyed reading the motivations that builds up nicely to the main theorems. The experiments also support most of the author's claims well.\n\n__Originality and Significance__: I personally do not work in this field, so I will only provide some brief observations on significance of results, while leaving more discussions to other expert reviewers: The proof of theorem 1 on neural collapse properties is not too different from existing techniques cited by this paper. Theorem 2, on the other hand, is nontrivial, and understanding the global landscape is indeed useful for developing guarantees for algorithms like Riemannian SGD.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept