A Geometric Analysis of Neural Collapse with Unconstrained Features
Authors: Zhihui Zhu
Tianyu DING
Jinxin Zhou
Xiao Li
Chong You
Jeremias Sulam
Qing Qu
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: neural collapse
nonconvex optimization
low-dimensional model

Abstract: We provide the first global optimization landscape analysis of Neural Collapse -- an intriguing empirical phenomenon that arises in the last-layer classifiers and features of neural networks during the terminal phase of training. As recently reported by Papyan et al., this phenomenon implies that (i) the class means and the last-layer classifiers all collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and (ii) cross-example within-class variability of last-layer activations collapses to zero. We study the problem based on a simplified unconstrained feature model, which isolates the topmost layers from the classifier of the neural network. In this context, we show that the classical cross-entropy loss with weight decay has a benign global landscape, in the sense that the only global minimizers are the Simplex ETFs while all other critical points are strict saddles whose Hessian exhibit negative curvature directions. Our analysis of the simplified model not only explains what kind of features are learned in the last layer, but also shows why they can be efficiently optimized, matching the empirical observations in practical deep network architectures. These findings provide important practical implications. As an example, our experiments demonstrate that one may set the feature dimension equal to the number of classes and fix the last-layer classifier to be a Simplex ETF for network training, which reduces memory cost by over 20% on ResNet18 without sacrificing the generalization performance. The source code is available at https://github.com/tding1/Neural-Collapse.

A Geometric Analysis of Neural Collapse with Unconstrained Features
Zhihui Zhu" University of Denver zhihui. zhu@du
Tianyu Ding Johns Hopkins University tding1@jhu. edu
Jinxin Zhou University of Denver
j jinxin. zhou@du. edu
Xiao Li
Chong You
Jeremias Sulam
University of Michigan xlxiaoQumico ch. edu
Google Research cyouβgoogle.cm com
Johns Hopkins University jsulam1 @jhu. edu edu
Qing Qu
University of Michigan qingquêumi ch. edu
Abstract
We provide the first global optimization landscape analysis of Neural Collapse- an intriguing empirical phenomenon that arises in the last-layer classifiers and features of neural networks during the terminal phase of training. As recently ported in [1]. this phenomenon implies that (i) the class means and the last-layer classifiers all collapse to the vertices of Simplex Equiangular Tight Frame (ETF) to scaling, and (ii) cross-example within-class variability of last-layer activa- tions collapses zero. We study the problem based on simplified unconstrained feature model, which isolates the topmost layers from the classifier of the neural network. this context, we show that the classical cross-entropy loss with weight decay has benign global landscape, in the sense that the only global minimizers are the Simplex ETFs while all other critical points are strict saddles whose Hes- sian exhibit negative curvature directions. Our analysis of the simplified model not only explains what kind of features learned in the last layer, but also shows why they can be efficiently optimized, matching the empirical observations in practical deep network architectures. These findings provide important practical implications. As example, our experiments demonstrate that one may set the feature dimension equal to the number classes and fix the last-layer classifier to be Simplex ETF for network training, which reduces memory cost by over 20% on R8 without sacrificing the generalization performance. The source code is available https:/ thub. coom/tdingg///eeeeee http//Neeaalllollllla -Colllpo
Introduction
In the past decade, the revival of deep neural networks (DNN) has led dramatic success in nu- merous applications ranging from computer vision, to natural language processing, to scientific discovery and beyond [2-5]. Nevertheless, the practice of deep networks has been shrouded with mystery
theoretical understanding for the success of deep learning remains elusive. There many intriguing phenomena, such as implicit algorithmic bias in training [6-10], and good general- ization of highly-overparameteriza networks [1-15, that seem often contradictory to, or cannot be explained by, classical optimization and learning theory.
'The first two authors contributed to this work equally.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Ryer
Oupui
M
Φ
A12
A4
W.
Figure 1: Illustration of Neural Collapse. Here (o)-) denotes the feature mapping of the network, i.. the output of the penultimate layer; see (1) for the formal definition.
Figure 2: Illustration the unconstrained feature model, where the gray box is peeled off so that the representation h is modeled by simple decision variable for every training sample.
Towards demystifying DNN, recent seminal work [1. 16] empirically discovered an intriguing phe- nomenon that persists across a range of canonical classification problems during the terminal phase of training. As illustrated in Figure I, it has been widely observed that last-layer features and clas- sifiers of a trained DNN exhibit simple but elegant mathematical structures:
Variability Collapse: cross-example within-class variability of last-layer features collapses to zero, as the individual features of each class themselves concentrate to their isolated class means. Convergence to Simplex ETF: the class-means centered their global mean not only linearly separable, but are actually maximally distant and located on sphere centered at the origin up to scaling (i.e., they form a Simplex Equiangular Tight Frame (ETF) or Simplex ETF, which is formally defined in Definition C.1 in the Appendix).
Convergence Self-duality: the last-layer linear classifiers, living in the dual vector space that of the class-means, are perfectly matched with their class-means.
Simple Decision Rule: the last- last- classifier is behaviorally equivalent a Nearest Class-Center decision rule.
These results suggest that deep networks are learning maximally separable features between classes, and a max- marxin classifier in the last layer upon these learned features, touching the ceiling in terms of the performance. This phenomenon is referred as Neural Collapse (NC) [1], and persists across a range of canonical classification problems, on different neural network architectures (e.g., VGG [17], ResNet [18), and DenseNet [19]) and on a variety of standard datasets (such as MNIST [20], CIFAR [21], and ImageNet [22]).
Fully demystifying the NC phenomenon in theory can be very challenging. Perhaps the most diffi- cult hurdle lies in the nonconvexity of the optimization problem for training neural networks, which, loosely speaking, stems from the nonlinear interaction across many different layers of neural net- works. Towards this goal, a recent line of work [23-29] studied the properties of last-layer clas- sifiers and features based on the assumption of the so-called unconstrained feature model [23] or layer-peeled model [26]. At a high level, the unconstrained feature model takes a top-down ap- proach the analysis of deep neural networks 223-26,3 29-31], wherein the last-layer features are modeled as free optimization variables (hence we call them unconstruined features) along with the last-layer classifiers (see Figure 2 for an illustration); this in contrast to the conventional bottom- up approach that studies the problem starting from the input [32-42].2 The underlying reasoning that modern deep networks are often highly overparameterized with the capacity of learning any representations [43-46], so that the last-layer features can approximate, or interpolate, any point in the feature space. In this way, the model simplifies the study of last-layer features, enabling us to analyze the interaction between them and the last-layer classifiers.
Nonetheless, the simplified unconstrained feature model still leaves us a highly nonconvex training loss to be dealt with. Despite the nonconvexity, recent work [23-28] studied the global minimizers, proving that Simplex ETFs (CC) are indeed global solutions the nonconvex loss. In particular, the work [23, 47] studied the training problem with the least-squared loss, proving that the gradient flow converges to NCC solutions with extra assumptions. Another line of work [24-27] considered the commonly used cross-entropy loss for classification, showing that the only global minimizers of the loss function are Simplex ETFs with different constraints on the weights and features." 3 However,
Here, top-down means that we study the network starting from the last- last-layer down to the input layer, whereas bottom-up refers to an approach from the input up the last layer.
'The constraints on the features are mainly used to prevent it from approaching infinity since the objective function, with the cross- cossopy loss, is not coercive. Note that we still refer to this model as an unconstrained feature model even if they include norm constraints or regularization.

---

Table 1: Comparison of the setup and results under the unconstrained feature model with cross-entropy loss.
Regularizer
Weight decay Bias term Global minimizer
Results
Constraint
Landscape
[24-27] This paper
these results still suffer from several limitations: (i) due to the nonconvex nature, only characterizing optimality conditions is not enough to explain the empirical convergence of iterative algorithms to NC, such as stochastic gradient descent (SGD); (ii) the problem formulations differ from those typically used in practice, which deploy norm regularization (i.e., weight decay) on the weights, rather than enforcing constraints, for the ease of optimization."
Contributions of This Work. Inspired by these pioneering results [1,23-26, 29], in this work we take step further by characterizing the global optimization landscape of the network training loss based on the unconstrained feature model. Our contributions are summarized as follows.
Benign Global Landscape. For the unconstrained feature model, we provide the first result show- ing that a commonly used, regularized cross-entropy loss is a strict saddle function [49-51]. In other words, every critical point is either global solution (corresponding a strict saddle point' with negative curvature, so that there is no spurious local minimizer on the optimization landscape. As summarized in Table 1, this in contrast to previous work [23-26] that only characterizes global minimizers.
Simplex ETFs) or
Efficient, Algorithmic Independent, Global Optimization. The benign global landscape implies that any method that can escape strict saddle points (e.g. stochastic gradient descent) converges to global solution [52] that exhibits NC. This result supports our empirical observation, as shown in Section 4. that practical overparameterized networks always converge to Simplex ETF solutions with a diverse choice of optimization algorithms.
Cost Reduction for Practical Network Training. Moreover, the universality of A'C implies that there is no need of training the last-layer classifiers since the weights can be simply fixed as a Simplex ETF throughout the training process. On the other hand, since NC happens whenever d 2 K, this implies that we can choose the feature dimension d comparable to the number of classes K, reducing the feature dimension for further computational benefits. In Section 4.3, our experiments demonstrate that such a strategy achieves on par performance with classical training methods, leading to substantial cost reductions on both memory and computation.
Our results shed new light on the question raised in the recent paper [53] on the role of the optimiza- tion strategy (e.g., stochastic gradient descent) for achieving NCC in training practical deep networks. This question also relates to the recent highly influential work [7] on the implicit algorithmic bias. For multi-class classification problems with linearly separable data, this work [7] showed that lin- ear predictors optimized by gradient descent converge to the max-margin classifiers even without adding any explicit regularization on the cross-entropy loss. Based on this result, a sequence of works [54-61] laid great emphasis on the notion of "inductive bias" of particular optimization al- gorithms as a reason for the surprising success in training deep learning models. 6 contrast, both theoretical result the global landscape for the unconstrained feature model and the empirical evidence on practical deep models demonstrate that ACC in network training is facilitated not only by the choice of the optimization methods, but more importantly, by the choice of loss functions and the power of overparameterization in the network architecture.
2 The Problem Setup
A deep neural network is essentially nonlinear mapping (6()) RR r R2K which can be modeled by a composition of simple maps: v(x) = v- o ... o 2 (x)) for x E RD,, where (-) L) called "layers". Each layer is composed of an affine transform, represented by
(1 < l
'A recent concurrent work [48] studied the gradient flow as well landscape of the cross-entropy loss under the unconstrained feature model, but without using any constraints or regularizers the weights. 'Throughout the paper, for minimization problem, we will not distinguish between local maxima and sad- dle points. We call critical point strict saddle if the Hessian at this point has at least one negative eigenvalue. 'While (stochastic) gradient descent and generic steepest descent methods can converge to max margin classifiers, it should be noted that other commonly used optimization algorithms in deep learning, such as AdaGrad [62] and Adam [63]. do not have max margin properties in general and their solutions depend upon initialization, step-size and other algorithm parametrrsss (64-66].
3

---

some weight matrix We, and bias be. followed by a simple nonlinear? activation function o(-).
More precisely, a vanilla L-layer neural network can be written as
(x) = WL a (WL-1 0 (W1 + b1) P +bL.
(1)
do (z)
For convenience, we use Q (W&: k=1 to denote all the network parameters, and use 0 {Wx, by bx]k=] to denote the network parameters up to the last layer. The output of the penultimate layer, denoted by Φ0(x), is usually referred to the representation or feature of the input æ learned from the network. In this way, the function implemented by a neural network classifier can also be expressed as a linear classifier acting upon (x
The goal of deep learning is fit the parameters @ so that the output of the model on an input samples x approximates the corresponding output y, i.e. so that the (œ) R y, in expectation over a distribution of input-output pairs, D. This can be achieved by optimizing an appropriate loss function L(ve (ar), y) which quantífies this approximation. In this work, we focus on multi-class classification tasks (say, with K classes), where the class label of a sample x' is given by a one-hot vector y E _ representing its membership to one of the K classes, In this setting, cross-entropy is one of the most popular choices for the loss function. Naturally, the distribution D is unknown, but we have access to training samples that are drawn i.i.d. from D. In this way, one can minimize the empirical risk over these samples by optimizing the following problem
K
A
min θ
Lee (vD yk)
+
A F 2
(2)
A= 1=1
where y E RK is hot vector with only the kthh entry equal to unity (1 S k K), {nx are the numbers of training samples in each class, and À > is the regularization parameter (or weight decay penalty), and (CE(:.) is the cross-entropy loss. As introduced in Section I, recent work [1] showed that the features learned by minimizing the above objective showcase the NCC phenomenon: their within-class variability vanishes, and the features converge to Simplex ETF.
2.1 Problem Formulation Based on Unconstrained Feature Models
In deep network models, the nonlinearity and interaction between a large number of layers results in tremendous challenges for analyzing this learning problem. Since modern networks are often highly overparameterized to approximate any continuous function and the characterization of NC only involves the last-layer features Φ0(x), a natural idea to simplify the analysis to treat these features as free optimization variables h = (x Rd, which motivates the name unconstrained feature model" [23] (see Figure 2 for an illustration). In this way, we can rewrite the network output as vhe (z) Wrh + bt.
For simplicity, we consider the setting where the number of training samples in each class is balanced (i.e.. n4 = n for all E [K] := 2........ K}). ......,,)..............>... We also write W = WL and b = by for conciseness. Based on the unconstrained feature model, we consider a slight variant of (2), given by
min w H
=
1 Kn i=1
LOB CCE(WWWW) (Whss b, + Aw 2
+
dy
2
P
+
d
2
bb (3)
[11.1..... hx,a] RdxN (here, we denote N = nK), b E R and Xw. AH A > 0 are the penalty parameters for the weight decay.
with W E RKxd
As summarized in Table 1, similar optimization problems have been considered in [24-26]. In contrast to these, our problem formulation here (3), with bias and weight decay, is closer to the loss used in practice for training neural networks; existing work [24-26] considered constrained variants of (3) and without the bias term, which can be implemented but seldom used in practice due to the difficulty of optimization. In the following, we briefly discuss the differences between our simplification and practical settings for training neural networks.
7 The nonlinear operator may include activations such as ReLU [67], pooling, and normalization [68]. etc. 'This model is also called layer-peeled model in [26], where "peels" off the first L 1 layers. It has also been recently studied in [24.25]. Throughout the paper, we will simply call unconstrained feature model. For example, the work [26] considers inequality constraints such that the energy of W and H are bounded; the other work [24,25] enforces W and H on the spheres up scaling.

---

Weight Decay on W and H. One simplification of our formulation is in the weight decay. In practice, weight decay is usually imposed on the network parameters O while we enforce weight decay on the last layer's classifier, W. and features, H. However, this idealization is reasonable since the energy of the features (i.e., || H ()) can indeed be upper bounded by the energy of the weights at every layer if the inputs are bounded (which holds in practice), implying that the norm of H is implicitly penalized by penalizing O Our experiments the Appendix demonstrate that both approaches exhibit similar NCC phenomena and comparable performance in practice.
Treating the Last-layer Features as Optimization Variables. One may question that "peeling off" the L 1 layers might oversimplify the problem. Nonetheless, this simplification (which is also adopted in [23-26]) is based on the fact that neural networks with sufficient overparameteriza- tion can approximate any function in Section 4.2, numerically demonstrate that .CC persists even when we train overparametrized networks on randomly generated labels. Moreover, as we shall in the following sections, both our theory and experiments demonstrate that sim- plification preserves the core properties of last-layer classifiers and features during training - the NC phenomenon. More specifically, in Section 3 we show that Simplex ETFF the only global minimizers to our simplified loss function (3), and the loss function is strict saddle function with no other spurious local minimizers so that it can be optimized efficiently global optimality.
3 Main Theoretical Results
In this section, we present our study on global optimality conditions as well as the optimization landscape of the nonconvex loss in (3).
Theorem 3.1 (Global Optimality Conditions) Assume that the feature dimension d is smaller than the number of classes K, i.e. 2 K 1, and the number of training samples in each class is balanced, n2
RK. Then any global minimizer (W* _H", bb) of in (3) satisfies
w² : 1 112 = | = = l
l1e and b' = b'1,
h V
dw
1 K
(4)
w
WkE [K], ie [n], and
h
hij 0, Vie [n],
AHA
where either b" 0 or d 0, and the matrix W*T B forms K K-Simplex ETF (defined in Definition C.I) up to some scaling, in the sense that the normalized matrix M == W satisfies
M
K K-1
IK 1 1
(5)
At high level, our proof (in Appendix D) finds lower bounds for the loss in (3) and studies the conditions for the lower bounds to be achieved, similar to 26]. As can be seen in this result, any global solution of the loss function (3) exhibits ACC in the sense that the variability of output features (h of each class l (1 < 1s < K) collapses to zero, and any pair of features (ht, .: h from different classes 1 * 12 are maximally separated. Similar results have been obtained in [24-26], which considered different problem formulations,
have discussed in Section 2.1.
Relationship between Class Number K and Feature Dimension d. The requirement that d > K - 1 is necessary for Theorem 3.1 to hold, simply because vectors in IR cannot form a K- Simplex ETF if K > d + 1. However, the relationship d K is often true in practice. In general, and in overparameterized models in particular, the feature dimension, d, is significantly larger than the number of classes, K. For example, the dimension of the features of ResNet [18] is typically set to d' = 512 for CIFARIO [21], a dataset with K = 10 classes. This dimension grows d 2048 for ImageNet [22], dataset with K 1000 classes.
Interpretations the Bias Term b' In contrast to previous works [24-26), consider the bias term in the unconstrained feature model (3). Our result indicates that a collapsing phenomenon also exists in the bias term b* in the sense that all the elements of b* are identical. When the features H are completely unconstrained, our result implies that removing the bias term b has no influence on the performance of the classifier. However, it should be noted that the ReLU unit is often applied at the end of the penultimate layer, so that H should be constrained to be nonnegative, H > 0. In such cases, h will no longer be zero, and neither will by. Here, the bias
5

---

term 6" will compensate for the global mean of the features, so that the globally-centera features still form a Simplex ETF [1]. 10
3.1
Characterizations of the Benign Global Landscape for (3)
The global optimality condition in Theorem 3.1 does not necessarily mean that we can achieve these global solutions efficiently, as the problem is still nonconvex. We now investigate the global opti- mization landscape of (3) by characterizing all of its critical points. Our next result implies that the training loss is strict saddle function, and every critical point is either global minimizer strict saddle point that can be escaped using negative curvatures. As a consequence, this implies that the global solutions of the training problem in (3) can be efficiently found from random initializations.
Theorem 3.2 (No Spurious Local Minima and Striet Saddle Property) Assume that the feature dimension is larger than the number of classes, d >> K, and the number of training samples in each ng. Then the function f(W, H. H.b) in (3) strict saddle function with no spurious local minimum, in the sense that
class is balanced n ny
Any local minimizer of (3) is a global minimizer of the form shown in Theorem 3.1.
Any critical point (W, H. b) of (3) that is not a local minimizer is strict saddle with negative curvature, i.e. the Hessian 2 f(W. H, b). at this critical point, is non-degenerate and has at least one negative eigenvalue, i.e. 3d: > ( H. < O.
In a nutshell, our proof relies on connecting the original nonconvex optimization problem (3) to its corresponding low-rank convex counterpart, so that we can obtain the global optimality conditions for (3) based on the latter. With this, we can then characterize the properties of all critical points based on the optimality conditions. We defer all details of this proof to Appendix D.
Existing results [24-26] have only studied the global minimizers of the original problem, which has limited implication for optimization. In contrast, Theorem 3.2 characterizes the properties for all critical points of the function in (3). As a consequence of this result, many first-order and second- order optimization methods [69] optimizing (W, H,, b) are guaranteed to converge to global solu- tion of (3). In particular, the result in [49, [2] ensures that (stochastic) gradient descent with random initialization, the de facto optimization algorithm used in deep learning, almost surely escapes strict saddles and converges to a second-ore order critical point which happens to be a global minimizer of form showed in Theorem 3.1 for our problem (3).
Constructing the Negative Curvature Direction for Strict Saddles. One of the major difficul- ties in our proof is to construct the negative curvature direction for strict saddle points. Here, we exploit the fact that the feature dimension d is larger than the number of classes K, and construct the negative curvature direction within the null space of W E g xd This also the main reason for the requirement d > K in Theorem 3.2, but we conjecture the results also hold for d = K and could be proved with more sophisticated analysis, which is left as future work.
Relationship to Low-Rank Matrix Recovery. As discussed in Appendix A, it has been recently shown that the strict saddle property holds for a wide range of nonconvex problems in machine learning [70-83 833, including low-rank matrix recovery [78, 80, 84- As we know that 2
|| 222)) (see [32] for a proof), our formulation in (3) is closely related to low-rank matrix problems [78, 80, 84-87] with the Burer-Moneirt, factorization approach [88], by viewing W and H as two factors of matrix Z W H. The differences lie in the loss functions and statistical properties of the problem." 1I Thus, our result establishes a connection between the study of low-rank matrix factorization and neural networks under the unconstrained feature model.
ming= и [(] [|
1|ee +
Suppose that an optimal solution to (3) is (W* HHH (b)), satisfying the conditions in Theorem 3.1. There exists nonzero vector a E Id such that H' = HH + x1 2 0. Here, a can be viewed as the global mean of H since H" has mean zero Then, let 6* = - at so that W'H + b.1 = W' + (W' a b²)1 = W Therefore, we can see that (W H' bb) achieves the same cross-entropy loss as (W', H .b).
We consider the cross-entropy loss rather than the least-squares loss due the differences in the task we focus on classification instead recovery problems. On the other hand, the results on low-rank matrix recovery are often based on certain statistical properties, such as the randomness in the measurements [84, 83J. or restricted well-coo coll--ooiiiondnesss property of the objective function [77, 87]. In contrast, these statistical properties do not exist in our problem, where the model and analysis are purely deterministic.

---

Comparison to Existing Landscape Analysis on Neural Network. Section provided a com- prehensive discussion on the relationship between our result and previous works on landscape analysis for deep neural networks. Although the unconstrained feature model can be viewed as two-layer linear network with input being the columns of an identity matrix, as preluded Sec- tion 1, our result has much broader implications than the previous results 3,,,,,,,,,,,444 48,,00,41,800 41,91. First, our problem formulation (3) is closer to practical settings for classification tasks, which con- siders the widely adopted cross-entropy loss while including weight decay and a bias term, while most existing results [33,34.37, 40,4 either do not incorporate any regularization and bias, focus on the squared loss for the regression problem. More importantly, our result characterizes the precise form of the global solutions (i.e., NC) for the last layer features and classifiers, and shows that they can be efficiently achieved. Moreover, convincing numerical results in [1] and the next section demonstrate that the global solutions do appear and can be achieved by practical networks on various standard image datasets. Our study of last-layer features could have profound implications for studying generalization and robustness of the deep networks.
4 Experiments
In this section, we run extensive experiments not only verifying our theoretical results on mod- em neural networks, but also demonstrating the potential practical benefits of understanding NC. More specifically, while Theorem 3.2 holds true for the simplified unconstrained feature model, in Section 4.1 we run experiments on practical network architectures and show that our analysis of simplified models captures the gist of NC. In particular, we demonstrate that this depends on the geometry of the problem rather than the algorithmic bias, by showing that different types of opti- mization algorithms all achieve NC during the terminal phase of training. In Section 4.2, we verify the validity of the simplification based on the unconstrained feature model. Moreover, the univer- sality of NC implies that there is no need for training the last-layer classifiers since the weights can be simply fixed as a Simplex ETF throughout the training process. Section 4.3, we demonstrate that such a strategy achieves essentially the same generalization performance as classical training algorithms, while improving on memory and computation. We begin by describing the basic ex- perimental setup, including the network architectures, evaluation datasets, training procedures, and metries for measuring NC.
Setup of Network Architectures, Dataset, and Training. In Section 4. and Section 4.2, we train a ResNet18 architecture [18] on CIFARIO [21] for image classification using the cross-entropy loss (2). Due to limited space, we present all the results on MNIST [90] in the Appendix. As is standard, images are normalized schannel-wise by their mean and standard deviation. We include no data augmentation in this section, as our focus is study the behavior associated with instead of obtaining -fre--ff-theattt performance. We train the network for 200 epochs with three distinct optimizers: two first-order methods (SGD and Adam) and one second-order method (LBFGS [69]). In particular, we use SGD with momentum 0.9, Adam with 3, = 0.9, B2 = 0.9 0.999, and LBFGS with memory size of 10. The initial learning rates for SGD and Adam are set to 0.05 and 0.001, respectively, and decreased by a factor of 10 for every 40 epochs. For LBFGS, we use an initial learning rate of 0.1 and employ a strong Wolfe line-search strategy for subsequent iterations. Except otherwise specified, the weight decay is set to 5 x 10-4 for all the experiments.
Metries for Measuring NC During Network Training. We measure NC for the learned last- layer classifiers and features based on the properties presented in Section 1. Some of the metrics are similar those presented in [1]. We first measure the within-class variability collapse by measuring the magnitude of the netween-class covariance Σg E dddd compared to the within-clase covariance Ww E d of the learned features via AC2 := 1
K trace(Ew Σ/D, where Σ/ B denotes the pseudo Simplex ETF [WW IF -K-I (Ix - } 1 || Fs where we rescale the ETF in (5) so that -Kー [IK 1 1x1x1 k has unit energy (in Frobenius norm). should be noted that our metric NC2 combines two metrics used in [1] to quantify to what extent the classifier approaches equiangularity and maximal-angle equiangularity. We then measure the duality between the clas- sifiers W and the centered class-means means H by NC3- := WH
inverse of Σg. For the learned classifier W e R xd we quantify its closeness
up to scaling by NC2 :- 11 WW
K
-K-T (IK-
1 1 1x11))]
WH]
7

---

LBFGS
466
0.6
SGD
SGD
SGD
Adam LBPGS
9.2
Adlami LBPGS
Adam LBRSS
100
150
150
0.0
50
150
280
103 Epoch
150
Epoch
Epoch
Epech
Figure 3: Illustration of NCC across different training algorithms with ResNet18 CIFARIO. From the left to the right, the plots show the four metrics, 22, and WCC, respectively,
1.
wisth
witth with
wisth
40h (log scale)
witth
with
with
with
Trai Error
5.5
0.4
0.2
230
4033 Epoch
610
600
202
503
800
200
21.
6.00
850
200
(03
605
500
Epoch
Epoch
Epoch
Figure 4: Training results of ResNet18 with various feature width on CIFART R10 with completely random label. From the left to the right: NC,, NCs, and the misclassification pereentage of training samples.
In many cases, the global mean hg of the features might not be zero, 12 and the bias term b would compensate for the global mean hg. Thus, we capture this collapsing phenomenon by measuring NC4 = ||b Whall2 The detailed descriptions of the four metries are given Appendix B.
4.1 The Prevalence of NC Across Different Optimization Algorithms
We show different types of training methods (e.g., SGID, Adam, and LBFGS) all achieve NCC during the terminal phase of training. Figure shows the evolution of the four metrics NV,,,NC,,,NC, and NC4, for measuring NC as training progresses. We consistently observe that all four metrics collapse to zero, trained by different types of algorithms. This implies that NCC occurs regardless of the choice of training methods. The last-layer features learned by the network are always maximally linearly separable, and correspondingly the last-layer classifier is a perfect linear classifier for the features. See Appendix for the testing performance the networks learned by different algorithms.
4.2 The Validity of (3) Based on Unconstrained Feature Models for NCC
The premise of our global landscape analysis of (3) for studying NC in deep neural networks is based upon the unconstrained feature model introduced in Section 2.1, which simplifies the net- work by synthesizing the first L - layers as universal approximator that generates a simple decision variable for each training sample. Here, we demonstrate through experiments that such simplification is reasonable for overparameterized networks, in the sense that they are sufficient for characterizing NC practical network training. In particular, we demonstrate that overparam- eterization is crucial for NC phenomenon during network training, while the input plays minimal influence. To that goal, we modify the training dataset CIFARIO by replacing all the correct label for each training sample with random counterpart. We report the corresponding NCC behaviors in Figure 4, which shows how training misclassification rate and NCC evolve over epochs of training for networks with different widths As the network is sufficiently large, it has enough capacity to memorize the training data and achieves zero training error, which is consistent with the observa- find from Figure 4 that the training accuracy is highly correlated with
tions in [11]. Moreover, NC in the sense that a larger network (i.e., larger width) tends to exhibit severe NCC and achieves smaller training error. In other words, while the emerging consensus is that the network can inter- polate any training data, our results show that such interpolation happens in a particular way the features maximally separated, followed by max-margin linear classifier. In Appendix B, also report experiments on weight decay imposed on the features H, as in (3).
12 For example, as discussed after Theorem 3.1 all the feature vectors in H would be nonnegative, because the nonnegative nonlinear operator ReLU has been applied at the end of the penultimate layer. We also conducted experiments on completely random dataset where each training image is generated with pixels uniformly from [0, 1] and we observed similar results.
14 Here, for ResNet18 adopt the method in [15] change its network width.
8

---

12
dassifer, 2-2048
1.2
130
Seed desalier d-205a
1.0
Training accuracy
Tosting sccuracy
isur nod dasilfer d=12
dasilter
60
fxed dasalier d-10
dasill
0.2
0.4
12
leamce cussifirr de10
leamat Caositer 4-2051
cussillrr de200z
leamed riusotirr 202048
clasifier, cm2246
class de10
0
300
220
200
0.0g
30
100
153
200
200
250
202
20
50
103
150
250
Epoch
Epoch
Fpoch
Epach
Figure 5: Comparison of the performances of ResNet 50 with learned vs. fixed last-layer classifiers on CIFARIO. From left to right): NC1.NCs, Training Accuracy, Testing Accuracy.
4.3 Insights from NC for Improving Network Designs
Finally, we conduct exploratory experiments to demonstrate the practical benefits of NC phe- nomenon. The universality of NC implies that the final classifier (i.e. the L-th layer) of neural network always converges to a Simplex ETF, which is fully determined up to an arbitrary rotation and happens when K S d. Thus, based on these understandings, we show that we can substantially improve the computational cost by modifying the architecture without the sacrificing performance, by (i) fixing the last-layer classifier as a Simplex ETF1S, and (ii) reducing the feature dimension K. Here, to demonstrate our method can achieve state-of-the- performance, we do include data augmentation in the training of our ResNet50 model [91] on CIFARIO, achieving around 95% test accuracy. See Appendix B for the results on MNIST and CIFARIO with ResNet18.
Fixing the Last-layer Classifier as a Simplex ETE. Figure 5 presents comparison of learned and fixed classifiers in terms of within-class variation collapse (NC,), self-duality (AC3), training accuracy, and test accuracy. These results imply that the fixed classifier exhibits the same within- class variation collapse for the features H. and achieves the same classification accuracy as the fully-trained classifier. On the other hand, fixing the classifier can reduce the number of parameters and the computational complexity for training. The number of parameters in the classifier can be significant for tasks with large number of classes and large feature dimensions. For example, for ImageNet, a dataset with K = 1000 classes, fixing the classifier can reduce 8.01%, 11.76%, and 52.56% of total learning parameters for ResNer50, DenseNet169 [19], and ShuffleNet [92], respectively. We note that our result also provides a theoretical justification for the work in [93] that fixes the classifier orthonormal matrices. Indeed, these close to simplex ETFs, particularly when the number of classes is large.
Feature Dimension Reduction for H E Rdxan by Choosing d = d.K.' In many classification problems, the practice of deep learning typically uses feature dimension d that is much larger than the number of classes K. In contrast, NCC implies that there is no need to choose a d that is much larger than the number of classes K. Reducing the dimension d can lead to substantial reductions in memory and computation cost. As shown in Figure 5, we also train all the weights of ResNet50 on CIFARIO using SGD with = K. The results demonstrate that ACC persists even when we choose d = K, and the network achieves on-par performance with networks of large d, in terms of training and test accuracy. This implies that when the number of classes K is small, we can choose a small feature dimension d = K (or d 2 K) instead of using large universal d to reduce the computation and memory costs for training. By setting d = K, this reduces the amount of parameters and hence the memory cost in ResNet18 and ResNet50 by 20.70% and 4.45% respectively.
5 Conclusion
In this work, we have provided an in-depth analysis to demystify the ACC phenomenon, which ap- pears during the terminal phase of training deep networks in classification problems. Based on the unconstrained feature model [24-26], we proved that Simplex ETFs are the only global minimiz- ers of the cross cross-entropy training loss with weight decay and bias. Moreover, we showed that the loss function is strict saddle function with respect to the last-layer features and classifiers, with no other spurious local minimizers. In contrast to existing landscape analyses for deep neural networks, which mostly focus on the optimization perspective, our simplified analysis not only characterizes
K 15 "Specifically, we set W - V R of a d d identity matrix, which lifts K x K ETF to d x K matrix. For simplicity, we also learn the bias term the last layer, though our result indicates it can set as Wh, where h the glohal mean the features. Though Theorem 3.2 requires d > K, conjecture also holds for d = K discussed in Section 3.1.
P (Ix + where P E [2dx contains the first K columns

---

the features that are learned in the last layer, but also explains why they can be efficiently optimized. This provides support for empirical observations in practical deep network architectures. Moreover, the study of last-layer features could have profound implications for optimization, generalization, and robustness of broad interests, which are the subjects of future work. It is also of interest to extend the current study to the case where d < K, which is the case in contrastive learning [94,95] 95] and many applications, such as recommendation systems [96] and document retrieval [97].
Acknowledgment
777 acknowledges support from NSF grant CCF 2008460. XI. and QQ acknowledge support from NSF grant DMS 2009752. IS acknowledges support from NSF grant CCF 2007649. We would like to thank Qinqing Zheng (Facebook Al Research), Vardan Papyan (U. Toronto), and Felix Yu (Google Research) for timely pointing us to some important references and valuable feedback on the final draft. We thank Christina Baek (UC Berkeley) and Sam Buchanan (Columbia U.) for fruitful discussions during various stages the work. We also thank Zhexin Wu (U. Michigan) for proofreading and pointing out several typos in the draft, and the four anonymous reviewers for their constructive comments.
References
[1] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117644))2462--26663, 2020.
[2] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classificatiin with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
[3] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436444, 2015.
[4] Tan Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT
press Cambridge, 2016.
[5] Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Zídck, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure predic- tion using potentials from deep learning. Nature, 577(7792):706-777 700,70006777777770 2020.
[6] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization deep learning. arXiv preprint arXiv:1412.6611 2014.
[7] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1 )0:2222 2878, 2018
[8] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factor- ization. Advances in Newral Information Processing Systems, 32, 2019.
[9] Edward Moroshko, Blake E Woodworth, Suriya Gunasckar, Jason D Lec, Nati Srebro, and Daniel Soudry, Implicit bias in deep linear classification: Initialization scale training accuracy. Advances in Neural Information Processing Systems, 33, 2020.
[10] Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by norms. Advances in Neural Information Processing Systems, 33, 2020.
[11] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 1rXivv1611.03530 2016.
[12] Mikhail Belkin, Daniel Hsu, Siyuan Ma. and Soumik Mandal. Reconciling modern machine-learing practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116632) 2019.
[13] Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and double descent curve. arXiv preprint arXiv: 05355, 2019.
[14] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Πlya Sutskever. Deep double descent: Where bigger models and more data hurt. arXiv preprint arXiv:1912.02299 2019.
[15] Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking viai-vari trade- off for generalization of neural networks. In International Conference on Machine Learning, pages 10767-10777. PMLR, 2020.
[16] Vardan Papyan. Traces of class/cross-clas class structure pervade deep learning spectra. Journal of Machine Learning Research, 21(252):1 64, 2020.
10

---

[17] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recog- nition. arKiv preprint arXiv:1409.1556 2014.
[18] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778,
2016.
[19] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convo- lutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[20] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. at&t labs, 2010.
[21] Alex Krizhevsky, Geoffry Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[22] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248 255, leee, 2009.
[23] Dustin G Mixon, Hans Parshall, and Jianzong Pi, Neural collapse with unconstrained features. arXiv preprint 2r 1619, 2020.
[24] Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss. arXiv preprint arXiv:2012.0844 0.8465,, 2020.
[25] E Weinan and Stephan Wojtowytsch. On the emergence of tetrahedral symmetry in the final and penul- timate layers of neural network classifiers. arXiv preprint arXiv:2012.05440 2020.
[26] Cong Fang. Hangfeng He, Qi Long, and Weijie Su. Layer-peeled model: Toward understanding well- trained deep neural networks. arXiv preprint arXiv:2101.122
2021.
[27] Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised constrastive learning. In International Conference on Machine Learning, pages 3821-3830 PMLR, 2021.
[28] Tolga Ergen and Mert Pilanci. Revealing the structure of deep neural networks via convex duality. In International Conference on Machine Learning, pages 1004-3014 PMLR, 2021.
[29] John Zarka, Florentin Guth, and Stéphane Mallat, Separation and concentration in deep networks, arXiv preprint arXiv:2012.10422 2020.
[30] Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. arXiv preprint arXiv:2006.88558 2020.
[31] Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Ma. Deep networks from the principle of rate reduction. arXiv preprint arXiv:2010. 14765, 2020.
[32] Benjamin D Haeffele and René Vidal. Global optimality in tensor factorization, deep learning, and beyond. arXiv preprint 156.0:556600540) 2015.
[33] Pierre Baldi and Kurt Homikk. Neural networks and principal component analysis: Learning from exam- ples without local minima. Neural networks, 2(1):53-58, 1989.
[34] Kenji Kawaguchi. Deep learning without poor local minima. Advances in neural information processing systems, 29:586-594, 2016.
[35] Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks. In International Conference on Machine Learning, pages 4433-4441 PMIR, 2018.
[36] Chulhee Yun. Suvrit Sra, and Ali Jadbabaie. Small nonlinearities in activation functions create bad local minima in neural networks. In International Conference on Learning Representations, 2018.
[37] Thomas Laurent and James Brecht. Deep linear networks with arbitrary loss: All local minima are global. In International conference on machine learning, pages 2902-2907. PMLR, 2018.
[38] Maher Nouiehed and Meisam Razaviyayn. Learning deep models: Critical points and local openness. arXiv preprint
2018.
[39] Shiyu Liang, Ruayu Sun, Jason Lee, and R Srikant. Adding one neuron can eliminate all bad local minima. Advances in Neural Information Processing Systems, 2018:4300-4300, 2018.
[40] Zhihui Zhu, Daniel Soudry. Yonina C Eldar, and Michael B Wakin. The global optimization geometry of shallow lincar neural networks. Journal of Mathematical Imaging and Vision, pages 1-14, 2019.
[41] Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Global optimality conditions for deep neural networks. In International Conference on Learning Representations, 2018.
[42] Mo Zhou, Rong Ge, and Chi Jin, A local convergence theory for mildly over-parameterized two-layer neural network. arXiv preprint 12240.2002.02100, 2021.

---

[43] G Cybenko. Approximation by superposition of sigmoidal functions. Mathematics of Control, Signals and Systems, 2(4):303-314, 1989.
[44] Kurt Homik, Approximation capabilities of multilayer feedforward networks. Neural networks,
4(2):251 257, 1991.
[45] Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: a view from the width. In Proceedings of the s1 International Conference on Neural Information Processing Systems, pages 6232-6240, 2017.
[46] Uni Shaham, Alexander Cloninger, and Ronald Coifman. Provable approximation properties for deep neural networks. Applied and Computational Harmonic Analysis, 44(3):537-557, 2018
[47] XY Han, Vardan Papyan, and David I. Donoho. Neural collapse under mse loss: Proximity to and dynamics the central path. arXiv preprint wwXiv2210600207. 2021.
[48] Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng. and Weijie J Su. An unconstrained layer-peeled perspective on neural collapse. arXiv preprint 2rXiv:2110002700 2021.
[49] Rong Ge. Furong Huang. Chi Jin, and Yang Yuan. Escaping from saddle points online stochastic gradient for tensor decomposition. In Proceedings of The 28th Conference on Learning Theory, pages 797-842, 842, 2015.
[50] Ju Sun, Qing Qu, and John Wright. When are nonconvex problems not scary? arXiv: 15106060966 2015.
arXiv preprint
[51] Yugian Zhang, Qing Qu, and John Wright. From symmetry to geometry: Tractable nonconvex problems. arXiv preprint arXiv:2007.06553, 2020.
[52] Jason D Lee, Max Simchowitz, Michael I Jordan, and Benjamin Recht. Gradient descent only converges to minimizers. In Conference on learning theory, pages 1246-1257, PMLR, 2016.
[53] Michael Elad, Dror Simon, and Aviad Aberdam. Another step toward demystifying deep neural net- works. Proceedings of the National Academy of Sciences, 117(44):2707-2777 27072, 2020.
[54] Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems, 2018.
[55] Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradient dynamies in linear neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bir Buc, Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, 2019.
[56] Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3420-3428. PMLR, 2019.
[57] Ziwei Ji, Miroslav Dudík, Robert E Schapire, and Matus Telgarsky. Gradient descent follows the regu- larization path for general losses. In Conference on Learning Theory, pages 2109-2136. PMLR, 2020.
[58] Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations, 2019.
[59] Ohad Shamir. Gradient methods never overtit separable data. Journal of Machine Learning Research, 22(85):1-20, 2021.
[60] Meena Jagadecsan, Ilya Razenshteyn, and Suriya Gunasckar. Inductive bias of multi-channel linear convolutional networks with bounded weight norm. arXiv preprini arXiv:2102.1 122., 2021.
[61] Chong You, Zhihui Zhu, Qing Qu, and Yi Ma. Robust recovery via implicit bias of discrepant learning rates for double over-parameteriiaaion. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 17733-17744. Curran Associates, 202..
[62] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
[63] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
[64] Mor Shpigel Nacson, Nathan Srebro, and Daniel Soudry. Stochastic gradient descent on separable data: Exact convergence with fixed learning rate. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3051-3059. PMLR, 2019.
[65] Qian Qian and Xiaoyuan Qian. The implicit bias of adagrad separable data. arXiv preprint 4rXiv:1906.03559, 2019.
[66] Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning, pages PMLR, 2018.
12

---

[67] Vinod Nair and Geoffrey Hinton. Rectified linear units improve restricted boltzmann machines. vol- ume 27, pages 807-814, 2010.
[68] Sergey loffe and Christian Szegedy, Batch normalization: Accelerating deep network training by reduc- ing internal covariate shift. In ICML, pages 448-456, 2015.
[69] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Stam Review, 20(2):23-311, 2018.
[70] Qing Qu, Ju Sun, and John Wright. Finding a sparse vector in a subspace: Linear sparsity using alter- nating directions. In Advances in Neural Information Processing Systems, pages 3401-3409, 2014.
[71] Tuo Zhao, Zhaoran Wang, and Han Liu. A nonconvex optimization framework for low rank matrix estimation. In Advances in Neural Information Processing Systems, pages 559-567, 2015.
[72] Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere I: Overview and the geometric picture. IEEE Transactions Information Theory, 63(2):853-884, 2016.
[73] Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere II: Recovery by Riemannian trust rreginn method. IEEF Transactions on Information Theory, 63(2):853-884 2017.
[74] Qiuwei Li, Zhihui Zhu, and Gongguo Tang. The non-convex geometry of low-rank matrix optimization. Information and Inference: Journal of the IMA, 8(1):51-96, 2018.
[75] Qing Qu, Yuqian Zhang, Yonina Eldar, and John Wright. Convolutional phase retrieval. In Advances in Neural Information Processing Systems, pages 6086-6096, 2017.
[76] Sun, Qing Qu, and John Wright. A geometric analysis of phase retrieval. Foundations of Computa- tional Mathematics, 18(5):131-11999 2018.
[77] Zhihui Zhu, Qiuwei Li, Gongguo Tang, and Michael B Wakin. Global optimality in low-rank matrix optimization. IEEE Transactions on Signal Processing, 66(13))3614-36) 2018.
[78] Xingguo Li, Junwei Lu, Raman Arora, Jarvis Haupt, Han Liu, Zhaoran Wang, and Tuo Zhao. Symmetry, saddle points, and global optimization landscape of nonconvex matrix factorization. IEEE Transactions on Information Theory, 6586):3489-351, 2019.
[79] Zhihui Zhu, Qiuwei Li, Xinshuo Yang. Gongguo Tang, and Michael B Wakin. Distributed low-rank matrix factorization with exact consensus. In Advances in Neural Information Processing Systems, pages 8422-8432, 2019.
[80] Yuejie Chi, Yue M Lu, and Yuxin Chen. Nonconvex optimization meets low-rank matrix factorization: An overview. IEEE Transactions on Signal Processing, 67(20):5239-526) 2019.
[81] Qing Qu Yuexiang Zhai, Xiao Li, Yuqian Zhang, and Zhihui Zhu Geometric analysis of nonconvex optimization landscapes for overcomplete learning. In International Conference on Learning Represen- tations, 2020.
[82] Qing Qu, Zhihui Zhu, Xiao Li, Manolis C. Tsakiris, John Wright, and René Vidal. Finding the sparsest vectors in a subspace: Theory, algorithms, and applications. arXiv preprint 200790000.,6677, 2020.
[83] Qing Qu, Xiao Li, and Zhihui Zhu. Exact recovery of multichannel sparse blind deconvolution via gradient descent. SIAM Journal on Imaging Sciences, 1303)1163- 2020.
[84] Rong Ge, Jason D Lec, and Tengyu Ma. Matrix completion has no spurious local minimum. arXiv preprint arXiv: 00772, 2016
[85] Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Global optimality of local search for low rank matrix recovery. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 3880-3888, 2016.
[86] Rong Ge, Chi Jin, and Yi Zheng. No spurious local minima in nonconvex low rank problems: unified geometric analysis. In International Conference on Machine Learning, pages 1233-1242. PMLR, 2017.
[87] Qiuwei Li, Zhihui Zhu, and Gongguo Tang. The non-convex geometry of low-rank matrix optimization. Information and Inference: A Journal of the IMA, 8(1):51 96, 2019.
[88] Samuel Burer and Renato DC Monteiro. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, 95(2):329-357, 2003.
[89] Daniel Kunin, Jonathan Bloom, Aleksandrina Goeva, and Cotton Seed. Loss landscapes of regularized linear autoencoders. In International Conference on Machine Learning, pages 3560-3569, PMLR, 2019.
[90] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmiss/, 1998.
[91] kuang Liu, Wei Yang, Yang Peiwen and Felipe Ducau. Train cifar10 with pytorch. haw.a//wwanilliuoacccccrrbbaaaae
[92] Xiangyu Zhang. Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convo- lutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848 6856, 2018
13

---

[93] Elad Hoffer, Ita Hubara, and Daniel Soudry. Fix your classifier: the marginal value of training the last weight layer. In International Conference on Learning Representations. 2018.
[94] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.057000 2020.
[95] Kaiming He, Hanqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsuper- vised visual representation learning. In Proceddiggs of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7729-973. 2020.
196] Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, pages 191-198, 2016.
[97] Wei-Cheng Chang, X Felix, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre tre-training tasks for embedding- large-scale retrieval. In International Conference on Learning Representations 2019.
[98] Ruo-Yu Sun. Optimization for deep learning: An overview. Journal of the Operations Research Society of China, (2):24492294, 2020.
[99] Ruoyu Sun, Dawei Li, Shiyu Liang, Tian Ding, and Rayadurgam Srikant. The global landscape of neural networks: An overview. IEEE Signal Processing Magazine, 37(5):95 -08, 2020.
[100] Zeyuan Allen Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning, pages 242-252 2019.
[101] Simon Du, Jason Lee. Haochuan Liwei Wang, and Xiyu Zhai, Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning, pages 1675-1685, 2019.
[102] Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural langent kernel: Convergence and general- ization in neural networks. NeurlPS, 2018
[103] Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over- parameterized deep relu networks. Machine Learning, 1096))4677-92, 2020.
[104] Andrea Montanari and Yiqiao Zhong. The interpolation phase transition in neural networks: Memoriza- tion and generalization under lazy training. arXiv preprint arXiv:2007. 2020.
[105] Sam Buchanan, Dar Gilboa, and John Wright. Deep networks and the multiple manifold problem. arXiv preprint arXiv:2008. 1245, 2020.
[106] Lénaic Chizat, Edouard Oyallon, Francis Bach. lazy training in differentiable programming. Advances in Neural Information Processing Systems, 3 2947, 2019.
[107] Marina Danilova, Pavel Dvurechensky. Alexander Gasnikov, Eduard Gorbunov, Sergey Guminov, Dmitry Kamzolov, and Innokentiy Shibaev. Recent theoretical advances non-convex optimization. arXiv preprint arXiv:2012.061 2020.
[108] Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M. Kakade, and Michael I. Jordan. On nonconvex op- timization for machine learning: Gradients, stochasticity, and saddle points. ACM, 68(2), February 2021.
[109] Yenson Lau, Qing Qu, Han- Kuo, Pengcheng Zhou, Yuqian Zhang, and John Wright. Short and sparse deconvolution geometric approach. International Conference on Learning Representa- tions, 2020.
[110] Han-Wen Kuo, Yenson Lau, Yuqian Zhang, and John Wright. Geometry and symmetry in short-and- sparse deconvolution. In International Conference on Machine Learning, pages 3570-3580. PMLR, 2019.
[111] Yugian Zhang, Ken Kuo, and John Wright. Structured local optima in sparse blind deconvolution. IEEE Transactions on Information Theory, 66(1):419 -522, 2019.
[112] Raghu Bollapragada, Jorge Nocedal, Dheevatsa Mudigere, Hao-Jun Shi, and Ping Tak Peter Tang. A pro- gressive batching I-bígs method for machine learning. In International Conference on Machine Learn- ing. pages 620-629. PMLR, 2018
[113] Yi Ren and Donald Goldfarb. Exonecker-factoree quasi-newton methods for convolutional neural net- works. arXiv preprint arXiv:2102.06737, 2021.
[114] Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge University Press, 2012.
[115] Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Reexamining low rank matrix factorization for trace norm regularization. arXiv preprint arXiv: (tv:/700 7706.08944 2017.
[116] G Alistair Watson. Characterization of the subdiffercntial of some matrix norms, Linear algebra and its applications, 170:33 45, 1992.
[117] Benjamin Recht, Maryam Fazel, and Pablo A Parrilo. Guaranteed minimum-rand solutions of linear matrix equations via nuclear norm minimization. SIAM Review, 52(3):77--01,
2010. 2010.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In the abstract, introduction,
main results, and conclusion, we explicitly stat that our results are about unconstrained feature model.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This paper mainly focuses on understanding the neural collapse phenomena observed in practical neural networks. Based on this understanding, we propose to fix the last layer classifier as a Simplex ETF. So no potential negative societal impact is expected of this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We explicitly mention the assumptions in Theorem 3.1 and Theorem 3.2.
(b) Did you include complete proofs of all theoretical results? [Yes] We include all the proofs in the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Section and Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] All the results are reported in terms of learning curves and each figure includes many plots, error bars experiments multiple times, and observe very similar performance in terms of NC, testing accuracy, etc.
reported. Bud we do the
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix B.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite them in Section 4.
(b) Did you mention the license of the assets? [Yes] This is mentioned in Appendix B.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you't re using/curating? [Yes] This discussed in Appendix B.
(c) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "Uf_75S93nOw", "reviewer": "Reviewer_Rfbc", "summary": "Neural collapse is an intriguing phenomenon that has been observed in neural network classifiers as the training error goes to 0 where the last layer of the neural network has collapsed input points to corners of a simplex, thereby minimizing variability, and maximizing separability. While this phenomenon seems well studied the authors extend it by studying the critical points of this optimization geometrically by relating it to a convex problem. They show in Thrm 3.2 that the critical points are either global minimal or saddle points with negative curvature which can be escaped by a minimizer. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work builds incremental, but important advances on what seems to be well-investigated theory regarding deep neural networks trained for K-class classification via a cross entropy loss, neural collapse. Neural Collapse was first reported in [1, 16] Key parts of the theory include the unconstrained feature model which involves analyzing only the last layer , and relating the non-convex optimization to a convex problem to characterize the minima. These parts of the theory have all been introduced and used in various works such as [23] and [30] respectively. To that this work extends these ideas a bit. They show that using [30] one can characterize the critical points as having at least one direction of negative curvature, by constructing a negative curvature direction. They then show that this that sufficiently powerful neural networks can escape these saddle points and reach a global optimum, and that this is independent of the optimization algorithm. They also propose an architectural choice of d=K in the last dimension, which is used frequently regardless. A more interesting idea is the fixing of the classifier. \n\nThe writing of the paper however seems list-like and with a piece of theory and minor ideas for practice. Instead I would consider first explaining the previous theory fully, particularly the unconstrained feature model, which as the authors say in the supplement may be an alternative condition to the \"infinitely wide condition of the NTK\" for casting the neural network as a linear model in an alternative feature space. What is the intuition behind the unconstrained feature model? Does it imply that those layers are not necessary or that they can be trained randomly? Some experiments here would help as well, changing the width for example of previous layers. \n\nSecond, the authors need to provide the background of [30] to address their geometric idea, which is in the title but not covered in the main paper. I believe this is the main contribution of the paper. This is what makes this idea practical and apply to modern DL training methods. Perhaps an additional metric would be to actually estimate the curvature of the landscape using Olivier-Ricci curvature or other data estimators of curvature would emphasize this points. See: https://www.sciencedirect.com/science/article/pii/S1631073X07004414?via%3Dihub.\n\nFinally, I think that d=K is somewhat obvious as an architectural choice, particularly given one hot encodings commonly used in classifications. I would rather understand more about fixing the last layer. Does the exact simplex used matter? How is it enforced?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "Nq-qQAjB-ge", "reviewer": "Reviewer_xttS", "summary": "This is an extremely well written (articulate and readable) and diligently prepared (completely\nby the book) manuscript. For example, it draws carefully on more than 100 other manuscripts.\nThis alone makes the manuscript valuable to other people, as it teaches how work can\nbe exposited and papers can be written.\n\nThe results are interesting and instructive, and in some way connect\nthese questions to some recent work in non-convex optimization.\nThe practical benefit of this work -- tens of percentage points speedups --\nare meaningful.\n\n", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: yes this is original.\nQuality: this is very well prepared as an academic work.\nClarity: this is exceptionally clear.\nSignificance: this paper shows that in this field,\ntheory can eventually make sense \nof phenomena which have been discovered\nand carefully documented empirically. The\nmessage of science progressing in this way is\nworth spreading far and wide. Generally speaking theorists\nworking in this area have not been concerned with explaining empirically\nuncovered phenomena. So this example may lead to a new\nsynergy.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In a few cases the paper betrays its authors' obvious theoretical \norigin. Theorists still believe that deep learning is somehow\nin a problematic situation because of its reliance on empiricism and\nphenomenology and the \"mysteries\" unearthed by empirical science.\nThis is simply a parochial viewpoint. It would be better for theorists to recognize that\nthe state of the art in deep learning is an empirical triumph and that \nit's better for theory to contribute to systematizing the various lessons that \nare being discovered by active machine learning researchers.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 "}, "extra_text": {}}, {"review_id": "FRFksYoVT3z", "reviewer": "Reviewer_HfZN", "summary": "The paper studies collapse phenomena of representations from neural classification models trained beyond zero training error with cross entropy. The theoretical contribution of the paper is an analysis of such phenomena in the so called unconstrained feature model. In this setting, the authors derive the global loss minimizers, and show that these can be obtained via gradient based optimization, as the critical points of the loss function are benign. These theoretical results are underpinned by experiments, which investigate the collapse phenomena for a ResNet-18 architecture when trained via different optimization algorithms or with randomly labeled data.\nThe theoretical results also indicate, that in practice, the last linear layer of a neural network can be fixed a priory and of low dimension.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Overall assessment:\n\nThis is an interesting paper which promotes the understanding of loss minimizers and loss optimization in the supervised classification setting. There are several issues regarding the assumptions and claims made in the paper, but I think these can be cleared with a revision. Furthermore, its connection to related work is described kind of superficially in the paper. In general, this is a good submission and I am willing to raise my score, once the major issues are resolved.\n\n\n\n### Significance/Originality:\n\nInspired by the observed neural collapse phenomenon in Papyan et al, there is a recent stream of research, which studies such phenomena from a theoretical perspective. Mainly, by deriving the loss minimizers under various assumptions in the unconstrained feature model. The first theoretical result in this paper is quite similar to these works. \nThe second theoretical result provides a real addition to the literature. There, in contrast to previous work, the authors do analyze the critical points in the unconstrained feature model and thus take into account that neural networks are usually trained via gradient based optimization.\nIn summary, the paper promotes our understanding of the neural collapse phenomenon.\n---\n\nWhile previous work relies on norm constraints on the representations, the authors argue, that weight decay yields an implicit L2-penalty on the representations. This is a quite interesting and original assumption. The authors argue that the assumption is meaningful, but unfortunately, this was not discussed more in detail, see Quality.\n\n---\nFrom the paper itself, it is difficult for readers to assess its role in the literature. This is because the authors tend to simultaneously cite multiple references which are only summarized as a whole. As a consequence, most references and how they relate to the submitted work are not discussed properly, leading to superficial related work sections. In the following paragraphs, I will precise this with some examples from the text.\n\nFor example, the section related to existing landscape analysis consists mainly of the statement that the authors' \"result has much broader implications than the previous results [31, 32, 35, 36, 38, 39, 85]\" (line 244), however the contribution of these earlier works, and thus the reason for the, (according to the authors) limited implications of these works remains unclear. Addendum: There is a section in the supplementary material which discusses these works in more detail. The existence of this section and its most important parts should be highlighted in the main text.\n\nThe relationship to low rank-matrix recovery section suffers from similar problems. The authors write \"As discussed in Section 1, it has been recently shown that the strict saddle property holds for a wide range of nonconvex problems in machine learning [66–79], including low-rank matrix recovery [74, 76, 80–83]\" (line 233). (I did not find this in section 1, maybe the authors could specify the exact location of the discussion in section 1.) \\\nFirst of all, I don't think that there is much benefit in simply listing machine learning papers which study critical points of some functions. The authors should either expand on the relationship of these references to the submitted work, or remove them.\\\nFurthermore, this list ([66–79]) is far from complete.  To name a few additional references, critical points were also studied for residual networks, see [5'] and [6']. There is also the work by Kawaguchi, which the authors already cite (reference [32]), but not in this context. \\\nIn fact, judging from the titles of [66–79], this list seems limited to the subarea of low rank matrix optimization and is due to mainly the same group of authors. If the point of including these references is to illustrate the existence of strict saddle points in many machine learning problems, then it is counterproductive to limit the references to a particular subfield. But as already stated, the main problem is not the lack of the references, but the superficial treatment of the provided ones.\\\nI am aware, that the works [66–79] are also presented in the supplementary material (starting at line 725). But even there, only the application areas of these works are named, but how these works relate to each other and the submitted work is not discussed. For example, it remains unclear, whether there is a central paper which introduced the required mathematics and the others are just applications of it, or if these works all build on each other and, if yes, how and to which extend. At the end of this paragraph in the supplementary material, two papers are highlighted; \"In fact, our proof techniques are inspired by recent results on low-rank matrix recovery [73, 76]\". But again, what these results entail, and how they inspired the authors' proof techniques remains unspecified.\n---\n\nI also want to highlight some related work not mentioned in the paper.\n\nAssuming that the representations and classifier weights are constrained on a sphere and discarding bias terms, Wang et al.[1'] already showed in 2017 that if the representations are already collapsed, then the class means need to be maximally separated in order to minimize the cross entropy loss.\n\nClosely related is also recent work [2'], which shows that in the unconstrained feature model loss minimization leads to neural collapse for models trained with cross entropy or with a supervised contrastive loss function. They assumed representations on a sphere, but allowed for weight decay. Similar to this paper, they also experimented on randomly labeled data and with fixed classifier weights. It would be worth adding a discussion about it.\n\nFurthermore, there is [4'], in which deep neural networks are studied using a convex analytic framework. Therein, it is shown that the weight matrices align with the previous layers at optimality. This serves as an explanation of neural collapse (see Corollary 4.3 in the reference). It would be interesting to discuss similarities and differences between [4'] and this work, as convexity is central to both.\n\n\n### Clarity:\n\nThe paper is well structured (global optima result -> local critical points result -> experiments) and easy to read. Good job. \nIn particular, I like the bullet points after the problem formulation and theorems, which discuss the assumptions of the theorems or point to related work (As written above, the related work sections need more details).\\\nI also think that it was a good idea to graphically compare the works [24-26] on neural collapse in table 1. (minor remark to the authors: insert some additional white space between the caption and the table to improve readability.)\n---\n\nThe proofs in the supplementary material are a bit difficult to follow. \\\nIn general, the proof of theorem 3.1 seems to be more involved than the proof of similar results, for example in [2'], [3'].  For the reader it remains unclear, why this is the case and I would appreciate if the authors comment on this in their response to the review.\\\nIn particular, I do not understand, why Lemma D.5 is in the form of \"for any $c_1>0$, the cross entropy loss can be lower bounded by...\".  As consequence,  all other intermediate results building on lemma D.5 are also of this form, which makes the proofs hard to read.\nI might be wrong, but to me it seems, that $c_1$ is introduced artificially, but never really used, as one is only interested in the equality case between lines 945 and 946 anyways, and this does not depend on $c_1$.\n\n### Quality:\n\nThe paper includes well designed experiments.\\\nExperiment 4.1 establishes, that neural collapse occurs independently from the optimization algorithm. Experiment 4.2 studies the unconstrained feature model assumption in greater detail. As one might already suspect, it clearly demonstrates that increasing the model capacity (measured by the width of the convolutional layers) leads to more simplex-ETF like representations. Thus confirming the unconstrained feature model assumption for highly overparametrized networks.\nI really liked that the authors introduced metrics (NC1-NC4) for measuring each of the respective neural collapse properties separately.\n---\n\nHowever, I am not convinced, that the experiments demonstrate that the input \"plays minimal influence\" (line 310) for reaching neural collapse. This is simply not studied in the experiments, as they all operate on the same input data.\n\n---\nWhile I do not doubt the results, the experiments would definitely benefit from more statistical power. So far, only the results of a single run (in each setup) is presented, as acknowledged by the authors in the checklist. However, reporting learning curves does not automatically prohibit to report error bars. For example, one could report mean learning curves and shade the error regions.\n\n---\nThe theory appears to be correct. However, I have to admit, that I was not able to check all of it thoroughly.\n\n---\nThe main weakness of the paper is that its fundamental assumption (separating it from other works using the unconstrained feature model) is not justified enough.\n\nThe theory relies on the assumption, that the representations are subject to an L2-penalty in the unconstrained feature model. As the authors argue, it seems plausible, that this assumption is  meaningful, i.e. it provides a valid approximation of the representations of a neural network. However, on a closer look, it appears that this assumption is made only so that a connection to a convex optimization problem can be established via Lemma C.3. From this perspective, the validity of the this assumption needs to be discussed in greater details and to be confirmed in experiments.\\\nFor example (other strategies are possible as well), the proof of theorem D.1 could be adapted such that optimal norms of the representations can be predicted (compare Corollary 1 in [2']). Thus, if one knows how the regularization strength $\\lambda_H$ depends on the model depth and the weight decay parameter used in practice, this prediction on the norms could then be tested in experiments.\n\n---\nThe authors state that they make the assumption of balanced class multiplicities in the training set due to simplicity (line 140). However, this is misleading, as this assumption is crucial for the theory (e.g. the proof of Lemma D.3 uses this assumption). If the class multiplicities are unbalanced, the loss minimizer is not a simplex ETF.\n\n\n### Minor remarks:\n\nthe links within the paper do not work \n\nline 289: typo quantity -> quantify\n\nThe most left (and maybe also the most right) plots in figures 3 and 4 would probably benefit from a log scale. As of now, the most right plot in figure 4 is unreadable.\\\nIn figure 4 (most left) the values t on the y-axis seems to be labeled incorrectly. What is meant there? t (i.e., as it is now), e^t, 10^t, or something else?\\\nI might have missed it, but please mention the width of a standard resnet-18 model for comparison.\n\nline 185: \"The requirement that d≥K is  necessary  for  Theorem  3.1  to  hold,  simply  because K vectors  in R^d cannot  form  a K-Simplex ETF if K > d.\"\\\nI don't think this is true. K vectors can still form a K-Simplex ETF if K=d+1, for example an equilateral triangle in R^2.\n\n**References:** \\\n[1'] Wang et al., NormFace:l2hypersphere embedding for face verification, ACM Multimedia, 2017\\\n[2'] Graf et al., Dissecting Supervised Contrastive Learning, ICML, 2021\\\n[3'] Lu and Steinerberger, Neural collapse with cross-entropy loss, arxiv, 2020\\\n[4'] Ergen and Pilanci, Revealing the Structure of Deep Neural Networks via Convex Duality, ICML 2021\\\n[5'] Yun et al., Are deep ResNets provably better than linear predictors?, NeurIPS 2019\\\n[6'] Hardt and Ma, Identity Matters in Deep Learning, ICLR 2017\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I agree with the authors, that there is no negative societal impact to be expected from this work. As acknowledged by the authors, the work is limited to the unconstrained feature model.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "I don't really know"}, "extra_text": {}}, {"review_id": "0S0lwuSkxiy", "reviewer": "Reviewer_7gsu", "summary": "This paper provides a landscape analysis for neural collapse with regularization analysis. The theoretical results are nice, thus I voted an accept decision. But reviewer has a concern about the novelty of the proof and the implications of the theoretical results.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "this is a nice paper analysis the landscape of the neural collapse model, The main concern reviewer have are following\n[1] The novelty of no strictly bad local minimum:\nThe analysis is actually the same as the landscape analysis of the matrix decomposition model in my mind. The paper deal with the problem as a convex_function(WH)+\\lambda ||W||^2_F +\\lambda ||H||^2_F where the nonconvexity is introduced by WH. I wonder whether the landscape results are the same as the previous matrix decomposition paper\n\n[2] The novelty of the proves of the unconstrained model. I wonder can we use the Lagrange dual relationship turns the global minimum of the unconstrained one to a constrained one. In practice, no one use regularization on features, I still can't see the benefit and novelty to turn the constrained model to a unconstrained one.\n\n[3] The implications of the theoretical results are separated from the theoretical results in the paper. The \"new method\" you can do that even after the first neural collapse paper and the \"new\" methods can also be understood as changing a loss function[using target feature as loss] and this leads to question 5.\n\n[4] From the theoretical result, can we say the layer before the last layer is also neural collapse? I wonder will the proof stays true while we inductive on the depth.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "see detailed review", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides an interesting well presented study of neural collapse, mostly focusing on solid theoretical results, but also demonstrating their empirical implication. All reviewers agree the paper should be accepted, commending the high quality of the presentation in the paper. Further, it is clear from the reviews and the paper that this is an important problem, and the results are sufficiently significant to be of interest to a wide audience at NeurIPS (as evidence, two reviewers marked the paper as top 50% of accepted NeurIPS papers with high or absolute confidence). Therefore, I recommend acceptance as a spotlight.

To the authors: please read carefully the reviewer comments and follow up on your responses with appropriate revisions to the paper.
Decision: Accept (Spotlight)