A Geometric Analysis of Neural Collapse with Unconstrained Features
Year: 2021

Abstract:
We provide the first global optimization landscape analysis of Neural Collapse— an intriguing empirical phenomenon that arises in the last-layer classifiers and features of neural networks during the terminal phase of training. As recently reported in [1], this phenomenon implies that (i) the class means and the last-layer classifiers all collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and (ii) cross-example within-class variability of last-layer activations collapses to zero. We study the problem based on a simplified unconstrained feature model, which isolates the topmost layers from the classifier of the neural network. In this context, we show that the classical cross-entropy loss with weight decay has a benign global landscape, in the sense that the only global minimizers are the Simplex ETFs while all other critical points are strict saddles whose Hessian exhibit negative curvature directions. Our analysis of the simplified model not only explains what kind of features are learned in the last layer, but also shows why they can be efficiently optimized, matching the empirical observations in practical deep network architectures. These findings provide important practical implications. As an example, our experiments demonstrate that one may set the feature dimension equal to the number of classes and fix the last-layer classifier to be a Simplex ETF for network training, which reduces memory cost by over 20% on ResNet18 without sacrificing the generalization performance. The source code is available at https://github.com/tding1/Neural-Collapse.

heading: 1 Introduction
text: In the past decade, the revival of deep neural networks (DNN) has led to dramatic success in numerous applications ranging from computer vision, to natural language processing, to scientific discovery and beyond [2–5]. Nevertheless, the practice of deep networks has been shrouded with mystery as our theoretical understanding for the success of deep learning remains elusive. There are many intriguing phenomena, such as implicit algorithmic bias in training [6–10], and good generalization of highly-overparameterized networks [7,11–15], that seem often contradictory to, or cannot be explained by, classical optimization and learning theory. ∗The first two authors contributed to this work equally.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
… … … … …
Input Hidden layer 1 OutputHidden layer L-1
Figure 2: Illustration of the unconstrained feature model, where the gray box is peeled off so that the representation h is modeled by a simple decision variable for every training sample.
Towards demystifying DNN, recent seminal work [1, 16] empirically discovered an intriguing phenomenon that persists across a range of canonical classification problems during the terminal phase of training. As illustrated in Figure 1, it has been widely observed that last-layer features and classifiers of a trained DNN exhibit simple but elegant mathematical structures: • Variability Collapse: cross-example within-class variability of last-layer features collapses to
zero, as the individual features of each class themselves concentrate to their isolated class-means. • Convergence to Simplex ETF: the class-means centered at their global mean are not only linearly
separable, but are actually maximally distant and located on a sphere centered at the origin up to scaling (i.e., they form a Simplex Equiangular Tight Frame (ETF) – or Simplex ETF, which is formally defined in Definition C.1 in the Appendix). • Convergence to Self-duality: the last-layer linear classifiers, living in the dual vector space to that of the class-means, are perfectly matched with their class-means. • Simple Decision Rule: the last-layer classifier is behaviorally equivalent to a Nearest Class-Center decision rule.
These results suggest that deep networks are learning maximally separable features between classes, and a max-margin classifier in the last layer upon these learned features, touching the ceiling in terms of the performance. This phenomenon is referred to as Neural Collapse (NC) [1], and it persists across a range of canonical classification problems, on different neural network architectures (e.g., VGG [17], ResNet [18], and DenseNet [19]) and on a variety of standard datasets (such as MNIST [20], CIFAR [21], and ImageNet [22]).
Fully demystifying the NC phenomenon in theory can be very challenging. Perhaps the most difficult hurdle lies in the nonconvexity of the optimization problem for training neural networks, which, loosely speaking, stems from the nonlinear interaction across many different layers of neural networks. Towards this goal, a recent line of work [23–29] studied the properties of last-layer classifiers and features based on the assumption of the so-called unconstrained feature model [23] or layer-peeled model [26]. At a high level, the unconstrained feature model takes a top-down approach to the analysis of deep neural networks [23–26, 29–31], wherein the last-layer features are modeled as free optimization variables (hence we call them unconstrained features) along with the last-layer classifiers (see Figure 2 for an illustration); this is in contrast to the conventional bottomup approach that studies the problem starting from the input [32–42].2 The underlying reasoning is that modern deep networks are often highly overparameterized with the capacity of learning any representations [43–46], so that the last-layer features can approximate, or interpolate, any point in the feature space. In this way, the model simplifies the study of last-layer features, enabling us to analyze the interaction between them and the last-layer classifiers.
Nonetheless, the simplified unconstrained feature model still leaves us a highly nonconvex training loss to be dealt with. Despite the nonconvexity, recent work [23–28] studied the global minimizers, proving that Simplex ETFs (i.e.,NC) are indeed global solutions to the nonconvex loss. In particular, the work [23, 47] studied the training problem with the least-squared loss, proving that the gradient flow converges to NC solutions with extra assumptions. Another line of work [24–27] considered the commonly used cross-entropy loss for classification, showing that the only global minimizers of the loss function are Simplex ETFs with different constraints on the weights and features.3 However,
2Here, top-down means that we study the network starting from the last-layer down to the input layer, whereas bottom-up refers to an approach from the input up to the last layer.
3The constraints on the features are mainly used to prevent it from approaching infinity since the objective function, with the cross-entropy loss, is not coercive. Note that we still refer to this model as an unconstrained feature model even if they include norm constraints or regularization.
these results still suffer from several limitations: (i) due to the nonconvex nature, only characterizing optimality conditions is not enough to explain the empirical convergence of iterative algorithms to NC, such as stochastic gradient descent (SGD); (ii) the problem formulations differ from those typically used in practice, which deploy norm regularization (i.e., weight decay) on the weights, rather than enforcing constraints, for the ease of optimization.4
Contributions of This Work. Inspired by these pioneering results [1, 23–26, 29], in this work we take a step further by characterizing the global optimization landscape of the network training loss based on the unconstrained feature model. Our contributions are summarized as follows. • Benign Global Landscape. For the unconstrained feature model, we provide the first result show-
ing that a commonly used, regularized cross-entropy loss is a strict saddle function [49–51]. In other words, every critical point is either a global solution (corresponding to Simplex ETFs) or a strict saddle point5 with negative curvature, so that there is no spurious local minimizer on the optimization landscape. As summarized in Table 1, this is in contrast to previous work [23–26] that only characterizes global minimizers. • Efficient, Algorithmic Independent, Global Optimization. The benign global landscape implies that any method that can escape strict saddle points (e.g. stochastic gradient descent) converges to a global solution [52] that exhibits NC. This result supports our empirical observation, as shown in Section 4.1, that practical overparameterized networks always converge to Simplex ETF solutions with a diverse choice of optimization algorithms. • Cost Reduction for Practical Network Training. Moreover, the universality of NC implies that there is no need of training the last-layer classifiers since the weights can be simply fixed as a Simplex ETF throughout the training process. On the other hand, since NC happens whenever d ≥ K, this implies that we can choose the feature dimension d comparable to the number of classes K, reducing the feature dimension for further computational benefits. In Section 4.3, our experiments demonstrate that such a strategy achieves on par performance with classical training methods, leading to substantial cost reductions on both memory and computation.
Our results shed new light on the question raised in the recent paper [53] on the role of the optimization strategy (e.g., stochastic gradient descent) for achievingNC in training practical deep networks. This question also relates to the recent highly influential work [7] on the implicit algorithmic bias. For multi-class classification problems with linearly separable data, this work [7] showed that linear predictors optimized by gradient descent converge to the max-margin classifiers even without adding any explicit regularization on the cross-entropy loss. Based on this result, a sequence of works [54–61] laid great emphasis on the notion of “inductive bias” of particular optimization algorithms as a reason for the surprising success in training deep learning models.6 In contrast, both our theoretical result on the global landscape for the unconstrained feature model and the empirical evidence on practical deep models demonstrate that NC in network training is facilitated not only by the choice of the optimization methods, but more importantly, by the choice of loss functions and the power of overparameterization in the network architecture.

heading: 2 The Problem Setup
text: A deep neural network is essentially a nonlinear mapping ψ(·) : RD 7→ RK , which can be modeled by a composition of simple maps: ψ(x) = ψL ◦ · · · ◦ ψ2 ◦ ψ1(x) for x ∈ RD, where ψ`(·) (1 ≤ ` ≤ L) are called “layers”. Each layer is composed of an affine transform, represented by
4A recent concurrent work [48] studied the gradient flow as well as landscape of the cross-entropy loss under the unconstrained feature model, but without using any constraints or regularizers on the weights.
5Throughout the paper, for a minimization problem, we will not distinguish between local maxima and saddle points. We call a critical point strict saddle if the Hessian at this point has at least one negative eigenvalue.
6While (stochastic) gradient descent and generic steepest descent methods can converge to max-margin classifiers, it should be noted that other commonly used optimization algorithms in deep learning, such as AdaGrad [62] and Adam [63], do not have max-margin properties in general and their solutions depend upon initialization, step-size and other algorithm hyper-parameters [64–66].
some weight matrix W`, and bias b`, followed by a simple nonlinear7 activation function σ(·). More precisely, a vanilla L-layer neural network can be written as
ψΘ(x) = WL σ (WL−1 · · ·σ (W1x+ b1) + bL−1)︸ ︷︷ ︸ φθ(x) +bL. (1)
For convenience, we use Θ = {Wk, bk}Lk=1 to denote all the network parameters, and use θ = {Wk, bk}L−1k=1 to denote the network parameters up to the last layer. The output of the penultimate layer, denoted by φθ(x), is usually referred to as the representation or feature of the input x learned from the network. In this way, the function implemented by a neural network classifier can also be expressed as a linear classifier acting upon φθ(x).
The goal of deep learning is to fit the parameters Θ so that the output of the model on an input samples x approximates the corresponding output y, i.e. so that ψΘ(x) ≈ y, in expectation over a distribution of input-output pairs, D. This can be achieved by optimizing an appropriate loss function L(ψΘ(x),y) which quantifies this approximation. In this work, we focus on multi-class classification tasks (say, with K classes), where the class label of a sample x is given by a one-hot vector y ∈ RK representing its membership to one of the K classes. In this setting, cross-entropy is one of the most popular choices for the loss function. Naturally, the distribution D is unknown, but we have access to training samples that are drawn i.i.d. from D. In this way, one can minimize the empirical risk over these samples by optimizing the following problem
min Θ K∑ k=1 nk∑ i=1 LCE (ψΘ(xk,i),yk) + λ 2 ‖Θ‖2F , (2)
where yk ∈ RK is a one-hot vector with only the kth entry equal to unity (1 ≤ k ≤ K), {nk}Kk=1 are the numbers of training samples in each class, and λ > 0 is the regularization parameter (or weight decay penalty), and LCE(·, ·) is the cross-entropy loss. As introduced in Section 1, recent work [1] showed that the features learned by minimizing the above objective showcase theNC phenomenon: their within-class variability vanishes, and the features converge to a Simplex ETF.

heading: 2.1 Problem Formulation Based on Unconstrained Feature Models
text: In deep network models, the nonlinearity and interaction between a large number of layers results in tremendous challenges for analyzing this learning problem. Since modern networks are often highly overparameterized to approximate any continuous function and the characterization of NC only involves the last-layer features φθ(x), a natural idea to simplify the analysis is to treat these features as free optimization variables h = φθ(x) ∈ Rd, which motivates the name unconstrained feature model8 [23] (see Figure 2 for an illustration). In this way, we can rewrite the network output as ψΘ(x) =WLh+ bL.
For simplicity, we consider the setting where the number of training samples in each class is balanced (i.e., nk = n for all k ∈ [K] := {1, 2, . . . ,K}). We also write W = WL and b = bL for conciseness. Based on the unconstrained feature model, we consider a slight variant of (2), given by
min W ,H,b
f(W ,H, b) := 1
Kn K∑ k=1 n∑ i=1 LCE (Whk,i + b,yk) + λW 2 ‖W ‖2F + λH 2 ‖H‖2F + λb 2 ‖b‖22 , (3)
with W ∈ RK×d, H = [h1,1 · · ·hK,n] ∈ Rd×N (here, we denote N = nK), b ∈ RK , and λW , λH , λb > 0 are the penalty parameters for the weight decay.
As summarized in Table 1, similar optimization problems have been considered in [24–26]. In contrast to these, our problem formulation here (3), with bias and weight decay, is closer to the loss used in practice for training neural networks; existing work [24–26] considered constrained9 variants of (3) and without the bias term, which can be implemented but seldom used in practice due to the difficulty of optimization. In the following, we briefly discuss the differences between our simplification and practical settings for training neural networks.
7The nonlinear operator may include activations such as ReLU [67], pooling, and normalization [68], etc. 8This model is also called layer-peeled model in [26], where one “peels” off the first L − 1 layers. It has also been recently studied in [24,25]. Throughout the paper, we will simply call it unconstrained feature model. 9For example, the work [26] considers inequality constraints such that the energy ofW andH are bounded; the other work [24, 25] enforcesW andH on the spheres up to scaling.
• Weight Decay on W and H . One simplification of our formulation is in the weight decay. In practice, weight decay is usually imposed on the network parameters Θ, while we enforce weight decay on the last layer’s classifier, W , and features, H . However, this idealization is reasonable since the energy of the features (i.e., ‖H‖F ) can indeed be upper bounded by the energy of the weights at every layer if the inputs are bounded (which holds in practice), implying that the norm of H is implicitly penalized by penalizing Θ. Our experiments in the Appendix demonstrate that both approaches exhibit similar NC phenomena and comparable performance in practice. • Treating the Last-layer Features as Optimization Variables. One may question that “peeling off” the L − 1 layers might oversimplify the problem. Nonetheless, this simplification (which is also adopted in [23–26]) is based on the fact that neural networks with sufficient overparameterization can approximate any function – in Section 4.2, we numerically demonstrate thatNC persists even when we train overparametrized networks on randomly generated labels. Moreover, as we shall see in the following sections, both our theory and experiments demonstrate that our simplification preserves the core properties of last-layer classifiers and features during training – the NC phenomenon. More specifically, in Section 3 we show that Simplex ETFs are the only global minimizers to our simplified loss function (3), and the loss function is a strict saddle function with no other spurious local minimizers so that it can be optimized efficiently to global optimality.

heading: 3 Main Theoretical Results
text: In this section, we present our study on global optimality conditions as well as the optimization landscape of the nonconvex loss in (3).
Theorem 3.1 (Global Optimality Conditions) Assume that the feature dimension d is no smaller than the number of classes K, i.e. d ≥ K − 1, and the number of training samples in each class is balanced, n = n1 = · · · = nK . Then any global minimizer (W ?,H?, b?) of f in (3) satisfies
w? : = ∥∥w?1∥∥ 2 = ∥∥w?2∥∥ 2 = · · · = ∥∥w?K∥∥ 2 , and b? = b?1,
h?k,i = √ λW λHn w?k, ∀ k ∈ [K], i ∈ [n], and h?i := 1 K K∑ j=1 h?j,i = 0, ∀ i ∈ [n], (4) where either b? = 0 or λb = 0, and the matrix W ?> ∈ Rd×K forms a K-Simplex ETF (defined in Definition C.1) up to some scaling, in the sense that the normalized matrixM := 1w?W ?> satisfies
M>M = K
K − 1
( IK − 1
K 1K1
> K
) . (5)
At a high level, our proof (in Appendix D) finds lower bounds for the loss in (3) and studies the conditions for the lower bounds to be achieved, similar to [24,26]. As can be seen in this result, any global solution of the loss function (3) exhibitsNC in the sense that the variability of output features {h?k,i}ni=1 of each class k (1 ≤ k ≤ K) collapses to zero, and any pair of features (h?k1,i,h ? k2,j
) from different classes k1 6= k2 are maximally separated. Similar results have been obtained in [24–26], which considered different problem formulations, as we have discussed in Section 2.1.
• Relationship between Class Number K and Feature Dimension d. The requirement that d ≥ K − 1 is necessary for Theorem 3.1 to hold, simply because K vectors in Rd cannot form a KSimplex ETF if K > d+ 1. However, the relationship d ≥ K is often true in practice. In general, and in overparameterized models in particular, the feature dimension, d, is significantly larger than the number of classes, K. For example, the dimension of the features of a ResNet [18] is typically set to d = 512 for CIFAR10 [21], a dataset with K = 10 classes. This dimension grows to d = 2048 for ImageNet [22], a dataset with K = 1000 classes.
• Interpretations on the Bias Term b?. In contrast to previous works [24–26], we consider the bias term in the unconstrained feature model (3). Our result indicates that a collapsing phenomenon also exists in the bias term b?, in the sense that all the elements of b? are identical. When the features H are completely unconstrained, our result implies that removing the bias term b has no influence on the performance of the classifier. However, it should be noted that the ReLU unit is often applied at the end of the penultimate layer, so that H should be constrained to be nonnegative,H ≥ 0. In such cases, h?i will no longer be zero, and neither will b?. Here, the bias
term b? will compensate for the global mean of the features, so that the globally-centered features still form a Simplex ETF [1].10
3.1 Characterizations of the Benign Global Landscape for (3)
The global optimality condition in Theorem 3.1 does not necessarily mean that we can achieve these global solutions efficiently, as the problem is still nonconvex. We now investigate the global optimization landscape of (3) by characterizing all of its critical points. Our next result implies that the training loss is a strict saddle function, and every critical point is either a global minimizer or a strict saddle point that can be escaped using negative curvatures. As a consequence, this implies that the global solutions of the training problem in (3) can be efficiently found from random initializations.
Theorem 3.2 (No Spurious Local Minima and Strict Saddle Property) Assume that the feature dimension is larger than the number of classes, d > K, and the number of training samples in each class is balanced n = n1 = · · · = nK . Then the function f(W ,H, b) in (3) is a strict saddle function with no spurious local minimum, in the sense that
• Any local minimizer of (3) is a global minimizer of the form shown in Theorem 3.1.
• Any critical point (W ,H, b) of (3) that is not a local minimizer is a strict saddle with negative curvature, i.e. the Hessian ∇2f(W ,H, b), at this critical point, is non-degenerate and has at least one negative eigenvalue, i.e. ∃ i : λi ( ∇2f(W ,H, b) ) < 0.
In a nutshell, our proof relies on connecting the original nonconvex optimization problem (3) to its corresponding low-rank convex counterpart, so that we can obtain the global optimality conditions for (3) based on the latter. With this, we can then characterize the properties of all critical points based on the optimality conditions. We defer all details of this proof to Appendix D.
Existing results [24–26] have only studied the global minimizers of the original problem, which has limited implication for optimization. In contrast, Theorem 3.2 characterizes the properties for all critical points of the function in (3). As a consequence of this result, many first-order and secondorder optimization methods [69] optimizing (W ,H, b) are guaranteed to converge to a global solution of (3). In particular, the result in [49,52] ensures that (stochastic) gradient descent with random initialization, the de facto optimization algorithm used in deep learning, almost surely escapes strict saddles and converges to a second-order critical point – which happens to be a global minimizer of form showed in Theorem 3.1 for our problem (3). • Constructing the Negative Curvature Direction for Strict Saddles. One of the major difficul-
ties in our proof is to construct the negative curvature direction for strict saddle points. Here, we exploit the fact that the feature dimension d is larger than the number of classes K, and construct the negative curvature direction within the null space ofW ∈ RK×d. This is also the main reason for the requirement d > K in Theorem 3.2, but we conjecture the results also hold for d = K and could be proved with more sophisticated analysis, which is left as future work.
• Relationship to Low-Rank Matrix Recovery. As discussed in Appendix A, it has been recently shown that the strict saddle property holds for a wide range of nonconvex problems in machine learning [70–83], including low-rank matrix recovery [78, 80, 84–87]. As we know that ‖Z‖∗ = minZ=WH 1 2 (‖W ‖ 2 F +‖H‖ 2 F ) (see [32] for a proof), our formulation in (3) is closely related to
low-rank matrix problems [78,80,84–87] with the Burer-Moneirto factorization approach [88], by viewingW andH as two factors of a matrix Z =WH . The differences lie in the loss functions and statistical properties of the problem.11 Thus, our result establishes a connection between the study of low-rank matrix factorization and neural networks under the unconstrained feature model. 10Suppose that an optimal solution to (3) is (W ?,H?, b?), satisfying the conditions in Theorem 3.1. There exists a nonzero vector α ∈ Rd such that H̃? = H? + α1> ≥ 0. Here, α can be viewed as the global mean of H̃? since H? has mean zero. Then, let b̃? = −W ?α, so that W̃ ?H̃? + b̃?1> = W ?H? + (W ?α+ b̃?)1> = W ?H?. Therefore, we can see that (W̃ ?, H̃?, b̃?) achieves the same cross-entropy loss as (W ?,H?, b?).
11We consider the cross-entropy loss rather than the least-squares loss due to the differences in the task – we focus on classification instead of recovery problems. On the other hand, the results on low-rank matrix recovery are often based on certain statistical properties, such as the randomness in the measurements [84, 85], or restricted well-conditionedness property of the objective function [77, 87]. In contrast, these statistical properties do not exist in our problem, where the model and analysis are purely deterministic.
• Comparison to Existing Landscape Analysis on Neural Network. Section 1 provided a comprehensive discussion on the relationship between our result and previous works on landscape analysis for deep neural networks. Although the unconstrained feature model can be viewed as a two-layer linear network with input being the columns of an identity matrix, as preluded in Section 1, our result has much broader implications than the previous results [33,34,37,38,40,41,89]. First, our problem formulation (3) is closer to practical settings for classification tasks, which considers the widely adopted cross-entropy loss while including weight decay and a bias term, while most existing results [33,34,37,38,40,41,89] either do not incorporate any regularization and bias, or focus on the squared loss for the regression problem. More importantly, our result characterizes the precise form of the global solutions (i.e., NC) for the last layer features and classifiers, and shows that they can be efficiently achieved. Moreover, convincing numerical results in [1] and the next section demonstrate that the global solutions do appear and can be achieved by practical networks on various standard image datasets. Our study of last-layer features could have profound implications for studying generalization and robustness of the deep networks.

heading: 4 Experiments
text: In this section, we run extensive experiments not only verifying our theoretical results on modern neural networks, but also demonstrating the potential practical benefits of understanding NC. More specifically, while Theorem 3.2 holds true for the simplified unconstrained feature model, in Section 4.1 we run experiments on practical network architectures and show that our analysis of simplified models captures the gist of NC. In particular, we demonstrate that this depends on the geometry of the problem rather than the algorithmic bias, by showing that different types of optimization algorithms all achieve NC during the terminal phase of training. In Section 4.2, we verify the validity of the simplification based on the unconstrained feature model. Moreover, the universality of NC implies that there is no need for training the last-layer classifiers since the weights can be simply fixed as a Simplex ETF throughout the training process. In Section 4.3, we demonstrate that such a strategy achieves essentially the same generalization performance as classical training algorithms, while improving on memory and computation. We begin by describing the basic experimental setup, including the network architectures, evaluation datasets, training procedures, and metrics for measuring NC.
Setup of Network Architectures, Dataset, and Training. In Section 4.1 and Section 4.2, we train a ResNet18 architecture [18] on CIFAR10 [21] for image classification using the cross-entropy loss (2). Due to limited space, we present all the results on MNIST [90] in the Appendix. As is standard, images are normalized (channel-wise) by their mean and standard deviation. We include no data augmentation in this section, as our focus is to study the behavior associated withNC instead of obtaining state-of-the-art performance. We train the network for 200 epochs with three distinct optimizers: two first-order methods (SGD and Adam) and one second-order method (LBFGS [69]). In particular, we use SGD with momentum 0.9, Adam with β1 = 0.9, β2 = 0.999, and LBFGS with a memory size of 10. The initial learning rates for SGD and Adam are set to 0.05 and 0.001, respectively, and decreased by a factor of 10 for every 40 epochs. For LBFGS, we use an initial learning rate of 0.1 and employ a strong Wolfe line-search strategy for subsequent iterations. Except otherwise specified, the weight decay is set to 5× 10−4 for all the experiments.
Metrics for Measuring NC During Network Training. We measure NC for the learned lastlayer classifiers and features based on the properties presented in Section 1. Some of the metrics are similar to those presented in [1]. We first measure the within-class variability collapse by measuring the magnitude of the between-class covariance ΣB ∈ Rd×d compared to the within-class covariance ΣW ∈ Rd×d of the learned features via NC1 := 1K trace(ΣWΣ † B), where Σ † B denotes the pseudo inverse of ΣB . For the learned classifier W ∈ RK×d, we quantify its closeness to a Simplex ETF up to scaling by NC2 := ∥∥ WW> ‖WW>‖F − 1√ K−1 ( IK − 1K1K1 > K ) ∥∥ F , where we rescale the ETF in (5) so that 1√ K−1 (IK − 1 K1K1 > K) has unit energy (in Frobenius norm). It should be noted that our metric NC2 combines two metrics used in [1] to quantify to what extent the classifier approaches equiangularity and maximal-angle equiangularity. We then measure the duality between the classifiers W and the centered class-means H by NC3 := ∥∥ WH ‖WH‖F − 1√ K−1 (IK − 1 K1K1 > K) ∥∥ F .
In many cases, the global mean hG of the features might not be zero,12 and the bias term b would compensate for the global mean hG. Thus, we capture this collapsing phenomenon by measuring NC4 := ‖b+WhG‖2 . The detailed descriptions of the four metrics are given in Appendix B.

heading: 4.1 The Prevalence of NC Across Different Optimization Algorithms
text: We show different types of training methods (e.g., SGD, Adam, and LBFGS) all achieveNC during the terminal phase of training. Figure 3 shows the evolution of the four metrics NC1,NC2,NC3, and NC4, for measuring NC as training progresses. We consistently observe that all four metrics collapse to zero, trained by different types of algorithms. This implies that NC occurs regardless of the choice of training methods. The last-layer features learned by the network are always maximally linearly separable, and correspondingly the last-layer classifier is a perfect linear classifier for the features. See Appendix for the testing performance of the networks learned by different algorithms.
4.2 The Validity of (3) Based on Unconstrained Feature Models for NC
The premise of our global landscape analysis of (3) for studying NC in deep neural networks is based upon the unconstrained feature model introduced in Section 2.1, which simplifies the network by synthesizing the first L − 1 layers as a universal approximator that generates a simple decision variable for each training sample. Here, we demonstrate through experiments that such a simplification is reasonable for overparameterized networks, in the sense that they are sufficient for characterizing NC in practical network training. In particular, we demonstrate that overparameterization is crucial for NC phenomenon during network training, while the input plays minimal influence. To that goal, we modify the training dataset CIFAR10 by replacing all the correct label for each training sample with a random counterpart.13 We report the corresponding NC behaviors in Figure 4, which shows how training misclassification rate andNC evolve over epochs of training for networks with different widths14. As the network is sufficiently large, it has enough capacity to memorize the training data and achieves zero training error, which is consistent with the observations in [11]. Moreover, we find from Figure 4 that the training accuracy is highly correlated with NC in the sense that a larger network (i.e., larger width) tends to exhibit severe NC and achieves smaller training error. In other words, while the emerging consensus is that the network can interpolate any training data, our results show that such interpolation happens in a particular way – the features are maximally separated, followed by a max-margin linear classifier. In Appendix B, we also report experiments on weight decay imposed on the featuresH , as in (3).
12For example, as discussed after Theorem 3.1 all the feature vectors in H would be nonnegative, because the nonnegative nonlinear operator ReLU has been applied at the end of the penultimate layer.
13We also conducted experiments on completely random dataset where each training image is generated with pixels uniformly from [0, 1] and we observed similar results.
14Here, for ResNet18 we adopt the method in [15] to change its network width.

heading: 4.3 Insights from NC for Improving Network Designs
text: Finally, we conduct exploratory experiments to demonstrate the practical benefits of NC phenomenon. The universality of NC implies that the final classifier (i.e. the L-th layer) of a neural network always converges to a Simplex ETF, which is fully determined up to an arbitrary rotation and happens when K ≤ d. Thus, based on these understandings, we show that we can substantially improve the computational cost by modifying the architecture without the sacrificing performance, by (i) fixing the last-layer classifier as a Simplex ETF15, and (ii) reducing the feature dimension d = K. Here, to demonstrate our method can achieve state-of-the-art performance, we do include data augmentation in the training of our ResNet50 model [91] on CIFAR10, achieving around 95% test accuracy. See Appendix B for the results on MNIST and CIFAR10 with ResNet18.
Fixing the Last-layer Classifier as a Simplex ETF. Figure 5 presents a comparison of learned and fixed classifiers in terms of within-class variation collapse (NC1), self-duality (NC3), training accuracy, and test accuracy. These results imply that the fixed classifier exhibits the same withinclass variation collapse for the features H , and achieves the same classification accuracy as the fully-trained classifier. On the other hand, fixing the classifier can reduce the number of parameters and the computational complexity for training. The number of parameters in the classifier can be significant for tasks with a large number of classes and large feature dimensions. For example, for ImageNet, a dataset with K = 1000 classes, fixing the classifier can reduce 8.01%, 11.76%, and 52.56% of total learning parameters for ResNet50, DenseNet169 [19], and ShuffleNet [92], respectively. We note that our result also provides a theoretical justification for the work in [93] that fixes the classifier as orthonormal matrices. Indeed, these are close to simplex ETFs, particularly when the number of classes is large.
Feature Dimension Reduction for H ∈ Rd×nK by Choosing d = K.16 In many classification problems, the practice of deep learning typically uses a feature dimension d that is much larger than the number of classes K. In contrast, NC implies that there is no need to choose a d that is much larger than the number of classes K. Reducing the dimension d can lead to substantial reductions in memory and computation cost. As shown in Figure 5, we also train all the weights of ResNet50 on CIFAR10 using SGD with d = K. The results demonstrate that NC persists even when we choose d = K, and the network achieves on-par performance with networks of large d, in terms of training and test accuracy. This implies that when the number of classes K is small, we can choose a small feature dimension d = K (or d & K) instead of using a large universal d to reduce the computation and memory costs for training. By setting d = K, this reduces the amount of parameters and hence the memory cost in ResNet18 and ResNet50 by 20.70% and 4.45% respectively.

heading: 5 Conclusion
text: In this work, we have provided an in-depth analysis to demystify the NC phenomenon, which appears during the terminal phase of training deep networks in classification problems. Based on the unconstrained feature model [24–26], we proved that Simplex ETFs are the only global minimizers of the cross-entropy training loss with weight decay and bias. Moreover, we showed that the loss function is a strict saddle function with respect to the last-layer features and classifiers, with no other spurious local minimizers. In contrast to existing landscape analyses for deep neural networks, which mostly focus on the optimization perspective, our simplified analysis not only characterizes
15Specifically, we set W> = √
K K−1P ( IK − 1K 1K1 > K ) where P ∈ Rd×K contains the first K columns
of a d×d identity matrix, which lifts aK×K ETF to d×K matrix. For simplicity, we also learn the bias term in the last layer, though our result indicates it can be set asWh, where h is the global mean of the features.
16Though Theorem 3.2 requires d > K, we conjecture it also holds for d = K as discussed in Section 3.1.
the features that are learned in the last layer, but also explains why they can be efficiently optimized. This provides support for empirical observations in practical deep network architectures. Moreover, the study of last-layer features could have profound implications for optimization, generalization, and robustness of broad interests, which are the subjects of future work. It is also of interest to extend the current study to the case where d < K, which is the case in contrastive learning [94, 95] and many applications, such as recommendation systems [96] and document retrieval [97].

heading: Acknowledgment
text: ZZ acknowledges support from NSF grant CCF 2008460. XL and QQ acknowledge support from NSF grant DMS 2009752. JS acknowledges support from NSF grant CCF 2007649. We would like to thank Qinqing Zheng (Facebook AI Research), Vardan Papyan (U. Toronto), and Felix Yu (Google Research) for timely pointing us to some important references and valuable feedback on the final draft. We thank Christina Baek (UC Berkeley) and Sam Buchanan (Columbia U.) for fruitful discussions during various stages of the work. We also thank Zhexin Wu (U. Michigan) for proofreading and pointing out several typos in the draft, and the four anonymous reviewers for their constructive comments.

Review summaries:
Neural collapse is an intriguing phenomenon that has been observed in neural network classifiers as the training error goes to 0 where the last layer of the neural network has collapsed input points to corners of a simplex, thereby minimizing variability, and maximizing separability. While this phenomenon seems well studied the authors extend it by studying the critical points of this optimization geometrically by relating it to a convex problem. They show in Thrm 3.2 that the critical points are either global minimal or saddle points with negative curvature which can be escaped by a minimizer.  

The paper studies collapse phenomena of representations from neural classification models trained beyond zero training error with cross entropy. The theoretical contribution of the paper is an analysis of such phenomena in the so called unconstrained feature model. In this setting, the authors derive the global loss minimizers, and show that these can be obtained via gradient based optimization, as the critical points of the loss function are benign. These theoretical results are underpinned by experiments, which investigate the collapse phenomena for a ResNet-18 architecture when trained via different optimization algorithms or with randomly labeled data. The theoretical results also indicate, that in practice, the last linear layer of a neural network can be fixed a priory and of low dimension. 

This paper provides a landscape analysis for neural collapse with regularization analysis. The theoretical results are nice, thus I voted an accept decision. But reviewer has a concern about the novelty of the proof and the implications of the theoretical results.

This is an extremely well written (articulate and readable) and diligently prepared (completely by the book) manuscript. For example, it draws carefully on more than 100 other manuscripts. This alone makes the manuscript valuable to other people, as it teaches how work can be exposited and papers can be written.  The results are interesting and instructive, and in some way connect these questions to some recent work in non-convex optimization. The practical benefit of this work -- tens of percentage points speedups -- are meaningful.