Are All Losses Created Equal: A Neural Collapse Perspective
Authors: Jinxin Zhou
Chong You
Xiao Li
Kangning Liu
Sheng Liu
Qing Qu
Zhihui Zhu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep Learning
Neural Collapse
Loss Functions

Abstract: While cross entropy (CE) is the most commonly used loss function to train deep neural networks for classification tasks, many alternative losses have been developed to obtain better empirical performance.  Among them, which one is the best to use is still a mystery, because there seem to be multiple factors affecting the answer, such as properties of the dataset, the choice of network architecture, and so on.  This paper studies the choice of loss function by examining the last-layer features of deep networks, drawing inspiration from a recent line work showing that the global optimal solution of CE and mean-square-error (MSE) losses exhibits a Neural Collapse phenomenon.  That is, for sufficiently large networks trained until convergence, (i) all features of the same class collapse to the corresponding class mean and (ii) the means associated with different classes are in a configuration where their pairwise distances are all equal and maximized.  We extend such results and show through global solution and landscape analyses that a broad family of loss functions including commonly used label smoothing (LS) and focal loss (FL) exhibits Neural Collapse. Hence, all relevant losses (i.e., CE, LS, FL, MSE) produce equivalent features on training data.  In particular, based on the unconstrained feature model assumption, we provide either the global landscape analysis for LS loss or the local landscape analysis for FL loss and show that  the (only!) global minimizers are neural collapse solutions, while all other critical points are strict saddles whose Hessian exhibit negative curvature directions either in the global scope for LS loss or in the local scope for FL loss near the optimal solution.  The experiments further show that Neural Collapse features obtained from all relevant losses (i.e., CE, LS, FL, MSE) lead to largely identical performance on test data as well, provided that the network is sufficiently large and trained until convergence. 

Are All Losses Created Equal: A Neural Collapse Perspective
Jinxin Zhou
Chong You Google Research cyouggoogle. com
Xiao University of Michigan xlxi aodumi ch. edu
Kangning Liu New York University x131410nyu.en
Ohio State University 38206osu edu
Sheng Liu
Qing Qu
Zhihui Zhu"
New York University shengliugnyu. edu
University of Michigan qingquBumich. edu
Ohio State University zhu 34408osu, edu
Abstract
While cross entropy (CE) is the most commonly used loss function to train deep neural networks for classification tasks, many alternative losses have been devel- oped to obtain better empirical performance. Among them, which one is the best use is still a mystery, because there seem be multiple factors affecting the answer, such as properties of the dataset, the choice of network architecture, and on. This paper studies the choice of loss function by examining the last-layer features of deep networks, drawing inspiration from a recent line work showing that the global optimal solution of CE and mean-square-eres eeror (MSE) losses ex- hibits a Neural Collupse (AC) phenomenon. That is, for sufficiently large net- works trained until convergence, (i) all features of the same class collapse the corresponding class mean and (ii) the means associated with different classes are in configuration where their pairwise distances are all equal and maximized. We extend such results and show through global solution and landscape analyses that broad family loss functions including commonly used label smoothing (LS) and focal loss (FL) exhibits NC. Hence, all relevant losses (i.e., CE, LS, FL, MSE) produce equivalent features on training data. particular, based on the unconstrained feature model assumption, we provide either the global land- scape analysis for LS loss or the local landscape analysis for FL loss and show that the (only!) global minimizers are NCC solutions, while all other critical points are strict saddles whose Hessian exhibit negative curvature directions either in the global scope for ISS loss in the local scope for FL. loss near the optimal solution. The experiments further show that NCC features obtained from all relevant losses (i.e., CE, LS, FL, MSE) lead to largely identical performance on test data well, provided that the network is sufficiently large and trained until convergence. The source code is available https: git.ub....... com/ iixinzzou/neen no_ss..
Introduction
Loss function is an indispensable component in the training deep neural networks (DNNs). While cross-entropy (CE) loss is one of the most popular choices for classification tasks, studies over the past few years have suggested many improved versions of CE that bring better empirical perfor- mance. Some notable examples include label smoothing (L.S) [1] where one-hot label is replaced by a smoothed label, focal loss (FL) [2] which puts more emphasis on hard misclassified samples and
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

reduces the relative loss on the already well-classifedd samples, and so on. Aside from CE and its variants, the mean squared error (MSE) loss which was typically used for regression tasks is recently demonstrated to have competitive performance when compared to CE for classification tasks [3].
Despite the existence of many loss functions there is however lack of consensus as to which one is the best to use, and the answer seems to depend on multiple factors such as properties of the dataset, choice of network architecture, and so on [4]. In this work, we aim understand the effect of loss function in classification tasks from the perspective of characterizing the last-layer features and classifier of a DNN trained under different losses. Our study is motivated by sequence of recent work that identify an intriguing Neural Collapse (NC) phenomenon in trained networks, which refers to the following properties of the last- features and classifier:
all features of the same class collapse to the corresponding class mean. (ii) Convergence to Simplex ETF: the means associated with different classes are in a Simplex Equiangular Tight Frame (ETF) configuration where their pairwise distances are all equal and maximized.
(i) Variability Collapse:
(iii) Convergence to Self-duality: the class means are idcally aligned with the last-layer linear classifiers.
(iv) Simple Decision Rule: the last-layer classifier is equivalent to a Nearest Class-Center decision rule.
This NC phenomena is first discovered by Papyan et al. [5,6] under canonical classification prob- lems trained with the CE loss. Following with the CE loss, Han et al. [7] recently reported that DNNs trained with MSE loss for classification problems also exhibit similar NCC phenomena. These results imply that deep networks are essentially learning maximally separable features between classes, and max- marxmn classifier in the last layer upon these learned features. The intriguing empirical obser- vation motivated a surge of theoretical investigation [7-22], mostly under a simplified anconstrained feature model [10] or layer-peeled model [12] that treats the last-layer features of each samples be- fore the final classifier as free optimization variables. Under the simplified unconstrained feature model, it has been proved that the NC solution is the only global optimal solution for the CE and MSE losses which are also proved to have benign global landscape, explaining why the global NCC solution can be obtained.
Contributions. While previous work provide thorough analysis for NC under CE and MSE losses, the theoretical analysis beyond CE and MSE losses is still limited, and their work only focus on one specific loss without a general format. In this paper, we consider broad family of loss functions that includes CE and some other popular loss functions such as LS and FL as special cases. Under the unconstrained feature model, we theoretically demonstrate in Section 3 that the NCC solution the only global optimal solution to the family of loss functions. Moreover, we provide global landscape analysis, showing that the LS loss function is a strict saddle function and FL loss function a local strict saddle function [23-25]. A (local) strict saddle function is function for which every critical point is either a global solution or a strict saddle point with negative curvature (locally). Hence, our result suggests that any optimizer can escape strict saddle points and converge to the global solution responding to .CC for LS and FL. As far we know, this paper is the first work that conducts global optimal solution and benign optimization landscape analysis beyond the scope of CE and MSE losses.
Our theoretical results explained above have important implications for understanding the role of loss function in training DNNs for classification tasks. Because all losses lead to NC solutions, their corresponding features are equivalent up to a rotation of the feature space. In other words, our analysis provides a theoretical justification for the following claim:
All losses (i... CE, LS, FL, MSE) lead largely identical features on training data by large DNNs and sufficiently many iterations.
We also provide experimental verification of this claim through experiments in Section 4.1.
While NCC reveals that all losses are equivalent at training time, does not have a direct implication for the features associated with test data as well as the generalization performance [26]. In partic- ular, a recent work [27] shows empirically that ACC does not occur for the features associated with test data. Nonetheless, we show through empirical evidence that for large DNNs, NC on training

---

data well predicts the test performance. In particular, our empirical study in Section 4.2 shows the following:
All losses (CE, LS, FL, MSE) lead to largely identical performance on test data by large DNNsand sufficiently many iterations.
Our conclusion that all losses are created equal appears to go against existing evidence on the ad- vantages of some losses over the others. Here we emphasize that our conclusion has an important premise, namely the neural network has sufficient approximation power and the training is per- formed for sufficiently many iterations. Hence, our conclusion implies that the better performance with particular choices of loss functions comes as a result that the training does not produce a glob- ally optimal (i.e., NC) solution. In such cases different losses lead to different solutions on the training data, and correspondingly different performance on test data. Such an understanding may provide important practical guidance on what loss to choose in different cases (e.g.. different model sizes and different training time budgets), as well as for the design of new and better losses in the future. We note that our conclusion is based on natural accuracy, rather than model transferability or robustness, which is worth additional efforts to exploit and is left as future work.
2 The Problem Setup
A typical deep neural network V(()) : RD. + R consists multi-layer nonlinear compositional feature mapping (-)) E. + IRd and a linear classifier (W, b), which can be generally expressed as
(e ())(x) +
where we use 0 represent the network parameters in the feature mapping and W Rxxd and b E RK to represent the linear classifier's 's weight and bias, respectively. Therefore, all the network parameters are the set of O {0, W, b}. For the input æ, the output of the feature mapping (o(x) is usually termed as the representation or feature learned from the network.
With an appropriate loss function, the parameters θ of the whole network are optimized to learn the underlying relation from the input sample x to their corresponding target y so that the output of the network V (z) approximates the corresponding target, i.e. (x) 2 y in term of the expectation over a distribution D of input-output data pairs (æ,, y). While it is hard to get access to the ground- truth distribution D in most cases, one can approximate the distribution DD through sampling enough data pairs i.i.d. from D. In this paper, we study the multi-class balanced classification tasks with K class and n samples per class, where we use the one-hot vector yı: R with unity only in k-th entry (1 < k < K) to denote the label of the i-th sample Ø2,6 E IR" in the k-th class. We then learn the parameters O via minimizing the following empirical risk over the total N nK training samples
K ΣD 2 L (Ve(ZE)) yk) + 2
min 6
| l
(2)
N
where A > O is the regularization parameter (a.k.a., the weight decay parameter") and L. ( (V , yk)) is predefined loss function that appropriately measures the difference between the output V and the target yv. Some common loss functions used for training deep neural networks will be specified in the next section.
2.1 Commonly Used Training Losses
In this subsection, we first present four common loss functions for classification task. To simplify the notation, let W W (x) b denote the network's 's output ("logit") vector for the input æ. Assume z belongs to the k-th class. Also let y smooth = (1 a)yx= + 11 denote the smoothed targets of k-th class, where 0 < a < 1 and 1x E R is vector with all entries equal to one. We will use zy, yk.t and y/2 smmooth denote the f-th entry of z, y) and y smootth respectively, where smooth 1 K-1 a and yk yk
smooth a for k LL.
K
K
Without weight decay, the features and classifiers will tend to blow up for CE and many other losses.
3

---

Cross entropy (CE) is perhaps the most common loss for multi-class classification in deep learn- ing. measures the distance between the target distribution ys and the network output distribution
obtained by applying the softmax function resulting in the following expression
exp(z*)
Le yu) = log
(3)
E exp(2,)
Focal loss (FL) [2] is first proposed to deal with the extreme foreground baruuundd--agrouuddd class imbalance in dense object detection, which adaptively focuses less on the well-classiied samples. Recent work [28, 29] reports that focal loss also improves calibration and automatically forms curriculum learning in multi-class classification setting. Letting y 2 denote the tunable focusing parameter, the focal loss can be expressed as:
exp(zx) LFL (2, = - - exp(2))
exp(z*) ΣE= exp(z;)
log
(4)
Label smoothing (LS) [1] replaces the hard targets in CE with smoothed targets y y obtained from mixing the original targets yv with uniform distribution over all entries 1x- Experiments in [30, 31] find that classification models trained with label smoothing have better calibration and generalization. Denoting by 0 a S 1 the tunable smoothing parameter, the label smoothing loss function can be formulated as:
K Σ 2-1
exp(ze)
(zs(2, yle) =
smooth yk
log
(5)
Σj=
j=1 exp(2,)
When a = the above label smoothing loss reduces to the CE loss.
Mean square error (MSE) is often used for regression but not classification task. The recent work [3] shows that classification networks trained with MSE loss achieve on par performance compared
to those trained with the CE loss. Throughout our paper, we use the rescaled MSE version [3]:
K
(2, yv) = x(zx - 3)² +
(6)
where x > 0 and | > O are hyperparameters.
2.2 Problem Formulation Based on Unconstrained Feature Models
Because of the interaction between a large number of nonlinear layers in the feature mapping Pa- it is tremendously challenging analyze the optimization of deep neural networks. To simplify the difficulty of deep neural network analysis. a series of recent works of theoretically studying NC phenomenon use so-called unconstrained feature model (or layer-peeled model [12]) which treats the last-layer features as free optimization variables h = (xx) E Rd The reason behind the unconstrained feature model is that modern highly overparameterized deep networks are able to approximate any continuous functions [32-35] and the characterization of NC are only related with the last layer features. We adopt the same approach and study the effects of different training losses on the last-layer representations of the network under the unconstrained feature model, For convenient, let us denote
W = fw'
2²
w K T
E Rxd
H H
H2 H E RdNN, and
Y YY
Y2
Yk) ERKN
where = is the k-tth vector of W, all the features in the k-th class are denoted H; (hi hg.)) E R and h is the feature of the i-th sample in the k-th class, and Y,- yyk y&] E Rxx" for all & feature model, consider slight variant of (2), given by
K and é = 1,2, ... n. Based on the unconstrained
K
min J(W, m H.O
H)
1
ΣΣ L (Wh + UR) +
Aw
W
2 +
HY 2
N |
|
N
2
2
7

---

where Aw, λH, A > O are the penalty parameters for W. H, and b, respectively.
By viewing the last-layer feature H free optimization variable, the simplified objective function (7) consider the weight decay about W and H, which is slightly different from practice that the weight decay is imposed all the network parameters @ as shown in (2). Nonetheless, the un- derlying rationale is that the weight decay on O implicitly penalizes the energy of the features (i.e., || [16].
As NC phenomena for the learned features and classifiers is first discovered for neural networks trained with the CE loss [5], the CE loss has been mostly studied through the above simplified unconstrained feature model [8, 9, 11, 12, 16] to understand the NCC phenomena. The work [7, 10, 14, 22] also studied the MSE loss, but the analysis there shows the solutions of the learned features and classifiers depend crucially on the bias term, while for CE loss with or without the bias term have no effect on the learned features and classifiers under the unconstrained feature model. The other losses such as focal loss and label smoothing have been less studied, though they are widely employed in practice obtain better performance. This will be the subject of next section.
3 Understanding Loss Functions Through Unconstrained Features Model
In this section, we study the effect of different loss functions through the unconstrained features model. We will first present contrastive property for general loss function LGL. in Definition 1. We will then study the global optimality conditions in terms of the learned features and classifiers as well as geometric properties for (7) with such a general loss function LGL.
3.1 A Contrastive Property for the Loss Functions
In this paper, we aim to provide unified analysis for different loss functions. Towards that goal, we first present some common properties behind the CE, FL and FL to motivate the discussion. Taking CE as example, we can lower bound it by
K-1 Eitales -2x) 2x) =
)) ÓCE
Loe(2,yN) 2 log [1+ (K - ex
Σ 2x) jotk
(8)
xー and the inequality achieves equality when 2j 2y for all j.J # k. This requirement is reasonable because the commonly used losses treat all the outputs except for the k-th output z identically. Since ΦCR is an increasing function, minimizing the CE loss LCE(Z, y&) is equivalent maximizing (K" 1)z
where (te(t) log [1+ (K
exp exp
Ljtk Zj. which contrasts the
k-th output 2 simultaneously all the other outputs 2j for all * k. Thus, contrastive property. Maximizing (K- - 1)2k - Ejj 2j would lead to a positive (and relatively large) and negative (and relatively small) 2j. In particular, within the unit sphere ||=||, = 1,
call (8)
(K- 1)2k EjØK žj achieves its maximizer when 2 = V
K-1 and 2j
K(K 1j for
all * h, which satisfies the requirement 2j zy for all j.J *k. Thus, 2k
K-1 and
2j K(K-T is also the global minimizer for OCR within the unit sphere ||=||, 1. As the global minimizer is unique for each class, it encourages intra-class compactness. On the other hand, the minimizers to different classes are maximally distant, promoting inter-class separability.
Motivated by the above discussion, we now introduce the following properties for a general loss function LeL (z, yv).
Definition 1 (Contrastive property). We say a loss function Lor(z yu) satisfies the contrastive property if there exists a function o such that (z,(Z, yv) can be lower bounded by
CGL 4k) ) 2 Φ
Σ 2x)
j/h
where the equality holds only when 2, = 2/1 for all j.3 # k. Moreover, e(t) satisfies 1" = arg min @ (1) + c/1l is unique for any e > 0, and 1" < 0.
(10)
5

---

In the appendix, we show that CE, FL and LS all satisfy this property. The motivation for (9) follows from the above discussion. In particular, (9) achieves equality when all the outputs except for the k-th one are identical, which holds for common loss functions since those outputs are treated identically. In (10), e is constant related with the weight decay penalty parameters. By (9), we can find the global minimizer for (ZL(Z, yk.) by minimizing the right hand side since the equality in (9) is achievable. Thus, the requirement of a unique minimizer (10) ensures unique minimizer for the regularized (z, y.). This condition can be easily satisfied. For example, ΦCE (t) defined in (8) for the CE loss an increasing and strictly convex function and thus has unique minimizer for (Cn(() + clt]. Along the same line, we require a negative minimizer (* to ensure that the minimizer for the regularized (z, yk) has k-th entry being its largest entry, which is required ensure correct prediction since the largest entry predicts the class membership. Therefore, such a condition generally satisfied by the common losses. For example, ΦCE(t) is an increasing function and thus must have a non-positive minimizer for ΦCB(t) + c/tl. Finally, we note that the MSE loss is not included since it has different form than others and thus the analysis will be different. But as mentioned above, the MSE loss has been studied in [7, 10, 14, 22].
3.2 Landscape Analysis for the Unconstrained Features Model
We now study the global optimality conditions in terms of the learned features and classifiers as well geometric properties for the training problem (7) with the general loss function CGL. satisfying the above contrastive property.
Theorem (Global Optimality Condition). Assume that the number of classes K is smaller than the feature dimension d, i.e., K < d, and the dataset is balanced for each class, n n. Then any global minimizer (W*, , b²) of f in (7) with a loss function LGL satisfying the contrastive property in Definition I has following properties:
| ||2 = //=111
= || 11 and b" b'1,
=
K O, Vie [n),
dw
hi
V
w XmM
[ i E [n]. and h
h
K
where either b" = 1 λb = O. and the matrix W is in form of K K-simplex ETF structure (see appendix for the formal definition) in the sense that
W T T' = =1100112
K K-1 Ix
K 1 (111))
Its proof is given in Appendix C. At a high level, we lower bound the general loss function based on the contrastive property (9) and then check the equality conditions hold for the lower bounds. While similar strategy has been used for CE loss [12, 13, 16], our proof is different from previous work in terms of dealing with the nuclear norm and checking the structures of the minimizer per sample and class, enabling the global optimality analysis for general loss functions. Theorem implies that for all the loss functions (e.g.. CE, LS, and FL) satisfying the contrastive property, they share similar global solutions with NC property in the learned features and classifiers.
While Theorem 1 shows that NC features and classifiers are the only global minimizers to (7), it not obvious whether local search algorithms (such as gradient descent) can efficiently find these benign global solutions. The reason is that the training problem (7) is nonconvex due to the bilinear form between W and H. To address this challenge, we use the recent advances on the geometric analysis for nonconvex optimization [23-25, 233 to guarantee that the global solutions of (7) can be efficiently achieved by iterative algorithms. Towards that goal, we first present the following general results concerning the global landscape for (7).
Theorem 2 (Benign Landscape). Assume that the feature dimension d is larger than the number of K. Also assume L(z, y) is a convex function in terms of z. Then the objective function f in (7) is a strict saddle function with no spurious local minimum. That is, any of its critical point is either a global minimizer, or it is striet saddle point whose Hessian has a strictly negative eigenvalue.
classes K, i.e,,

---

This result is similar to [16, Theorem 3.2] which studies the particular CE loss. Though the result in [16] is about the CE loss, we checked its proof and only uses convexity and smoothness and thus the result can be applied more generally for any smooth convex loss function C(,, yy) So we omit the proof of Theorem 2. We note that the geometric analysis is also closed related to nonconvex low-rank matrix problems [37-43] with the Burer-Moneirto factorization approach [44] if one views W and H as two factors of a matrix Z = W.. We refer to [16] for more discussions about the connections and differences.
We now exploit Theorem 2 for the label smoothing and focal loss. In the supplementary material, we show that LS is convex function. Thus, the following result establishes global optimization landscape for the training problem (7) with such a loss.
Corollary (Benign Landscape with LS). Assume that the feature dimension d is larger than the mumber of classes K, i.e., d >> K. Then the objective function f in (7) with LS loss LLs is a strict saddle function with no spurious local minimum.
Unlike LS, focal loss (FL(z, is convex only in a local region rather than the entire space. For example, we can show that (FL(2, is convex within the region Q = {= E RK exp(<x)/ / exp(z,) 2 0.213. The set Q contains a relative large region including the global minimizer which has the value exp(ee)
Σ/=n exp(2;) approaching 1. Thus, we obtain a benign local land-
scape for the training problem (7) with FL.
Corollary 2 (Benign Landscape with FL). Assume that the feature dimension d is larger than the number of classes K, i.e., d >> K. Then the objective function in (7) with FL loss LF1, has benign local landscape: J is a strict saddle function with no spurious local minimum within the region {(W, b) Whe, + E Q, 1 1 S 1i n).
While Corollary 2 only provides a local benign landscape for the FL, we observe from experiments that gradient descent with random initialization always converges to a global solution with NC properties for (7). So expect the training problem in (7) with FL loss has benign landscape in much larger region. One direction is to show CFL(. (., yk) is locally convex in a much larger region Ω but we leave the thorough investigation to future work. Noting that the CE and MSE losses are also convex, these results imply that (stochastic) gradient descent with random initialization [23,36] almost surely finds the global solutions of the training problem in (7) with different training losses. This together with Theorem 1 implies that for different losses, gradient descent will always learn similar features and thassifiers-those that exhibit the A'C phenomenon.
Experiments
We conduct experiments with practical network architectures on standard image classification datasets to study the effect of different loss functions. First, Section 4.1 provides results to show that the NCC phenomena are not restricted to networks trained via the CE and MSE losses. Rather, there is a family of loss functions, and for the purpose of illustration we pick FL and LS as two prominent special cases, that exhibit the same NC phenomena. Such results verify our theoretical results in Section 3. To demonstrate the implication of NC for test performance, we present exper- imental results in Section 4.2 with a varying number of training iterations and a varying width of networks, showing that all losses with ACC global optimality have similar performance on the test dataset when the network is sufficiently large and trained long enough.
Before presenting the experiment results, we first introduce our experimental setup, including datasets, network architectures. training procedure, and metrics for measuring NC.
Setup of Loss Function, Network Architecture, Dataset, and Training We focus the CE, FL, LS and MSE loss functions for which we use y = for FL, a = 0.1 for LS, and x = 1 and 3 = 15 for MSE, except otherwise specificed. We train Net50 network [45] CIFAR10 and CIFAR100 100 datasets [46] and a WideResNet18 network on minilmageNet [47] with various widths and number of iterations for image classification using these four different losses.? To examine the
'Similar results are expected on other architectures and dataset as NC is observed across a range of archi- tectures and dataset in [5].
7

---

effect of model size, we experiment with four versions of WideResNet, denoted as WideResNet-A. X,, where E {0.25, 0.5, 1, 2) a multiplier on the width its corresponding standard WideResNet. Due to the page limit, we put all results on CIFAR 100 and minilmageNet in the Appendix. We use standard preprocessing such that images are normalized (channel-wise) by their mean and standard deviation, well as standard data augmentation. For optimization, we use SGD with momentum 0.9 and an initial learning rate 0.1 decayed by a factor of 0.1 at & and & of the total number of iterations. Following [28]. the norm of gradient is clipped at 2 which can improve performance for all losses. For CIFARIO and minilmageNet, the weight decay set to x 10 for configurations with all losses. For CIFARIO0 100, the weight decay is fine-tuned to achieve best accuracy for every configuration and loss.
Three NC Metries NC11-CC3 during Network Training We use the same three metrics NC1- NC3 for the last-layer features and classifier as [5, 16, to measure the first three NC properties in Section 1. Before describe these three metrics, let denote the global mean fg and k-th class mean he of last-layer features (hr as
n hkgi h Σ has (ISks K).
hg
nk
Within-class variability collapse is measured by NC1 which depicts the relative magnitude of nk Ek Eimi (hr,: h2) (h2 - -hx) E IRdxd w.r.t. the
the within-class covariance Σw I
between-class covariance EB R E (hs- -ha) (hx- hc) E Rddd of the last-layer fea- tures as following:
NCI = K
trace Ew
where Σ/ is the pseudo inverse of E.
Convergence to simplex ETF is measured by AC2 which reflects the & distance between the normalized simplex ETF and the normalized W W as following:
WWT
+
K k
NC2
IK
WW
1F
VK
where W E Kxd is the weight matrix of learned classifier.
Convergence to self-duality is measured by NCcss which calculates the lg distance between the normalized simplex ETF and the normalized W H as following:
NC,-
==
WH
Ik
K lx1k)
W H
VK
where H
h1 hG
h ha] E Raxk is the centered class-mean matrix.
4.1 Prevalence of NC Across Varying Training Losses
We show that all loss functions lead to NC solutions during the terminal phase of training. The results on CIFAR10 10 using 5e15-2 and different loss functions is provided in Figure 1. We consistently observe that all three ACC metrics across different losses converge to a small value as training progresses. This supports learned under different losses are always maximally linearly separable and perfectly aligned with the linear classifier, and the features and the weight of linear classifier learned by different losses are almost equivalent up to rotation and scale of the feature space. The evolution of three NCC metrics across different losses CIFAR100 is in Appendix A.2.
theoretical results in Section 3 that the last-layer features
4.2
All Losses Lead to Largely Identical Performance
We show that all loss functions have largely identical performance once the training procedure con- verges to the ACC global optimality. In Figure 2, we plot the evolution of the training accuracy,

---

2.0
1.2
CF
CF
CF
1.0
1.0
L5
1.5
MSE
0.8
MSE
0.8
MSE
1.0
0.6
0.5
www
0.4
0.4
0.2
0.7
0.0
400
600
0.0
200
400
600
800
0.0
200
400
600
800
Epoch
Epoch
Epoch
N's (CIFARIO)
(b) N/C2 (CIFARIO)
(6) NC (CIFARIO)
Figure 1: The evolution of NC metrics across different loss functions. We train the WideResNee50-2 on CIFAR10 10 dataset for 800 epochs using different loss functions. From left to right: NCc, (variability collapse), NC (convergence simplex ETF) and NCCa (convergence self-duality).
100
100
100
Training accuracy
Testing accuracy
Val accuracy
90
90
00
Ch
L5
70
L5
MSE
MSE
MSE
200
600
6%
200
600
800
6%
800
Epoch
Epoch
Epoch
(a) Train Ace (CIFARIO)
(b) Acc (CIFARIO)
(c) Test Ace (CIFARIO)
Figure 2: The evolution of performance across different loss functions. We train the WideResNee50022 on CIFAR10 10 dataset for 800 epochs using different loss functions. From left to right: training accuracy, validation accuracy and test accuracy.
validation accuracy and test accuracy as training progresses, where all losses are optimized on the same WideResNet50-2 architecture and CIFARIO for 800 epochs. To reduce the randommess, we average the results from 3 different random seeds per width-iteration configuration, and the test ac- curacy is reported based on the model with best accuracy on validation set, where we organize validation set by holding out 10 percent data from the training set. The results consistently show that for all cases the training accuracy converges to one hundred percent (reaching to terminal phase), and the validation accuracy and test accuracy are largely the same, as long as the network is trained longer enough and converges to the NCC global solution.
While previous work advocates the advantage of some losses over other others, our experiments show that when conditions between dataset and model allow for SGD find an ACC solution, all losses we tested produced indistinguishable results. In Figure 3, we plot the average test accuracy of different losses under different pairs of width and iterations. We consistently observe three phe- nomenon. First, with fixed number of iterations, increasing the width of network improves the test accuracy for all losses. This because the wider networks (more over-parameterized) are more powerful to fit the underlying mapping from input data to the targets. Second, with a fixed width of network, increasing the number of iterations improves the test accuracy for all losses. This because the longer optimization leads the last-layer features and the linear classifer closer to the NC global solutions. Finally, while there are some unignorable difference between different losses in some width-iteration contigurations, the results consistently show that all losses lead to largely identical performance when the network is sufficiently large and trained long enough to achieve a global NC solution (e.g. width=2 and epochs=800).
5 Conclusion
In this work we provided a theoretical study to extend the scope of NC, curious phenomenon associated with last-layer features and classifier weight classification network, from networks trained with particular losses (i.e., CE and MSE) to those trained via a broad family of loss functions including the popular LS and FL as special cases. Our theory not only establishes NCC as the only global solutions, but also shows benign optimization landscape that explains why NCC. solutions

---

best AcCce
test ACCISE
test AcCr
test AcciE
800
94.477 94.863
05.523
95.097
800
98:013
04 400
04 670
95.190
800
94.079
94.850
94.993
05.277
800
94.485
04 04.807
04 34:053
05.427
Egochs
400
94.160
94.897
95 260
95, 303
Egocho
400
98 08.763
94 94.410
94 570 94.960
Egochs
400
93.973
175 94 727 95.060
Egoche
400
94
24.290
94. 780
95 017 95.259
200
98.567
94.297
94 860 95.047
200
98 163
03:839
94 167 04 93.537
200
92.063
03.887
94 040
94.238
200
93.733
94 403
94 600 05,057
100 92.517
93.363
98.707
94.237
100
92 09.090
92
02.630
93 240 93.293
100
91.499
92.250
92.827
93.093
100
92.463
93
03.230
03 03:683
04. 06.260
0.25
0.5
0.25
0.5
2
0.25
0.5
0.25
0.5
2
Width
Width
Width
Width
(CE)
(b) Test (MSE)
(c) (FL)
(d) Text (LS)
Figure 3:
Illustration of test accuracy across different iterations width settings. The figure depicts the test accuracy of various iteration-width configurations for different loss functions on CIFARIO.
are easy to obtain in practice. Such results readily suggest that all relevant losses (i.e., CE, MSE, L.S, and FL.) produce entirely equivalent features on the training data. Although NC is an optimization phenomenon pertaining to training data only, we found through experiments that all relevant losses (i.e., CE, MSE, LS, and FL) lead to very similar test performance well. Such result may come as a surprise to the common belief that some losses are intrinsically better than the others, and clarify some mystery on how different losses affect the performance.
The family of loss functions considered in this paper by no means inclusive of all possible loss functions that lead to NC. There are many other popular loss functions, such as center loss [48], trgge-marinn softmax (L-Softmax) loss [49] and many of its variants [50-52], which are all de- signed with the intuition of encouraging intra-class compactness and inter-class separability between learned features. In addition, many generalized versions of the cross-entropy loss such as those for robust learning under label noise [53-55] and long-tail distribution [56,57] may have similar prop- erty as the vanilla cross-entropy loss. We conjecture that many of them provably produce NC solu- tions under unconstrained feature models, while leave formal justification future work. Beyond losses for classification task, NC may also arise with popular losses used metric learning [58, 59] evidenced by recent study [60]. This means that the observations from this paper, namely all losses lead to largely the same test performance, may apply for all such losses as well.
Loss functions that do not lead to NC. While the study in this paper covers many of the most commonly used loss functions for classification tasks, we note that there are alternative choices the literature which do not induce NC features. Many of such losses such as [60-63] are particularly designed to discourage variability collapse and learn diverse features, which are shown to benefit model transferability [64] and robustness.
Acknowledgment
JZ. and 777 acknowledge support from NSF grants CCF-2240708 and CCF-2241298. XL and QQ acknowledge support from U-M START & PODS grants, NSF CAREER CCF 2143904, NSF CCF 2212066, NSF CCF 2212326, and ONR N00014-22-1-2522 grants. SI acknowledges support from NSF NRT-HDR Award 1922658 and Alzheimer's Association grant AARG- NTF-21-848627.
References
[1] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 2818-2826, 2016.
[2] Tsung- Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[3] Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks. arXiv preprint arXiv:2006.073:: 073322, 2020.
[4] Katarzyna Janocha and Wojciech Marian Czarnecki. On loss functions for deep neural networks in clas- sification. Schedae Informaticae, 25:49-59, 2016.
10

---

[5] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 1 117740):24652-2466 2020.
[6] Vardan Papyan. Traces of class/cross- class structure pervade deep learning spectra. Journal of Machine Learning Research, 21(252):1 64. 2020.
[7] XY Han, Vardan Papyan, and David I. Donoho. Neural collapse under mse loss: Proximity to and dynamies on the central path. arXiv preprint arXiv:2106.0221 02073, 2021.
[8] Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss. arXiv preprint arXiv:2012.084 08465, 2020.
[9] E. Weinan and Stephan Wojtowytsch. On the emergence of tetrahedral symmetry in the final and penulti- mate layers of neural network classifiers. arXiv preprint arXiv:20/2.054200 2020.
[10] Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint 1. 1669, 2020.
[11] Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised constrastive learning. In International Conference on Machine Learning, pages 3821-3830. PMLR, 2021.
[12] Cong Fang, Hangfeng Hc, Qi Long, and Weijie J Su. Layer-peeled model: Toward understanding well- trained deep neural networks. arXiv e-prints. pages arXiv-2101, 2021.
[13] Wenlong Ji, Yiping Lu, Yiliang Zhang. Zhun Deng, and Weijie J Su. An unconstrained layer-peeled perspective on neural collapse. arXiv preprint arxiv:21 2021.
[14] Tom Tirer and Joan Bruna. Extended unconstrained fcatures model for exploring deep neural collapse. arXiv preprint wxxiv:202.08000 2022.
[15] Tolga Ergen and Mert Pilanci. Revealing the structure of deep neural networks via convex duality. In International Conference on Machine Learning, pages 3004-3014, PMLR, 2021.
[16] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li. Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34, 2021.
[17] Akshay Rangamani and Andrzej Banburski-Fahey Neural collapse in deep homogeneous classifiers and the role of weight decay. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4243 4244. IEEE, 2022.
[18] Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint arXiv:2101 1000000,, 2020.
[19] Tomaso Poggio and Qianli Liao. Implicit dynamic regularization in deep networks. Technical report, Center for Brains, Minds and Machines (CBMM), 2020.
[20] Akshay Rangamani, Mengjia Xu, Andrzej Banburski, Qianli Liao, and Tomaso Poggio. Dynamics and neural collapse in deep classifiers trained with the square loss. Technical report, Center Brains, Minds and Machines (CBMM), 2021.
[21] Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Neural collapse in deep homoge- neous claaifiers and the role of weight decay. In International Conference on Machine Learning, pages 3821-3830. PMLR, 2021.
[22] Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You. Qing Qu, and Zhihui Zhu. On the optimization land- scape of neural collapse under mse loss: Global optimality with unconstrained features. arXiv preprint arXiv:2203.0/23: 2022.
[23] Rong Ge. Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points-online stochastic gradi- ent for tensor decomposition. In Proceedings of The 28th Conference on Leurning Theory.pages 797-842, 2015.
[24] Ju Sun, Qing Qu, and John Wright. When are nonconvex problems not scary? arXiv preprint arXiv:1510.06099 2015.
[25] Yuqian Zhang, Qing Qu, and John Wright. From symmetry geometry: Tractable nonconvex problems. arXiv preprint arXiv:2007.06753 2020.
[26] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications the ACM, 64(3):17- -115,, 2021.
[27] Like Hui, Mikhail Belkin, and Prectum Nakkiran. Limitations of neural collapse for understanding gen- eralization in deep learning. arXiv preprint arXiv:2202.08334 2022.
[28] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puncet Dokania. Calibrating deep neural networks using local loss. Advances in Neural Information Processing Systems, 33:15288-1525 15299, 2020.
[29] Leslie N Smith. Cyclical focal loss. arXiv preprint
2022.
11

---

[30] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
[31] Blair Chen, Liu Ziyin, Zihao Wang, and Paul Pu Liang. An investigation of how label smoothing affects generalization. arXiv preprint arXiv:2010. 12648, 2020.
[32] G Cyhenko. Approximation by superposition of sigmoidal functions. Mathematics of Control, Signals and Systems, 2(4):303-314, 1989.
[33] Kurt Hornik. Approximation capabilittes of multilayer feedforward networks. Newral networks, 4(2):251-
257, 1991.
[34] Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neu- ral networks: view from the width. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6232-6240, 2017.
[35] Uri Shaham, Alexander Cloninger, and Ronald Coifman. Provable approximation properties for deep neural networks. Applied and Computational Harmonic Analysis, 44(3):537-555 557, 2018.
[36] Jason D Lee, Max Simchowitz, Michael I Jordan, and Benjamin Recht, Gradient descent only converges to minimizers. In Conference on learning theory, pages 1246-1257. PMI.R 2016.
[37] Benjamin D Haeffele and René Vidal. Global optimality in tensor factorization, deep learning, and be- yond. arXiv preprint arXiv: 07540, 2015.
[38] Rong Ge, Jason D Lee, and Tengyu Ma. Matrix completion has no spurious local minimum. arXiv preprint 160i 07272. 2016.
[39] Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Global optimality of local scarch for low rank matrix recovery. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 3880-3888, 2016.
[40] Rong Ge, Chi Jin, and Yi Zheng. No spurious local minima in nonconvex low rank problems: A unified geometric analysis. In International Conference Machine Learning, pages 1233 1242. PMLR, 2017.
[41] Qiuwei Li, Zhihui Zhu, and Gongguo Tang. The non-convex geometry low-rank matrix optimization. Information and Inference: A Journal of the IMA, 8(1):51-96, 2019.
[42] Xingguo Li, Junwei Lu, Raman Arora, Jarvis Haupt, Han Liu, Zhaoran Wang, and Tuo Zhao. Symmetry, saddle points, and global optimization landscape of nonconvex matrix factorization. IEEE Transactions on Information Theory, 65(6):3489 3514, 2019.
[43] Yuejie Chi, Yue M Lu, and Yuxin Chen. Nonconvex optimization meets low-rank matrix factorization: An overview. IEEE Transactions Signal Processing, 67(20):523-5526) 2019.
[44] Samuel Burer and Renato DC Monteiro. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, 95(2):329-357, 2003.
[45] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[46] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[47] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
[48] Yandong Wen, Kaipeng Zhang, Zhifeng L.,, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European conference on computer vision, pages 499-515. Springer, 2016.
[49] Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In ICML, volume 2. page 7, 2016.
[50] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj. and Le Song. Sphereface: Deep hyper- sphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 212-220, 2017.
[51] Hao Wang. Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5265-5274, 2018.
[52] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pages 4690-4699, 2019.
[53] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
[54] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 322-330, 2019.
12

---

[55] Jiaheng Wei and Yang Liu. When optimizing f-divergence is robust with label noise. In International Conference Learning Representations, 2021.
[56] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and San- jiv Kumar. Long-tail tail learning via logit adjustment. In International Conference on Learning Representa- tions, 2021.
[57] Wittawat Jitkrittum, Aditya Krishna Menon, Ankit Singh Rawat, and Sanjiv Kumar. Elm: Embedding and logit margins for long-tail learning. arXiv preprint arXiv:2204. 13208, 2022.
[58] Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, 11(3), 2010.
[59] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Advances in neural information proessing systems, pages 1857-1865, 2016.
[60] Elad Levi, Tete Xiao, Xiaolong Wang, and Trevor Darrell. Reducing class collapse in metric lcarning with easy positive sampling. arXiv preprint arXiv:2006.05/62 2020.
[61] Zhang. Felix Yu, Sanjiv Kumar, and Shih-Fu Chang. Learning spread-out local feature descriptors. In Proceedings of the IEEE international conference on computer vision, pages 4595-4603, 2017.
[62] Qi Qian, Lei Shang, Baigui Sun, Juhua Hu, Hao Li, and Rong Jin. Softtriple loss: Deep metrie learning without triplet sampling. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6450-6458, 2019.
[63] Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via principle of maximal coding rate reduction. arXiv preprint arXiv:2006. 08558, 2020,
[64] Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? Advances in Neural Information Processing Systems, 34, 2021.
[65] Jan JM Cuppen. A divide and conquer method for the symmetric tridiagonal cigenproblem. Numerische Mathematik, 36(2):177-195, 1980.
[66] N Jakovčević Stor, Slapničar, and Jesse L Barlow, Forward stable eigenvalue decomposition of rank-one modifications of diagonal matrices. Linear Algebra and its Applications, 487:301-315, 2015.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In the abstract, introduction,
main results, and conclusion, we explicitly stat that our results are about unconstrained feature model.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This paper mainly focuses on understanding the neural collapse phenomena observed in practical neural networks. Based on this understanding, we propose to fix the last layer classifier as a Simplex ETF. So no potential negative societal impact is expected of this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We explicitly mention the assumptions in Theorem and Theorem 2.
(b) Did you include complete proofs of all theoretical results? [Yes] We include all the proofs in the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes) See Section 4
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? ? [Yes] All the results obtained by averaging the resluts from 3 different random seeds. See Section 4
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.
4. If you using existing assets (c.g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite them in Appendix
(b) Did you mention the license of the assets? [Yes] This is mentioned Appendix A.
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'te re using/curating? [Yes] This is discussed in Appendix A.
(e) Did you discuss whether the data you using/curating contains personally identi- fiable information or offensive content? [N/A] The CIFAR datasets do not contain personally identifiable information or offensive content
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] Our research does not involve with participants and screenshots (b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] Our research does not include any poten- tial participant risks, with links to Institutional Review Board (IRB) approvals
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] Our work does require participants.
14


Reviews:
[{"review_id": "ysKH21XeM_o", "reviewer": "Reviewer_VAGF", "summary": "This paper did global solution and landscape analyses and extend the neural collapse phenomenon (on training data) to a broad family of loss functions, focusing on the less-studied label smoothing and focal loss.\n\nBy landscape analyses, they show the global minimizers are neural collapse solutions. Also, empirically they show NC features obtained from all relevant losses (till convergence) lead to largely identical test performance.\n\n", "questions": "The context of the paper is clear and no many questions.\n\nMain concern is about the claim on test data performance mentioned above, which I think is not well-supported.\nOne question relates to this:\n\n  How to make this claim convincing? Other than the empirical evidence, could you provide more rigorous theory results on a simplified setting?", "limitations": "The claim on \"largely identical performance on test data\" is not convincing. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n  1) The paper looks at different losses from the neural collapse perspective and conclude that all losses can be equal (at convergence) as they produce equivalent features on training data. This perspective is interesting and the conclusion is strong.\n\n  2) The writing is clear and it is easy to follow the ideas.\n\nWeaknesses:\n  The claim on the \"largely identical performance on test data\" of different losses is not convincing, as it only comes from experimental results on CIFAR-10 and CIFAR-100. Also, on CIFAR-100 (Figure 6 in Appendix), the difference between MSE (78.0%) and FL (79.1%) is 1.1%, which is not negligible.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Foh-cxnwamD", "reviewer": "Reviewer_8maL", "summary": "This paper mainly focuses on the choice of the loss function in the training of neural networks. Specifically, at the end of the convergence of the training process, all the features collapse to the class mean, and the pairwise distances of different classes are maximized and equal. Moreover, the author proposes a new insight that different losses can generate similar neural collapse features when the network is sufficiently large and converged.", "questions": "1. I wonder if the $\\mathcal{NC}$  phenomenon is more common in other datasets and tasks or some more classic model architectures like MLP.\n2. Is the regularization term (weight decay) required to give rise to such a phenomenon? Ablation studies are required to eliminate the effect of the regularization term. A net loss function is better to verify the statements in the paper.", "limitations": "1. The datasets and models are limited to support all the claims. More models and datasets are required.\n2. There are no further statements linking the losses from the $\\mathcal{NC}$ perspective and the real training scenario. In other words, given that all the losses are the same from the view of $\\mathcal {NC} $, how can we guide our training in a real situation? How can we choose the loss function?", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\nThis paper experimentally shows an interesting phenomenon that different losses are the same if the network is sufficiently trained until convergence. The model can learn the same neural collapse features with different losses.\n\n[Weakneses]\n1. The paper discusses an over-restricted situation in which all models can learn well towards convergence. In real tasks and big models, however, the models can be poorly trained and cannot converge. \n2. The assumption behind $\\mathcal{NC}$ is questionable. Firstly, I doubt whether the former layers can learn well in difficult tasks. In other words, the premise that the former layers can approximate continuous function is unreliable. Secondly, even if the former layers learn good feature representations, the interaction between the last layer and the former layers still cannot be ignored. Separating the former and last layers is a nice try but have no theoretical guarantee.\n3. Minor: some figures need modification due to the small size of the fonts, e.g., Fig 3", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2rCw5gsk39i", "reviewer": "Reviewer_cCWw", "summary": "The authors study the phenomenon of neural collapse (NC) for a wider variety of loss functions for classification (as well as MSE adapted for classification) in the balanced multi-class problem.  They broaden the set of objectives used for NC study to include modifications of the standard CE loss such as Focal Loss and Label Smoothing.  Building on the Unconstrained Features Model for studying NC, they develop a property these loss functions all satisfy (the Contrastive property), and go on to explore the structures of over-parameterized networks trained using SGD to optimize losses that satisfy this property.  They conclude that all of the losses they study induce structural similarity in the solution to the unconstrained features model, and furthermore that neural collapse is observed not only during training, but also on test sets. \n", "questions": "Lines 77-84: For the uninitiated, it would help to more precisely refine the NC phenomenon from the more generally understood overfitting phenomenon.  How are they different?\n\nI would like to have seen the authors extend beyond networks for vision and into other settings like LLMs, or to better contrast *what* about some losses do not lead to NC? \n\nSection [5]: the paragraph beginning on line 355 was somewhat confusing; the authors cite 61 as an example of more involved loss functions (contrasted with CE) that \"discourage variability collapse and learn diverse features, which are shown to benefit model transferability and robustness\".  But [61] concludes that actually most of the loss functions they examined confer better on-task performance relative to CE, but the features yielded from training with them performed *worse than CE* when transferring to other tasks, which seems to contradict the authors claims here.  Could the authors clarify this?", "limitations": "Yes, they have.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I encourage the authors to build on their intuition in lines 348 - 350, and to formalize the property of losses that encourage intra-class compactness and inter-class separability.\n\nSection 3.2 [lines 222,239] While their own proof is slightly different for Theorems 1,2 (by the authors’ own admission), they are using tools developed by previous studies of NC.  But they are invoked to justify application to other losses (e.g lines 240-244) with only the briefest of explanation. This makes it difficult for the reader to determine the correctness of this extension, and also how much of what follows is an original contribution here, versus a restatement from previous recent papers studying NC. \n\nSection 4.1 [lines 298,299] The definitions of NC1,2,3 are  really important!  It’s a shame they have to be shunted to the appendix, makes it hard to get much from the paper alone.\n\nI was really hoping that the apparent conflicting results on lines 81-84 would be resolved (i.e the surprising findings that go against those of [26]). \nRead closely: do lines 321 - 331 support the conclusion that all losses are created equal?  It seems like it should be plausible to evaluate when H,W exhibit NC, and thus support their claim that NC happens on the test set regardless of the loss chosen.\n\nSection 4.2 [lines 324-331]: I think the set of experiments is too narrow to conclude the “all losses created equal” claim.  I tdon’t doubt that there is something here.  But I would rephrase the message in this section to say something like “when conditions between dataset and model allow for SGD to find an NC solution, all losses we tested produced indistinguishable results”\n\nAn experiment I would have liked to see, would be to reduce the model capacity until NC solutions become less likely to be found by NC.\n\nSection 5 [line 340]: missing a \"to\" in the phrase \"pertaining training data only,\"\n\nRegarding neural collapse on the test set.  Hui et al. refine the definitions of NC on the test set as strong & weak forms.  Weak test collapse (defintion 2 in that work) states taht a training procedure exhibits weak test-collapse on a distribution D if for all sample sizes n, each test point in each sample is mapped to one of the k means with probability 1.  Crucially, they note, that collapse must be observed for *every finite sample test set drawn from D*.  \n\nMoreover, they go on to define tests for when collapse can be said to be observed (see the definitions for train collapse and weak test collapse in section 3.1): for each train or test point, you can measure the variability of each $h_{S}^{t}(x)$ from the corresponding collapsed point $\\mu_{y}$.  They take the expection over the whole set $S$ (or training set $T$), and scale by the expected deviation of each $\\mu_{y}$ from the marginal mean of all the data $\\mu := \\sum_{i} \\frac{\\mu_{i}}{|\\mathcal{Y}|}$.  While the results presented in Figure 3 are *consistent* with observing NC on the test set, they are not nearly so convincing as simply measuring the degree of observed NC directly.\n\nOverall, I think this paper makes important original contributions by revealing the structural similarities of different loss functions for networks trained to NC conditions, but that these contributions are not very well delineated from prior work, and are lacking some critical empirical evidence to support the claims (especially the claims regarding loss equivalency under NC conditions at test time).  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept