Are All Losses Created Equal: A Neural Collapse Perspective

Abstract: While cross entropy (CE) is the most commonly used loss function to train deep neural networks for classification tasks, many alternative losses have been developed to obtain better empirical performance.  Among them, which one is the best to use is still a mystery, because there seem to be multiple factors affecting the answer, such as properties of the dataset, the choice of network architecture, and so on.  This paper studies the choice of loss function by examining the last-layer features of deep networks, drawing inspiration from a recent line work showing that the global optimal solution of CE and mean-square-error (MSE) losses exhibits a Neural Collapse phenomenon.  That is, for sufficiently large networks trained until convergence, (i) all features of the same class collapse to the corresponding class mean and (ii) the means associated with different classes are in a configuration where their pairwise distances are all equal and maximized.  We extend such results and show through global solution and landscape analyses that a broad family of loss functions including commonly used label smoothing (LS) and focal loss (FL) exhibits Neural Collapse. Hence, all relevant losses (i.e., CE, LS, FL, MSE) produce equivalent features on training data.  In particular, based on the unconstrained feature model assumption, we provide either the global landscape analysis for LS loss or the local landscape analysis for FL loss and show that  the (only!) global minimizers are neural collapse solutions, while all other critical points are strict saddles whose Hessian exhibit negative curvature directions either in the global scope for LS loss or in the local scope for FL loss near the optimal solution.  The experiments further show that Neural Collapse features obtained from all relevant losses (i.e., CE, LS, FL, MSE) lead to largely identical performance on test data as well, provided that the network is sufficiently large and trained until convergence. 

Reviews and author responses:

 We thanks for the reviewer for the appreciation and support of our work. We also appreciate the reviewer’s constructive suggestions in the review, which we will incoporated in our camera-ready version.

 We thank the reviewer for the appreciation of our efforts and recognition of our work in the response, and again, we thank the reviewer for the detailed and thoughtful comments, which helped us to improve our work. 

 > For the correctness of Theorem 2, as stated right after Theorem 2, its proof follows directly from [A]. In particular, we checked the proof of [A, Theorem 3.2] and it only uses smoothness and convexity of CE loss (the detailed proof can be found in the appendix of [A]), and thus the result can be applied more generally for any smooth convex loss function. We have included this discussion right after Theorem 2 in the revision.

Thanks, I appreciate your work to clarify this (for me :) )

> A6: Thanks for the suggestion. In the revision, we have plotted the NC1 of pairs of width-iteration configurations in figure 4 (for CIFAR10) and figure 7 (for CIFAR100) in the appendix. Comparing the NC1 and test accuracy, we can see that the values of NC1 for width = 0.25 and epochs=100 configuration is around three orders of magnitude larger than those for width=2 and epochs=800 configuration, which indicates the network of the former configuration is less likely collapsed.

This strengthens your case for loss-equality by establishing more clearly when NC can be said to occur in the experiments.



 Thank authors for your rebuttal and new experiments. After reading your rebuttal, I think the authors have addressed many questions in my review with solid experiments and additional references. Nice work.

 We thank the reviewer for the appreciation of our work and for detailed and thoughtful comments. In the following, we address the reviewer’s comments in detail.
***
_Q1: The claim on the "largely identical performance on test data" of different losses is not convincing, as it only comes from experimental results on CIFAR-10 and CIFAR-100. Also, on CIFAR-100 (Figure 6 in Appendix), the difference between MSE (78.0%) and FL (79.1%) is 1.1%, which is not negligible._

A1: Thanks for the comments. We have conducted additional experiments by training ResNet18 on miniImageNet to further support our claims. The results are plotted in Figure 8 in the appendix of the revision, and we observed a similar phenomenon as on CIFAR10 and CIFAR100. In particular, the last-iteration test accuracy of training WideResNet18-0.25 (small width) for 100 epochs is 0.7195, 0.6915, 0.7020 and 0.7040, respectively for CE, MSE, FL, and LS.  The test accuracy of training WideResNet18-2 (large width) for 800 epochs is 0.7930, 0.7962, 0.7932 and 0.8020 for CE, MSE, FL722 and LS, respectively. Not only all the test accuracies increase, but also the gap between them becomes much smaller. 
In terms of the performance of MSE on CIFAR-100, we do not tune the performance and its performance can be improved by tuning the rescaling parameters in the rescaled MSE (not the training parameters like learning rates and weight decay). For example, by setting the $\kappa=5, \beta=30$ of rescaled MSE equation (6), we can obtain an average test accuracy 78.7% of width=2, epochs =800 configuration for MSE. Due to limited time and computational resources, we will update all the results of MSE in the final version. In addition, we think that the network ResNet50-2 is not large enough since MSE still increases from width 1 to width 2. We conjecture the gap will decrease if we further increase the network width.
***
_Q2: Main concern is about the claim on test data performance mentioned above, which I think is not well-supported. One question relates to this: how to make this claim convincing? Other than the empirical evidence, could you provide more rigorous theory results on a simplified setting?_

A2: We have theoretical results for training under unconstrained features models, but only empirical results on testing performance. We believe it is an important question for future work to have a theoretical understanding of the test data performance (such as NC and accuracy). 
***


 We thank the reviewer for the appreciation of our work and for detailed and thoughtful comments. In the following, we address the reviewer’s comments in detail.
***
_Q1: The paper discusses an over-restricted situation in which all models can learn well towards convergence. In real tasks and big models, however, the models can be poorly trained and cannot converge._

A1: We agree that it is not easy to train a large model. But thanks to useful techniques such as batch-normalization, in general now we can train a deep network until convergence with properly chosen hyperparameters. In this paper, we study the effect of different loss functions under the assumptions that the networks are trained until convergence, which we believe is a reasonable assumption. Also we note for all the losses, the models are relatively easy to be trained and we use the same parameters such as learning rates, weight decay, data augmentation and so on for all the losses.
***
_Q2: The assumption behind NC is questionable. Firstly, I doubt whether...unreliable. Secondly, even if ... ignored. Separating ...  guarantee._

A2: We agree that our theoretical analysis is based on the unconstrained feature model, which is a simplified model. However, recent work indicates that the strong assumption of the unconstrained feature model is reasonable for explaining NC during the training phase. For example, the experiments in [A,B] show that a sufficiently large network can learn neural collapse features even for random labels.
Though the unconstrained feature model is a simplified model, it does not ignore the interaction between the last layer (linear classifier) and the former layer. Indeed, it is the goal to study the interaction between the classifiers and the features in the penultimate layer and show that the linear classifiers are perfectly matched with class-mean features, i.e., the third property of neural collapse. 
***
_Q3: I wonder if the NC phenomenon is more common in other datasets and tasks or some more classic model architectures like MLP._

A3: The seminal work [C] shows NC phenomenon across a wide range of common network architectures (including VGG, ResNet, and DenseNet) and datasets (including MNIST, CIFAR10, CIFAR100, SVHN, STL10 and ImageNet), though some simple datasets (like MNIST) are easier to have NC than the othrs (like ImageNet). The work [D]  shows that NC also exists for MLP on CIFAR10. We have added this discussion in Footnote 1 in Section 4.
***
_Q4: Is the regularization term (weight decay) required to give rise to such a phenomenon? Ablation studies are required to eliminate the effect of the regularization term._ 

A4: Theoretically, without weight decay, the features and classifiers will blow up to infinity for CE and other losses. This is the reason that either a regularizer (as in [B]) or a norm constraint (as in [A]) is used to study the unconstrained feature model. In experiments, we also observe that the features and classifiers keep increasing without weight decay. We have included this discussion in the footnote right after eq. (1) and will add such as ablation study in the camera-ready version.   
***
_Q5:The datasets and models are limited to support all the claims. More models and datasets are required._

A5: Thanks for the suggestion. Due to time and computational limitations, we have conducted additional experiments by training ResNet18 on miniImageNet to further support our claims. The results are plotted in Figure 8 in the appendix of the revision, and we observed a similar phenomenon as on CIFAR10 and CIFAR100. 
***
_A6: Given that all the losses are the same from the view of NC, how can we guide our training in a real situation? How can we choose the loss function?_

Q6: When the network is large enough and trained longer enough, all losses could lead to similar performance. This is verified on real training scenario (Resnet on CIFAR10, CIFAR100, and Mini-ImageNet). But for limited resources, different losses may lead to different performance. It is of interest to find the best loss function for this case. We leave it as future work.
***
```
[A]  Florian Graf, Christoph Hofer, Marc Niethammer, and Roland Kwitt. Dissecting supervised constrastive learning. In International Conference on Machine Learning. PMLR, 2021.
[B] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 2021.
[C] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020.
[D] Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization land413 scape of neural collapse under mse loss: Global optimality with unconstrained features. In International Conference on Machine Learning. PMLR, 2022.
```


 ***
_Q9: Lines 77-84: For the uninitiated, it would help to more precisely refine the NC phenomenon from the more generally understood overfitting phenomenon. How are they different?_

A9: We think that we still lack a clear answer to whether a deep network ovefits. On one hand, we observe that increasing network size and training epochs improves the testing performance on CIFAR10, CIFAR100, and miniImageNet. So there appears to be no overfitting issue. On the other hand, previous works have shown that deep networks can fit the labels and establish NC on a training set even with random labels [A,C], which may be said as overfitting. So we don't specify overfitting issues to avoid confusion. We may misinterpret the reviewer's comment; if so, we are happy to answer the reviewer’s question in continuing discussion.
***
_Q10: I would like to have seen the authors extend beyond networks for vision and into other settings like LLMs, or to better contrast what about some losses do not lead to NC?_

A10: Thanks for the suggestion. We are aware of other’s unpublished work showing that NC also exists for the Roberta and BERT networks on six classification tasks of GLUE benchmark: CoLA, MNLI, MRPC, QNLI, QQP, SST-2. So we expect similar phenomena may exist for large language models (LLMs), but we leave a thorough study as future work due to lack of expertise in LLMs and limited time and computational resources. We also believe it is a very interesting direction to formally compare the losses that lead to NC and those that do not lead to NC, such as the MCR^2 [D]. As it requires a substantial effort to exploit, we leave the thorough study as future work. 
***
_Q11:  Section [5]: the paragraph beginning on line 355 was somewhat confusing.. cite [61] as an example …But [61] concludes that … worse than CE when transferring to other tasks, which seems to contradict the authors claims here. Could the authors clarify this?_

A11: Thanks for the question. We test the performance of the learned deep networks on the same dataset as training, not for a new task. So there is no contradiction with the other work on model transferability. We have clarified this at the end of introduction. 
***
```
[A] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 2021.
[B] Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding426 generalization in deep learning. arXiv preprint arXiv:2202.08384, 2022.
[C] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep424 learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115, 2021.
[D] Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. arXiv preprint arXiv:2006.08558,507 2020.
```


 We thank the reviewer for the detailed and thoughtful comments. In the following, we address the reviewer’s comments in detail.
***
_Q1: I encourage the authors to build on their intuition in lines 348 - 350, and to formalize the property of losses that encourage intra-class compactness and inter-class separability._

A1: Thanks for the suggestion. In the revision, right before Definition 1, we have included this discussion about encouraging intra-class compactness and inter-class separability to formalize the property of losses. 
***

_Q2: “Section 3.2 [lines 222,239] While their own proof is slightly different… with only the briefest of explanation …, previous recent papers studying NC”._

A2: We first note that Theorem 1 is a new result,  which generalizes the previous work and can be applied to general loss functions. Our proof is different from previous work in terms of dealing with the nuclear norm and checking the structures of the minimizer per sample and class, enabling the global optimality analysis for general loss functions. We included the full proof in the Appendix.

For the correctness of Theorem 2, as stated right after Theorem 2, its proof follows directly from [A]. In particular, we checked the proof of [A, Theorem 3.2] and it only uses smoothness and convexity of CE loss (the detailed proof can be found in the appendix of [A]), and thus the result can be applied more generally for any smooth convex loss function. We have included this discussion right after Theorem 2 in the revision.
***

_Q3: Section 4.1 [lines 298,299] The definitions of NC1,2,3 are really important! It’s a shame they have to be shunted to the appendix, makes it hard to get much from the paper alone._

A3: Thanks for your suggestion. We agree and will include them in the final version, if we are allowed an additional content page for the camera-ready version as previous years.
***

_Q4: I was really hoping that the apparent conflicting …Read closely: do lines 321 - 331 …the test set regardless of the loss chosen._

A4: We first note that our results require the network to be sufficiently large, which is different from [26]. Second, our main point is that all losses will lead to the NC solutions on the training dataset, and further lead to the identical test accuracy, as long as the training processes to NC global optimality. We do not study the NC on the test data (which will be left as future work). So the results do not go against those of [26]. We have highlighted the assumption of using large model in lines 73, 78-80.
***

_Q5: Section 4.2 [lines 324-331]: I think the set of experiments is too narrow…But I would rephrase the message like “when conditions between dataset and model allow for SGD to find an NC solution, all losses we tested produced indistinguishable results”_

A5: We appreciate your suggestion and we have rephrased this sentence following your suggestion in lines 323-324 in the revision. 
***

_Q6: An experiment I would have liked to see, would be to reduce the model capacity until NC solutions become less likely to be found by NC._

A6: Thanks for the suggestion. In the revision, we have plotted the NC1 of pairs of width-iteration configurations in figure 4 (for CIFAR10) and figure 7 (for CIFAR100) in the appendix. Comparing the NC1 and test accuracy, we can see that the values of NC1 for width = 0.25 and epochs=100 configuration is around __three orders__ of magnitude larger than those for width=2 and epochs=800 configuration, which indicates the network of the former configuration is less likely collapsed.
***
_Q7: While the results presented in Figure 3 are consistent with observing NC on the test set, they are not nearly so convincing as simply measuring the degree of observed NC directly._

A7: We actually don’t consider the correlation between NC on the training set and test set, but instead, we focus on the connections between test accuracy and NC on the training set. Nonetheless we agree that the NC on testing data may be used to connect these two. As a future work, we will exploit this direction by using the definition defined in [B] for NC on testing data.
***
_Q8:  these contributions are not very well delineated from prior work, and are lacking some critical empirical evidence to support the claims(especially the claims regarding loss equivalency under NC conditions at test time)._ 

A8: As answered in Q7, this work only considers the NC on training set and the test accuracy after training exhibits NC. We do not study the relationship between the NC on the training set and test set.  But we think it is an important direction to study NC for testing data. To avoid confusion with the current work, we leave a thorough study as future work. Due to limited time and computational resources, we conduct additional experiments by training ResNet18 on miniImageNet and include the results in figure 8 in the revision to further support our claims in paper. 
***
**to be continued in the following**


 The authors study the phenomenon of neural collapse (NC) for a wider variety of loss functions for classification (as well as MSE adapted for classification) in the balanced multi-class problem.  They broaden the set of objectives used for NC study to include modifications of the standard CE loss such as Focal Loss and Label Smoothing.  Building on the Unconstrained Features Model for studying NC, they develop a property these loss functions all satisfy (the Contrastive property), and go on to explore the structures of over-parameterized networks trained using SGD to optimize losses that satisfy this property.  They conclude that all of the losses they study induce structural similarity in the solution to the unconstrained features model, and furthermore that neural collapse is observed not only during training, but also on test sets. 
 I encourage the authors to build on their intuition in lines 348 - 350, and to formalize the property of losses that encourage intra-class compactness and inter-class separability.

Section 3.2 [lines 222,239] While their own proof is slightly different for Theorems 1,2 (by the authors’ own admission), they are using tools developed by previous studies of NC.  But they are invoked to justify application to other losses (e.g lines 240-244) with only the briefest of explanation. This makes it difficult for the reader to determine the correctness of this extension, and also how much of what follows is an original contribution here, versus a restatement from previous recent papers studying NC. 

Section 4.1 [lines 298,299] The definitions of NC1,2,3 are  really important!  It’s a shame they have to be shunted to the appendix, makes it hard to get much from the paper alone.

I was really hoping that the apparent conflicting results on lines 81-84 would be resolved (i.e the surprising findings that go against those of [26]). 
Read closely: do lines 321 - 331 support the conclusion that all losses are created equal?  It seems like it should be plausible to evaluate when H,W exhibit NC, and thus support their claim that NC happens on the test set regardless of the loss chosen.

Section 4.2 [lines 324-331]: I think the set of experiments is too narrow to conclude the “all losses created equal” claim.  I tdon’t doubt that there is something here.  But I would rephrase the message in this section to say something like “when conditions between dataset and model allow for SGD to find an NC solution, all losses we tested produced indistinguishable results”

An experiment I would have liked to see, would be to reduce the model capacity until NC solutions become less likely to be found by NC.

Section 5 [line 340]: missing a "to" in the phrase "pertaining training data only,"

Regarding neural collapse on the test set.  Hui et al. refine the definitions of NC on the test set as strong & weak forms.  Weak test collapse (defintion 2 in that work) states taht a training procedure exhibits weak test-collapse on a distribution D if for all sample sizes n, each test point in each sample is mapped to one of the k means with probability 1.  Crucially, they note, that collapse must be observed for *every finite sample test set drawn from D*.  

Moreover, they go on to define tests for when collapse can be said to be observed (see the definitions for train collapse and weak test collapse in section 3.1): for each train or test point, you can measure the variability of each $h_{S}^{t}(x)$ from the corresponding collapsed point $\mu_{y}$.  They take the expection over the whole set $S$ (or training set $T$), and scale by the expected deviation of each $\mu_{y}$ from the marginal mean of all the data $\mu := \sum_{i} \frac{\mu_{i}}{|\mathcal{Y}|}$.  While the results presented in Figure 3 are *consistent* with observing NC on the test set, they are not nearly so convincing as simply measuring the degree of observed NC directly.

Overall, I think this paper makes important original contributions by revealing the structural similarities of different loss functions for networks trained to NC conditions, but that these contributions are not very well delineated from prior work, and are lacking some critical empirical evidence to support the claims (especially the claims regarding loss equivalency under NC conditions at test time).   Lines 77-84: For the uninitiated, it would help to more precisely refine the NC phenomenon from the more generally understood overfitting phenomenon.  How are they different?

I would like to have seen the authors extend beyond networks for vision and into other settings like LLMs, or to better contrast *what* about some losses do not lead to NC? 

Section [5]: the paragraph beginning on line 355 was somewhat confusing; the authors cite 61 as an example of more involved loss functions (contrasted with CE) that "discourage variability collapse and learn diverse features, which are shown to benefit model transferability and robustness".  But [61] concludes that actually most of the loss functions they examined confer better on-task performance relative to CE, but the features yielded from training with them performed *worse than CE* when transferring to other tasks, which seems to contradict the authors claims here.  Could the authors clarify this? Yes, they have.

 This paper mainly focuses on the choice of the loss function in the training of neural networks. Specifically, at the end of the convergence of the training process, all the features collapse to the class mean, and the pairwise distances of different classes are maximized and equal. Moreover, the author proposes a new insight that different losses can generate similar neural collapse features when the network is sufficiently large and converged. [Strengths]
This paper experimentally shows an interesting phenomenon that different losses are the same if the network is sufficiently trained until convergence. The model can learn the same neural collapse features with different losses.

[Weakneses]
1. The paper discusses an over-restricted situation in which all models can learn well towards convergence. In real tasks and big models, however, the models can be poorly trained and cannot converge. 
2. The assumption behind $\mathcal{NC}$ is questionable. Firstly, I doubt whether the former layers can learn well in difficult tasks. In other words, the premise that the former layers can approximate continuous function is unreliable. Secondly, even if the former layers learn good feature representations, the interaction between the last layer and the former layers still cannot be ignored. Separating the former and last layers is a nice try but have no theoretical guarantee.
3. Minor: some figures need modification due to the small size of the fonts, e.g., Fig 3 1. I wonder if the $\mathcal{NC}$  phenomenon is more common in other datasets and tasks or some more classic model architectures like MLP.
2. Is the regularization term (weight decay) required to give rise to such a phenomenon? Ablation studies are required to eliminate the effect of the regularization term. A net loss function is better to verify the statements in the paper. 1. The datasets and models are limited to support all the claims. More models and datasets are required.
2. There are no further statements linking the losses from the $\mathcal{NC}$ perspective and the real training scenario. In other words, given that all the losses are the same from the view of $\mathcal {NC} $, how can we guide our training in a real situation? How can we choose the loss function?

 This paper did global solution and landscape analyses and extend the neural collapse phenomenon (on training data) to a broad family of loss functions, focusing on the less-studied label smoothing and focal loss.

By landscape analyses, they show the global minimizers are neural collapse solutions. Also, empirically they show NC features obtained from all relevant losses (till convergence) lead to largely identical test performance.

 Strengths: 
  1) The paper looks at different losses from the neural collapse perspective and conclude that all losses can be equal (at convergence) as they produce equivalent features on training data. This perspective is interesting and the conclusion is strong.

  2) The writing is clear and it is easy to follow the ideas.

Weaknesses:
  The claim on the "largely identical performance on test data" of different losses is not convincing, as it only comes from experimental results on CIFAR-10 and CIFAR-100. Also, on CIFAR-100 (Figure 6 in Appendix), the difference between MSE (78.0%) and FL (79.1%) is 1.1%, which is not negligible. The context of the paper is clear and no many questions.

Main concern is about the claim on test data performance mentioned above, which I think is not well-supported.
One question relates to this:

  How to make this claim convincing? Other than the empirical evidence, could you provide more rigorous theory results on a simplified setting? The claim on "largely identical performance on test data" is not convincing. 

Meta-review: This paper demonstrates that the neural collapse phenomenon - first observed under cross entropy and MSE losses - occurs with a broad family of other loss functions. The authors include a theoretical justification of this phenomenon as well as experiments exploring its implications. Overall, this paper makes an important contribution to the study of loss functions and raises interesting questions about neural network training. To strengthen the paper, I encourage the authors to incorporate the reviewer’s suggestions for additional experiments (ablation studies and additional datasets). Nevertheless, this paper is well suited for publication at NeurIPS and will be of interest to the neural network community.

Decision: Accept