Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors study the phenomenon of neural collapse (NC) for a wider variety of loss functions for classification (as well as MSE adapted for classification) in the balanced multi-class problem.  They broaden the set of objectives used for NC study to include modifications of the standard CE loss such as Focal Loss and Label Smoothing.  Building on the Unconstrained Features Model for studying NC, they develop a property these loss functions all satisfy (the Contrastive property), and go on to explore the structures of over-parameterized networks trained using SGD to optimize losses that satisfy this property.  They conclude that all of the losses they study induce structural similarity in the solution to the unconstrained features model, and furthermore that neural collapse is observed not only during training, but also on test sets. 

strengths_and_weaknesses: I encourage the authors to build on their intuition in lines 348 - 350, and to formalize the property of losses that encourage intra-class compactness and inter-class separability.

Section 3.2 [lines 222,239] While their own proof is slightly different for Theorems 1,2 (by the authors’ own admission), they are using tools developed by previous studies of NC.  But they are invoked to justify application to other losses (e.g lines 240-244) with only the briefest of explanation. This makes it difficult for the reader to determine the correctness of this extension, and also how much of what follows is an original contribution here, versus a restatement from previous recent papers studying NC. 

Section 4.1 [lines 298,299] The definitions of NC1,2,3 are  really important!  It’s a shame they have to be shunted to the appendix, makes it hard to get much from the paper alone.

I was really hoping that the apparent conflicting results on lines 81-84 would be resolved (i.e the surprising findings that go against those of [26]). 
Read closely: do lines 321 - 331 support the conclusion that all losses are created equal?  It seems like it should be plausible to evaluate when H,W exhibit NC, and thus support their claim that NC happens on the test set regardless of the loss chosen.

Section 4.2 [lines 324-331]: I think the set of experiments is too narrow to conclude the “all losses created equal” claim.  I tdon’t doubt that there is something here.  But I would rephrase the message in this section to say something like “when conditions between dataset and model allow for SGD to find an NC solution, all losses we tested produced indistinguishable results”

An experiment I would have liked to see, would be to reduce the model capacity until NC solutions become less likely to be found by NC.

Section 5 [line 340]: missing a "to" in the phrase "pertaining training data only,"

Regarding neural collapse on the test set.  Hui et al. refine the definitions of NC on the test set as strong & weak forms.  Weak test collapse (defintion 2 in that work) states taht a training procedure exhibits weak test-collapse on a distribution D if for all sample sizes n, each test point in each sample is mapped to one of the k means with probability 1.  Crucially, they note, that collapse must be observed for *every finite sample test set drawn from D*.  

Moreover, they go on to define tests for when collapse can be said to be observed (see the definitions for train collapse and weak test collapse in section 3.1): for each train or test point, you can measure the variability of each $h_{S}^{t}(x)$ from the corresponding collapsed point $\mu_{y}$.  They take the expection over the whole set $S$ (or training set $T$), and scale by the expected deviation of each $\mu_{y}$ from the marginal mean of all the data $\mu := \sum_{i} \frac{\mu_{i}}{|\mathcal{Y}|}$.  While the results presented in Figure 3 are *consistent* with observing NC on the test set, they are not nearly so convincing as simply measuring the degree of observed NC directly.

Overall, I think this paper makes important original contributions by revealing the structural similarities of different loss functions for networks trained to NC conditions, but that these contributions are not very well delineated from prior work, and are lacking some critical empirical evidence to support the claims (especially the claims regarding loss equivalency under NC conditions at test time).  
questions: Lines 77-84: For the uninitiated, it would help to more precisely refine the NC phenomenon from the more generally understood overfitting phenomenon.  How are they different?

I would like to have seen the authors extend beyond networks for vision and into other settings like LLMs, or to better contrast *what* about some losses do not lead to NC? 

Section [5]: the paragraph beginning on line 355 was somewhat confusing; the authors cite 61 as an example of more involved loss functions (contrasted with CE) that "discourage variability collapse and learn diverse features, which are shown to benefit model transferability and robustness".  But [61] concludes that actually most of the loss functions they examined confer better on-task performance relative to CE, but the features yielded from training with them performed *worse than CE* when transferring to other tasks, which seems to contradict the authors claims here.  Could the authors clarify this?
limitations: Yes, they have.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper mainly focuses on the choice of the loss function in the training of neural networks. Specifically, at the end of the convergence of the training process, all the features collapse to the class mean, and the pairwise distances of different classes are maximized and equal. Moreover, the author proposes a new insight that different losses can generate similar neural collapse features when the network is sufficiently large and converged.
strengths_and_weaknesses: [Strengths]
This paper experimentally shows an interesting phenomenon that different losses are the same if the network is sufficiently trained until convergence. The model can learn the same neural collapse features with different losses.

[Weakneses]
1. The paper discusses an over-restricted situation in which all models can learn well towards convergence. In real tasks and big models, however, the models can be poorly trained and cannot converge. 
2. The assumption behind $\mathcal{NC}$ is questionable. Firstly, I doubt whether the former layers can learn well in difficult tasks. In other words, the premise that the former layers can approximate continuous function is unreliable. Secondly, even if the former layers learn good feature representations, the interaction between the last layer and the former layers still cannot be ignored. Separating the former and last layers is a nice try but have no theoretical guarantee.
3. Minor: some figures need modification due to the small size of the fonts, e.g., Fig 3
questions: 1. I wonder if the $\mathcal{NC}$  phenomenon is more common in other datasets and tasks or some more classic model architectures like MLP.
2. Is the regularization term (weight decay) required to give rise to such a phenomenon? Ablation studies are required to eliminate the effect of the regularization term. A net loss function is better to verify the statements in the paper.
limitations: 1. The datasets and models are limited to support all the claims. More models and datasets are required.
2. There are no further statements linking the losses from the $\mathcal{NC}$ perspective and the real training scenario. In other words, given that all the losses are the same from the view of $\mathcal {NC} $, how can we guide our training in a real situation? How can we choose the loss function?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper did global solution and landscape analyses and extend the neural collapse phenomenon (on training data) to a broad family of loss functions, focusing on the less-studied label smoothing and focal loss.

By landscape analyses, they show the global minimizers are neural collapse solutions. Also, empirically they show NC features obtained from all relevant losses (till convergence) lead to largely identical test performance.


strengths_and_weaknesses: Strengths: 
  1) The paper looks at different losses from the neural collapse perspective and conclude that all losses can be equal (at convergence) as they produce equivalent features on training data. This perspective is interesting and the conclusion is strong.

  2) The writing is clear and it is easy to follow the ideas.

Weaknesses:
  The claim on the "largely identical performance on test data" of different losses is not convincing, as it only comes from experimental results on CIFAR-10 and CIFAR-100. Also, on CIFAR-100 (Figure 6 in Appendix), the difference between MSE (78.0%) and FL (79.1%) is 1.1%, which is not negligible.
questions: The context of the paper is clear and no many questions.

Main concern is about the claim on test data performance mentioned above, which I think is not well-supported.
One question relates to this:

  How to make this claim convincing? Other than the empirical evidence, could you provide more rigorous theory results on a simplified setting?
limitations: The claim on "largely identical performance on test data" is not convincing.
review_title: null

Meta-review: metareview: This paper demonstrates that the neural collapse phenomenon - first observed under cross entropy and MSE losses - occurs with a broad family of other loss functions. The authors include a theoretical justification of this phenomenon as well as experiments exploring its implications. Overall, this paper makes an important contribution to the study of loss functions and raises interesting questions about neural network training. To strengthen the paper, I encourage the authors to incorporate the reviewer’s suggestions for additional experiments (ablation studies and additional datasets). Nevertheless, this paper is well suited for publication at NeurIPS and will be of interest to the neural network community.
confidence: Certain
Decision: Accept