Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies the Neural-collapse (NC) phenomenon for data with class imbalances. The authors study two versions of the unconstrained feature model (UFM), which is a theoretical model for studying neural collapse, and introduce Simplex-Encoded-Labels Interpolation (SELI) geometry. The SELI geometry is a generalization of the equiangular tight frame (ETF) in the NC with balanced data. For the UF-SVM problem, they show that any solution follows the SELI geometry, and the major classes and embeddings are assigned with larger norms. For the UFM with ridge-regularized cross-entropy loss, they show that no finite regularization leads to the SELI geometry. Instead, as regularization vanishes, the solution converges in direction to the SEL matrix. The authors also provide experiments supporting that SELI is a better characterization than ETF for class-imbalanced data. The experiments also indicate that the convergence worsens with increasing level of imbalance. 
strengths_and_weaknesses: The paper studies data with class imbalances, which is an important extension to the existing NC theory. The paper is well organized and clearly written, making the contribution and methodology easy to understand. The proposed SELI geometry is justified as it serves as the optimal solution for the UF-SVM model. The authors also connect the SELI geometry to the UFM with ridge-regularized CE model, where the solution converges in direction to the SELI geometry when the regularization vanishes. As a result, it seems convincing to use SELI as a generalization for the ETF for class-imbalanced data. Lemma 1 showing that UF-SVM assigns major classes with larger classifiers and embeddings is also interesting. 

Weakness: 

1. The main weakness of the work is that it only studies the UFM. Even though the UFM is used in other analyses for the NC, it is an oversimplified model and is not equivalent to real neural networks. As a result, it is possible that the SELI geometry characterized in the paper is only true for the UFM instead of empirical neural networks. 

2. The main technical contribution of the work is theorem 1, where the authors use convex relaxation and the dual problem to show that $\hat{Z}$ is the unique minimizer. The analysis for the UFM with ridge-regularized CE model, however, is mostly based on earlier analysis. Theorem 2 is a natural extension of [Zhu et al, 2021], and proposition 2 is an extension of [Rosset et al, 2003]. This may undermine the theoretical contribution of the work. 

3. The early works of the NC are mostly empirical observations, showing that one can observe the NC phenomenon in empirical classification tasks. This paper tries to argue that SELI serves as a generalization of the ETF for class-imbalanced data. So it would be nice if authors can provide empirical results where the SELI geometry can be observed over realistic datasets. Compared to Figure 2, a demonstration looks like Figure 1 would be more convincing. Still, the theoretical contribution of the paper can have independent interest to the field, so the additional experiment is not a hard requirement. 
questions: 1. In the experiments, it is reported that increasing levels of imbalance would worsen the convergence. This imbalance is characterized by the imbalance ratio $R$. There is another term $\rho$ that describes the minority fraction, and it is not addressed in the theorem or the experiments. I am wondering how the $\rho$ affects the SELI geometry or the speed of convergence. 

2. The NC phenomenon happens for very deep networks. The UFM can be viewed as an approximation for infinitely deep neural networks. Will the imbalance level affects the depth required to observe NC? In other words, will one need a deeper network to observe neural collapse for imbalanced data? It is fine if the authors don't have a good answer for this as the paper does not study the effect of the depth of the networks. 
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors study the geometric of neural network's learned embeddings and classifier's weights. They investigates a simplified neural network model: unconstrained feature model. Unlike the existing works that focuses on balanced dataset, they focus on imbalanced dataset and establish a novel geometric characterization of NN's learned embeddings and classifier's weights. The results are insightful and can have strong impact towards better understanding of neural network's characteristics in practice.
strengths_and_weaknesses: With existing work on neural collapse on balanced dataset, this paper moves an important step forward to the case of imbalanced dataset.
1. The paper uses novel technical approaches and proposes SELI, a more generalized description of NN's geometric patterns instead of ETF. A connection between SELI and ETF is also established.
2. A detailed analysis of how regularization influences the minimizer of UFM is provided, which I think might be insightful for NN as well.
3. The paper extend the results of minority collapse from previous work and provides a more thorough classification of the UFM minimizer's geometry under imbalanced datasets.
4. the paper is well written and clearly organized.

With above said, I believe that the practical impact of the results would be more significant if more experiments are conducted on various neural networks and different datasets, in addition to CIFAR-10 and MNIST.

Overall, I believe that the analysis on imbalanced dataset is a significant step towards the understanding of NN, compared to the balanced dataset. With solid theoretical results, empirical experiment support and a clear presentation of the content, this is a very good paper and I anticipate it to have strong impact on the future study of NN's geometric patterns and dynamics.
questions: It would be interesting if author can discuss about the challenges to generalize the result to a more general imbalanced dataset. 
limitations: No negative impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work studies the neural collapse for classification problems where the training data is imbalanced. Speciﬁcally, under the unconstrained feature model (UFM), the work theoretically studies the problem with cross-entropy loss and vanishing regularization. Irrespective of class imbalances, the work proved that embeddings and classiﬁers always interpolate a simplex-encoded label matrix and that their individual geometries are determined by the SVD factors of this same label matrix. The study is empirically verified by experiments on synthetic and real datasets that conﬁrm convergence to the SELI geometry and the convergence rate depends on the imbalance ratio.

strengths_and_weaknesses: * Strengths: the work studies an interesting problem of neural collapse under imbalanced data and an unconstrained feature model. It shows a more general phenomenon of SELI geometry instead of Simplex ETF, when the training data is imbalanced. The study is well supported by experiments.

* Weakness: The reviewer finds the paper is a little hard to follow, and the writing needs to be significantly polished and the paper needs to be reorganized to be more clear to the readers. 
Also, the reviewer cannot find too much practical guidance from this study. For training on imbalanced data, some recent works on neural collapse are missing in the reference and have not been compared/discussed:
https://arxiv.org/abs/2203.09081
https://arxiv.org/abs/2204.08735
https://arxiv.org/abs/2206.04041
Although it is nice to show the convergence to SELI, what would be the benefits of understanding this? For example, one of the recent above works showed that fixing the linear classifier as an ETF can improve generalization performance. 

The current analysis seems to be some trivial extension of [9] and [24], without very clear justifications/motivations behind them. For example, the paper studied two formulations (1) and (2). The KKT condition in Theorem 1, and Theorem 2 for (1) can be directly generalized from [9] and [24] for SELI.

questions: * can the authors provide more intuitions behind the SELI? It is a bit difficult to understand based on the current statement of geometry.

* The authors study the neural collapse under the UFM-SVM formulation, arguing that neural collapse is happening with vanishing regularizations. Is there empirical support for this? With moderate regularization, neural collapse still happens. The UFM-SVM is not the commonly used formulation for training neural networks. Or is this for the ease of analysis?

* can the authors explain more on the intuitions behind the SELI? It is a bit difficult to understand based upon the current statement.

* what is the practical benefits of such a study for training deep networks with imbalanced data? It is a bit unclear to the reviewer.

limitations: n/a
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies the effect of class-imbalanced data on neural collapse (NC) of DNNs through the unconstrained features model (UFM).
It does it by identifying the structure of the minima of the UFM in a max-margin optimization problem and connecting it to those of the minima of CE loss with vanishing regularization.
It shows that the new results are in some sense complementary to the "minority collapse" that has been shown before (they are two opposite extreme cases of the level of regularization) and shows correlation of the theory with practical NC of DNNs.
In my opinion, these are significant contributions.

strengths_and_weaknesses: Strengths: Apart from the novel significant results that are mentioned above, I also find the paper to be nicely written and organized.

Weaknesses: I have mostly minor comments that are listed below.




questions: 1. 
What can you say about how your results may be changed if W and H in Eq. 1 and in Eq. 2 have different regularization hyperparameters?
The reason I ask this is because only the dimension of H depends on the number of samples (unlike W), while it seems "closer" to the weight-decay that is used in practice if both regularization terms will be scaled similarly with respect to n.
Many previous UFM works provide results that hold for separate hyperparameters for H and W and thus do not have this issue.

2. 
I disagree with the claim that one can interpret the "UFM as two-layer linear net" (page 4, line 120), because it is limited to "data" $X=I_n$ which is extremely naïve. Also, in a two-layer linear net the dimensions of the weights do not depend on the number of samples n.
This problematic interpretation appears also in Section 4.4, which requires some rephrasing of this section in my opinion. 
Also, you don't really need to repeat again this perspective for the "UFM: SGD solutions" that are presented in Figure 4 and are interesting on their own. 

3. 
In page 5 line 188, what do you mean in: "U has repeated columns"? it has only k-1 columns. Perhaps you mean rows? Add more details how can you see it from Theorem 1.
Also, consider rewriting the "more straightforward argument" in a more mathematical/formal way (e.g., do you use Jensen's inequality per class?).

4. 
In page 6 line 205, note that [18] claims that W is aligned with the features' means after reducing their global mean. Mention why you do not do reduce the global mean in M. (Is it zero like in previous UFM works with CE loss?)
Also, don't the equalities hold (only) up to some scalar factors?

5. 
You should also refer in the paper to the following related work, which also analyzes the UFM (showing effect of bias regularization on the solution structure) and further extends it (to "two-layer (non)linear net" on-top of the features): 

Tirer, T. and Bruna, J., 2022. Extended unconstrained features model for exploring deep neural collapse. arXiv preprint arXiv:2202.08087

limitations: n/a
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies neural collapse for classification with class imbalanced data. The authors propose a novel characterization of the neural collapse phenomenon which is called Simplex-Encoded-Labels Interpolation (SELI). It has been shown both theoretically and experimentally that SELI is invariant across different imbalance levels.
strengths_and_weaknesses: Strengths:
1. This paper is well-written and easy to follow.
2. While the problem of neural collapse with class imbalance has been investigated in the previous work [1], this work derives an novel characterization that is invariant to class imbalances.
3. The authors extensively studied the role of regularization in tweaking the geometry.

Weaknesses: 

1. It is not clear how the proposed description of the geometric structure helps to explain the generalization problem in learning with imbalanced data, and how it helps to mitigate the problem of data imbalance.

[1] Fang, Cong, et al. "Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training." Proceedings of the National Academy of Sciences 118.43 (2021): e2103091118.
questions: It would be helpful if there is a comparison between the proposed SELI and the ETF geometry in Figure 1.
limitations: The authors have adequately addressed the limitations and potential negative social impact of their work.
review_title: null

Meta-review: metareview: This work makes interesting contributions toward understanding the geometric properties of well-trained neural networks by analyzing the neural collapse phenomenon on imbalanced datasets. The proposed SELI is novel. I encourage the author(s) to increase the practical impact of the results by conducting experiments with more architectures and datasets.
confidence: Certain
Decision: Accept