Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: On balanced datasets and when using cross-entropy loss for learning, it is observed that [28] the last layer features from the same class converge to their within-class means and the weights the network in the last layer collapses to the vertices of an equiangular tight frame (ETF). Thus, a natural question to ask is: if this empirical observation can be used to train better deep learning models? The current paper uses this insight to design a network where instead of training for a cross-entropy loss, a loss that regresses the deep features against  a randomly sampled set of vectors (corresponding to the vertices of an ETF) is used for classification. In situations when the data is imbalanced, the paper claims that this prior knowledge of ETF allows better learning of the model. Experiments are provided on small scale datasets (such as CIFAR, STL, etc.) and demonstrate promise of the method in imbalanced settings. 
strengths_and_weaknesses: I think the insights underlying the paper are encouraging and the application of the ETF model for imbalanced datasets is reasonable. The experiments show improvements on a variety of datasets under imbalanced settings. 

While, the insights are interesting, I do not think the contributions are compelling for the following reasons.

1.  The key question the paper asks is L58: "Do we really need to learn a linear classifier at the end of a deep neural network for classification?". While, in the context of ETF, it is an interesting question, however when thinking of this question in the context of learning in general, it is obviously not needed to learn a linear classifier at the end of a DNN. The one-hot encoding typically used on labels in cross-entropy loss based training is one way of encoding with its own geometric meaning, while the one proposed using ETF in the paper is yet another encoding. In that sense, I do not really see a significant contribution the paper brings in. Of course, when using an LPM model, we may get rid of learning of the classifier weights, but when using the method in an end-to-end learning framework, doesn't one still need to learn a method that maps the features into one of the vertices of the ETF? 

2.  While ETF is a structure that evolves from the use of CE loss, why not consider potentially other structures if we do not assume any CE loss? Why not a regression into the columns of an arbitrary orthogonal matrix (assuming there is no need to interpret the classification as probabilities any more, and thus there is no need to use a softmax)? 

3. There are standard approaches for dealing with the problem of imbalanced datasets, such as choosing weights during training, focal loss, etc. The paper appears to have not considered or compared the proposed approach to any of such alternatives and thus limits the understanding of the benefits to a narrow scope (e.g., Figure 3, Tables 2,3,4, etc.).

4. More experiments, for example in settings when there are very large number of classes as in ImageNet, can make the analysis significantly better. I wonder if in such settings a cross-entropy loss provides better "push" and "pull" gradients (due to the log term) than the proposed linear alternative (14).

Other comments:
1. For the layer-peeled model in (4), it is not clear how precisely are the features produced? If you optimize over W and H, then how do you associate the features to be from the data x? When optimizing for h, are you optimizing the parameters of the underlying network? If so, rewriting h to h_\theta(x) and optimizing over \theta would make the math better. If it is being optimized only on W and H, then how is H conditioned on the data X, and how do you prevent H from being arbitrary?
questions: Please see above. 
limitations: The paper does not describe any limitations as such. However, I believe experiments on larger datasets such as ImageNet could bring out the limitations of the approach better.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies neural collapse in the case in which the final classifier is not subject to back-propagation and is initialized with the vertices of a simplex equiangular tight frame (ETF). In particular the paper studies the unbalanced learning regime and shows that the neural collapse properties hold also in this case. The proof is based on the layered-peeled model assumption.

strengths_and_weaknesses: STRENGTHS:

1) The paper is very interesting as it provides a theoretical analysis of the class imbalance problem with a fixed classifier inspired by the recent neural collapse phenomenon.

2) The push-pull mechanism from the gradient analysis is novel and it is a clear contribution of the paper.

3) The idea may also have practical implications related to some recent works about learning compatible features representations [C, D]. Compatible features are learned so that they can be directly compared even if they are learned from different time instants or from different network architectures. The method [D] specifically takes advantage of a fixed classifier exactly defined as the one proposed in [47] and in this submission.

WEAKNESSES:

1) The paper contributions should be improved. It seems that the main contribution of the paper is about the combination of the following three aspects: neural collapse, unbalanced data and fixed classifiers. The title does not convey a direct relationship with these three aspects. In its current shape it seems more referring to the fixed classifier aspect, like in [29] and [12], rather than to the imbalanced Neural Collapse training regime. This is mostly due to the fact that if a learnable classifier is not “really needed” it is supposed that a fixed one would be preferred. In the reviewer's opinion the title should be improved reflecting more the main content of the paper.
In this respect, the differences with [47] and [29] should be better highlighted. In particular a question that should be answered is: what is the difference between an ETF simplex and the classic d-simplex regular polytope? The question is valid either in the fixed case and in the learnable one. Finally, as the paper [5] is very related with the imbalance aspect of the paper, it should be discussed in the related work section and possibly differences highlighted.

2) It seems that in many papers (including the seminal one [28] and [47]) the relationship between the number of classes K and the dimension d required to observe the neural Collapse phenomenon is d≥K while in the submitted manuscript and in [47] d≥K-1. Is there any specific motivation for these two cases?

3) Some important references are missing: [A,B]. Although not directly related to neural collapse, the paper [A] proposes a similar gradient analysis of the final classifier. A brief discussion highlighting differences and contributions should be given. The paper [B] seems to be one of the first published papers on maximal separation and fixed classifiers with the simplex geometry.

4) Is the random ETF initialization needed to avoid possible biases in its choice? Is there any evaluation quantifying this dependency?

5) Eq. 15 is referring to the optimization of the angle between features and classifier prototypes. What is the difference between that loss and the classic entropy loss in which both features and prototypes are normalized? Both minimize the angles between features and prototypes.

References

[A] P2SGrad: Refined Gradients for Optimizing Deep Face Models Xiao Zhang, Rui Zhao, Junjie Yan, Mengya Gao, Yu Qiao, Xiaogang Wang, Hongsheng Li; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019

[B] Maximally Compact and Separated Features with Regular Polytope Networks Federico Pernici, Matteo Bruni, Claudio Baecchi, Alberto Del Bimbo; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019, pp. 46-53

[C] Shen Y, Xiong Y, Xia W, Soatto S. Towards backward-compatible representation learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020 (pp. 6368-6377).

[D] Biondi N, Pernici F, Bruni M, Del Bimbo A. CoReS: Compatible Representations via Stationarity. arXiv preprint arXiv:2111.07632. 2021 Nov 15.

questions: See above.
limitations: No noteworthy limitations or potential negative societal impacts.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper proposes a loss function which projects the features to a pre-defined simplex and computes the gradients w.r.t it. It claims that neural networks use linear classifiers at the end and this is not needed if features are computed for each class and projected to a simplex. A theoretical explanation for this approach is provided and experimental results are presented.
strengths_and_weaknesses: The idea to try to solve classification using a simplex is interesting but I believe there are several papers in the deep learning community for large-scale classification problems (like face/few-shot recognition) which have evaluated similar loss functions which have not been cited or compared with in this paper. I am mentioning a few of these papers and they use similar kinds of projection functions but also present more variants of the version presented in this paper. For example, Wen et. al. (ECCV 2016) and Snell et. al. Neurips 2017, use class means/prototypes as a loss function (called center/prototype loss) to improve recognition performance. The class means are then constrained to a sphere by Liu et. al in SphereFace (CVPR 2017) which further improves the performance (like the ETF proposed in this paper). Deng et al. in ArcFace (CVPR 2019) further add an angular penalty between points on a sphere. There have been several papers after this (look for papers citing these 4) which explore ways to do classification without a linear layer followed by softmax+CE which are based on per-class features and their margins.

[1] A Discriminative Feature Learning Approach for Deep Face Recognition, Wen et. al, ECCV 2016

[2] Prototypical networks for few-shot learning, Snell et. al, Neurips 2017

[3] SphereFace: Deep Hypersphere Embedding for Face Recognition, Liu et. al, CVPR 2017

[4] ArcFace: Additive Angular Margin Loss for Deep Face Recognition, Deng et. al, CVPR 2019
questions: Line 23:
The paper starts by citing a reference [28] where a phenomenon called neural collapse is mentioned. For example, it is not clear why would features collapse to within-class mean? What is the loss function being used to do so? Are class labels even used during training which leads to this equiangular tight frame? The reader needs to be given a high level/intuitive introduction to ETF at this stage of the paper even though these details may be described in [28]. It could be like, once a neural net is trained, [28] observes 4 properties NC 1,2,3,4, which are the properties for an ETF and we can describe what is NC 1/2/3/4.

Please clarify how the work is different from the papers which I mentioned or if it would be better compared to these in a given scenario.
limitations: yes, it has been addressed.
review_title: null

Meta-review: metareview: This paper examines the use of a random equiangular tight frame (ETF) as a replacement mechanism for the final classification layer in a deep neural network, and demonstrates experimental advantages in class-imbalanced training scenarios.

Reviewers gave drastically different assessments of this paper, with ratings ranging from reject to weak accept.  The authors provided extensive responses to all reviewers, and Reviewer amWi participated in an extended discussion with the authors.  Author responses directly addressing concerns raised by other reviewers, such as pointing to ImageNet results in response to Reviewer Nj4c asking for such experiments, appear not to have received subsequent engagement from reviewers.

The Area Chair has taken an detailed look at the paper and the entirety of the discussion, and agrees with Reviewer amWi's assessment.  The work provides an interesting examination of ETF as a novel mechanism to address class imbalanced training; the contributions meet the bar for acceptance to NeurIPS.

Reviewer amWi makes several suggestions regarding presentation of the main contributions as well as additional papers for citation and discussion, which the authors may want to take into consideration when preparing the final version of the paper.
confidence: Certain
Decision: Accept