Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Feature normalization has been a common practice. In order to justify neural collapse for normalized features, this paper formulate an unconstrained feature model (UFM) with spherical constraints. Theoretical results on the neural collapse global optimality and the benign landscape are performed. Experiments are conducted to show that UFM with feature normalization has better training and collapse performances than feature regularization.
strengths_and_weaknesses: Strengths:

-	The motivation to interpret feature normalization from the neural collapse perspective is interesting. 

-	The paper is well organized and presented. 

-	Theoretical work is sound. 

Weaknesses:

-	Limited contribution. The first theoretical result, the neural collapse global optimality of Eq. (3), is a minor extension of Theorem 1 in [1], which also has a spherical constraint. Besides, as stated by the author, the problem with spherical constraint in Eq. (3) has the same global solution as the one with ball constraint in [2,3]. But the feasible region of ball constraint in [2,3] is much larger. So, the result in this paper is less significant. 

-	In order to verify the theoretical results (Theorem 1 and 2) in this paper, the authors show that UFM with feature normalization has faster training and better collapse than feature regularization. But, the same theoretical results, including the same global optimality of neural collapse conditions and the same benign landscape properties, are also applicable to UFM with regularization [4], and UFM without either constraint or regularization [5]. In this case, how to explain the advantage of normalization over regularization in the experiments?

[1] Lu et al., Neural collapse with cross-entropy loss,

[2] Graf et al., Dissecting supervised contrastive learning, ICML 2021,

[3] Fang et al., Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training, PNAS,

[4] Zhu et al., A geometric analysis of neural collapse with unconstrained features, NeurIPS 2021,

[5] Ji et al., An unconstrained layer-peeled perspective on neural collapse, 

questions: For practical application, one is more interested in generalization ability. Regularization has been known as an effective scheme to improve generalization. It is not clear about this for feature normalization. So, I think the authors should compare validation/test accuracy in Figure 4 and 5. 
limitations: There are discussions about the limitations of this work. 

No concern about negative social impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper considers the landscape of the optimization problem when training neural networks with normalized features under the cross-entropy loss. Under the assumption of the unconstrained feature model, the authors reformulated the problem as a Riemannian optimization problem, and showed that (1) global solutions of the problem satisfy the neural collapse properties and (2) the optimization problem has a benign global landscape (i.e. local minimizers are global minimizers, and all other critical points are strict saddle points).
strengths_and_weaknesses: __Quality and Clarity__: The paper is written clearly, and I enjoyed reading the motivations that builds up nicely to the main theorems. The experiments also support most of the author's claims well.

__Originality and Significance__: I personally do not work in this field, so I will only provide some brief observations on significance of results, while leaving more discussions to other expert reviewers: The proof of theorem 1 on neural collapse properties is not too different from existing techniques cited by this paper. Theorem 2, on the other hand, is nontrivial, and understanding the global landscape is indeed useful for developing guarantees for algorithms like Riemannian SGD.
questions: This is probably a minor question regarding the presentation of Figure 2.

In the end of section 3, the "Limitations on the assumptions of the feature dimension d" part, the authors stated that "We believe the bound can be improved (from d>N) to d>K+1, .... corroborated by our experimental results (See Figure 2)". 
 - I am expecting Figure 2 to show an experiment with K+1<d<N, which would empirically support the author's claims that we do not necessarily need d>N.
 - But instead, it looks like the authors chose d=100, n=5 for that Figure, which still guarantees d>n, so it does not really corroborate the claim above. Am I missing something, or should the graph be modified?
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper provides theoretical justification of the neural collapse phenomenon for normalized features. The proof is based on the unconstrained feature model (UFM) and the resulting optimization problem is analyzed over the Riemannian manifold by constraining features and classifiers prototypes over the sphere. The paper basically shows that critical points for the optimization (i.e. potential minima) are saddles (i.e. negative curvature critical points) while all others are global minimizers for which the neural collapse properties hold. The paper concludes with an empirical evaluation confirming the proposed theory.

strengths_and_weaknesses: STRENGTHS:

1) Interesting formulation of neural collapse in the case in which features are normalized. Feature normalization is a quite classic strategy/practice that has been used in several deep learning contexts (i. e. face recognition) in which maximal separation between learned features is required and therefore it naturally matches with the concept of maximal separation which is inherent to the neural collapse phenomenon. The provided theoretical analysis is very useful. The fact that a standard practice has a convergent geometric discriminative structure is a clear strength of this paper. 

2) The oblique manifold formulation is interesting and it appears to be well suited to the problem.

WEAKNESSES:

1) The paper is well written, but many parts seem to require further work. The main result (the proof according to the oblique manifold formulation) of the paper is neither introduced nor motivated and no intuitions are given. Although very interesting, it is hard for the reviewer and, possibly for the reader, to fully grasp the real contribution of the paper. The paper refers to a book (specifically to some exercises) from which the reader should possibly get some intuition and motivations. This part should be “softened”.

2) The related work section in its current shape is somewhat short and without specific comments. It should be improved by following a thread with respect to what is proposed. The reviewer agrees that feature normalization can improve separability, however the four cited works (i.e. [22,23,24,7]) are not discussed in this regard. In which way those four works achieve separability? What is the quality of the representation? Does it depend on the task addressed? Is there any trade-off in the achievement of separability? Important related works are also missing. Many papers, especially from the feature learning and visual search literature are completely missing [A,B,C,D,E,F]. In particular it seems that [A] is one the first works which studied classifier prototypes and their relationship with the learned features. The works [B,C] and [D] firstly introduced feature normalization and their feature separability (i.e., discriminative features) behavior. Finally, [E] and [F] seem to be the first papers to apply maximal separability in a simplex shaped structure (and other regular polytopes). As recent neural collapse literature used the term simplex ETF, what is the difference between the  simplex regular polytopes and a simple ETF? Are they equivalent?

3) The introduction should be better focused on the related work. On line 37 a large number of works are cited in a single shot as related to NC. However no comments are explicitly given. Given the diversity of the cited papers a discussion should be given.

4) What data is used in Fig.1 and Tab.1? It seems that it is not real data. How is the data generated?

5) The most important part related to the proof, should be improved. Some motivations and intuitions about the oblique manifold should be given. The cited book is addressing a dictionary learning problem. Although the affinity is clear, the paper in its current shape does not provide an intuitive access to the oblique manifold formulation. A potential reader of this paper would be forced to read too many references. The reviewer's suggestion is to increase the level of self-containment of the paper.

References

[A] Liu, W., Wen, Y., Yu, Z., & Yang, M. (2016). Large-margin softmax loss for convolutional neural networks. arXiv preprint arXiv:1612.02295.

[B] Ranjan, R., Castillo, C. D., & Chellappa, R. (2017). L2-constrained softmax loss for discriminative face verification. arXiv preprint arXiv:1703.09507.

[C] Wang, F., Xiang, X., Cheng, J., & Yuille, A. L. (2017, October). Normface: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international conference on Multimedia (pp. 1041-1049).
 
[D] Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., & Song, L. (2017). Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 212-220).

[E] Pernici, F., Bruni, M., Baecchi, C., & Del Bimbo, A. (2019, January). Maximally Compact and Separated Features with Regular Polytope Networks. In CVPR Workshops (pp. 46-53).

[F] Pernici, F., Bruni, M., Baecchi, C., & Del Bimbo, A. (2021). Regular polytope networks. IEEE Transactions on Neural Networks and Learning Systems.

questions: This form is intentionally left blank as questions are included in the previous form.

limitations: No negative societal impacts. Limitations are included into the Strengths And Weaknesses form.
review_title: null

Meta-review: metareview: The paper studies a matrix decomposition problem and shows the problem is in a strict-saddle type. All the reviewers tend to accept the paper. I recommend an acceptance.
confidence: Less certain
Decision: Accept