Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a closed-form design of the classifier layer, which encourages the maximal separation between any two classifiers on a hypersphere. The design is simple and straighforward, and it introduces a recursive algorithm to place $k+1$ classifiers in $k$ dimensional space. The experimental results demonstrate effectiveness in standard visual recognitio, long-tailed recognitio, OOD detection and open-set recognition.
strengths_and_weaknesses: Strengths:

- This paper is well written and structured. I generally enjoy reading this paper and find this idea quite interesting. Placing static classifiers and encourage their maximal separability is very natural and well motivated. The inductive bias from hyperspherical uniformity is encoded in the classifier layer which is used to generate corresponding gradients to update the whole network. Due to such a design, I think the learned features should also be quite discriminative (maybe with some sort of angular margin). I would suggest the authors to conduct some visualizations or Fisher discrimination analysis (say angular Fisher score in Appendix E of [1]) to demonstrate such advantages in a more intuitive way.

- The resursive algorithm is neat and efficient. I like its simplicity and the theoretical analysis that motivates it. The analysis is intuitive and should be correct as far as I'm concerned. I am wondering how it compares to gradient-based optimization. For example, you can directly minimize a matrix's hyperspherical energy to obtain its maximally separable column vectors, and then keep them fixed during the network training. Although it may not be the optimal design, I am curious how it compares empirically to the proposed method.

- I believe the static design of the classifier layer is of sufficient interest to the community and is of sufficient novelty, because it may be a potential solution to both encode better inductive bias and address the computational difficulty of training million-level number of classes. Besides the proposed static design, I would point the authors to a probabilistic design (in Section 6.4 of [2]) that also pre-specifies a set of fixed classifiers (which are randomly initialized) and then learns an orthogonal matrix that applies to them. I find the same idea can also be used in this paper, meaning that you learn an orthogonal matrix (using the same way as [2]) just to rotate/reflect these maximally separable classifiers. This does not change their pairwise distance / similarity, which means their hyperspherical energy stays the same. I think it will further improve the proposed method by granting it more flexibilty (while their maximal separability does not change).

- The paper demonstrates the effectiveness of the hyperspherical uniformity inductive bias in a number of recognition experiments. Its generalization ability is well verified. To me, the experiments should also be easily reproducible.


Weaknesses:

- While I like the general idea of static classifier design, the proposed method has an obvious weakness: the dimensionality of classifier scales linearly with the number of classifiers. This may be okay for benchmarks like CIFAR and ImageNet, but it still constrains its practical usage in large categorical training (say extreme classification). I can understand this particular algorithm design (ie, $k+1$ classifiers in $k$-dimensional space) is due to the fact that static assignment of maximally separable classifiers is very challenging when the dimensioanltiy is way smaller than the number of classifiers. I think this limitation, as a future direction to study, should be explicitly discussed in the paper.

- I appreciate the simplicity of the static classifier design, but the lack of flexibility (learnability) may limit its performance. Learning an orthogonal transformation (or even simpler, learning a rotation matrix) can be beneficial, as I mention in the third point in Strengths.

- As Lemma 1 shows, the inner product approaches zero as $k$ becomes larger. Does it mean that we can use orthogonal matrix (which is $k\times k$ instead of $k+1 \times k$) to replace the maximally separated matrix? It seems to be an interesting conclusion that one can take advantage of. This is not exactly a weakness, but it could be an interesring connection to orthogonality. Maybe the authors can comment / elaborate on this.

- The definition of maximally seprated matrix is a bit inappropriate in the sense that the equal inner product between any two classifiers does not always hold for matrices of any size. It only holds when the matrix is of the size $(k+1)\times k$. I suggest the authors to clarify this in order to avoid confusion.


Summary:
- I am in favor of the core idea (as well as the overall direction in static classifier design) and find this paper a solid work in general. I would vote for clear acceptance given the authors properly address my concerns.


[1] SphereFace: Deep Hypersphere Embedding for Face Recognition, CVPR 2017

[2] Orthogonal Over-Parameterized Training, CVPR 2021
questions: - My major questions are given in the "Strengths And Weaknesses" section.

- Some visualization on the learned features could really improve the paper. I am also curious about how the learned features will be given a fixed set of classifiers.

- While the classifiers are no longer learned, I am wondering whether the convergence performance will also be improved (since less number of parameters are needed to learn)? Some experiments on convergence speed (eg, iteration vs. testing accuracy) would be nice.
limitations: For technical limitations, I have discussed them in the "Strengths And Weaknesses" section. For potential negative societal impact, I am unaware of any.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes to bake maximum separation between classes as an inductive prior to a deep learning model. The proposed approach maximally separates class vectors, which is cleverly proven to have a closed form (assuming hyperspherical uniformity), and is implementable as a simple multiplication with a fixed matrix. Results are good on conventional tasks (Cifar, ImageNet), alongside long-tailed and open set problems. 
strengths_and_weaknesses: Strengths:
The method improves classification in the imbalanced, long tailed setting, alongside OOD and open set settings. 
The approach easily slots in with what appears to be any/most existing approach, as seen by the comprehensive experiments with various different methods on various problems. 
Good theoretical analysis. It appears correct to the best of my understanding.  
Utilizing a fixed matrix that you do not learn and improving results in a deep learning framework is an interesting approach with some novelty/originality to it. 
The result of the ablation with learnable prototypes is interesting, highlighting the importance of class separation as a solid inductive prior. 
How to run the code is very clearly laid out in README.md (albeit with minor mistakes). 
I was able to reproduce similar numbers to a subset of the results (table 1, alexnet) using the code with minor fixes. Did not have time to try more.

Weakness:
Diagrams are confusing and could be better labeled. For example, in Figure 1, it would be most useful to label what is $P_k$ and what is $p_k$.  Figure 3 could be much more clear (sub-title each subplot with amount of imbalance).
Some figures appear to contain the wrong description i.e. figure 4 "green" (there is no green). 
The code was troublesome to get running; many simple errors to fix. 
 
questions: Can this applied to other tasks, like segmentation?
Can an approach like this be modified to work well in an unsupervised setting? 
limitations: I'd be interested in knowing if this work can be applied to other different tasks, like segmentation, or be applied in other settings, like unsupervised. 
There are obviously improvements with this method, but when I quickly ran the code a couple of times, there seemed to be a large amount of variance of the test performance, exp in the imbalanced setting. Calculating some error bars/confidence intervals would be helpful.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes to integrate a closed-form structural prior in deep learning architectures, which has a clear geometric interpretation. For any neural network designed for classification, hence having a final softmax layer, the contribution takes the form of a fixed (non-learnable) matrix multiplication before the computation of the softmax. For a classification problem with K classes, the construction of the matrix consists of deriving K vectors on a (K-1)-dimensional hypersphere that are maximally separated. The authors provide a simple recursive procedure to construct such a matrix that only needs to be called once before training. The proposed contribution is evaluated on different learning tasks, showcasing its relevance in various common benchmarks.
strengths_and_weaknesses: The main strength of the paper is to provide a simple geometric interpretation of maximum class separation within the context of neural architecture design. Contrary to many engineering tricks used in deep learning to stabilize training or improve performances, the presented contribution does not contain any hyperparameter that needs to be fine-tuned. The originality and quality of the proposed contribution lies in the fact that the authors addressed the question of maximum separation from a theoretical point of view, and proves that their matrix design is relevant when followed by a softmax activation function. The recursive procedure used to construct the matrix closely resembles an algorithm that could be used to build an orthonormal basis for a vector space, except that in the presented case, the class vectors are required to have a fixed pairwise scalar product (equal to $-\frac{1}{K}$) instead of being orthogonal to each others. The experiments are well-chosen to prove the significance of the contribution across different challenging learning tasks, and authors focus on proving that their contribution improves on many baselines, rather than seeking for a single state-of-the-art metric in a very specific setting. Additionally, the authors shows that making their matrix learnable, either with their initialization or with random initialization, can even degrade performances, thus showing the optimality of their contribution to a certain extent. Such paper shows that it is possible to make consistent improvements in neural architectures when the chosen operations are motivated by strong theoretical arguments, somehow deviating from the traditional viewpoint that neural networks are differentiable hence optimizable black box functions. One criticism that could be made is that settings where the proposed contribution falls short are only mentioned in one sentence at the conclusion.
questions: Since the authors experimented with various learning tasks to showcase the relevance of their contribution, it could be of interest to additional information on the limitation of the proposed contribution.
Can the authors provide examples showcasing such limits ?
limitations: The authors stated that their work is limited to supervised settings, and cannot be applied to settings that require relational information between classes. However, in the case of unsupervised learning with variational-auto-encoders, could such a matrix be integrated at the top of the encoder for better latent space covering ?
Answering such a question could additionally broaden the significance of the contribution, since performing clustering in the latent space of an auto-encoder can provide additional information that can be very valuable in a few-shot learning setting.
Generally speaking, even if the construction of the matrix is somehow tied to a classification problem with $K$ classes, the fact that the matrix impose a geometric structure in the resulting output space should also be beneficial for unsupervised learning.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the author explores how to achieve maximizing class separation by directly adding a pre-calculated layer into the deep neural networks. The weights of the layer are solved in closed-form by a recurve algorithm. The effectiveness of the maximally separable vectors is verified on various datasets from long-tailed recognition, out-of-distribution and open-set recognition tasks.
strengths_and_weaknesses: Strengths
- The topic of maximum class separation is interesting.
- The experimental evaluation is adequate.
- The paper is well written and easy to follow.

Weaknesses
- Limited novelty and contribution. To my knowledge, the maximum class separation has been discussed by a few works [1-2]. Moreover, the proposed method is similar with [1], and the matrix estimation in [1] does not need recursive calculation.
- Lack of insights from the experiments. The experiments mainly focus on the performance evaluation. 

[1] Do We Really Need a Learnable Classifier at the End of Deep Neural Network.

[2] Dissecting Supervised Contrastive Learning.

questions: - As shown in Table.1, the performance gap of adding maximum separation layer over baseline enlarges on more imbalanced data. However, the phenomenon does not hold in Table. 2. More explanations are needed.
- Why the proposed methods can benefit OOD detection and OSR recognition? More explanations and evidences are needed.

limitations: Yes.
review_title: null

Meta-review: metareview: This paper aims at introducing a criterium for class separation. The paper demonstrates high performance, by proposing an affine transformation of the canonical embedding of labels, which lead to a maximal separation between those new vectors.
Given the simplicity and good numerical results, I recommend accepting this paper; however, the minor revisions suggested by reviewer BhPp needs to be addressed in the camera-ready version.
confidence: Certain
Decision: Accept