Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a method to combine the margin maximization in both Euclidean and angular spaces. More specifically, the method maps the labels to the vertices of a regular simplex as new labels.

Section 2: 
1) explains the motivation of the method, i.e. the two distances are the same once the data lie on the boundary of a hyper-sphere. 
2) more details about the method, including the vertices of a regular simplex, the case that uses the background class samples and the Dimension Augmentation Module(which is used for increasing the dimension, since the method requires the dimension of data is nearly the data size).

Section 3 and Appendix provide more details about the experiments:
1) feature representations learned by different loss function.
2) results of Open Set Recognition and Closed Set Recognition.

strengths_and_weaknesses: *The proposed method is interesting(also the DAM is interesting) and simple, and the simplicity brings several advantages (few hyper-parameters, proper acceptance regions, suit for unbalanced datasets), thus can be easily applied for classification problems. The experiments show that the method achieves good accuracy in several cases (especially the Open Set Recognition).
*The paper is well organized(especially the Section 2, which explains the main idea of the paper).

*Although the paper explains the benefits of the setting, it may need more explains or experiments or theorems to support the method. It's hard to convince me that the method works well now.
*The DAM seems need large size when the C is large.
questions: 1. Is there more explains about the hyperparameters such as, how there were chosen? Especially the u, which is the size of the hypersphere(it seems the u doesn't affect the classification?) . In contrast, is the m very sensitive? And what's the initialization?(what do the paper means by 'completely random'). 
2. Does DAM work better than the method in [1], or just the method in [1] is not suitable?

[1]Almost-equidistant sets.
limitations: No. The authors partly mentioned the limitations in the experiments (Section 3, especially 3.3.2) and the summary (Section 4). The limitation focus on the large-scale problem and the DAM.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Sorry for misunderstanding the paper. Revised review is here.

This paper proposed novel loss function for training deep neural network classifier, which can be generalizably applied to many domains such as evaluation for representation learning such as MoCo, BYOL or Masked Auto-Encoder. The loss function does not need any hyper-parameter tuning since the class centers are given as vertices of simplex and each data samples are trained to map to its corresponding class center. This way, traditional way of maximizing margin between samples from different classes and minimizing margin between samples from same class can be achieved in simple and effective way. While the limitation is the dimension of output feature of deep neural net should be larger or equal to C-1, authors mitigated the issue using DAM which can be further improved. In the experiment section, the authors gave intuition on how DSC clusters the data samples compared to other method, and how the learned representation can be semantically meaningful on unknown classes. The method achieved SOTA on most classification dataset which seems promising.
strengths_and_weaknesses: strengths: 
1. The paper is well-written and easy to follow.
2. The proposed method DSC generalizes to domains where the classification is required as a downstream task.
2. Proposed method achieves SOTA on various dataset

weaknesses:
1. More experiments on validating the proposed method seems required. For example, by getting learned representations from self-supervised pre-trained model (e.g., MoCo, BYOL, etc), the evaluation on downstream classification task with various loss functions will provide more strength on the proposed method. 
2. As authors pointed out in the contribution, experimental evidence on how this method is robust to unbalanced dataset needs to be addressed.
3. The solution of limitation seems not practical, since simple multi-layer perceptron is used between d-dimensional and C-1 dimensional vector which requires a lot of parameters of d and C-1 is large.
4. It would be better to show distance matrix of other baselines. Although baselines' distance between class centers from known classes exhibit difference, semantic property would be preserved. Reviewer is not sure if the learned representation with DSC is actually semantically meaningful since this is not preserved for known classes. Giving more explanations on this would be nice.
questions: 1. Does the length of the center vector u affect the performance of the DSC (or need tuning)? How does this value affect the performance?
2. In equation (4), does the dimension of f_i, and s_{y_i} match? Since it seems that f_i is d-dimensional vector while s_{y_i} is C-dimensional vector. Please clarify this part.
limitations: None.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies an interesting problem of fixing a set of equi-distance classifiers and then learning features to the corresponding class center (i.e., a vertice in a simplex). The specific form of the loss function is simple and aims to minimize the Euclidean distance between learned feature and the class center (on a hypersphere). This will automatically guarantee maximum separation of class centers and also lead to discriminative feature representation. The paper uses this method in both closed-set and open-set recognition, and shows some improvements.
strengths_and_weaknesses: I general, I think this is an interesting idea by taking advantages of the simplex and using it as a set of classifiers to guide the feature learning.

Strengths:

- The core idea is interesting. Connecting the simplex and maximum separated classifiers is interesting. Using it as an inductive bias to guide the feature learning is intuitive and seems to make lot of senses.

- The paper also considers the weakness of this method due to the limitation in dimension. The propose dimension augmentaion module aims to address this limitation (however, it inevitably introduces additional trainable parameters and it might not be a fair comparison to the other baselines any more).

- The paper considers both closed-set and open-set recognition and show some improvements over existing baselines. 

Weaknesses:

- As the authors have mentioned, the biggest weakness of this method is its restriction to feature dimension. Although the dimension augmentation module can partially address this, it does not solve the problem. For example, if you have a million-level classification (e.g., many face recognition datasets have this size), you have to map the feature to a similar scale of dimension, which is computationally intractable and introduces many additional overhead. This limitation largely restricts the application of the proposed method.

- Some important references are missing from the paper. What the simplex is actually doing is to encourage hyperspherical uniformity. This have a large body of works on this, e.g. [1], [2], [3]. I think they are closely related, since the simplex classifier is essentially a special case of hyperspherical uniformity (I understand that the construction procedure could vary).

- The empirical gain seems to be somewhat marginal, especially on closed-set recognition.

[1] Learning towards Minimum Hyperspherical Energy, NeurIPS 2018

[2] Learning with Hyperspherical Uniformity, AISTATS 2021

[3] Regularizing Neural Networks via Minimizing Hyperspherical Energy, CVPR 2020
questions: - I am wondering whether the authors only uses the proposed Euclidean loss in closed-set recognition (essentially center loss on a simplex)? It is a bit difficult to believe that it can actually outperform Softmax-based margin losses (say ArcFace). Using only Euclidean losses is actually ineffective in classification problems in my experience (it may work for clustering problems like face verification).

-  In the CIFAR-10 experiment, what is feature dimension in the proposed method? How the simplex is constructed if you are using the same feature dimension as the others? Are the authors using 9-dimension features?
limitations: n/a
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper introduces a Deep Simplex Classiﬁer (DSC) that maximizes the inter-class margins in both Euclidean and angular spaces for the open set recognition problem. Specifically, this method regards the vertices of a regular C-simplex in the (C-1)-dimensional feature space as the class centers of known C classes in the dataset and then encourages the class samples to be as close to their corresponding centers as possible. In cases that the feature dimension is smaller than C-1, a Dimension Augmentation Module (DAM) is further proposed to expand the feature dimension to C-1 to make it possible to construct the C-simplex. The open set and closed set recognition experiments are conducted on Mnist, Cifar10, SVHN, Cifar100, and Tiny ImageNet datasets while the face recognition experiments are conducted on MS1Mv2 dataset to show the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths:
1. The practice of regarding the vertices of a regular C-simplex in (C-1)-dimensional space as the C classes’ centers in the training dataset is new in open set recognition task, which has a clear geometric interpretation. 
2. The proposed method is clearly explained and easy to follow.
3. The authors conduct the illustration experiment to ease the understanding of the effect of DSC and evaluate the proposed method extensively on open set recognition, closed set recognition, and large-scale face recognition tasks.

Weaknesses:
1. The idea of maximizing the inter-class margins by explicitly manipulating the class centers is not new. For fixed centers, in Section 3.2 of [1], Do et al. propose to regard the basis vectors of the C-dimensional space as the C classes’ centers which are also the vertices of a C-simplex. The only difference is that the feature dimension in [1] is one larger than that in this paper. For learnable centers, Hayat et al. [2] also encourage the pairwise distances between any two class centers to be similar in a learnable way to bring uniformly separated centers in Euclidean space. Besides, Liu et al. [3] propose a minimum hyperspherical energy (MHE) regularizer to uniformly distribute the class centers on a hypersphere for maximizing the inter-class separability. These two methods share similar goals with Deep Simplex Classiﬁer (DSC) but do not require that the feature dimension is larger than C-2. As the relations between each pair of classes are different, is it necessary to make the pairwise distances between any two centers identical as in DSC or what is its advantage? I think more discussions and empirical comparisons are needed considering these strongly related existing works. 
2. The motivation of this paper is not clearly expressed yet. The authors claim that the Euclidean and Cosine distances would complement each other in abstract and finally choose to maximize the margin in both the Euclidean and angular spaces in introduction. However, the authors mostly emphasize the complexity of hyperparameters of existing methods in introduction while do not explain how would two kinds of distances complement each other and what is the key advantage of considering both spaces in open set recognition? Besides, the authors simply list and explain a series of existing works one by one in introduction with few summaries, which is kind of redundant, lacks logical coherence, and could be further improved. 
3. The proposed DSC is strictly restricted by the requirement that the feature dimension is not less than C-1. The Dimension Augmentation Module (DAM) would only alleviate this issue when C is small while with a large C, DAM will introduce a large number of parameters. In the face recognition experiment, with 18.6K identities, the number of parameters of the last linear layer in DAM is about 18.6K*18.6K \approx 346M, which is seven times more than the number of parameters of the ResNet-101 backbone (45M) and hinders the practical usage. I think a better solution is needed and essential for the general usage of DSC.
4. The authors claim that DSC does not have any hyperparameters but actually there is a hyperparameter u denoting the radius of the hypersphere. The authors simply set it to 64 following the practice in face recognition, which lacks theoretical and empirical guarantee as this paper does not focus on the task of face recognition and uses datasets beyond face recognition datasets. For this reason, an ablation study is needed to investigate the effect of the value of u. Moreover, the ablation study for the hyperparameters \lambda and m in Equation 5 in open set recognition is also needed. Besides, I also have some concerns about the experiments. First, for the open set recognition task, the proposed DSC uses a large amount of data from 80 Million Tiny Images dataset as the background class in Section 3.2.1, but the existing methods do not adopt this setting, for which I think the comparison in Table 1 is unfair. In line 269, a deeper network is employed for the Tiny ImageNet dataset, is it the same with the settings in existing works? Second, for the closed set recognition task in Table 2, do authors tune the hyperparameters in SphereFace, CosFace, and ArcFace for current classification datasets or directly follow the settings in original papers? Besides, the methods mentioned in the first comment should also be compared in the experiments since they are quite related with the proposed method.

[1] Do, Thanh-Toan, et al. "A theoretically sound upper bound on the triplet loss for improving the efficiency of deep distance metric learning." In CVPR. 2019.
[2] Hayat, Munawar, et al. "Gaussian affinity for max-margin class imbalanced learning." In ICCV. 2019.
[3] Liu, Weiyang, et al. "Learning towards minimum hyperspherical energy." In NeurIPS. 2018.

questions: 1. The novelty of the proposed method needs further clarification. More discussions and empirical comparisons between the proposed method and [1], [2], [3] are needed since they are quite related in methodology or goal. As the relations between each pair of classes are different, is it necessary to make the pairwise distances between any two centers identical as in DSC or what is its key advantage?
2. The motivation should be expressed more clearly. The authors mostly emphasize the complexity of hyperparameters of existing methods in introduction while do not explain how would two kinds of distances complement each other and what is the key advantage of considering both spaces in open set recognition? Besides, the presentation of introduction could be further improved.
3. How to better alleviate the feature dimension restriction for the proposed method without introducing too many parameters, especially when there are a large number of classes as in face recognition?
4. The ablation studies for hyperparameters u, \lambda and m in Equation 5 are needed. 
5. For the open set recognition task, the proposed DSC uses a large amount of data from 80 Million Tiny Images dataset as the background class in Section 3.2.1, but the existing methods do not adopt this setting, for which I think the comparison in Table 1 is unfair. In line 269, a deeper network is employed for the Tiny ImageNet dataset, is it the same with the settings in existing works? 
6. For the closed set recognition task in Table 2, do authors tune the hyperparameters in SphereFace, CosFace, and ArcFace for current classification datasets or directly follow the settings in original papers? 

limitations: No. The authors do not adequately address the feature dimension restriction of the proposed DSC. The proposed DAM introduces a large number of parameters when the number of classes is large.
review_title: null

Meta-review: metareview: This paper proposed to use least-squares loss functions in training deep neural networks. The main idea is to encode class means, whose mutual distances are equivalent. The method is simple but efficient. However, the similar idea has been widely used in multi-class classification (SVM and Fisher discriminant analysis) and spectral clustering. More specifically,  one reviewer commented that this work encodes class labels as high-dimensional vectors  similar one-hot, and then uses a least-squares loss. Although the authors did not admit this comment, but essecially this comment is indeed right. This idea has been used such as in the following references
1) Multicategory Support Vector Machines: Theory and Application to the Classification of Microarray Data and Satellite Radiance Data
Yoonkyung Lee, Yi Lin & Grace Wahba
2) Prevalence of neural collapse during the terminalphase of deep learning training  Vardan Papyana, X. Y. Hanb, and David L. Donoho

confidence: Certain
Decision: Reject