Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper analyzes novel loss functions to promote large margins during the training of a supervised (softmax-based) classification loss, under various conditions such as balanced and imbalanced number of samples per class. They propose a number of novel losses to promote large margins, and show analytical and empirical proof of the optimality of their proposed losses.
main_review: Strengths:

1. The paper is well-written, with nice motivations and clear theorems (no unnecessary "mathification" of proofs or writing).

2. Analytical results are clear and to-the-point, showing the motivation for their losses.

3. Empirical results are extensive, and show the benefits of their margin-promoting losses.

Weaknesses:

1. It would have been nice to show at least some empirical result for the case when k > d+1 (large number of classes compared to embedding dimensionality). Do their losses work "reasonably" in this case? Such a scenario may arise in modern deep learning methods when we train on datasets with many fine-grained classes e.g. ImageNet-21k.

2. It's slightly awkward that different losses are used for different datasets. How can we decide whether to use GM-Softmax, LM-Softmax or R_sm for a given dataset? Why not try all 3 together?

summary_of_the_review: The paper as it's well written, and shows novel analytical and empirical results. I think the losses are simple and easy to implement in practice.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes large-margin based loss functions. The authors formulate two types of margins, class- and sample-margins, and then analyze lower bounds of various margin-based losses in a unified framework to show that those losses are minimized by the optimizers of class-/sample-margins. While inspecting the existing margin-based losses, they also formulate two practical methods of sample-margin regularization and largest margin softmax loss to further enhance margins of classifiers. The experimental results on some image classification tasks demonstrate that those methods contribute to performance improvement, while favorably being compared with the other methods that induce large-margin classification.
main_review: ### PROS:
+ The authors proved that existing margin-based losses have the same lower bound and accordingly the same optimizers through connecting them to the optimizers of class-/sample-margins.
+ Some practical methods regarding classifier margins are introduced and empirically evaluated to produce favorable performance improvement on image classification tasks.

### CONS:
- Practical method.

The paper presents some practical methods. They, however, are less novel and their effectiveness are not sufficiently validated in the experiments.
Sample margin regularization in Eq.(3.4) is the classical loss such as used in MC-SVM [R1].
LM-softmax loss in Eq.(3.5) is a simple reformulation of the standard softmax loss; we can easily see that L_{softmax loss} = softplus(sL). While the softmax loss is lower-bounded by softplus, the LM-softmax loss is not by softplus but by normalization of features and classifier weights, due to which we can guess LM-softmax is less stable as shown in Table 2. 
Zero-center regularization is somewhat interestingly formulated for imbalanced learning. It, however, lacks comparison/analysis to the other related works of imbalanced learning such as [R2,R3]. Besides, the regularization would not work in the framework of cRT [R4] which re-trains classifiers on 'balanced' dataset; balanced mini-batch sampling would naturally render zero-centered classifiers even by the softmax loss.

In the experiments, the authors apply simple softmax loss for comparison. However, toward fair comparison, the softmax loss should be applied to normalized features and classifiers with the tuned temperature as in the other losses. Without comparison to the well-calibrated softmax loss, it is hard to recognize the effectiveness of the proposed method, especially LM-softmax.


[R1] K. Crammer & Y. Singer. On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines. JMLR, No. 2, pp.265-292, 2001.

[R2] H.-J. Ye et al. Identifying and Compensating for Feature Deviation in Imbalanced Deep Learning, arXiv:2001.01385.

[R3] B. Kim & J. Kim. Adjusting Decision Boundary for Class Imbalanced Learning, IEEE Access 2020.

[R4] B. Kang et al. Decoupling representation and classifier for long-tailed recognition. ICLR 2020.

- Theoretical analysis.

Considering the limited novelty of the presented methods as mentioned above, the paper's contribution would be rather theoretical materials. Though, those materials seem not to be so effective for further understanding/analyzing the large-margin losses.
As to normalization of features and classifiers, empirical analysis in Fig.1 is well-known in the literature of imbalanced learning [R2-R4] and Theorem 3.1 is rather trivial.
The analysis about the margin-based losses through Theorem 3.2 (Proposition 3.3) is somewhat interesting, but Theorem 3.2 is too rough to understand the roles of those losses deeply; actually, Theorem 3.2 is applicable even to the standard softmax loss with \alpha=1 and \beta=0, not only to the margin-aware ones. Thus, the theoretical part could not provide us with deep analysis specific to the margin-based losses such as XFace losses.

The paper lacks analysis and/or discussion about the case that some classes are correlated as frequently observed in real-world tasks. In that case, maximizing class-margin by force toward (k-1)-simplex classifier would be less effective, degrading intrinsic characteristics of those class categories; the correlated classes would intrinsically provide smaller margin.

### MINOR COMMENTS:
- In p.3, "intra-class compactness and inter-class separability" -> "inter-class separability and intra-class compactness"
- In Eq.3.5, -\frac{1}{s} -> \frac{1}{s}, and y_i -> y
- In Eq.3.6, \frac{1}{t} -> \frac{1}{s}
- In Table 1, no comparison to CE+R_{sm} nor LM-softmax.
- In Sec.4.3, it is unclear why the authors consider the task of person reID. How much effective is it for evaluating the margin-based losses?
- In p.13, Lemma 2.3 -> Lemma 2.1
- At the equations in top of p.15, "k \log" -> "\log".
summary_of_the_review: This paper presents several theoretical and practical materials regarding large-margin based losses.
They, however, are of limited novelty and are less contributive to theoretical understanding of margin-based losses.
Thus, my rating score of the paper is leaning toward weak rejection.


-- After rebuttal --

I appreciate the authors' effort to provide detailed response.
It nicely addresses my concerns about Theorem 3.2 and the practical methods.
Based on the rebuttal, I upgrade my score to 'weak accept' and recommend the authors to put those materials into a revised paper for clarifying the theoretical and practical contributions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper developed a principled mathematical framework for a better understanding and design of loss functions.

Based on the class and sample margins, the proposed method formulated the objective as learning towards the largest margins, and offer rigorously theoretical analysis as support.

For class balanced cases, this paper proposed an explicit sample margin regularization term and a novel largest margin softmax loss; for the class imbalanced cases, this paper proposed a simple but effective zero centroid regularization term, which restricts the centroid of prototypes to be zero.

Extensive experimental results demonstrate that the proposed strategy significantly improves the performance in accuracy and margins for various tasks.
main_review: Most of the former margin-based softmax methods (e.g. sphereface, cosface and arcface) are experiment-driven. By contrast, this paper offers rigorous theoretical analysis, which is the main contribution of this paper.

The writing of this paper is clear and all Theorems in the main paper are proved in the Supplementary Materials.

Limitations:
(1) The class margin (Eq 2.4), sample margin (Eq 3.4), generalized margin softmax loss (Eq 3.3), and zero centroid regularization (Eq 3.7) are not novel, but the theoretical explanation is nice.

(2) For face verification, which evaluation metric is employed to report the performance on IJB-C? We usually report TAR@FAR=1e-4/1e-5.

---------------post rebuttal----------------------

After reading the reply from the author, I have understood the evaluation metric used on IJB-C. I suggest the author change the AUC to TAR@FAR=1e-4/1e-5 in the camera-ready version.  The improvement is not obvious ($<0.5\\%$ IJB-C TAR@FAR=1e-4/1e-5) but it is ok for a theoretical paper. This paper can enhance our understanding of margin-based face recognition. Therefore, I still vote for acceptance.
summary_of_the_review: The theoretical explanation in this paper is nice and the experiments on person Re-Id and face verification confirm the effectiveness of the proposed method. In the NeurIPS 2021 review, this paper accepted two marginally above and one borderline result. In this ICLR submission, the main concern from the former reviewers has been fixed. In ICCV 2021, face-related papers have the highest rejection rate (>80%). There are two main groups of researchers in the face recognition community. One group of researchers focus on high performance under large-scale settings. Their papers look like experiment reports, but the performance is impressive. Another group of researchers focus on theoretical analysis. Their papers look more rigorous, but the performance is not satisfying and the proposed methods are not orthogonal to current state-of-the-art solutions. Even though I am from the first group, and I am not persuaded that the proposed method can obviously improve verification accuracy (in Table 4), I believe the theoretical explanation in this paper can enrich the research community and enhance the deep understanding of margin-based softmax loss designs. Both of the researchers from the practical and theoretical sides need to appreciate the contributions and novelties from the other side.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper develops a principled mathematical framework for better understanding and design of margin-based loss functions, where the principled optimization objective is formulated as learning towards the largest margins. In the proposed method, class margin and the sample margin are defined as the measure of inter-class separability and the measure of intra-class compactness. Furthermore, the sample margin regularization and zero-centroid regularization are introduced for the class-balanced case and the class-imbalanced case. Experimental results show the effectiveness of the proposed method on imbalanced classification, person re-identification, and face verification.
main_review: The strengths of the paper:
+ The paper is well-written.
+ The paper develops a principled mathematical framework and formulates it as the problem of learning towards the largest margins for better understanding and design of margin-based loss functions.
+ The sample margin regularization and zero-centroid regularization are proposed for the class-balanced case and the class-imbalanced case.
+ The paper offers rigorously theoretical analysis as well as extensive experiments to support the proposed method.

The weaknesses of the paper
- In the proposed Generalized Margin Softmax Loss (GM-Softmax), there are more parameters than a unified framework (Deng et al., 2019), how to effectively set them in the experiments?
- The improvement of the proposed is not significant. The proposed method results in both more larger class margin and more larger sample margin than the compared methods, however, the accuracy of the proposed method is slightly better than accuracies of the compared methods, and the current version does not analysis this.
summary_of_the_review: The paper is well-written and the novelty/contribution is significant and somewhat new. However, the weaknesses of the paper should be addressed to improve the paper.
review_title: null

Meta-review: This work presents a principled objective function for large margin learning. Specifically, it introduces class margin and sample margin, both of which it aims to promote. It also derives a generalized margin softmax loss which to draw general conclusions on the existing margin-based losses. The effectiveness of the proposed theory is empirically verified in visual classification, imbalanced classification, person re-identification, and face verification.

The reviewers initially raised some concerns, but most of them were well addressed in the rebuttal and convinced the reviewers. Specifically, pU1u was satisfied by authors' reply on Theorem 3.2 and the practical methods. pGzf appreciated clarifications around the evaluation metric used on IJB-C and believes this work can improve our understanding of margin-based face recognition. Finally, 3YiD had some reservations about number of parameters which got clarified by the authors. 

In sum, all post rebuttal ratings fall in the accept zone, and the reviewers find the paper interesting and insightful. In concordance with them, I recommend this paper for publication. Please make sure to include suggestions made by reviewers in the camera ready version.
Decision: Accept (Poster)