Learning Towards The Largest Margins
Authors: Xiong Zhou
Xianming Liu
Deming Zhai
Junjun Jiang
Xin Gao
Xiangyang Ji
Venue: ICLR 2022 Poster (2022)
Keywords: loss function design
margin-based loss
classification

Abstract: One of the main challenges for feature representation in deep learning-based classification is the design of appropriate loss functions that exhibit strong discriminative power. The classical softmax loss does not explicitly encourage discriminative learning of features. A popular direction of research is to incorporate margins in well-established losses in order to enforce extra intra-class compactness and inter-class separability, which, however, were developed through heuristic means, as opposed to rigorous mathematical principles. In this work, we attempt to address this limitation by formulating the principled optimization objective as learning towards the largest margins. Specifically, we firstly propose to employ the class margin as the measure of inter-class separability, and the sample margin as the measure of intra-class compactness. Accordingly, to encourage discriminative representation of features, the loss function should promote the largest possible margins for both classes and samples. Furthermore, we derive a generalized margin softmax loss to draw general conclusions for the existing margin-based losses. Not only does this principled framework offer new perspectives to understand and interpret existing margin-based losses, but it also provides new insights that can guide the design of new tools, including \textit{sample margin regularization} and \textit{largest margin softmax loss} for class balanced cases, and \textit{zero centroid regularization} for class imbalanced cases. Experimental results demonstrate the effectiveness of our strategy for multiple tasks including visual classification, imbalanced classification, person re-identification, and face verification.

Published as a conference paper at ICLR 2022
LEARNING TOWARDS THE LARGEST MARGINS
Xiong Zhou's Xianming Liu 122 Deming Zhai', Junjun Jiang" Xin G Xiangyang Ji
Harbin Institute of Technology Peng Cheng Laboratory 'King Ahdullah University of Science and Technology $ Gaollnn School of Artificial Intelligence, Renmin University of China TTsinghua University
ABSTRACT
One of the main challenges for feature representation in deep learning-basee clas- sification is the design of appropriate loss functions that exhibit strong discrimina- tive power. The classical softmax loss does not explicitly encourage discriminative learning of features. A popular direction of research is to incorporate margins in well-established losses in order to enforce extra intra-class compactness and inter- class separability, which, however, were developed through heuristic means, opposed to rigorous mathematical principles. In this work, we attempt to address this limitation by formulating the principled optimization objective as learning towards the largest margins. Specifically, we firstly define the class margin as the measure of inter-class separability, and the sample margin as the measure of intra-class compactness. Accordingly, to encourage discriminative representation of features, the loss function should promote the largest possible margins for both classes and samples. Furthermore, we derive a generalized margin softmax loss to draw general conclusions for the existing margin- losses. Not only does this principled framework offer new perspectives to understand and interpret ex- isting margin-based -baeed losses, but it also provides new insights that can guide the design of new tools, including sample margin regularization and largest margin softmax loss for the class-balanced case, and zero-centroid regularization for the class-imbalanced case. Experimental results demonstrate the effectiveness of our strategy on a variety of tasks, including visual classification, imbalanced classifi- cation, person re-identification, and face verification.
INTRODUCTION
Recent years have witnessed the great success of deep neural networks (DNNs) in a variety of tasks, especially for visual classification (Simonyan & Zisserman, 2014; Szegedy et al., 2015; He et al., 2016; Howard et al., 2017; Zoph et al., 2018; Touvron et al., 2019; Brock et al., 2021; Dosovitskiy al., 2021). The improvement in accuracy is attributed not only to the use of DNNs, but also the elaborated losses encouraging well-separated features (Elsayed et al., 2018; Musgrave et al., 2020).
In general, the loss is expected to promote the learned features have maximized intra-class com- pactness and inter-class separability simultaneously, so as to boost the feature discriminativeness. Softmax loss, which the combination of a linear layer, a softmax function, and cross-entropy loss, is the most commonly-used ingredient in deep learning-bait based classification. However, the softmax loss only learns separable features that are not discriminative enough (Liu et al., 2017). To rem- edy the limitation of softmax loss, many variants have been proposed. Liu et al. (2016) proposed a generalized large-margin softmax loss, which incorporates a preset constant m multiplying with the angle between samples and the classifier weight of the ground truth class, leading potentially larger angular separability between learned features. SphereFace (Liu et al., 2017) further improved the performance of L-Softmax by normalizing the prototypes in the last inner-product layer. Sub- sequently, Wang et al. (2017) exhibited the usefulness of feature normalization when using feature vector dot products in the softmax function. Coincidentally, in the field of contrastive learning, Chen et al. (2020) also showed that normalization of outputs leads to superior representations. Due to its effectiveness, normalization on either features or prototypes or both becomes a standard procedure in margin- based losses, such as SphereFace (Liu et al., 2017). 201 8b;a) and ArcFace (Deng et al., 2019). However, there is no theoretical guarantee provided yet.
SeaFaace/M-Softmaaa (Wang et al.,
"This work was done as intern at Peng Cheng Laboratory. Corresponneecce to: Xianming Liu <esxm @ bht.ccuu.cmm.

---

Published as a conference paper at ICLR 2022
Despite their effectiveness and popularity, the existing margin- based losses were developed through heuristic means, as opposed to rigorous mathematical principles, modeling and analysis. Although they offer geometric interpretations, which are helpful to understand the underlying intuition, the theoretical explanation and analysis that can guide the design and optimization is still vague. Some critical issues are unclear, e-g., why is the normalization of features and prototypes necessary? How can the loss be further improved or adapted to new tasks? Therefore, it naturally raises a fundamental question: how to develop principled mathematical framework for better understanding and design of margin-based loss functions? The goal of this work is to address these questions by formulating the objective as learning towards the largest margins and offering rigorously theoretical analysis as well as extensive empirical results to support this point.
To obtain an optimizable objective, firstly, we should define measures of intra-class compactness and inter-class separability. To this end, we propose to employ the class margin as the measure of inter- class separability, which is defined as the minimal pairwise angle distance between prototypes that reflects the angular margin of the two closest prototypes. Moreover, we define the sample margin following the classic approach in (Koltchinskii et al., 2002, Sec. 5), which denotes the similarity difference of sample to the prototype of the class it belongs to and to the nearest prototype of other classes and thus measures the intra-class compactness. We provide a rigorous theoretical guarantee that maximizing the minimal sample margin over the entire dataset leads maximizing the class margin regardless of feature dimension, class number, and class balancedness. It denotes that the sample margin also has the power of measuring inter-class separability.
According to the defined measures, we can obtain categorical discrininativeness of features by the loss function promoting the largest margins for both classes and samples, which also meets to tighten the margin-based generalization bound in (Kakade et al., 2008; Cao et al., 2019). The main contributions of this work are highlighted as follows:
For a better understanding of margin- based losses, we provide rigorous analysis about the necessity of normalization on prototypes and features. Moreover, we propose a generalized margin softmax loss (GM-Softmax), which can be derived to cover most of existing margin- based losses. We prove that, for the leads to maximizing both class margin and sample margin under mild conditions.
case, learning with the GM-Softmax loss
We show that learning with existing margin-based based loss functions, such as SphereFace, NormFace, CosFace, AM-Softmax and ArcFace, would share the same optimal solution. In other words, all of them attempt learn towards the largest margins, even though they are tailored to obtain different desired margins with explicit decision boundaries. However, these losses do not always maximize margins under different hyper-parameter settings. In- stead, we propose an explicit sample margin regularization term and a novel largest margin softmax loss (LM-Sfftmax) derived from the minimal sample margin, which significantly improve the class margin and the sample margin.
We consider the class-imbalanced case, which the margins are severely affected. We provide sufficient condition, which reveals that, if the centroid of prototypes is equal to zero, learning with GM-Sofumax will provide the largest margins. Accordingly, we propose a simple but effective zero-centroid regularization term, which can be combined with commonly-used losses to mitigate class imbalance.
Extensive experimental results are offered to demonstrate that the strategy of learning towards the largest margins significantly can improve the performance in accuracy and class/sample margins for various tasks, including visual classification, imbalanced classifi- cation, person re-identification, and face verification.
2 MEASURES OF INTRA-CLASS COMPACTNESS AND INTER-CLASS SEPARABILITY
With labeled dataset D {(x, J (where x; denotes a training example with label ys, and ya E [1, k] = 21, ..., kt), the softmax loss for a k-classification problem is formulated as
N
exp(w (w 21)
N
exp( wy || 2 2|2 cos cos(O.p.))
log Σ/= (xp(u=f22) 21)
- log k
(2.1)
exp(1Iur un و|2||111 | co,)))
2

---

Published as a conference paper at ICLR 2022
where 2; (2,));) E R (usually & Σ d 1) is the learned feature representation vector; Φe denotes the feature extraction sub-network; W = (wi...., Wy) E Rd denotes the linear classifier which is implemented with linear layer at the end of the network (some works omit the bias and use an inner-product layer); fiy denotes the angle between 2; and wy; and || |/2 denotes the Euclidean norm, where w1,-..; uy be regarded the class centers or prototypes (Mettes et al., 2019). For simplicity, we use prototypes to denote the weight vectors in the last inner-product layer.
The softmax loss intuitively encourages the learned feature representation z; to be similar to the corresponding prototype wy, while pushing 2, away from the other prototypes. Recently, some works (Liu et al., 2016; 2017; Deng et al.. 2019) aim to achieve better performance by modifying the softmax loss with explicit decision boundaries to enforce extra intra-class compactness and inter- class separability. However, they do not provide the theoretical explanation and analysis about the newly designed losses. In this paper, we claim that loss function to obtain better inter-class separability and intra-class compactness should learn towards the largest class and sample margin, and offer rigorously theoretical analysis as support. All proofs can be found in the Appendix A.
In the following, we define class margin and sample margin as the measures of inter-class separabil- ity and intra-class compactness, respectively, which serve as the base for our further derivation.
2.1 CLASS MARGIN
With prototypes wT, Wy Rdd the class margin is defined as the minimal pairwise angle distance:
WD! T
mc((w;)/m)) = min L(w,, wj) = arccos
max (
ww 11
W
(2.2)
where Z(w,, wy) denotes the angle between the vec- tors w, and wy. Note that we omit the magnitudes of the prototypes in the definition, since the magnitudes tend to be very close according to the symmetry prop- erty. To verify this, compute the ratio between the maximum and minimum magnitudes, which tends to be close to 1 on different datasets, shown in Fig.
MNIST
CIFAR-10
CIFAR-100 100
1.3
1.225
1.15
To obtain better inter-class separability, we seek the largest class margin, which can be formulated
1.075
max mo(( (: (/: max min w,) [m=
22
43
64
85
106
127
148
169
190
[w. (S)
Epochs
Since magnitudes do not affect the solution of the max- min problem, we perform Eg normalization for each w, to effectively restrict the prototypes on the unit sphere gd-1 with center at the origin. Under this constraint,
Figure 1: The curves of ratio between max- imum and minimum magnitudes of prototypes on MNIST and CIFAR- using the CE loss. The ratio is roughly close to (< 1.3).
the maximization of the class margin is equivalent to the configuration points on Sd-1 to maximize their minimum pairwise distance:
arg max min L(w,, wx) {w. C8 $#1
arg max min [w; -w w0llla,
(2.3)
{w); ifj
The right-hand side well known as the k-points best-packing problem on spheres (often called the Tammes problem), whose solution leads to the optimal separation of points (Borodachov et al., 2019). The best-packing problem turns out to be the limiting case of the minimal Riesz energy:
arg lim Σ
1
arg max min llw, wy 112- (w)f CBu-1 (nf
(2.4)
wlw- w
(w) {w){_, CBd-1
Interestingly, Liu et al. (2018) utilized the minimum hyperspherical energy as a generic regulariza- tion for neural networks to reduce undesired representation redundancy. When Wi...., Wa E Sd-1 k & d +++ 1, and f. > 0, the solution of the best- problem leads to the minimal Riesz t-energy: Lemma 2.1. For any wi...., w. 1 d 2 2, and 2 < k d + 1, the solution of minimal Riesz t-energy and k- k-points best best-packing configurations are uniquely given by the vertices of regular (I- 1)-simplices inscribed in gd-1, Furthermore, w.. offw - Vi + j.
3

---

Published as a conference paper at ICLR 2022
This lemma shows that the maximum of me((w;) )--))) is arccos( - when & I d + 1, which is analytical and can be constructed artificially. However, when k > d + 1, the optimal k-point configurations on the sphere Sd have no generic analytical solution, and are only known explicitly for handful of cases, even for d = 3.
2.2 SAMPLE MARGIN
According to the definition in (Koltchinskii et al., 2002), for the network f(æ; O W) WT (2(x): 1 + E that outputs & logits, the sample margin for (x, y) is defined
=
(). = f(æ)y max f(z); = w max w/
(2.5)
jfy
where z de (x) denotes the corresponding feature. Let nj be the number of samples in class and S, {i Vi j} denote the sample indices corresponding to class j. We can further define the sample margin for samples in class j as
%
min (2 9:)- EES
(2.6)
Accordingly, the minimal sample margin the entire dataset is Tmin min(n, 7k }- In- tuitively, learning features and prototypes to maximize the minimum of all sample margins means making the feature embeddings close their corresponding classes and far away from the others:
Theorem 2.2. For W1: ..., Wk: 21, ..... NN gd-1 (where ny 0 for each j E [L, the optimal solution {wo } 1 { arg max (u. Ymin is obtained if and only if {wt } max- w.:
imizes the class margin mo((w,) (w; the vectors {wy j maximizes W., wj.j j + y.
and z
where wy denotes the centroid of w b.
Two
As shown in the proof A, Theorem 2.2 guarantees that maximizing Tmin will provide the solution of the Tammes problem with respect to any feature dimension class number k. and both class- balanced and class-imbalance cases. When 2 I & S d 1, we can derive the following proposition:
Proposition 2.3. For any Wi,..., 21, i..... NN E S d 2 2, and 2 < k < d + 1, the maximum of Ymin is E which is obtained if and only if Vi + j. w. w - and z; Wy.
Theorem 2.2 and Proposition 2.3 show that the best separation of prototypes is obtained when max- imizing the minimal sample margin Ymin.
On the other hand, let Ly Præup, [maxy'4 f(x) f(x); - > denote the hard margin loss on samples from class j. and Lny. denote its empirical variant. When the training dataset is separable (which indicates that there exists f such that Tinin > 0), Cao et al. (2019) provided a fine- grained generalization error bound under the setting with balanced test distribution by considering the margin of each class, i.e., for 7 > and all f E F, with a high probability, we have
(f < max (x))
5 Te
Σ j
4 (F) ( [m
1 (F) +
(2.7)
Lzy
%
In the right-hand side, the empirical Rademacher complexity 4 (F) has big impact. From the 7o
perspective of our work, straightforward way to tighten the generalization bound is to enlarge the minimal sample margin Ymin. which further leads to the larger margin by for each class j.
3 LEARNING TOWARDS THE LARGEST MARGINS
3.1 CLASS-BALANCED CASE
According to the above derivations, to encourage discriminative representation of features, the loss function should promote the largest possible margins for both classes and samples. In (Mettes et al., 2019), the pre-defined prototypes positioned through data-indeepndent optimization are used to obtain a large class margin. As shown in Figure 2, although they keep the particularly large margin
4

---

Published as a conference paper at ICLR 2022
CIFAR-10 fi
CIFAR-IO w-f
CIPAR-10 w/ Ex
CIAM-110 w/o fx
CIFAR-IO CIFAR-10 wlo
100
CIEAR 100 w/ Bx
CIEAR 100 w/o lix
100
CIPAR-100 w/ lis
CIFAR- 100 who Ft
1.00
CHARR 100 wt Bx
CIRAR 100 wro tix
78
0.69
.55
55
33
33
10
10
40.25
32
112 146
16 32
16 64 80
96 112128144 190
(a) Test Accuracy
(b) Class Margin
(c) Sample Margin
Figure 2: Test accuracies, class margins and sample margins on CIFAR-I0 and CIFAR-100 with and without fixed prototypes, where fixed prototypes are pre-trained for very large class margins.
from the beginning. the sample margin is smaller than that optimized without fixed prototypes, leading to insignificant improvements in accuracy.
In recent years, in the design of variants of softmax loss, one popular approach (Bojanowski & Joulin, 2017; Wang et al., 2017; Mettes al., 2019; Wang & Isola, 2020) is to perform normaliza- tion on prototypes or/and features, leading to superior performance than unnormalized counterparts (Parkhi et al., 2015; Schroff et al., 2015; Liu et al., 2017). However, there is no theoretical guarantee provided yet. In the following, we provide rigorous analysis about the necessity of normalization. Firstly, we prove that minimizing the original softmax loss without normalization for both features and prototypes may result in very small class margin:
Theorem 3.1. Vs E (0, x /2. if the range of Wi.... Wk or Z1..., &N is Rd (2 S. k. S d + 1), then there exists prototypes that achieve the infimum of the softmax loss and have the class margin E.
normalized, the original softmax loss may produce an arbitrary small class margin E. As a corroboration of this conclusion, L-Softmax (Liu et al., 2016) and A-Softmax (Liu et al., 2017) that do not perform any normalization only do on prototypes, cannot guarantee to maximize the class margin. To remedy this issue, some works (Wang et al., 2017; 201 8a;;; Deng et al., 2019) proposed normalize both features and prototypes.
This theorem reveals that, unless both features and prototypes
A unified framework (Deng et al., 2019) that covers A A-Softmax (Liu et al., 2017) with feature nor- malization, NormFace (Wang et al., 2017), CosFace/AM-Softma (Wang al., 2018b:a), ArcFace (Deng et al., 2019) as special cases can be formulated with
m1, m2 and my!
exp(s(cos(m,,)) 0iy. + m2) - -ma))
L' = - -log
(3.1)
exp(s(cos(m)10)) m2) m2)) + /iv exp(s cos cos):;)
where Oij = (w,, 2.). The
setting usually guarantees that cos(m1 + m2)
my S cos m2 cos 8gg. my, and my is usually set to satisfy cos m2 2 2. Let a cos my and B = -mg < 0, then we have
exp(s(a cos Ozy, + +3)
4 2 - log exp(s(a cos Org. + B)) + Ejty. exp(s cos 0o.)
(3.2)
which indicates that the existing well-designed normalized softmax loss functions are all considered as the upper bound of the right-hand side, and the equality holds if and only if 0ig, = 0.
Generalized Margin Softmax Loss. Based on the right-hand side of (3.2), we can derive a more general formulation, called Generalized Margin Softmax (GM-Softmax) loss:
exp(s(a; cos p.y, + Ba)))
L = - log exp(s(ag cos 0.g, + B:2)) + Ejty. exp(scos 0,y))
(3.3)
where a,,, 022 A1 and 3i2 hyper-paramette parameters to handle the margins in training, which are set specifically for each sample instead of the same in (3.2). We also require that 0; 2 J. a;2 S aa. > 0, Bar 8o2 R. For class balance case, each sample is treated equally, thus setting 041 on,
5

---

Published as a conference paper at ICLR 2022
0z C22, 3n 31 and 822 B2, Vi. For class-imbalen iass-imbalanced case, the setting relies the data distribution, e.g., the LDAM loss (Cao et al., 2019) achieves the trade-off of margins with an =
a,2 = 1 and 3.1 3:2- = -Ony. -14 It is worth noting that we merely use the GM-Softmax loss as a theoretical formulation and will derive a more efficient form for the practical implementation.
Wang et al. (2017) provided lower bound for normalized softmax loss, which relies on the as- sumption that all samples are well-separated. ie., each sample's ' feature is exactly the same as its corresponding prototype. However, this assumption could be invalid during training, e.g., for binary
classification, the best feature of the first class & obtained by minimizing - log exp(swi exxexp(
exp(suu] 2)
is 111 110 | walla rather than W1. In the following, we provide a more general theorem, which does not rely on such a strong assumption. Moreover, we prove that the solutions {wo k j=11 (2ī J.V mini- mizing the GM-Softmax loss will maximize both class margin and sample margin.
Theorem 3.2. For class-balanen datasets, W1, Wk: ø1, NN &gd-1, d 2 2, and 2 S k S
d + 1, learning with GM-Softmux (where out a1, 0:2 (22, Bi1 31 and Bi2 32) leads to maximizing both class margin and sample margin.
As can be seen, for any 0 2 3 02 S 511$ > 0, and 31. 3 E I. minimizing the GM-Softmax loss produces the same optimal solution or leads to neural collapse (Papyan et al., 2020)), even though they are intuitively designed to obtain different decision boundaries. Moreover, we have
datasets, Wi...., Wg: 21 .... NN E & 2 2, and S k S d+ 1, learning with the loss functions A-Softmax (Liu et al., 2017) with feature normalization, NormFace (Wang et al., 2017), CosFace ( (Wang et al., 2018b) or AM-Softmax (Wang et al., 2018a), and ArcFace (Deng et al., 2019) share the same optimal solution.
Proposition 3.3. For dlass-balanee
Although these losses theoretically share the same optimal solution, in practice they usually meet sub-optimal solutions under different hyper-parameter settings when optimizing neural network, which is demonstrated Table 1. Moreover, these losses are complicated and possibly redundantly designed, leading to difficulties in practical implementation. Instead, suggest a concise and easily implemented regularization term and a loss function in the following.
Sample Margin Regularization. In order to encourage learning towards the largest margins, may explicitly leverage the sample margin (2.5) as the loss, which is defined as:
Rum (æ, y) = (w) z max w/ 2). jFy
(3.4)
Noticeably, the empirical risk 1/ E." Rsm (z,, 34) is lower-bounded surrogate of -Tmin i.e. N Em= Rum (z), while directly minimizing -Ymin is too difficult to optimize neural networks. When k. S d+1, learning with Rum will promote the learning towards the largest margins:
-Yfmin 2 1
Theorem 3.4. For class-balance datasets, Wi..... Wk. Z1,..., zn E s d 2 2, and 2 S k S ++ 1, learning with Rym leads to the maximization of both class margin and sample margin.
Although learning with R.m theoretically achieves the largest margins, in practical implementation, the optimization by the gradient-basee methods shows unstable and non-convergent results for large scale datasets. Alternatively, we turn to combine Rsm as a regularization or complementary term with commonly-used losses, which referred to as sample margin regularization. The empirical results demonstrate its superiority in learning towards the large margins, as depicted in Table 1.
Largest Margin Softmax Loss (LM-SSftmax)), Theorem 2.2 provides a theoretical guarantee that maximizing Ymin will obtain the maximum of class margin regardless of feature dimension, class number, and class balancedness. It offers straightforward approach to meet our purpose, i.e. learning towards the largest margins. However, directly maximizing 7min is difficult to optimize neural network with only one sample margin. As a consequence, we introduce a surrogate loss for balanced datasets, which is called the Largest Margin Softmax (LM-Softmax) loss:
exp(swā<) x) 8 log Σ exp(sw3xx
L(x, 8) =
log Σ exp(s(w; - T2)
(3.5)
j
which is derived by the limiting case of the logsumexp operator, i.e.. we have -7min lim 1 Iig Ejty, exp(s(10]22 [(])) Moreover, since log is strictly concave,

---

Published as a conference paper at ICLR 2022
derive the following inequality
1 logí Σ
1
exp(s(w32, -.
(.,,,))) 2,))) N (1:88 + log N.
(3.6)
Minimizing the right-hand side of (3.6) usually leads to that Ejzy. exp(s(w) w z))
constant, while the equality of (3.6) holds if and only if Litt, exp(s(wf= w w. x]) is constant. Thus, achieve the maximum of "imin by minimizing L(x, y: 8) defined in (3.5).
can be found that, LM-Softmax can be regarded as a special case of the GM-Softmax loss when aq or B2 approaches oc, which can be more efficiently implemented than the GM-Softmar loss. With respect to the original softmax loss, LM-Softmax removes the term exp( J z) in the denominator.
3.2 CLASS-IMEE IMBALANCED CASE
Class imbalance is ubiquitous and inherent in real-world classification problems (Buda al., 2018; Liu et al., 2019). However, the performance of deep learning-based classification would drop sig- nificantly when the training dataset suffers from heavy class imbalance effect. According to (2.7), enlarging the sample margin can tighten the upper bound in case of class imbalance. To learn towards the largest margins on Jass-imballance datasets, we provide the following sufficient condition:
Theorem 3.5. For class-balanced or -imbalanced datasets, wi,..., WK, ....... NN E &d-11 2 2, and 2 S k S. d + 1, if i w; O, learming with GM-Softmax in (3.3) leads to maximizing both class margin and sample margin.
This theorem reveals that, if the centroid of prototypes is equal to zero, learning with GM-Softmax will provide the largest margins.
Zero-centroid Regularization. As consequence, we propose a straight regularization term as follows, which can be combined with commonly-uset losses to remedy the class imbalance effect:
Rw Rw(w a|l
Σ
(3.7)
The zero-centroid regularization is only applied to prototypes the last inner-product layer.
4 EXPERIMENTS
In this section, we provide extensive experimental results to show superiority of our method on a variety of tasks, including visual classification, imbalanced classification, person RelD, and face verification. More experimental analysis and implementation details can be found in the appendix.
4.1 VISUAL CLASSIFICATION
To verify the effectiveness of the proposed sample margin regularization in improving inter-class separability and intra-class compactness, we conduct experiments classification on balanced datasets MNIST (LeCun et al., 1998), CIFAR-I0 and CIFAR-100 (Krizhevsky & Hinton, 2009). We evaluate performance with three metrics: 1) top-1 validation accuracy acc; 2) the class margin mats defined in (2.2); 3) the average of sample margins m.amp. We use a 4-layer CNN, ResNet-18, and ResNet-34 on MNIST, CIFAR-10, 10, and CIFAR-100 respectively. Moreover, some commonly- used neural units are considered, such as ReLU, BatchNorm, and cosine learning rate annealing. We use CE, CosFace, ArcFace, NormFace as the compared baseline methods. Note that CosFace, ArcFace, NormFace have one identical hyper-paramett 8, which is used for comprehensive study.
Results. As shown in Table 1, all baseline losses fail in learning with large margins for all in which the class margin decreases as 8 increases. There is no significant performance difference among them. In contrast, by coupling with the proposed sample margin regularization Rmm, the losses turn to have larger margins. The results demonstrate that the proposed sample margin regularization is really beneficial to learn towards the possible largest margins. Moreover, the enlargement class margin and sample margin means better inter-class separability and intra-class compactness, which further brings the improvement of classification accuracy in most cases.

---

Published as a conference paper at ICLR 2022
Table 1: Test accuracies (acc), class margins (mats) and sample margins (msamp) on MNIST. CIFAR- 10 and CIFAR-100 using loss functions with/without Rum in (3.4). The results with positive gains highlighted.
Dataset
MNIST
CIFAR-10
CIFAR 100
Metric
0.c
mals
Mammy
acc
Mata
Myamp
acc
Wets
79 Maany
CE
99.11
87.39
0.5014
94,12
81.73
0.6203
74.56
65,38
0.1612
CE 0.5R.n
99.13
95.41
1.026
94.45
96.31
0.9744
74.96
90.00"
0.4955
CosFace (s = 10)
98.98
95.93*
0,9839
94.39
96.00'
0.9168
74.44
83,31
0,4578
CosFace
99.06
93.24
0.8376
94.13
222
0.7955
73,26
79.17
0.3078
CosFace (# 64)
99.25
89,50"
0.7581
93.53
64.14
0.6969
73.87
72,56
0,2233
CosFace 10) R.5
99.16
95,56
1,033
94,42
96.26
0.9675
73.76
90.21
0,5089
CosFace (s 20) + 0.5
99.24
95.41
1,030
94.27
96.18'
0.9490
74.41
89.02
0.4780
CosFace
R.5
99.27
95.35
1.019
94.20
95.48"
0.9075
74.53
85.31
0.3817
ArcFace (s = 10)
99.05
94.64"
0.8225
94.50
91.23"
0.8501
73.96
76.91
0.4313
ArcFace 20)
99.11
90,84%
0.6091
94.11
53.98
0.5707
74.74
60.91
0,3010
AreFace (a = 64)
99.21
82.639
0.4038
ArcFace (+ 10) + 0.5
99.14
95,42"
1,034
94.21
96.27
0,9651
74.47
90.13
0,5143
ArcFace (s = 20) 0.5
99.19
91.38"
1,030
94.32
96.15
0,9571
74.64
88.73
0.4804
ArcFace 64) 0.5
99.14
95,29
1.019
NornFace (s 10)
99.06
94.34
0.7750
94,16
94.40"
0.8004
74.23
79.
0,4250
NorPFace (+ 20)
99.09
89. 272
0,5263
94.09
74.32
0.6001
73.87
77.47%
0,2498
NormFace (+ = 64)
99.00
82.08
0.2621
94.01
36.50"
0.2633
73.42
52.37"
0.0993
NornFacee (s = 10) + f.5m
99.16
95,38
1,034
94.23
96.28
0.9650
74.54
90,10
0,5160
NornFace (s 20) + 0.5 Run
99.19
95.37
1,031
94.38
96.17%
0.9519
74.75
88.86
0.4773
NomFacce (a 64) + R.m.
99.34
98.29
1.021
94,42
93.87
0.9508
74.33
76.02
0.3665
4.2 IMBALANCED CLASSIFICATION
To verify the effectiveness of the proposed zero-centroid regularization handling class-imbalanced effect, we conduct experiments on imbalanced classification with two imbalance types: long-tailed imbalance (Cui et al., 2019) and step imbalance (Buda et al., 2018). The compared baseline losses include CE. Focal Loss, NormFace, CosFace, ArcFace, and the Loss (LDAM) with hyper-parameter 8 = 5. We follow the controllable data imbalance strategy in (Maas et al., 201 1; Cao al., 2019) create the imbalanced CIFAR-10/-100 by reducing the number of training examples per class and keeping the validation set unchanged. The imbalance ratio p- max; n min, nı is used to denote the ratio between sample sizes of the most frequent and least frequent classes. We add zero-centroid regularization to the margin-based based baseline losses and the proposed LM--oftmax to verify its validity. We report the top-1 validation accuracy acc and class margin mais of compared methods.
Aware Margin
Table 2: Test accuracies (acc) and class margins (mass) on imbalanced CIFAR-10. The results with positive gains are highlighted (where denotes coupling with zero-centroid regularization term).
Dataset
Imbalanced CIFAR-10
Imbalanced CIFAR-100
Inhalance Type
long-tailed
step
long-tailed
step
Imbalance Ratio]
100
10
100
10
100
10
100
10
Metric
acc
mala
acc
mata
acc
Mal
acc
acc
mata
DCc
Mela
70.88 77.41*
88.17
79.639
62.21
76.50
88.06
82.24
140.38
64.73
60.42
66.249
42,36
60.32
56.88
62,82
Focal
66.30 74.14"
87.33 74.48"
60.55
63.31
84.49
75.16"
38.04
54.67"
60.09
59.29
41.90 55.98
57.84 55.72"
CosFace
69.28 58.77"
87.02
81.61"
53.64
19.78"
84.86
75.96
[34.91
4.731"
60.60
70.81"
40.36
0.764
47.56 8.559"
CosFacee*
69.52 91.90*
87.55
95.46"
62.49
95.86'
[85.59
92.
40.98
80.93
60.77
84.97
41.17
41.59
57.97 83.93
ArcFace
72.20 86"
89.00
85.23
62.48
54.29"
186.32
80.51"
422.7
13.22"
63.21
67.73"
41.47
0.497"
58.89 0.369"
AreFace
72.23 92.30*
89.22
96.23
64.38
93.51
86.65
96.23
44.68
56.60"
63.80
73:45"
44.26
32.10"
60.79 79.85
NormFace
72.37 62.72"
89,19
82.60"
63.69
51.00%
186.37
77.82%
|43.71
16.
11
63.50
71.26%
71
41.93
1,3639
55.885 21.32
NormFFace*
72.07 94.95*
89.30
94.50
64.07
93.06"
886.49
96.28
44.25
64.85"
63.81
79.85
44.51
36.30
60.22 80.83
LDAM
72.86 73.30"
88.92
88.19
63.27
61.42"
187.04
K5.21
(43.28
7.733"
63.62
73.19"
41.65
0.852
58.32 6.085"
LDAM
72.86 91.75"
96.26
96.04"
886.74
96.26"
45.23
70.96"
64.18
85.03"
43.26"
75.22"
I.M-Sofimas
665.32 4.420"
88.69
68 98.91
50.47
0.452
886.08
$2.20"
[41 52.52
4.500
63.26
68.31
41.53
0.4677
55.44 1.3729
LM-Softmax*
73.21 92.57
89.12
95.73
65.91
93.84
87.00
96.05
45.28
69.53'
63.77
$1.99
46.23
43.15
60.73 74.78
Results. As can be seen from Table 2, the baseline margin- losses have small class margins, although their classification performances are better than CE and Focal, which largely attribute to the normalization feature and prototype. We can further improve their classification accuracy by
8

---

Published as a conference paper at ICLR 2022
enlarging their class margins through the proposed zero-centroid regularization. as demonstrated by results in Table 2. Moreover, can be found that the class margin of our LM-Softmax loss is fairly low in the severely imbalanced cases, since it is tailored for balanced case. We can also achieve significantly enlarged class margins and improved accuracy by the zero-centroid regularization.
4.3 PERSON RE-IDENTIFICATION
We conduct experiments on the task of per- son re-identification. Specifically, we use the off-the-shelf baseline (Luo et al., 2019) as the main code verify the effectiveness of our proposed LM-Softmax. We follow the default parameter settings and training strategy, and train the ResNet50 with Triplet Loss Schroff al. (2015) coupling with the compared losses, including the Soft- max loss (CE), ArcFace, CosFace, Norm- Face, and our proposed L.M-Softmx. Ex- periments are conducted on Market- 1501 (Zheng et al., 2015) and DukeMTMC (Ris- tani et al., 2016). As shown in Table 3.
Table 3: The results on Market-1501 and DukeMTMC for person re-identification task. The best three results are highlighted.
Dataset
Market-1501
DukeMTMC
Method
|MAP
Rankl
R5
mAP Rank@1 I Rank
CE
822.8
92.7
97.5
73.0
83.5
93.0
ArcFace (a 10)
67.5
84.1
92.1
37.7
58.7
72.7
ArcFace (s 20)
79.1
90.8
96.5
61.4
78.3
88.6
ArcFace
80.4
92.6
97.4
67.6
83.4
91.4
CosFace 10)
68.0
84,9
92.7
60.6
73.1
CosPace 20)
80.5
92.0
97.1
64.2
81.3
89.7
CosFace = 644)
78.7
92.0
97.1
68.2
83.1
92.5
NormFiace (s = 10)
81.2
91.6
96.3
63.7
79.3
88.5
NormFace 20)
93.5
97.9
83.8
93.3
NormFace (a 6-1)
77.5
90.0
96.9
60.1
75.2
88.1
proposed LM-Softmax obtains obvi- ous improvements in mean Average Preci- sion (mAP), rank-1 1), and rank-5 (Rank @5) matching rate. Moreover, LM-Softmax exhibits significant robustness for different pa-
LM-Softmax 10)
92.8
97.1
72.2
85.8
92.4
LM-SSffmmxx (s = 20)
84.7
93.8
97.6
74.1
86.4
93.5
LM-Softmux (s 64)
93.9
98.1
86.6
93.5
rameters, while ArcPace, CosFace, and NormFace show worse performance than ours and are more sensitive to parameter settings.
4.4 FACE VERIFICATION
Table 4: Face verification results on IJCCC, Age-DB30, CFP-FP and LFW. The results with positive gains are highlighted.
We also verify our method on face verification that highly depends the discriminability of feature em- beddings. Following the settings in (An et al., 2020), we train the compared models on a large-scale dataset MSIMV3 (85K IDs/ 5.8M images) (Guo et al., 2016) and test on LFW (Huang et al., 2008), CFP-FP (Sen- gupta et al., 2016), AgeDB-30 (Moschoglou et al., 2017) and IJCC (Maze et al., 2018). We use ResNet34 as the backhone, and train it with batch size 512 for all compared methods. The comparison study includes CosFace, ArcFace, NormFace, and our LM-Softmax. As shown in Table 4, Ram (sample margin regularization) and RN (zero-centroid regularization) can improve the performance of these baselines in most cases. Moreover, it is worth noting that the results of LM-Softmax are slightly worse than ArcFace and CosFace, which is due to that in these large-scale datasets there exists class imbalanced effect more less. We can alleviate this issue by adding Rw. which can improve the performance further.
Method
UB-C Age-DR30 CFP-FP
LFW
ArcPace CosFace
99.4919
98.067
91.371
99,800
99,4922
98.033
97.300
99,800
LM Softmax
99.4721
97.917
97.057
99.817
ArcFacef
99.5011
98.117
97.400
99.817
ArcPacet
99,5133
98.083
97.471
99.817
CosFacef
99,5112
98.150
97.371
99.817
CosFacet
99.5538
97.900
97.500
99 800
I.M. SoMmaaa
99.5086
98.167
97.429
99,833
7 and denotes training with R.n and Ra, respectively.
5 CONCLUSION
In this paper, we attempted to develop a principled mathematical framework for better understanding and design of margin-based loss functions, in contrast to the existing ones that are designed heuris- tically. Specifically, based on the class and sample margins, which are employed as measures of intra-class compactness and inter-class separability, we formulate the objective as learning towards the largest margins, and offer rigorously theoretical analysis as support. Following this principle, for class-balance case, we propose an explicit sample margin regularization term and a novel largest margin softmax loss; for diaaaliialeanee case, we propose simple but effective cero-centroid reg- ularization term. Extensive experimental results demonstrate that the proposed strategy significantly improves the performance in accuracy and margins various tasks.

---

Published as a conference paper at ICLR 2022
Acknowledgements. This work was supported by National Key Research and Development Project under Grant 2019YFE0109600, National Natural Science Foundation of China under Grants 61922027, 6207115 and 61932022.
REFERENCES
Xiang An, Xuhan Zhu, Yang Xiao, Lan Wu, Ming Zhang, Yuan Gao, Bin Qin, Debing Zhang. and Fu Ying. Partial fe: Training 10 million identities on single machine. In Arxiv 2010.05222, 2020.
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In International Conference on Machine Learning, pp. 517-526. PMLR, 2017.
Sergiy V Borodachov, Douglas P Hardin, and Edward Saff. Discrete energy on rectifiable sets. Springer, 2019.
Andrew Brock, Soham De, Samuel Smith, and Karen Simonyan. High-peerformance large-scale image recognition without normalization. arXiv preprint arXiv:2102.0617 2021.
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-255 2599992599999 2018.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced
datasets with labelldistribution-awww margin loss. In Advances in Neural Information Processing Systems, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning. pp. 1597-1607. PMLR, 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Blass-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9677-977, 2019.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. Advances in neural information processing systems, 31, 2018.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 315-323. JMLR Workshop and Conference Proceedings, 2011.
Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb- Im: dataset and benchmark for large-scale face recognition. In European conference on computer vision, pp- 87-102. Springer, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and patlern recognition, pp- 770-778, 2016.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: S0Miv:/704.000 1704.04611, 2017.
10

---

Published as a conference paper at ICLR 2022
Gary B Huang. Marwan Mattar, Tamara Berg, and Eric Lealed-Miller. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in Real-Life Images: detection, alignment, and recognition, 2008
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Sham M Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction: risk bounds, margin bounds, and regularization. In Proceedings of the 21st International Conference on Neural Information Processing Systems, pp. 793-800, 2008
Bingyi Kang, Saining Xie, Mareus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yan- nis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In Eighth International Conference on Learning Representations (ICLR), 2020.
Vladimir Koltchinskii, Dmitry Panchenko, et Empirical margin distributions and bounding the generalization error of combined classifiers. Annals of statistics, 30(1):1-50, 2002.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Computer Science Department, University of Toronto, Tech. Rep, 1, 01 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2222222242222 1998.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convo- lutional neural networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning- Volume 48, pp. 507-516, 2016.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li. Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 212-220. 2017.
Weiyang Liu, Rongmei Lin. Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. Advances in Neural Information Processing Systems, 31:6222-622 2018.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella Yu. Large- scale long-tailed recognition in open world. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR 2017 (5th International Conference Learning Representations), 2016.
Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. Bag of tricks and strong baseline for deep person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Andrew Maas, Raymond Daly, Peter Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Brianna Maze, Jocelyn Adams, James A Duncan, Nathan Kalka, Tim Miller, Charles Otto, Anil K Jain, W Tyler Niggel, Janet Anderson, Jordan Cheney, et al. larpa janus benchmark-e: Face dataset and protocol. In 20/8 International Conference on Biometrics (ICB), pp. 158-165. IEEE, 2018.
Pascal Mettes, Elise van der Pol, and Cees G M Snoek. Hyperspherical prototype networks. In Advances in Neural Information Processing Systems, 2019.
Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp- 51-59, 2017.
11

---

Published as a conference paper at ICLR 2022
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In European Conference on Computer Vision, pp. 681-699. Springer, 2020.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In British Ma- chine Vision Association, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. arXiv preprint arXiv: 2019.
Ergys Ristani, Francesco Solera, Roger Zou, R. Cucchiara, and Carlo Tomasi. Performance measures and data set for multi-target, multi-camera tracking. In ECCV Workshops, 2016.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015.
Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1-9. IEEE, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test resolution discrepancy. In H. Wallach, H. Larochelle, Beygelzimer, F. d'Alché-Bus Buché-Bue,, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: 1.2 hypersphere embed- ding for face verification. In Proceedings of the 25th ACM international conference on Multime- dia, pp. 1041-1049, 2017.
Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verifica- tion. IEEE Signal Processing Letters, 25(7):926 930, 2018a.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5265-5274, 2018b.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9929-9939, PMLR, 2020.
L. L. Whyte. Unique arrangements of points on a sphere. The American Mathematical Monthly, 59 (9):606-611, 1952. ISSN 00029890, 19300972. URL http: hwtp///wwww////// jstor org/ wtwpg/////aale...eee stable/ 2306764.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In Proceedings of the IEEE international conference on computer vision, pp. 11 11-41124 2015.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tain long-tailed recogni- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16489-16499. 2021.
12

---

Published as a conference paper at ICLR 2022
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
13

---

Published as a conference paper at ICLR 2022
Appendix for "Learning Towards the Largest Margin"
A PROOFS
Lemma 2.1. For any Wi..., we S d and 2 k dd++ the solution of minimal Riest t-energy and k-points best- best-packing configurations are uniquely given by the vertices of regular (k- simppliss inscribed in S Furthermore, w w E-T Vi
Proof. See in Borodachov et al. (2019, Theorem 3.3.1).
Theorem 2.2. For Wi.... Wh, ø1, ... NN E gd-1 (where nj 0 for each j E [L, kl), the optimal Tmin is obtained if and only if (w; [w] max-
solution {wo Ji= (zt ( arg imizes the class margin me((w:) and 2
wy. w
Ti w 22 where w denotes the centroid of
the vectors {wyy: j maximizes wy, u # V.
Proof. According the definition of 7min: have
arg max max Ymin arg max max min wi! bi
2; max wo 2; プサ
arg max min max un
Ei 3.
2,
max un 2i 3#
arg max min max un 2, un 2
arg max min || wt 2|
where k = arg j w/ 2i, and k = arg minjæy. | w2ll2. Therefore, we have
l,,, wella Notice that w. --211 - 2xx122, then
wv; we
arg max max Ymin max min min [| wu lll = arg max min l|w, wy w,l12,
w
kz
ie., maximizing Tmin will provide the solution of the Tammes Problem, which also maximizes the class margin.
On the other hand, 2 maximizes w 2; maxjyy; W 21, i.e.,
2 arg max W TT y.
2: max w TT j7V
2.881-1
= arg max w TT
w TT zi
2;
2.684
" W
I
w y:
112
where wn, denotes the centroid of the vectors (wy : maximizes un], u.f. yi).
Proposition 2.3. For any wi,..., Wks 21...., ..., NN gd-1, d2 2. and 2 < k: < d + 1, the maximum of Ymin is k-1' which is obtained if and only if vi + j. w. w k-1- and z; = wy.-
Proof. Based on Theorem 2.2, the maximum of Amin is obtained if and only if [w: maximizes we, Wy i.e., wi! w. =- according to Lemma 2.3. At this time,
the class margin and z; Iw,, w
have z;
Wy ll
10 wo 21
uy my
Wy.
Theorem 3.1. VE E (0, x 2/2,. if the range of wi. wy or 2i,... nn is dd (2 < k S d 1), then there exists prototypes that achieve the infimum the softmax loss and have the class margin
14

---

Published as a conference paper at ICLR 2022
Proof.
With the softmax loss, the goal is to optimize the following problem
1
N log
exp(w W zi)
min (
N
j==1 exp(w)
Ve E (0, 311 we can easily obtain A: S d + vectors wi, ..., w/g on the unit sphere gd-1, such that the angle between any two of them is E E (0, 2)
(1) the domain of W1, wy is R", then let wj = swyy, and z; = wy In this way, we have " 2, > wf Zi. vj F yi. The infimum of softmax loss can be obtained by directly increasing s.
(2) If the domain of 2 2x is Rd, then let wi w/j and z; swa, In this way, have w. 2, > w 2, Vj A y/. The infimum of softmax loss can be obtained by directly increasing s.
In conclusion, without both normalization for both features and prototypes, the original softmax loss may produce an arbitrary small class margin E.
Theorem 3.2. For class-balanced datasets (i.e., each class has the same number of samples), W1. .... Wk: ø1, ..... nn S d 2 2, and 2 k I d learning with GM- CM--offmax (where an = an 0:2 = a2 B1 = 31 and 322 = 32) leads to maximizing both the class margin and the sample margin. More specifically, the optimal solution
N
xp(s(anw]]) 2; + 31))
(w)] 11 12
min E3d-1
log
N
exp(8(022)] z; B2)) Estu. exp(sw0f 2:)
have the largest class margin m arccos - and the largest sample margin Tunn E-I. The lower bound of the risk is loglexp(s(e) B1 - 022 - ()))) (k: 1) exeeep(=0p) - which is obtained if and only if vi + j. wo! w, = -1
k
+ an + 31))
k- and = Wy.
Proof. Since the function exp is strictly convex, using the Jensen's 's inequality, we have
exp(s(arw) 2, + B:))
L
log
exp(s(a2w) /1 2i + B2)) E,fi e 2i)
z, +1))
log
exp exp(s(a2 2, 32)) (k. 1) expl
# k- E j w/ zi)
Let w = 1
Wi, a 02 a1 B B2 B1, σ k-
I * and ā k- + aj then we have
L
log [exp(s(aww)] 3) + (k 1) exp(s(oww Sw TT s31)]
N
loglexp(so sa + +B) + (k 1) exp( - sloow Sw, 1l2 - -331)l
N
loglexp( sa + +3)) + (k- - 1) exp(-sllow - Swal12 s3311
where we use the facts that aw. 2; 2 a when a S 0, (ow - Swy, TT, 2 -llow Sw;ll2 when z; 8d-1 Due the convexity of the function log(1 + exp(ar + b)] (a 0). we the Jensen's
15

---

Published as a conference paper at ICLR 2022
inequality and obtain that
L 2 log
exp(s(a+ + (k: - exepp(
exp(-E-
low Swill2 - -331)
i=1
2 log
exp(s(a + (A: 1) expp(
k:
oo 8m0:13 -/81)
1:
log
exp(s(a ()) + exp((
k(k622 2kool ko ko?l| (3] sB1))
7:
> loglexp(s(a 8)) + (k 1) exx((=5((( (- =6++ + B1))]
= log
exp(s(a22 + 82 B1)) + (k - exp( - 8( k- + t. +311))
1
where in the second inequality we used the Cauchy-Schwarzz Schwarz inequality, and the third inequality is k-2 which holds since a1 2 2- based on that a & 28 + a1 2 2k-2'
According the above derivation, the equality holds if and only if Vi, w z = ... loo ow 112
w - 12
TT 1/4 + 2 ... 1 2,, w. y 2; 1,
-w 5wn, I wow- ll2 1ow 5w l|2
and w 0. The condition can be simplified as Vi + w w k-1 and 2; wy. when d and < K < +.
Proposition 3.3. For
datasets, Wi...., Wk: 21, .... 2N Sd-1, 2 2, and S
k S d 1, learning with the loss functions A A-Softmax (Liu et al., 2017) with feature normalization, NormFace (Wang el al., 2017), CosFace (Wang el al., 2018b) or AM-Sofimax (Wang al., (0), and ArcFace (Deng et al., 2019) share the same optimal solution.
Proof. A unified framework for A-Softmax with feature normalization, NormFace, LMLC/AM- m1, m2 and my, i.e.,
Softmax and ArcFace can be implemented with
exp(s(cos(m))) Oey. + m2) -ma) exp(s(cost))) (cosscomm m2) - ms)) Σj exp(s cos 0:j))
L' - llg
where 01j / (w), 2.). The setting of these hyper-parameters always guarantees that cos(m) + m2) m3 cos m2 cos oyy. my, and m2 is usually set to satisfy m2 2 1.- Let m2 and B = mm < 0, then we have
exp(s(a cos 0zy, + +))
L log
(A.1)
exp(s(a cos Osg. + B)) Ejtn. exp(s cos cos 0:j)
where the equality holds if and only if 0.g. 0.
According to Theorem 3.2, we know that the empirical risk of the loss function in the right-hand side of (3.2) has lower bound, then we obtain
N
exp(s(a + +1)
L Z - log
(A.2)
5 exp(s(a + B)) + Ejty. exp( F-T
The equality holds if and only if Vi + j. i w I and z, Wg. Since z; Wy. means kyy 0, indicating that the equality in (3.2) holds. Then the optimal solution the same for A A-Sofumax with feature normalization, NormFace, CosFace, and ArcFace.
Theorem 3.4. For class-balance datasets, Wk, 2i..... NN + 1, learning with R.m = - w. x + maxj w x leads to the maximization of the class margin and the sample margin.
2. and 2 < k <
16

---

Published as a conference paper at ICLR 2022
Proof. Let L(z,)) = w + max
w|
2,
k
+/= WA then have
U(z 1
Wyy 2i
T
max w 2;)
N
N
j$
1=1
N
W
T
V:
Zi
h:
Σ
un zi)
1=1
k
(1y,
w TT)
N(k
N
10y,
w 122
N(k
(A.3)
1=1
20 12
k
ww
w
k
1
|k(k k| 1 13)
k
k k-1
where the equality holds if and only if vi + j. u w
ww
and 2, = Wy:
Theorem 3.5. For class-balanced or -imbalanced cases, WT, Wkg Z1,..., EN E S d d 2, and 2 S k S d + 1, if Ei w, = 0, then learning with the GM-Sofimux loss (3.3) leads to maximizing both the class margin and the sample margin. More specifically, the optimal solution {w]) {2, NN has the largest class margin m (*) = arccos マー and the largest sample
margin Yiin = R-T- The lower bound of the risk is N En= log[exp(s(0s) + +B31 - 022 /322) +
(I 1a-11 1) cxp(=s( k-1 + an + 3a)),, which is obtained if and only vi + j. ww w = K- and 2; Wyz ie., the optimal solution maximizes that class margin and sample margin.
-1
exp(slann 2 2(1)) Proof. For the GM-Softmax loss L log exp(s(a.2 +.22))+++ exp(sw z.) let a; 0:22
0, S 0, 8 8,2 811. If Ei K w; 0, then we have
exp(s(a, w. y, + B:1))
L log
exp(s(a,22)) M1 2, + Bi2)) + Eitn jty. exp(suf22) 2,)
exp(s(a.) w. 21 + Bia))
2 - log
2 B:2)) + (k + 1) expl - Eitty swft 2:)
(A.4)
= log
exp(s(a.20 v + B:2)) (k 1) expl (oopp------0
R- (---TW w zi)
log
exp(sa, u n. + sB,) + (k- 1) exp( k
zi s(a, w 1. 2; + 3,1)) wh y1
2 log (exp(s(a)) + 3a 0:2 B12)) + (k: 1) exp( (00((1s100 (1/(k - 1) + on + Bn))]
where in the first inequality we used the Jensen's 's inequality, and the last inequality comes from the facts that aw y: 2; 2 a and - z; a; wi y: z; 2 - Qil
Therefore, we have the lower bound of the risk / E Li 2 + +M log(exp(s(av) + Ba 0:2 BBa)) + (k- 1) 1)exp(=s(x_) + aa + Ba))], where the equality holds if and only if V
17

---

Published as a conference paper at ICLR 2022
T w =w w w 4.+12 =..= =w zg. and w. y. 2 1. The condition can be simplified
as Vi # j. w w = T- and 2; Wy. when S and 2 s k 1
B MORE ANALYSIS
In this section, we provide analysis about the unified framework of margin-based losses in (3.2), Sample Margin Regularization, Largest-Margin Softmax (LM-Softmax) loss.
B.1 A UNIFIED FRAMEWORK
unified framework that covers A-Softmax (Liu al., 2017) with feature normalization, NormFace
(Wang et al., 2017), CosFacc/AM-Softmnn AM-Softmax (Wang et al., 201 8b;a) and ArcFace (Deng et al., 2019) as special cases can be formulated with hyper-parameters m1, m2 and my:
exp(s(cos(m)0)) + m2) - -ma))
E - -log exp(s (oppsccssm))0p + m2) - m3)) + Σj exp(s cos 0.j))
(B.1)
where O2g = Lww, z.). In the following, we provide the details of the derivation from (3.1) to (3.2)
For the parameter mi satisfies that cos (m; 0) S cos(0) in SphereFace Liu et al. (2017). There- fore, based on the definition of the multiplicativ-aanulaa operator, we have cos(m; + m2) S cos(fry, + m2). To better understand the theoretical optimal solution, we make the constraint that Big. E [0, 21. which is reasonable because the unique minimizer of these losses, like SphereFace, CosFace, and ArcFace, should satisfy 0g = 0, rather than belongs to m3.
As for m2, ArcFace did not analyze its range. Instead, we can easily derive that 0 < m2 2- Otherwise, the minimum of AreFace will be obtained at Pzy, = π, since cos(Oiy. + m2) S cos + m2) when m2 z. which is ridiculous. Therefore, for 0ig, m2 E [0, 5 we have cos(Or,, + cos Ozg. cos m2 sin Oig, sin sinm2 S cos m2 cos Oig. which is the main derivation from (3.1) (3.2).
B.2 ON THE SAMPLE MARGIN REGULARIZATION AND BEYOND
The sample margin regularization term in (3.4) actually encourages the feature representation = to be similar to the corresponding prototype wy. and push z away from the most similar one of the other prototypes. This concept is similar to contrastive learning, where the most similar of the other prototypes can be regarded as the hardest negative representation. And we also have
Ram (2 1) S - - = +
(B.2)
k
jøy
where the right side can be regarded as pushing = away from the centroid - Σ Ejty W, or pushing away from other negative representations. Intuitively, we can also use the right side of Eq. B.2 as a sample margin regularization.
B.3 MORE CLARIFICATIONS
As shown in the main paper. GM-Softmax loss, LM-Softmax loss, Sample Margin regularization, and Zero-centroid regularization serve different purposes. More specifically,
The GM-Softmax loss is only derived as a thcorctical formulation, which not used for practical implementation.
The LM-Softmax loss is tailored to obtain large margins with only one hyper-parameter. It can be used to replace popular margin-based losses, such as CosFace, and ArcFace, to obtain better discriminativeness of feature representations. Compared with NormFace Wang al. (2017), LM-Softmax achieves much better performance the task of person ReID, as shown in Table 3. This demonstrates that removing the term exp(sw (sw, x) in the denominator is helpful, which enforces LM-Softmax to have stronger fitting ability.
18

---

Published as a conference paper at ICLR 2022
(a) NormFace (68.50")
(b) CosFace (71.86")
(c) ArcFace (67.74")
(d) LM Softmmx (74.46")
Figure 3: Visualization of the learned prototypes (red arrows) and features (green points) using NormFace, CosFace, ArcFace and LM-Softmax on 5 for eight classes. The optimal solution of Tammes problem for N = 8 have the class margin 74.86" (Whyte, 1952), where the class margin of learning with the losses NormFace, CosFace, AreFace and LM-Softmax are 68.50", 71.86", 67.74 and 74.46,, respectively. We note that this phenomenon coincides with the recent popular concept neural collapse (Papyan al., 2020).
The sample margin regularization Ram serves as a general regularization term to signif- icantly improve the ability of learning towards the largest margins by combining it with the commonly-used losses. Sample margin is not new, but to the best of our knowledge, we are the first one use it in deep learning to obtain feature representations with inter- class separability and intra-class compactness. Although theoretically learning with Rum can achieve the largest margins, we verify by experiments that directly maximizing sample margin cannot optimize neural networks well complex datasets, such as CIFAR- 100, as shown in Table 5. can be found that learning with Ram suffers from the underfitting problem CIFAR-100 100, whose performance is much worse than CE. Alternatively, turn to use R.m as a regularization term, which can significantly improve the performance of commonly-used CE loss. These results demonstrate that using sample margin as the regularization term is more beneficial than using it as the loss. This is our new contribution to the classical sample margin.
The zero-centroid regularization Rw is specially tailored for class-imbalanced cases, which is only applied to prototypes the last inner-product layer. Therefore, it can be easily embedded into the DNN-based methods to handle class imbalance.
C EXPERIMENTS
In this section, we provide the experimental details, including datasets, network architectures, pa- rameter settings, analysis, and more results. All codes are implemented by PyTorch (Paszke al., 2019).
We first recall the sample margin regularization R.m = - + w z and the zero-centroid regularization R. = 1 Σt= w;ll2, which used enlarge margins for baseline methods. As for the trade-off parameter settings μ and in the following experiments. We use μ and X 100A denote the trade-off parameters for Rsm and Ry, i.e., L + R. and L 100AR respectively. In the following, we set μ 0.5, 1.0 for Pom, and X = 1, 5, 10, 20 for Rw.
C.1 TOY EXPERIMENT
We conduct toy experiment to show the inter-class separability and intra-class compactness using different losses, where we randomly generate prototypes W kxd (we set and = 8), and initialize features Z E R xd (we set N 10k). Our goal is to optimize both W and Z to learn the largest class margin and sample margin with different losses. According to the Tammes problem 74.86* (Whyte, 1952). The
for N 8, the optimal solution of W and Z satisfies that (W) number training epochs is set 500,000. We use cosine learning rate annealing with T 1m,x010,000, and SGD optimizer with momentum 0.9 and weight decay l 4.
19

---

Published as a conference paper at ICLR 2022
Tran
Tuun
Pock
122
5000
230
190
1530
2000
332
-24
-02
62
on
0.6
24
10
-24
-12
0.
02
0.
06
64
10
-04
-02
60
02
64
0G
10
(a) CosFace (s 64)
(b) CosFace (s
64) + 0.577mm
(c) CosFace (s 64) R.m
Text
1422
1422
213
1202
1202
153
1002
1003
402
402-
202
233
34
(d) CosFace (s = 64)
(c) CosFace (8 = 64) + Rgm
(f) CosFace (s = 64) + Ram
Figure 4: Histogram of similarities and sample margins for CosFace with/without sample margin regularization R.m on CIF (a-c) denote the cosine similarities between samples and their corresponding prototypes, and (d-f) denote the sample margins.
Results. We use green points and red arrows to denote the learned feature vectors and prototype vectors, respectively. As shown in Fig. 3, the learned prototypes are separated well with Norm- Face, CosFace, ArcFace, and LM-Softmax. Specifically, the class margin of learning with the losses NormFace, CosFace, ArcFace, and LM-Softmal are 68.50%, 71.86", 67.74% and 74.46', respec- tively. As we can seen, ArcFace has a smaller class margin (67.749) than the others, and the intra- class compactness for NormFace and CosFace is worse than LM-Softmax. The features in the blue box of Fig. 3(a) and Fig. 3(b) are not compact enough, but ArcFace and LM-Softmax do. Moreover, our proposed LM-Soffmax shows better performance in class margin and sample margins, where the learned prototypes have the class margin close to the theoretical optima, and the features are perfectly optimized be their corresponding prototypes.
C.2 VISUAL CLASSIHICATION
We introduce three metrics to evaluate whether a loss function owns good inter-class separabil- ity and intra-class compactness. The first one is the top-1 test accuracy generalization of the trained models. The second one is the class margin mais defined in Eq. (2). And the last one we define as the average of sample margins with cosine similarities, i.e.
measure the
wo (x.)
w (2,))
Msamp = N Σi= W., w,, [..)] maxj Tw, (a))) Then we experiments with 4-layer CNN, ResNet- 18 and R4 (He et al., 2016) on MNIST (LeCun et al., 1998), CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009), respectively. Moreover, some commonly-used neural layers are considered, such as ReLU (Glorot et al., 2011), BatchNorm (loffe & Szegedy, 2015), and cosine learning rate annealing (Loshchilov & Hutter, 2016).
Datasets.
We empirically investigate the performance of learning towards the largest margins on benchmark datasets including MNIST (LeCun et al., 1998), CIFAR- 10 and CIFAR- 100 (Krizhevsky & Hinton, 2009).
Training details. We use a simple CNN which consists of Conv(1, 32, 3) - BatchNorm (Loffe & Szegedy, 2015) - ReLU (Glorot et al., 2011) + MaxPool(2.2)) > Comy(3,, 64, 3) + BatchNorm - ReLU + MaxPool(2,2) - Linear() for MNIST, a ResNet-18 (He et al., 2016) for CIFAR-10 10, and 34 (He al., 2016) for CIFAR-100. The number of training epochs is set 100, 200 and
20

---

Published as a conference paper at ICLR 2022
Table 5: Test accuracies, class margins and sample margins on MNIST, CIFAR-10 and CIFAR-
100 using loss functions with/without sample margin regularization Ran where we simply set the regularization parameter to 0.5. The results with positive gains are highlighted.
Dataset
MNIST
CIFAR-I0
CIFAR-100 100
Motric
acc
Mete
Maamp
acc
Meta
Maamp
acc
Mal
Maamp
CE
99.11
87.39
0.5014
94.12
81.73
0.6205
74.56
63.38
0.1612
Run
95.38
96.28
0.9791
58.58
CE O.5RRa
99.13
95.41
1.026
94.45
96.31
0.9744
74.96
90.00
0.4955
CnsFace (s 5)
99.11
95.85
1.020
94.02
96.33
0.9619
75.37
84,20
0.5037
CusFace (s 10)
98.98
95,93
0.9839
94.39
96.00"
0.9168
74.44
83.31
0.4578
CasFace 20)
99.06
93.24
0.8176
94.13
91,22"
0.7955
73.26
77.
0.3078
CosFace (+ 40)
99.18
90.69"
0,7650
93.84
76.09"
0.7617
73.54
77.48"
0,2380
CosFace
89.50"
0.7581
64.14
0.6969
72.56
0.2233
CosFace (s 5)+ f.n
99.07
95.60"
1.036
94,20
96.32
0.9740
75.52
90.41
0.5230
CosFace 10) R.5
99.16
95.56
1,033
94.42
96,26'
0.9675
73.76
90.21
0.5089
CosFace (s 20) 0.5 Run
99.24
95.41
1.030
94.27
96.
0.9490
74.41
89 89.020
0.4780
CosFace 10) 0.5
95.41
1,026
95.93*
0,9238
74.58
86,91
0,4251
CosFace (a 64) U.5.Hen Hon
99.27
95.35"
1.019
94.20
95.48"
0,9075
74.53
85.31
0.3817
CusFace 5)+
99.15
95.59
1.032
94,38
96.35'
0,9817
75.18
90.44
0.5228
CaeFace (s 10) Rus
99.09
95.48
1.029
94.49
96.32
0.9770
73.93
90.36
0.5237
CosFace 20) Run
95.37"
1.028
96.24
0.9640
73.79
89.632
0,4958
CosFace (a 20) Run
99.12
95.38"
1.027
94.31
96.18"
0,9510
74.43
88.83"
0.4736
64) Run
87.83"
ArePace 5)
95.46"
0.9956
96.33"
0.9473
28.28"
ArcFace 10)
99.05
94.64"
0.8225
94.50
91,23"
0,8501
73.96
76.91 9
0.4313
ArcFace 20)
99.11
90.84
0.6091
94.11
53.98"
0.5707
74.74
60.91
0.3010
ArcFace
99.13
16.
0.4606
93.88
35.68"
0.3195
AreFace (s 64)
99.21
82.63
0,4038
ArcPace 5) R.5
99.00
95,59
1.034
94.17
96.32"
0,9731
74.72
90.37
0.5081
ArcFace 10) 0..
99.14
95.42
1.034
94.21
96.27
0.9651
74.47
90.13
0.5143
ArcFace
0.5
99.19
91.38"
1.030
94,32
96,15"
0,9571
74.64
88.73
0.4804
AreFace (s 40) 0.5
99.24
95.34
1.026
94.07
95.69
0.9434
64) 0.5
95.29
ArcFace 5)+ Ren
99.17
95.53"
1.030
94.40
96.35"
0.9825
74.85
90.41
0.5156
ArcFace 10) Run
99.09
95.37"
1.029
94.14
96.32"
0,9713
73.76
90.30"
0.5259
AreFace (s 20) Rsn
99.11
95.36
1.028
94.45
96.25
0.9676
74.61
89.65
0.50333
ArcFace
Ran
95.34
96.04
0.9621
ArcFace 64) Ren
99.13
95.30)
1.024
NormFace (s = 5)
99.03
95.68"
0.9836
94.34
96.34
0.9452
75.56
85.37
0.5076
NormFace 10)
94,34
0.7750
94.40%
74.23
0.4250
NormFace
99.09
89,279
0,5263
94,09
74.32
0.6001
73.87
77.47%
0.2498
NormFace (a =40)
99.06
85.44
0,3473
94.11
47.52"
0.3825
73.73
66.67
0. 1439
NormFace
82.08
0,2621
36.50"
0.2633
73.42
$2.37"
0.0993
NormFace 5) R..
95.55
NormFace (+ 100 0.5.R..
99.16
95,38"
1.034
94,23
96,28
0.96550
74.54
90.10"
0.5160
NormFace (s 20) 01.5 0.5
99.19
95.37"
L031
94.38
96.
0.9519
74.75
88.86"
0.4773
NormFace
R..
99.14
95.36"
1.026
95.59"
0.9495
74.48
84,78"
0.4181
NormFace (a (611) R.5.Ran
99.34
95.29
1,021
94,44
93.87"
0,9508
74.33
76,02"
0.3665
NormFaae 3)+ Ron
99.14
95.48
1.029
94,42
96.34
0,9798
74,89
90.45
0.5134
NormFace (s 10) Run
99.12
95.37"
1,028
94.31
96.32
0.9758
73.16
90.31
0.5183
NormFace
R.n
99.11
95.35"
1.028
94.16
96.28"
0.9656
74.23
89,72
0.5004
NormFace (8 = =10) Run
99.11
95.36
1.026
93.98
95.87"
0.9583
74.22
88,73
0.4731
NormFace
21

---

Published as a conference paper at ICLR 2022
3000
LM-Softmax (a 10) (b) LM-Softmax (s 20) LM-Softmax (s 10) (d) LM-Softmax (s 64)
Pame
1300
Tipe
300
-10
(c) LM-Softmax (s 10) (f) LM- L.M-Softmax (s 20) (g) LM-Softmax (s 40) (h) LM-Softmax 64)
Figure 5: Histogram of similarities and sample margins for LM- A-Sooftmax CIFAR-10. (a-d) denote the cosine similarities between samples and their corresponding prototypes, and (c-h) denote the sample margins.
250 for MNIST, CIFAR-10 10, and CIFAR-100, respectively. For all training, we use SGD optimizer with momentum 0.9 and cosine learning rate annealing (Loshchilov & Hutter, 2016) when T TMax is equal to the corresponding epochs. Weight Decay is set to 1 x 10-4 for MNIST, CIFAR-10, and CIFAR-100. The initial learning rate set to 0.01 for MNIST and 0.1 for CIFAR-10 and CIFAR- Moreoor,,,er, batch size is set to 256. Typical data augmentations including random width/height shift and horizontal flip are applied.
Baselines and hyper- parameter settings. We consider the baseline methods, including the commonly-used loss function CE, and margin-based loss functions NormFace, CosFace, and Ar- cFace with normalization for both feature vectors and class centers, and our proposed LM-Softmax loss. We have tuned their spper-prrrmeters for the best performance, and the specific settings are: for CosFace, we m 0.1; for ArcFace, we set m 0.1. To learn towards the largest margins, we boost them with the sample margin regularization, and the trade-off parameter is set to 0.5 and Moreover, we tune their identical arper-parameter s, and show them for a comprehensive study.
The test accuracy, class margin and the average of all sample margins are reported in Table As we can see, the baseline methods fail in learning large margins for all s, and there no sig- nificant difference in the performance of these losses. More specifically, the class margin decreased as s increases, while the losses with the sample margin regularization R.m usually remain the large class margins, and the class margins are close to the optimal results (arccos(-1/95) MNIST and CIFAR- 10, and arccos(-1/99) = 90.57 for CIFAR-100), To better describe the the inter-class separability and intra-class compactness, and similarities between the learned features and their corresponding prototype that they belong to. In Fig. 9, the similarities in Fig. 9(a) are mainly concentrated in 0.8 for CosFace with while the similarities in Fig. 9(b) and 9(c) are very close to 1. This indicates that the sample margin regularization significantly improves the inter-class compactness (the leamndd features in the same class are very similar to their corresponding prototype.) Moreover, the histograms of our proposed LM-Softmax on CIFAR-10 and CIFAR-100 are reported in Fig. 5 and 6, respectively. The similar- ities and sample margins keep very large with different More visualizations are provided in the following figures.
Results.
96.37 for
provide the histograms of sample margins
64,
Clarification. As shown in table 5, the proposed method results in both more larger class margin and more larger sample margin than the compared methods, however, the accuracy of the proposed method is slightly better than accuracies of the compared methods. acc actually evaluate the propor- tion of samples whose sample margin is larger than 0, i.c., good evaluation criterion for classification but is not good enough to measure the quality of feature representation. This is also one of the motivations of the previous works to improve the original
N Dm I(>(2;, yı) > 0). acc is
22

---

Published as a conference paper at ICLR 2022
softmax loss. In this paper, we measure the inter-class separability and intra-class compactness by class margin and sample margin, which can be used as two criteria to evaluate the quality of feature representations. Thus, acc, class margin, and sample margin can be regarded as different criteria.
Although the relationship of acc and margins is not so straightforward, enlarging the margins can improve acc some extent. As shown in Table we see that enlarging the margins of other losses by adding the sample margin regularization Rmm can improve the accuracy in most cases. Moreover, as shown in Table 2, the results on imbalanced learning are noteworthy, where the zero- centroid regularization for learning towards the largest margins on imbalanced classification shows obvious improvements in both class margins and accuracy most cases, and even can improve the performance of LDAM that is tailored for imbalanced learning
400
300
300
Co
(a) LM Softmax (s 10) (b) LM-Soffmax (s = 20) (c) LM Sof-Sof (s 40) (d) LM-Soofmax (s = 64)
100
300
(e) LM-Softmax (s = 10) (f) LM-Softmax (s = 20) (g) LMf-Soft (s - 40) (b) LM-Softmax (s 64)
Figure 6: Histogram of similarities and sample margins for LM-Softmax on CIFAR-100. (a-d) denote the cosine similarities between samples and their corresponding prototypes, and (e-h) denote the sample margins.
C.3 IMBALANCED CLASSIFICATION
Imbalanced CIFAR-10 and CIFAR-100. The original version of CIFAR-10 and CIFAR- 100 con- tains 50,000 training images and 10,000 test images of size 32 x 32 with 10 and 100 classes, re- spectively. To create their imbalanced version, we follow the setting in (Buda et al., 2018; Cui et al., 2019; Cao et al., 2019), where we reduce the number of training examples per class, and keep the test set unchanged. To ensure that our methods apply to variety of settings, of imbalance: long-tailed imbalance (Cui et al., 2019) and step imbalance (Buda et al., 2018). We the imbalance ratio p to denote the ratio between sample sizes of the most frequent and least frequent class, i.e., = max; [n+]/ min, ]. Long-tailed imbalance utilizes an exponential decay in sample sizes across different classes. For step imbalance setting, all minority classes have the same sample size, as do all frequent classes. This gives a clear distinction between minority classes and frequent classes, and the fraction for minority classes defined as μ.. We follow (Cao et al., 2019) and set = 0.5 by default.
types consider
We report the top-1 test accuracy ace and class margin meis of various baseline methods, includ- ing CF, Focal Loss, NormFace, CosFace, ArcFace, and the Label-DistrbbutionnAwaa Margin Loss (LDAMM) with hyper-parameter s = 5. Moreover, the proposed I.-SSftmax loss actually greatly affected by data imbalance since it will pay much attention to enlarge the margin between frequent classes and minority classes than other losses rather than any two classes. And we experiment with the LM-Softmax to verify the validity of the enlarging margin method. Moreover, add the zero- centroid regularization to the losses whose feature and prototypes are normalized for better margins.
We use ResNet-18 for imbalanced CIFAR-10, and ResNet-34 for imbalanced CIFAR- 100. Following in (Cao et al., 2019), we use SGD optimizer with momentum 0.9 and weight decay 2 x 10-1, The number of training epochs is set 200, and batch size is 128. The initial learning
Training details.
23

---

Published as a conference paper at ICLR 2022
Londe"
(030
1545
Unow
203
225
(a) Test Accuracy
(b) Class Margin
(c) Test Accuracy
(d) Class Margin
Figure 7: Test accuracies and class margins using different loss functions with and without the zero- centroid regularization on imbalanced CIFAR-10 and CIFAR-100. (a) and (b) are test accuracies and class margins on imbalanced CIFAR-10, 10, respectively. (c) and (d) are test accuracies and class margins on imbalanced CIFAR-10, respectively.
rate is set to 0.1. Moreover, we use the cosine learning rate annealing strategy (Loshchilov & Hutter, 2016) when T Taax is equal to the corresponding epochs.
Baselines and their hyper-parameter settings. We consider the baseline methods, including CE, Focal loss, CosFace, NormFace, AreFace, LM-Softmax, and the dabel-distrbbution-a aware margin (LDAM) loss. We set 1 for Folcal, 0.35 for CosFace, results, and the identical hyper-parameter is set to 5.
0.1 for ArcFace with stable
Results. The experimental results of imbalanced CIFAR-10 and CIFAR-100 are reported in Ta- ble 6. As we can see, the class margin of the LM-Softmax loss is fairly low in the severely imbalanced cases, while the other losses with feature and weight normalization have better per- formance than CE and Focal. However, their class margins are still small. With the role of the zero-centroid regularization, the class margin has very obvious improvement in all cases, where the class margins are close to the optimal one (arccos(- -1/9) = 96.37 for imbalanced CIFAR- 10, and arccos(-1/99) of X. As for the accuracy, there are also good improvements in most cases, especially for imbal- anced CIFAR-100. Moreover, compared with the performance of NormFace, it is worth noticing that the improvements of LDAM may heavily rely on the features and prototype normalization even margin trade-off. As illustrated in Fig. 20-28,
90.57 for imbalanced CIFAR-100). This conclusion holds for any choice
LDAM is designed for distrrdiiiiutin-aaaeee the zero-centroid regularization improves the intra-class compactness, where the cosine similarities between features and their corresponding prototypes they belong to are more concentrated around The experimental results the task of imbalanced classification. In the class imbalanced scenario, the stronger fitting ability of LM-Softmax however would make the learner care more about the majority classes but neglect the minority classes. This is the reason why L.M-Softax is less stable, which can be alleviated by applying the proposed zero-centroid regularization.
More Comparisons. To better show the effectiveness of zero-centroid regularization, we also con- struet more comparison to other related works imbalanced learning, including two-stage methods cRT (Kang al., 2020) and MiSLAS (Zhong al., 2021). cRT works in two-stage manner: firstly learn feature representation from the original imbalanced data, and then retrain the classifier us- ing class-balanced sampling with the first-stage representation frozen. Our proposed zero-centroid regularization R2 can not only render zero-centroid classifier but also produce feature represen- tations with larger margins when directly learning with imbalanced datasets. Thus, our proposed zero-cente pero-centroid regularization can benefit these two-stage methods. To verify this point, we conduct experiments on ImageNet-LTT with backbone ResNet-50 where the experimental settings follow recent two-stage decoupling method MiSLAS. As shown in the following table, the performance comparison of CE and CE + Ry demonstrate that zero-centroid regularization can significantly im- prove the representation learning ability of the first stage. Moreover, our zero-centroid regularization Rw can be easily integrated into well-developed two-stage decoupling methods, such as cRT, MiS- LAS. As demonstrated by the following results, adding R., into 1st stage (representation learning only) or both stages (representation learning and classifier learning) all can improve the performance of the original methods.
24

---

Published as a conference paper at ICLR 2022
Table 6: Test accuracies (acc) and class margins (mets) on imbalanced CIFAR-10. The results with positive gains are highlighted (where A denotes the regularization coefficient of the zero-centroid regularization term).
Dataset
Imbulanced CIFAR-10
Imbulanced CIFAR 100
Imbulance Type
long-talled
step
long-tailed
stop
Imbulance Ratio
100
10
100
100
10
100
Metric
acc
Mula
Meta
acc
Mele
Masi
acc
Mota
0.c
acc
Mal
CE Focal
70.88
87.41"
88.17 79.63"
76.50% 60.55
64.21
85.06
82.24
40.38
64.73 38.04
60:42
66.24"
42.36
60.32
56.88
62.839
66.30
74.14
87.33
74.48
63.30
84.49
75.16'
54.67
60.09
59.30"
41.90
55.98
57.84
55.72
CosFace 0)
69.28 58.77
87.02
81 $1.619
19.78
75.96
4.73
60.60
70.82
40.36
47.56
8.56
CosFace (A 1)
68.86 96.17'
87.24
96.16"
62.24
95.93'
84,98
96.26
40.53
65.42
60.37
84.34
40.90
42.85
56.50
74.41
CosFace (A 2)
69.40 95.61
87.16
96.26
62.49
95.86
84.69
95.88
40.53
65.13
60.77
84.97"
40.84
42.96
56.73
71.08"
CosFace 5)
69.18 93.73"
87.34
96. 24"
62.13
95.84
85.07
96.24
40.5%
55.27
60.34
84.47"
41.12
43,79
57.22
75.42"
CosFace (A 10)
68.83 92.49
86.94
96.23
61.99
95.35
96.129 85.59
80.93 40.98
59.15
85.070
40.97
34.65
56.97
84.09
CosFace 20)
69.52
91.90
95.46 87.55
94.36
95.88
80.30
59.66
83.46
41.59
83.93
ArcFace (A = ()
72.20 65.86
89.00
85 23"
62.44
54, 29
80.51 86.49
86.32
42.77
13.22
63.21
67.73"
41.47
0.50
58.89
0.37%
ArcFace (A 1)
71.69
95.08"
88 86
96. 26
63.10
95.83
96.23
43.97
52.75
63.67
71.52"
44.45
0,71
61.11
62.38
ArcFace (A- = 2)
71.91
93.78
88.78
96.24
94.84 63.05
86.18
96.23'
44.19
55.95
63.54
72.68 8
44,41
0.81
60,71
0.62
ArcFace 5)
72.23
89.22 96.23
95.01 633.59
96.24 86.65
56.60' 43.89
63.80
73.45
43.79
0.61
63.68
AreFace (A 10)
71.99
91.92
88.99 94.68
94.97 63.56
96.23'
75.58
63.55 82.
44.11
31.54
60.44
69.63'
ArcFace (A 20)
71.75
88.99
92.85
93.29
95.83'
75.28
62. $1.00
44.26
32.10
60.79
NormFace (A = 0)
72,37
62.72
89.19
82. 60
63:69
$1.00
86.37
77.82"
43.71
16.11 1
63
71 26
41..93
1.36
39.85
21.32
NormFace (A 1)
72.07
94.95
96.27
96.15
96.29 86.26 96.30'
59.42
63.81
89
43.77
41.25
64.55
NormFaace (A 2)
71.92
94.29
88.93
96. 28
63:21
94.14
44.20
60.39
63,90 77.69
44.51
36.30
60.49
71.70"
NormFace (A 5)
70.79 92.37'
96.
95.38
96.28 86.06
64.85
63.60
77.74
44.14
60.30
73.08
NormFace (A 10)
72.04
91.95
89.30
94,50
63:45
94.75 64.07
96.29 86.50
43.71
74.87
63 82,71 1
43.61
36.47
60.22
80.83
NormFace (A- = 20)
71.36 91.14
89.08
93.40"
93.06
95.94
43.67
75.71
62.66
82.18
43.70
28.94
60.16
81.66"
LDAM (A- = 0)
72.86
73.30
88.92
88. 19
63:27
61.42
87.0% 85.21
43,228
7.73
63.62
73.19
41.65
0.85
58.32
6.08
1)
72.50
88.97
96.10
71.00
84.49
60.52
LDAM (A- 2)
72.41
95.85
89.01
96.24
64.99
96.04
86,55
96.28
44,90
67.95
64.12
85.81
44,40
36.96
60.83
75.22
LDAM (A- 5)
71.99
93.83
89.51
96.25
64.79
96.12
96.16 86.69
86.62
45,23
70.96"
64.18
85.03
43,80
40.03
60.83
72.27
LDAM (A- 10)
72.21
92.49
88.92
1
64.48
96.16
96.29 86.60
81.42 43,53
63.05 85.62"
43,26 44.48
60.39 83,06
LDAM (A 20)
72.86
91.75
89,20
95.59'
64.66
94.55
96.05"
43.85
79.68
62.64
84,87
37.66 44.17
60.28
84.31
L. Softmax (A 0)
65.32
4.42
88, 69
68, 91
50:47
0.45
$6.08
22.20"
41.52
4. 4.50
63, 26
68. 68.31
411.53
0.47
55
1.37
LM-Softmax (A 1)
72.25
96.06
88.47
96.26
64.18
91.44
86.66
96.14
45,22
68.02
63.77
81.99
45.40
39.87
60.57
73.19'
LM MM-Soommax
2)
72.57
95.83"
88. 69
96.31"
65.58
93.23
86,70
96
44.90
67.90
63.39
82,93
45.17
38.29
60.73
74.78"
LM-Softmax (A = -5)
72.53 93.68"
88.60
96.26
65.18
95.20
96.05" 87.07
45.28
69.53
63.60
83.32"
46.23
43.15
60.22
74.37
LM M--Sofmax -10) 73.21 92.57
96.25
65.91
93.84
86.96
96.09
78.90
62.89
85.39
45.06
46.69
7.94
LM Softmax (A- = -20) 73.20 91.95
89.12
95.73
65.39
93.23
86.95
96.03'
44.22
80.53
63.40
83.80
45.97
64.84
60.23
76.77
Table 7: Top-1 validation accuracy on ImageNet-LT, where * denotes that the results are borrowed from MiSLAS, X+I., denotes adding Ry to the corresponding stages, and the trade-off parameter is set 100. The results with positive gains are highlighted.
Method
Many
Medium
Few
All
CE
66.76
36.87
7.06
43.61
CE+R
68.42
39.42
10.69
45.90
cRT*
62.5
47.4
29.5
50.3
cRT+mixup*
63.9
49.1
30.2
51.7
cRT+mixup
65.72
48.78
25.89
51.61
cRT+mixup+RR Ry, (adding R,, for 1st stage)
64.03
49.89
32.81
52.59
cRT+mixup+RR (adding R,, for 1st and 2nd stage)
64.12
49.99
32.73
52.65
MiSLAS
61.7
51.3
35.8
52.7
MiSLAS
63.30
50.06
33.52
52.50
MiSLAS+ Ra (adding Rw for 1st stage)
63.11
50.56
34.24
52.76
MiSLAS+RI RR, (adding Ra for 1st and 2nd stage)
63.20
50.69
34.21
52.85
25

---

Published as a conference paper at ICLR 2022
(a) 0
(h)
(c) 2
(d) 5
(c) 10
(f) 20
(g) À = 0 (h) =
(i) A =
(1) A =
(k) = 10
(1) = 20
Figure 8: Histogram of similarities and sample margins for LMM-offmax using the zero-centroid regularization with different À on long-tailed imbalanced CIFAR- 10 with 10. (a-f) denote the cosine similarities between samples and their corresponding prototypes, and (g-I) denote the sample margins.
C.4 PERSON RE-IDENTIFICATION
We conduct experiments on the task of person re-identification. Specifically, use the off-the-shelf baseline (Luo et al., 2019) as the main code to verify the efficiency our proposed LM-Softmax.
Training Details. We followed the default parameter settings and training strategy. More specif- ically, we train the ResNet50 with pre-traine pre-trained parameters for 60 epochs. Two benchmark datasets Market-1501 (Zheng et al., 2015) and DukeMTMC (Ristani al., 2016) evaluated. More- over, all models are trained with Triplet Loss + the compared losses, including CE, AreFace, Cos- Face, NormFace, and the proposed L.M-Softmxx Experiments DukeMTMC. As shown Table 3, our proposed L.M-Softmax obtains obvious improvements in mAP, Rank @ and Rank@5, which also exhibits significant robustness for different parameters. In contrast, ArcFace, CosFace, and NormFace show worse performance than ours and are more sensitive parameter settings.
conducted Market-1501 and
Table 8: The results on Market- and DukeMTMC for person re-identification task. The best four results are highlighted.
Dataset
Market-1501
DukeMTMC
Method
ImAP Rank@)
Rank Rank @ 10
mAP Rank Rank @5 Rank @ 10
Softmax
82.8
92.7
97.5
98.7
73.0
83.5
93.0
95.2
ArcFace (s 10)
67.5
84.1
92.1
94.9
37.7
58.7
72.7
77.8
ArcFace (s 20)
79.1
90.8
96.5
98.1
61.4
78.3
88.6
91.6
ArcFace (s 32)
80.5
92.1
97.1
98.4
66.7
82.9
91.2
93.4
ArcFace (s 64)
80.4
92.6
97.4
98.4
67.6
83.4
91.4
94.1
CosFace (s 10)
68.0
84.9
92.7
95.2
39.3
60.6
73.1
78.7
CosFace (s 20)
80.5
92.0
97.1
98.2
64.2
81.3
89.7
92.8
CosFace (s 32)
81.7
93.4
97.6
98.3
69.4
83.5
92.3
94.4
CosFace (s = 64)
78.7
92.0
97.1
98.3
68.2
83.1
92.5
94.4
NormFace (s 10)
81.2
91.6
96.3
98.0
63.7
79.3
88.5
91.0
NormFace (s 20)
83.2
93.5
97.9
98.8
71.6
83.8
93.3
95.1
NormFace (s 32)
77.5
90.0
96.9
98.3
66.2
80.2
90.5
93.8
NormFace (s 64)
77.5
90.0
96.9
98.3
60.1
75.2
88.1
91.7
LM-Softmax (s 10) 83.3
92.8
97.1
98.2
72.2
85.8
92.4
94.8
LM-Sofumax (s -20) 84.7
93.8
97.6
98.6
74.1
86.4
93.5
94.9
LM-Softmax (s -32) 84.3
93.4
97.7
98.4
73.3
86.0
93.2
95.1
LM- MM-oofmax (s -64) 84.6
93.9
98.1
98.8
74.2
86.6
93.5
95.2
26

---

Published as a conference paper at ICLR 2022
(4) 1 = 0
(b) /r = 0.5
(c) /
(d) /4 = 0
(c) /z = 0.5
( /
gg) (h) 0.5
(1 p=1
( 0
(k) = 0.5
() p=1
Figure 9: Histogram of similarities and sample margins for CosFace (s 64) with/without sample margin regularization Rum CIFAR-10 10 and CIFAR-100 (a-c) and (g-i) denote the cosine simi- larities on CIFAR-10 10 and CIFAR-100, respectively. (d-f) and (j-l) denote the sample margins on CIFAR-10 and CIFAR-100, respectively.
C.5 FACE VERIFICATION
Datasets. We also verify method on the task of face verification whose performance highly depends on the discriminability of feature embeddings. We follow the training settings (An al., 2020)'. The model is trained MS1MV3 with 5.8M images and 85K ids (Guo et al., 2016) and testing on LFW (Sengupta et al., 2016), CFP-FP [3I, AgeDB-30 30 (Moschoglou et al., 2017) and IJCC (Maze al., 2018). The detailed results IJBC-C are shown in Table 9
Training Details We use ResNet34 as the fea- ture embedding model and train it on two GPUs NVIDIA Tesla v100 with batch size 512 for all compared methods. The compared method in- cludes ArcFace, CosFace, NormFace, and our pro- posed LMS-oftmax.
Table 9: Different evaluation metries of fave verifi- cation on UB-C. The results with positive gains are highlighted.
Method
le-5
le-4
AUC
ArcFace
93.21
95.51
99.4919
ArcFace+ Rsm
93.26
95.41
99.5011
Baselines and hyper-parameter settings. We use the baseline methods including CosFace, Are- Face, NormFace, and our proposed LM-Softmax. For CosFace and ArcFace, we use the hyper- parameters followed their original paper, i.e., 64 and m = 0.35 for CosFace: = 64 and 0.5 for ArcFace; For NormFace and LM- Softmax, we set s 64 and s 32, respectively.
ArcFace+RR.
93.27
95.53
99.5133
CosFace
93.27
95.63
99.4942
CosFacc+ Roam
93.28
95.68
99.5112
CosFacct
93.29
95.69
99,5538
LM-Softmax
91.85
94.80
99.4721
IL
93.17
95.47
99,5086
27

---

Published as a conference paper at ICLR 2022
(a) ja 0
(b) μr = 0.5
(c)
(d) 0
(e) μ1 = 0.5
(1 P
(g) = 0
(h) 0.5
(1 =1
( 0
(k) 0.5
(1 =1
Figure 10: Histogram similarities and sample margins for ArcFace (s = 20) with/without sample margin regularization R&m on CIFAR-IO and CIFAR-100. (a-c) and (g-i) denote the cosine simi- larities on CIFAR-I0 and CIFAR-I00 respectively. (d-f) and (i-l) denote the sample margins on CIFAR-10 10 and CIFAR-100 respectively.
(a) / 1r 0.5
(c) /
(d) M 0
(c) J 0.5
(f) p
(g) μ (h) μ
(1 μ
h==
(k) / =
μ ==
Figure 11: Histogram of similarities and sample margins for NormFace (s 64) with/without sample margin regularization R.m on CIFAR-10 and CIFAR-100. (a-c) and (g-i) denote the cosine similarities on CIFAR-10 and CIFAR-100, respectively. (d-f) and (i-l) denote the sample margins on CIFAR-10 and CIFAR-100, respectively.
(a) A = 0
(b) A =1
(c) A= =
(d) = 5
(e) X 10
(f) À 20
(g) À 0
(h) A =1
(i) X =
(i) X
(k) X 10
(1) À 20
Figure 12: Histogram of similarities and sample margins for LM-Softmax using the zero-centroid
regularization with different on step imbalanced CIFAR-10 with p = 10. (a-f) denote the co- sine similarities between samples and their corresponding prototypes, and (g-l) denote the sample margins.
28

---

Published as a conference paper at ICLR 2022
(a) A = 0
(b) A ==
(c) =
(d) = 5
(e) 10
(1) 20
(g) À = 0 (h) A =1
(1 2
( A = 5 (k) X 10
()) = 20
Figure 13: Histogram of similarities and sample margins for CosFace using the zero-centroid reg- ularization with different on long-tailed imbalanced CIFAR-10 with p 10. (a-f) denote the
cosine similarities between samples and their corresponding prototypes, and (g-l) denote the sample margins.
(a) 0
(b) =
(c) 2
(d) 5
(c) 10
(f) 20
(g) =
(h) ==1
(i) ==
(i) A =
(k) 10
(1)
Figure 14: Histogram of similarities and sample margins for ArcFace using the zero-centroid reg- ularization with different À on long-tail longltailed imbalanced CIFAR-10 10 with 10. (a-f) denote the cosine similarities between samples and their corresponding prototypes, and (g-l) denote the sample margins.
(a) À 0
(b) A =
(c) A 2
(d) = 5
(e) X 10
(f) À 20
(g) À 0
(h) A (0) X
(i) À
(k) X 10
(1) À 20
Figure 15: Histogram of similarities and sample margins for NormFace using the zero-centroid regularization with different on long-tailed imbalanced CIFAR-10 with p 10. (a-f) denote the cosine similarities between samples and their corresponding prototypes, and (g-I) denote the sample margins.
29

---

Published as a conference paper at ICLR 2022
(a) 0
(b) 1
(c) A 2
(d) 5
(c) 10
(f) 20
(g) =
(h) =
(i) x=
(i) A =
(k) 10
(1)
Figure 16: Histogram of similarities and sample margins for LDAM using the zero-centroid reg- ularization with different À on longttailed imbalanced CIFAR-10 with 10. (a-f) denote the
cosine similarities between samples and their corresponding prototypes, and (g-l) denote the sample margins.
30


Reviews:
[{"review_id": "p0aeniOllJB", "reviewer": "Reviewer_3YiD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the paper:\n+ The paper is well-written.\n+ The paper develops a principled mathematical framework and formulates it as the problem of learning towards the largest margins for better understanding and design of margin-based loss functions.\n+ The sample margin regularization and zero-centroid regularization are proposed for the class-balanced case and the class-imbalanced case.\n+ The paper offers rigorously theoretical analysis as well as extensive experiments to support the proposed method.\n\nThe weaknesses of the paper\n- In the proposed Generalized Margin Softmax Loss (GM-Softmax), there are more parameters than a unified framework (Deng et al., 2019), how to effectively set them in the experiments?\n- The improvement of the proposed is not significant. The proposed method results in both more larger class margin and more larger sample margin than the compared methods, however, the accuracy of the proposed method is slightly better than accuracies of the compared methods, and the current version does not analysis this.", "summary_of_the_paper": "This paper develops a principled mathematical framework for better understanding and design of margin-based loss functions, where the principled optimization objective is formulated as learning towards the largest margins. In the proposed method, class margin and the sample margin are defined as the measure of inter-class separability and the measure of intra-class compactness. Furthermore, the sample margin regularization and zero-centroid regularization are introduced for the class-balanced case and the class-imbalanced case. Experimental results show the effectiveness of the proposed method on imbalanced classification, person re-identification, and face verification.", "summary_of_the_review": "The paper is well-written and the novelty/contribution is significant and somewhat new. However, the weaknesses of the paper should be addressed to improve the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jZhGDL2HHxR", "reviewer": "Reviewer_pU1u", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### PROS:\n+ The authors proved that existing margin-based losses have the same lower bound and accordingly the same optimizers through connecting them to the optimizers of class-/sample-margins.\n+ Some practical methods regarding classifier margins are introduced and empirically evaluated to produce favorable performance improvement on image classification tasks.\n\n### CONS:\n- Practical method.\n\nThe paper presents some practical methods. They, however, are less novel and their effectiveness are not sufficiently validated in the experiments.\nSample margin regularization in Eq.(3.4) is the classical loss such as used in MC-SVM [R1].\nLM-softmax loss in Eq.(3.5) is a simple reformulation of the standard softmax loss; we can easily see that L_{softmax loss} = softplus(sL). While the softmax loss is lower-bounded by softplus, the LM-softmax loss is not by softplus but by normalization of features and classifier weights, due to which we can guess LM-softmax is less stable as shown in Table 2. \nZero-center regularization is somewhat interestingly formulated for imbalanced learning. It, however, lacks comparison/analysis to the other related works of imbalanced learning such as [R2,R3]. Besides, the regularization would not work in the framework of cRT [R4] which re-trains classifiers on 'balanced' dataset; balanced mini-batch sampling would naturally render zero-centered classifiers even by the softmax loss.\n\nIn the experiments, the authors apply simple softmax loss for comparison. However, toward fair comparison, the softmax loss should be applied to normalized features and classifiers with the tuned temperature as in the other losses. Without comparison to the well-calibrated softmax loss, it is hard to recognize the effectiveness of the proposed method, especially LM-softmax.\n\n\n[R1] K. Crammer & Y. Singer. On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines. JMLR, No. 2, pp.265-292, 2001.\n\n[R2] H.-J. Ye et al. Identifying and Compensating for Feature Deviation in Imbalanced Deep Learning, arXiv:2001.01385.\n\n[R3] B. Kim & J. Kim. Adjusting Decision Boundary for Class Imbalanced Learning, IEEE Access 2020.\n\n[R4] B. Kang et al. Decoupling representation and classifier for long-tailed recognition. ICLR 2020.\n\n- Theoretical analysis.\n\nConsidering the limited novelty of the presented methods as mentioned above, the paper's contribution would be rather theoretical materials. Though, those materials seem not to be so effective for further understanding/analyzing the large-margin losses.\nAs to normalization of features and classifiers, empirical analysis in Fig.1 is well-known in the literature of imbalanced learning [R2-R4] and Theorem 3.1 is rather trivial.\nThe analysis about the margin-based losses through Theorem 3.2 (Proposition 3.3) is somewhat interesting, but Theorem 3.2 is too rough to understand the roles of those losses deeply; actually, Theorem 3.2 is applicable even to the standard softmax loss with \\alpha=1 and \\beta=0, not only to the margin-aware ones. Thus, the theoretical part could not provide us with deep analysis specific to the margin-based losses such as XFace losses.\n\nThe paper lacks analysis and/or discussion about the case that some classes are correlated as frequently observed in real-world tasks. In that case, maximizing class-margin by force toward (k-1)-simplex classifier would be less effective, degrading intrinsic characteristics of those class categories; the correlated classes would intrinsically provide smaller margin.\n\n### MINOR COMMENTS:\n- In p.3, \"intra-class compactness and inter-class separability\" -> \"inter-class separability and intra-class compactness\"\n- In Eq.3.5, -\\frac{1}{s} -> \\frac{1}{s}, and y_i -> y\n- In Eq.3.6, \\frac{1}{t} -> \\frac{1}{s}\n- In Table 1, no comparison to CE+R_{sm} nor LM-softmax.\n- In Sec.4.3, it is unclear why the authors consider the task of person reID. How much effective is it for evaluating the margin-based losses?\n- In p.13, Lemma 2.3 -> Lemma 2.1\n- At the equations in top of p.15, \"k \\log\" -> \"\\log\".", "summary_of_the_paper": "This paper analyzes large-margin based loss functions. The authors formulate two types of margins, class- and sample-margins, and then analyze lower bounds of various margin-based losses in a unified framework to show that those losses are minimized by the optimizers of class-/sample-margins. While inspecting the existing margin-based losses, they also formulate two practical methods of sample-margin regularization and largest margin softmax loss to further enhance margins of classifiers. The experimental results on some image classification tasks demonstrate that those methods contribute to performance improvement, while favorably being compared with the other methods that induce large-margin classification.", "summary_of_the_review": "This paper presents several theoretical and practical materials regarding large-margin based losses.\nThey, however, are of limited novelty and are less contributive to theoretical understanding of margin-based losses.\nThus, my rating score of the paper is leaning toward weak rejection.\n\n\n-- After rebuttal --\n\nI appreciate the authors' effort to provide detailed response.\nIt nicely addresses my concerns about Theorem 3.2 and the practical methods.\nBased on the rebuttal, I upgrade my score to 'weak accept' and recommend the authors to put those materials into a revised paper for clarifying the theoretical and practical contributions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZQcT2_TIgu-", "reviewer": "Reviewer_sx1i", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper is well-written, with nice motivations and clear theorems (no unnecessary \"mathification\" of proofs or writing).\n\n2. Analytical results are clear and to-the-point, showing the motivation for their losses.\n\n3. Empirical results are extensive, and show the benefits of their margin-promoting losses.\n\nWeaknesses:\n\n1. It would have been nice to show at least some empirical result for the case when k > d+1 (large number of classes compared to embedding dimensionality). Do their losses work \"reasonably\" in this case? Such a scenario may arise in modern deep learning methods when we train on datasets with many fine-grained classes e.g. ImageNet-21k.\n\n2. It's slightly awkward that different losses are used for different datasets. How can we decide whether to use GM-Softmax, LM-Softmax or R_sm for a given dataset? Why not try all 3 together?\n", "summary_of_the_paper": "This paper analyzes novel loss functions to promote large margins during the training of a supervised (softmax-based) classification loss, under various conditions such as balanced and imbalanced number of samples per class. They propose a number of novel losses to promote large margins, and show analytical and empirical proof of the optimality of their proposed losses.", "summary_of_the_review": "The paper as it's well written, and shows novel analytical and empirical results. I think the losses are simple and easy to implement in practice.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-PCKwHWu-Q6", "reviewer": "Reviewer_pGzf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Most of the former margin-based softmax methods (e.g. sphereface, cosface and arcface) are experiment-driven. By contrast, this paper offers rigorous theoretical analysis, which is the main contribution of this paper.\n\nThe writing of this paper is clear and all Theorems in the main paper are proved in the Supplementary Materials.\n\nLimitations:\n(1) The class margin (Eq 2.4), sample margin (Eq 3.4), generalized margin softmax loss (Eq 3.3), and zero centroid regularization (Eq 3.7) are not novel, but the theoretical explanation is nice.\n\n(2) For face verification, which evaluation metric is employed to report the performance on IJB-C? We usually report TAR@FAR=1e-4/1e-5.\n\n---------------post rebuttal----------------------\n\nAfter reading the reply from the author, I have understood the evaluation metric used on IJB-C. I suggest the author change the AUC to TAR@FAR=1e-4/1e-5 in the camera-ready version.  The improvement is not obvious ($<0.5\\\\%$ IJB-C TAR@FAR=1e-4/1e-5) but it is ok for a theoretical paper. This paper can enhance our understanding of margin-based face recognition. Therefore, I still vote for acceptance.", "summary_of_the_paper": "This paper developed a principled mathematical framework for a better understanding and design of loss functions.\n\nBased on the class and sample margins, the proposed method formulated the objective as learning towards the largest margins, and offer rigorously theoretical analysis as support.\n\nFor class balanced cases, this paper proposed an explicit sample margin regularization term and a novel largest margin softmax loss; for the class imbalanced cases, this paper proposed a simple but effective zero centroid regularization term, which restricts the centroid of prototypes to be zero.\n\nExtensive experimental results demonstrate that the proposed strategy significantly improves the performance in accuracy and margins for various tasks.", "summary_of_the_review": "The theoretical explanation in this paper is nice and the experiments on person Re-Id and face verification confirm the effectiveness of the proposed method. In the NeurIPS 2021 review, this paper accepted two marginally above and one borderline result. In this ICLR submission, the main concern from the former reviewers has been fixed. In ICCV 2021, face-related papers have the highest rejection rate (>80%). There are two main groups of researchers in the face recognition community. One group of researchers focus on high performance under large-scale settings. Their papers look like experiment reports, but the performance is impressive. Another group of researchers focus on theoretical analysis. Their papers look more rigorous, but the performance is not satisfying and the proposed methods are not orthogonal to current state-of-the-art solutions. Even though I am from the first group, and I am not persuaded that the proposed method can obviously improve verification accuracy (in Table 4), I believe the theoretical explanation in this paper can enrich the research community and enhance the deep understanding of margin-based softmax loss designs. Both of the researchers from the practical and theoretical sides need to appreciate the contributions and novelties from the other side.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a principled objective function for large margin learning. Specifically, it introduces class margin and sample margin, both of which it aims to promote. It also derives a generalized margin softmax loss which to draw general conclusions on the existing margin-based losses. The effectiveness of the proposed theory is empirically verified in visual classification, imbalanced classification, person re-identification, and face verification.

The reviewers initially raised some concerns, but most of them were well addressed in the rebuttal and convinced the reviewers. Specifically, pU1u was satisfied by authors' reply on Theorem 3.2 and the practical methods. pGzf appreciated clarifications around the evaluation metric used on IJB-C and believes this work can improve our understanding of margin-based face recognition. Finally, 3YiD had some reservations about number of parameters which got clarified by the authors. 

In sum, all post rebuttal ratings fall in the accept zone, and the reviewers find the paper interesting and insightful. In concordance with them, I recommend this paper for publication. Please make sure to include suggestions made by reviewers in the camera ready version.
Decision: Accept (Poster)