Year: 2022

Abstract:
One of the main challenges for feature representation in deep learning-based classification is the design of appropriate loss functions that exhibit strong discriminative power. The classical softmax loss does not explicitly encourage discriminative learning of features. A popular direction of research is to incorporate margins in well-established losses in order to enforce extra intra-class compactness and interclass separability, which, however, were developed through heuristic means, as opposed to rigorous mathematical principles. In this work, we attempt to address this limitation by formulating the principled optimization objective as learning towards the largest margins. Specifically, we firstly define the class margin as the measure of inter-class separability, and the sample margin as the measure of intra-class compactness. Accordingly, to encourage discriminative representation of features, the loss function should promote the largest possible margins for both classes and samples. Furthermore, we derive a generalized margin softmax loss to draw general conclusions for the existing margin-based losses. Not only does this principled framework offer new perspectives to understand and interpret existing margin-based losses, but it also provides new insights that can guide the design of new tools, including sample margin regularization and largest margin softmax loss for the class-balanced case, and zero-centroid regularization for the class-imbalanced case. Experimental results demonstrate the effectiveness of our strategy on a variety of tasks, including visual classification, imbalanced classification, person re-identification, and face verification.

heading: 1 INTRODUCTION
text: Recent years have witnessed the great success of deep neural networks (DNNs) in a variety of tasks, especially for visual classification (Simonyan & Zisserman, 2014; Szegedy et al., 2015; He et al., 2016; Howard et al., 2017; Zoph et al., 2018; Touvron et al., 2019; Brock et al., 2021; Dosovitskiy et al., 2021). The improvement in accuracy is attributed not only to the use of DNNs, but also to the elaborated losses that encourage well-separated feature embeddings (Musgrave et al., 2020).
In general, the loss is expected to promote the learned features to have maximized intra-class compactness and inter-class separability simultaneously, so as to boost the feature discriminativeness. Softmax loss, which is the combination of a linear layer, a softmax function, and a cross-entropy loss, is the most commonly-used ingredient in deep learning-based classification. However, the softmax loss only learns separable features that are not discriminative enough (Liu et al., 2017). To remedy the limitation of softmax loss, many variants have been proposed. Liu et al. (2016) proposed a generalized large-margin softmax loss, which incorporates a preset constant m multiplying with the angle between samples and the classifier weight of the ground truth class, leading to potentially larger angular separability between learned features. SphereFace (Liu et al., 2017) further improved the performance of L-Softmax by normalizing the prototypes in the last inner-product layer. Subsequently, (Wang et al., 2017) exhibited the usefulness of feature normalization when using feature vector dot products in the softmax function. Coincidentally, in the field of contrastive learning, Chen et al. (2020) also showed that normalization of outputs leads to superior representations. Due to its effectiveness, normalization on either features or prototypes or both becomes a standard procedure in margin-based losses, such as SphereFace (Liu et al., 2017), CosFace/AM-Softmax (Wang et al., 2018b;a) and ArcFace (Deng et al., 2019). However, there is no theoretical guarantee provided yet. ∗This work was done as intern at Peng Cheng Laboratory. †Correspondence to: Xianming Liu <csxm@hit.edu.cn>
Despite their effectiveness and popularity, the existing margin-based losses were developed through heuristic means, as opposed to rigorous mathematical principles, modeling and analysis. Although they offer geometric interpretations, which are helpful to understand the underlying intuition, the theoretical explanation and analysis that can guide the design and optimization is still vague. Some critical issues are unclear, e.g., why is the normalization of features and prototypes necessary? How can the loss be further improved or adapted to new tasks? Therefore, it naturally raises a fundamental question: how to develop a principled mathematical framework for better understanding and design of margin-based loss functions? The goal of this work is to address these questions by formulating the objective as learning towards the largest margins and offering rigorously theoretical analysis as well as extensive empirical results to support this point.
To obtain an optimizable objective, firstly, we should define measures of intra-class compactness and inter-class separability. To this end, we propose to employ the class margin as the measure of interclass separability, which is defined as the minimal pairwise angle distance between prototypes that reflects the angular margin of the two closest prototypes. Moreover, we define the sample margin following the classic approach in (Koltchinskii et al., 2002, Sec. 5), which denotes the similarity difference of a sample to the prototype of the class it belongs to and to the nearest prototype of other classes and thus measures the intra-class compactness. We provide a rigorous theoretical guarantee that maximizing the minimal sample margin over the entire dataset leads to maximizing the class margin regardless of feature dimension, class number, and class balancedness. It denotes that the sample margin also has the power of measuring inter-class separability.
According to the defined measures, we claim that, to encourage discriminative representation of features, the loss function should promote the largest possible margins for both classes and samples, which also meets to tighten the margin-based generalization bound in (Kakade et al., 2008; Cao et al., 2019). The main contributions of this work are highlighted as follows:
• For a better understanding of margin-based losses, we provide a rigorous analysis about the necessity of normalization on prototypes and features. Moreover, we propose a generalized margin softmax loss (GM-Softmax), which can be derived to cover most of existing marginbased losses. We prove that, for the class-balance case, learning with the GM-Softmax loss leads to maximizing both class margin and sample margin under mild conditions.
• We show that learning with existing margin-based loss functions, such as SphereFace, NormFace, CosFace, AM-Softmax and ArcFace, would share the same optimal solution. In other words, all of them attempt to learn towards the largest margins, even though they are tailored to obtain different desired margins with explicit decision boundaries. However, these losses do not always maximize margins under different hyper-parameter settings. Instead, we propose an explicit sample margin regularization term and a novel largest margin softmax loss (LM-Softmax) derived from the minimal sample margin, which significantly improve the class margin and the sample margin.
• We consider the class-imbalanced case, in which the margins are severely affected. We provide a sufficient condition, which reveals that, if the centroid of prototypes is equal to zero, learning with GM-Softmax will provide the largest margins. Accordingly, we propose a simple but effective zero-centroid regularization term, which can be combined with commonly-used losses to mitigate class imbalance.
• Extensive experimental results are offered to demonstrate that the strategy of learning towards the largest margins significantly can improve the performance in accuracy and class/sample margins for various tasks, including visual classification, imbalanced classification, person re-identification, and face verification.

heading: 2 MEASURES OF INTRA-CLASS COMPACTNESS AND INTER-CLASS SEPARABILITY
text: With a labeled dataset D = {(xi, yi)}Ni=1 (where xi denotes a training example with label yi, and yi ∈ [1, k] = {1, 2, ..., k}), the softmax loss for a k-classification problem is formulated as
L = 1
N N∑ i=1 − log exp(wTyizi)∑k j=1 exp(w T j zi) = N∑ i=1 − log exp(‖wyi‖2‖zi‖2 cos(θiyi))∑k j=1 exp(‖wj‖2‖zi‖2 cos(θij)) , (2.1)
where zi = φΘ(xi) ∈ Rd (usually k ≤ d + 1) is the learned feature representation vector; φΘ denotes the feature extraction sub-network; W = (w1, ...,wk) ∈ Rd×k denotes the linear classifier which is implemented with a linear layer at the end of the network (some works omit the bias and use an inner-product layer); θij denotes the angle between zi and wj ; and ‖ · ‖2 denotes the Euclidean norm, wherew1, ...,wk can be regarded as the class centers or prototypes (Mettes et al., 2019). For simplicity, we use prototypes to denote the weight vectors in the last inner-product layer.
The softmax loss intuitively encourages the learned feature representation zi to be similar to the corresponding prototype wyi , while pushing zi away from the other prototypes. Recently, some works (Liu et al., 2016; 2017; Deng et al., 2019) aim to achieve better performance by modifying the softmax loss with explicit decision boundaries to enforce extra intra-class compactness and interclass separability. However, they do not provide the theoretical explanation and analysis about the newly designed losses. In this paper, we claim that a loss function to obtain better inter-class separability and intra-class compactness should learn towards the largest class and sample margin, and offer rigorously theoretical analysis as support. All proofs can be found in the Appendix A.
In the following, we define class margin and sample margin as the measures of inter-class separability and intra-class compactness, respectively, which serve as the base for our further derivation.

heading: 2.1 CLASS MARGIN
text: With prototypesw1, ...,wk ∈ Rd, the class margin is defined as the minimal pairwise angle distance:
mc({wi}ki=1) = min i6=j ∠(wi,wj) = arccos [ max i 6=j wTi wj ‖wi‖2‖wj‖2 ] , (2.2)
where ∠(wi,wj) denotes the angle between the vectors wi and wj . Note that we omit the magnitudes of the prototypes in the definition, since the magnitudes tend to be very close according to the symmetry property. To verify this, we compute the ratio between the maximum and minimum magnitudes, which tends to be close to 1 on different datasets, as shown in Fig. 1.
To obtain better inter-class separability, we seek the largest class margin, which can be formulated as
max {wi}ki=1 mc({wi}ki=1) = max {wi}ki=1 min i 6=j ∠(wi,wj).
Since magnitudes do not affect the solution of the maxmin problem, we perform `2 normalization for eachwi to effectively restrict the prototypes on the unit sphere Sd−1 with center at the origin. Under this constraint, the maximization of the class margin is equivalent to
the configuration of k points on Sd−1 to maximize their minimum pairwise distance:
arg max {wi}ki=1⊂Sd−1 min i 6=j ∠(wi,wj) = arg max {w}ki=1⊂Sd−1 min i 6=j ‖wi −wj‖2, (2.3)
The right-hand side is well known as the k-points best-packing problem on spheres (often called the Tammes problem), whose solution leads to the optimal separation of points (Borodachov et al., 2019). The best-packing problem turns out to be the limiting case of the minimal Riesz energy:
arg min {w}ki=1⊂Sd−1 lim t→∞ ∑ i 6=j
1
‖wi −wj‖t2 = arg max {w}ki=1⊂Sd−1 min i 6=j ‖wi −wj‖2. (2.4)
Interestingly, Liu et al. (2018) utilized the minimum hyperspherical energy as a generic regularization for neural networks to reduce undesired representation redundancy. When w1, ...,wk ∈ Sd−1, k ≤ d+ 1, and t > 0, the solution of the best-packing problem leads to the minimal Riesz t-energy: Lemma 2.1. For any w1, ...,wk ∈ Sd−1, d ≥ 2, and 2 ≤ k ≤ d + 1, the solution of minimal Riesz t-energy and k-points best-packing configurations are uniquely given by the vertices of regular (k − 1)-simplices inscribed in Sd−1. Furthermore, wTi wj = −1k−1 , ∀i 6= j.
This lemma shows that the maximum of mc({wi}ki=1) is arccos( −1k−1 ) when k ≤ d + 1, which is analytical and can be constructed artificially. However, when k > d + 1, the optimal k-point configurations on the sphere Sd−1 have no generic analytical solution, and are only known explicitly for a handful of cases, even for d = 3.

heading: 2.2 SAMPLE MARGIN
text: According to the definition in (Koltchinskii et al., 2002), for the network f(x; Θ,W ) = WTφΘ(x) : Rm → Rk that outputs k logits, the sample margin for (x, y) is defined as
γ(x, y) = f(x)y −max j 6=y f(x)j = w T y z −max j 6=y wTj z, (2.5)
where z = φΘ(x) denotes the corresponding feature. Let nj be the number of samples in class j and Sj = {i : yi = j} denote the sample indices corresponding to class j. We can further define the sample margin for samples in class j as
γj = min i∈Sj γ(xi, yi). (2.6)
Accordingly, the minimal sample margin over the entire dataset is γmin = min{γ1, ..., γk}. Intuitively, learning features and prototypes to maximize the minimum of all sample margins means making the feature embeddings close to their corresponding classes and far away from the others: Theorem 2.2. For w1, ...,wk, z1, ...,zN ∈ Sd−1 (where nj > 0 for each j ∈ [1, k]), the optimal solution {w∗i }ki=1, {z∗i }Ni=1 = arg max{wi}ki=1,{zi}Ni=1 γmin is obtained if and only if {w ∗ i }ki=1 maximizes the class margin mc({wi}ki=1), and z∗i = w∗yi −w∗yi
‖w∗yi−w ∗ yi ‖2 , where w
∗ yi denotes the centroid of
the vectors {wj : j maximizes wTyiwj , j 6= yi}.
As shown in the proof A, Theorem 2.2 guarantees that maximizing γmin will provide the solution of the Tammes problem with respect to any feature dimension d, class number k, and both classbalanced and class-imbalance cases. When 2 ≤ k ≤ d+ 1, we can derive the following proposition: Proposition 2.3. For any w1, ...,wk, z1, ...,zN ∈ Sd−1, d ≥ 2, and 2 ≤ k ≤ d+ 1, the maximum of γmin is kk−1 , which is obtained if and only if ∀i 6= j, w T i wj = − 1k−1 , and zi = wyi .
Theorem 2.2 and Proposition 2.3 show that the best separation of prototypes is obtained when maximizing the minimal sample margin γmin.
On the other hand, let Lγ,j [f ] = Prx∼Pj [maxj′ 6=j f(x)j′ > f(x)j − γ] denote the hard margin loss on samples from class j, and L̂γ,j denote its empirical variant. When the training dataset is separable (which indicates that there exists f such that γmin > 0), Cao et al. (2019) provided a finegrained generalization error bound under the setting with balanced test distribution by considering the margin of each class, i.e., for γj > 0 and all f ∈ F , with a high probability, we have
Pr(x,y)[f(x)y < max l 6=y
f(x)l] ≤ 1
k k∑ j=1 ( L̂γj ,j [f ] + 4 γj R̂j(F) + εj(γj) ) . (2.7)
In the right-hand side, the empirical Rademacher complexity 4γj R̂j(F) has a big impact. From the perspective of our work, a straightforward way to tighten the generalization bound is to enlarge the minimal sample margin γmin, which further leads to the larger margin γj for each class j.

heading: 3 LEARNING TOWARDS THE LARGEST MARGINS

heading: 3.1 CLASS-BALANCED CASE
text: According to the above derivations, to encourage discriminative representation of features, the loss function should promote the largest possible margins for both classes and samples. In (Mettes et al., 2019), the pre-defined prototypes positioned through data-independent optimization are used to obtain a large class margin. As shown in Figure 2, although they keep the particularly large margin
from the beginning, the sample margin is smaller than that optimized without fixed prototypes, leading to insignificant improvements in accuracy.
In recent years, in the design of variants of softmax loss, one popular approach (Bojanowski & Joulin, 2017; Wang et al., 2017; Mettes et al., 2019; Wang & Isola, 2020) is to perform normalization on prototypes or/and features, leading to superior performance than unnormalized counterparts (Parkhi et al., 2015; Schroff et al., 2015; Liu et al., 2017). However, there is no theoretical guarantee provided yet. In the following, we provide a rigorous analysis about the necessity of normalization. Firstly, we prove that minimizing the original softmax loss without normalization for both features and prototypes may result in a very small class margin:
Theorem 3.1. ∀ε ∈ (0, π/2], if the range of w1, ...,wk or z1, ...,zN is Rd (2 ≤ k ≤ d + 1), then there exists prototypes that achieve the infimum of the softmax loss and have the class margin ε.
This theorem reveals that, unless both features and prototypes are normalized, the original softmax loss may produce an arbitrary small class margin ε. As a corroboration of this conclusion, L-Softmax (Liu et al., 2016) and A-Softmax (Liu et al., 2017) that do not perform any normalization or only do on prototypes, cannot guarantee to maximize the class margin. To remedy this issue, some works (Wang et al., 2017; 2018a;b; Deng et al., 2019) proposed to normalize both features and prototypes.
A unified framework (Deng et al., 2019) that covers A-Softmax (Liu et al., 2017) with feature normalization, NormFace (Wang et al., 2017), CosFace/AM-Softmax (Wang et al., 2018b;a), ArcFace (Deng et al., 2019) as special cases can be formulated with hyper-parameters m1, m2 and m3:
L′i = − log exp(s(cos(m1θiyi +m2)−m3)) exp(s(cos(m1θiyi +m2)−m3)) + ∑ j 6=yi exp(s cos θij) , (3.1)
where θij = ∠(wj , zi). The hyper-parameters setting usually guarantees that cos(m1θiyi +m2)− m3 ≤ cosm2 cos θiyi − m3, and m2 is usually set to satisfy cosm2 ≥ 12 . Let α = cosm2 and β = −m3 < 0, then we have
L′i ≥ − log exp(s(α cos θiyi + β)) exp(s(α cos θiyi + β)) + ∑ j 6=yi exp(s cos θij) , (3.2)
which indicates that the existing well-designed normalized softmax loss functions are all considered as the upper bound of the right-hand side, and the equality holds if and only if θiyi = 0.
Generalized Margin Softmax Loss. Based on the right-hand side of (3.2), we can derive a more general formulation, called Generalized Margin Softmax (GM-Softmax) loss:
Li = − log exp(s(αi1 cos θiyi + βi1)) exp(s(αi2 cos θiyi + βi2)) + ∑ j 6=yi exp(s cos θij) , (3.3)
where αi1, αi2, βi1 and βi2 are hyper-parameters to handle the margins in training, which are set specifically for each sample instead of the same in (3.2). We also require that αi1 ≥ 12 , αi2 ≤ αi1, s > 0, βi1, βi2 ∈ R. For class-balanced case, each sample is treated equally, thus setting αi1 = α1,
αi2 = α2, βi1 = β1 and βi2 = β2, ∀i. For class-imbalanced case, the setting relies on the data distribution, e.g., the LDAM loss (Cao et al., 2019) achieves the trade-off of margins with αi1 = αi2 = 1 and βi1 = βi2 = −Cn−1/4yi . It is worth noting that we merely use the GM-Softmax loss as a theoretical formulation and will derive a more efficient form for the practical implementation.
Wang et al. (2017) provided a lower bound for normalized softmax loss, which relies on the assumption that all samples are well-separated, i.e., each sample’s feature is exactly the same as its corresponding prototype. However, this assumption could be invalid during training, e.g., for binary classification, the best feature of the first class z obtained by minimizing− log exp(sw T 1 z)
exp(swT1 z)+exp(sw T 2 z)
is w1−w2‖w1−w2‖2 rather than w1. In the following, we provide a more general theorem, which does not rely on such a strong assumption. Moreover, we prove that the solutions {w∗j }kj=1, {z∗i }Ni=1 minimizing the GM-Softmax loss will maximize both class margin and sample margin. Theorem 3.2. For class-balanced datasets, w1, ...,wk, z1, ...,zN ∈ Sd−1, d ≥ 2, and 2 ≤ k ≤ d + 1, learning with GM-Softmax (where αi1 = α1, αi2 = α2, βi1 = β1 and βi2 = β2) leads to maximizing both class margin and sample margin.
As can be seen, for any α1 ≥ 12 , α2 ≤ α1, s > 0, and β1, β ∈ R, minimizing the corresponding GM-Softmax loss produces the same optimal solution, even though they are intuitively designed to handle different decision boundaries. Moreover, as a special case, we have Proposition 3.3. For class-balanced datasets, w1, ...,wk, z1, ...,zN ∈ Sd−1, d ≥ 2, and 2 ≤ k ≤ d+ 1, learning with the loss functions A-Softmax (Liu et al., 2017) with feature normalization, NormFace (Wang et al., 2017), CosFace (Wang et al., 2018b) or AM-Softmax (Wang et al., 2018a), and ArcFace (Deng et al., 2019) share the same optimal solution.
Although these losses theoretically share the same optimal solution, in practice they usually meet sub-optimal solutions under different hyper-parameter settings when optimizing a neural network, which is demonstrated in Table 1. Moreover, these losses are complicated and possibly redundantly designed, leading to difficulties in practical implementation. Instead, we suggest a concise and easily implemented regularization term and a loss function in the following.
Sample Margin Regularization. In order to encourage learning towards the largest margins, we can explicitly leverage the sample margin (2.5) as the loss, which is defined as:
Rsm(x, y) = −(wTy z −max j 6=y wTj z). (3.4)
Noticeably, the empirical risk 1N ∑N i=1Rsm(xi, yi) is a lower-bounded surrogate of −γmin, i.e.,
−γmin ≥ 1N ∑N i=1Rsm(xi, yi), while directly minimizing −γmin is too difficult to optimize neural networks. When k ≤ d+1, learning withRsm will promote the learning towards the largest margins: Theorem 3.4. For class-balanced datasets, w1, ...,wk, z1, ...,zN ∈ Sd−1, d ≥ 2, and 2 ≤ k ≤ d+ 1, learning with Rsm leads to the maximization of both class margin and sample margin.
Although learning with Rsm theoretically achieves the largest margins, in practical implementation, the optimization by the gradient-based methods shows unstable and non-convergent results for large scale datasets. Alternatively, we turn to combine Rsm as a regularization or complementary term with commonly-used losses, which is referred to as sample margin regularization. The empirical results demonstrate its superiority in learning towards the large margins, as depicted in Table 1.
Largest Margin Softmax Loss (LM-Softmax). Theorem 2.2 provides a theoretical guarantee that maximizing γmin will obtain the maximum of class margin regardless of feature dimension, class number, and class balancedness. It offers a straightforward approach to meet our purpose, i.e., learning towards the largest margins. However, directly maximizing γmin is difficult to optimize a neural network with only one sample margin. As a consequence, we introduce a surrogate loss for balanced datasets, which is called the Largest Margin Softmax (LM-Softmax) loss:
L(x, y; s) = −1 s log exp(swTy z)∑ j 6=y exp(sw T j z) = 1 s log ∑ j 6=y exp(s(wj −wy)Tz) (3.5)
which is derived by the limiting case of the logsumexp operator, i.e.. we have −γmin = lims→∞ 1 s log( ∑N i=1 ∑ j 6=yi exp(s(w T j zi − wTyizi))). Moreover, since log is strictly concave, we
can derive the following inequality
1 s log( N∑ i=1 ∑ j 6=yi exp(s(wTj zi −wTyizi))) ≥ 1 N N∑ i=1 L(xi, yi; s) + 1 s logN. (3.6)
Minimizing the right-hand side of (3.6) usually leads to that ∑ j 6=yi exp(s(w T j z − wTyiz)) is a
constant, while the equality of (3.6) holds if and only if ∑ j 6=yi exp(s(w T j z −wTyiz)) is a constant. Thus, we can achieve the maximum of γmin by minimizing L(x, y; s) defined in (3.5).
It can be found that, LM-Softmax can be regarded as a special case of the GM-Softmax loss when α2 or β2 approaches−∞, which can be more efficiently implemented than the GM-Softmax loss. With respect to the original softmax loss, LM-Softmax removes the term exp(swTy z) in the denominator.

heading: 3.2 CLASS-IMBALANCED CASE
text: Class imbalance is ubiquitous and inherent in real-world classification problems (Buda et al., 2018; Liu et al., 2019). However, the performance of deep learning-based classification would drop significantly when the training dataset suffers from heavy class imbalance effect. According to (2.7), enlarging the sample margin can tighten the upper bound in case of class imbalance. To learn towards the largest margins on class-imbalanced datasets, we provide the following sufficient condition: Theorem 3.5. For class-balanced or -imbalanced datasets, w1, ...,wk, z1, ...,zN ∈ Sd−1, d ≥ 2, and 2 ≤ k ≤ d + 1, if ∑K i=1wi = 0, learning with GM-Softmax in (3.3) leads to maximizing both class margin and sample margin.
This theorem reveals that, if the centroid of prototypes is equal to zero, learning with GM-Softmax will provide the largest margins.
Zero-centroid Regularization. As a consequence, we propose a straight regularization term as follows, which can be combined with commonly-used losses to remedy the class imbalance effect:
Rw{wj}kj=1 = λ ∥∥1 k k∑ j=1 wj ∥∥2 2 . (3.7)
The zero-centroid regularization is only applied to prototypes at the last inner-product layer.

heading: 4 EXPERIMENTS
text: In this section, we provide extensive experimental results to show superiority of our method on a variety of tasks, including visual classification, imbalanced classification, person ReID, and face verification. More experimental analysis and implementation details can be found in the appendix.

heading: 4.1 VISUAL CLASSIFICATION
text: To verify the effectiveness of the proposed sample margin regularization in improving inter-class separability and intra-class compactness, we conduct experiments of classification on balanced datasets MNIST (LeCun et al., 1998), CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009). We evaluate performance with three metrics: 1) top-1 validation accuracy acc; 2) the class margin mcls defined in (2.2); 3) the average of sample margins msamp. We use a 4-layer CNN, ResNet-18, and ResNet-34 on MNIST, CIFAR-10, and CIFAR-100, respectively. Moreover, some commonlyused neural units are considered, such as ReLU, BatchNorm, and cosine learning rate annealing. We use CE, CosFace, ArcFace, NormFace as the compared baseline methods. Note that CosFace, ArcFace, NormFace have one identical hyper-parameter s, which is used for comprehensive study.
Results. As shown in Table 1, all baseline losses fail in learning with large margins for all s, in which the class margin decreases as s increases. There is no significant performance difference among them. In contrast, by coupling with the proposed sample margin regularization Rsm, the losses turn to have larger margins. The results demonstrate that the proposed sample margin regularization is really beneficial to learn towards the possible largest margins. Moreover, the enlargement on class margin and sample margin means better inter-class separability and intra-class compactness, which further brings the improvement of classification accuracy in most cases.

heading: 4.2 IMBALANCED CLASSIFICATION
text: To verify the effectiveness of the proposed zero-centroid regularization in handling class-imbalanced effect, we conduct experiments on imbalanced classification with two imbalance types: long-tailed imbalance (Cui et al., 2019) and step imbalance (Buda et al., 2018). The compared baseline losses include CE, Focal Loss, NormFace, CosFace, ArcFace, and the Label-Distribution-Aware Margin Loss (LDAM) with hyper-parameter s = 5. We follow the controllable data imbalance strategy in (Maas et al., 2011; Cao et al., 2019) to create the imbalanced CIFAR-10/-100 by reducing the number of training examples per class and keeping the validation set unchanged. The imbalance ratio ρ = maxi ni/mini ni is used to denote the ratio between sample sizes of the most frequent and least frequent classes. We add zero-centroid regularization to the margin-based baseline losses and the proposed LM-Softmax to verify its validity. We report the top-1 validation accuracy acc and class margin mcls of compared methods.
Results. As can be seen from Table 2, the baseline margin-based losses have small class margins, although their classification performances are better than CE and Focal, which largely attribute to the normalization on feature and prototype. We can further improve their classification accuracy by
enlarging their class margins through the proposed zero-centroid regularization, as demonstrated by results in Table 2. Moreover, it can be found that the class margin of our LM-Softmax loss is fairly low in the severely imbalanced cases, since it is tailored for balanced case. We can also achieve significantly enlarged class margins and improved accuracy by the zero-centroid regularization.

heading: 4.3 PERSON RE-IDENTIFICATION
text: We conduct experiments on the task of person re-identification. Specifically, we use the off-the-shelf baseline (Luo et al., 2019) as the main code to verify the effectiveness of our proposed LM-Softmax. We follow the default parameter settings and training strategy, and train the ResNet50 with Triplet Loss Schroff et al. (2015) coupling with the compared losses, including the Softmax loss (CE), ArcFace, CosFace, NormFace, and our proposed LM-Softmax. Experiments are conducted on Market-1501 (Zheng et al., 2015) and DukeMTMC (Ristani et al., 2016). As shown in Table 3, our proposed LM-Softmax obtains obvious improvements in mean Average Precision (mAP), rank-1(Rank@1), and rank-5
(Rank@5) matching rate. Moreover, LM-Softmax exhibits significant robustness for different parameters, while ArcFace, CosFace, and NormFace show worse performance than ours and are more sensitive to parameter settings.
4.4 FACE VERIFICATION
We also verify our method on face verification that highly depends on the discriminability of feature embeddings. Following the settings in (An et al., 2020), we train the compared models on a large-scale dataset MS1MV3 (85K IDs/ 5.8M images) (Guo et al., 2016) and test on LFW (Huang et al., 2008), CFP-FP (Sengupta et al., 2016), AgeDB-30 (Moschoglou et al., 2017) and IJBC (Maze et al., 2018). We use ResNet34 as the backbone, and train it with batch size 512 for all compared methods. The comparison study includes CosFace, ArcFace, NormFace, and our LM-Softmax.
As shown in Table 4, Rsm (sample margin regularization) and Rw (zero-centroid regularization) can improve the performance of these baselines in most cases. Moreover, it is worth noting that the results of LM-Softmax are slightly worse than ArcFace and CosFace, which is due to that in these large-scale datasets there exists class imbalanced effect more or less. We can alleviate this issue by adding Rw, which can improve the performance further.

heading: 5 CONCLUSION
text: In this paper, we attempted to develop a principled mathematical framework for better understanding and design of margin-based loss functions, in contrast to the existing ones that are designed heuristically. Specifically, based on the class and sample margins, which are employed as measures of intra-class compactness and inter-class separability, we formulate the objective as learning towards the largest margins, and offer rigorously theoretical analysis as support. Following this principle, for class-balance case, we propose an explicit sample margin regularization term and a novel largest margin softmax loss; for class-imbalance case, we propose a simple but effective zero-centroid regularization term. Extensive experimental results demonstrate that the proposed strategy significantly improves the performance in accuracy and margins on various tasks.
Acknowledgements. This work was supported by National Key Research and Development Project under Grant 2019YFE0109600, National Natural Science Foundation of China under Grants 61922027, 6207115 and 61932022.

heading: Appendix for “Learning Towards the Largest Margin”

heading: A PROOFS
text: Lemma 2.1. For any w1, ...,wk ∈ Sd−1, d ≥ 2, and 2 ≤ k ≤ d + 1, the solution of minimal Riesz t-energy and k-points best-packing configurations are uniquely given by the vertices of regular (k − 1)-simplices inscribed in Sd−1. Furthermore, wTi wj = −1k−1 , ∀i 6= j.
Proof. See in Borodachov et al. (2019, Theorem 3.3.1).
Theorem 2.2. For w1, ...,wk, z1, ...,zN ∈ Sd−1 (where nj > 0 for each j ∈ [1, k]), the optimal solution {w∗i }ki=1, {z∗i }Ni=1 = arg max{wi}ki=1,{zi}Ni=1 γmin is obtained if and only if {w ∗ i }ki=1 maximizes the class margin mc({wi}ki=1), and z∗i = w∗yi −w∗yi
‖w∗yi−w ∗ yi ‖2 , where w
∗ yi denotes the centroid of
the vectors {wj : j maximizes wTyiwj , j 6= yi}.
Proof. According to the definition of γmin, we have
arg max w max z γmin = arg max w max z min i wTyizi −maxj 6=yi wTj zi
= arg max w min i max zi wTyizi −maxj 6=yi wTj zi
= arg max w min i max zi wTyizi −w T k zi
= arg max w min i ‖wyi −wk‖2
,
where k = arg maxj 6=yi w T j zi, and zi = wyi−wk ‖wyi−wk‖2 . Notice that wTk zi = − 12‖wyi −wk‖2, then k = arg minj 6=yi ‖wTyi −wj‖2. Therefore, we have
arg max w max z γmin = max w min i min k 6=yi ‖wyi −wk‖2 = arg max w min i 6=j ‖wi −wj‖2,
i.e., maximizing γmin will provide the solution of the Tammes Problem, which also maximizes the class margin.
On the other hand, z∗i maximizes w ∗T yi zi −maxj 6=yi w ∗T j zi, i.e.,
z∗i = arg max zi∈Sd−1 w∗Tyi zi −maxj 6=yi w∗Tj zi
= arg max zi∈Sd−1
w∗Tyi zi −w ∗T yi zi
= w∗yi −w ∗ yi
‖w∗yi −w ∗ yi‖2
,
where w∗yi denotes the centroid of the vectors {wj : j maximizes w T yiwj , j 6= yi}.
Proposition 2.3. For any w1, ...,wk, z1, ...,zN ∈ Sd−1, d ≥ 2, and 2 ≤ k ≤ d + 1, the maximum of γmin is kk−1 , which is obtained if and only if ∀i 6= j, w T i wj = − 1k−1 , and zi = wyi .
Proof. Based on Theorem 2.2, the maximum of γmin is obtained if and only if {wi}ki=1 maximizes the class margin and zi =
wyi−wyi ‖wyi−wyi‖2 , i.e.,wTi wj = − 1k−1 according to Lemma 2.3. At this time,
we have zi = wyi−wyi ‖wyi−wyi‖2 = wyi−(−wyi ) ‖wyi−(−wyi )‖2 = wyi .
Theorem 3.1. ∀ε ∈ (0, π/2], if the range of w1, ...,wk or z1, ...,zN is Rd (2 ≤ k ≤ d + 1), then there exists prototypes that achieve the infimum of the softmax loss and have the class margin ε.
Proof. With the softmax loss, the goal is to optimize the following problem
min {wj}kj=1,{zi}Ni=1
L = − 1 N N∑ i=1 log exp(wTyizi)∑K j=1 exp(w T j zi) .
∀ε ∈ (0, π2 ], we can easily obtain k ≤ d + 1 vectors w ′ 1, ...,w ′ k on the unit sphere Sd−1, such that the angle between any two of them is ε ∈ (0, π2 ).
(1) If the domain of w1, ...,wk is Rd, then let wj = sw′j , and zi = wyi . In this way, we have wTyizi > w T j zi, ∀j 6= yi. The infimum of softmax loss can be obtained by directly increasing s.
(2) If the domain of z1, ...,zk is Rd, then let wj = w′j , and zi = swyi . In this way, we have wTyizi > w T j zi, ∀j 6= yi. The infimum of softmax loss can be obtained by directly increasing s.
In conclusion, without both normalization for both features and prototypes, the original softmax loss may produce an arbitrary small class margin ε.
Theorem 3.2. For class-balanced datasets (i.e., each class has the same number of samples), w1, ...,wk, z1, ...,zN ∈ Sd−1, d ≥ 2, and 2 ≤ k ≤ d + 1, learning with GM-Softmax (where αi1 = α1, αi2 = α2, βi1 = β1 and βi2 = β2) leads to maximizing both the class margin and the sample margin. More specifically, the optimal solution
{w∗j }kj=1, {z∗i }Ni=1 = arg min wj ,zi∈Sd−1
1
N N∑ i=1 − log exp(s(α1w T yizi + β1)) exp(s(α2wTyizi + β2)) + ∑ j 6=yi exp(sw T j zi)
have the largest class margin m∗c = arccos −1 k−1 and the largest sample margin γ ∗ min = k k−1 . The lower bound of the risk is log[exp(s(α1 + β1 − α2 − β2)) + (k − 1) exp(−s( 1k−1 + α1 + β1))], which is obtained if and only if ∀i 6= j, wTi wj = −1k−1 , and zi = wyi .
Proof. Since the function exp is strictly convex, using the Jensen’s inequality, we have
L = 1
N N∑ i=1 − log exp(s(α1w T yizi + β1)) exp(s(α2wTyizi + β2)) + ∑ j 6=i exp(sw T j zi)
≥ 1 N N∑ i=1 − log exp(s(α1w T yizi + β1)) exp(s(α2wTyizi + β2)) + (k − 1) exp( s k−1 ∑ j 6=iw T j zi)
Let w = 1k ∑k i=1wi, α = α2 − α1, β = β2 − β1, σ = k k−1 , and δ = 1 k−1 + α1, then we have
L ≥ 1 N N∑ i=1 log [ exp(s(αwTyizi + β) + (k − 1) exp(s(σw − δwyi) Tzi − sβ1) ]
≥ 1 N N∑ i=1 log[exp(sα+ sβ) + (k − 1) exp(−s‖σw − δwyi‖2 − sβ1)]
= 1
k k∑ i=1 log[exp(sα+ sβ) + (k − 1) exp(−s‖σw − δwi‖2 − sβ1)]
,
where we use the facts that αwTyizi ≥ α when α ≤ 0, (σw − δwyi) Tzi ≥ −‖σw − δwi‖2 when zi ∈ Sd−1. Due the convexity of the function log[1 + exp(ax + b)] (a > 0), we use the Jensen’s
inequality and obtain that
L ≥ log [ exp(s(α+ β)) + (k − 1) exp(− s
k k∑ i=1 ‖σw − δwi‖2 − sβ1)
]
≥ log exp(s(α+ β)) + (k − 1) exp(− s k √√√√k k∑ i=1 ‖σw − δwi‖22 − sβ1)  = log [ exp(s(α+ β)) + (k − 1) exp(− s
k
√ k(kδ2 − 2kσδ‖w‖22 + kσ2‖w‖22)− sβ1) ] ≥ log[exp(s(α+ β)) + (k − 1) exp(−s(δ + β1))]
= log [ exp(s(α2 − α1 + β2 − β1)) + (k − 1) exp(−s( 1
k − 1 + α1 + β1))
] ,
where in the second inequality we used the Cauchy–Schwarz inequality, and the third inequality is based on that σ ≤ 2δ ⇔ α1 ≥ k−22k−2 , which holds since α1 ≥ 1 2 .
According to the above derivation, the equality holds if and only if ∀i, wT1 zi = ... = wTyi−1zi = wTyi+1zi = ... = w T k zi, w T yizi = 1, zi = − σw−δwyi ‖σw−δwyi‖2
, ‖σw − δw1‖2 = ... = ‖σw − δwk‖2, and w = 0. The condition can be simplified as ∀i 6= j, wTi wj = −1k−1 , and zi = wyi when 2 ≤ d and 2 ≤ k ≤ d+ 1.
Proposition 3.3. For class-balanced datasets, w1, ...,wk, z1, ...,zN ∈ Sd−1, d ≥ 2, and 2 ≤ k ≤ d+ 1, learning with the loss functions A-Softmax (Liu et al., 2017) with feature normalization, NormFace (Wang et al., 2017), CosFace (Wang et al., 2018b) or AM-Softmax (Wang et al., 2018a), and ArcFace (Deng et al., 2019) share the same optimal solution.
Proof. A unified framework for A-Softmax with feature normalization, NormFace, LMLC/AMSoftmax and ArcFace can be implemented with hyper-parameters m1, m2 and m3, i.e.,
L′i = − log exp(s(cos(m1θiyi +m2)−m3)) exp(s(cos(m1θiyi +m2)−m3)) + ∑ j 6=yi exp(s cos θij) ,
where θij = ∠(wj , zi). The setting of these hyper-parameters always guarantees that cos(m1θiyi + m2) −m3 ≤ cosm2 cos θiyi −m3, and m2 is usually set to satisfy cosm2 ≥ 12 . Let α = cosm2 and β = −m3 < 0, then we have
L′i ≥ − log exp(s(α cos θiyi + β)) exp(s(α cos θiyi + β)) + ∑ j 6=yi exp(s cos θij) , (A.1)
where the equality holds if and only if θiyi = 0.
According to Theorem 3.2, we know that the empirical risk of the loss function in the right-hand side of (3.2) has a lower bound, then we obtain
1
N N∑ i=1 L′i ≥ − log exp(s(α+ β)) exp(s(α+ β)) + ∑ j 6=yi exp(− s k−1 )
(A.2)
The equality holds if and only if ∀i 6= j, wTi wj = −1k−1 , and zi = wyi . Since zi = wyi means θiyi = 0, indicating that the equality in (3.2) holds. Then the optimal solution is the same for A-Softmax with feature normalization, NormFace, CosFace, and ArcFace.
Theorem 3.4. For class-balanced datasets, w1, ...,wk, z1, ...,zN ∈ Sd−1, d ≥ 2, and 2 ≤ k ≤ d + 1, learning with Rsm = −wTy z + maxj 6=ywTj z leads to the maximization of the class margin and the sample margin.
Proof. Let L(z, y) = −wTy z + maxj 6=ywTj z, w = 1k ∑k i=1wi, then we have
1
N N∑ i=1 L(zi, yi) = 1 N N∑ i=1 (−wTyizi + maxj 6=yi wTj zi)
≥ 1 N N∑ i=1 (−wTyizi + 1 k − 1 ∑ j 6=yi wTj zi)
= k
N(k − 1) N∑ i=1 −(wyi −w)Tzi
≥ k N(k − 1) N∑ i=1 −‖wyi −w‖2
= 1
k − 1 k∑ i=1 −‖wi −w‖2
≥ 1 k − 1 − √√√√k( k∑ i=1 ‖wi −w‖22)
= 1 k − 1 − √ k(k − k‖w‖22)
≥ − k k − 1
(A.3)
where the equality holds if and only if ∀i 6= j, wTi wj = −1k−1 , and zi = wyi .
Theorem 3.5. For class-balanced or -imbalanced cases, w1, ...,wk, z1, ...,zN ∈ Sd−1, d ≥ 2, and 2 ≤ k ≤ d + 1, if ∑K i=1wi = 0, then learning with the GM-Softmax loss in (3.3) leads to maximizing both the class margin and the sample margin. More specifically, the optimal solution {w∗j }Kj=1, {z∗i }Ni=1 has the largest class margin m(W ∗) = arccos −1K−1 and the largest sample margin γ∗min = k k−1 . The lower bound of the risk is 1 N ∑N i=1 log[exp(s(αi1 + βi1 − αi2 − βi2)) + (k − 1) exp(−s( 1k−1 + αi1 + βi1))], which is obtained if and only if ∀i 6= j, w T i wj = −1 K−1 , and zi = wyi , i.e., the optimal solution maximizes that class margin and sample margin.
Proof. For the GM-Softmax loss Li = − log exp(s(αi1w
T yi zi+βi1))
exp(s(αi2wTyi zi+βi2))+ ∑ j 6=yi exp(swTj zi) , let αi = αi2−
αi1 ≤ 0, βi = βi2 − βi1. If ∑K i=1wi = 0, then we have
Li = − log exp(s(αi1w
T yizi + βi1)) exp(s(αi2wTyizi + βi2)) + ∑ j 6=yi exp(sw T j zi)
≥ − log exp(s(αi1w
T yizi + βi1))
exp(s(αi2wTyizi + βi2)) + (k − 1) exp( 1 k−1 ∑ j 6=yi sw T j zi)
= − log exp(s(αi1w
T yizi + βi1))
exp(s(αi2wTyizi + βi2)) + (k − 1) exp(− s k−1w T yizi)
= log [ exp(sαiw T yizi + sβi) + (k − 1) exp(− s
k − 1 wTyizi − s(αi1w T yizi + βi1)) ] ≥ log [ exp(s(αi1 + βi1 − αi2 − βi2)) + (k − 1) exp(−s(1/(k − 1) + αi1 + βi1)) ]
. (A.4)
where in the first inequality we used the Jensen’s inequality, and the last inequality comes from the facts that αiwTyizi ≥ αi and − 1 k−1w T yizi − αi1w T yizi ≥ − 1 k−1 − αi1.
Therefore, we have the lower bound of the risk 1N ∑N i=1 Li ≥ 1 N ∑N i=1 log[exp(s(αi1 + βi1 − αi2 − βi2)) + (k − 1) exp(−s( 1k−1 + αi1 + βi1))], where the equality holds if and only if ∀ i,
wT1 zi = ... = w T yi−1zi = w T yi+1zi = ... = w T k zi, andw T yizi = 1. The condition can be simplified as ∀i 6= j, wTi wj = −1k−1 , and zi = wyi when 2 ≤ d and 2 ≤ k ≤ d+ 1.

heading: B MORE ANALYSIS
text: In this section, we provide more analysis about the unified framework of margin-based losses in (3.2), Sample Margin Regularization, Largest-Margin Softmax (LM-Softmax) loss.
B.1 A UNIFIED FRAMEWORK
A unified framework that covers A-Softmax (Liu et al., 2017) with feature normalization, NormFace (Wang et al., 2017), CosFace/AM-Softmax (Wang et al., 2018b;a) and ArcFace (Deng et al., 2019) as special cases can be formulated with hyper-parameters m1, m2 and m3:
L′i = − log exp(s(cos(m1θiyi +m2)−m3)) exp(s(cos(m1θiyi +m2)−m3)) + ∑ j 6=yi exp(s cos θij) , (B.1)
where θij = ∠(wj , zi). In the following, we provide the details of the derivation from (3.1) to (3.2)
For the parameter m1, it satisfies that cos(m1θ) ≤ cos(θ) in SphereFace Liu et al. (2017). Therefore, based on the definition of the multiplicative-angular operator, we have cos(m1θiyi + m2) ≤ cos(θiyi + m2). To better understand the theoretical optimal solution, we make the constraint that θiyi ∈ [0, π2 ], which is reasonable because the unique minimizer of these losses, like SphereFace, CosFace, and ArcFace, should satisfy θ∗iyi = 0, rather than belongs to ( π 2 , π].
As for m2, ArcFace did not analyze its range. Instead, we can easily derive that 0 ≤ m2 ≤ π2 . Otherwise, the minimum of ArcFace will be obtained at θiyi = π, since cos(θiyi +m2) ≤ cos(π + m2) whenm2 > π2 , which is ridiculous. Therefore, for θiyi ,m2 ∈ [0, π 2 ] , we have cos(θiyi +m2) = cos θiyi cosm2− sin θiyi sinm2 ≤ cosm2 cos θiyi , which is the main derivation from (3.1) to (3.2).
B.2 SAMPLE MARGIN REGULARIZATION
The sample margin regularization term in (3.4) actually encourages the feature representation z to be similar to the corresponding prototype wy , and push z away from the most similar one of the other prototypes. The concept is similar to contrastive learning, where the most similar one of the other prototypes can be regarded as the hardest negative representation. And we also have
Rsm(x, y) ≤ −wTy z + 1 k − 1 ∑ j 6=y wTj z, (B.2)
where the right side can be regarded as pushing z away from the centroid 1k−1 ∑ j 6=ywj . Therefore, we also use the right side as a sample margin regularization.
B.3 MORE CLARIFICATIONS
As shown in the main paper, GM-Softmax loss, LM-Softmax loss, Sample Margin regularization, and Zero-centroid regularization serve different purposes. More specifically,
• The GM-Softmax loss is only derived as a theoretical formulation, which is not used for practical implementation.
• The LM-Softmax loss is tailored to obtain large margins with only one hyper-parameter. It can be used to replace popular margin-based losses, such as CosFace, and ArcFace, to obtain better discriminativeness of feature representations. Compared with NormFace Wang et al. (2017), LM-Softmax achieves much better performance on the task of person ReID, as shown in Table 3. This demonstrates that removing the term exp(swTy z) in the denominator is helpful, which enforces LM-Softmax to have a stronger fitting ability.
• The sample margin regularization Rsm serves as a general regularization term to significantly improve the ability of learning towards the largest margins by combining it with
the commonly-used losses. Sample margin is not new, but to the best of our knowledge, we are the first one to use it in deep learning to obtain feature representations with interclass separability and intra-class compactness. Although theoretically learning with Rsm can achieve the largest margins, we verify by experiments that directly maximizing sample margin cannot optimize neural networks well on complex datasets, such as CIFAR-100, as shown in Table 5. It can be found that learning with Rsm suffers from the underfitting problem on CIFAR-100, whose performance is much worse than CE. Alternatively, we turn to use Rsm as a regularization term, which can significantly improve the performance of commonly-used CE loss. These results demonstrate that using sample margin as the regularization term is more beneficial than using it as the loss. This is our new contribution to the classical sample margin.
• The zero-centroid regularization Rw is specially tailored for class-imbalanced cases, which is only applied to prototypes at the last inner-product layer. Therefore, it can be easily embedded into the DNN-based methods to handle class imbalance.

heading: C EXPERIMENTS
text: In this section, we provide the experimental details, including datasets, network architectures, parameter settings, analysis, and more results. All codes are implemented by PyTorch (Paszke et al., 2019).
We first recall the sample margin regularizationRsm = −wTy z+maxj 6=ywTj z and the zero-centroid regularization Rw = ‖ 1k ∑k i=1wi‖22, which are used to enlarge margins for baseline methods. As for the trade-off parameter settings µ and λ in the following experiments. We use µ and λ′ = 100λ denote the trade-off parameters for Rsm and Rw, i.e., L + µRsm and L + 100λRw, respectively. In the following, we set µ = 0.5, 1.0 for Rsm, and λ = 1, 2, 5, 10, 20 for Rw.
C.1 TOY EXPERIMENT
We conduct a toy experiment to show the inter-class separability and intra-class compactness using different losses, where we randomly generate prototypes W ∈ Rk×d (we set d = 3 and k = 8), and initialize features Z ∈ RN×d (we set N = 10k). Our goal is to optimize both W and Z to learn the largest class margin and sample margin with different losses. According to the Tammes problem for N = 8, the optimal solution of W and Z satisfies that mc(W ) = 74.86◦ (Whyte, 1952). The number of training epochs is set 500,000. We use cosine learning rate annealing with Tmax=10,000, and SGD optimizer with momentum 0.9 and weight decay 1e− 4. Results. We use green points and red arrows to denote the learned feature vectors and prototype vectors, respectively. As shown in Fig. 3, the learned prototypes are separated well with NormFace, CosFace, ArcFace, and LM-Softmax. Specifically, the class margin of learning with the losses NormFace, CosFace, ArcFace, and LM-Softmax are 68.50◦, 71.86◦, 67.74◦ and 74.46◦, respec-
tively. As we can seen, ArcFace has a smaller class margin (67.74◦) than the others, and the intraclass compactness for NormFace and CosFace is worse than LM-Softmax. The features in the blue box of Fig. 3(a) and Fig. 3(b) are not compact enough, but ArcFace and LM-Softmax do. Moreover, our proposed LM-Softmax shows better performance in class margin and sample margins, where the learned prototypes have the class margin close to the theoretical optima, and the features are perfectly optimized to be their corresponding prototypes.
C.2 VISUAL CLASSIFICATION
We introduce three metrics to evaluate whether a loss function owns good inter-class separability and intra-class compactness. The first one is the top-1 test accuracy acc to measure the generalization of the trained models. The second one is the class margin mcls defined in Eq. (2). And the last one we define as the average of sample margins with cosine similarities, i.e.,
msamp = 1 N ∑N i=1 wTyi φΘ(xi)
‖wyi‖‖φΘ(xi)‖ − maxj 6=yi
wTyi φΘ(xi)
‖wj‖‖φΘ(xi)‖ . Then we experiments with a 4-layer CNN, ResNet-18 and ResNet-34 (He et al., 2016) on MNIST (LeCun et al., 1998), CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009), respectively. Moreover, some commonly-used neural layers are considered, such as ReLU (Glorot et al., 2011), BatchNorm (Ioffe & Szegedy, 2015), and cosine learning rate annealing (Loshchilov & Hutter, 2016).
Datasets. We empirically investigate the performance of learning towards the largest margins on benchmark datasets including MNIST (LeCun et al., 1998), CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009).
Training details. We use a simple CNN which consists of Conv(1, 32, 3)→ BatchNorm (Ioffe & Szegedy, 2015)→ ReLU (Glorot et al., 2011)→MaxPool(2,2)→ Conv(32, 64, 3)→ BatchNorm→ ReLU→ MaxPool(2,2)→ Linear() for MNIST, a ResNet-18 (He et al., 2016) for CIFAR-10, and a ResNet-34 (He et al., 2016) for CIFAR-100. The number of training epochs is set 100, 200 and 250 for MNIST, CIFAR-10, and CIFAR-100, respectively. For all training, we use SGD optimizer with momentum 0.9 and cosine learning rate annealing (Loshchilov & Hutter, 2016) when Tmax is equal to the corresponding epochs. Weight Decay is set to 1 × 10−4 for MNIST, CIFAR-10, and CIFAR-100. The initial learning rate is set to 0.01 for MNIST and 0.1 for CIFAR-10 and CIFAR-
100.Moreover, batch size is set to 256. Typical data augmentations including random width/height shift and horizontal flip are applied.
Baselines and hyper-parameter settings. We consider the baseline methods, including the commonly-used loss function CE, and margin-based loss functions NormFace, CosFace, and ArcFace with normalization for both feature vectors and class centers, and our proposed LM-Softmax loss. We have tuned their hyper-parameters for the best performance, and the specific settings are: for CosFace, we set m = 0.1; for ArcFace, we set m = 0.1. To learn towards the largest margins, we boost them with the sample margin regularization, and the trade-off parameter is set to 0.5 and 1. Moreover, we tune their identical hyper-parameter s, and show them for a comprehensive study.
Results. The test accuracy, class margin and the average of all sample margins are reported in Table 1. As we can see, the baseline methods fail in learning large margins for all s, and there is no significant difference in the performance of these losses. More specifically, the class margin decreased as s increases, while the losses with the sample margin regularization Rsm usually remain the large class margins, and the class margins are close to the optimal results (arccos(−1/9) = 96.37◦ for MNIST and CIFAR-10, and arccos(−1/99) = 90.57◦ for CIFAR-100). To better describe the the inter-class separability and intra-class compactness, we provide the histograms of sample margins and similarities between the learned features and their corresponding prototype that they belong to. In Fig. 9, the similarities in Fig. 9(a) are mainly concentrated in 0.8 for CosFace with s = 64, while the similarities in Fig. 9(b) and 9(c) are very close to 1. This indicates that the sample margin regularization significantly improves the inter-class compactness (the learned features in the same class are very similar to their corresponding prototype.) Moreover, the histograms of our proposed LM-Softmax on CIFAR-10 and CIFAR-100 are reported in Fig. 5 and 6, respectively. The similarities and sample margins keep very large with different s. More visualizations are provided in the following figures.
Clarification. As shown in table 5, the proposed method results in both more larger class margin and more larger sample margin than the compared methods, however, the accuracy of the proposed method is slightly better than accuracies of the compared methods. acc actually evaluate the proportion of samples whose sample margin is larger than 0, i.e., acc = 1N ∑N i=1 I(γ(xi, yi) > 0). acc is a good evaluation criterion for classification but is not good enough to measure the quality of feature representation. This is also one of the motivations of the previous works to improve the original softmax loss. In this paper, we measure the inter-class separability and intra-class compactness by class margin and sample margin, which can be used as two criteria to evaluate the quality of feature representations. Thus, acc, class margin, and sample margin can be regarded as different criteria.
Although the relationship of acc and margins is not so straightforward, enlarging the margins can improve acc to some extent. As shown in Table 1, we can see that enlarging the margins of other losses by adding the sample margin regularization Rsm can improve the accuracy in most cases. Moreover, as shown in Table 2, the results on imbalanced learning are noteworthy, where the zerocentroid regularization for learning towards the largest margins on imbalanced classification shows obvious improvements in both class margins and accuracy in most cases, and even can improve the performance of LDAM that is tailored for imbalanced learning
C.3 IMBALANCED CLASSIFICATION
Imbalanced CIFAR-10 and CIFAR-100. The original version of CIFAR-10 and CIFAR-100 contains 50,000 training images and 10,000 test images of size 32 × 32 with 10 and 100 classes, respectively. To create their imbalanced version, we follow the setting in (Buda et al., 2018; Cui et al., 2019; Cao et al., 2019), where we reduce the number of training examples per class, and keep the test set unchanged. To ensure that our methods apply to a variety of settings, we consider two types of imbalance: long-tailed imbalance (Cui et al., 2019) and step imbalance (Buda et al., 2018). We use the imbalance ratio ρ to denote the ratio between sample sizes of the most frequent and least frequent class, i.e., ρ = maxi{ni}/mini{ni}. Long-tailed imbalance utilizes an exponential decay in sample sizes across different classes. For step imbalance setting, all minority classes have the same sample size, as do all frequent classes. This gives a clear distinction between minority classes and frequent classes, and the fraction for minority classes is defined as µ. We follow (Cao et al., 2019) and set µ = 0.5 by default.
We report the top-1 test accuracy acc and class margin mcls of various baseline methods, including CE, Focal Loss, NormFace, CosFace, ArcFace, and the Label-Distribution-Aware Margin Loss (LDAM) with hyper-parameter s = 5. Moreover, the proposed LM-Softmax loss actually is greatly affected by data imbalance since it will pay much attention to enlarge the margin between frequent classes and minority classes than other losses rather than any two classes. And we experiment with the LM-Softmax to verify the validity of the enlarging margin method. Moreover, we add the zerocentroid regularization to the losses whose feature and prototypes are normalized for better margins.
Training details. We use ResNet-18 for imbalanced CIFAR-10, and ResNet-34 for imbalanced CIFAR-100. Following in (Cao et al., 2019), we use SGD optimizer with momentum 0.9 and weight decay 2×10−4. The number of training epochs is set 200, and batch size is 128. The initial learning rate is set to 0.1. Moreover, we use the cosine learning rate annealing strategy (Loshchilov & Hutter, 2016) when Tmax is equal to the corresponding epochs.
Baselines and their hyper-parameter settings. We consider the baseline methods, including CE, Focal loss, CosFace, NormFace, ArcFace, LM-Softmax, and the label-distribution-aware margin (LDAM) loss. We set γ = 1 for Folcal, m = 0.35 for CosFace, m = 0.1 for ArcFace with stable results, and the identical hyper-parameter s is set to 5.
Results. The experimental results of imbalanced CIFAR-10 and CIFAR-100 are reported in Table 6. As we can see, the class margin of the LM-Softmax loss is fairly low in the severely imbalanced cases, while the other losses with feature and weight normalization have better performance than CE and Focal. However, their class margins are still small. With the role of the zero-centroid regularization, the class margin has a very obvious improvement in all cases, where the class margins are close to the optimal one (arccos(−1/9) = 96.37◦ for imbalanced CIFAR-10, and arccos(−1/99) = 90.57◦ for imbalanced CIFAR-100). This conclusion holds for any choice of λ. As for the accuracy, there are also good improvements in most cases, especially for imbalanced CIFAR-100. Moreover, compared with the performance of NormFace, it is worth noticing that the improvements of LDAM may heavily rely on the features and prototype normalization even if LDAM is designed for label-distribution-aware margin trade-off. As illustrated in Fig. 20-28, the zero-centroid regularization improves the intra-class compactness, where the cosine similarities between features and their corresponding prototypes they belong to are more concentrated around 1. The experimental results on the task of imbalanced classification. In the class imbalanced scenario, the stronger fitting ability of LM-Softmax however would make the learner care more about the majority classes but neglect the minority classes. This is the reason why LM-Softmax is less stable, which can be alleviated by applying the proposed zero-centroid regularization.
More Comparisons. To better show the effectiveness of zero-centroid regularization, we also construct more comparison to other related works of imbalanced learning, including two-stage methods cRT (Kang et al., 2020) and MiSLAS (Zhong et al., 2021). cRT works in a two-stage manner: firstly learn feature representation from the original imbalanced data, and then retrain the classifier using class-balanced sampling with the first-stage representation frozen. Our proposed zero-centroid regularization Rw can not only render zero-centroid classifier but also produce feature representations with larger margins when directly learning with imbalanced datasets. Thus, our proposed zero-centroid regularization can benefit these two-stage methods. To verify this point, we conduct experiments on ImageNet-LT with backbone ResNet-50 where the experimental settings follow a recent two-stage decoupling method MiSLAS. As shown in the following table, the performance comparison of CE and CE + Rw demonstrate that zero-centroid regularization can significantly improve the representation learning ability of the first stage. Moreover, our zero-centroid regularization Rw can be easily integrated into well-developed two-stage decoupling methods, such as cRT, MiSLAS. As demonstrated by the following results, adding Rw into 1st stage (representation learning only) or both stages (representation learning and classifier learning) all can improve the performance of the original methods.
C.4 PERSON RE-IDENTIFICATION
We conduct experiments on the task of person re-identification. Specifically, we use the off-the-shelf baseline (Luo et al., 2019) as the main code to verify the efficiency of our proposed LM-Softmax.
Training Details. We followed the default parameter settings and training strategy. More specifically, we train the ResNet50 with pre-trained parameters for 60 epochs. Two benchmark datasets Market-1501 (Zheng et al., 2015) and DukeMTMC (Ristani et al., 2016) are evaluated. More-
over, all models are trained with Triplet Loss + the compared losses, including CE, ArcFace, CosFace, NormFace, and the proposed LM-Softmax. Experiments were conducted on Market-1501 and DukeMTMC. As shown in Table 3, our proposed LM-Softmax obtains obvious improvements in mAP, Rank@1 and Rank@5, which also exhibits significant robustness for different parameters. In contrast, ArcFace, CosFace, and NormFace show worse performance than ours and are more sensitive to parameter settings.
C.5 FACE VERIFICATION
Datasets. We also verify our method on the task of face verification whose performance highly depends on the discriminability of feature embeddings. We follow the training settings in (An et al., 2020)1. The model is trained on MS1MV3 with 5.8M images and 85K ids (Guo et al., 2016) and
1https://github.com/deepinsight/insightface/
testing on LFW (Sengupta et al., 2016), CFP-FP [3], AgeDB-30 (Moschoglou et al., 2017) and IJBC (Maze et al., 2018). The detailed results on IJBC-C are shown in Table 9
Training Details We use ResNet34 as the feature embedding model and train it on two GPUs NVIDIA Tesla v100 with batch size 512 for all compared methods. The compared method includes ArcFace, CosFace, NormFace, and our proposed LM-Softmax.
Baselines and hyper-parameter settings. We use the baseline methods including CosFace, ArcFace, NormFace, and our proposed LM-Softmax. For CosFace and ArcFace, we use the hyperparameters followed their original paper, i.e., s = 64 and m = 0.35 for CosFace; s = 64 and m = 0.5 for ArcFace; For NormFace and LMSoftmax, we set s = 64 and s = 32, respectively.

Review summaries:
This paper analyzes large-margin based loss functions. The authors formulate two types of margins, class- and sample-margins, and then analyze lower bounds of various margin-based losses in a unified framework to show that those losses are minimized by the optimizers of class-/sample-margins. While inspecting the existing margin-based losses, they also formulate two practical methods of sample-margin regularization and largest margin softmax loss to further enhance margins of classifiers. The experimental results on some image classification tasks demonstrate that those methods contribute to performance improvement, while favorably being compared with the other methods that induce large-margin classification.

This paper developed a principled mathematical framework for a better understanding and design of loss functions.  Based on the class and sample margins, the proposed method formulated the objective as learning towards the largest margins, and offer rigorously theoretical analysis as support.  For class balanced cases, this paper proposed an explicit sample margin regularization term and a novel largest margin softmax loss; for the class imbalanced cases, this paper proposed a simple but effective zero centroid regularization term, which restricts the centroid of prototypes to be zero.  Extensive experimental results demonstrate that the proposed strategy significantly improves the performance in accuracy and margins for various tasks.

This paper develops a principled mathematical framework for better understanding and design of margin-based loss functions, where the principled optimization objective is formulated as learning towards the largest margins. In the proposed method, class margin and the sample margin are defined as the measure of inter-class separability and the measure of intra-class compactness. Furthermore, the sample margin regularization and zero-centroid regularization are introduced for the class-balanced case and the class-imbalanced case. Experimental results show the effectiveness of the proposed method on imbalanced classification, person re-identification, and face verification.