Learning Towards The Largest Margins

1 INTRODUCTION . Recent years have witnessed the great success of deep neural networks ( DNNs ) in a variety of tasks , especially for visual classification ( Simonyan & Zisserman , 2014 ; Szegedy et al. , 2015 ; He et al. , 2016 ; Howard et al. , 2017 ; Zoph et al. , 2018 ; Touvron et al. , 2019 ; Brock et al. , 2021 ; Dosovitskiy et al. , 2021 ) . The improvement in accuracy is attributed not only to the use of DNNs , but also to the elaborated losses that encourage well-separated feature embeddings ( Musgrave et al. , 2020 ) . In general , the loss is expected to promote the learned features to have maximized intra-class compactness and inter-class separability simultaneously , so as to boost the feature discriminativeness . Softmax loss , which is the combination of a linear layer , a softmax function , and a cross-entropy loss , is the most commonly-used ingredient in deep learning-based classification . However , the softmax loss only learns separable features that are not discriminative enough ( Liu et al. , 2017 ) . To remedy the limitation of softmax loss , many variants have been proposed . Liu et al . ( 2016 ) proposed a generalized large-margin softmax loss , which incorporates a preset constant m multiplying with the angle between samples and the classifier weight of the ground truth class , leading to potentially larger angular separability between learned features . SphereFace ( Liu et al. , 2017 ) further improved the performance of L-Softmax by normalizing the prototypes in the last inner-product layer . Subsequently , ( Wang et al. , 2017 ) exhibited the usefulness of feature normalization when using feature vector dot products in the softmax function . Coincidentally , in the field of contrastive learning , Chen et al . ( 2020 ) also showed that normalization of outputs leads to superior representations . Due to its effectiveness , normalization on either features or prototypes or both becomes a standard procedure in margin-based losses , such as SphereFace ( Liu et al. , 2017 ) , CosFace/AM-Softmax ( Wang et al. , 2018b ; a ) and ArcFace ( Deng et al. , 2019 ) . However , there is no theoretical guarantee provided yet . ∗This work was done as intern at Peng Cheng Laboratory . †Correspondence to : Xianming Liu < csxm @ hit.edu.cn > Despite their effectiveness and popularity , the existing margin-based losses were developed through heuristic means , as opposed to rigorous mathematical principles , modeling and analysis . Although they offer geometric interpretations , which are helpful to understand the underlying intuition , the theoretical explanation and analysis that can guide the design and optimization is still vague . Some critical issues are unclear , e.g. , why is the normalization of features and prototypes necessary ? How can the loss be further improved or adapted to new tasks ? Therefore , it naturally raises a fundamental question : how to develop a principled mathematical framework for better understanding and design of margin-based loss functions ? The goal of this work is to address these questions by formulating the objective as learning towards the largest margins and offering rigorously theoretical analysis as well as extensive empirical results to support this point . To obtain an optimizable objective , firstly , we should define measures of intra-class compactness and inter-class separability . To this end , we propose to employ the class margin as the measure of interclass separability , which is defined as the minimal pairwise angle distance between prototypes that reflects the angular margin of the two closest prototypes . Moreover , we define the sample margin following the classic approach in ( Koltchinskii et al. , 2002 , Sec . 5 ) , which denotes the similarity difference of a sample to the prototype of the class it belongs to and to the nearest prototype of other classes and thus measures the intra-class compactness . We provide a rigorous theoretical guarantee that maximizing the minimal sample margin over the entire dataset leads to maximizing the class margin regardless of feature dimension , class number , and class balancedness . It denotes that the sample margin also has the power of measuring inter-class separability . According to the defined measures , we claim that , to encourage discriminative representation of features , the loss function should promote the largest possible margins for both classes and samples , which also meets to tighten the margin-based generalization bound in ( Kakade et al. , 2008 ; Cao et al. , 2019 ) . The main contributions of this work are highlighted as follows : • For a better understanding of margin-based losses , we provide a rigorous analysis about the necessity of normalization on prototypes and features . Moreover , we propose a generalized margin softmax loss ( GM-Softmax ) , which can be derived to cover most of existing marginbased losses . We prove that , for the class-balance case , learning with the GM-Softmax loss leads to maximizing both class margin and sample margin under mild conditions . • We show that learning with existing margin-based loss functions , such as SphereFace , NormFace , CosFace , AM-Softmax and ArcFace , would share the same optimal solution . In other words , all of them attempt to learn towards the largest margins , even though they are tailored to obtain different desired margins with explicit decision boundaries . However , these losses do not always maximize margins under different hyper-parameter settings . Instead , we propose an explicit sample margin regularization term and a novel largest margin softmax loss ( LM-Softmax ) derived from the minimal sample margin , which significantly improve the class margin and the sample margin . • We consider the class-imbalanced case , in which the margins are severely affected . We provide a sufficient condition , which reveals that , if the centroid of prototypes is equal to zero , learning with GM-Softmax will provide the largest margins . Accordingly , we propose a simple but effective zero-centroid regularization term , which can be combined with commonly-used losses to mitigate class imbalance . • Extensive experimental results are offered to demonstrate that the strategy of learning towards the largest margins significantly can improve the performance in accuracy and class/sample margins for various tasks , including visual classification , imbalanced classification , person re-identification , and face verification . 2 MEASURES OF INTRA-CLASS COMPACTNESS AND INTER-CLASS SEPARABILITY . With a labeled dataset D = { ( xi , yi ) } Ni=1 ( where xi denotes a training example with label yi , and yi ∈ [ 1 , k ] = { 1 , 2 , ... , k } ) , the softmax loss for a k-classification problem is formulated as L = 1 N N∑ i=1 − log exp ( wTyizi ) ∑k j=1 exp ( w T j zi ) = N∑ i=1 − log exp ( ‖wyi‖2‖zi‖2 cos ( θiyi ) ) ∑k j=1 exp ( ‖wj‖2‖zi‖2 cos ( θij ) ) , ( 2.1 ) where zi = φΘ ( xi ) ∈ Rd ( usually k ≤ d + 1 ) is the learned feature representation vector ; φΘ denotes the feature extraction sub-network ; W = ( w1 , ... , wk ) ∈ Rd×k denotes the linear classifier which is implemented with a linear layer at the end of the network ( some works omit the bias and use an inner-product layer ) ; θij denotes the angle between zi and wj ; and ‖ · ‖2 denotes the Euclidean norm , wherew1 , ... , wk can be regarded as the class centers or prototypes ( Mettes et al. , 2019 ) . For simplicity , we use prototypes to denote the weight vectors in the last inner-product layer . The softmax loss intuitively encourages the learned feature representation zi to be similar to the corresponding prototype wyi , while pushing zi away from the other prototypes . Recently , some works ( Liu et al. , 2016 ; 2017 ; Deng et al. , 2019 ) aim to achieve better performance by modifying the softmax loss with explicit decision boundaries to enforce extra intra-class compactness and interclass separability . However , they do not provide the theoretical explanation and analysis about the newly designed losses . In this paper , we claim that a loss function to obtain better inter-class separability and intra-class compactness should learn towards the largest class and sample margin , and offer rigorously theoretical analysis as support . All proofs can be found in the Appendix A . In the following , we define class margin and sample margin as the measures of inter-class separability and intra-class compactness , respectively , which serve as the base for our further derivation . 2.1 CLASS MARGIN . With prototypesw1 , ... , wk ∈ Rd , the class margin is defined as the minimal pairwise angle distance : mc ( { wi } ki=1 ) = min i6=j ∠ ( wi , wj ) = arccos [ max i 6=j wTi wj ‖wi‖2‖wj‖2 ] , ( 2.2 ) where ∠ ( wi , wj ) denotes the angle between the vectors wi and wj . Note that we omit the magnitudes of the prototypes in the definition , since the magnitudes tend to be very close according to the symmetry property . To verify this , we compute the ratio between the maximum and minimum magnitudes , which tends to be close to 1 on different datasets , as shown in Fig . 1 . To obtain better inter-class separability , we seek the largest class margin , which can be formulated as max { wi } ki=1 mc ( { wi } ki=1 ) = max { wi } ki=1 min i 6=j ∠ ( wi , wj ) . Since magnitudes do not affect the solution of the maxmin problem , we perform ` 2 normalization for eachwi to effectively restrict the prototypes on the unit sphere Sd−1 with center at the origin . Under this constraint , the maximization of the class margin is equivalent to the configuration of k points on Sd−1 to maximize their minimum pairwise distance : arg max { wi } ki=1⊂Sd−1 min i 6=j ∠ ( wi , wj ) = arg max { w } ki=1⊂Sd−1 min i 6=j ‖wi −wj‖2 , ( 2.3 ) The right-hand side is well known as the k-points best-packing problem on spheres ( often called the Tammes problem ) , whose solution leads to the optimal separation of points ( Borodachov et al. , 2019 ) . The best-packing problem turns out to be the limiting case of the minimal Riesz energy : arg min { w } ki=1⊂Sd−1 lim t→∞ ∑ i 6=j 1 ‖wi −wj‖t2 = arg max { w } ki=1⊂Sd−1 min i 6=j ‖wi −wj‖2 . ( 2.4 ) Interestingly , Liu et al . ( 2018 ) utilized the minimum hyperspherical energy as a generic regularization for neural networks to reduce undesired representation redundancy . When w1 , ... , wk ∈ Sd−1 , k ≤ d+ 1 , and t > 0 , the solution of the best-packing problem leads to the minimal Riesz t-energy : Lemma 2.1 . For any w1 , ... , wk ∈ Sd−1 , d ≥ 2 , and 2 ≤ k ≤ d + 1 , the solution of minimal Riesz t-energy and k-points best-packing configurations are uniquely given by the vertices of regular ( k − 1 ) -simplices inscribed in Sd−1 . Furthermore , wTi wj = −1k−1 , ∀i 6= j . This lemma shows that the maximum of mc ( { wi } ki=1 ) is arccos ( −1k−1 ) when k ≤ d + 1 , which is analytical and can be constructed artificially . However , when k > d + 1 , the optimal k-point configurations on the sphere Sd−1 have no generic analytical solution , and are only known explicitly for a handful of cases , even for d = 3 .

Summary: This paper developed a principled mathematical framework for a better understanding and design of loss functions.  Based on the class and sample margins, the proposed method formulated the objective as learning towards the largest margins, and offer rigorously theoretical analysis as support.  For class balanced cases, this paper proposed an explicit sample margin regularization term and a novel largest margin softmax loss; for the class imbalanced cases, this paper proposed a simple but effective zero centroid regularization term, which restricts the centroid of prototypes to be zero.  Extensive experimental results demonstrate that the proposed strategy significantly improves the performance in accuracy and margins for various tasks.