SphereFace2: Binary Classification is All You Need for Deep Face Recognition
Authors: Yandong Wen
Weiyang Liu
Adrian Weller
Bhiksha Raj
Rita Singh
Venue: ICLR 2022 Spotlight (2022)

Abstract: State-of-the-art deep face recognition methods are mostly trained with a softmax-based multi-class classification framework. Despite being popular and effective, these methods still have a few shortcomings that limit empirical performance. In this paper, we start by identifying the discrepancy between training and evaluation in the existing multi-class classification framework and then discuss the potential limitations caused by the "competitive" nature of softmax normalization. Motivated by these limitations, we propose a novel binary classification training framework, termed SphereFace2. In contrast to existing methods, SphereFace2 circumvents the softmax normalization, as well as the corresponding closed-set assumption. This effectively bridges the gap between training and evaluation, enabling the representations to be improved individually by each binary classification task. Besides designing a specific well-performing loss function, we summarize a few general principles for this "one-vs-all" binary classification framework so that it can outperform current competitive methods. Our experiments on popular benchmarks demonstrate that SphereFace2 can consistently outperform state-of-the-art deep face recognition methods.

Published as a conference paper at ICLR 2022
SPHEREFACE2: BINARY CLASSIFICATION IS ALL YOU NEED FOR DEEP FACE RECOGNITION
Yandong Wen',, Weiyang Liu Adrian Weller Bhiksha Raj', Rita Singh'
CCanegie Mellon University PUniversity of Cambridge 'MPI for Intelligent Systems "Alan Turing Institute
ABSTRACT
State-of-the-art deep face recognition methods are mostly trained with a softmax- based multi-class classification framework. Despite being popular and effective, these methods still have few shortcomings that limit empirical performance. In this paper, we start by identifying the discrepancy between training and evaluation in the existing multi-class classification framework and then discuss the potential limitations caused by the "competitive" nature of softmax normalization. Motivated by these limitations, we propose novel binary classification training framework, termed SphereFace2. In contrast to existing methods, SphereFace2 circumvents the softmax normalization, as well the corresponding closed-set assumption. This effectively bridges the gap between training and evaluation, enabling the representations be improved individually by each binary classification task. Besides designing a specific well-performing loss function, we summarize a few general principles for this "one-vs-all" binary classification framework so that it can outperform current competitive methods. Our experiments on popular benchmarks demonstrate that SphereFace2 can consistently outperform sate-offthhe-at deep face recognition methods. The code is available at OpenSphere.
INTRODUCTION
Recent years have witnessed the tremendous success of deep face recognition (FR), largely owing to rapid development in training objectives [4, 16-18, 31, 32, 37-39, 42, 43]. Current deep FR methods typically based a multi-class learning objective, e-g., softmax cross-entropy loss [4, 17, 18, 27, 37-39]. Despite its empirical effectiveness, there an obvious discrepancy between such a multi-class classification training and open-set pair-wise testing, as shown Fig. 1. In contrast to multi-class classification, pair-wise verification is a binary problem where we need to determine whether pair of face images belongs to the same person or not. This significant discrepancy may cause the training target to deviate from the underlying FR task and therefore limit the performance. This problem has also been noticed in [29, 31], but they still try to address it under the multi-class (or triplet) learning framework which still fundamentally differs from pair-wise verification in testing. On the other hand, multi-class classification training assumes a closed-set set environment where all the training data must belong to one of the known identities, which is also different from open-set testing.
In order to address these limita- tions, we propose a novel deep face recognition framework com- pletely based binary classifi- cation. SphereFace [17] is one of the earliest works that explic- itly performs multi-class classifi- cation on the hypersphere (i.e., angular space) in deep FR. In light of this, we name our frame- work SphereFace2 because it ex- clusively performs binary classi-
Multiclee Chocciiationn
Binary Coscifration
Pair Verificaiun
Cluss
Sootuux
Which clas
Class KT
Poetiive Sumples
Nagative Samplee
(6) Currest Malti-hhas Leamingg
(bo Our Binory Clasiffioinn Training (Spher
(e) Openvssi Tosing
Figure Comparison between current multi class classification training deep face recog- nition and our hinary classification training.
fications (hence the "2") on the hypersphere. Unlike multi-class classification training. SphereFace2 effectively bridges the gap between training and testing, because both training and testing perform pair- wise comparisons. Morcover, SphercFace2 alleviates the closed-set assumption and views the training as an open-set learning problem. For example, training samples that do not belong to any class still useful and can serve as negative samples in SphereFace2, while they cannot be used
Yandong Wen and Weiyang Liu are co-first authors and contributed equally to this work.

---

Published as a conference paper at ICLR 2022
at all in the multi-class classification framework. Specifically, given K classes in the training set, SphereFace2 constructs K binary classification objectives where data from the target class are viewed as positive samples and data from the remaining classes are negative samples. The spirit is similar to "one-vs-all" classification [28]. An intuitive illustration comparing typical multi-class training, our proposed SphereFace2, and open-set evaluation is given in Fig. 1.
In multi-class classification training, the softmax (cross-entropy) loss introduces competition among different classes, because the softmax normalization requires all the class logits are summed to one. Therefore, increasing the confidence of one class will necessarily decrease the confidence of some other classes, which can easily lead to ever-confident predictions [6]. In contrast to the competitive nature in all the softmax losses, our SphereFace2 encourages each binary classification task to individually improve the discriminativeness of the representation without any competition.
The significance of our method can also be understood from a different perspective. Depending on how samples interact with each other in training, deep FR methods can be categorized into triplet-based learning and pair-based learning. Triplet-based learning simultaneously utilizes an anchor, positive samples and negative samples in its objective, while based learning uses either positive pairs negative pairs in one shot. Based on whether a proxy is used to represent a set of samples, both triplet-based and pair-based methods can learn with without proxies. A categorization of deep FR methods is given in Ta- ble Proxy-free methods usually require expansive pair/triplet mining and most of them [8, 23, 26, 31, 33, 52] use a hybrid loss that includes the standard softmax loss. Typical examples of triplet-based learning with proxy include the softmax loss and its popular variants [4, 17, 37-39] where classifier is class proxy and the learning involves an anchor (i.e., the deep feature x), the positive proxy (i.., the target classifier Wg) and the negative proxies (i.e., . the other classifiers Wj.j * y). Most popular deep FR methods use proxies by default, since they greatly speed up training and improve data-efficiency (especially on large-scale datasets). Distinet from previous deep FR, we believe SphereFace2 is the first work to adopt pair-wise learning paradigm with proxies.
w/o Proxy
P/oxx
DecplD [32]
DeepFacc [35]
DeepIDZ [31]
FaceNet [29]
DecpID24 [33]
Triplet
VGGFace [26]
SphereFace [17]
Triplet [23]
NormFacee [38]
MTER [52]
CosFace [37, 39]
ArcFace [4]
Ring Loss [51]
CurriculaPacce [10]
Circle Loss [34]
Siamese [3]
DeeplD22 [31]
Pair
SphereFace2
DeepID2+ [33]
Coot. CNN [8]
Table I: Overview of deep FR. denotes that the method uses a hybrid loss that combines multi- class softmax loss and contrastive loss.
An outstanding difference be- tween triplet-based and pair- based learning is the usage of universal threshold. In Fig. 2, we show that triplet- based learning compares the similarity scores between dif- ferent pairs, while pair-based learning compares a similarity score and a universal thresh- old. As a pair-based method, SphereFace2 too optimizes the difference between similarity scores and a universal threshold. By learning the threshold to distinguish positive and negative pairs during training, it naturally also generalizes to open-set FR.
Postine Pair
Positise
Negative
Pusitive
Thesdeld
Smilaity
Soore
Negaiive Pair
Negative Pair
Negative Pair
Ponllive Pair
(n) Tripes-
Phir-rsedd Leaming
Figure Comparison between triplet based pair baved learning. Purple aroos optimization directions. Tripet-bused learning compares different similarity scores, while pair- based learning compares similarity soure and threshold.
In our framework, we cast a principled view on designing loss functions by systematically discussing the importance of positive/negative sample balance, easy/hard sample mining and angular margin, and combine these ingredients to propose an effective loss function for SphereFace2. In fact, most popular variants of the softmax loss [4, 37-39] also perform casy/hard sample mining and incorporate angular margin. Moreover, we observe that the distribution of similarity scores from positive pairs quite different from that of negative pairs, in that it exhibits larger variance, which makes the threshold hard to determine. To address this, we propose novel similarity adjustment method that can adjust the distribution of pair similarity and effectively improve the generalizability.
The flexibility of binary classifications in SphereFace2 brings few additional advantages. First, in contrast to the classifier layer with the softmax loss which highly non-trivial to parallelize [1], the
2

---

Published as a conference paper at ICLR 2022
classifier layer in SphereFace2 can be naturally parallelized across different GPUs. While the gradient update in the softmax loss needs to involve all the classifiers, the gradient update for classifiers in SphereFace2 is independent and fully decoupled from each other. The gradient with respect to each classifier only depends on the feature x: and the binary classifier itself. Therefore, SphereFace2 naturally addresses the problem of the softmax loss in training on a large number of face identities [1]. Second, the pair-wise labels used in SphereFace2 are in fact a weaker form of supervision than the class-wise labels used in all the softmax- based losses. This property is closely connected to the open-set assumption and also makes SphereFace2 less sensitive to class-wise label noise. For example, SphereFace2 does not require the class-wiss labels of the negative samples in each binary classification. Instead, long we know that the identities of negative samples are different from the positive samples, SphereFace2 will still work well. Our experiments also empirically verify the robustness of SphereFace2 against class-wise label noise. Our major contributions are:
SphereFace2 constructs a novel binary classification framework for deep FR. To our knowledge, SphereFace2 is the first work in deep FR to adopt a pair-wise learning paradigm with proxies. We further summarize a series of general principles for designing a good loss function for SphereFace2.
The decoupling of the binary classifications Icads to natural parallelization for the classifier layer, making SphereFace2 more scalable than softmax-base losses.
The waie labels used in SphereFace2 are weaker supervision than the class-wise labels used in the softmax loss, yielding better robustness to class-wise label noise.
Related Work. Deep FR methods are either proxy-based or proxy-free, shown in Table 1. Proxy- free learning includes contrastive loss [3, 7, 31J and triplet loss [29]. Both losses directly optimize the similarity between samples, so they are highly unstable and data nata-ineffficient. Proxy- free learning is more widely used in deep metric learning [25, 30, 36, 40, 41, 45]. Proxy-based learning uses a set of proxies represent different groups of samples and usually works better for large-scale datasets. Typical examples include the softmax loss and its variants [4, 17, 32, 34, 35, 37-39] where each proxy is used to represent a class. SphereFace [17] suggests that large-margin classification is more aligned with open-set FR, and proposes to learn features with large angular margin. CosFace [37, 39] and ArcFace [4] introduce alternative ways to learn large-margin features with improved stability. Although large-margin classification brings the training target closer to the task of open-set FR, the discrepancy still exists and it unclear how much the margin should be to close the gap. Moreover, a large margin inevitably leads to training instability. In contrast, SphereFace2 naturally avoids these problems and aligns the training target with open-set FR by adopting pair-wise learning with proxies.
2
THE SPHEREFACE2 FRAMEWORK
2.1
OVERVIEW AND PRELIMINARIES
The goal of SphereFace2 is to align the training target with open-set verification so that our training is more effective in improving open-set generalization in deep FR. To this end, SphereFace2 explicitly incorporates pair-wise comparison into training by constructing K binary classification tasks (K is the number of identities in the training set). The core of SphereFace2 is the binary classification reformulation of the training target. In the i-th binary classification, we construct the positive samples with the face images from the i-th class and the negative samples with face images from other classes. Specifically, we denote the weights of the i-th binary classifier by W., the deep feature by æ and its corresponding label by y. A naive loss formulation is
Ly log (1 exp( W,, bb)) + +Σ log exxe bc))
*
which is combination of K standard binary logistic regression losses. Instead of performing binary classification in a unconstrained space, we perform binary classification on the unit hypersphere by normalizing both classifiers W,, Vi and feature æ. The loss function now becomes
L log (1 + cos(e))) + > log (1 + exp(cco(B.)))
(1)
where 0, is the angle between the i-th binary classifier W, and the sample œ. The biases b,, Vi are usually removed in common practice [4, 17, 19, 37, 39], since they are learned for closed set and cannot generalize to unknown classes. However, we actually find them very useful in SphereFace2, as will be discussed later. For now, we temporarily remove them for notational convenience. One of the unique advantages of such a parameterization for binary classifiers is that it constructs the i-th class positive proxy with W; and the negative proxy with W, Depending on the label, the
3

---

Published as a conference paper at ICLR 2022
training will minimize the angle between and W; or between æ and - W; in order to minimize the loss. This parameterization of positive and negative proxies immediately guarantees minimum hyperspherical energy [12-15] that has been shown to effeetively benefit generalization. Moreover, our parameterization in SphereFace2 has the same number of parameters for the classifier layer as the previous multi-class training, and does not introduce extra overhead. However, naively minimizing the loss in Eq. (1) will not give satisfactory results. Therefore, we explore in the next subsection how to find desirable loss function that works well in the SphereFace2 framework.
2.2
PRINCIPLED VIEW ON LOSS FUNCTION
We emphasize that the exact form of our loss function is in fact not crucial, and the core of SphereFace2 is the spirit of binary classification (i.e., pair-wise learning with proxies), Following such spirit, there are likely many alternative losses that work as well as ours. Besides proposing a specific loss function, we summarize our reasoning for designing a good loss function via a few general principles.
Positive/negative sample balance. The first problem in SphereFace2 how to balance the positive and negative samples. In fact, balancing positive/negative samples has also been considered in triplet loss, contrastive loss and softmax based losses. Contrastive loss achieves the positive/ncgative sample balance by selecting the pairs. Based on [34], triplet-based methods including both triplet loss and softmax- based losses can naturally achieve positive/negative sample balance, since these losses require the presentation of balanced number of both positive and negative samples.
From Eq. (I), the gradients from positive samples and negative samples are highly imbalanced because only one out of K terms computes the gradient for positive samples. A simple yet effective remedy is to introduce a weighting factor À to balance gradients for positive and negative samples:
La = A log (1 [1+eep(-ccos/e
+ (1-A) Σ log ( (1+e)p(cos(A..)
*F where Ae [0, 1] is a hyperparameter that determines the balance between positive and negative samples. One of the simplest ways to set À is based on the fraction of the loss terms, ie., x K -1 when there are K classes in total.
K
3.4
Easy/hard sample mining. An- other crucial criterion for a good loss function is the strategy for mining easy/hard samples, since it closely related to the convergence speed and quality. It is commonly perceived that softmax-bases of easy/hard sample mining, unlike triplet and contrastive losses. How- ever, this is inaccurate. We construct simple example to illustrate how the softmax- -baed loss mines easy/hard samples. We assume a set of cosine logits from four classes is (cos(61),, cos(O2), cos(Os), cos(),) and the target class is exp(s-cos(O,) ))
Luo Olgestivv Valre
4 1.
Los Obgectie
0
Easy Samples
Easy Samples
Hard Samples
Hard Samples
losses are free
4.5
ccaith)
(4) Normalized Suthmax Loss
(b) Our propesed loss (2.)
Figure 3: Loss objectivee value under different target crsine values.
y y- Then we also compute the s-normalized softmax loss L,, = log( Σ. (00) by fixing cos 0..2 i +
and varying cos( from 1 to 1. From the results shown in Fig. 3(a), we can observe that as the scale s increases, the loss for hard samples will be higher and also more sensitive than the loss for easy samples. This makes the neural network focus on optimizing the angles of hard samples and therefore can improve convergence and generalization. The scaling strategy is widely used as common practice in most softmax-basee role of easy/hard sample mining. For the standard softmax cross- crosspcntrppy loss, easy/hard sample mining is dynamically realized by the norm of features and classifiers.
margin losses [4, 34, 37-39], playing an implicit
In our framework, we need a similar strategy to mine easy/hard samples. Inspired by the rescaled softplus function [5], use extra hyperparameter to adjust the curvature of the loss function Lş:
Le log (1 exp(-r-cos((,))) 1-> log (1 + exp(r- exp(rrccs((:)))
(2)
where larger r implies stronger focus on hard samples. We consider an example of one binary classification and Le becomes 1 -log (1+ exp(-r-cos(f))) co,())))) when >= 1. We then plot how the loss value changes by varying cos( from 1 to under different in Fig. 3(b). We observe that when r becomes larger, the loss for easy samples gets closer to zero while the loss for hard samples remains large. Therefore, r can help to mine and reweight easy/hard samples during training.
4

---

Published as a conference paper at ICLR 2022
Angular margin. Learning deep features with large angular margin is arguably one of the most effective criteria to achieve good generaliz- ability on open-set face recognition. SphereFace [17] introduced angular margin to deep face recognition by considering a multiplicative margin.
Angudar Margin
Angelar Margis ns
RF:
Anguis blaspit at
Class
At
Aupular Margin eo Other Clasas
Angilar Maraya fi.
Class
Margin
Anguler
Class
Mazzin
Otter Cloncs
Algular
JF.
Angular
J,
Angula Margin
Marrgin ro
Ocasies Beurddrr
Bountar)
(r) Margin-besod Sofimax
(b) Loss SphereFace2
(6) Less I. for SphereFroe2
CosFace [37, 39] and ArcFace [4] further considered an additive angu-
Figure 4: Intuitive comparison of angular margin in different losses.
lar margin which makes the loss function easier to train. light of these works, we introduce a novel
two-sided angular margin with two adjustable parameters to the SphereFace2 framework:
La log (1 + exp(-r - (cos( mp)))
log (1 + exp(r (cos((.) + m..))
(3)
where my controls the size of the margin for positive samples and ma controls the size of the margin for negative samples. Larger mp and m, lead to larger additive angular margin, sharing similar spirits to CasineFace [37, 39]. Our framework agnostic to different forms of angular margin and all types of angular margin are applicable here. We also apply ArcFace-type margin [4] and multiplicative margin [16, 17] to SphereFace2 and obtain promising results. Details are in Appendix F.
However, quite different from the angular margin in the softmax-basee losses, the angular margin in Eq. has a universal confidence threshold (i.e., cos(O.) 0). Both angular margins my and m. are introduced with respect to this decision boundary cos(0,) (0,)= =0, see Fig. 4(b). This property has both advantages and disadvantages. One of the unique advantages is that our angular margin for each class is added based universally consistent confidence threshold and does not depend the other classifiers, while the angular margin in softmax-basee classifiers. However, it is extremely challenging to achieve the universal threshold 0, which results in training difficulty and instability. To improve training stability, the bias term that has long been forgotten in softmax-based losses comes the rescue. We combine the biases back:
losses will be largely affected by the neighbor
Σ log (1 (1+exp(r-(cosl) + m..) + +b)) (4) Kiyy where b, denotes the bias term for the binary classifier of the i-th class. Since the class-specific bias is not useful in the open-set testing, we will simply use the same bias b for all the classes. The bias b now becomes the universal confidence threshold for all the binary classifications and the baseline decision boundary becomes y co,) -b= instead of r cos((,,) =0, making the training more stable and flexible. The final decision boundary is r (0,)(0,)) mp) + b-=0 for the positive samples and (cos (0,) +m,) + 0 for the negative samples. More importantly, the threshold b is still universal and consistent for each class. An illustration of Eq. 4 is given in Fig. 4. Alternatively, we learnable angular margin where the confidence threshold is still O. Then we can view m - as the positive margin and m, + as the negative margin. Because b is learnable, we only need to focus on my + m, which yields only one effective hyperparameter, For convenience, we simply let mp mn = m and only need to tune m in practice.
Ls log (1 (1+exp(-r- (cos((O)))) m..) b,))
can interpret Eq.
Similarity adjustment. Fig. 5(a),
we observe a large inconsistency be- tween positive and negative pairs
Nanber oc
Napaia Pain
Nagali Paka
Number of Phane
Negadsa Pairs
Nunter
the distribution of cosine similarity. The similarity distribution of negative pairs has smaller variance and is more concentrated, while the positive pairs exhibit much larger variation in sim- ilarity score. The distributional dis- crepancy between positive and nega- tive pairs leads to a large overlap of similarity scores between them, making it difficult to give a clear threshold to separate the positive and negative pairs. This is harmful to generalization. Fig. 5(a) also empirically shows the similarity scores mostly lie in the range of [-0.2, 1]. These observations motivate us to (1) reduce the overlap of the similarity scores between positive and negative pairs, and (2) increase the empirical dynamic range of the similarity score such that the pair similarity can be distributed in a larger space.
Codre Sindiiity (a)t=]
Conter Similrit (b):-3
Costas Simibeity 501--5
Figure 5: Similarity score distribation of positive and negative pairs trained with dif- ferent The evaluation pairs combined from 4 I.FW Age-DB30 [24], CA-LFW [19] and CP-LFW [50].
5

---

Published as a conference paper at ICLR 2022
To this end, propose novel similarity adjustment method. Our basic idea is to construct a monotonic decreasing function g(z)
Adjested Similerlty glowni))
where 2 E [-1, 1, 1] and g(2) [ [-1,1] 1] and then use ((cos(0)) instead
of the original cos(O) ()) to adjust the mapping from angle to similarity
score during training. Considering that the originally learned cos (0)
mostly lies in the range of [-0.2, 1], we require g(z) to map [-0.2, 2, 1] larger range (e.g. [-0.9, 1D. so that if cos(() is learned similarly
as before and still gives the empirical dynamic range of [-0.2, 11, we can end up with g (cos(#)) whose empirical dynamic range becomes
4.2
Simillriy coard)
[-0.9, 1]. Specifically, g(z) is parameterized as g(z) 22 (
Higure g(cos(0)) of diffferent t.
where we typically use = == =0) E [, 1]. In practice, we simply replace the original cosine similarity with g (0)) during training. 1 controls the strength of similarity adjustment. When 1= (()) reduces to the standard cosine similarity. In Fig. 5, we show that the similarity distribution can be modified by increasing the parameter As increases, the overlap between the positive and negative pairs is reduced and their similarity distributions also become more separable. Moreover, the empirical dynamic range of the similarity score is also approximately increased from [-0.2, 1] to 1-0.4, 1]. The empirical results validate the effectiveness of the proposed similarity adjustment.
Final loss function. After combining all the principles and simplifying hyperparameters, we have
L L= log eexx (-r-(g(cox(O,)) -m) mm) 1
log 1exxp (r-(cooss00))) +m))))) (5)
where g(-) has hyperparameter L. In total, there are four hyperparameters X, r, m and l. Each has a unique geometric interpretation, making them easy to tune. Following our design principles, there could be many potential well-performin loss functions that share similar properties to the proposed one. Our framework opens up new possibilities advance deep face recognition.
2.3 GEOMETRIC INTERPRETATION
This subsection provides a comprehensive discussion and visualization to justify our designs and explain the geometric meaning of each hyperparameter. By design, r is the radius of the hypersphere where all the learned features live and is also the magnitude of the features. The bias for the i-th class moves the baseline decision boundary along the direction of its classifier W, The parameter m controls the size of the induced angular margin. We set the output feature dimension as 2 and plot the 2D features trained by SphereFace2 with different margin in Fig. 7. The visualization empirically verifies the following arguments.
The bias moves the decision boundary. From Fig. 7, we can observe that the bias b can be ef- fectively learned to move the de- cision boundary along the classi- fier direction and lead to new universal confidence -6 for all the classes. The bias makes the train- ing easier and more stable while still preserving the unique property that all classes share the same confi- dence for classification. Compared to other deep FR methods, the uni- versal confidence SphereFace2 can help learn a consistent positive/negative pair separation and explicitly encourage a unique and consistent verification threshold during training.
Uirtonieeppt hoooyoyeeee and
-15
Clss:
Angulor Morgin
(b) Margin (.2)
Figure 7: deep learned by SphereFaee22 construct small consisting of 6 facc identities from VGGFawe2 [2]. Dots with the same culor represent samples from face identity.
m. r control the angular margin. Fig. visualizes the baseline decision boundary (denoted as (2) in Fig. 7(b)) and the decision boundary for the positive/negative samples (denoted as (1)((3 in Fig. 7(b)). The distance between the positive and negative decision boundary is 2rm, producing an effective angular margin. The results show that the empirical margin matches our expected size well.
Larger m leads larger angular margin. From Fig. 7, we can empirically compare the deep features learned with m = and m 0.2 and verify that larger m indeed incorporates larger angular margin between different classes. Large inter-class separability also encouraged.

---

Published as a conference paper at ICLR 2022
We also visualize 3D deep features and the decision planes for class with =0 and m= 0.2 in Fig. 8. We can observe that samples from each class well separated with large angular margins, which is consistent with the 2D case. The empirical angular margin also perfectly matches the induced one (i.e., the distance between the positive and negative plane). The results further verify our empirical conclusions drawn from the 2D visualization.
2.4 EFFICIENT MODEL PARALLELIZATION ON GPUS
As it becomes increasingly important for deep FR methods to train on large- scale datasets with million-level identities, bottleneck is the storage of the classifier layer since its space complexity grows linearly with the number of identities. A common solution is to distribute the storage of the classifiers W,, vi and their gradient computations to multiple GPUs. Then each GPU only needs to compute the logits for a subset of the classes. However, the normalization in softmax-bases losses inevitably requires some data communication overhead across different GPUs, resulting in less efficient parallelization. In contrast, the gradient computations in SphereFace2 communication cost is needed. The decoupling among different classifiers makes SphereFace2 suitable for multi-GPU model parallelization.
Figure 8: 3D features.
class-independent and be performed locally within one GPU. Thus no
Specifically, the softmax normalization in the softmax loss involves the computation of all the classifiers, so computing the gradient w.r.f. any classifier will still require the weights of all the other classifiers, which introduces communication overhead across GPUs. In contrast, the loss for SphereFace2 (Eq. (5)) can be rewritten as L = Σ (:W,, a) where fi(-.-) is some differentiable W,
OL function. To compute DW, we only need to compute (f:(W..) which does not involve any other classifiers. Therefore, this gradient computation can be performed locally and does not require any communication overhead. Appendix E compares the gradient of Sphere and the softmax loss.
EXPERIMENTS AND RESULTS
Preprocessing. Each face image is cropped based on the 5 face landmarks detected by MTCNN [48] using a similarity transformation. The cropped image is of size 112 x 112. Each RGB pixel (J0, 255)) normalized to [-1, 1]. We put the details of training and validation datasets in Appendix A.
CNNs. We adopt the same CNNs from [17, 39] for fair comparison. We use 20-layer CNNs in ablations and 64-layer CNNs for the comparison to existing state-of-the-arr methods.
Training and Testing, We use SGD with momentum 0.9 by default. We adopt VGGFace2 [2] as the same training set for all the methods. VGGFace2 contains 3. 1M images from 8.6K identities, as shown in Table 9. The training faces are horizontally flipped for data augmentation. We strictly follow the specific protocol provided in each dataset for evaluations. Given face image, we extract 512D embedding. The final score is computed by the cosine distance of two embeddings. The nearest neighbor classifier and thresholding are used for face identification and verification, respectively.
3.1 ABLATION STUDY AND EXPLORATORY EXPERIMENTS
We perform ablation study on four validation sets: LFW, Age CA-LFW, and CP-LFW. The statistics of these datasets are summarized in Table 9. Following the provided evaluation protocols, we report 1:1 verification accuracy of 6,000 pairs (3,000 positive and 3,000 negative pairs) for each dataset. In addition, we combine these datasets and compute the overall verification accuracy, which serves more accurate metric to evaluate the models.
Design Principles. We start with abla- tions on the designing principles of the loss function. As shown in Table 2, it is quite effective to improve the perfor- mance following the principles to design a loss function. Specifically, the naive binary classification (Eq. 1) fails to con- verge, since the learning objective and the gradients are both dominated by the negative pairs. To address this, set À to K results in the first row of Table 2. As can be seen, while the model is trainable and starts to converge, the results show significant room for improvement. After some tuning of the hyperparameter X,
PN
EH
AM
SA
LFW
AgeDR.30
CA-I.FW
CP-LFFW
Combined
K
X
93.60
71.67
68.40
74.07
74,49
/
x
x
X
95.37
72.90
72.93
76.90
78.03
x
X
98.60
88.10
89.98
85.23
89.65
99.62
92.82
93.07
/
99.50
93.68
93,47
91.07
94,28
Table 2: Ablations of designing principles. PN, BII, AM and SA are the ahbre- viations posirivehegativve sample balance, easy/hard sample mining, angular margin, and similarity adjustment (9)).
K-1 and report the
7

---

Published as a conference paper at ICLR 2022
SphereFace2 can achieve 78.03% accuracy on the combined dataset. Then we gradually incorporate the hard sample mining and angular margin into SphereFace2, improving the verification accuracy from 78.03% to 89.65%, and then to 93.87% With similarity adjustment, SphereFace2 yields 94.28% accuracy on the combined dataset. Table 2 clearly shows the effectiveness of each ingredient.
Hyperparameters. There are four hyperparameters (A, m, and t) in SphereFace2. Since r and have been extensively explored in [4, 37-39], we follow previous practice to fix and m 30 and 0.4 respectively. Our experiments mainly focus on analyzing the effect of À and t.
LFW
AgeDB-30
CA-LFW
CP-LFW
Combined
0.3
99.38
91.38
92.93
88,88
92.72
0.4
99.42
92.30
92.85
89.97
93.38
99.55
92.77
93.32
90.20
0.6
99.48
92.67
93.40
90.10
93.63
0.7
99.58
92.63
93.30
90.33
93.73
0.8
99.62
92.81
93.07
90.85
93.87
0.9
99. 50
92.57
92.82
90.53
93.62
99.53
90.37
91.68
89.33
92.31
From Table 3, the performance of SphereFace2 re- mains stable for À from 0.5 to 0.9, where a good bal- ance for the positive and negative pairs is achieved. Then we evaluate different t for À = [0.6, 0.7, 0.8] and report the results in Table 4. As can be seen from the results, adjusting the similarity scores fur- ther boosts model accuracy. The effect of different t is also illustrated in Fig. 5. More detailed ablation studies are included in Appendix C and D.
Table 3: Effectt of different A. We fix t= and explore how the
model performs with different Às. Results S6.
A
LFW
AgeDB-30
CA-LFW
CP-LFW
Combined
0.6
99.53
93,30
93.37
90.65
94,02
0.6
99.48
93,80
93.53
91.08
94.28
0.7
99.62
93.22
93:35
91.02
94,05
0.7
99.50
93.68
93.47
91.07
94,28
0.8
99.57
93.55
93.28
90.72
94.03
0.8
99.62
93.58
93.38
91.12
94.23
Table 4: Effect of different We explore different ts for several
besting performing As. Results are % and higher better.
State-of-art loss functions. We com- pare SphereFace2 with current state- of-art loss functions in Table 5, We note that these current best- persfrerfrring methods [4, 17, 34, 39] are based on multi-class classification and belong to the triplet learning paradigm. In contrast, our SphereFace2 is the only method that is based on binary classi- fication and adopts the pair-wise learn- ing paradigm. Following [16], we re-
Loss Function
LFW
AgeDB-30
CA-LFW
CP-LFW
Combined
Softmax Loss
98.20
87.23
88.
84.85
89.05
Coco Loss [20]
99. 16
90.23
91.47
89.53
92.4
SphereFace [16, 17]
99.55
92.88
92.55
90.90
93,75
CosFace [39]
99.51
92.98
92.83
91.03
93.89
ArcFace [4]
99.47
91.97
92.47
90.85
93.97
Circle Loss [34]
99.48
92.23
92,90
91.17
93.78
CurricularFace [10]
99.53
92.47
92.90
90.65
93,70
SphereFace2
99.50
93.68
93.47
91.07
94,28
functions. We take the released source code of these metheds and carefully tune the hyperparameters to achieve optimal perfor- mance. Results are in and higher values are
Table Comparison of different
implement SphereFace [17] with hard feature normalization for fair comparison. We observe that the verification accuracies on LFW are saturated around 99.5%. On both AgeDB-30 and CA-LFW datasets, SphereFace2 achieves the best accuracy, outperforming the second best results by a signifi- cant margin. The results on the combined datasets also validate the effectiveness of SphereFace2.
3.2 EVALUATIONS ON LARGE-SCALL BENCHMARKS
We use three challenging face recognition benchmarks, IJB-B, IJB-C, and MegaFace to evaluate SphereFace2 (with A= 0.7, r 40, m 004, 33)). We use 64-layer CNNs for all the methods here.
IJB datasets. UB-B [44] has 21.8K still images (including 11.K faces and 10k non-faces) and 55K frames from 7K videos. The total number of identities is 1,845. We follow the standard 1:1 verification and I:N identification pro- tocols for experiments. The protocol de- fines 12. templates, where each tem- plate consists of multiple images and/or frames. Matching is performed based on the defined templates. Specifically, 10,270 genuine matches and 8M impos- tor matches are constructed in 1:1 verification protocol, and 10,270 probes and 1,875 galleries are constructed in I:N identification protocol. IJB-C is an extension of IJB-B, comprising 3,531 identities with 31.3K still images and 11 17..5K frames from 1 1.8KK videos. The evaluation protocols of IJB-C are similar to IJB-B. The details of the protocols are summarized in Appendix. We report the true accept rates (TAR) at different false accept rates (FAR) for verification, and true positive identification rates (TPIR) at different false positive identification rates (FPIR) for identification, as shown in Table
1:1 Veri. TAR @ FAR
EN Iden. TPIR
@ FPIR
Method
le.5
le.4
le:3
rank-1
le-2
le-1
VGGFaae22 (SENet) [2]
67.1
80.0
88.8
90.1
70.6
83,9
MN-ve [47]
83.1
90.9
Comparator Nets [46]
84,9
93.7
SphereFace
80.75
89.41
94.18
93.49
73.49
87.70
CosFace
79.62
88.61
94.10
92.90
73,80
86.89
ArcFace
80.59
89.11
94.25
93.23
74.81
87.28
Circle Loss
78.34
88.56
94.12
92.54
72.54
86.46
SpbereFaee2
85.40
91.31
94,80
93.32
76.89
89.91
Table 6: Results on UJB-B We cite the results from the original papers for [2, 46, 47], For the re-implemented methods, we the hyperparameters that lead to the hest results on the validation set. Results are in % and higher values are hetter.
8

---

Published as a conference paper at ICLR 2022
We make several observations based the evaluation results of JB-B (Table 6). First, SphereFace2 produces significant improvements over other state-of-the-art methods, especially low FARs and FPIRs. Specifically, SphereFace2 outper- forms CosFace by 5.37% at FAR=le-5 and 2.70% at FAR=1e-4 in 1:1 verifica- tion, 3.09% FPIR=1e-2 and 3.02% at FPIR=1e-1 in I:N identification. These significant performance gains suggest that the pair-wise learning paradigm is very useful in improving the robustness of face recog- nition system. Similar observations can also found in the results of IJB-C (Table 7) and the ROC curves results (Fig. 9). Second, the performance is getting saturated for verification rate at FAR=1c-3, Compared to other methods, SphereFace2 can still improve the results by 0.67% - 1.02% on IJB-B and 0.43% 1.04% IJB-C. Third, the rank-1 identification performance of SphereFace2 is slightly better than CosFace, ArcFace, Circle Loss, and comparable to SphereFace. Overall, Spherelacce, performs significantly better than current best-performing methods these challenging datasets.
1:1 Veri.
TAR
FAR
LN Iden.
TPIR
@
FPIR
Methed
1o-5
lo-4
lc-3
rank-1
1c-2
10-1
VGGFace2 (SENct) [2]
84.0
91.0
91.2
74.6
84.2
MN-ve [47]
86.2
92.7
Comparator Nets [46]
88.5
94.7
SphereFace
86.07
91.96
66
94.83
83.15
89.45
CosFace
85.13
90.98
95.47
94.25
80.89
88.17
ArcFace
86.12
91.60
95.72
94.71
82.21
88,94
Circle Loss
84.05
90.83
95.44
93.64
80.07
87.61
SphereFaac22
89.04
93.25
96.03
94.70
85,65
91.08
Table 7: Resalts on UB-C. The testing instances of UBB-C are twice as many % and higher better.
MegaFace. We further evaluate SphereFace2 on the MegaFace dataset. This challenging test- ing benchmark to evaluate the performance of face recognition methods at the million scale of distrac- tors. It contains a gallery set with more than I million images from 690K different identities, and
Iden. (10% Distractors)
Veri. (FPR=1c-6)
lahel refined
X
/
SphereFace
71.53
86.49
85.02
88.48
CosFace
71.65
86.21
85.45
88.
ArcFace
73.65
87
87.77
89.88
Circle Loss
71.32
85.94
84.34
87.96
SphereFuce2
74.38
89.84
89.19
91.94
probe set with 3,530 images from 530 identities. MegaFace provides two testing protocols for iden- tification and verification. We evaluate on both and report the results in Table 8. The gains are consistent with the IJB datasets. Under the same training setup, SphereFace2 outperforms current state-of state-of-the-ar methods by a large margin.
Table
Results MogaFace. Because of mislabeled samples
Megulaace, we present the results before and after label relinement.
3.3 NOISY LABEL LEARNING
Since the pair-wise labels used in SphereFace2 provide weaker supervi- sion than the class-wise labels, per- form experiments to evaluate the ro- bustness of SphereFace2 in label-noisy training. We randomly alter 20%, 40%, 60% and 80% of the labels for each class. The four noisy datasets are used to train CosFace, ArcFace, and SphereFace2 separately. We evaluate the trained models on the combined validation set. From Fig 10 (left), SphereFace2 shows stronger robustness noisy labels than CosPace and ArcFace, as its performance degrades significantly more slowly as the ratio of noisy labels increases from 0 to 0.8.
Clili
Pale Puikii (778)
Poskin (aa
Figure 9: The ROC curves of Spherelacce2 and other start of methods on IJB.B (left) UBBC (right)
3.4 MODEL PARALLELIZATION
1200
We follow [4] to parallelize the loss computations for CosFace, ArcFace, and SphereFace2. These methods are trained with 1 million identities. Fig. 10 (right) shows how the number of pro- cessed images per second changes with different numbers of GPUs. Note that we do not include the feature extraction here. CosFace and ArcFace have negli- gible difference on running time. When a single GPU is used (i.e., no model parallelization), CosFace and ArcFace are slightly faster than SphereFace2. As the number of GPUs increases, the acceleration of SphereFace2 is more significant, due to less communication cost over GPUs. The near linear acceleration for SphereFace2 is owing to its proxy-based pair-wise formulation.
Sphenfucc
0.5
Ratovo Nalg
Naaher Citts
Figure 10: Lell: evaluations on the robusiness to noisy labels. Right: evaluations on the multi-GPU model parallelization.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
AW acknowledges support from a Turing AT Fellowship under grant EP/VO25379/1, The Alan Turing Institute, and the Leverhulme Trust via CFI. RS is partially supported by the Defence Science and Technology Agency (DSTA), Singapore under contract number A025959, and this paper does not reflect the position policy of DSTA and no official endorsement should be inferred.
REFERENCES
[1] Xiang An, Xuhan 7hu, Yang Xiao, Lan Wu, Ming Zhang, Yuan Gao, Bin Qin, Dehing Zhang, and Ying Fu. Partial fe: Training 10 million identities on a single machine. arXiv preprint arXiv:2010.0522. 2020. 2, 3
[2] Qiong Cao, Li Shen, Weidi Xie, Omkar Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pages 67-74, IEEE, 2018. 6, 7, 8, 9, 13
[3] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In CVPR, 2005. 2,
[4] Jiankang Deng, Jia Guo, Niannan Xue, Stefanos Zafeiriou, Areface: Additive angular margin loss for deep face recognition. In CVPR, 2019. 1, 2, 3, 4, 5, 8, 9, 14, 15
[5] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. AISTATS, 2011. 1011.4 4
[6] Chuan Guo, Geoff Pleiss, Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017.
[7] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In CVPR, 2006. 3
Chunrui Han, Shiguang Shan, Mcina Kan, Shuzhe Wu, and Xilin Chen. Face recognition with contrastive convolution. ECCV, 2018.
[9] Gary B Huang. Manu Ramesh, Tamara Berg, and Erik Learned-Mlllr. Labeled faces in the wild: A database for studying face recognition unconstrained environments. Technical report, Technical Report, 2007. 5, 13
[10] Yuge Huang. Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li. Jilin Li. Feiyue Huang. Curricularface: adaptive curriculum learning loss for deep face recognition. In CVPR, 2020. 2, 8, 15
[11] Kemelmaacher-Shiizermaa Steven M. Seitz, Daniel Miller, and Evan Brossard. The megaface bench- mark: million faces for recognition at scale. In CVPR, 2016. 13
[12] Rongmei Lin, Weiyang Liu, Zhen Liu, Chen Feng, Zhiding Yu, James Rehg. Xiong, and Le Song. Regularizing neural networks via minimizing hyperspherical energy. In CVPR, 2020. 4
[13] Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Dai, and Song. Learning towards minimum hyperspherical energy. In NeurlPS, 2018.
[14] Weiyang Liu, Rongmei Lin, Zhen Liu, James M Rehg, Liam Paull, Xiong, Song, and Adrian Weller. Orthogonal over parameterizee training. In CVPR, 2021.
[15] Weiyang Liu, Rongmei Lin, Zhen Liu, Xiong, Bernhard Schölkopf, and Adrian Weller. Learning with hyperspherical uniformity. In AISTATS, 2021. 4
[16] Weiyang Liu, Yandong Wen, Bhiksha Raj, Rita Singh, and Adrian Weller. Sphereface revived: Unifying hyperspherical face recognition. TPAMI, 2022. 1, 8, 14, 15
[17] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In CVPR, 2017. 2, 3, 7, 8, 13, 14, 15
[18] Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. ICML, 2016. 1
[19] Weiyang Liu, Yan-Ming Zhang, Xingguo Li, Zhiding Yu, Bo Dai, Tuo Zhao, and Le Song. Deep hyperspherical learning. In NIPS, 2017.
10

---

Published as a conference paper at ICLR 2022
[20] Yu Liu, Hongyang Li, and Xiaogang Wang. Learning deep features via congenerous cosine loss for person recognition. arXiv preprint vrXiv:/70.00688 068900 2017. 15
[21] Brianna Maze, Jocelyn Adams, James Duncan, Nathan Kalka, Tim Miller, Charles Otto, Anil Jain, W Tyler Niggel. Janet Anderson, Jordan Cheney, et al. larpa janus benchmark-c: Face dataset and protocol. In 2018 International Conference on Biometrics (ICB), pages 158-165. IEEE, 2018. 13
[22] Daniel Miller, Brossard, S Seitz, and recognition at scale. arXiv preprint: 150r:it:::5000222 1505.02 2015. 13
Megaface: A million faces for
[23] Zuheng Ming, Joseph Chazalon, Muhammad Muzzamil Luqman, Muriel Visani, and bean-Chrittophe Burie. Simple triplct loss based on intra/inter-class metric learning for face verification. In ICCVW, 2017.
[24] Stylianos Moschoglou. Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In Proceedings of the IEFE Conference on Computer Vision and Pottern Recognition Workshops, pages 51-59, 2017. 5, 13
[25] Hyun Oh Song, Yu Xiang. Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. CVPR, 2016.
[26] Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In BMVC, 2015. 2
[27] Rajecv Ranjan, Carlos D Castillo, and Rama Chellappa. L2-constrained softmax loss for discriminative face verification. arXiv preprint 1eXXiv1703.0900 2017.
[28] Ryan Rifkin and Aldebaro Klautau. In defense of one-vs-all classification. JMLR, 2004. 2
[29] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, 2015. 1,
[30] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 1857-1865, 2016.
[31] Yi Sun, Yuheng Chen, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation by joint identification veriiicatin........ In NIPS, 2014. 2,
[32] Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deep Icarning face representation from predicting 10,000 classes. In CVPR, 2014. 1.2,3
[33] Sun, Xiaogang Wang, and Xiaoou Tang. Deeply learned face representations are sparse, selective, and robust. In CVPR, 2015. 2
[34] Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In CVPR, 2020. 2, 3, 4, 8, 15
[35] Yaniv Taigman, Ming Yang, Mare' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In CVPR, 2014. 2,
[36] Evgeniya Ustinova and Victor Lempitsky. Learning deep embeddings with histogram loss. In NIPS, 2016.
[37] Feng Wang, Weiyang Liu, Haijun Liu, and Jian Cheng. Additive margin softmax for face verification. arXiv preprint arXiv:1801.05533 2018 1. 2, 4. 5, 8, 14
[38] Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: 1.2 hypersphere embedding for face verification. In ACM-MM, 2017. 2
[39] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingehao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In CVPR, 2018. 1, 3, 5, 7, 14, 15
[40] Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanging Lin. Deep metric learning with angular loss. In ICCV, 2017. 3
[41] Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In CVPR, 2019.
[42] Yandong Wen, Kaipeng Zhang, Zhifeng I.,, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In FCCV, 2016.
11

---

Published as a conference paper at ICLR 2022
[43] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A comprehensive study center loss for deep face recognition. IJCV, 2019.
[44] Cameron Whitelam, Emma Taborsky, Austin Blanton, Brianna Maze, Jocelyn Adams, Tim Miller, Nathan Kalka, Anil K Jain, James Duncan, Kristen Allen, al. larpa janus benchmark-b face dataset. In proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 90-98, 2017. 8, 13
[45] Chao-Yuan Wu, R Manmatha, Alexander Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. In ICCV, 2017. 3
[46] Weidi Xie, Li Shen, and Andrew Zisserman. Comparator networks. Proceedings of the European Conference on Computer Vision (ECCV), pages 782-797, 2018. 8, 9
[47] Weidi Xie and Andrew Zisserman. Multicolumn networks for face recognition. arXiv preprint MMXiv:187.00912. 2018. 8,
[48] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multi-task cascaded convolutional networks. arXiv 160.00n::604008877. 2016. 7
[49] Tianyue Zheng and Weihong Deng. Cross-pose Ifw: database for studying cross-pose face recognition in unconstrained environments, Beijing University of Posts and Telecommunicaations, Tech. Rep, 5, 2018. 5,
13
[50] Tianyue Zheng, Weihong Deng, and Jiani Hu. Cross-age Ifw: A database for studying cross-age face recognition in unconstrained environments. arXiv preprint arXiv:1708.088 2017. 5, 13
[51] Yutong Zheng, Dipan K Pal, and Marios Savvides. Ring loss: Convex feature normalization for face recognition. In CVPR, 2018. 2
[52] Yaoyao Zhong and Weihong Deng, Adversarial learning with margin- based triplet embedding regulariza- tion. In ICCV, 2019.
12

---

Published as a conference paper at ICLR 2022
Appendix
A STATISTICS FOR THE USED DATASETS
on
split
VGGFace2 [3]
8.6K
3.1M
train
LFWW [9]
5,749
13,233
val
Age-DB30 [24]
568
16,488
val
CA-LFW [49]
5,749
11,652
val
CP-LPW
5,749
12,174
val
[44]
1,845
test
IIB-C [21]
3,531
148.8K
test
MegaFace (pro.) [11]
530
3,530
test
MegaFace (dis.) [22]
690K
test
Table 9: Statistics for the used datasets.
B STATISTICS OF IJB TEST PROTOCOLS
IJB [44]
IJB [[1]]
# of templates
12,115
23,124
A of genuine matches
10,270
19,557
1:1 Verification
of imposter matches
15M
* of probes
10,270
19,593
Identification
# of galleries
1,875
3,531
The evaluation statistics
C MORE HYPERPARAMETER EXPERIMENTS
We additionally provide the results under different hyperparameters. We follow the same settings in the main paper by training a 20-layer CNN [17] on VGGFace2. First, we vary the hyperparameter from 20 to 50 and the results in Table 11 show that both r 30 and 40 work reasonably well. Second, we vary the hyperparameter m from 0.2 to 0.5 and evaluate how the size of margin affects the performance. The results in Table 11 show that m 0.3 generally achieves the best performance. Last, we evaluate how the hyperparameter t in similarity adjustment may affect the performance. Table 11 shows that similarity adjustment generally helpful for performance (i.e., 2,3, 3,3,4, 4,5 works better than / 1) and = 3 achieves the best combined performance.
LFW
Ag:DB-30
CA-LEW
CP-LFW
Combined
0.7
0.4
99.56
92.97
92.85
90.97
93.96
0.7
0.4
99.62
93.22
93.35
91.02
94.28
0.4
93.92
93.73
93.42
93.53
0.7
0.2
92.45
93.40
93.93
0.7
0.3
93.47
93.83
94.52
93.68
93.47
92.97
93.38
0.7
0.4
86.03
87.48
86.21
0.7
0.4
99.06
86.46
88.88
89.57
92.36
93.55
92.63
93.30
93.73
93.68
93.47
0.7
93.58
93.48
90.98
94.17
93.45
Table 11: Ablations parameter
Results % and higher values better.
13

---

Published as a conference paper at ICLR 2022
D SIMILARITY ADJUSTMENT FOR OTHER METHODS
To empirically show the comparison between SphereFace2 and other methods with SA, we present the experiments on UJBB and UJC datasets. As shown in Table 12, similarity adjustment works well for SphereFace2, and applying it to multi-class classification losses is not as useful as in SphereFace2.
Methods
I:1 Veri. TAR @ FAR
IN Iden. TPIR @ FPIR
Methods
1:1 Veri.
TAR
FAR
I-N Iden. TPIR @ FPIR
on LBB
le-5
le-4
le.3
rank-1
le-2
le-1
on UBC
le-5
1e-4
1e-3
rank-1
le-2
le-1
CosFace w/o SA
79.35 88.05
93.71
92.54
72.20
86.40
CosFace w/o SA
84,20
90.53
95.12
93.73
80.04
87.46
CosFace SA
75.72
86.57
93.07
92.59
68.48
84.38
CodFace w/ SA
82.92
89.83
94.54
93.77
78.46
86.26
AreFace w/o SA
78.12
87.11
93.29
92.18
69.86
84.88
AreFace w/o SA
82.54
89.53
94.79
93.20
78.35
86.09
ArcFace w/ SA
78.88
88.06
93.71
92.65
71.62
86.08
ArcFace w/ SA
84.62
90.53
95.12
93.78
80.52
87.55
SphereFace2
81.11 89.05
94.11
92.66
73.54
87.32
SphereFaos2
85.38
91.32
95.24
93.74
81.98
88.54
Table 12: Comparison of SphereFacc22, CosFace and ArePace (with or without similarity adjustment).
E GRADIENT COMPARISON
We compare the gradient between the multi-class softmax-based loss and SphereFace2 in Table 13. We observe that the gradient updates SphereFace2 can be performed with only the corresponding classifier weights (i.e., no summation over classifier weights of different classes). Therefore, the classifier layer in the SphereFace2 framework can be back-propagated with the local GPU and involve no communication overhead.
Multi-class Softmax based Loss
SphereFace2
log exp(W, a) Σ, exp[W, Tx)
log (1 exp( w)) + Eisy log (1 (xpp(W"xx))
ow, a y)
(W m)
exp: W. =)
Σg expp (xp(W)
cssp (W,T w)
21 o
E
(W Σ, (W TT)
1)
y
14 W.
Table 13: Gradient comparisons between the multi-class softmax-besed loss and SphercFace2. Here we omit the constant terms, bias. margin, etc., since they do not affect the conclusion.
F DIFFERENT FORMS OF ANGULAR MARGIN IN SPHEREFACE2
Although the additive margin example in the main paper, it is natural to consider the other forms of angular margin in SphereFace2. Specifically, we first revisit the final loss function that uses particular type of additive margin [37, 39] (also used as the example in the main paper):
> 2.og [+exxp (-r-(g(cos(O.)) m)-b) Lsnc
1-1 Σ log (1+exp (r- (g(cc((())) mm) +m)
For another type of additive margin [4] and the multiplicative margin [16, 17], implement them using the Characteristic Gradient Detachment (CGD) trick [16] to enable stable training. Therefore, use the following loss function implement the ArcFace-type additive margin in SphereFace2:
LSFz- = log (1 1++ex exp ( g(cos(((), Detach( (g (cos (min (z, 0, + m))) g(cos((0,))
A
Σ log 1 [++exp exp (r g(cos(0:)) + +b)
where Detach(-) is a gradient detachment operator that stops the back-propagated gradients. For details of how CGD works, refer to [16]. For the multiplicative margin, we adopt an improved version from SphereFace-R [16] (i.e., SphereFace-R vl), which yields
LSFAM
log
+ exp - g(cos(B,)) - Detach( g (cos(min(m, ,,) (-0))) - g(coos(,))) -b)
1 x
log 1 + exp (r (0:)) + b)
14

---

Published as a conference paper at ICLR 2022
For both ArcFace-type and multiplicative margin, we do not inject the angular margin to the negative samples. Considering the cases with or without angular margin for the negative samples, note that a learnable bias makes both cases equivalent. The only difference is that the optimal choice for the margin parameter m may vary for the two cases.
Then we conduct experiments to empirically compare them. We adopt the same training settings as Ta- ble 5 (i.e., SFNet-20 [16, 17] without batch normalization). The results are given in Table 14, Table 15, and Table 16. Here we term SphereFace2 with CosFace-type additive margin as SphereFace22 SphereFace2 with ArcFace-type additive margin as SphereFace2-A and SphereFace2 with multiplica- tive additive margin as SphereFace2-M. The margins for SphereFace2-A and SphereFace2-M are 0.5 and 1.7, respectively. We observe that different types of angular margin perform similarly in general. Note that we did not carefully tune the hyperparameters for SphereFace2-A and Sphereface2-N and the performance already very competitive. We believe that the performance can be further improved by more systematic hyperparameter search.
Loss Function
LFW
AgeDB-30
CA-LFW
CP-LFW
Combined
Softmax Loss
98.20
87.23
88.17
84.85
89.05
[20]
92.4
SpherePace [16, 17]
92:55
90.90
93.75
CosFace [39]
99.51
92.98
92.83
91.03
93.89
ArcFace [4]
99.47
91.97
92.47
90.85
93.97
Circle Loss [34]
99.48
92.23
92.90
91.17
93.78
CurriculaFaace [10]
99.53
92.47
92.90
90.65
93.70
SphereFace2.C
99.50
93.68
93.47
91.07
94.28
SphereFace2-A
99.51
93.53
93.75
91.01
94.19
SphereFace22 M
99.58
93.63
93.66
90.95
94.19
Table 14: Comparison of different loss functions, Results are in % and higher values are hetler.
Methods
1:1 Veri, TAR @ FAR
1:N Iden.
TPIR
FPIR le:2
on JJBB
le.5
le.4
1e-3
rank-1
le
CosFaace
79.35
88.05
93.71
92.54
72.20
86.40
AndFaaee
78.12
87.11
93.29
92.18
69.86
84.88
SphereFace2-C
82.36
89.54
92.61
88.22
SphereFace2-A
92.68
SphereFace2-M
92.53
Table Comparison CosFace, AreFace, and SphereFace2 different margin types BJB-B
Methods
1:1 Veri. TAR @ FAR
I.N Iden.
TPIR
FPIR
JBC
1c-5
1c-4
10-3
rank-1
1e-2
16-1
CosFacc
84,20
90.53
95.12
93.73
80.04
87.46
ArcFace
82.54
89.53
94.79
93.20
78.35
86.09
SphereFace2-CC
86.78
91.78
95.26
93.77
83,20
89.36
SphereFaec2-A
86.30
91.68
95.33
93.72
82.82
89.24
SphereFace2
Table Comparison CosFFace
and SpherelFace2 differenn margin types
15


Reviews:
[{"review_id": "u_kDknWCbo1", "reviewer": "Reviewer_iYQo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "** Strengths\n- The discussion on the advantages of the multi-binary classification strategy is interesting and persuasive. I like the comparison between triplet-based learning and pair-based learning in Figure 2.\n- Since the naive solution does not work, the authors proposed some principles in designing an effective loss function. These principles are effective in designing a good loss function, as shown in Table 2.\n- The authors did a good job on explaining the proposed components via geometric interpretation.\n- SphereFace2 outperforms the state-of-the-art face recognition models on 1:1 verification and 1:N open-set evaluation benchmarks. \n- It also allows model parallelization that effectively speeds up the training process when multiple GPUs are available.\n\n** Weaknesses\n- The Positive/negative sample balance technique is a bit simple. In fact, the authors manually tuned the hyper-parameter \\lambda to get a desirable results while the simplest configuration does not work well (the first row in Table 2). There are existing techniques to handle data imbalance such as focal loss. It is better to have a comparison to these available options.\n- The authors did not provide the final values of the hyper-parameters (λ, r, m, and t)\n- SphereFace2 performs worse than SphereFace in 1-N rank-1 Indentification tests\n- In Section 3.2 and 3.3, the authors used CosFace as the baseline for comparison. ArcFace is a better option.", "summary_of_the_paper": "This paper proposes a novel face recognition model called SphereFace2. The key idea is to replace the softmax-based multi-class training with multiple binary classification training, which is more consistent with the testing phrase and generalized to both closed/open-set evaluations. To make this novel strategy works, the authors defined some principles in designing an effective loss function, including (1) Positive/negative sample balance, (2) Easy/hard sample mining, (3) Angular margin, and (4) Similarity adjustment. SphereFace2 outperforms the state-of-the-art face recognition models on 1:1 verification and 1:N open-set evaluation benchmarks. It also allows model parallelization that effectively speeds up the training process when multiple GPUs are available.", "summary_of_the_review": "Overall, this paper presents a novel face recognition model, called SphereFace2, with interesting insights and persuasive explanations. The proposed model outperforms the baselines on popular 1-1 verification and 1-N open-set evaluation. Despite few minor issues, I lean towards accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fBfVQIkAhCD", "reviewer": "Reviewer_6DJe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- Different from the commonly used multi-class classification, the proposed method uses binary classification to reduce the gap between training and open-set testing, which addresses the face recognition problem from a new perspective.\n- The principles for the designed loss function are well-motivated and clearly explained. The geometric interpretations also help to better understand.\n- The experiments are comprehensive, and show the state-of-the-art performance of the proposed method.\n- The paper is well-written and easy to follow.\n\nCons:\n- In Sec 3.1, the hyperparameters are analyzed with different values. In Sec 3.2, since the backbone network has changed, which hyperparameter setting is adopted? It would be better to clarify this.\n- Some results in the text seem to be inconsistent with those in the tables. For examples, in p .8 l. 3, 93.73% (93.87% in Table 2); in p. 8 l. 7-9, 5.37%, 2.74%, 3.50%, and 3.52% (5.78%, 2.70%, 3.09%, and 3.02% in Table 6).\n- A few typos, e.g., p. 9 l. 13, Table 6 -> Table 7.\n", "summary_of_the_paper": "This paper presents a binary classification training framework, SphereFace2, for face recognition. Instead of multi-class classification, the proposed method adopts pair-wise comparisons learned through a well-designed loss function, which is more suitable for open-set evaluation. Comprehensive experiments demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "The idea of using binary classification learning to cope with face recognition is well-motivated and reasonable. The framework and loss function design are also clearly explained and further demonstrated by experiments on benchmark datasets. Although there may be some minor errors, I think this is a good paper, and would vote for accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8vO6sAHvhYd", "reviewer": "Reviewer_4dBH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n1. The paper is well written with nice presentation and clear description.\n2. The proposed method has sound motivation, i.e., bridging the gap between the training process and the actual test protocol when deployed for open-set applications. The effectiveness of the key components are well illustrated from both intuitive and empirical perspectives with figures and statistics. \n3. The new loss formulation also benefits parallel training setup which are widely adopted nowadays.\n4. Extensive experiments have been conducted to ascertain the final outstanding performance of the proposed method.\n\n### Weaknesses:\n1. The beginning of this paper may be misleading. The problem of multi-class softmax-based training when switched to the open-set deployment has been noticed in many early works, such as [25] [27], where contrastive loss and triplet loss were used to address this problem. In this paper, however, the Abstract, Paragraph 1 and the first sentence of Paragraph 2 did not mention this fact and therefore may mislead readers to believe that this work is the first to address this issue.\n2. Some statements are vague or unclear, such as:\n    * The authors claim that softmax loss introduces competition among different classes. What is the definition of \"competition\"? Even using the proposed loss function, the feature embedding still receives gradients from all the classes (1 positive and N-1 negative classes) in one training iteration, could that also be a kink of inter-class competition?\n    * The categorization of proxy-free and proxy based is unclear. Why is using softmax loss and its variants considered proxy-based (like [28]) but using combination of softmax and contrastive loss is considered proxy-free (like [27])?\n    * I didn't understand the example given for similarity adjustment, where the text says \"if g(cos(theta)) is ... range of [-0.2, 1], we can end up with cos(theta) ... becoms [-0.9, 1].\" But looking at the formulation and Figure 6, it seems that when t > 1, cos(theta) is within [-0.2, 1] while g(cos(theta)) is within [-0.9, 1]. I am not sure if I misunderstood anything and expect more details from the authors feedback.\n    * In the end, how is the verification threshold is determined? Based on the validation set or is there a theoretical value? If the threshold is determined from the results on the validation set, how to understand the claim that the proposed method eases the difficulty of choosing a verification threshold compared to other methods?\n3. In addition to new formulation of the binary classification training protocol, there are multiple components contributing to the final performance, including sample balancing, easy/hard sample mining, angular margin and similarity adjustment. It is hard to judge which component is play a key role to the results. Looking at Table 2, without SA, the performance (93.87%) is on par with other state-of-the-art works such as ArcFace (93.97%). However, SA can also be applied to other loss functions, then what is the actual benefit by using the binary classification loss formulation?", "summary_of_the_paper": "This paper has proposed SphereFace2, a new face recognition training scheme that applies a set of binary classification losses to each identities in the dataset, to learn better face embeddings. The proposed method has addressed some limitations of existing training schemes, such as discrepancy between the softmax-based loss and the actual open-set face recognition scenarios. The proposed binary classification protocol works together with a series of optimization strategies, such as easy/hard sample mining and similarity adjustment, to achieve superior performance over state-of-the-art methods as shown in the extensive experiments.", "summary_of_the_review": "This paper has proposed a relatively new idea on training face recognition models by using multiple binary classification losses. The proposed method has convincing motivation and the technical design is reasonable. Experimental results ascertain the superiority of the proposed method as a unified solution. However, some statements of the paper are misleading or vague. Moreover, as the proposed method comprises multiple effective components, it is also unclear that whether the performance improvements come from the new formulation or from one or two incremental components.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the proposed SphereFace2 approach - training face recognition models by using multiple binary classification losses - is interesting and innovative. The reviewers agree that the paper is well written and are satisfied with the presented experimental study. The rebuttal addressed all additionally raised questions. I believe that the paper will be of interest to the audience attending ICLR and would recommend a presentation of the work as a spotlight.
Decision: Accept (Spotlight)