Year: 2021

Abstract:
State-of-the-art deep face recognition methods are mostly trained with a softmaxbased multi-class classification framework. Despite being popular and effective, these methods still have a few shortcomings that limit empirical performance. In this paper, we start by identifying the discrepancy between training and evaluation in the existing multi-class classification framework and then discuss the potential limitations caused by the “competitive” nature of softmax normalization. Motivated by these limitations, we propose a novel binary classification training framework, termed SphereFace2. In contrast to existing methods, SphereFace2 circumvents the softmax normalization, as well as the corresponding closed-set assumption. This effectively bridges the gap between training and evaluation, enabling the representations to be improved individually by each binary classification task. Besides designing a specific well-performing loss function, we summarize a few general principles for this “one-vs-all” binary classification framework so that it can outperform current competitive methods. We conduct comprehensive experiments on popular benchmarks to demonstrate that SphereFace2 can consistently outperform current state-of-the-art deep face recognition methods.

heading: 1 INTRODUCTION
text: Recent years have witnessed the tremendous success of deep face recognition (FR), largely owing to rapid development in training objectives [4, 13, 14, 27, 28, 33–35, 38]. Current deep FR methods are typically based on a multi-class learning objective, e.g., softmax cross-entropy loss [4, 13, 14, 23, 33– 35]. Despite its empirical effectiveness, there is an obvious discrepancy between such a multi-class classification training and open-set pair-wise testing, as shown in Fig. 1. In contrast to multi-class classification, pair-wise verification is a binary problem where we need to determine whether a pair of face images belongs to the same person or not. This significant discrepancy may cause the training target to deviate from the underlying FR task and therefore limit the performance. This problem has also been noticed in [25, 27], but they still try to address it under the multi-class (or triplet) learning framework which still fundamentally differs from pair-wise verification in testing. On the other hand, multi-class classification training assumes a closed-set environment where all the training data must belong to one of the known identities, which is fundamentally different from open-set testing.
In order to address these limitations, we propose a novel deep face recognition framework completely based on binary classification. SphereFace [13] is one of the earliest works that explicitly performs multi-class classification on the hypersphere (i.e., angular space) in deep FR. In light of this, we name our framework SphereFace2 because it exclusively performs binary classi-
fications (hence the “2”) on the hypersphere. Unlike multi-class classification training, SphereFace2 effectively bridges the gap between training and testing, because both training and testing perform pair-wise comparisons. Moreover, SphereFace2 alleviates the closed-set assumption and views the training as an open-set learning problem. For example, training samples that do not belong to any class are still useful and can serve as negative samples in SphereFace2, while they cannot be used at all in the multi-class classification framework. Specifically, given K classes in the training set, SphereFace2 constructs K binary classification objectives where data from the target class are viewed
as positive samples and data from the remaining classes are negative samples. The spirit is similar to “one-vs-all” classification [24]. An intuitive illustration comparing typical multi-class training, our proposed SphereFace2, and open-set evaluation is given in Fig. 1.
In multi-class classification training, the softmax (cross-entropy) loss introduces competition among different classes, because the softmax normalization requires all the class logits are summed to one. Therefore, increasing the confidence of one class will necessarily decrease the confidence of some other classes, which can easily lead to over-confident predictions [6]. In contrast to the competitive nature in all the softmax losses, our SphereFace2 encourages each binary classification task to individually improve the discriminativeness of the representation without any competition.
The significance of our method can also be understood from a different perspective. Depending on how samples interact with each other in training, deep FR methods can be categorized into triplet-based learning and pair-based learning. Triplet-based learning simultaneously utilizes an anchor, positive samples and negative samples in its objective, while pair-based learning uses either positive pairs or negative pairs in one shot. Based on whether a proxy is used to represent a set of samples, both triplet-based and pair-based methods can learn with or without proxies. A categorization of deep FR methods is given in Table 1. Proxy-free methods usually require expansive pair/triplet mining and most of them [8, 19, 22, 27, 29, 47] use a hybrid loss that includes the standard softmax loss. Typical examples of triplet-based learning with proxy include the softmax loss and its popular variants [4, 13, 33–35] where a classifier is a class proxy and the learning involves an anchor (i.e., the deep feature
x), the positive proxy (i.e., the target classifier Wy) and the negative proxies (i.e., the other classifiers Wj , j 6= y). Most popular deep FR methods use proxies by default, since they greatly speed up training and improve data-efficiency (especially on large-scale datasets). Distinct from previous deep FR, we believe SphereFace2 is the first work to adopt a pair-wise learning paradigm with proxies.
An outstanding difference between triplet-based and pairbased learning is the usage of a universal threshold. In Fig. 2, we show that tripletbased learning compares the similarity scores between different pairs, while pair-based learning compares a similarity score and a universal threshold. As a pair-based method, SphereFace2 too optimizes the difference between similarity
scores and a universal threshold. By learning the threshold to distinguish positive and negative pairs during training, it naturally also generalizes to open-set FR.
In our framework, we cast a principled view on designing loss functions by systematically discussing the importance of positive/negative sample balance, easy/hard sample mining and angular margin, and combine these ingredients to propose an effective loss function for SphereFace2. In fact, most popular variants of the softmax loss [4, 33–35] also perform easy/hard sample mining and incorporate angular margin. Moreover, we observe that the distribution of similarity scores from positive pairs is quite different from that of negative pairs, in that it exhibits larger variance, which makes the threshold hard to determine. To address this, we propose a novel similarity adjustment method that can adjust the distribution of pair similarity and effectively improve the generalizability.
The flexibility of binary classifications in SphereFace2 brings a few additional advantages. First, in contrast to the classifier layer with the softmax loss which is highly non-trivial to parallelize [1], the classifier layer in SphereFace2 can be naturally parallelized across different GPUs. While the gradient update in the softmax loss needs to involve all the classifiers, the gradient update for classifiers in
SphereFace2 is independent and fully decoupled from each other. The gradient with respect to each classifier only depends on the feature x and the binary classifier itself. Therefore, SphereFace2 naturally addresses the problem of the softmax loss in training on a large number of face identities [1]. Second, the pair-wise labels used in SphereFace2 are in fact a weaker form of supervision than the class-wise labels used in all the softmax-based losses. This property is closely connected to the open-set assumption and also makes SphereFace2 less sensitive to class-wise label noise. For example, SphereFace2 does not require the class-wise labels of the negative samples in each binary classification. Instead, as long as we know that the identities of negative samples are different from the positive samples, SphereFace2 will still work well. Our experiments also empirically verify the robustness of SphereFace2 against class-wise label noise. Our major contributions are:
• SphereFace2 constructs a novel binary classification framework for deep FR. To our knowledge, SphereFace2 is the first work in deep FR to adopt a pair-wise learning paradigm with proxies. We further summarize a series of general principles for designing a good loss function for SphereFace2.
• The decoupling of the binary classifications leads to a natural parallelization for the classifier layer, making SphereFace2 more scalable than softmax-based losses.
• The pair-wise labels used in SphereFace2 are a weaker supervision than the class-wise labels used in the softmax loss, yielding better robustness to class-wise label noise.
Related Work. Deep FR methods are either proxy-based or proxy-free, as shown in Table 1. Proxyfree learning includes contrastive loss [3, 7, 27] and triplet loss [25]. Both losses directly optimize the similarity between samples, so they are highly unstable and data-inefficient. Proxy-free learning is more widely used in deep metric learning [21, 26, 32, 36, 37, 40]. Proxy-based learning uses a set of proxies to represent different groups of samples and usually works better for large-scale datasets. Typical examples include the softmax loss and its variants [4, 13, 28, 30, 31, 33–35] where each proxy is used to represent a class. SphereFace [13] suggests that large-margin classification is more aligned with open-set FR, and proposes to learn features with large angular margin. CosFace [33, 35] and ArcFace [4] introduce alternative ways to learn large-margin features with improved stability. Although large-margin classification brings the training target closer to the task of open-set FR, the discrepancy still exists and it is unclear how much the margin should be to close the gap. Moreover, a large margin inevitably leads to training instability. In contrast, SphereFace2 naturally avoids these problems and aligns the training target with open-set FR by adopting pair-wise learning with proxies.

heading: 2 THE SPHEREFACE2 FRAMEWORK

heading: 2.1 OVERVIEW AND PRELIMINARIES
text: The goal of SphereFace2 is to align the training target with open-set verification so that our training is more effective in improving open-set generalization in deep FR. To this end, SphereFace2 explicitly incorporates pair-wise comparison into training by constructing K binary classification tasks (K is the number of identities in the training set). The core of SphereFace2 is the binary classification reformulation of the training target. In the i-th binary classification, we construct the positive samples with the face images from the i-th class and the negative samples with face images from other classes. Specifically, we denote the weights of the i-th binary classifier by Wi, the deep feature by x and its corresponding label by y. A naive loss formulation is
Lf =log ( 1 + exp(−W>y x− by) ) + ∑ i 6=y log ( 1 + exp(W>i x+ bi) ) which is a combination of K standard binary logistic regression losses. Instead, we perform the binary classification on the unit hypersphere by normalizing both classifiers Wi,∀i and feature x. The loss function now becomes
Ls = log ( 1 + exp(− cos(θy)) ) + ∑ i6=y log ( 1 + exp(cos(θi)) ) (1)
where θi is the angle between the i-th binary classifier Wi and the sample x. The biases bi,∀i are usually removed in common practice [4, 13, 15, 33, 35], since they are learned for a closed set and cannot generalize to unknown classes. However, we actually find them very useful in SphereFace2, as will be discussed later. For now, we temporarily remove them for notational convenience. One of the unique advantages of such a parameterization for binary classifiers is that it constructs the i-th class positive proxy with Wi and the negative proxy with −Wi. Depending on the label, the training will minimize the angle between x and Wi or between x and −Wi in order to minimize the loss. This parameterization of positive and negative proxies immediately guarantees minimum hyperspherical
energy [12] that has been shown to benefit generalization. Moreover, our parameterization in SphereFace2 has the same number of parameters for the classifier layer as the previous multi-class training, and does not introduce extra overhead. However, naively minimizing the loss in Eq. (1) will not give satisfactory results. Therefore, we explore in the next subsection how to find a desirable loss function that works well in the SphereFace2 framework.

heading: 2.2 A PRINCIPLED VIEW ON LOSS FUNCTION
text: We emphasize that the exact form of our loss function is in fact not crucial, and the core of SphereFace2 is the spirit of binary classification (i.e., pair-wise learning with proxies). Following such a spirit, there are likely many alternative losses that work as well as ours. Besides proposing a specific loss function, we summarize our reasoning for designing a good loss function via a few general principles.
Positive/negative sample balance. The very first problem we need to address for SphereFace2 is how to balance the positive and negative samples. In fact, balancing positive/negative samples has also been considered in triplet loss, contrastive loss and softmax-based losses. Contrastive loss achieves the positive/negative sample balance by selecting the pairs. Based on the unified view in [30], triplet-based methods including both triplet loss and softmax-based losses can naturally achieve positive/negative sample balance, since these losses require the presentation of a balanced number of both positive and negative samples.
From Eq. (1), the gradients from positive samples and negative samples are highly imbalanced because only one out of K terms computes the gradient for positive samples. A simple yet effective remedy is to introduce a weighting factor λ to balance gradients for positive and negative samples:
Lb=λ log ( 1+exp(− cos(θy)) ) + (1−λ) ∑ i6=y log ( 1+exp(cos(θi)) ) where λ∈ [0, 1] is a hyperparameter that determines the balance between positive and negative samples. One of the simplest ways to set λ is based on the fraction of the loss terms, i.e., λ = K−1K when there are K classes in total.
Easy/hard sample mining. Another crucial criterion for a good loss function is the strategy for mining easy/hard samples, since it is closely related to the convergence speed and quality. It is commonly perceived that softmax-based losses are free of easy/hard sample mining, unlike triplet and contrastive losses. However, this is inaccurate. We construct
a simple example to illustrate how the softmax-based loss mines easy/hard samples. We assume a set of cosine logits from four classes is [cos(θ1), cos(θ2), cos(θ3), cos(θ4)] and the target class is y=1. Then we also compute the s-normalized softmax loss Ln=− log( exp(s·cos(θy))∑
i exp(s·cos(θi)) ) by fixing
cos(θi)=0.2, i 6=y and varying cos(θy) from −1 to 1. From the results shown in Fig. 3(a), we can observe that as the scale s increases, the loss for hard samples will be higher and also more sensitive than the loss for easy samples. This makes the neural network focus on optimizing the angles of hard samples and therefore can improve convergence and generalization. The scaling strategy is widely used as a common practice in most softmax-based margin losses [4, 30, 33–35], playing an implicit role of easy/hard sample mining. For the standard softmax cross-entropy loss, easy/hard sample mining is dynamically realized by the norm of features and classifiers.
In our framework, we need a similar strategy to mine easy/hard samples. Inspired by the rescaled softplus function [5], we use an extra hyperparameter r to adjust the curvature of the loss function based on the objective Lb:
Le = λ
r · log
( 1 + exp(−r · cos(θy)) ) +
1− λ r · ∑ i 6=y log ( 1 + exp(r · cos(θi)) ) (2)
where larger r implies stronger focus on hard samples. We consider an example of one binary classification and Le becomes 1r · log ( 1+exp(−r ·cos(θy)) ) when λ=1. We then plot how the loss value changes by varying cos(θy) from −1 to 1 under different r in Fig. 3(b). We observe that when r becomes larger, the loss for easy samples gets closer to zero while the loss for hard samples remains large. Therefore, r can help to mine and reweight easy/hard samples during training.
Angular margin. Learning deep features with large angular margin is arguably one of the most effective criteria to achieve good generalizability on open-set face recognition. SphereFace [13] introduced angular margin to deep face recognition by considering a multiplicative margin. CosFace [33, 35] and ArcFace [4] further considered an additive angu-
lar margin which makes the loss function easier to train. In light of these works, we introduce a novel two-sided angular margin with two adjustable parameters to the SphereFace2 framework:
La = λ
r · log
( 1 + exp(−r · (cos(θy)−mp)) ) +
1− λ r · ∑ i 6=y log ( 1 + exp(r · (cos(θi) +mn)) ) (3)
where mp controls the size of the margin for positive samples and mn controls the size of the margin for negative samples. Largermp and mn lead to larger additive angular margin, sharing similar spirits to CosineFace [33, 35]. The other forms of angular margin are also applicable in our framework, but exhaustively trying them is simple repetitive work and also beyond the scope of this paper.
However, quite different from the angular margin in the softmax-based losses, the angular margin in Eq. 3 has a universal confidence threshold 0 (i.e., cos(θi)=0). Both angular margins mp and mn are introduced with respect to this decision boundary cos(θi)=0, see Fig. 4(b). This property has both advantages and disadvantages. One of the unique advantages is that our angular margin for each class is added based on a universally consistent confidence threshold and does not depend on the other classifiers, while the angular margin in softmax-based losses will be largely affected by the neighbor classifiers. However, it is extremely challenging to achieve the universal threshold 0, which results in training difficulty and instability. To improve training stability, the bias term that has long been forgotten in softmax-based losses comes to the rescue. We combine the biases back:
Lb = λ
r · log
( 1+exp(−r · (cos(θy)−mp)−by) ) +
1− λ r · ∑ i 6=y log ( 1+exp(r · (cos(θi)+mn)+bi) ) (4)
where bi denotes the bias term for the binary classifier of the i-th class. Since the class-specific bias is not useful in the open-set testing, we will simply use the same bias b for all the classes. The bias b now becomes the universal confidence threshold for all the binary classifications and the baseline decision boundary becomes r ·cos(θy)+b=0 instead of r ·cos(θy)=0, making the training more stable and flexible. The final decision boundary is r ·(cos(θy)−mp)+b=0 for the positive samples and r ·(cos(θy)+mn)+b=0 for the negative samples. More importantly, the threshold b is still universal and consistent for each class. An illustration of Eq. 4 is given in Fig. 4. Alternatively, we can interpret Eq. 4 as a learnable angular margin where the confidence threshold is still 0. Then we can view mp− br as the positive margin and mn+ b r as the negative margin. Because b is learnable, we only need to focus on mp+mn which yields only one effective hyperparameter. For convenience, we simply let mp=mn=m and only need to tune m in practice.
Similarity adjustment. In Fig. 5(a), we observe a large inconsistency between positive and negative pairs in the distribution of cosine similarity. The similarity distribution of negative pairs has smaller variance and is more concentrated, while the positive pairs exhibit much larger variation in similarity score. The distributional discrepancy between positive and negative pairs leads to a large overlap of
similarity scores between them, making it difficult to give a clear threshold to separate the positive and negative pairs. This is harmful to generalization. Fig. 5(a) also empirically shows the similarity scores mostly lie in the range of [−0.2, 1]. These observations motivate us to (1) reduce the overlap of the similarity scores between positive and negative pairs, and (2) increase the empirical dynamic range of the similarity score such that the pair similarity can be distributed in a larger space.
To this end, we propose a novel similarity adjustment method. Our basic idea is to construct a monotonic decreasing function g(z) where z∈ [−1, 1] and g(z)∈ [−1, 1] and then use g(cos(θ)) instead of the original cos(θ) to adjust the mapping from angle to similarity score during training. Considering that the originally learned cos(θ) mostly lies in the range of [−0.2, 1], we require g(z) to map [−0.2, 1] to a larger range (e.g., [−0.9, 1]), so that if cos(θ) is learned similarly as before and still gives the empirical dynamic range of [−0.2, 1], we can end up with g(cos(θ)) whose empirical dynamic range becomes [−0.9, 1]. Specifically, g(z) is parameterized as g(z)=2 ( z+1 2
)t−1
where we typically use z=cos(θ)∈ [−1, 1]. In practice, we simply replace the original cosine similarity with g(cos(θ)) during training. t controls the strength of similarity adjustment. When t=1, g(cos(θ)) reduces to the standard cosine similarity. In Fig. 5, we show that the similarity distribution can be modified by increasing the parameter t. As t increases, the overlap between the positive and negative pairs is reduced and their similarity distributions also become more separable. Moreover, the empirical dynamic range of the similarity score is also approximately increased from [−0.2, 1] to [−0.4, 1]. The empirical results validate the effectiveness of the proposed similarity adjustment.
Final loss function. After combining all the principles and simplifying hyperparameters, we have
L = λ
r ·log
( 1+exp ( −r·(g(cos(θy))−m)−b )) + 1− λ r · ∑ i 6=y log ( 1+exp ( r·(g(cos(θi))+m)+b )) (5)
where g(·) has a hyperparameter t. In total, there are four hyperparameters λ, r, m and t. Each has a unique geometric interpretation, making them easy to tune. Following our design principles, there could be many potential loss functions that share similar properties to ours. Our framework opens up new possibilities to advance deep face recognition.

heading: 2.3 GEOMETRIC INTERPRETATION
text: This subsection provides a comprehensive discussion and visualization to justify our designs and explain the geometric meaning of each hyperparameter. By design, r is the radius of the hypersphere where all the learned features live and is also the magnitude of the features. The bias b for the i-th class moves the baseline decision boundary along the direction of its classifier Wi. The parameter m controls the size of the induced angular margin. We set the output feature dimension as 2 and plot the 2D features trained by SphereFace2 with different margin m in Fig. 7. The visualization empirically verifies the following arguments.
The bias b moves the decision boundary. From Fig. 7, we can observe that the bias b can be effectively learned to move the decision boundary along the classifier direction and lead to a new universal confidence −b for all the classes. The bias b makes the training easier and more stable while still preserving the unique property that all classes share the same confidence for classification. Compared to other deep FR methods, the uni-
versal confidence in SphereFace2 can help to learn a consistent positive/negative pair separation and explicitly encourage a unique and consistent verification threshold during training.
m, r control the angular margin. Fig. 7 visualizes the baseline decision boundary (denoted as (2) in Fig. 7(b)) and the decision boundary for the positive/negative samples (denoted as (1)/(3) in Fig. 7(b)). The distance between the positive and negative decision boundary is 2rm, producing an effective angular margin. The results show that the empirical margin matches our expected size well.
Larger m leads to larger angular margin. From Fig. 7, we can empirically compare the deep features learned with m=0 and m=0.2 and confirm that larger m indeed incorporates larger angular margin between different classes.
We also visualize 3D deep features and the decision planes for one class with r=30 and m=0.2 in Fig. 8. We can observe that samples from each class are well separated with large angular margins, which is consistent with the 2D case. The empirical angular margin also perfectly matches the induced one (i.e., the distance between the positive and negative plane). The results further verify our empirical conclusions drawn from the 2D visualization.

heading: 2.4 EFFICIENT MODEL PARALLELIZATION ON GPUS
text: As it becomes increasingly important for deep FR methods to train on largescale datasets with million-level identities, a bottleneck is the storage of the
classifier layer since its space complexity grows linearly with the number of identities. A common solution is to distribute the storage of the classifiersWi,∀i and their gradient computations to multiple GPUs. Then each GPU only needs to compute the logits for a subset of the classes. However, the normalization in softmax-based losses inevitably requires some data communication overhead across different GPUs, resulting in less efficient parallelization. In contrast, the gradient computations in SphereFace2 are class-independent and can be performed locally within one GPU. Thus no communication cost is needed. The decoupling among different classifiers makes SphereFace2 suitable for multi-GPU model parallelization.
Specifically, the softmax normalization in the softmax loss involves the computation of all the classifiers, so computing the gradient w.r.t. any classifier will still require the weights of all the other classifiers, which introduces communication overhead across GPUs. In contrast, the loss for SphereFace2 (Eq. (5)) can be rewritten as L = ∑K i=1 fi(Wi,x) where fi(·, ·) is some differentiable function. To compute ∂L∂Wi , we only need to compute ∂fi(Wi,x) Wi which does not involve any other classifiers. Therefore, this gradient computation can be performed locally and does not require any communication overhead. Appendix E compares the gradient of SphereFace2 and the softmax loss.

heading: 3 EXPERIMENTS AND RESULTS
text: Preprocessing. Each face image is cropped based on the 5 face landmarks detected by MTCNN [43] using a similarity transformation. The cropped image is of size 112× 112. Each RGB pixel ([0, 255]) is normalized to [−1, 1]. We put the details of training and validation datasets in Appendix A. CNNs. We adopt the same CNNs from [13, 35] for fair comparison. We use 20-layer CNNs in ablations and 64-layer CNNs for the comparison to existing state-of-the-art methods.
Training and Testing. We use SGD with momentum 0.9 by default. We adopt VGGFace2 [2] as the same training set for all the methods. VGGFace2 contains 3.1M images from 8.6K identities, as shown in Table 9. The training faces are horizontally flipped for data augmentation. We strictly follow the specific protocol provided in each dataset for evaluations. Given a face image, we extract a 512D embedding. The final score is computed by the cosine distance of two embeddings. The nearest neighbor classifier and thresholding are used for face identification and verification, respectively.

heading: 3.1 ABLATION STUDY AND EXPLORATORY EXPERIMENTS
text: We perform an ablation study on four validation sets: LFW, AgeDB-30, CA-LFW, and CP-LFW. The statistics of these datasets are summarized in Table 9. Following the provided evaluation protocols, we report 1:1 verification accuracy of 6,000 pairs (3,000 positive and 3,000 negative pairs) for each dataset. In addition, we combine these datasets and compute the overall verification accuracy, which serves as a more accurate metric to evaluate the models.
Design Principles. We start with ablations on the designing principles of the loss function. As shown in Table 2, it is quite effective to improve the performance following the principles to design a loss function. Specifically, the naive binary classification (Eq. 1) fails to converge, since the learning objective and
the gradients are both dominated by the negative pairs. To address this, we set λ to K−1K and report the results in the first row of Table 2. As can be seen, while the model is trainable and starts to converge, the results show significant room for improvement. After some tuning of the hyperparameter λ,
SphereFace2 can achieve 78.03% accuracy on the combined dataset. Then we gradually incorporate the hard sample mining and angular margin into SphereFace2, improving the verification accuracy from 78.03% to 89.65%, and then to 93.87% With similarity adjustment, SphereFace2 yields 94.28% accuracy on the combined dataset. Table 2 clearly shows the effectiveness of each ingredient.
Hyperparameters. There are four hyperparameters (λ, r, m, and t) in SphereFace2. Since r and m have been extensively explored in [4, 33–35], we follow previous practice to fix r and m to 30 and 0.4 respectively. Our experiments mainly focus on analyzing the effect of λ and t.
From Table 3, the performance of SphereFace2 remains stable for λ from 0.5 to 0.9, where a good balance for the positive and negative pairs is achieved. Then we evaluate different t for λ = [0.6, 0.7, 0.8] and report the results in Table 4. As can be seen from the results, adjusting the similarity scores further boosts model accuracy. The effect of different t is also illustrated in Fig. 5. More detailed ablation studies are included in Appendix C and C.
State-of-art loss functions. We compare SphereFace2 with current state-ofart loss functions in Table 5. We note that these current best-performing methods [4, 13, 30, 35] are based on multiclass classification and belong to the triplet learning paradigm. In contrast, our SphereFace2 is based on binary classification and adopts the pair-wise learning paradigm. We re-implement SphereFace [13] with feature normaliza-
tion for fair comparison. We observe that the verification accuracies on LFW are saturated around 99.5%. On both AgeDB-30 and CA-LFW datasets, SphereFace2 achieves the best accuracy, outperforming the second best results by a significant margin. The results on the combined datasets also validate the effectiveness of SphereFace2.

heading: 3.2 EVALUATIONS ON LARGE-SCALE BENCHMARKS
text: We use three challenging face recognition benchmarks, IJB-B, IJB-C, and MegaFace to evaluate SphereFace2 (with λ=0.7, r=40,m=0.4, t=3). We use 64-layer CNNs for all the methods here.
IJB datasets. IJB-B [39] has 21.8K still images (including 11.8K faces and 10k non-faces) and 55K frames from 7K videos. The total number of identities is 1,845. We follow the standard 1:1 verification and 1:N identification protocols for experiments. The protocol defines 12,115 templates, where each template consists of multiple images and/or frames. Matching is performed based on the defined templates. Specifically, 10,270 genuine matches and 8M impos-
tor matches are constructed in 1:1 verification protocol, and 10,270 probes and 1,875 galleries are constructed in 1:N identification protocol. IJB-C is an extension of IJB-B, comprising 3,531 identities with 31.3K still images and 117.5K frames from 11.8K videos. The evaluation protocols of IJB-C are similar to IJB-B. The details of the protocols are summarized in Appendix. We report the true accept rates (TAR) at different false accept rates (FAR) for verification, and true positive identification rates (TPIR) at different false positive identification rates (FPIR) for identification, as shown in Table 6.
We make several observations based on the evaluation results of IJB-B (Table 6). First, SphereFace2 produces significant improvements over other state-of-the-art methods, especially at low FARs and FPIRs. Specifically, SphereFace2 outperforms CosFace by 5.37% at FAR=1e-5 and 2.70% at FAR=1e-4 in 1:1 verification, 3.09% at FPIR=1e-2 and 3.02% at FPIR=1e-1 in 1:N identification. These significant performance gains suggest
that the pair-wise learning paradigm is very useful in improving the robustness of a face recognition system. Similar observations can also be found in the results of IJB-C (Table 7) and the ROC curves results (Fig. 9). Second, the performance is getting saturated for verification rate at FAR=1e-3. Compared to other methods, SphereFace2 can still improve the results by 0.67% - 1.02% on IJB-B and 0.43% - 1.04% on IJB-C. Third, the rank-1 identification performance of SphereFace2 is slightly better than CosFace, ArcFace, Circle Loss, and comparable to SphereFace. Overall, SphereFace2 performs significantly better than current best-performing methods on these two challenging datasets.
MegaFace. We further evaluate SphereFace2 on the MegaFace dataset. This is a challenging testing benchmark to evaluate the performance of face recognition methods at the million scale of distractors. It contains a gallery set with more than 1 million images from 690K different identities, and a probe set with 3,530 images from 530 identities. MegaFace provides two testing protocols for identification and verification. We evaluate on both and
report the results in Table 8. The gains are consistent with the IJB datasets. Under the same training setup, SphereFace2 outperforms current state-of-the-art methods by a large margin.

heading: 3.3 NOISY LABEL LEARNING
text: Since the pair-wise labels used in SphereFace2 provide weaker supervision than the class-wise labels, we perform experiments to evaluate the robustness of SphereFace2 in label-noisy training. We randomly alter 20%, 40%, 60% and 80% of the labels for each class. The four noisy datasets are used to train CosFace, ArcFace, and
SphereFace2 separately. We evaluate the trained models on the combined validation set. From Fig 10 (left), SphereFace2 shows stronger robustness to noisy labels than CosFace and ArcFace, as its performance degrades significantly more slowly as the ratio of noisy labels increases from 0 to 0.8.
3.4 MODEL PARALLELIZATION
We follow [4] to parallelize the loss computations for CosFace, ArcFace, and SphereFace2. These methods are trained with 1 million identities. Fig. 10 (right) shows how the number of processed images per second changes with different numbers of GPUs. Note that we do not include the feature extraction here. CosFace and ArcFace have negli-
gible difference on running time. When a single GPU is used (i.e., no model parallelization), CosFace and ArcFace are slightly faster than SphereFace2. As the number of GPUs increases, the acceleration of SphereFace2 is more significant, due to less communication cost over GPUs. The near linear acceleration for SphereFace2 is owing to its proxy-based pair-wise formulation.

heading: 4 REPRODUCIBILITY STATEMENT
text: Experiments. To ensure the reproducibility of the empirical results, we include all the necessary details (including all hyperparameters) in the experiments. On acceptance, we will publicly release our codebase to ensure reproducibility of our experiments.

heading: A STATISTICS FOR THE USED DATASETS

heading: B STATISTICS OF IJB TEST PROTOCOLS

heading: C MORE HYPERPARAMETER EXPERIMENTS
text: We additionally provide the results under different hyperparameters. We follow the same settings as in the main paper by training a 20-layer CNN [13] on VGGFace2. First, we vary the hyperparameter r from 20 to 50 and the results in Table 11 show that both r = 30 and r = 40 work reasonably well. Second, we vary the hyperparameter m from 0.2 to 0.5 and evaluate how the size of margin affects the performance. The results in Table 11 show that m = 0.3 generally achieves the best performance. Last, we evaluate how the hyperparameter t in similarity adjustment may affect the performance. Table 11 shows that similarity adjustment is generally helpful for performance (i.e., t = 2, 3, 4, 5 works better than t = 1) and t = 3 achieves the best combined performance.

heading: D SIMILARITY ADJUSTMENT FOR OTHER METHODS
text: To empirically show the comparison between SphereFace2 and other methods with SA, we present the experiments on IJBB and IJBC datasets. As shown in Table 12, similarity adjustment works well for SphereFace2, and applying it to multi-class classification losses is not as useful as in SphereFace2.

heading: E GRADIENT COMPARISON
text: We compare the gradient between the multi-class softmax-based loss and SphereFace2 in Table 13. We observe that the gradient updates in SphereFace2 can be performed with only the corresponding classifier weights (i.e., no summation over classifier weights of different classes). Therefore, the classifier layer in the SphereFace2 framework can be back-propagated with the local GPU and involve no communication overhead.

Review summaries:
This paper has proposed SphereFace2, a new face recognition training scheme that applies a set of binary classification losses to each identities in the dataset, to learn better face embeddings. The proposed method has addressed some limitations of existing training schemes, such as discrepancy between the softmax-based loss and the actual open-set face recognition scenarios. The proposed binary classification protocol works together with a series of optimization strategies, such as easy/hard sample mining and similarity adjustment, to achieve superior performance over state-of-the-art methods as shown in the extensive experiments.

This paper proposes a novel face recognition model called SphereFace2. The key idea is to replace the softmax-based multi-class training with multiple binary classification training, which is more consistent with the testing phrase and generalized to both closed/open-set evaluations. To make this novel strategy works, the authors defined some principles in designing an effective loss function, including (1) Positive/negative sample balance, (2) Easy/hard sample mining, (3) Angular margin, and (4) Similarity adjustment. SphereFace2 outperforms the state-of-the-art face recognition models on 1:1 verification and 1:N open-set evaluation benchmarks. It also allows model parallelization that effectively speeds up the training process when multiple GPUs are available.

This paper presents a binary classification training framework, SphereFace2, for face recognition. Instead of multi-class classification, the proposed method adopts pair-wise comparisons learned through a well-designed loss function, which is more suitable for open-set evaluation. Comprehensive experiments demonstrate the effectiveness of the proposed method.