SphereFace2: Binary Classification is All You Need for Deep Face Recognition

1 INTRODUCTION . Recent years have witnessed the tremendous success of deep face recognition ( FR ) , largely owing to rapid development in training objectives [ 4 , 13 , 14 , 27 , 28 , 33–35 , 38 ] . Current deep FR methods are typically based on a multi-class learning objective , e.g. , softmax cross-entropy loss [ 4 , 13 , 14 , 23 , 33– 35 ] . Despite its empirical effectiveness , there is an obvious discrepancy between such a multi-class classification training and open-set pair-wise testing , as shown in Fig . 1 . In contrast to multi-class classification , pair-wise verification is a binary problem where we need to determine whether a pair of face images belongs to the same person or not . This significant discrepancy may cause the training target to deviate from the underlying FR task and therefore limit the performance . This problem has also been noticed in [ 25 , 27 ] , but they still try to address it under the multi-class ( or triplet ) learning framework which still fundamentally differs from pair-wise verification in testing . On the other hand , multi-class classification training assumes a closed-set environment where all the training data must belong to one of the known identities , which is fundamentally different from open-set testing . In order to address these limitations , we propose a novel deep face recognition framework completely based on binary classification . SphereFace [ 13 ] is one of the earliest works that explicitly performs multi-class classification on the hypersphere ( i.e. , angular space ) in deep FR . In light of this , we name our framework SphereFace2 because it exclusively performs binary classi- fications ( hence the “ 2 ” ) on the hypersphere . Unlike multi-class classification training , SphereFace2 effectively bridges the gap between training and testing , because both training and testing perform pair-wise comparisons . Moreover , SphereFace2 alleviates the closed-set assumption and views the training as an open-set learning problem . For example , training samples that do not belong to any class are still useful and can serve as negative samples in SphereFace2 , while they can not be used at all in the multi-class classification framework . Specifically , given K classes in the training set , SphereFace2 constructs K binary classification objectives where data from the target class are viewed as positive samples and data from the remaining classes are negative samples . The spirit is similar to “ one-vs-all ” classification [ 24 ] . An intuitive illustration comparing typical multi-class training , our proposed SphereFace2 , and open-set evaluation is given in Fig . 1 . In multi-class classification training , the softmax ( cross-entropy ) loss introduces competition among different classes , because the softmax normalization requires all the class logits are summed to one . Therefore , increasing the confidence of one class will necessarily decrease the confidence of some other classes , which can easily lead to over-confident predictions [ 6 ] . In contrast to the competitive nature in all the softmax losses , our SphereFace2 encourages each binary classification task to individually improve the discriminativeness of the representation without any competition . The significance of our method can also be understood from a different perspective . Depending on how samples interact with each other in training , deep FR methods can be categorized into triplet-based learning and pair-based learning . Triplet-based learning simultaneously utilizes an anchor , positive samples and negative samples in its objective , while pair-based learning uses either positive pairs or negative pairs in one shot . Based on whether a proxy is used to represent a set of samples , both triplet-based and pair-based methods can learn with or without proxies . A categorization of deep FR methods is given in Table 1 . Proxy-free methods usually require expansive pair/triplet mining and most of them [ 8 , 19 , 22 , 27 , 29 , 47 ] use a hybrid loss that includes the standard softmax loss . Typical examples of triplet-based learning with proxy include the softmax loss and its popular variants [ 4 , 13 , 33–35 ] where a classifier is a class proxy and the learning involves an anchor ( i.e. , the deep feature x ) , the positive proxy ( i.e. , the target classifier Wy ) and the negative proxies ( i.e. , the other classifiers Wj , j 6= y ) . Most popular deep FR methods use proxies by default , since they greatly speed up training and improve data-efficiency ( especially on large-scale datasets ) . Distinct from previous deep FR , we believe SphereFace2 is the first work to adopt a pair-wise learning paradigm with proxies . An outstanding difference between triplet-based and pairbased learning is the usage of a universal threshold . In Fig . 2 , we show that tripletbased learning compares the similarity scores between different pairs , while pair-based learning compares a similarity score and a universal threshold . As a pair-based method , SphereFace2 too optimizes the difference between similarity scores and a universal threshold . By learning the threshold to distinguish positive and negative pairs during training , it naturally also generalizes to open-set FR . In our framework , we cast a principled view on designing loss functions by systematically discussing the importance of positive/negative sample balance , easy/hard sample mining and angular margin , and combine these ingredients to propose an effective loss function for SphereFace2 . In fact , most popular variants of the softmax loss [ 4 , 33–35 ] also perform easy/hard sample mining and incorporate angular margin . Moreover , we observe that the distribution of similarity scores from positive pairs is quite different from that of negative pairs , in that it exhibits larger variance , which makes the threshold hard to determine . To address this , we propose a novel similarity adjustment method that can adjust the distribution of pair similarity and effectively improve the generalizability . The flexibility of binary classifications in SphereFace2 brings a few additional advantages . First , in contrast to the classifier layer with the softmax loss which is highly non-trivial to parallelize [ 1 ] , the classifier layer in SphereFace2 can be naturally parallelized across different GPUs . While the gradient update in the softmax loss needs to involve all the classifiers , the gradient update for classifiers in SphereFace2 is independent and fully decoupled from each other . The gradient with respect to each classifier only depends on the feature x and the binary classifier itself . Therefore , SphereFace2 naturally addresses the problem of the softmax loss in training on a large number of face identities [ 1 ] . Second , the pair-wise labels used in SphereFace2 are in fact a weaker form of supervision than the class-wise labels used in all the softmax-based losses . This property is closely connected to the open-set assumption and also makes SphereFace2 less sensitive to class-wise label noise . For example , SphereFace2 does not require the class-wise labels of the negative samples in each binary classification . Instead , as long as we know that the identities of negative samples are different from the positive samples , SphereFace2 will still work well . Our experiments also empirically verify the robustness of SphereFace2 against class-wise label noise . Our major contributions are : • SphereFace2 constructs a novel binary classification framework for deep FR . To our knowledge , SphereFace2 is the first work in deep FR to adopt a pair-wise learning paradigm with proxies . We further summarize a series of general principles for designing a good loss function for SphereFace2 . • The decoupling of the binary classifications leads to a natural parallelization for the classifier layer , making SphereFace2 more scalable than softmax-based losses . • The pair-wise labels used in SphereFace2 are a weaker supervision than the class-wise labels used in the softmax loss , yielding better robustness to class-wise label noise . Related Work . Deep FR methods are either proxy-based or proxy-free , as shown in Table 1 . Proxyfree learning includes contrastive loss [ 3 , 7 , 27 ] and triplet loss [ 25 ] . Both losses directly optimize the similarity between samples , so they are highly unstable and data-inefficient . Proxy-free learning is more widely used in deep metric learning [ 21 , 26 , 32 , 36 , 37 , 40 ] . Proxy-based learning uses a set of proxies to represent different groups of samples and usually works better for large-scale datasets . Typical examples include the softmax loss and its variants [ 4 , 13 , 28 , 30 , 31 , 33–35 ] where each proxy is used to represent a class . SphereFace [ 13 ] suggests that large-margin classification is more aligned with open-set FR , and proposes to learn features with large angular margin . CosFace [ 33 , 35 ] and ArcFace [ 4 ] introduce alternative ways to learn large-margin features with improved stability . Although large-margin classification brings the training target closer to the task of open-set FR , the discrepancy still exists and it is unclear how much the margin should be to close the gap . Moreover , a large margin inevitably leads to training instability . In contrast , SphereFace2 naturally avoids these problems and aligns the training target with open-set FR by adopting pair-wise learning with proxies . 2 THE SPHEREFACE2 FRAMEWORK . 2.1 OVERVIEW AND PRELIMINARIES . The goal of SphereFace2 is to align the training target with open-set verification so that our training is more effective in improving open-set generalization in deep FR . To this end , SphereFace2 explicitly incorporates pair-wise comparison into training by constructing K binary classification tasks ( K is the number of identities in the training set ) . The core of SphereFace2 is the binary classification reformulation of the training target . In the i-th binary classification , we construct the positive samples with the face images from the i-th class and the negative samples with face images from other classes . Specifically , we denote the weights of the i-th binary classifier by Wi , the deep feature by x and its corresponding label by y . A naive loss formulation is Lf =log ( 1 + exp ( −W > y x− by ) ) + ∑ i 6=y log ( 1 + exp ( W > i x+ bi ) ) which is a combination of K standard binary logistic regression losses . Instead , we perform the binary classification on the unit hypersphere by normalizing both classifiers Wi , ∀i and feature x . The loss function now becomes Ls = log ( 1 + exp ( − cos ( θy ) ) ) + ∑ i6=y log ( 1 + exp ( cos ( θi ) ) ) ( 1 ) where θi is the angle between the i-th binary classifier Wi and the sample x . The biases bi , ∀i are usually removed in common practice [ 4 , 13 , 15 , 33 , 35 ] , since they are learned for a closed set and can not generalize to unknown classes . However , we actually find them very useful in SphereFace2 , as will be discussed later . For now , we temporarily remove them for notational convenience . One of the unique advantages of such a parameterization for binary classifiers is that it constructs the i-th class positive proxy with Wi and the negative proxy with −Wi . Depending on the label , the training will minimize the angle between x and Wi or between x and −Wi in order to minimize the loss . This parameterization of positive and negative proxies immediately guarantees minimum hyperspherical energy [ 12 ] that has been shown to benefit generalization . Moreover , our parameterization in SphereFace2 has the same number of parameters for the classifier layer as the previous multi-class training , and does not introduce extra overhead . However , naively minimizing the loss in Eq . ( 1 ) will not give satisfactory results . Therefore , we explore in the next subsection how to find a desirable loss function that works well in the SphereFace2 framework .

Summary: This paper proposes a novel face recognition model called SphereFace2. The key idea is to replace the softmax-based multi-class training with multiple binary classification training, which is more consistent with the testing phrase and generalized to both closed/open-set evaluations. To make this novel strategy works, the authors defined some principles in designing an effective loss function, including (1) Positive/negative sample balance, (2) Easy/hard sample mining, (3) Angular margin, and (4) Similarity adjustment. SphereFace2 outperforms the state-of-the-art face recognition models on 1:1 verification and 1:N open-set evaluation benchmarks. It also allows model parallelization that effectively speeds up the training process when multiple GPUs are available.