SphereFace2: Binary Classification is All You Need for Deep Face Recognition

1 INTRODUCTION . Recent years have witnessed the tremendous success of deep face recognition ( FR ) , largely owing to rapid development in training objectives [ 4 , 13 , 14 , 27 , 28 , 33–35 , 38 ] . Current deep FR methods are typically based on a multi-class learning objective , e.g. , softmax cross-entropy loss [ 4 , 13 , 14 , 23 , 33– 35 ] . Despite its empirical effectiveness , there is an obvious discrepancy between such a multi-class classification training and open-set pair-wise testing , as shown in Fig . 1 . In contrast to multi-class classification , pair-wise verification is a binary problem where we need to determine whether a pair of face images belongs to the same person or not . This significant discrepancy may cause the training target to deviate from the underlying FR task and therefore limit the performance . This problem has also been noticed in [ 25 , 27 ] , but they still try to address it under the multi-class ( or triplet ) learning framework which still fundamentally differs from pair-wise verification in testing . On the other hand , multi-class classification training assumes a closed-set environment where all the training data must belong to one of the known identities , which is fundamentally different from open-set testing . In order to address these limitations , we propose a novel deep face recognition framework completely based on binary classification . SphereFace [ 13 ] is one of the earliest works that explicitly performs multi-class classification on the hypersphere ( i.e. , angular space ) in deep FR . In light of this , we name our framework SphereFace2 because it exclusively performs binary classi- fications ( hence the “ 2 ” ) on the hypersphere . Unlike multi-class classification training , SphereFace2 effectively bridges the gap between training and testing , because both training and testing perform pair-wise comparisons . Moreover , SphereFace2 alleviates the closed-set assumption and views the training as an open-set learning problem . For example , training samples that do not belong to any class are still useful and can serve as negative samples in SphereFace2 , while they can not be used at all in the multi-class classification framework . Specifically , given K classes in the training set , SphereFace2 constructs K binary classification objectives where data from the target class are viewed as positive samples and data from the remaining classes are negative samples . The spirit is similar to “ one-vs-all ” classification [ 24 ] . An intuitive illustration comparing typical multi-class training , our proposed SphereFace2 , and open-set evaluation is given in Fig . 1 . In multi-class classification training , the softmax ( cross-entropy ) loss introduces competition among different classes , because the softmax normalization requires all the class logits are summed to one . Therefore , increasing the confidence of one class will necessarily decrease the confidence of some other classes , which can easily lead to over-confident predictions [ 6 ] . In contrast to the competitive nature in all the softmax losses , our SphereFace2 encourages each binary classification task to individually improve the discriminativeness of the representation without any competition . The significance of our method can also be understood from a different perspective . Depending on how samples interact with each other in training , deep FR methods can be categorized into triplet-based learning and pair-based learning . Triplet-based learning simultaneously utilizes an anchor , positive samples and negative samples in its objective , while pair-based learning uses either positive pairs or negative pairs in one shot . Based on whether a proxy is used to represent a set of samples , both triplet-based and pair-based methods can learn with or without proxies . A categorization of deep FR methods is given in Table 1 . Proxy-free methods usually require expansive pair/triplet mining and most of them [ 8 , 19 , 22 , 27 , 29 , 47 ] use a hybrid loss that includes the standard softmax loss . Typical examples of triplet-based learning with proxy include the softmax loss and its popular variants [ 4 , 13 , 33–35 ] where a classifier is a class proxy and the learning involves an anchor ( i.e. , the deep feature x ) , the positive proxy ( i.e. , the target classifier Wy ) and the negative proxies ( i.e. , the other classifiers Wj , j 6= y ) . Most popular deep FR methods use proxies by default , since they greatly speed up training and improve data-efficiency ( especially on large-scale datasets ) . Distinct from previous deep FR , we believe SphereFace2 is the first work to adopt a pair-wise learning paradigm with proxies . An outstanding difference between triplet-based and pairbased learning is the usage of a universal threshold . In Fig . 2 , we show that tripletbased learning compares the similarity scores between different pairs , while pair-based learning compares a similarity score and a universal threshold . As a pair-based method , SphereFace2 too optimizes the difference between similarity scores and a universal threshold . By learning the threshold to distinguish positive and negative pairs during training , it naturally also generalizes to open-set FR . In our framework , we cast a principled view on designing loss functions by systematically discussing the importance of positive/negative sample balance , easy/hard sample mining and angular margin , and combine these ingredients to propose an effective loss function for SphereFace2 . In fact , most popular variants of the softmax loss [ 4 , 33–35 ] also perform easy/hard sample mining and incorporate angular margin . Moreover , we observe that the distribution of similarity scores from positive pairs is quite different from that of negative pairs , in that it exhibits larger variance , which makes the threshold hard to determine . To address this , we propose a novel similarity adjustment method that can adjust the distribution of pair similarity and effectively improve the generalizability . The flexibility of binary classifications in SphereFace2 brings a few additional advantages . First , in contrast to the classifier layer with the softmax loss which is highly non-trivial to parallelize [ 1 ] , the classifier layer in SphereFace2 can be naturally parallelized across different GPUs . While the gradient update in the softmax loss needs to involve all the classifiers , the gradient update for classifiers in SphereFace2 is independent and fully decoupled from each other . The gradient with respect to each classifier only depends on the feature x and the binary classifier itself . Therefore , SphereFace2 naturally addresses the problem of the softmax loss in training on a large number of face identities [ 1 ] . Second , the pair-wise labels used in SphereFace2 are in fact a weaker form of supervision than the class-wise labels used in all the softmax-based losses . This property is closely connected to the open-set assumption and also makes SphereFace2 less sensitive to class-wise label noise . For example , SphereFace2 does not require the class-wise labels of the negative samples in each binary classification . Instead , as long as we know that the identities of negative samples are different from the positive samples , SphereFace2 will still work well . Our experiments also empirically verify the robustness of SphereFace2 against class-wise label noise . Our major contributions are : • SphereFace2 constructs a novel binary classification framework for deep FR . To our knowledge , SphereFace2 is the first work in deep FR to adopt a pair-wise learning paradigm with proxies . We further summarize a series of general principles for designing a good loss function for SphereFace2 . • The decoupling of the binary classifications leads to a natural parallelization for the classifier layer , making SphereFace2 more scalable than softmax-based losses . • The pair-wise labels used in SphereFace2 are a weaker supervision than the class-wise labels used in the softmax loss , yielding better robustness to class-wise label noise . Related Work . Deep FR methods are either proxy-based or proxy-free , as shown in Table 1 . Proxyfree learning includes contrastive loss [ 3 , 7 , 27 ] and triplet loss [ 25 ] . Both losses directly optimize the similarity between samples , so they are highly unstable and data-inefficient . Proxy-free learning is more widely used in deep metric learning [ 21 , 26 , 32 , 36 , 37 , 40 ] . Proxy-based learning uses a set of proxies to represent different groups of samples and usually works better for large-scale datasets . Typical examples include the softmax loss and its variants [ 4 , 13 , 28 , 30 , 31 , 33–35 ] where each proxy is used to represent a class . SphereFace [ 13 ] suggests that large-margin classification is more aligned with open-set FR , and proposes to learn features with large angular margin . CosFace [ 33 , 35 ] and ArcFace [ 4 ] introduce alternative ways to learn large-margin features with improved stability . Although large-margin classification brings the training target closer to the task of open-set FR , the discrepancy still exists and it is unclear how much the margin should be to close the gap . Moreover , a large margin inevitably leads to training instability . In contrast , SphereFace2 naturally avoids these problems and aligns the training target with open-set FR by adopting pair-wise learning with proxies . 2 THE SPHEREFACE2 FRAMEWORK . 2.1 OVERVIEW AND PRELIMINARIES . The goal of SphereFace2 is to align the training target with open-set verification so that our training is more effective in improving open-set generalization in deep FR . To this end , SphereFace2 explicitly incorporates pair-wise comparison into training by constructing K binary classification tasks ( K is the number of identities in the training set ) . The core of SphereFace2 is the binary classification reformulation of the training target . In the i-th binary classification , we construct the positive samples with the face images from the i-th class and the negative samples with face images from other classes . Specifically , we denote the weights of the i-th binary classifier by Wi , the deep feature by x and its corresponding label by y . A naive loss formulation is Lf =log ( 1 + exp ( −W > y x− by ) ) + ∑ i 6=y log ( 1 + exp ( W > i x+ bi ) ) which is a combination of K standard binary logistic regression losses . Instead , we perform the binary classification on the unit hypersphere by normalizing both classifiers Wi , ∀i and feature x . The loss function now becomes Ls = log ( 1 + exp ( − cos ( θy ) ) ) + ∑ i6=y log ( 1 + exp ( cos ( θi ) ) ) ( 1 ) where θi is the angle between the i-th binary classifier Wi and the sample x . The biases bi , ∀i are usually removed in common practice [ 4 , 13 , 15 , 33 , 35 ] , since they are learned for a closed set and can not generalize to unknown classes . However , we actually find them very useful in SphereFace2 , as will be discussed later . For now , we temporarily remove them for notational convenience . One of the unique advantages of such a parameterization for binary classifiers is that it constructs the i-th class positive proxy with Wi and the negative proxy with −Wi . Depending on the label , the training will minimize the angle between x and Wi or between x and −Wi in order to minimize the loss . This parameterization of positive and negative proxies immediately guarantees minimum hyperspherical energy [ 12 ] that has been shown to benefit generalization . Moreover , our parameterization in SphereFace2 has the same number of parameters for the classifier layer as the previous multi-class training , and does not introduce extra overhead . However , naively minimizing the loss in Eq . ( 1 ) will not give satisfactory results . Therefore , we explore in the next subsection how to find a desirable loss function that works well in the SphereFace2 framework .

Summary: This paper has proposed SphereFace2, a new face recognition training scheme that applies a set of binary classification losses to each identities in the dataset, to learn better face embeddings. The proposed method has addressed some limitations of existing training schemes, such as discrepancy between the softmax-based loss and the actual open-set face recognition scenarios. The proposed binary classification protocol works together with a series of optimization strategies, such as easy/hard sample mining and similarity adjustment, to achieve superior performance over state-of-the-art methods as shown in the extensive experiments.