Title: UniTSFace: Unified Threshold Integrated Sample-to-Sample Loss for Face Recognition
Venue: NeurIPS 2023
Accepted: yes
Keywords: Face Recognition

Unified Threshold

USS Loss
Abstract: Sample-to-class-based face recognition models can not fully explore the cross-sample relationship among large amounts of facial images, while sample-to-sample-based models require sophisticated pairing processes for training. Furthermore, neither method satisfies the requirements of real-world face verification applications, which expect a unified threshold separating positive from negative facial pairs. In this paper, we propose a unified threshold integrated sample-to-sample based loss (USS loss), which features an explicit unified threshold for distinguishing positive from negative pairs. Inspired by our USS loss, we also derive the sample-to-sample based softmax and BCE losses, and discuss their relationship. Extensive evaluation on multiple benchmark datasets, including MFR, IJB-C, LFW, CFP-FP, AgeDB, and MegaFace, demonstrates that the proposed USS loss is highly efficient and can work seamlessly with sample-to-class-based losses. The embedded loss (USS and sample-to-class Softmax loss) overcomes the pitfalls of previous approaches and the trained facial model UniTSFace exhibits exceptional performance, outperforming state-of-the-art methods, such as CosFace, ArcFace, VPL, AnchorFace, and UNPG. Our code is available at https://github.com/CVI-SZU/UniTSFace.

UniTSFace: Unified Threshold Integrated Sample-to-Sample Loss for Face Recognition

Abstract

Sample-to-class-based face recognition models can not fully explore the crosssample relationship among large amounts of facial images, while sample-to-samplebased models require sophisticated pairing processes for training. Furthermore, neither method satisfies the requirements of real-world face verification applications, which expect a unified threshold separating positive from negative facial pairs. In this paper, we propose a unified threshold integrated sample-to-sample based loss (USS loss), which features an explicit unified threshold for distinguishing positive from negative pairs. Inspired by our USS loss, we also derive the sample-to-sample based softmax and BCE losses, and discuss their relationship. Extensive evaluation on multiple benchmark datasets, including MFR, IJB-C, LFW, CFP-FP, AgeDB, and MegaFace, demonstrates that the proposed USS loss is highly efficient and can work seamlessly with sample-to-class-based losses. The embedded loss (USS and sample-to-class Softmax loss) overcomes the pitfalls of previous approaches and the trained facial model UniTSFace exhibits exceptional performance, outperforming state-of-the-art methods, such as CosFace, ArcFace, VPL, AnchorFace, and UNPG. Our code is available at https://github.com/CVI-SZU/UniTSFace .

1 Introduction

Modern deep facial recognition systems, involving an enormous number of facial images and identities, essentially rely on discriminative feature learning: the facial images from the same identity should be close while those from different identities should be distant in the feature space. That is, the similarity of a positive pair (two facial images from the same identity) is required to be larger than any negative pair (two facial images from different identities). In other words, a unified threshold is expected to distinguish positive from negative pairs.

The feature learning process of general deep face recognition models is either based on sample-toclass losses (such as Softmax loss (27; 28)) or sample-to-sample losses (such as contrastive (4; 10; 25)

and triplet loss (22)). Inspired by the success of large-scale image classification, the softmax loss and its extensions have become popular in deep face recognition systems. In the softmax loss, each weight vector can be regarded as a proxy of the corresponding class (face identity), the classification-based face recognition models are demanded to learn the class proxy and image features simultaneously. However, these models possess a significant drawback, namely a domain gap between the training and testing stages of face models. This gap arises from the reliance on limited identity proxies for computing and optimizing feature similarity in sample-to-class models. Conversely, in real-world scenarios, feature similarity is computed and compared across various samples from diverse facial identities. In other words, the sample-to-class-based classification strategy may not entirely explore the variances across samples ( Problem 1 ). Therefore, its efficacy in accurately reflecting real-world scenarios is questionable. To tackle this drawback, VPL (8) considers a small variation around the class proxy, which implicitly introduces more samples during the optimization and improves the face recognition performance. However, while this small variation extends the capability of softmax loss, it cannot essentially represent all the samples in this class.

Face models based on sample-to-sample losses (4; 10; 22; 3; 24) learn facial identity features and optimize feature similarities by comparing positive and negative samples, which is closer to real-world face recognition applications than sample-to-class losses. The majority of sample-to-sample losses aim to maximize inter-class discrepancy while minimizing intra-class distances, which may need a meticulous sampling/paring step for every mini-batch. Moreover, in face verification tasks, a single threshold is required to distinguish positive facial pairs from negative ones. Unfortunately, none of the aforementioned methods incorporate such an explicit constraint ( Problem 2 ).

In this study, we commence our research by analyzing a reasonable, albeit naive, sample-to-sample loss, which enables us to straightforwardly investigate the variances across facial samples, thereby resolving Problem 1 . To address Problem 2 , we integrate a unified learnable threshold into the naive loss, resulting in a novel sample-to-sample loss function that we term the unified threshold integrated sample-to-sample (USS) loss. We provide a mathematical elaboration of the relationship between our USS loss and the sample-to-sample BCE loss and softmax loss, from the perspective of the naive sample-to-sample loss. To evaluate the effectiveness of the USS loss, we conduct experiments on various benchmark datasets and qualitatively demonstrate that it meets the requirements of real-world applications. Additionally, we find that incorporating a margin can easily enhance the USS loss. Our USS loss also works seamlessly with the sample-to-class based losses and shows significant improvements when they are combined together to train a face model called UniTSFace. To summarize, the contributions of this work are

We introduce a unified threshold integrated sample-to-sample loss (USS) derived from a reasonable naive loss for face recognition. We also derive the sample-to-sample based softmax and binary-cross-entropy (BCE) losses from the naive loss and reveal the mathematical relationship among the softmax, BCE, and our USS losses.

We demonstrate that a unified threshold can be learned by adopting the proposed USS loss and quantitatively and qualitatively demonstrate that the learned threshold aligns with face verification expectations in experiments.

The proposed USS loss can be enhanced with an auxiliary margin and is compatible with existing sample-to-class based losses. We show that USS loss, when used jointly with sampleto-class based losses such as CosFace and ArcFace, leads to a continuous improvement.

Our UniTSFace outperforms state-of-the-art methods on the Megaface 1 dataset and ranks first place on the MFR ongoing challenge till the submission of this work (May 17 '23, academic track): http://iccv21-mfr.com/#/leaderboard/academic .

2 Related Works

Sample-to-Sample based Methods. DeepID2 (25) uses a contrastive loss(4; 10) to encourage the features learned from the same identity to be close while that learned from different identities are distant. FaceNet(22) constructs three-element tuples and minimizes the distance between an anchor and a positive sample and maximizes the distance between the anchor and a negative sample. The effectiveness of contrastive/triplet losses relies on the meticulous selection of pairs/triplets. Furthermore, neither method explicitly enforces that all positive sample-to-sample similarities are greater than negative similarities.

Sample-to-Class based Methods. While Softmax loss is frequently utilized in deep recognition models, it only promotes separability and does not learn discriminative features. Various approaches have been proposed to enhance the feature learning of softmax loss. Some methods (32; 9; 34; 16) proposed extra constraints imposed on the softmax loss, such methods normally have a class proxy (class center/prototype) and maximize (minimize) the similarity (distances) between a facial feature and its corresponding class proxy (32), as well as maximize the distances between all class proxies(16; 9; 34). However, the training process of such methods needs careful balancing of the softmax loss and the extra constraints. Alternatively, some works directly improve the softmax by normalizing the facial features and adding margins between positive and negative sample-to-class pairs (18; 29; 30; 7; 17). While sample-to-class methods demonstrate excellent performance in deep face verification, they may not entirely explore the variability across various facial samples.

Hybrid (/combined) Methods. Circle loss (26) is one of the first works that discussed the two elemental learning paradigms, i.e., learning with class-level labels and pair-wise labels, in a unified framework. However, the two learning paradigms are respectively used in circle loss. Following (26), the combination of the two learning paradigms to solve the shortcoming of either method become popular, e.g., VPL(8), AnchorFace(15), and UNPG(13). VPL extends the softmax loss by considering a small feature variation around the class proxy W . The embedding of a small feature variation cannot essentially represent all samples, VPL still presents a large gap between training a face recognition model and testing over the open sets. UNPG jointly optimizes the distance between a sample x i with negative class proxies W j and negative samples x j in the same softmax loss using batch processing, focusing on the negative pairs during training. AnchorFace, however, uses a combination of softmax loss and different sample-to-sample losses, such as TAR Loss and FAR Loss, to target specific testing protocols. In contrast, our USS loss is more general and compatible with existing sample-to-class losses. We demonstrate our USS loss can lead to a steady improvement when used with sample-to-class losses jointly in experiments.

3 Methods

Suppose M is a deep face model trained on a facial sample set D = ⋃ N i =1 D i captured from N subjects, where D i denotes the subset containing the facial samples captured from the same subject i . Then, we can get a feature set F = ⋃ N i =1 F i = ⋃ N i =1 { x ( i ) = M ( X ( i ) ) : X ( i ) ∈ D i } , where x ( i ) is the feature vector of the sample X ( i ) and X ( i ) denotes the sample captured from subject i .

For any two samples X , X ∗ ∈ D , we apply a bivariate operator g ( x , x ∗ ) ∈ [ -1 , 1] denoting their feature similarity, where x = M ( X ) , x ∗ = M ( X ∗ ) are features of X , X ∗ . We term g ( x , x ∗ ) the positive sample-to-sample similarity if the two samples are captured from the same subject, while the negative sample-to-sample similarity if they are from different subjects. Then, for any sample X ( i ) ∈ D i , ∀ i , we define its positive and negative similarity sets as

̸

where x ( i ) = M ( X ( i ) ) .

In real applications of face verification, a threshold ˆ t ∈ [ -1 , 1] is chosen to verify whether two samples X and X ∗ are from the same subject or not. Specifically, the two facial samples X , X ∗ are with the same identity if g ( x , x ∗ ) ≥ ˆ t , while they are from two different identities when g ( x , x ∗ ) < ˆ t .

̸

To be in line with face verification applications, during the training process of M , we expect a unified threshold t such that any two samples ( X ( i ) ∈ D i and X ( j ) ∗ ∈ D j ) conform to the above rules. If they share the same identity (i.e., i = j ), then g ( x ( i ) , x ( j ) ∗ ) ≥ t , in the case of i = j , g ( x ( i ) , x ( j ) ∗ ) < t , where x ( i ) = M ( X ( i ) ) , x ( j ) ∗ = M ( X ( j ) ∗ ) . The unified threshold t satisfies

However, the existing sample-to-sample losses (4; 10; 22; 24) fail to explicitly include and learn the unified threshold. In this paper, by explicitly defining the unified threshold t , we design a unified threshold integrated sample-to-sample loss.

3.1 Unified Threshold Integraed Sample-to-Sample Loss (USS Loss)

Clearly, in the training of model M , it expects large positive sample-to-sample similarities but small negative ones, and then, for any sample X ( i ) , a naive loss could be reasonably defined as,

̸

̸

where γ is a scale factor, |S| denotes the element number of a set S , and F - F i denotes the complementary set of F i in F , i.e., F - F i = ⋃ N j =1 j = i F j .

The loss in Eq. (4) computes the feature similarities of all positive sample-to-sample pairs and all negative ones for the sample X ( i ) , which is not easily implemented in practice. In this paper, without loss of generality, we consider the feature similarities of one positive sample pair and N -1 negative sample pairs for the sample X ( i ) in the naive loss,

̸

̸

where x ( i ) ∗ = M ( X ( i ) ∗ ) , x ( j ) ∗ = M ( X ( j ) ∗ ) , and X ( i ) ∗ and X ( j ) ∗ are randomly taken from the subject i, j , with j = i .

Using the inequality of arithmetic and geometric means 1 , we derive two inequalities about L naive ,

̸

̸

According to Eqs. (6), (7), and the unified threshold t in Eq. (3), one can get

̸

̸

̸

̸

̸

We define the U nified threshold integrated S ample-toS ample (USS) loss L uss ( X ( i ) ) as

̸

where b = γt is a constant to be learned (depicted in Fig. 1). The detailed derivations of the above inequalities are described in the supplementary (appendix).

We here analyze that the unified threshold t could be learned. Suppose that the model M has been perfectly trained, and L uss has reached its minimum point after the training, which means (i) the positive sample-to-sample similarity g ( x ( i ) , x ( i ) ∗ ) tends to 1 , and the negative ones g ( x ( i ) , x ( j ) ∗ ) tends to -1 ; and (ii) L uss reaches its stationary point in terms of variable b . From (ii), one can deduce that

̸

̸

If N < e 2 γ +3 2 , the final learned threshold t = b γ will locate between -1 and 1 , which means now the leaned t has correctly separated the negative sample-to-sample similarities, g ( x ( i ) , x ( j ) ∗ ) , and the positive ones, g ( x ( i ) , x ( i ) ∗ ) . In the practice, following (30; 7), we set γ = 64 , then, according to the above analysis, the unified threshold t could be leaned if N < 1 . 9 × 10 55 .

3.2 Sample-to-Sample Based Softmax and BCE Losses

Sample-to-class based softmax and BCE losses are widely applied in the image classification. For the study of face verification, we here deduce the sample-to-sample based softmax and BCE losses from the naive loss L naive .

Similar to the deduction of USS loss, using the inequality of arithmetic and geometric means, we first present another two inequalities about L naive ,

̸

Softmax Loss. For sample X ( i ) ∈ D i with x ( i ) = M ( X ( i ) ) , we define its sample-to-sample softmax loss as

Then, according to Eq. (15), one can get

Similar to the naive loss L naive, the design of softmax loss L soft does not consider the unified threshold among the sample-to-sample pairs.

BCE Loss. For all samples captured from subject i , we assume the existence of a threshold t i , which could separate their all positive sample-to-sample pairs and negative ones, i.e.,

then, according to Eqs. (16) and (6),

̸

We define BCE loss for sample X ( i ) as

̸

where b i = γt i are parameters to be learned. Note that the t i can be different for different identities and therefore are not unified.

After the training of the model M , i.e., the threshold t i = b i γ were learned, then,

3.3 Marginal Sample-to-Sample Based Losses

Our USS loss, as well as the deduced sample-to-sample based L soft and L bce, only encourage the separability between positive and negative sample pairs. To further improve the discriminative ability of such losses, i.e., to encourage the positive features to be distant from the negative ones, we further proposed the marginal extensions for such losses.

By introducing a margin on the feature similarities, we can have the marginal USS loss as:

̸

where the m is the introduced hyper-parameter on the margin. Proper adjusting of such a parameter can improve the recognition performance, as discussed in Sec. 4.3. Similarly, we can extend the vanilla L soft and L bce to the marginal version of L soft-m and L bce-m, the full equations are given in the supplementary materials (and appendix).

4 Experiments

4.1 Datasets and Evaluations

Datasets. We utilize four publicly available datasets for training, namely, CASIA-WebFace(33) (consisting of 0.5 million images of 10K identities), Glint360K(2) (comprising 17.1 million images of 360K identities), WebFace42M(35) (containing 42.5 million images of 2 million identities), and WebFace4M, which is a subset of WebFace42M with 4.2 million images of 0.2 million identities. For evaluating the face verification performance, the ICCV-2021 Masked Face Recognition Challenge (MFR Ongoing)(5) is adopted. The MFR ongoing testing protocol includes various popular benchmarks, such as LFW (11), CFP-FP (23), AgeDB (21), and IJB-C (20), along with its own MFR benchmarks, such as the Mask, Children, and Globalized Multi-Racial (GMR) test sets. The Mask set comprises 13.9K positive pairs and 96.9 million negative pairs (including 6.9K masked images and

̸

̸

13.9K non-masked images) of 6.9K identities. The Children set contains 157K images (totaling 1.7 million positive pairs and 24.7 billion negative pairs) of 14K identities. The Globalized Multi-Racial sets comprise 1.6 million images in total, consisting of 4.6 million positive pairs and 2.6 trillion negative pairs, and representing 242K identities across four races: African, Caucasian, South-Asian, and East-Asian. For face identification, the MegaFace Challenge 1(14) is employed as the test set, which comprises a gallery set with over 1 million images from 690K different identities, and a probe set with 3,530 images from 530 identities. It is worth noting that the MegaFace Challenge 1 also has a verification track, and the verification performance is included in the experiments.

Evaluation and Metrics. For the MFR Ongoing Challenge, the trained models are submitted to and evaluated by the online server. Specifically, we report 1:1 verification accuracy for LFW, CFP-FP, and AgeDB. We report True Accept Rate (TAR) at False Accept Rate (FAR) levels of 1e-4 and 1e-5 for IJB-C. We report TARs at FAR=1e-4 for the Mask and Children test sets, and TARs at FAR=1e-6 for the GMR test sets. For the MegaFace Challenge 1, we report Rank1 accuracy for identification and TAR at FAR=1e-6 for verification.

4.2 Implementation Details

Preprocessing . Firstly, we aligned all face images using the 5 landmarks detected by RetinaFace (6) and cropped the center 112 × 112 patch. We then normalized the cropped images by first subtracting 127.5 and then dividing 128. Finally, we augmented the training images with horizontal flipping.

Training . We adopt customized ResNets as our backbone following (7). We implement all models using Pytorch and train them using the SGD optimizer with a weight decay of 5e-4 and momentum of 0.9. For the face models on CASIA-WebFace, we train them over 28 epochs with a batch size of 512. The learning rate starts at 0.1 and is reduced by a factor of 10 at the 16 th and 24 th epoch. For both Glint360K and WebFace4M, we train the ResNets for 20 epochs using a batch size of 1024. The learning rate is initially set at 0.1, while a polynomial decay strategy (power=2) is applied for the learning rate schedule. In the case of WebFace42M, we train the ResNets for 20 epochs, using a larger batch size of 4096. The learning rate linearly warms up from 0 to 0.4 during the first epoch, followed by a polynomial decay (power=2) for the remaining 19 epochs. We include the detailed settings of all hyper-parameters used in Sec. 4 and Sec. 5 in the appendix for further reference.

Testing . For a given facial image, we extract two 512-dimensional features from the original image and its horizontally flipped counterpart. These features are then combined together as the final representation with element-wise addition. To assess the similarity between two images, we utilize the cosine similarity metric.

4.3 Ablation and Parameter Study

Effectiveness of unified threshold. We first demonstrate the necessity of learning a unified threshold by comparing our proposed L uss with two other sample-to-sample losses, L soft and L bce, in Table 1. We additionally report the performance of the model trained with L naive . The first to third rows respectively show the results from L naive , L soft and L bce, while the fourth row lists the performance of our L uss. We observe that L soft performs 4 . 68% better than our proposed L uss on IJB-C. However, our USS loss outperforms L soft on the other four

datasets, particularly on MFR-All where it achieves a 12 . 09% higher TAR@FAR=1e-6. Compared to the L bce, the gains achieved by our L uss are more significant, with performance improvements of 29 . 52% on MFR-All, 33 . 01% in terms of TAR@FAR=1e-4 on IJB-C, 30 . 10% on CFP-FP, and 19 . 69% on AgeDB. The L naive, however, is difficult to train and the trained model barely learns any useful features. We note that the results are not cherry-picked. For a fair comparison, we adopted the same experimental setting when we trained the model using the different losses. During our experiments, we found that the model is easy to converge using L soft and L uss, and the training process is stable, which leads to more favorable results. The model, however, is difficult to train with L naive and L bce. We have dedicated many efforts to these two losses, but the convergence is still problematic.

When comparing the marginal softmax loss L soft-m, marginal BCE loss L bce-m with marginal USS loss L uss-m (last three rows), we observe that the marginal USS loss L uss-m achieves the highest performance across all five reported datasets, which further confirms the effectiveness of our proposed unified threshold strategy.

It is noteworthy that all the methods in this table are based on the same ResNet-50 network trained on the CASIA-WebFace dataset.

Qualitative study of unified threshold. To better understand the effects of unified threshold, in Fig. 1, we randomly select 20 identities (5,074 facial images) from the whole training dataset with N = 10572 subjects, and then construct 1 positive sample pair and N -1 negative pairs for each of the selected images. For each of the 20 identities, we compute the optimal threshold to separate the positive pair and the hardest negative pair, i.e., the most similar negative pair.

Lsoft

Lbce

Luss

0.25

0.30

0.35

0.40

0.45

0.50

0.55

We respectively plot threshold distributions of the 20 identities for L soft , L bce, and L uss . In Fig. 1, each dot denotes the optimal threshold for each identity. The blue dashed line is the unified threshold t = b γ = 31 . 3344 / 64 = 0 . 4896 exclusively learned by our L uss in Eq. (11). From this figure, we can clearly observe that our L uss has the most compact threshold distribution, while the threshold distributions from L bce and L soft are relatively loose. Most importantly, for L uss, the median threshold from the randomly sampled 20 identities is in line with the learned unified threshold 0 . 4896 from the face model trained with L uss, which proves the effects of including an explicit unified threshold.

Impact of Margins. The results in Table 1 (with m = 0 . 1 ) demonstrate that a margin improves the performance of L uss. However, the introduced margin m remains a hyperparameter that needs to be tuned. In Table 2, we investigate the effects of using different margins in L uss-m. Specifically, we experiment with four more settings that increase m to 0.2, 0.3, 0.4, and 0.5. We observe that i) the highest performance on MR-ALL, IJB-C, LFT, CFP-FP, and AgeDB are respectively achieved when m equals 0.2, 0.4, 0.1, 0.1, and 0.3; ii) the performance of the four marginal L uss-m losses are all higher than that of the original L uss (i.e., m = 0 in L uss-m). Although the auxiliary margin improves the overall performance, the performance seems to saturate when m is set to 0.4. Note that in our later experiments, we use m = 0 . 1 .

Compatibility. In Table 1 and Table 2, we have shown that the proposed L uss is superior to other sample-to-sample based losses and can be boosted by adding a proper margin. In Table 3, we investigate the effects of combining L uss with two marginal softmax losses, i.e., ArcFace and CosFace. We find that the model trained with the combined loss significantly outperforms the original ArcFace and CosFace models, as well as our USS loss. For example, the model trained with ArcFace + USS respectively outperforms ArcFace and USS with 6.71% and 6.37% gains on MR-ALL, and 41.07% and 5.64% gains on IJB-C. The model trained with CosFace + USS respectively outperforms CosFace and USS with 5.16% and 7.73% gains on MR-ALL, and 33.19% and 5.92% gains on IJB-C. Such significant improvements suggest the compatibility of our USS loss. In this work, we refer to the model trained using a combination of a sample-to-class method and our USS as UniTSFace. For our experiments, we choose to use CosFace as the sample-to-class method.

5 Comparison with the State-Of-The-Art

MegaFace Challenge 1. In Table 4, we compare both the identification and verification performance of our UniTSFace with several state-of-the-art methods on MegaFace Challenge 1. These methods include sample-to-sample based Contrastive Loss (4; 10) and Triplet Loss (22), sample-to-class based CosFace(30) and ArcFace(7), as well as the hybrid/combined approaches: VPL(8) and UNPG(13).

To ensure a fair comparison, as per the official protocols, we compare our UniTSFace trained on the CASIA-WebFace with the models trained on 'Small' datasets, while UniTSFace trained on Glint360K is compared with the models trained on 'Large' datasets in Table 4. When label refinement (7) is not used, our UniTSFace achieves the highest accuracy among the compared models trained on 'Large' datasets, with an 85.01% identification accuracy and a 97.85% verification accuracy. When label refinement (7) is used, the accuracies can be further increased to 99.27% and 99.19% respectively. Though our UniTSFace is slightly lower than CurricularFace (12) in terms of identification performance on 'Small' datasets, UniTSFace achieves the highest accuracy on the verification track, regardless of whether label refinement is used or not. These results demonstrate the effectiveness of our UniTSFace in both identification and verification tasks.

MFR Ongoing Benchmarks. We then compare the proposed UniTSFace with Contrastive Loss(4; 10), (N+1)-Tuplet Loss(24), CosFace(30), ArcFace(7), VPL(8), AnchorFace(15), and UNPG(13) on the MFR Ongoing benchmark. We re-implement these methods with the optimal hyper-parameters recommended in their original papers. All the compared models are trained with a ResNet-50 backbone and the CASIA-WebFace dataset. As reported in Table 5, our UniTSFace achieves clear improvement over other methods.

Additionally, we compare our UniTSFace with the recent Partial FC(1), which is the leading method on the MFR ongoing challenge. Following the settings of Partial FC(1), we train the proposed UniTSFace on the WebFace4M and WebFace42M datasets using two different architectures, i.e., ResNet-50 and ResNet-200. We can observe that, on average, our performance is better than that of the Partial FC. Till the submission of this work (May 17 '23), the proposed UniTSFace ranks first place on the academic track of the MFR-ongoing leaderboard.

6 Discussion and Conclusion

Discussion.

1) Though the threshold range of USS is narrowed compared to other losses in Fig. 1, is it correct to claim the word 'unified'? Firstly, our theoretical objective is to learn a unified threshold that satisfies Eq.(3) during the training of a facial model. Therefore, we first assume the existence of such a unified threshold, and then propose the unified threshold integrated sample-to-sample (USS) loss. We have proven through our analysis in Section 3.1 that, ideally, a model trained by USS could learn a unified threshold for the training dataset.

However, we must admit that achieving this ideal goal is subjective to the model capacity, training hyper-parameters, and even the training dataset itself, which are all independent from our USS loss. For example, if the backbone network only uses one single linear neural layer, our USS loss definitely cannot guarantee a unified threshold either. In Fig. 1, using the same backbone architecture and training hyper-parameters, i) our USS loss is able to achieve a more compact threshold distribution than the other losses, moreover, and ii) the learned threshold denoted by the blue dashed line lies around the median of the boxplot, these two observations suggest the superiority of imposing the unified threshold and are consistent with our expectations.

2) In the testing stage, how to determine the threshold? The threshold learned by the training stage cannot be directly used in testing. In the testing stage, the threshold is determined according to the specific testing criteria. For example, when reporting the 1:1 verification accuracy on LFW, CFP-FP, AgeDB, 10-fold validation is used. We first select the threshold that achieves the highest accuracy in the first 9 folds and then adopt this threshold to calculate the accuracy in the leave-out fold.

3) How does UniTSFace compare to other methods such as CosFace in terms of computational efficiency and memory usage? UniTSFace utilizes the ResNet architecture as its backbone and optimizes the parameters using the algorithmic average of the cosine-margin Softmax loss and the proposed USS loss. When compared to CosFace, the extra computational cost brought by USS loss is relatively small, as the computational consumption and memory usage mostly depend on the convolutional operations inherent to the selected network architecture and the resolutions of the input images. In experiments, we indeed found that the computational differences between these methods during both the testing and training stages are negligible.

Conclusion.

We propose the USS loss for deep face recognition by explicitly defining a unified threshold that separates all positive sample-to-sample pairs from negative ones. Though this unified threshold learned in the training stage cannot be directly applied to the testing stage, the model trained by USS is desired to extract more discriminative features and subsequently improve the face recognition performance in various testing scenarios, which have been demonstrated with extensive experiments.

Furthermore, the proposed USS loss can be effortlessly extended to the Marginal USS loss and can also be seamlessly combined with other sample-to-class losses. In our experiments, we have combined the USS loss with ArcFace and CosFace methods, and the combined approaches consistently surpass their respective individual counterparts. We denote the fusion of the CosFace and USS losses as 'UniTSFace' which we have compared with other sophisticated combinations such as VPL, UNPG, and AnchorFace. The experimental results on multiple benchmark datasets further suggest the superiority of our UniTSFace.

In conclusion, we believe our USS loss provides the research community with a much more versatile and effective solution for face recognition tasks.

References

Xiang An, Jiankang Deng, Jia Guo, Ziyong Feng, XuHan Zhu, Jing Yang, and Tongliang Liu. Killing two birds with one stone: Efficient and robust training of face recognition cnns by partial fc. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4042-4051, 2022.

Xiang An, Xuhan Zhu, Yuan Gao, Yang Xiao, Yongle Zhao, Ziyong Feng, Lan Wu, Bin Qin, Ming Zhang, Debing Zhang, et al. Partial fc: Training 10 million identities on a single machine. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1445-1449, 2021.

Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang. Beyond triplet loss: a deep quadruplet network for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 403-412, 2017.

Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) , volume 1, pages 539-546. IEEE, 2005.

Jiankang Deng, Jia Guo, Xiang An, Zheng Zhu, and Stefanos Zafeiriou. Masked face recognition challenge: The insightface track report. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1437-1444, 2021.

Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5203-5212, 2020.

Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4690-4699, 2019.

Jiankang Deng, Jia Guo, Jing Yang, Alexandros Lattas, and Stefanos Zafeiriou. Variational prototype learning for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11906-11915, 2021.

Yueqi Duan, Jiwen Lu, and Jie Zhou. Uniformface: Learning deep equidistributed representation for face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) , volume 2, pages 1735-1742. IEEE, 2006.

Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in'RealLife'Images: detection, alignment, and recognition , 2008.

Yuge Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricularface: adaptive curriculum learning loss for deep face recognition. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5901-5910, 2020.

Junuk Jung, Seonhoon Lee, Heung-Seon Oh, Yongjun Park, Joochan Park, and Sungbin Son. Unified negative pair generation toward well-discriminative feature space for face recognition. arXiv preprint arXiv:2203.11593 , 2022.

Ira Kemelmacher-Shlizerman, Steven M Seitz, Daniel Miller, and Evan Brossard. The megaface benchmark: 1 million faces for recognition at scale. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4873-4882, 2016.

Jiaheng Liu, Haoyu Qin, Yichao Wu, and Ding Liang. Anchorface: Boosting tar@ far for practical face recognition. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 1711-1719, 2022.

Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. Advances in neural information processing systems , 31, 2018.

Weiyang Liu, Yandong Wen, Bhiksha Raj, Rita Singh, and Adrian Weller. Sphereface revived: Unifying hyperspherical face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 212-220, 2017.

Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In ICML , volume 2, page 7, 2016.

Brianna Maze, Jocelyn Adams, James A Duncan, Nathan Kalka, Tim Miller, Charles Otto, Anil K Jain, WTyler Niggel, Janet Anderson, Jordan Cheney, et al. Iarpa janus benchmark-c: Face dataset and protocol. In 2018 international conference on biometrics (ICB) , pages 158-165. IEEE, 2018.

Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 51-59, 2017.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 815-823, 2015.

Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE winter conference on applications of computer vision (WACV) , pages 1-9. IEEE, 2016.

Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems , 29, 2016.

Yi Sun, Yuheng Chen, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation by joint identification-verification. Advances in neural information processing systems , 27, 2014.

Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6398-6407, 2020.

Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation from predicting 10,000 classes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1891-1898, 2014.

Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to humanlevel performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1701-1708, 2014.

Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international conference on Multimedia , pages 1041-1049, 2017.

Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5265-5274, 2018.

Yandong Wen, Weiyang Liu, Adrian Weller, Bhiksha Raj, and Rita Singh. Sphereface2: Binary classification is all you need for deep face recognition. In International Conference on Learning Representations , 2022.

Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European conference on computer vision , pages 499-515. Springer, 2016.

Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. Learning face representation from scratch. arXiv preprint arXiv:1411.7923 , 2014.

Kai Zhao, Jingyi Xu, and Ming-Ming Cheng. Regularface: Deep face recognition via exclusive regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Jiwen Lu, Dalong Du, et al. Webface260m: A benchmark unveiling the power of million-scale deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10492-10502, 2021.

A Inequalities about L naive

We here present the detailed derivations for the inequalities about the naive loss L naive . For any sample X ( i ) captured from subject i , with feature x ( i ) = M ( X ( i ) ) , the naive loss L naive comprises of one positive sample-to-sample similarity and N -1 negative similarities,

̸

̸

where x ( i ) ∗ = M ( X ( i ) ∗ ) , x ( j ) ∗ = M ( X ( j ) ∗ ) , and X ( i ) ∗ and X ( j ) ∗ are randomly taken from the subject i, j , with j = i .

Using the inequality of arithmetic and geometric means, i.e., n √∏ n i =1 a i ≤ 1 n ∑ n i =1 a i for a i ≥ 0 , we derive four inequalities about L naive .

̸

̸

̸

L naive ( X ( i ) ) = -γg ( x ( i ) , x ( i ) ∗ ) + 1 N -1 N ∑ j =1 j = i γg ( x ( i ) , x ( j ) ∗ )

̸

̸

̸

Appendix for USS

=

̸

(

i

∗

N

1

-

γg

x

i

̸

̸

̸

) +

L

naive

-

(

γg

X

(

x

(

(

i

i

)

)

̸

N

∑

j

=1

j

̸

=

i

̸

̸

̸

̸

̸

̸

̸

and

)

,

x

)

1

(

(

)

,

x

(

j

∗

)

)

̸

̸

̸

̸

̸

̸

̸

̸

B Marginal Sample-to-Sample Based Losses

We have derived three sample-to-sample based losses in the manuscript, i.e., USS loss, sample-tosample based softmax, and BCE losses. We hereby present their respective marginal versions:

̸

̸

The experimental evaluations of such marginal losses have been included in Sec. 4.3 of the manuscript.

C Training Details

In our work, we choose the cosine function to represent the similarity of two features, i.e.,

Following (30; 7), we use customized ResNets (such as ResNet-50, ResNet-100, and ResNet-200) as our backbone networks. We implement all models using Pytorch and train them using the SGD optimizer with a weight decay of 5e-4 and momentum of 0.9. We use γ = 64 for L uss , L soft, and L bce in all experiments. Note that, we use a combination of CosFace( m = 0 . 4 ) and our USS( m = 0 . 1 ) as UniTSFace in Tables 4 and 5.

For the face models (using ResNet-50 as the backbone) on CASIA-WebFace, we train them over 28 epochs with a batch size of 512. The learning rate starts at 0.1 and is reduced by a factor of 10 at the 16 th and 24 th epoch. All models in ablation and parameter study were trained on CASIA-WebFace.

In Table 1, the margin m of L soft-m , L bce-m and L uss-m are set to be 0.1. In Table 3, the margin m of ArcFace and CosFace are set to be 0.5 and 0.4 respectively. The UniTSFace under the 'Small' protocol of MegaFace Challenge 1 in Table 4 and the models re-implemented in MFR Ongoing (the first ten rows in Table 5) were also trained on CASIA-WebFace.

For Glint360K, we train the models(ResNet-100) for 20 epochs using a batch size of 1024. Initially, the learning rate was set at 0.1, and a polynomial decay strategy (power=2) was applied to the learning rate schedule. The UniTSFace under the 'Large' protocol of MegaFace Challenge 1 (as shown in Table 4) was trained on Glint360K.

For WebFace4M, we train the models(ResNet-50) for 20 epochs using a batch size of 1024. The learning rate was initially set at 0.1, while a polynomial decay strategy (power=2) was applied to the learning rate schedule. The UniTSFace at the 12 th row of Table 5 was trained on WebFace4M.

In the case of WebFace42M, we train the models(ResNet-200) for 20 epochs, using a larger batch size of 4096. The learning rate linearly warmed up from 0 to 0.4 during the first epoch, followed by a polynomial decay (power=2) for the remaining 19 epochs. The UniTSFace at the last row of Table 5 was trained on WebFace42M.

D Megaface Challenge 1

We notice that the official MegaFace challenge website has been decommissioned and MegaFace data are no longer being distributed. However, despite these changes, we can still utilize the previously released data and development kit to evaluate the performance of trained models. Specifically, we have adopted the MegaFace testsuite provided by InsightFace ( https://github.com/deepinsight/ insightface/tree/master/recognition/_evaluation_/megaface ), which also includes the official devkit.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nI thank the authors for interesting paper and the effort in expanding the field of face recognition. \nThe paper proposes a new loss function USS (and its variants) for face recognition. The motivation is to jointly learn a threshold that can be used during the verification process. The authors show the efficacy of UniTSFace when USS is used jointly with ArcFace or CosFace (margin based softmax loss) in various evaluation sets.\n\nSTRENGTHS:\n- The performance gain in various datasets using UniTSFace is clearly shown. \n- The derivation of USS loss starting from contrastive objective is interesting. \n- The motivation of trying to learn one threshold that can be used in all evaluation dataset is interesting.\n\nWEAKNESSES:\n- The authors do not show whether learning the threshold t is really necessary. The readers cannot distinguish whether the performance gain comes from making the threshold t, a learnable parameter or it comes from the addition of the supervised contrastive learning objective which is a loss function that boosts the performance when combined with margin based softmax losses [1].\n- The authors do not show whether the learned threshold is indeed better than the more common method which is to calculate the optimal threshold specific to the validation dataset. \n- Figure 1 seems to suggest that despite the L_uss loss, the variance in threshold remains therefore, it is better to calibrate different threshold for different datasets. \n\n[1] CoReFace: Sample-Guided Contrastive Regularization for Deep Face Recognition\n\nQUESTIONS:\n- It was not stated clearly in the text what unified refers to. If it is correct that the authors use the term unified because the learned threshold is used for all evaluation settings *(in contrast to the normal scenario where the threshold is computed for each dataset), then it would be good to say it in the beginning. \n- Why does the unified threshold satisfy the equation 3? There may not exist t that satisfies this for a given dataset. \n- How does UniTSFace compare with ArcFace + Supervised Contrasitve Loss (InfoNCE) and ArcFace + L_naive (which does not learn t). In such case, t is computed for each validation dataset as is customary for face recognition.\n\nLIMITATIONS:\nYes. The limitation is discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper proposes a unified threshold integrated sample-to-sample loss for face recognition, which addresses the limitations of existing methods in exploring the cross-sample relationship and setting a unified threshold. The proposed loss function achieves exceptional performance on multiple benchmark datasets.

STRENGTHS:
1) The paper introduces an approach called UniTSFace, which combines a unified threshold with a sample-to-sample loss for face recognition. This approach addresses the limitations of existing methods and proposes a new loss function that achieves exceptional performance on multiple benchmark datasets.
2) The experimental results of the paper demonstrate significant improvement, especially compared to the baseline, on highly discriminative datasets such as MFR.

WEAKNESSES:
1) It is unnecessary to use complex symbolic formulas to express simple concepts, such as in lines 105 to 124.
2) Adding a sample-to-sample loss in the field of face recognition is not novel enough.
3) The training method of using a fixed threshold may appear to be consistent with testing, but it actually conflicts with it. In testing, different false acceptance rates (FAR) are used on different scenarios, meaning that for the same model, different thresholds will be used to determine the predicted results.

QUESTIONS:
Please refer to the Weaknesses section.

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes UniTSFace, a new approach to face recognition that uses a unified threshold integrated sample-to-sample loss (USS loss). The USS loss features a unified threshold for distinguishing positive from negative pairs and can be enhanced with an auxiliary margin. The authors show that the USS loss can be integrated with sample-to-class based losses and evaluate its effectiveness on various benchmark datasets, demonstrating its suitability for real-world applications. The contributions of this work include introducing the USS loss, demonstrating that it can learn a unified threshold, and showing that it can be enhanced with an auxiliary margin and is compatible with existing sample-to-class based losses.\n\nIn addition, UniTSFace outperforms state-of-the-art methods on multiple benchmark datasets, including the Megaface 1 dataset, and the proposed approach is evaluated on the large-scale WebFace42M dataset, which contains 42.5 million images of 2 million identities, demonstrating its efficacy for real-world applications.\n\nSTRENGTHS:\n1. The advantages of the USS loss over other loss functions are numerous. First, it is highly efficient and can work seamlessly with sample-to-class-based losses. Second, it overcomes the limitations of previous sample-to-sample losses by explicitly incorporating a learnable threshold that separates positive and negative pairs. Third, it achieves state-of-the-art performance on multiple benchmark datasets, demonstrating its effectiveness in real-world face recognition applications.\n2. The experiments in this paper are solid and have been conducted on the largest-scale dataset WebFace42M, achieving the state-of-the-art performance on the most challenging benchmark dataset MFR in the academic track, and the comparisons are fair.\n3. The proposed method is original\n4. There are no significant flaws in the experiments conducted in this work.\n\nWEAKNESSES:\nThe paper does not explicitly mention any weaknesses or limitations of the proposed UniTSFace approach or the USS loss function.\n\nQUESTIONS:\n1. The paper mentions that the unified threshold is limited to a sample-to-sample loss. Have you considered extending the unified threshold to sample-to-class loss, and if so, what are the potential benefits and challenges of doing so?\n2. The paper mentions that UniTSFace achieves state-of-the-art performance on multiple benchmark datasets. However, how does UniTSFace compare to other state-of-the-art methods in terms of computational efficiency and memory usage?\n\nLIMITATIONS:\nThe paper does not explicitly mention any limitations", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an interesting idea by focusing on the unified threshold of distinguishing positive from negative pairs in face recognition. Although face recognition has undergo a big step towards real application driven by deep learning, this submission has some new proposal for sample-to-sample based loss with unified threshold. Experiments on typical facial image datasets show the effectiveness.\n\n\n===============\nAfter reading the authors' rebuttals to my concerns and dicussions, I upgrade my rating as \"weak accept\".\n\nSTRENGTHS:\n+It is interesting to focus on the threshold for distinguishing positive from negative pairs, which is paid less attention.\n\n+The derivation of the USS loss makes sense by defining the  upperbound of the naive loss.\n\n+Experiments by deploying USS into ArcFace and CosFace show the effectiveness of the proposed USS. The threshold range is narrowed, which shows the certainty of threshold is improved.\n\nWEAKNESSES:\n-From Table 1, it shows that BCE loss is much worse than USS loss. This seems strange and I concern if the experiment is wrong for this loss. The reasons should be discussed.\n\n-From Fig.1, the threshold range is narrowed compared to other losses, but I concern if it is correct to claim the word \"unified\".\n\n-Face recognition models are explosive increased with deep learning. A number of advanced methods are developed. The necessity of this submission can be discussed in the paper. The contribution of the proposed method to the community can be discussed by combining with previous advanced face recognition models.\n\nQUESTIONS:\nIn inference stage, how to determine the threshold? pre-computed? It is better to clarify this point because the title is unified theshold, which gives me the first expression about the pre-trained threshold.\n\nLIMITATIONS:\nNot observed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}