UniTSFace: Unified Threshold Integrated Sample-to-Sample Loss for Face Recognition
Authors: Qiufu Li
Xi Jia
Jiancan Zhou
Linlin Shen
Jinming Duan
Venue: NeurIPS 2023 poster (2023)
Keywords: Face Recognition
Unified Threshold
USS Loss

Abstract: Sample-to-class-based face recognition models can not fully explore the cross-sample relationship among large amounts of facial images, while sample-to-sample-based models require sophisticated pairing processes for training. Furthermore, neither method satisfies the requirements of real-world face verification applications, which expect a unified threshold separating positive from negative facial pairs. In this paper, we propose a unified threshold integrated sample-to-sample based loss (USS loss), which features an explicit unified threshold for distinguishing positive from negative pairs. Inspired by our USS loss, we also derive the sample-to-sample based softmax and BCE losses, and discuss their relationship. Extensive evaluation on multiple benchmark datasets, including MFR, IJB-C, LFW, CFP-FP, AgeDB, and MegaFace, demonstrates that the proposed USS loss is highly efficient and can work seamlessly with sample-to-class-based losses. The embedded loss (USS and sample-to-class Softmax loss) overcomes the pitfalls of previous approaches and the trained facial model UniTSFace exhibits exceptional performance, outperforming state-of-the-art methods, such as CosFace, ArcFace, VPL, AnchorFace, and UNPG. Our code is available at https://github.com/CVI-SZU/UniTSFace.

UniTSFace: Unified Threshold Integrated Sample-to-Sample Loss for Face Recognition
Qiufu Li 1,2.6,# liqiufulszu. edu. cn
Xi Jia 1,2,3,
Jiancan Zhou 1.2.4.# zhoujiancan@r oxmail. com
x. jia. ..a.100 10cs. bham ac uk
Linlin Shen 1,2,6, 11shen@szu. eetrezzuedd..co cn
Jinming Duan 3,5 j duan@bham. ac. uk
National Enginecring Laboratory for Big Data System Computing Technology, Shenzhen University, China
Computer Vision Institute, Shenzhen University, China School of Computer Science, University of Birmingham, UK Aqara, Lumi United Technology Co., Ltd, China Alan Turing Institute, UK SZU Branch, Shenzhen Institute of Artificial Intelligence and Robotics for Society, China
Abstract
Sample-to-clant toape--t--llss-bbased face recognition models can not fully explore the cross- sample relationship among large amounts of facial images, while sample-to-sappll based models require sophisticated pairing processes for training. Furthermore, neither method satisfies the requirements of real-world face verification applica- tions, which expect unified threshold separating positive from negative facial pairs. In this paper, we propose unified threshold integrated sample-to-same tample based loss (USS loss), which features an explicit unified threshold for distinguishing positive from negative pairs. Inspired by our USS loss, also derive the sumple-to-sampe based softmax and BCE losses, and discuss their relationship. Extensive evaluation multiple benchmark datasets, including MFR, IJBC,, LFW, CFP-FP, AgeDB, and MegaFace, demonstrates that the proposed USS loss is highly efficient and can work seamlessly with sample-to-class toaape-----css--bseeee losses. The embedded loss (USS and sample-to-class Softmax loss) overcomes the pitfalls of previous approaches and the trained facial model UniTSFace exhibits exceptional performance, outperform- ing state-of-the-art methods, such as CosFace, ArcFace, VPL, AnchorFace, and UNPG. Our code is available lattps://githu:: Sttp//VVIISUUUUISSSII/S
Introduction
Modern deep facial recognition systems, involving an enormous number of facial images and identities, essentially rely on discriminative feature learning: the facial images from the same identity should be close while those from different identities should be distant in the feature space. That is, the similarity of a positive pair (two facial images from the same identity) is required to be larger than any negative pair (two facial images from different identities). In other words, unified threshold is expected to distinguish positive from negative pairs.
The feature learning process of general deep face recognition models is either based sample-to- class losses (such as Softmax loss (27 or sample-to-sampe losses (such as contrastive 4 10 25
# Equal contribution: "Corresponding author. Parts of work were done when X. and were students at Shenzhen University.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

and triplet loss (22)). Inspired by the success of large-scale scale image classification, the softmax loss and its extensions have become popular in deep face recognition systems. In the softmax loss, each weight vector can be regarded as proxy of the corresponding class (face identity), the classification-bai based face recognition models are demanded to learn the class proxy and image features simultaneously. However, these models possess a significant drawback, namely a domain gap between the training and testing stages of face models. This gap arises from the reliance on limited identity proxies for computing and optimizing feature similarity in sample-to-class models. Conversely, in real-world scenarios, feature similarity is computed and compared across various samples from diverse facial identities. In other words, the topple--t-cllss-bbee classification strategy may not entirely explore the variances across samples (Problem 1). Therefore, its efficacy in accurately reflecting real-world scenarios is questionable. To tackle this drawback, VPL (8) considers small variation around the class proxy, which implicitly introduces more samples during the optimization and improves the face recognition performance. However, while this small variation extends the capability of softmax loss, it cannot essentially represent all the samples in this class.
Face models based on sample-to- tamplee losses 4 10 22 24) learn facial identity features and optimize feature similarities by comparing positive and negative samples, which is closer to real- face recognition applications than sample-to-class losses. The majority of sample-to-sample losses aim to maximize inter-class discrepancy while minimizing intra-class distances, which may need a meticulous sampling/paring step for every batch Moreover, in face verification tasks, single threshold is required to distinguish positive facial pairs from negative ones. Unfortunately, none of the aforementioned methods incorporate such explicit constraint (Problem 2).
In this study, we commence our research by analyzing a reasonable, albeit naive, sample-to-sample loss, which enables us to straightforwardly investigate the variances across facial samples, thereby resolving Problem To address Problem 2, we integrate a unified learnable threshold into the naive loss, resulting in novel sample-to-sampl loss function that we term the unified threshold integrated sample-to-sample (USS) loss. We provide a mathematical elaboration of the relationship between our USS loss and the sample-to-sampl BCE loss and softmax loss. from the perspective of the naive sumpl-to-ssappl loss. To evaluate the effectiveness of the USS loss, we conduct experiments on various benchmark datasets and qualitatively demonstrate that it meets the requirements of real-world applications. Additionally, we find that incorporating a margin can easily enhance the USS loss. Our USS loss also works seamlessly with the sample-to-class based losses and shows significant improvements when they are combined together to train a face model called FaCeS To summarize, the contributions of this work are
We introduce a unified threshold integrated caaple-to-samplee loss (USS) derived from a reasonable naive loss for face recognition. We also derive the sample soplle-to-sane to-sample based soft- max and binary-cross-entrpp (BCE) losses from the naive loss and reveal the mathematical relationship among the softmax, BCE, and our USS losses.
We demonstrate that a unified threshold can be learned by adopting the proposed USS loss and quantitatively and qualitatively demonstrate that the learned threshold aligns with face verification expectations in experiments.
The proposed USS loss can be enhanced with an auxiliary margin and is compatible with existing sample-to-class based losses. We show that USS loss, when used jointly with sample- to-class based losses such as CosFace and ArcFace leads to a continuous improvement.
Our Unie outperforms state-of- of-te-of-the-ar methods on the Megaface dataset and ranks first place on the MFR ongoing challenge till the submission of this work (May 17 '23, academic track): httpp://ccv22 -mfr. com/ htte////eeeebbed/////e hww.eerrbbrrrraaaaaeici.a academic
2 Related Works
uses contrastive loss 10) to encourage the features learned from the same identity to be close while that learned from different identities are distant. (2ceNet(22) constructs three-element tuples and minimizes the distance between an anchor and a positive sample and maximizes the distance between the anchor and a negative sample. The effectiveness of contrastive/triplet losses relies on the meticulous selection of pairs/triplets. Furthermore, neither method explicitly enforces that all positive sample-to-sappe similarities are greater than negative similarities.
Sample-to-Sampll based Methods. DeepID2 25
2

---

Sample-to-Clasy based Methods. While Softmax loss is frequently utilized in deep recognition models, it only promotes separability and does not lear discriminative features. Various approaches have been proposed to enhance the feature learning of softmax loss. Some methods (32 p 34
16) proposed extra constraints imposed on the softmax loss, such methods normally have a class proxy (class center/prototype) and maximize (minimize) the similarity (distances) between facial feature and its corresponding class proxy (32), as well as maximize the distances between all class proxies(16: 9 (34). However, the training process of such methods needs careful balancing of the softmax loss and the extra constraints. Alternatively, some works directly improve the softmax by normalizing the facial features and adding margins between positive and negative sample-to-clas pairs [88 29: 30 Z (7). While sample-to-clas toeclsss methods demonstrate excellent performance in deep face verification, they may not entirely explore the variability across various facial samples.
Hybrid (/combined) Methods. Circle loss (26) is one of the first works that discussed the two elemental learning paradigms, i.e., learning with class-level labels and pair-wise labels, in a unified framework. However, the two learning paradigms are respectively used in circle loss. Following (26). the combination of the two learning paradigms to solve the shortcoming of either method become popular, e.g., (8L(8), AnchorFace([5), and (3) VPL extends the softmax loss by considering small feature variation around the class proxy W. The embedding of a small feature variation cannot essentially represent all samples, VPL still presents a large gap between training a face recognition model and testing over the open sets. UNPG jointly optimizes the distance between a sample I; with negative class proxies W, and negative samples z, in the same softmax loss using batch processing, focusing the negative pairs during training. AnchorFace, however, uses a combination of softmax loss and different comple-to-sample losses, such as TAR Loss and FAR Loss, target specific testing protocols. In contrast, our USS loss is more general and compatible with existing sample-to-clast losses. We demonstrate our USS loss can lead to a steady improvement when used with sample-to-class losses jointly in experiments.
3 Methods
Suppose M is a deep face model trained on a facial sample set D = U Di captured from N subjects, where D; denotes the subset containing the facial samples captured from the same subject i. Then, we can get feature set F - Fi U (x(*) M x Di). where (i)
is the feature vector of the sample ( and ( denotes the sample captured from subject i.
For any two samples X, X. E DD we apply bivariate operator g(z æ. E [-1, denoting their feature similarity, where x .M(X), xt _M(X) are features of X, X.. We term g(æ, x.) the positive sample-to-samp the negative sumple-to-samlle similarity if they are from different subjects. Then, for any sample ( E D., Vi, we define its positive and negative similarity sets as
similarity if the two samples are captured from the same subject, while
(pos) ()) { (0(x(()) x..)) x. E Fi).
8
(neg)
U {9lo000 xx.): : x. E FFi
(2)
(())
where (+) = (x ()).
In real applications of face verification, a threshold E [-1,1] is chosen to verify whether two samples X and X, are from the same subject or not. Specifically, the two facial samples X,X, X, are with the same identity if g(x,, ax.)) 2 i, while they are from two different identities when g x.) < i.
To be in line with face verification applications, during the training process of M, we expect unified threshold such that any two samples (X (1) E De and x!i) E EDi) conform to the above rules. they share the same identity (i.e.. i = j), then ((x(()) xz{i) t, in the case of + j. (() x(11 t.
where (x) M ()), (0( (()))), The unified threshold t satisfies
max U
(neg) x
< S min (Ü
U (D)D,
(pos) X
1=1 ED.
3

---

However, the existing sumple-o-samppe losses 10 22 24) fail to explicitly include and learn the unified threshold. In this paper, by explicitly defining the unified threshold t, we design unified threshold integrated sample-to-sample loss.
3.1 Unified Threshold Integraed Sample-to-Sample Loss (USS Loss)
Clearly, in the training of model M, expects large positive sample-to- -o--s similarities but small negative ones, and then, for any sample ((), naive loss could be reasonably defined as,
NAVVE X x0)
(1( .x) WEF.
Σ
x)
(g( (z) 2x)
F
F-
WEF,
where Y is scale factor, [S] denotes the element number of S, and F Fi denotes the complementary set of F; in J, i.e., F-
FjJ
The loss in Eq. 4 computes the feature similarities of all positive sample-to-sample pairs and all negative ones for the sample X which is not easily implemented in practice. In this paper, without loss of generality, we consider the feature similarities of one positive sample pair and N negative
sample pairs for the sample X in the naive loss,
N
L L X (1)) = - -- ,x) +
(g( ,.)))
(5)
where xr! - M(x!'), = (( and x!!! and x! are randomly taken from the subject i,j, with j # i.
Using the inequality of arithmetic and geometric means we derive inequalities about Lnaive
N exp ΣΣ
20(m .n.
Tnnave (x(9) S 2log (1
N-1
)
2 log 2,
(6)
(z1(()(1)) ar!")))
exp
N
e (a(())
2
Inaive (1))
log
2 log 2.
(7)
N
erg(ar())
According to Eqs.
66. and the unified threshold in Eq.
one can get
N
2 Lnnive
(x + N Nlog2
N $ log (1+ exp
1-1
N-1
N
log (1+ exp(1g((00) ar!
(8)
exp (2) ar!
exp(1g((((1) ar
11 < log ( exp
N-I
78
N
log (1+ exp(7g((0(1) 2!!i)))
(9)
exp (7g( x..1)
exp(7t)
log 20(0 ( )
+ log (1 r0(000 (0)0))
(10)
We define the Unified threshold integrated Cample-to-Sample (USS) loss Luss (uss((()) (x(1)) as
Luss (x = log (1 + 79 . 2 +
log (1 + (7)(2(() (2(0))=))
(11)
VIl ai 5 L for a;

---

where b = yt is constant to be learned (depicted in Fig. The detailed derivations of the above inequalities are described in the supplementary (appendix).
We here analyze that the unified threshold could be learned. Suppose that the model M has been perfectly trained, and Tuss has reached its minimum point after the training, which means (i) the
positive sample-to-sampe similarity g(x(1) ) tends to 1, and the negative ones ((x(f), x
( tends
to 1; and (ii) Luss reaches its stationary point in terms of variable b. From (īi), one can deduce that
ULyss bb
79( _ )46
N
e7g( (0) )
0
(12)
+ e (0(((1))))))))))))))))))
148
+ eng(((()) )
e 77
N 6
i
(13)
+ e --7
e-7-6
(N 2)e
(N 27 +4
= b = log
(14)
If N < +.3 the final learned threshold / = will locate between -1 and 1, which means now the leaned t. has correctly separated the negative sample-to-sampe similarities, g(x(()) . ) and the positive ones, g(x(), 2 In the practice, following (30) 77), we set 7 64, then, according the above analysis, the unified threshold t could be leaned if N < 1.9 x 1055.
2
3.2 Sample-to-Sample Based Softmax and BCE Losses
Sample-to-class based softmax and BCE losses are widely applied in the image classification. For the study of face verification, we here deduce the sample-to-sen sommpll based softmax and BCE losses from the naive loss Luaive.
Similar to the deduction of USS loss, using the inequality of arithmetic and geometric means, we first present another two inequalities about Lnaive.
N
e ( (.(())
N log N N
Luaive (X (1)
N log
(15)
Σド= eng((a(() 22
N
N
N
N
log 1+ (7 (x(
Luaive (1)
2N log 2.
(16)
N
Softmax Loss. softmax loss as
For sample X E Dj with x(1) M(X we define its sumple-to-sample
e (1)((())) ,m(
Lsoft (1) log
(17)
Σラ=1 e7
N
Then, according to Eq.
(15). one can get
N
N N - log N.
Lnaive ( N-
Lunn X (3)
(18)
Similar to the naive loss L the design of softmax loss Lsan does not consider the unified threshold among the cample-to-samplee pairs.
BCE Loss. For all samples captured from subject i, we assume the existence of threshold 4, which could separate their all positive sample-to-sample pairs and negative ones, i.e..
max
(neg) x t S min U
x
(pos)
A ii,
(19)
X
X
5

---

then, according to Eqs. (6] and 6
nnive [X (1)) + N22 log 2
2
log 11 exp Σ أسر 79(0<0) (0!!)
N
N
log (1+ exp()g(e(0) xx.i))
N-1
(20)
exp (1g(2((1) xx!")))
exp(gg(z()) ب. )(
N
11 1 (a())))
Σ log 1 e7 _00!) (=0(a(00_) )
log
(21)
We define BCE loss for sample (+) as
Locce (x = log (1 +
70(a0 an -
+ Σ log (1 > -b)))
(22)
where b, = 74 are parameters be learned. Note that the l can be different for different identities and therefore are not unified.
After the training of the model M, i.c., the threshold & hy were learned, then,
N
N Locel (x)) - 2N log
2
nnivee (X
(1)
(23)
N
3.3 Marginal Sample-to-Sample Based Losses
Our USS loss, as well as the deduced sample-to toople-to-ssmple based Loo and Live, only encourage the separability between positive and negative sample pairs. To further improve the discriminative ability of such losses, i.e., to encourage the positive features be distant from the negative ones, we further proposed the marginal extensions for such losses.
By introducing margin the feature similarities, we can have the marginal USS loss as:
Luss-m ( (x)
-log (1 + (((((()) (n())
m)+b) -
log (1 + (m
(24)
where the is the introduced hyper- bypee-paramettr the margin. Proper adjusting of such a parameter can improve the recognition performance, as discussed in Sec. 4.3] Similarly, we can extend the vanilla Lostt and Lice the marginal version of Leoft-m and Loce-m- the full equations are given the supplementary materials (and appendix).
4 Experiments
4.1 Datasets and Evaluations
Datasets. We utilize four publicly available datasets for training, namely,
(33)
(consisting of 0.5 million images of 10K identities), Glint360K(2) (comprising 17.1 million images of 360K identities). WebFace422 (containing 42.5 million images of 2 million identities), and WebFace4M., which subset WebFace42MM with 4.2 million images 0.2 million identities. For evaluating the face verification performance, the ICCV-2021 Masked Face Recognition Chal- lenge (MFR Ongoing) (5) adopted. The MFR ongoing testing protocol includes various popular benchmarks, such as LFW (1I. CFP-FP (23), AgeDB (21). and IJB-C (20). along with its own MFR benchmarks, such as the Mask, Children, and Globalized Multi-Racial (GMR) test sets. The Mask set comprises 13.9K positive pairs and 96.9 million negative pairs (including 6.9K masked images and

---

13.9K non masked images) of 6.9K identities. The Children set contains 157K images (totaling 1.7 million positive pairs and 24.7 billion negative pairs) of 14K identities. The Globalized Multi-Racial sets comprise 1.6 million images in total, consisting of 4.6 million positive pairs and 2.6 trillion negative pairs, and representing 242K identities across four races: African, Caucasian, South-Asian, and East-Asian. For face identification, the MegaFace Challenge 1 (14) is employed as the test set, which comprises a gallery set with over million images from 690K different identities, and a probe set with 3,530 images from 530 identities. It is worth noting that the MegalFace Challenge I also has a verification track, and the verification performance is included in the experiments.
Evaluation and Metrics. For the MFR Ongoing Challenge. the trained models are submitted to and evaluated by the online server. Specifically, we report 1:1 verification accuracy for LFW, CFP-FP, and AgeDB. We report True Accept Rate (TAR) at False Accept Rate (FAR) levels of le-4 and le-5 for IJB-C. We report TARs at FAR=le-4 for the Mask and Children test sets, and TARs at FAR=1e-6 for the GMR test sets. For the MegaFace Challenge 1, we report Rank1 accuracy for identification and TAR at FAR=1e-6 for verification.
4.2 Implementation Details
Preprocessing. Firstly, we aligned all face images using the 5 landmarks detected by RetinaFace (6) and cropped the center 112 112 patch. We then normalized the cropped images by first subtracting 127.5 and then dividing 128. Finally, we augmented the training images with horizontal flipping.
Training. We adopt customized ResNets as our backbone following 77). We implement all models using Pytorch and train them using the SGD optimizer with a weight decay of 5e-4 and momentum 0.9. For the face models on SSSIA--eFFace, we train them over 28 epochs with batch size of 512. The learning rate starts at 0.1 and is reduced by a factor of 10 at the 16th and 24th epoch. For both Glint360K and WebFace4M, we train the ResNets for 20 epochs using a batch size of 1024. The learning rate is initially set at 0.1, while polynomial decay strategy (power=2) is applied for the learning rate schedule. the case of WebFace42M, we train the ResNets for 20 epochs, using a larger batch size of 4096. The learning rate linearly warms up from 0 to 0.4 during the first epoch, followed by a polynomial decay (power=2) for the remaining 19 epochs. We include the detailed settings of all Syyper-paramete
used in Sec. 4 and Sec. in the appendix for further reference.
Testing. For a given facial image, we extract two 512-dimensional features from the original image and its horizontally flipped counterpart. These features are then combined together as the final representation with element-wise addition. To assess the similarity between two images, we utilize the cosine similarity metric.
4.3 Ablation and Parameter Study
Effectiveness of unified threshold. We first demonstrate the necessity of learning a unified threshold by comparing our proposed Luss with two other tompeleo-ssample losses, Tooht and Thces in Table [ We additionally report the performance of the model trained with T The first to third rows respectively show the results from Luaive- Lioot and Lose, while the fourth row lists the performance of our Luss. We observe that Loof performs 4.68% better than our proposed Luss on IJBC. However, our USS loss outperforms Lyot on the other four datasets, particularly MFR-All where it achieves 12.09% higher @ FAR@FFR=1C-6.. Compared to the Lice, the gains achieved by our Luss are more significant, with performance improvements of 29.52% MFR-AII, 33.01% in terms of TAR @ FAR=1e-4 19.69% on AgeDB. The Tinives however, is difficult to train and the trained model barely learns any useful features. We note that the results are not cherry-picked For fair comparison, we adopted the same experimental setting when we trained the model using the different losses. During our experiments, we found that the model is easy to converge using Lioht and Lusss and the training process is stable, which leads to more favorable results. The model, however, is difficult to train with Lnaive and Lose. We have dedicated many efforts to these two losses, but the convergence is still problematic.
Loss
MR-ALL
LJB-C
LFW
CFP
Age
Tenaive
0.0
0.35
50.0
50.0 50.0
T:
26.34
76.88
98.80
95.50 93.48
Lice
8.91
39.19
92.60
66.41 74.36
Lisss
38.43
72.20
99.40
96.51 94.05
T m
38.11
83.60
99.18
96.32 94.18
Tree-m
9.01
45.08
92.48
65.34 76.30
Liuss-m
42.55
83.92
99.46 96.81 94.28
Table 1: Ablation study of the proposed Taus-
IJBC, 30% on CFP-FP, and
7

---

When comparing the marginal softmax loss Li marginal BCE loss L with marginal USS loss Tuss-m (last three rows), we observe that the marginal USS loss Tuss.m achieves the highest performance across all five reported datasets, which further confirms the effective- ness of our proposed unified threshold strategy,
0.55
0.50
0.45
0.40
0.35
noteworthy that all the methods in this ta- ble are based on the same ResNet-50 network trained on the Sesee-eeFaaee dataset.
0.30
0.25
Qualitative study of unified threshold. To better understand the effects of unified thresh- old, in Fig. D we randomly select 20 identities (5,074 facial images) from the whole training dataset with N = 10572 subjects, and then con- struct 1 positive sample pair and N - negative pairs for each of the selected images. For each of the 20 identities, we compute the optimal thresh- old to separate the positive pair and the hardest negative pair, i... the most similar negative pair. We respectively plot threshold distributions of the 20 identities for Townn Trcee, and Tauss In Fig. 1 each dot denotes the optimal threshold for each identity. The blue dashed line the unified threshold t 31.3344/66
L
Lice
Luss
Figure 1: Threshold distributions of Lsoft, Loee and Luss on 20 random selected identities in CASIA- WebFace. Each dot denotes the optimal threshold for each identity. The dashed blue line represents the threshold that was learned solely by Luss with == b 31.3344/64 0.4896 in Eq. (1).
0.4896 exclusively learned by our Luss in Eq. 41). From this figure, we can clearly observe that our Luss has the most compact threshold distribution, while the threshold distributions from Lice and Lon are relatively loose. Most importantly, for Lusss the median threshold from the randomly sampled 20 identities is in line with the learned unified threshold 0.4896 from the face model trained with Tu which proves the effects of including an explicit unified threshold.
Impact of Margins. The results in Table | (with m = 0.1) demonstrate that margin improves the performance of Luss However, the introduced margin m remains a hyperparameter that needs to be tuned. In Table 2 we investigate the effects of using different margins in Luss-m- Specifically, we experiment with four more settings that increase m to 0.2, 0.3, 0.4, and 0.5. We observe that i) the highest performance on MR-ALL, IJB C, LFT, CFP-FP, and AgeDB are respectively achieved when m equals 0.2, 0.4, 0.1, 1.1, and 0.3: ii) the performance of the four marginal Luss-m losses are all higher than that of the original Luss (i.c., m 0 in Lus--m). Although the auxiliary margin improves the overall performance, the performance seems to saturate when m is set to 0.4. Note that in our later experiments, we use m 0.1.
Margin
MR-ALL
UB-C LFW CFP
Age
MR-ALL
JBBC LFW CFP
Age
m 0.0
38.43
72.20
99.4 40 96.51 94.05
USS
42.55
83.92 99.46 96.81 94.28
m 0.1
42.55
83.92
99.46 96.81 94.28
ArcFace
42.21
48.49 99.31 97.07 94.5T
m = 0.2
44.74
87.65
99.26 96.65 94.16
UrcFace+USS
48.92
89.56
99.40 97.22 95.20
0.3
44.14
87.58
99.30 96.41 94.38
CosFace
45.12
56.65
99.36 97.30 94.98
0.4
43.38
87.82
26 96.04 94.23
CosFace+USS
50.28
89.84
99.41 97.35 95.13
m 0.5
43.91
87.62
99.13 96.05 94.23
Table 3: Combination of USS and two sample-to- class based methods.
Table 2: Parameter study of margin in Luss-m-
Compatibility. In Table and Table 2 we have shown that the proposed Luss is superior to other cample-to-smppl based losses and can be boosted by adding a proper margin. In Table 3 we investigate the effects of combining Luss with two marginal softmax losses, i.c., ArcFace and CosFace. We find that the model trained with the combined loss significantly outperforms the original ArcFace and CosFace models, as well as our USS loss. For example, the model trained with ArcFace + USS respectively outperforms ArcFace and USS with 6.71% and 6.37% gains on MR-ALL, and 41.07% and 5.64% gains on IJB-C. The model trained with CosFace + USS respectively outperforms CosFace and USS with 16% and 7.73% gains on -R-ALL, and 33.19% and 5.92% gains on IJB-C. Such significant improvements suggest the compatibility of our USS loss. In this work, we refer to the model trained using a combination of a sample-to-elass method and our USS as FnceeFace. For our experiments, choose to use CosFace the sample-to-clas -laas method.
8

---

Method
P
R
Iden.
Veri.
Method
P
R
Iden.
Veri.
Softmax
(8)
S
54.85
65.92
ArcFace
L
X
81.03
96.98
Triplet Loss [8 (2)
S
64.79
78.32
CurricularFace 12
L X
81.26
97.26
Contrastive 18 25
S
X
65.21
78.86
CosFace (30)
L X 82.72
96.65
Center (18] 32
S
X 63.49
80.14
UniTSFace
L
X
85.01
97.85
L-Softmax [8 19
S
X
67.12
80.42
SphereFace2 (31)
L
89.84
91.94
SphereFace (8)
S
72.72
85.56
CosFace
L
V
97.91
97.91
SphereFace+ (16
S
X 73.03
SphereFace 18
L
98.16
98.46
CosFace (30)
S
X 77.11
89.88
ArcFace 7
L
V
98.35
98.48
ArcFace 7
S
X
77.50
92.34
Circle Loss (26
L V
98.50
98.73
CurricularFace 12)
S
77.65
92.91
CurricularFace 12
L V
98.71
98.64
UniTSFace
S
77.41
93.50
VPL (8)
L V
98,80
98.97
ArcFace (
S
/
91.75
93.69
Partial FC (2)
L V
98,94
99.10
CurricularFace 12)
S
92.48
94.55
UNPG (13)
L V
99.27
UniTSFace
S
92.36
94.84
UnTTSFace
I. 99.27
99.19
Table 4: Comparisons between different methods the MegaFace Challenge 1. The letter P indicates the 'Small' or 'Large' protocols, R denotes whether the label refinement is used. All reported results, with the exception of our own, were directly taken from their respective papers.
Net. Data.
MFR
LJB-C
Verification Acc.
Method
Mask Child. Afri Cau. S-A.
E-A.
MR-AII
lc-4 le-5
LFW CFP Age
Contrastive(4)
6.67 10.58 12.40 18.84 13.57
10.38
12.38
58.47
46.75
95.50 74.65 82.28
(N+1)-Tuplet (24)
26.56 28.23 39.16 50.92 47.71
24.06
38.11
83.60
73.93
99.18 96.32 94.18
ArcFace
38.52 31.42 45.87 63.69 59.85
7.66
42.21
9.18 48.49
99.31
97.07 94.51
CosFace 300
38.79 31.33 48.06 63.56 58.71
15.08
45.12
11.30 56.65
99.36 97.30 94.98
Sphere-Rv R
R50
32.80 28.09 40.24 57.24 50.38 22.30
39.92
75.81 86.35
99.38 96.95 94.48
SphereFace2211
CASIA
35.40 30.55 46.65 62.69 56.23 26.65
44.20
88.41 79.18
99.46 97.42 94.96
VPL
33.86 31.39 46.52 59.93 54.07 27.18
47.02
88.44 81.38
99.30
97.07 94.75
AnchorFace
37.04 32.28 49.60
63.17 59.80 28.88
48.44
88.81 77.82
99.56
97.48 95.18
33)
38.62
33.24
49.94
63,85 59.60 29.21
48.66
88.17 77.73
99.45
97.25 94.83
UnieSFace
37.98 31.73
51.45
64.89 59.73 29.56
50.28
89.84 82.64
99.41 97.35 95.13
Partial FC 1
R50
72.28
84.86
91.57 88.57 67.52
86.85
SSace
WF4M
75.93
72.00
88.17
93.68 91.40 70.55
89.65
97.03
95.18
99.80
99.04 97.93
Partial FC (I)
R200
91.87 .7
97.79
98.70 98.54 89.52
97.70
97.97
96.93
99.83
99.51 98.70 70
SniTSFace
WF42M
92.87 93.51
98.35 99.03 98.99 90.76
98.16
97.99 97.00
99.83 99.47 98.71
Table 5: Comparisons between different methods on MFR-Ongoing.
Comparison with the State-OffThe-Art
MegaFace Challenge 1. In Table compare both the identification and verification performance of our UniTSFace with several oftte-of-thh-an methods on MegaFace Challenge 1. These methods include sample-to-sample based Contrastive Loss (4 10) and Triplet Loss (22), sample-to-class based CosFace 30) and (r)Face as well as the hybrid/combined approaches: VPL and (3).
To ensure fair comparison, as per the official protocols, we compare our UniTSFace trained on the CASIA-WebFasd with the models trained on 'Small' datasets, while UniTSFace trained on Glint360K is compared with the models trained on 'Large' datasets in Table 4 When label refinement 7 is not used, our UniTSFace achieves the highest accuracy among the compared models trained 'Large' datasets, with 85.01% % identification accuracy and a 97. 97.85% verification accuracy. When label refinement (7) is used, the accuracies can be further increased to 99.27% and 99.19% respectively. Though our Unce is slightly lower than CurricularFace (12 in terms of identification performance on 'Small' datasets, Unce achieves the highest accuracy on the verification track, regardless of whether label refinement is used not. These results demonstrate the effectiveness of our UnieSFace in both identification and verification tasks.
MFR Ongoing Benchmarks. We then compare the proposed UniTSFace with Contrastive Loss(4) 10), IN+1)-Tuplet (2s)(24), CosFace(30), ArcFace(Z). VPL(8), AnchorFace(15), and UNPG(13) on the MFR Ongoing benchmark. We re-implement these methods with the optimal hyper-parameters recommended in their original papers. All the compared models are trained with ResNet-50 backbone and the CASIA- WebFace dataset. As reported in Table our UFae achieves clear improvement over other methods.

---

Additionally, we compare our Unie with the recent Partial FC which is the leading method on the MFR ongoing challenge. Following the settings of Partial FC(I, we train the proposed UniTSFace on the WebFace4M and WebFace42M datasets using two different architectures, i.e., ResNet-50 and ResNet-200. We can observe that, on average, our performance is better than that of the Partial FC. Till the submission of this work (May 17 '23), the proposed UniT ranks first place on the academic track of the MFR-ongoing leaderboard.
Discussion and Conclusion
Discussion.
1) Though the threshold range of USS is narrowed compared other losses in Fig. is it correct to claim the word "unified"? Firstly, our theoretical objective is to learn a unified threshold that satisfies Eq. during the training of facial model. Therefore, we first assume the existence of such unified threshold, and then propose the unified threshold integrated sample-to-sappe (USS) loss. We have proven through our analysis in Section 3.1 that, ideally, a model trained by USS could learn a unified threshold for the training dataset.
However, must admit that achieving this ideal goal is subjective to the model capacity, training prrer-parameters and even the training dataset itself, which are all independent from our USS loss. For example, if the backbone network only uses one single linear neural layer, our USS loss definitely cannot guarantee a unified threshold either. In Fig. [ using the same backbone architecture and training parammeaa,,t,sss our USS loss is able achieve more compact threshold distribution than the other losses, moreover, and ii) the learned threshold denoted by the blue dashed line lies around the median of the boxplot, these two observations suggest the superiority of imposing the unified threshold and are consistent with our expectations.
2) In the testing stage, how to determine the threshold? The threshold learned by the training stage cannot be directly used in testing. In the testing stage, the threshold is determined according to the specific testing criteria. For example, when reporting the 1:1 verification accuracy LFW, CFP-FFI FP, AgeDB, 10-fold validation is used. We first select the threshold that achieves the highest accuracy in the first 9 folds and then adopt this threshold to calculate the accuracy in the leave-out fold.
3) How does UniTSFace compare to other methods such as CosFace in terms of computational efficiency and memory usage? UniTSFace utilizes the ResNet architecture as its backbone and optimizes the parameters using the algorithmic average of the margin Softmax loss and the proposed USS loss. When compared to CosFace, the extra computational cost brought by USS loss relatively small, as the computational consumption and memory usage mostly depend on the convolutional operations inherent to the selected network architecture and the resolutions of the input images. In experiments, we indeed found that the computational differences between these methods during both the testing and training stages are negligible. Conclusion.
We propose the USS loss for deep face recognition by explicitly defining a unified threshold that separates all positive sample-to-sampe learned in the training stage cannot be directly applied to the testing stage, the model trained by USS is desired to extract more discriminative features and subsequently improve the face recognition performance in various testing scenarios, which have been demonstrated with extensive experiments.
pairs from negative ones. Though this unified threshold
Furthermore, the proposed USS loss can be effortlessly extended to the Marginal USS loss and can also be seamlessly combined with other sample-to-class losses. In our experiments, we have combined the USS loss with ArcFace and CosFace methods, and the combined approaches consistently surpass their respective individual counterparts. We denote the fusion of the CosFace and USS losses as "UniTSFace" which we have compared with other sophisticated combinations such VPL, UNPG, and AnchorFace. The experimental results on multiple benchmark datasets further suggest the superiority of our UniTSFace.
In conclusion, we believe our USS loss provides the research community with a much more versatile and effective solution for face recognition tasks.
10

---

Acknowledgement
This work was supported by the National Natural Science Foundation of China under Grants 82261138623 and 62006156, Guangdong Basic and Applied Basic Research Foundation under Grants 1023A151501000 0688 and 2022A1515012125, and Shenzhen Municipal Science and Technology Innovation Council under Grant JCYI2022053 10141200300
References
[1] Xiang An. Jiankang Deng, Jia Guo, Ziyong Feng. XuHan Zhu, Jing Yang, and Tongliang Liu. Killing two birds with one stone: Efficient and robust training of face recognition cnns by partial fc. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4042-4051, 2022.
2022. [2] Xiang An, Xuhan Zhu, Yuan Gao, Yang Xiao, Yongle Zhao, Ziyong Feng, Lan Wu, Bin Qin, Ming Zhang, Debing Zhang. et al. Partial fe: Training 10 million identities on a single machine. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1445-1449, 2021.
[3] Weihua Chen, Xiaotang Chen, Jianguo Zhang. and Kaiqi Huang. Beyond triplet loss: a deep quadruplet network for person re-identification. In Proceedings of the IEEE conference computer vision and
pattern recognition, pages 403-412, 2017.
[4] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 05), volume pages 539 546. IEEE, 2005.
[5] Jiankang Deng. Jia Guo, Xiang An, Zheng Zhu, and Stefanos Zafeiriou. Masked face recognition challenge:
The insightface track report. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1437-1444, 2021.
[6] Jiankang Deng. Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Partern Recognition, pages 5203-5212, 2020.
[7] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 4690-46666 4699. 2019.
[8] Jiankang Deng, Jia Guo, Jing Yang, Alexandros Lattas, and Stefanos 7afeiriou. Variational prototype learning for deep face recognition. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 11 11906-11915 2021.
[9] Yueqi Duan, Jiwen Lu, and Jie Zhou. Uniformface: Learning deep equidistributed representation for facc recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2019.
[10] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), volume 2. pages 1735-1742. IEEE, 2006.
[11] Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Mller. Labeled faces in the wild: database forstudying face recognition in unconstrained environments. Workshop on faces in 'Real- Life Immges: detection, alignment, and recognition, 2008
[12] Yuge Huang. Yuhan Wang. Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricularface: adaptive curriculum learning loss for deep face recognition. In proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 5901-5910, 2020.
[13] Junuk Jung, Seonhoon Lee, Heung- Seon Oh, Yongjun Park, Joochan Park. and Sungbin Son. Unified negative pair generation toward cell-diseriminative feature space for face recognition. arXiv preprint arXiv:2203. 2022.
[14] SamnelnacherSShizzemmee Steven M Seitz, Daniel Miller, and Evan Brossard. The megaface benchmark: million faces for recognition at scale. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4873-4882, 2016.
[15] Jiaheng Liu. Haoyu Qin, Yichao Wu, and Ding Liang. Anchorface: Boosting tar@è far for practical face recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages
1711-1719, 2022.
[16] Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. Advances in neural information processing systems, 31, 2018.
[17] Weiyang Liu, Yandong Wen, Bhiksha Raj. Rita Singh, and Adrian Weller. Sphereface revived: Unifying hyperspherical face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[18] Weiyang Liu. Yandong Wen, Zhiding Yu, Ming Li. Bhiksha Raj. and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 212-220, 2017.
[19] Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In /CML. volume 2, page 7, 2016.
[20] Brianna Maze, Jocelyn Adams, James A Duncan, Nathan Kalka, Tim Miller, Charles Otto, Anil Jain, W Tyler Niggel, Janet Anderson, Jordan Cheney, et al. larpa janus benchmark-cc: Face dataset and protocol. In 2018 international conference on biometrics (ICB), pages 158- 165. IEEE, 2018.
11

---

[21] Stylianos Moschoglou. Athanasios Papaioannou, Christos Sagonas, Jiankang Deng. Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In proceedings of the IEEE conference on computer vision and paltern recognition workshops, pages 51-59, 2017.
[22] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015.
[23] Soumyadip Sengupta. Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa. and David W Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE winter conference on applications of computer vision (WACV), pages 1-9, IEEE, 2016.
[24] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
[25] Yi Sun, Yuheng Chen, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation by joint Advances in neural information processing systems, 27, 2014.
identificatin-vvrriiccll
[26] Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang. Liang Zheng, Zhongdao Wang. and Yichen Wei. Circle loss: unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
pattern [27] Sun, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation from predicting 10,000 classes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1891-1898, 2014.
2020.
[28] Yaniv Taigman. Ming Yang, Marc Aurelio Ranzato, and Lior Wolf. Deepface:
Closing the gap to human- level performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1701-1708, 2014.
[29] Feng Wang. Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international conference on Multimedia, pages
1041-1049 2017.
[30] Hao Wang. Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingehao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition. pages 5265-5274 2018
[31] Yandong Wen, Weiyang Liu, Adrian Weller, Bhiksha Raj. and Rita Singh. Sphereface2: Binary classifica- tion is all you need for deep face recognition. In International Conference on Learning Representations, 2022.
[32] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. discriminative feature learning approach for deep face recognition. In European conference on computer vision, pages 499- 499-515, Springer, 2016.
[33] Dong Yi, Zhen Icci, Shengeai Liao, and Stan 7. Li. Learning face representation from scratch. arXiv preprint
2014.
[34] Kai Zhao, Jingyi Xu, and Ming-Ming Cheng. Regularface: Deep face recognition via exclusive regulariza- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[35] Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang. Jiwen Lu, Dalong Du, et al. Wehface260m: A benchmark unveiling the power of million-scale deep face recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 10492- 10502, 2021.
12

---

Appendix for USS
A Inequalities about Laive
We here present the detailed derivations for the inequalities about the naive loss Lnaive. For any sample X captured from subject i, with feature x (() the naive loss Lmaive comprises of one positive cample-to-sampee similarity and N 1 negative similarities,
N (0) x.!))
Lnaive (x))) = - -g)(x(") xx (1)
+ N
(9(x
(25)
where x. M(X a.6) = 30==M(X' M(x and x! and x are randomly taken from the subject i,j, with j # i.
Using the inequality of arithmetic and geometric means, i.e, VII as S Ei= as for a, 2 0. we derive four inequalities about Lnaive-
Luaive (X (1)
N 7g (,) x.."))
- -0)(zx() xx.2))
+
N
N N
(x)
x
N
gg (x) .x.
(26)
N
N
N
N
xo.i)) _x, (79(0(1)
(0),
1
Σ
gg( (1) xz!h)))
(27)
N
N
j=1
N
log logexp (zg((0 x. _..')) log exp
7g
x
x
(28)
N
j=1
N
N
log exp (xg(x((), (,,') log
1
Σ
exp
(7g(2((1
(29)
N
N
er ((0)) 20
N
log
+ log N))
(30)
N
N j=1 e7 [
N
(00((((0)) .m(())
N Nlog
log
(31)
N -
Σ)= e7g(a(1)) (0))
N --1
Lnavve (iaive(X())) (x))
N (9(x (1) x(1)
- (g( (0) x.())))
N
S+i
N
()) x.!)
+
(1) - 27g(x11))
(32)
N 1
5-1
N
22/2( ( (1)
xx
(3)
(9(x ((,a,,))))))) -09 xx
(33)
N
2-1
221ogoox exp
1
[ (19(()(0))
(2)
+
1
Σ
(xx ,aa ) log exp (z) xx(
(7)(2(0),x()))
N
3-1
(34)
13

---

N
(19(()(0) _o!! )) - log exp (1g(20 (00(=((,,))))) xx!
2f log 1// (e(ppx exp (1g(x(0), + exp Σ
N --1
(35)
(19(2((), (0xe))))))))) -
2 1og ( (cxp (00(=(),==!))) exp Σ
N N --1-
log exp (19(z:()) =x."))) log 2
328
(36)
2og (1+ exp [>x N
1=1
N-1
2 log 2,
(37)
exp (1g(x((), ( ,0,e)))))))))) ))
Luavve (X (x))
N N (g (t) x.,)))
- 7g(x(() (1) (0)
+
1
(7g(2!0) ( + (g(x xx!")))))) 2(N - 1))g(x(0)) xx
(38)
N
N
(x((() 2x) + (g(x((1) x.())
2
(N - ()))(()))))) (x,
(39)
N
- 11
2
N
cxp 7g(x((), (10) + (x(()())))))))))))))))))))))(( ,0! (N log exp (rg( 2!(1))
2
exp
N
2
(40)
(7((2()) x(l)) "0g(=(" (1!)))
2
(N log exp (), xc("
log
(41)
N
e 4 an(1) + ()((((()) ((())
2
log (2)((( ae!")
log
(42)
N
2
(( b6 "r( ((())
2
log 207 ("))
(43)
N
2
log
(44)
N
2 e7s(a(1),00)
e (+) 2o(3))
2
log 1
2 loog 2,
(45)
N
e79(a(())) (
and
Lnaive X (1)
N N
N Σ
(1) (1)
(1))) - (N - 1)
( (1) ax!)))
(46)
N
1=1
14

---

N
N
N N
1g(x x!!
gg(x (1)
, az!1)
(47)
N N
(rg(2(1), ,x(,)))) 7g
-29(()))))))) x..2)))
(48)
N
N
(19(0((), xx( + (g(xx(011)) xz!!)) 2rgg( Drg(x(1)) (1()) xI!1))
(49)
N
N
N
logexp 70(x:1), x.. + (g(x(11 _x.1))
log exp (19(x(0), x()))))
(50)
N
2
N
N
(00(0(0)))11) + erv( e
log exp (19(x(0), ,z.")))
log
(51)
N
2
NV
N
log (ers(al) (ام (
+ (70(000),a!)) ,at")) - log exp (zg(200), xx x011)]] 2N log
N
(52)
N
N
log (1 (1 2m!
2
2.N log 2.
(53)
B Marginal Sample-to-Sampe Based Losses
We have derived three comple-to-sappe based losses in the manuscript, i.e., USS loss, sample-to- sample based softmax, and BCE losses. We hereby present their respective marginal versions:
Luss-m (X (1)) log (1 + (0)) N
log (I (r
(54)
( 10 -m)
Lznft-m (x (()) = 1 log
(55)
Σ/= ((.(
Lice-m X (() log (1 e (l((e(
log (1+ +e eng(a() ( (b))))-)))
(56)
The experimental evaluations of such marginal losses have been included Sec. 4.3 of the manuscript.
C Training Details
In our work, we choose the cosine function to represent the similarity of two features, i.e.
(ar, v..)
g(x, æ.) = cos x. x.) = |||||| |x,||'
for V x. x. F.
(57)
Following 30 7/. we use customized ResNets (such as ResNet-50, ResNet-100, and ResNet-200) as our backbone networks. We implement all models using Pytorch and train them using the SGD optimizer with weight decay of 5e-4 and momentum of 0.9. We use ? = 64 for Lusss Luwht and Lice in all experiments. Note that, use combination of CosFace(m 0.4) and our USS(m 0.1) as UniTSFace in Tables 4 and 5.
For the face models (using ResNet-50 as the backbone) on -ebbaeeeeaac,,, we train them over 28 epochs with a batch size 512. The learning rate starts 0.1 and is reduced by a factor of 10 the th and th epoch. All models in ablation and parameter study were trained on CASIA-WebFace.
15

---

In Table 1, the margin m of Lsont--- Lice-m and Luss-m are set to be 0.1. In Table 3, the margin m of ArcFace and CosFace are set be 0.5 and 0.4 respectively. The UniTSFace under the 'Small' protocol of MegaFace Challenge 1 in Table 4 and the models re-implemented in MFR Ongoing (the first ten rows in Table 5) were also trained on CASIA-WebFian
For Glint360K, we train the models(ResNet- 100) for 20 epochs using a batch size of 1024. Initially, the learning rate was set 0.1, and polynomial decay strategy (power=2) was applied the learning rate schedule. The UneeSFace under the 'Large' protocol of MegaFacc Challenge (as shown in Table was trained on Glint360K.
For WebFace4M, we train the models(ResNet-50) for 20 epochs using a batch size of 1024. The learning rate was initially set at 0.1, while polynomial decay strategy (power=2) was applied the learning rate schedule. The Uniee at the 12th row of Taalle 5 was trained on WebFace4M.
In the case of WebFace42M, we train the models(ResNet-220) for 20 epochs, using larger batch size of 4096. The learning rate linearly warmed up from 0 to 0.4 during the first epoch, followed by polynomial decay (power=2) for the remaining 19 epochs. The UnieSFace at the last row of Table 5 was trained on WebFace42M.
D Megaface Challenge
We notice that the official MegaFace challenge website has been decommissioned and MegaFace data are no longer being distributed. However, despite these changes, we can still utilize the previously released data and development kit evaluate the performance of trained models. Specifically, we have adopted the MegaFace testsuite provided by InsightFace (https://githan) com/ deepinsiigniii httee///r///mmmtt/r//tttttttto// wttp://owooovalaaaaaaoom///////e
which also includes the
official devkit.
16


Reviews:
[{"review_id": "qGyHjNwecM", "reviewer": "Reviewer_nR9r", "summary": "This paper presents an interesting idea by focusing on the unified threshold of distinguishing positive from negative pairs in face recognition. Although face recognition has undergo a big step towards real application driven by deep learning, this submission has some new proposal for sample-to-sample based loss with unified threshold. Experiments on typical facial image datasets show the effectiveness.\n\n\n===============\nAfter reading the authors' rebuttals to my concerns and dicussions, I upgrade my rating as \"weak accept\".", "questions": "In inference stage, how to determine the threshold? pre-computed? It is better to clarify this point because the title is unified theshold, which gives me the first expression about the pre-trained threshold.", "limitations": "Not observed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+It is interesting to focus on the threshold for distinguishing positive from negative pairs, which is paid less attention.\n\n+The derivation of the USS loss makes sense by defining the  upperbound of the naive loss.\n\n+Experiments by deploying USS into ArcFace and CosFace show the effectiveness of the proposed USS. The threshold range is narrowed, which shows the certainty of threshold is improved.\n", "weaknesses": "-From Table 1, it shows that BCE loss is much worse than USS loss. This seems strange and I concern if the experiment is wrong for this loss. The reasons should be discussed.\n\n-From Fig.1, the threshold range is narrowed compared to other losses, but I concern if it is correct to claim the word \"unified\".\n\n-Face recognition models are explosive increased with deep learning. A number of advanced methods are developed. The necessity of this submission can be discussed in the paper. The contribution of the proposed method to the community can be discussed by combining with previous advanced face recognition models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. From Table 1, it shows that BCE loss is much worse than USS loss. This seems strange and I concern if the experiment is wrong for this loss. The reasons should be discussed.**\n\nFirstly, we believe that the results are correct and we want to emphasize that the results are not cherry-picked. For a fair comparison, in Table 1, we adopted the same experimental setting when we trained the model using three different losses: $L_{\\text{soft}}$, $L_{\\text{bce}}$, and $L_{\\text{uss}}$.\n\nSpecifically, during our experiments, we found that the model is easy to converge using $L_{\\text{soft}}$ and $L_{\\text{uss}}$, and the training process is stable, which leads to more favorable results at the end.\n\nHowever, we found that the model is extremely hard to converge when using the $L_{\\text{bce}}$. Despite many efforts such as prolonged iterations, have been dedicated, the convergence is still problematic. We reckon that $L_{\\text{bce}}$ has respective one bias term for each subject, resulting in $N$ different explicit thresholds as $b_i = \\gamma t_i$, which lead to instable training. In contrast, the $L_{\\text{uss}}$ has only one explicit threshold and $L_{\\text{soft}}$ does not integrate any explicit threshold.\n\n**Q2. From Fig.1, the threshold range is narrowed compared to other losses, but I concern if it is correct to claim the word \"unified\".**\n\nWe thank the reviewer for pointing this out.\n\nFirstly, our theoretical objective is to learn a unified threshold that satisfying Eq. (3) during the training of a facial model, which is consistent with the requirement of the testing. Therefore, we first assume the existence of such a unified threshold, and then propose the unified threshold integrated sample-to-sample (USS) loss.\n\nWe have proven through our analysis in lines 141-148 of the paper that, ideally, a model trained by USS could learn a unified threshold for the training dataset.\n\nHowever, we must admit that achieving this ideal goal is subjective to the model capacity, training hyper-parameters, and even the training dataset itself, which are all independent from our USS loss.  For example, if the backbone network only uses one single linear neural layer, our USS loss definitely cannot guarantee a unified threshold either. In Fig. 1, however, using the same backbone architecture and training hyper-parameters, our USS loss are able to achieve a more compact threshold distribution than the other losses, which suggests the superiority of imposing the unified threshold and is consistent with our expectation.\n\nTherefore, we believe it is reasonable to retain the word \"unified\".\n\n\n**Q3. Face recognition models are explosive increased with deep learning. A number of advanced methods are developed. The necessity of this submission can be discussed in the paper. The contribution of the proposed method to the community can be discussed by combining with previous advanced face recognition models.**\n\nWe thank the advice from the reviewer.\n\nSignificant advancements in deep face recognition have been introduced to the community in the past decade, including sample-to-class-based diagrams, sample-to-sample-based diagrams, and the combinations of the two diagrams. However, none of them can explicitly learn a unified threshold to separate the positive sample-to-sample similarities from the negative similarities among all samples in the whole training dataset. An optimal threshold separating all positive sample-to-sample pairs from the negative ones is also in demand during testing. \n\nIn our submission, we propose the USS loss to explicitly learn a unified threshold for the training dataset. Though this unified threshold learned in the training stage cannot be directly applied to the testing stage, the model trained by USS is desired to extract more discriminative features and subsequently improves the face verification performance in various testing scenarios, which have been demonstrated through our extensive experiments.\n\nFurthermore, the proposed USS loss can be effortlessly extended to the Marginal USS loss and can also be seamlessly jointly used with other sample-to-class losses. In our experiments, we have combined the USS loss with ArcFace and CosFace methods, and the combined approaches consistently surpass their respective individual counterparts. We denote the fusion of the CosFace and USS losses as \"UniTSFace,\" which we have compared with other sophisticated combinations such as VPL, UNPG, and AnchorFace.  The experimental results on multiple benchmark datasets further suggest the superiority of our UniTSFace.\n\nIn conclusion, we believe our USS loss provides the research community with a much more versatile and effective solution for face recognition tasks.\n\n**Q4. In inference stage, how to determine the threshold? pre-computed? It is better to clarify this point because the title is unified threshold, which gives me the first expression about the pre-trained threshold.**\n\nWe thank the reviewer for this suggestion. We will clarify this in the revised version.\n\nYes, the threshold learned by the training stage cannot be directly used in testing. In the testing/inference stage, we first extract the features using trained models for all testing images, and then the threshold is determined according to the specific testing criteria. For example, when reporting the 1:1 verification accuracy on LFW, CFP-FP, AgeDB in Table 5, 10-fold validation is used. We first select the threshold which achieves the highest accuracy in the first 9 folds, and then adopt this threshold to calculate the accuracy in the leave-out fold."}, {"review_id": "nuOZefKcFs", "reviewer": "Reviewer_qAqH", "summary": "This paper proposes UniTSFace, a new approach to face recognition that uses a unified threshold integrated sample-to-sample loss (USS loss). The USS loss features a unified threshold for distinguishing positive from negative pairs and can be enhanced with an auxiliary margin. The authors show that the USS loss can be integrated with sample-to-class based losses and evaluate its effectiveness on various benchmark datasets, demonstrating its suitability for real-world applications. The contributions of this work include introducing the USS loss, demonstrating that it can learn a unified threshold, and showing that it can be enhanced with an auxiliary margin and is compatible with existing sample-to-class based losses.\n\nIn addition, UniTSFace outperforms state-of-the-art methods on multiple benchmark datasets, including the Megaface 1 dataset, and the proposed approach is evaluated on the large-scale WebFace42M dataset, which contains 42.5 million images of 2 million identities, demonstrating its efficacy for real-world applications.", "questions": "1. The paper mentions that the unified threshold is limited to a sample-to-sample loss. Have you considered extending the unified threshold to sample-to-class loss, and if so, what are the potential benefits and challenges of doing so?\n2. The paper mentions that UniTSFace achieves state-of-the-art performance on multiple benchmark datasets. However, how does UniTSFace compare to other state-of-the-art methods in terms of computational efficiency and memory usage?\n", "limitations": "The paper does not explicitly mention any limitations ", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The advantages of the USS loss over other loss functions are numerous. First, it is highly efficient and can work seamlessly with sample-to-class-based losses. Second, it overcomes the limitations of previous sample-to-sample losses by explicitly incorporating a learnable threshold that separates positive and negative pairs. Third, it achieves state-of-the-art performance on multiple benchmark datasets, demonstrating its effectiveness in real-world face recognition applications.\n2. The experiments in this paper are solid and have been conducted on the largest-scale dataset WebFace42M, achieving the state-of-the-art performance on the most challenging benchmark dataset MFR in the academic track, and the comparisons are fair.\n3. The proposed method is original\n4. There are no significant flaws in the experiments conducted in this work.", "weaknesses": "The paper does not explicitly mention any weaknesses or limitations of the proposed UniTSFace approach or the USS loss function. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.The paper mentions that the unified threshold is limited to a sample-to-sample loss. Have you considered extending the unified threshold to sample-to-class loss, and if so, what are the potential benefits and challenges of doing so?**\n\nWe thank the reviewer for pointing this out. We are actually in the process of integrating the unified threshold into a sample-to-class loss, which we term USC loss here in coordination with USS loss for convenience.\n\nThe sample-to-class-based losses are conventionally built on Softmax loss in face recognition, including the well-known Large Margin Softmax loss, normalized Softmax loss, cosine-margin and angular-margin based Softmax loss, such losses only learn a class proxy conveyed in the weight vector, instead of exploring the cross-sample relationship. To incorporate the unified threshold into the sample-to-class Softmax loss, it is straightforward that we directly introduce a unified threshold:\n$$L_{\\text{usc}}(\\textbf X^{(i)})=\n\\log(1+e^{-\\gamma\\ g(\\textbf x^{(i)}, \\textbf c^{(i)})+b})\n+\\sum_{j\\neq i\\atop j=1}^N\\log(1+e^{\\gamma\\ g(\\textbf x^{(i)}, \\textbf c^{(j)}) - b}),$$ \nwhere $b = \\gamma t$ is a constant to be learned, $t$ hereby is the desired unified threshold, and $\\textbf c^{(i)}$ is the feature proxy of the $i$-th class.\n\nSince the unified threshold in USC provides a stricter constraint than the original normalized Softmax loss, i.e., we anticipate that such a unified threshold can separate all the positive sample-to-class similarities from the negative ones. The USC loss is expected to outperform the original normalized Softmax counterpart in face recognition.\n\nAdditionally, the USC is essentially a sample-to-class-based classification loss, it can naturally be used in general object classification tasks.\n\nOn the other hand, we must admit that the USC loss has its own shortcomings in face recognition. Different from the sample-to-sample loss (such as $L_{\\text{naive}}$, $L_{\\text{uss}}$) used in this submission, it cannot directly explore the relationships among the facial samples in the training of the facial model, as it only optimizes the class proxy and sample features during training. The key challenge towards robust face recognition is how to learn a representative class proxy $\\textbf c^{(i)}$ that can benefit from the abundant training facial samples. Therefore, as we stated in the conclusion of the manuscript, effectively combing the two approaches into one framework will be our future work.\n\n\n**Q2. The paper mentions that UniTSFace achieves state-of-the-art performance on multiple benchmark datasets. However, how does UniTSFace compare to other state-of-the-art methods in terms of computational efficiency and memory usage?**\n\nWe thank the reviewer for pointing this out. UniTSFace utilizes the ResNet architecture as its backbone and optimizes the parameters using the algorithmic average of the cosine-margin Softmax loss and the proposed USS loss.\n\nIn the inference stage, the trained UniTSFace model is solely a convolutional ResNet used to extract latent facial features for testing images. Naturally, the computational consumption and memory usage totally depend on the convolutional operations inherent to the selected ResNet architecture and the resolutions of the input images used during testing.\n\nSince all the methods under comparison in Tables 4 and 5 adopt the same ResNet-50, ResNet-100, and ResNet-200 backbones and the same input images for testing, the differences in computational and memory usage are negligible. This is affirmed in **Table C**, where we present the inference runtime across various methods. The Table clearly indicates that the variations in inference times among different methods are negligible, measured in fractions of a millisecond.\n\nContrastingly, in the training stage, different models employ different loss functions to train the convolutional ResNet backbone, resulting in a diverse computational overhead and memory usage. Theoretically, \n\n1. when comparing to deep models based on the purely sample-to-class losses, such as CosFace and ArcFace, UniTSFace requires additional $O(N)$ logarithmic/multiply-addition operations attributable to the USS loss. The joint USS loss entails additional memory usage to accommodate $N$ facial features, where $N$ denotes the number of subjects within the training dataset.\n2. when comparing to deep models based on the purely sample-to-sample losses, e.g., $L_{\\text{soft}}$ or $L_{\\text{bce}}$, UniTSFace only requires $O(N)$ extra logarithmic/multiply-addition operations to account for the cosine-marginal Softmax loss.\n3. when comparing to deep models trained with a combination of the two kinds of losses like VPL, UNPG, and AnchorFace, the computational requirements and memory consumption of our UniTSFace remain at par.\n\nIn experiments, since the forward and backward propagation of convolutional operations (including the activations functions and normalization layers) accounts for the majority of computations and memory usage, we found that the computational difference between these methods during the training stage is minimal as well. As reported in **Table C**, the memory usage (in GB) and training speeds (in seconds per batch, i.e., seconds per 512 images) across different methods similarly exhibit minimal discrepancies."}, {"review_id": "dRZiT2RAGE", "reviewer": "Reviewer_XuAx", "summary": "The paper proposes a unified threshold integrated sample-to-sample loss for face recognition, which addresses the limitations of existing methods in exploring the cross-sample relationship and setting a unified threshold. The proposed loss function achieves exceptional performance on multiple benchmark datasets.", "questions": "Please refer to the Weaknesses section.", "limitations": "None", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper introduces an approach called UniTSFace, which combines a unified threshold with a sample-to-sample loss for face recognition. This approach addresses the limitations of existing methods and proposes a new loss function that achieves exceptional performance on multiple benchmark datasets.\n2) The experimental results of the paper demonstrate significant improvement, especially compared to the baseline, on highly discriminative datasets such as MFR.", "weaknesses": "1) It is unnecessary to use complex symbolic formulas to express simple concepts, such as in lines 105 to 124.\n2) Adding a sample-to-sample loss in the field of face recognition is not novel enough.\n3) The training method of using a fixed threshold may appear to be consistent with testing, but it actually conflicts with it. In testing, different false acceptance rates (FAR) are used on different scenarios, meaning that for the same model, different thresholds will be used to determine the predicted results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. It is unnecessary to use complex symbolic formulas to express simple concepts, such as in lines 105 to 124.**\n\nThanks for the comments. The reviewer appears to be well-versed in face recognition research and reckons that these formulas present a simple concept. However, it is essential for us to cater to a wider range of readers, including beginners in this field, and accurately and clearly present preliminary concepts in a comprehensive manner.\n\nMoreover, through these formulas, we aim to mathematically present our motivation for this work, i.e., explicitly learning a unified threshold to distinguish the positive sample-to-sample similarities from the negative ones on the training set, which is absent in existing sample-to-sample losses. \n\nAdditionally, these notations are consistently used in sections 3.1, 3.2, and 3.3 to formulate different losses, namely $L_{\\text{naive}}$, $L_{\\text{soft}}$, $L_{\\text{bce}}$, and $L_{\\text{uss}}$. Removing these notations/formulas directly would adversely affect the presentation of these losses and lead to misunderstandings.\n\nTherefore, we believe it is crucial to retain the formulas in lines 105-124 to maintain a coherent and informative paper that caters to both experienced researchers and newcomers to the field of face recognition.\n\n**Q2. Adding a sample-to-sample loss in the field of face recognition is not novel enough.**\n\nSample-to-sample based loss has been a crucial research topic in deep face recognition for the past decades. Researchers have dedicated substantial efforts to this area and have introduced prestigious losses and methods such as DeepID2, Triplet Loss, and (N+1)-Tuplet Loss, just to name a few. The majority of them aim to maximize inter-subject discrepancy while minimizing intra-subject distances, which may need a meticulous sampling/paring step for every mini-batch. Moreover, in face verification tasks, a single threshold is required to distinguish positive facial pairs from negative ones. Unfortunately, none of the aforementioned methods incorporate such an explicit constraint.\n\nTo address this limitation, we propose the USS, which explicitly incorporates a learnable unified $t$ during the training process. By encouraging to separate positive and negative image pairs by the $t$, we expect that the model is able to learn more discriminative features, which thereafter can perform better on the unseen testing sets. \n\nAdditionally, through our derivations, the $t$ can be embedded in the bias term $b=\\gamma t$. After the model is trained, we are able to directly investigate the value of this $t$, which can help us to quantitively and qualitatively understand what the model is learning, as shown in Fig. 1 in the paper.\n\nFurthermore, the USS can be effortlessly extended to the Marginal USS loss and can be jointly used with other sample-to-class losses. In our experiments, we have combined the USS loss with CosFace as UniTSFace, which surpassed other sophisticated combinations such as VPL, UNPG, and AnchorFace. In fact, as the reviewer pointed out, \"**the experimental results of the paper demonstrate significant improvement**\".\n\nReviewer nR9r agrees with us: “**It is interesting to focus on the threshold for distinguishing positive from negative pairs, which is paid less attention.**” Reviewer qAqH also stated that “**The advantages of the USS loss over other losses are numerous**”. Overall, we assert that our USS is novel and provides the research community with a much more general and effective solution for face recognition tasks.\n\n**Q3. The training method of using a fixed $t$ may appear to be consistent with testing, but it actually conflicts with it. In testing, different FAR are used on different scenarios, meaning that for the same model, different thresholds will be used.**\n\nWe thank the reviewer for pointing this out. We will clarify this part in the revised version.\n\nFirstly, we note that the threshold $t$ is not fixed during training but a learnable parameter embedded in $b=\\gamma t$. For different training datasets, the learned $t$ might be different.\n\nSecondly, it is true that different thresholds will be selected based on various criteria such as different FAR. This however, is not conflicting with our goal of learning a unified threshold in our USS loss. We explain the reasons as follows:\n\n1. The training and testing stages in the context of deep face recognition are two completely separate stages. During the training stage, a face model is supervised and trained using either sample-to-class or sample-to-sample loss functions. Once the model is trained, it will solely be employed to extract latent facial features for testing samples (which may not necessarily belong to the subjects in the training set). The extracted features will be used to evaluate the performance of the trained face model. In other words, the performance of a face model essentially relies on effective feature learning, which is, features corresponding to the same subject are brought closer together, while features from different subjects are pushed apart. In testing, based on the features extracted by the learned deep face models, different thresholds will be determined for best performance in different scenarios.\n\n2. We propose to explicitly learn a unified threshold $t$ through our USS loss.  The training process towards the unified threshold ensures that the model not only promotes proximity among intra-subject features and discrepancy among inter-subject features, but also explicitly encourages all positive sample-to-sample similarities to exceed all negative sample-to-sample similarities. By imposing this more stringent constraint, the features learned using our USS loss are anticipated to be more discriminative.\n\n3. Though the learned unified $t$ might not be directly used in various testing scenarios, the ultimate goal of separating the negative from positive pairs using a unified threshold is consistent in both training and testing."}, {"review_id": "JtiH08Njym", "reviewer": "Reviewer_m9uU", "summary": "I thank the authors for interesting paper and the effort in expanding the field of face recognition. \nThe paper proposes a new loss function USS (and its variants) for face recognition. The motivation is to jointly learn a threshold that can be used during the verification process. The authors show the efficacy of UniTSFace when USS is used jointly with ArcFace or CosFace (margin based softmax loss) in various evaluation sets. ", "questions": "- It was not stated clearly in the text what unified refers to. If it is correct that the authors use the term unified because the learned threshold is used for all evaluation settings *(in contrast to the normal scenario where the threshold is computed for each dataset), then it would be good to say it in the beginning. \n- Why does the unified threshold satisfy the equation 3? There may not exist t that satisfies this for a given dataset. \n- How does UniTSFace compare with ArcFace + Supervised Contrasitve Loss (InfoNCE) and ArcFace + L_naive (which does not learn t). In such case, t is computed for each validation dataset as is customary for face recognition. \n", "limitations": "Yes. The limitation is discussed.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The performance gain in various datasets using UniTSFace is clearly shown. \n- The derivation of USS loss starting from contrastive objective is interesting. \n- The motivation of trying to learn one threshold that can be used in all evaluation dataset is interesting. ", "weaknesses": "- The authors do not show whether learning the threshold t is really necessary. The readers cannot distinguish whether the performance gain comes from making the threshold t, a learnable parameter or it comes from the addition of the supervised contrastive learning objective which is a loss function that boosts the performance when combined with margin based softmax losses [1].\n- The authors do not show whether the learned threshold is indeed better than the more common method which is to calculate the optimal threshold specific to the validation dataset. \n- Figure 1 seems to suggest that despite the L_uss loss, the variance in threshold remains therefore, it is better to calibrate different threshold for different datasets. \n\n[1] CoReFace: Sample-Guided Contrastive Regularization for Deep Face Recognition", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The authors do not show whether learning the $t$ is really necessary.**\n\nThanks for the suggestion.\n\nWe have compared the performance of $L_{\\text{uss}}$ and $L_{\\text{naive}}$ in **Table A**, it is evident that $L_{\\text{uss}}$, by adding the $t$, achieves a remarkable enhancement over $L_{\\text{naive}}$. We note that we dedicated extensive efforts to train the model with $L_{\\text{naive}}$, but the model struggled to converge, resulting in inferior results. \n\nIn **Tables A and B**, we further compared the performance of both $L_{\\text{uss}}$ and $L_{\\text{naive}}$, when employed in conjunction with the CosFace and ArcFace. Similarly, we observed that the combination of ArcFace/CosFace with $L_{\\text{naive}}$ resulted in notably inferior outcomes compared to the baseline ArcFace/CosFace. This starkly contrasts with the more favorable performance achieved by pairing ArcFace/CosFace with $L_{\\text{uss}}$. These results show that the $t$ plays a pivotal role in boosting performance.\n\nMoreover, we note that in Table 1 of our paper, we have compared our $L_{\\text{uss}}$ with the other two baseline sample-to-sample losses, $L_{\\text{soft}}$ and $L_{\\text{bce}}$ (neither has a unified $t$). Our $L_{\\text{uss}}$ again outperforms these two losses, which also shows the effectiveness of incorporating the $t$.\n\n**Q2. Whether the learned $t$ is indeed better than the more common method which is to calculate the optimal threshold specific to the validation dataset.**\n\nThanks for the question and we will clarify this in the revision.\n\nFirstly, the learned $t$ is not directly used in testing. In our experiments, the optimal threshold is indeed calculated specific to the validation dataset as well as the testing criteria. \n\nSecondly, though the unified $t$ learned by the model on the training dataset cannot be directly applied to the validation dataset, the model itself has learned compact and discriminative features. Specifically, without this explicit $t$, the losses, $L_{\\text{naive}}$, $L_{\\text{soft}}$, and $L_{\\text{bce}}$, only promote proximity among intra-subject features and discrepancy among inter-subject features for each subject. With this explicit $t$, our USS loss aims to learn features to distinguish all the positive sample-to-sample similarities from all the negative ones on the whole training dataset. In other words, the learning objective of USS loss is more stringent, the features learned using our USS loss are therefore expected to be more discriminative. The results reported in Table 1 of the paper, as well as in the **Tables A and B**, demonstrate the benefits of the features learned by adding the unified $t$.\n\n**Q3. Figure 1, the variance in threshold remains therefore, it is better to calibrate different threshold for different datasets.**\n\nAs per our response to **Q2**, the optimal testing threshold is indeed calibrated for specific dataset.\n\nOn the other hand, though there are variances in Fig. 1, we note that the distribution from $L_{\\text{uss}}$ is much more compact than the other two counterparts, which qualitatively support our claim that the model has managed to learn more compact and discriminative features, due to the application of $L_{\\text{uss}}$ loss to learn a unified threshold.\n\n**Q4. It was not stated clearly in the text what unified refers to.**\n\nThanks for the advice and we will clarify this in the revision.\n\nIn the case of training, the unified threshold $t$ distinguishes our $L_{\\text{uss}}$ from the other sample-to-sample loss such as $L_{\\text{naive}}$, $L_{\\text{soft}}$, and $L_{\\text{bce}}$, which only care about the subject-level separation: i.e., intra-subject similarity is higher than inter-subject similarity. They do not pursue that, for all subjects, all the positive sample-to-sample similarity should be larger than all the negative sample-to-sample similarity. The $t$ in our $L_{\\text{uss}}$, however, explicitly requires that all the positive sample-to-sample similarity should be larger than all the negative sample-to-sample similarity for all samples. We, therefore, name our loss as a unified threshold integrated loss.\n\nIn the case of testing, the learned $t$ on the training set cannot be directly applied to the testing datasets. The optimal testing threshold will be calculated based on different datasets as well as the application scenarios. \n\n**Q5. Why does the unified $t$ satisfy the equation 3? There may not exist a $t$ that satisfies this for a given dataset.**\n\nWe agree that there might not exist a unified $t$ that satisfies Eq. (3) for a given dataset, which have noise samples, or wrong labels. \n\nIdeally, if the dataset is clean and the face model is well-trained, all positive sample-to-sample similarity should be close to 1 and negative similarity should be close to -1. Then the $t$ will exist.\n\nNaturally, our USS loss is built on the assumption that there exists such a unified $t$ for a given dataset, then it is our objective to find the unified threshold satisfying Eq. (3) in the training of a facial model. We have demonstrated that the unified $t$ can be learned after the training of the model by adopting the USS loss, in lines 141-148, and the unified $t$ is evolved into the learnable parameter bias $b=\\gamma t$ in the USS loss.\n\nEven though the unified $t$ may not exist, the learning process towards this ideal condition still enables the model to learn more compact and discriminative features than the other losses, which can lead to better performance on various testing datasets.\n\n**Q6. Compare UniTSFace with ArcFace + InfoNCE / + L_naive.**\n\nAs per the reviewer's request, we have conducted the related experiments and reported the results in **Table A**. It is clear that ArcFace + $L_{\\text{naive}}$ produced significantly inferior results than the plain ArcFace, while ArcFace + InfoNCE improves the performance of ArcFace, it is still outperformed by our UniTSFace, demonstrating the effectiveness of our method."}]

Author rebuttal:
We thank all reviewers for appreciating the state-of-the-art performance of our UniTSFace; especially the recommendations from reviewers qAqH and nR9r that “It is interesting to focus on the threshold for distinguishing positive from negative pairs, which is paid less attention,” and “the advantages of our USS loss over other losses are numerous”. We provide point-to-point responses to the main concerns raised by the reviewers. We here present the experimental results requested by the reviewers and attach them in the PDF file.


**Table A. Comparisons of different losses/methods on the MFR-Ongoing dataset. We note that model trained with Lnaive encountered challenges in convergence and exhibited the least favorable results.**
|     Method                 |     MR-ALL    |     IJB-C    |     LFW      |     CFP      |     Age      |
|----------------------------|---------------|--------------|--------------|--------------|--------------|
|     $L_{\text{naive}}$                 |     0.0       |     0.35     |     50.0     |     50.0     |     50.0     |
|     $L_{\text{uss}}$                   |     38.43     |     72.20    |     99.40    |     96.51    |     94.05    |
|     ArcFace                |     42.21     |     48.49    |     99.31    |     97.07    |     94.51    |
|     ArcFace   + InfoNCE    |     45.47     |     88.00    |     99.40    |     97.11    |     94.71    |
|     ArcFace   + $L_{\text{naive}}$     |     0.48      |     3.52     |     98.21    |     80.42    |     84.13    |
|     ArcFace   + $L_{\text{uss}}$       |     48.76     |     89.06    |     99.58    |     97.40    |     94.73    |
|     ArcFace   + $L_{\text{soft-m}}$    |     46.03     |     88.16    |     99.45    |     97.20    |     95.03    |
|     ArcFace   + $L_{\text{bce-m}}$     |     17.28     |     65.10    |     96.38    |     74.40    |     82.81    |
|     ArcFace   + $L_{\text{uss-m}}$     |     48.92     |     89.56    |     99.40    |     97.22    |     95.20    |

**Table B. Comparisons of different losses in combination with CosFace on the MFR-Ongoing dataset.**
|     Method                 |     MR-ALL    |     IJB-C    |     LFW      |     CFP      |     Age      |
|----------------------------|---------------|--------------|--------------|--------------|--------------|
|     CosFace                |     45.12     |     56.65    |     99.36    |     97.30    |     94.98    |
|     CosFace   + $L_{\text{naive}}$     |     1.82      |     13.03    |     98.98    |     95.47    |     93.05    |
|     CosFace   + $L_{\text{uss}}$       |     49.75     |     89.62    |     99.41    |     96.78    |     95.30    |
|     CosFace   + $L_{\text{soft-m}}$   |     47.90     |     88.71    |     99.46    |     97.12    |     95.50    |
|     CosFace   + $L_{\text{bce-m}}$     |     15.53     |     61.53    |     96.15    |     73.20    |     80.36    |
|     CosFace   + $L_{\text{uss-m}}$      |     50.28     |     89.84    |     99.41    |     97.35    |     95.13    |

**Table C. Comparisons of different methods/losses in terms of the Inference Time (in millisecond), Training Memeory usage (in GB), and Training Speeds (in seconds per batch i.e., seconds per 512 images). All experiments are conducted on the same machine with Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz; TITAN RTX, 24GB.**
|     Method        |     Inference Time     (ms)    |     Training Memory   (G)    |     Training Speeds   (s/batch, s/512images)    |
|-------------------|--------------------------------|------------------------------|-------------------------------------------------|
|     ArcFace       |     14.42                      |     9.20                     |     0.93                                        |
|     CosFace       |     14.25                      |     9.20                     |     0.92                                        |
|     $L_{\text{soft}}$         |     14.50                      |     9.19                     |     0.96                                        |
|     $L_{\text{bce}}$          |     14.42                      |     9.19                     |     0.97                                        |
|     $L_{\text{uss}}$          |     14.33                      |     9.22                     |     0.97                                        |
|     VPL           |     14.25                      |     9.19                     |     0.97                                        |
|     UNPG          |     14.66                      |     9.24                     |     0.96                                        |
|     AnchorFace    |     14.25                      |     9.24                     |     0.99                                        |
|     UniTSFace     |     14.60                      |     9.22                     |     0.97                                        |

Meta-review: Most of the reviewers gave positive ratings on this submission while the Reviewer XuAx had some concerns on writing and fixed threshold. After carefully read the submission, comments and rebuttals, the AC agree with most of the reviewers and suggest accepting this submission.
Decision: Accept (poster)