SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
L _ {\mathrm {b c e}} \left(\boldsymbol {X} ^ {(i)}\right) = \log \left(1 + \mathrm {e} ^ {- \gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(i)}\right) + b _ {i}}\right) + \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \log \left(1 + \mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(j)}\right) - b _ {j}}\right), \tag{22}
$$
where  $b_{i} = \gamma t_{i}$  are parameters to be learned. Note that the  $t_i$  can be different for different identities and therefore are not unified.
After the training of the model  $\mathcal{M}$ , i.e., the threshold  $t_i = \frac{b_i}{\gamma}$  were learned, then,
$$
\sum_ {i = 1} ^ {N} L _ {\text {n a i v e}} \left(\boldsymbol {X} ^ {(i)}\right) \leq \frac {2}{N} \sum_ {i = 1} ^ {N} L _ {\mathrm {b c e}} \left(\boldsymbol {X} ^ {(i)}\right) - 2 N \log 2. \tag {23}
$$
3.3 Marginal Sample-to-Sample Based Losses
Our USS loss, as well as the deduced sample-to-sample based  $L_{\mathrm{soft}}$  and  $L_{\mathrm{bce}}$ , only encourage the separability between positive and negative sample pairs. To further improve the discriminative ability of such losses, i.e., to encourage the positive features to be distant from the negative ones, we further proposed the marginal extensions for such losses.
By introducing a margin on the feature similarities, we can have the marginal USS loss as:
$$
L _ {\mathrm {u s s - m}} \left(\boldsymbol {X} ^ {(i)}\right) = \log \left(1 + \mathrm {e} ^ {- \gamma \left(g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(i)}\right) - m\right) + b}\right) + \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \log \left(1 + \mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(j)}\right) - b}\right) \tag{24}
$$
where the  $m$  is the introduced hyper-parameter on the margin. Proper adjusting of such a parameter can improve the recognition performance, as discussed in Sec. 4.3. Similarly, we can extend the vanilla  $L_{\mathrm{soft}}$  and  $L_{\mathrm{bce}}$  to the marginal version of  $L_{\mathrm{soft - m}}$  and  $L_{\mathrm{bce - m}}$ , the full equations are given in the supplementary materials (and appendix).
4.1 Datasets and Evaluations
Datasets. We utilize four publicly available datasets for training, namely, CASIA-WebFace(33) (consisting of 0.5 million images of 10K identities), Glint360K(2) (comprising 17.1 million images of 360K identities), WebFace42M(35) (containing 42.5 million images of 2 million identities), and WebFace4M, which is a subset of WebFace42M with 4.2 million images of 0.2 million identities. For evaluating the face verification performance, the ICCV-2021 Masked Face Recognition Challenge (MFR Ongoing)(5) is adopted. The MFR ongoing testing protocol includes various popular benchmarks, such as LFW (11), CFP-FP (23), AgeDB (21), and IJB-C (20), along with its own MFR benchmarks, such as the Mask, Children, and Globalized Multi-Racial (GMR) test sets. The Mask set comprises 13.9K positive pairs and 96.9 million negative pairs (including 6.9K masked images and
13.9K non-masked images) of 6.9K identities. The Children set contains 157K images (totaling 1.7 million positive pairs and 24.7 billion negative pairs) of 14K identities. The Globalized Multi-Racial sets comprise 1.6 million images in total, consisting of 4.6 million positive pairs and 2.6 trillion negative pairs, and representing 242K identities across four races: African, Caucasian, South-Asian, and East-Asian. For face identification, the MegaFace Challenge 1(14) is employed as the test set, which comprises a gallery set with over 1 million images from 690K different identities, and a probe set with 3,530 images from 530 identities. It is worth noting that the MegaFace Challenge 1 also has a verification track, and the verification performance is included in the experiments.
Evaluation and Metrics. For the MFR Ongoing Challenge, the trained models are submitted to and evaluated by the online server. Specifically, we report 1:1 verification accuracy for LFW, CFP-FP, and AgeDB. We report True Accept Rate (TAR) at False Accept Rate (FAR) levels of 1e-4 and 1e-5 for IJB-C. We report TARs at  $\mathrm{FAR} = 1\mathrm{e} - 4$  for the Mask and Children test sets, and TARs at  $\mathrm{FAR} = 1\mathrm{e} - 6$  for the GMR test sets. For the MegaFace Challenge 1, we report Rank1 accuracy for identification and TAR at  $\mathrm{FAR} = 1\mathrm{e} - 6$  for verification.
Preprocessing. Firstly, we aligned all face images using the 5 landmarks detected by RetinaFace (6) and cropped the center  $112 \times 112$  patch. We then normalized the cropped images by first subtracting 127.5 and then dividing 128. Finally, we augmented the training images with horizontal flipping.
Training. We adopt customized ResNets as our backbone following (7). We implement all models using Pytorch and train them using the SGD optimizer with a weight decay of 5e-4 and momentum of 0.9. For the face models on CASIA-WebFace, we train them over 28 epochs with a batch size of 512. The learning rate starts at 0.1 and is reduced by a factor of 10 at the  $16^{th}$  and  $24^{th}$  epoch. For both Glint360K and WebFace4M, we train the ResNets for 20 epochs using a batch size of 1024. The learning rate is initially set at 0.1, while a polynomial decay strategy (power=2) is applied for the learning rate schedule. In the case of WebFace42M, we train the ResNets for 20 epochs, using a larger batch size of 4096. The learning rate linearly warms up from 0 to 0.4 during the first epoch, followed by a polynomial decay (power=2) for the remaining 19 epochs. We include the detailed settings of all hyper-parameters used in Sec. 4 and Sec. 5 in the appendix for further reference.
Testing. For a given facial image, we extract two 512-dimensional features from the original image and its horizontally flipped counterpart. These features are then combined together as the final representation with element-wise addition. To assess the similarity between two images, we utilize the cosine similarity metric.
4.3 Ablation and Parameter Study
Effectiveness of unified threshold. We first demonstrate the necessity of learning a unified threshold by comparing our proposed  $L_{\mathrm{uss}}$  with two other sample-to-sample losses,  $L_{\mathrm{soft}}$  and  $L_{\mathrm{bce}}$ , in Table 1. We additionally report the performance of the model trained with  $L_{\mathrm{naive}}$ . The first to third rows respectively show the results from  $L_{\mathrm{naive}}$ ,  $L_{\mathrm{soft}}$  and  $L_{\mathrm{bce}}$ , while the fourth row lists the performance of our  $L_{\mathrm{uss}}$ . We observe that  $L_{\mathrm{soft}}$  performs  $4.68\%$  better than our proposed  $L_{\mathrm{uss}}$  on IJB-C. However, our USS loss outperforms  $L_{\mathrm{soft}}$  on the other four
Loss	MR-ALL	IJB-C	LFW	CFP	Age
Lnaive	0.0	0.35	50.0	50.0	50.0
Lsoft	26.34	76.88	98.80	95.50	93.48
Lbce	8.91	39.19	92.60	66.41	74.36
Luss	38.43	72.20	99.40	96.51	94.05
Lsoft-m	38.11	83.60	99.18	96.32	94.18
Lbce-m	9.01	45.08	92.48	65.34	76.30
Luss-m	42.55	83.92	99.46	96.81	94.28
Table 1: Ablation study of the proposed  ${L}_{\text{uss }}$  .
datasets, particularly on MFR-All where it achieves a  $12.09\%$  higher TAR@FAR=1e-6. Compared to the  $L_{\mathrm{bce}}$ , the gains achieved by our  $L_{\mathrm{uss}}$  are more significant, with performance improvements of  $29.52\%$  on MFR-All,  $33.01\%$  in terms of TAR@FAR=1e-4 on IJB-C,  $30.10\%$  on CFP-FP, and  $19.69\%$  on AgeDB. The  $L_{\mathrm{naive}}$ , however, is difficult to train and the trained model barely learns any useful features. We note that the results are not cherry-picked. For a fair comparison, we adopted the same experimental setting when we trained the model using the different losses. During our experiments, we found that the model is easy to converge using  $L_{\mathrm{soft}}$  and  $L_{\mathrm{uSS}}$ , and the training process is stable, which leads to more favorable results. The model, however, is difficult to train with  $L_{\mathrm{naive}}$  and  $L_{\mathrm{bce}}$ . We have dedicated many efforts to these two losses, but the convergence is still problematic.
When comparing the marginal softmax loss  $L_{\mathrm{soft - m}}$ , marginal BCE loss  $L_{\mathrm{bce - m}}$  with marginal USS loss  $L_{\mathrm{uss - m}}$  (last three rows), we observe that the marginal USS loss  $L_{\mathrm{uss - m}}$  achieves the highest performance across all five reported datasets, which further confirms the effectiveness of our proposed unified threshold strategy.
It is noteworthy that all the methods in this table are based on the same ResNet-50 network trained on the CASIA-WebFace dataset.
Qualitative study of unified threshold. To better understand the effects of unified threshold, in Fig. 1, we randomly select 20 identities (5,074 facial images) from the whole training dataset with  $N = 10572$  subjects, and then construct 1 positive sample pair and  $N - 1$  negative pairs for each of the selected images. For each of the 20 identities, we compute the optimal threshold to separate the positive pair and the hardest negative pair, i.e., the most similar negative pair.
We respectively plot threshold distributions of the 20 identities for  $L_{\mathrm{soft}}$ ,  $L_{\mathrm{bce}}$ , and  $L_{\mathrm{uss}}$ . In Fig. 1, each dot denotes the optimal threshold for each identity. The blue dashed line is the unified threshold  $t = \frac{b}{\gamma} = 31.3344 / 64 = 0.4896$  exclusively learned by our  $L_{\mathrm{uss}}$  in Eq. (11). From this figure, we can clearly observe that our  $L_{\mathrm{uss}}$  has the most compact threshold distribution, while the threshold distributions from  $L_{\mathrm{bce}}$  and  $L_{\mathrm{soft}}$  are relatively loose. Most importantly, for  $L_{\mathrm{uss}}$ , the median threshold from the randomly sampled 20 identities is in line with the learned unified threshold 0.4896 from the face model trained with  $L_{\mathrm{uss}}$ , which proves the effects of including an explicit unified threshold.
Impact of Margins. The results in Table 1 (with  $m = 0.1$ ) demonstrate that a margin improves the performance of  $L_{\mathrm{uss}}$ . However, the introduced margin  $m$  remains a hyperparameter that needs to be tuned. In Table 2, we investigate the effects of using different margins in  $L_{\mathrm{uss - m}}$ . Specifically, we experiment with four more settings that increase  $m$  to 0.2, 0.3, 0.4, and 0.5. We observe that i) the highest performance on MR-ALL, IJB-C, LFT, CFP-FP, and AgeDB are respectively achieved when  $m$  equals 0.2, 0.4, 0.1, 0.1, and 0.3; ii) the performance of the four marginal  $L_{\mathrm{uss - m}}$  losses are all higher than that of the original  $L_{\mathrm{uss}}$  (i.e.,  $m = 0$  in  $L_{\mathrm{uss - m}}$ ). Although the auxiliary margin improves the overall performance, the performance seems to saturate when  $m$  is set to 0.4. Note that in our later experiments, we use  $m = 0.1$ .
Margin	MR-ALL	IJB-C	LFW	CFP	Age
m = 0.0	38.43	72.20	99.40	96.51	94.05
m = 0.1	42.55	83.92	99.46	96.81	94.28
m = 0.2	44.74	87.65	99.26	96.65	94.16
m = 0.3	44.14	87.58	99.30	96.41	94.38
m = 0.4	43.38	87.82	99.26	96.04	94.23
m = 0.5	43.91	87.62	99.13	96.05	94.23
Table 2: Parameter study of margin in  ${L}_{\text{uss-m }}$  .
MR-ALL	IJB-C	LFW	CFP	Age
USS	42.55	83.92	99.46	96.81	94.28
ArcFace	42.21	48.49	99.31	97.07	94.51
ArcFace+USS	48.92	89.56	99.40	97.22	95.20
CosFace	45.12	56.65	99.36	97.30	94.98
CosFace+USS	50.28	89.84	99.41	97.35	95.13
Table 3: Combination of USS and two sample-to-class based methods.
Compatibility. In Table 1 and Table 2, we have shown that the proposed  $L_{\mathrm{uss}}$  is superior to other sample-to-sample based losses and can be boosted by adding a proper margin. In Table 3, we investigate the effects of combining  $L_{\mathrm{uss}}$  with two marginal softmax losses, i.e., ArcFace and CosFace. We find that the model trained with the combined loss significantly outperforms the original ArcFace and CosFace models, as well as our USS loss. For example, the model trained with ArcFace + USS respectively outperforms ArcFace and USS with  $6.71\%$  and  $6.37\%$  gains on MR-ALL, and  $41.07\%$  and  $5.64\%$  gains on IJB-C. The model trained with CosFace + USS respectively outperforms CosFace and USS with  $5.16\%$  and  $7.73\%$  gains on MR-ALL, and  $33.19\%$  and  $5.92\%$  gains on IJB-C. Such significant improvements suggest the compatibility of our USS loss. In this work, we refer to the model trained using a combination of a sample-to-class method and our USS as UniTSFace. For our experiments, we choose to use CosFace as the sample-to-class method.
Figure 1: Threshold distributions of  $L_{\mathrm{soft}}$ ,  $L_{\mathrm{bce}}$ , and  $L_{\mathrm{uss}}$  on 20 random selected identities in CASIA-WebFace. Each dot denotes the optimal threshold for each identity. The dashed blue line represents the threshold that was learned solely by  $L_{\mathrm{uss}}$  with  $t = \frac{b}{\gamma} = 31.3344 / 64 = 0.4896$  in Eq. (11).
Method	P	R	Iden.	Veri.	Method	P	R	Iden.	Veri.
Softmax Loss (18)	S	✗	54.85	65.92	ArcFace (7)	L	✗	81.03	96.98
Triplet Loss (18; 22)	S	✗	64.79	78.32	CurricularFace (12)	L	✗	81.26	97.26
Contrastive (18; 25)	S	✗	65.21	78.86	CosFace (30)	L	✗	82.72	96.65
Center (18; 32)	S	✗	65.49	80.14	UniTSFace	L	✗	85.01	97.85
L-Softmax (18; 19)	S	✗	67.12	80.42	SphereFace2 (31)	L	✓	89.84	91.94
SphereFace (18)	S	✗	72.72	85.56	CosFace (7; 30)	L	✓	97.91	97.91
SphereFace+ (16)	S	✗	73.03	-	SphereFace (18)	L	✓	98.16	98.46
CosFace (30)	S	✗	77.11	89.88	ArcFace (7)	L	✓	98.35	98.48
ArcFace (7)	S	✗	77.50	92.34	Circle Loss (26)	L	✓	98.50	98.73
CurricularFace (12)	S	✗	77.65	92.91	CurricularFace (12)	L	✓	98.71	98.64
UniTSFace	S	✗	77.41	93.50	VPL (8)	L	✓	98.80	98.97
ArcFace (7)	S	✓	91.75	93.69	Partial FC (2)	L	✓	98.94	99.10
CurricularFace (12)	S	✓	92.48	94.55	UNPG (13)	L	✓	99.27	-
UniTSFace	S	✓	92.36	94.84	UniTSFace	L	✓	99.27	99.19
Table 4: Comparisons between different methods on the MegaFace Challenge 1. The letter  $\mathbf{P}$  indicates the 'Small' or 'Large' protocols,  $\mathbf{R}$  denotes whether the label refinement is used. All reported results, with the exception of our own, were directly taken from their respective papers.
Method	Net. Data.	MFR	IJB-C	Verification	Acc.
Mask	Child.	Afri.	Cau.	S-A.	E-A.	MR-All	1e-4	1e-5	LFW	CFP	Age
Contrastive(4)	6.67	10.58	12.40	18.84	13.57	10.38	12.38	58.47	46.75	95.50	74.65	82.28
(N+1)-Tuplet (24)	26.56	28.23	39.16	50.92	47.71	24.06	38.11	83.60	73.93	99.18	96.32	94.18
ArcFace(7)	38.52	31.42	45.87	63.69	59.85	7.66	42.21	48.49	9.18	99.31	97.07	94.51
CosFace(30)	38.79	31.33	48.06	63.56	58.71	15.08	45.12	56.65	11.30	99.36	97.30	94.98
Sphere-Rv1(17)	R50	32.80	28.09	40.24	57.24	50.38	22.30	39.92	86.35	75.81	99.38	96.95	94.48
SphereFace2(31)	CASIA	35.40	30.55	46.65	62.69	56.23	26.65	44.20	88.41	79.18	99.46	97.42	94.96
VPL(8)	33.86	31.39	46.52	59.93	54.07	27.18	47.02	88.44	81.38	99.30	97.07	94.75
AnchorFace(15)	37.04	32.28	49.60	63.17	59.80	28.88	48.44	88.81	77.82	99.56	97.48	95.18
UNPG(13)	38.62	33.24	49.94	63.85	59.60	29.21	48.66	88.17	77.73	99.45	97.25	94.83
UniTSFace	37.98	31.73	51.45	64.89	59.73	29.56	50.28	89.84	82.64	99.41	97.35	95.13
Partial FC (1)	R50	72.28	-	84.86	91.57	88.57	67.52	86.85	-	-	-	-	-
UniTSFace	WF4M	75.93	72.00	88.17	93.68	91.40	70.55	89.65	97.03	95.18	99.80	99.04	97.93
Partial FC (1)	R200	91.87	-	97.79	98.70	98.54	89.52	97.70	97.97	96.93	99.83	99.51	98.70
UniTSFace	WF42M	92.87	93.51	98.35	99.03	98.99	90.76	98.16	97.99	97.00	99.83	99.47	98.71
Table 5: Comparisons between different methods on MFR-Ongoing.
5 Comparison with the State-Of-The-Art
MegaFace Challenge 1. In Table 4, we compare both the identification and verification performance of our UniTSFace with several state-of-the-art methods on MegaFace Challenge 1. These methods include sample-to-sample based Contrastive Loss (4; 10) and Triplet Loss (22), sample-to-class based CosFace(30) and ArcFace(7), as well as the hybrid/combined approaches: VPL(8) and UNPG(13).
To ensure a fair comparison, as per the official protocols, we compare our UniTSFace trained on the CASIA-WebFace with the models trained on 'Small' datasets, while UniTSFace trained on Glint360K is compared with the models trained on 'Large' datasets in Table 4. When label refinement (7) is not used, our UniTSFace achieves the highest accuracy among the compared models trained on 'Large' datasets, with an  $85.01\%$  identification accuracy and a  $97.85\%$  verification accuracy. When label refinement (7) is used, the accuracies can be further increased to  $99.27\%$  and  $99.19\%$  respectively. Though our UniTSFace is slightly lower than CurricularFace (12) in terms of identification performance on 'Small' datasets, UniTSFace achieves the highest accuracy on the verification track, regardless of whether label refinement is used or not. These results demonstrate the effectiveness of our UniTSFace in both identification and verification tasks.
MFR Ongoing Benchmarks. We then compare the proposed UniTSFace with Contrastive Loss(4; 10),  $(\mathrm{N} + 1)$ -Tuplet Loss(24), CosFace(30), ArcFace(7), VPL(8), AnchorFace(15), and UNPG(13) on the MFR Ongoing benchmark. We re-implement these methods with the optimal hyper-parameters recommended in their original papers. All the compared models are trained with a ResNet-50 backbone and the CASIA-WebFace dataset. As reported in Table 5, our UniTSFace achieves clear improvement over other methods.
Additionally, we compare our UniTSFace with the recent Partial FC(1), which is the leading method on the MFR ongoing challenge. Following the settings of Partial FC(1), we train the proposed UniTSFace on the WebFace4M and WebFace42M datasets using two different architectures, i.e., ResNet-50 and ResNet-200. We can observe that, on average, our performance is better than that of the Partial FC. Till the submission of this work (May 17 '23), the proposed UniTSFace ranks first place on the academic track of the MFR-ongoing leaderboard.
1) Though the threshold range of USS is narrowed compared to other losses in Fig. 1, is it correct to claim the word "unified"? Firstly, our theoretical objective is to learn a unified threshold that satisfies Eq.(3) during the training of a facial model. Therefore, we first assume the existence of such a unified threshold, and then propose the unified threshold integrated sample-to-sample (USS) loss. We have proven through our analysis in Section 3.1 that, ideally, a model trained by USS could learn a unified threshold for the training dataset.
However, we must admit that achieving this ideal goal is subjective to the model capacity, training hyper-parameters, and even the training dataset itself, which are all independent from our USS loss. For example, if the backbone network only uses one single linear neural layer, our USS loss definitely cannot guarantee a unified threshold either. In Fig. 1, using the same backbone architecture and training hyper-parameters, i) our USS loss is able to achieve a more compact threshold distribution than the other losses, moreover, and ii) the learned threshold denoted by the blue dashed line lies around the median of the boxplot, these two observations suggest the superiority of imposing the unified threshold and are consistent with our expectations.
2) In the testing stage, how to determine the threshold? The threshold learned by the training stage cannot be directly used in testing. In the testing stage, the threshold is determined according to the specific testing criteria. For example, when reporting the 1:1 verification accuracy on LFW, CFP-FP, AgeDB, 10-fold validation is used. We first select the threshold that achieves the highest accuracy in the first 9 folds and then adopt this threshold to calculate the accuracy in the leave-out fold.
3) How does UniTSFace compare to other methods such as CosFace in terms of computational efficiency and memory usage? UniTSFace utilizes the ResNet architecture as its backbone and optimizes the parameters using the algorithmic average of the cosine-margin Softmax loss and the proposed USS loss. When compared to CosFace, the extra computational cost brought by USS loss is relatively small, as the computational consumption and memory usage mostly depend on the convolutional operations inherent to the selected network architecture and the resolutions of the input images. In experiments, we indeed found that the computational differences between these methods during both the testing and training stages are negligible.
We propose the USS loss for deep face recognition by explicitly defining a unified threshold that separates all positive sample-to-sample pairs from negative ones. Though this unified threshold learned in the training stage cannot be directly applied to the testing stage, the model trained by USS is desired to extract more discriminative features and subsequently improve the face recognition performance in various testing scenarios, which have been demonstrated with extensive experiments.
Furthermore, the proposed USS loss can be effortlessly extended to the Marginal USS loss and can also be seamlessly combined with other sample-to-class losses. In our experiments, we have combined the USS loss with ArcFace and CosFace methods, and the combined approaches consistently surpass their respective individual counterparts. We denote the fusion of the CosFace and USS losses as "UniTSFace" which we have compared with other sophisticated combinations such as VPL, UNPG, and AnchorFace. The experimental results on multiple benchmark datasets further suggest the superiority of our UniTSFace.
In conclusion, we believe our USS loss provides the research community with a much more versatile and effective solution for face recognition tasks.
This work was supported by the National Natural Science Foundation of China under Grants 82261138629 and 62006156, Guangdong Basic and Applied Basic Research Foundation under Grants 2023A1515010688 and 2022A1515012125, and Shenzhen Municipal Science and Technology Innovation Council under Grant JCYJ20220531101412030.
Appendix for USS
A Inequalities about  $L_{\mathrm{naive}}$
We here present the detailed derivations for the inequalities about the naive loss  $L_{\mathrm{naive}}$ . For any sample  $\mathbf{X}^{(i)}$  captured from subject  $i$ , with feature  $\mathbf{x}^{(i)} = \mathcal{M}(\mathbf{X}^{(i)})$ , the naive loss  $L_{\mathrm{naive}}$  comprises of one positive sample-to-sample similarity and  $N - 1$  negative similarities,
$$
L _ {\text {n a i v e}} \left(\boldsymbol {X} ^ {(i)}\right) = - \gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(i)}\right) + \frac {1}{N - 1} \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(j)}\right), \tag{25}
$$
Using the inequality of arithmetic and geometric means, i.e.,  $\sqrt[n]{\prod_{i=1}^{n} a_i} \leq \frac{1}{n} \sum_{i=1}^{n} a_i$  for  $a_i \geq 0$ , we derive four inequalities about  $L_{\mathrm{naive}}$ .
$$
\begin{array}{l} \sum_ {i = 1} ^ {N} L _ {\text {n a i v e}} \left(\boldsymbol {X} ^ {(i)}\right) \\ = \frac {1}{N - 1} \left(\sum_ {i = 1} ^ {N} \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(j)}\right) - (N - 1) \sum_ {i = 1} ^ {N} \gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(i)}\right)\right) \tag{46} \\ \end{array}
$$