SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qiufu Li 1,2,6,#
liqiufu@szu.edu.cn
Xi Jia 1,2,3,#
x.jia.1@cs.bham.ac.uk
Jiancan Zhou 1,2,4,#
zhoujiancan@foxmail.com
Linlin Shen 1,2,6,*
llshen@szu.edu.cn
Jinming Duan 3,5
j.duan@bham.ac.uk
$^{1}$  National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University, China
$^{2}$  Computer Vision Institute, Shenzhen University, China
$^{3}$  School of Computer Science, University of Birmingham, UK
<sup>4</sup> Aqara, Lumi United Technology Co., Ltd, China
5 Alan Turing Institute, UK
$^{6}$  SZU Branch, Shenzhen Institute of Artificial Intelligence and Robotics for Society, China
Sample-to-class-based face recognition models can not fully explore the cross-sample relationship among large amounts of facial images, while sample-to-sample-based models require sophisticated pairing processes for training. Furthermore, neither method satisfies the requirements of real-world face verification applications, which expect a unified threshold separating positive from negative facial pairs. In this paper, we propose a unified threshold integrated sample-to-sample based loss (USS loss), which features an explicit unified threshold for distinguishing positive from negative pairs. Inspired by our USS loss, we also derive the sample-to-sample based softmax and BCE losses, and discuss their relationship. Extensive evaluation on multiple benchmark datasets, including MFR, IJB-C, LFW, CFP-FP, AgeDB, and MegaFace, demonstrates that the proposed USS loss is highly efficient and can work seamlessly with sample-to-class-based losses. The embedded loss (USS and sample-to-class Softmax loss) overcomes the pitfalls of previous approaches and the trained facial model UniTSFace exhibits exceptional performance, outperforming state-of-the-art methods, such as CosFace, ArcFace, VPL, AnchorFace, and UNPG. Our code is available at https://github.com/CVI-SZU/UniTSFace.
Modern deep facial recognition systems, involving an enormous number of facial images and identities, essentially rely on discriminative feature learning: the facial images from the same identity should be close while those from different identities should be distant in the feature space. That is, the similarity of a positive pair (two facial images from the same identity) is required to be larger than any negative pair (two facial images from different identities). In other words, a unified threshold is expected to distinguish positive from negative pairs.
The feature learning process of general deep face recognition models is either based on sample-to-class losses (such as Softmax loss (27; 28)) or sample-to-sample losses (such as contrastive (4; 10; 25)
Equal contribution; *Corresponding author. Parts of the work were done when X. J and J. Z were students at Shenzhen University.
and triplet loss (22)). Inspired by the success of large-scale image classification, the softmax loss and its extensions have become popular in deep face recognition systems. In the softmax loss, each weight vector can be regarded as a proxy of the corresponding class (face identity), the classification-based face recognition models are demanded to learn the class proxy and image features simultaneously. However, these models possess a significant drawback, namely a domain gap between the training and testing stages of face models. This gap arises from the reliance on limited identity proxies for computing and optimizing feature similarity in sample-to-class models. Conversely, in real-world scenarios, feature similarity is computed and compared across various samples from diverse facial identities. In other words, the sample-to-class-based classification strategy may not entirely explore the variances across samples (Problem 1). Therefore, its efficacy in accurately reflecting real-world scenarios is questionable. To tackle this drawback, VPL (8) considers a small variation around the class proxy, which implicitly introduces more samples during the optimization and improves the face recognition performance. However, while this small variation extends the capability of softmax loss, it cannot essentially represent all the samples in this class.
Face models based on sample-to-sample losses (4; 10; 22; 3; 24) learn facial identity features and optimize feature similarities by comparing positive and negative samples, which is closer to real-world face recognition applications than sample-to-class losses. The majority of sample-to-sample losses aim to maximize inter-class discrepancy while minimizing intra-class distances, which may need a meticulous sampling/paring step for every mini-batch. Moreover, in face verification tasks, a single threshold is required to distinguish positive facial pairs from negative ones. Unfortunately, none of the aforementioned methods incorporate such an explicit constraint (Problem 2).
In this study, we commence our research by analyzing a reasonable, albeit naive, sample-to-sample loss, which enables us to straightforwardly investigate the variances across facial samples, thereby resolving Problem 1. To address Problem 2, we integrate a unified learnable threshold into the naive loss, resulting in a novel sample-to-sample loss function that we term the unified threshold integrated sample-to-sample (USS) loss. We provide a mathematical elaboration of the relationship between our USS loss and the sample-to-sample BCE loss and softmax loss, from the perspective of the naive sample-to-sample loss. To evaluate the effectiveness of the USS loss, we conduct experiments on various benchmark datasets and qualitatively demonstrate that it meets the requirements of real-world applications. Additionally, we find that incorporating a margin can easily enhance the USS loss. Our USS loss also works seamlessly with the sample-to-class based losses and shows significant improvements when they are combined together to train a face model called UniTSFace. To summarize, the contributions of this work are
- We introduce a unified threshold integrated sample-to-sample loss (USS) derived from a reasonable naive loss for face recognition. We also derive the sample-to-sample based softmax and binary-cross-entropy (BCE) losses from the naive loss and reveal the mathematical relationship among the softmax, BCE, and our USS losses.
- We demonstrate that a unified threshold can be learned by adopting the proposed USS loss and quantitatively and qualitatively demonstrate that the learned threshold aligns with face verification expectations in experiments.
- The proposed USS loss can be enhanced with an auxiliary margin and is compatible with existing sample-to-class based losses. We show that USS loss, when used jointly with sample-to-class based losses such as CosFace and ArcFace, leads to a continuous improvement.
- Our UniTSFace outperforms state-of-the-art methods on the Megaface 1 dataset and ranks first place on the MFR ongoing challenge till the submission of this work (May 17 '23, academic track): http://iccv21-mfr.com/#/leaderboard/academic.
Sample-to-Sample based Methods. DeepID2 (25) uses a contrastive loss(4; 10) to encourage the features learned from the same identity to be close while that learned from different identities are distant. FaceNet(22) constructs three-element tuples and minimizes the distance between an anchor and a positive sample and maximizes the distance between the anchor and a negative sample. The effectiveness of contrastive/triplet losses relies on the meticulous selection of pairs/triplets. Furthermore, neither method explicitly enforces that all positive sample-to-sample similarities are greater than negative similarities.
Sample-to-Class based Methods. While Softmax loss is frequently utilized in deep recognition models, it only promotes separability and does not learn discriminative features. Various approaches have been proposed to enhance the feature learning of softmax loss. Some methods (32; 9; 34; 16) proposed extra constraints imposed on the softmax loss, such methods normally have a class proxy (class center/prototype) and maximize (minimize) the similarity (distances) between a facial feature and its corresponding class proxy (32), as well as maximize the distances between all class proxies(16; 9; 34). However, the training process of such methods needs careful balancing of the softmax loss and the extra constraints. Alternatively, some works directly improve the softmax by normalizing the facial features and adding margins between positive and negative sample-to-class pairs (18; 29; 30; 7; 17). While sample-to-class methods demonstrate excellent performance in deep face verification, they may not entirely explore the variability across various facial samples.
Hybrid (/combined) Methods. Circle loss (26) is one of the first works that discussed the two elemental learning paradigms, i.e., learning with class-level labels and pair-wise labels, in a unified framework. However, the two learning paradigms are respectively used in circle loss. Following (26), the combination of the two learning paradigms to solve the shortcoming of either method become popular, e.g., VPL(8), AnchorFace(15), and UNPG(13). VPL extends the softmax loss by considering a small feature variation around the class proxy  $W$ . The embedding of a small feature variation cannot essentially represent all samples, VPL still presents a large gap between training a face recognition model and testing over the open sets. UNPG jointly optimizes the distance between a sample  $x_{i}$  with negative class proxies  $W_{j}$  and negative samples  $x_{j}$  in the same softmax loss using batch processing, focusing on the negative pairs during training. AnchorFace, however, uses a combination of softmax loss and different sample-to-sample losses, such as TAR Loss and FAR Loss, to target specific testing protocols. In contrast, our USS loss is more general and compatible with existing sample-to-class losses. We demonstrate our USS loss can lead to a steady improvement when used with sample-to-class losses jointly in experiments.
Suppose  $\mathcal{M}$  is a deep face model trained on a facial sample set  $\mathcal{D} = \bigcup_{i=1}^{N} \mathcal{D}_i$  captured from  $N$  subjects, where  $\mathcal{D}_i$  denotes the subset containing the facial samples captured from the same subject  $i$ . Then, we can get a feature set  $\mathcal{F} = \bigcup_{i=1}^{N} \mathcal{F}_i = \bigcup_{i=1}^{N} \left\{\boldsymbol{x}^{(i)} = \mathcal{M}(\boldsymbol{X}^{(i)}): \boldsymbol{X}^{(i)} \in \mathcal{D}_i\right\}$ , where  $\boldsymbol{x}^{(i)}$  is the feature vector of the sample  $\boldsymbol{X}^{(i)}$  and  $\boldsymbol{X}^{(i)}$  denotes the sample captured from subject  $i$ .
For any two samples  $X, X_* \in \mathcal{D}$ , we apply a bivariate operator  $g(\boldsymbol{x}, \boldsymbol{x}_*) \in [-1, 1]$  denoting their feature similarity, where  $\boldsymbol{x} = \mathcal{M}(\boldsymbol{X})$ ,  $\boldsymbol{x}_* = \mathcal{M}(\boldsymbol{X}_*)$  are features of  $X, X_*$ . We term  $g(\boldsymbol{x}, \boldsymbol{x}_*)$  the positive sample-to-sample similarity if the two samples are captured from the same subject, while the negative sample-to-sample similarity if they are from different subjects. Then, for any sample  $X^{(i)} \in \mathcal{D}_i$ ,  $\forall i$ , we define its positive and negative similarity sets as
$$
\boldsymbol {s} _ {\boldsymbol {X} ^ {(i)}} ^ {(\text {p o s})} = \left\{g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*}\right): \boldsymbol {x} _ {*} \in \mathcal {F} _ {i} \right\}, \tag {1}
$$
$$
\boldsymbol {s} _ {\boldsymbol {X} ^ {(i)}} ^ {(\text {n e g})} = \bigcup_ {\substack {j = 1 \\ j \neq i}} ^ {N} \left\{g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*}\right): \boldsymbol {x} _ {*} \in \mathcal {F} _ {j} \right\}, \tag{2}
$$
where  $\pmb{x}^{(i)} = \mathcal{M}(\pmb{X}^{(i)})$ .
In real applications of face verification, a threshold  $\hat{t} \in [-1, 1]$  is chosen to verify whether two samples  $\mathbf{X}$  and  $\mathbf{X}_{*}$  are from the same subject or not. Specifically, the two facial samples  $\mathbf{X}, \mathbf{X}_{*}$  are with the same identity if  $g(\mathbf{x}, \mathbf{x}_{*}) \geq \hat{t}$ , while they are from two different identities when  $g(\mathbf{x}, \mathbf{x}_{*}) < \hat{t}$ .
To be in line with face verification applications, during the training process of  $\mathcal{M}$ , we expect a unified threshold  $t$  such that any two samples  $(\pmb{X}^{(i)} \in \mathcal{D}_i$  and  $\pmb{X}_{*}^{(j)} \in \mathcal{D}_{j})$  conform to the above rules. If they share the same identity (i.e.,  $i = j$ ), then  $g(\pmb{x}^{(i)}, \pmb{x}_{*}^{(j)}) \geq t$ , in the case of  $i \neq j$ ,  $g(\pmb{x}^{(i)}, \pmb{x}_{*}^{(j)}) < t$ , where  $\pmb{x}^{(i)} = \mathcal{M}(\pmb{X}^{(i)})$ ,  $\pmb{x}_{*}^{(j)} = \mathcal{M}(\pmb{X}_{*}^{(j)})$ . The unified threshold  $t$  satisfies
$$
\max  \left(\bigcup_ {i = 1} ^ {N} \bigcup_ {\boldsymbol {X} ^ {(i)} \in \mathcal {D} _ {i}} \boldsymbol {s} _ {\boldsymbol {X} ^ {(i)}} ^ {(\text {n e g})}\right) <   t \leq \min  \left(\bigcup_ {i = 1} ^ {N} \bigcup_ {\boldsymbol {X} ^ {(i)} \in \mathcal {D} _ {i}} \boldsymbol {s} _ {\boldsymbol {X} ^ {(i)}} ^ {(\text {p o s})}\right). \tag {3}
$$
However, the existing sample-to-sample losses (4; 10; 22; 24) fail to explicitly include and learn the unified threshold. In this paper, by explicitly defining the unified threshold  $t$ , we design a unified threshold integrated sample-to-sample loss.
3.1 Unified Threshold Integraed Sample-to-Sample Loss (USS Loss)
Clearly, in the training of model  $\mathcal{M}$ , it expects large positive sample-to-sample similarities but small negative ones, and then, for any sample  $X^{(i)}$ , a naive loss could be reasonably defined as,
$$
L _ {\text {N A I V E}} \left(\boldsymbol {X} ^ {(i)}\right) = - \frac {1}{\left| \mathcal {F} _ {i} \right|} \sum_ {\boldsymbol {x} \in \mathcal {F} _ {i}} \gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x}\right) + \frac {1}{\left| \mathcal {F} - \mathcal {F} _ {i} \right|} \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \sum_ {\boldsymbol {x} \in \mathcal {F} _ {j}} \gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x}\right), \tag{4}
$$
where  $\gamma$  is a scale factor,  $|\mathcal{S}|$  denotes the element number of a set  $\mathcal{S}$ , and  $\mathcal{F} - \mathcal{F}_i$  denotes the complementary set of  $\mathcal{F}_i$  in  $\mathcal{F}$ , i.e.,  $\mathcal{F} - \mathcal{F}_i = \bigcup_{\substack{j = 1\\ j\neq i}}^{N}\mathcal{F}_j$ .
The loss in Eq. (4) computes the feature similarities of all positive sample-to-sample pairs and all negative ones for the sample  $\mathbf{X}^{(i)}$ , which is not easily implemented in practice. In this paper, without loss of generality, we consider the feature similarities of one positive sample pair and  $N - 1$  negative sample pairs for the sample  $\mathbf{X}^{(i)}$  in the naive loss,
$$
L _ {\text {n a i v e}} \left(\boldsymbol {X} ^ {(i)}\right) = - \gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(i)}\right) + \frac {1}{N - 1} \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(j)}\right), \tag{5}
$$
where  $\pmb{x}_{*}^{(i)} = \mathcal{M}(\pmb{X}_{*}^{(i)})$ ,  $\pmb{x}_{*}^{(j)} = \mathcal{M}(\pmb{X}_{*}^{(j)})$ , and  $\pmb{X}_{*}^{(i)}$  and  $\pmb{X}_{*}^{(j)}$  are randomly taken from the subject  $i, j$ , with  $j \neq i$ .
Using the inequality of arithmetic and geometric means<sup>1</sup>, we derive two inequalities about  $L_{\mathrm{naive}}$ ,
$$
L _ {\text {n a i v e}} \left(\boldsymbol {X} ^ {(i)}\right) \leq 2 \log \left(1 + \frac {\exp \left(\sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \frac {\gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(j)}\right)}{N - 1}\right)}{\exp \left(\gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(i)}\right)\right)}\right) - 2 \log 2, \tag{6}
$$
$$
L _ {\text {n a i v e}} \left(\boldsymbol {X} ^ {(i)}\right) \leq \frac {2}{N - 1} \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \log \left(1 + \frac {\mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(j)}\right)}}{\mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(i)}\right)}}\right) - 2 \log 2. \tag{7}
$$
According to Eqs. (6), (7), and the unified threshold  $t$  in Eq. (3), one can get
$$
\begin{array}{l} \frac {N}{2} L _ {\mathrm {n a i v e}} \left(\boldsymbol {X} ^ {(i)}\right) + N \log 2 \\ \leq \log \left(1 + \frac {\exp \left(\sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \frac {\gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(j)}\right)}{N - 1}\right)}{\exp \left(\gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(i)}\right)\right)}\right) + \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \log \left(1 + \frac {\exp \left(\gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(j)}\right)\right)}{\exp \left(\gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(i)}\right)\right)}\right) (8) \\ \leq \log \left(1 + \frac {\exp \left(\sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \frac {\gamma t}{N - 1}\right)}{\exp \left(\gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(i)}\right)\right)}\right) + \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \log \left(1 + \frac {\exp \left(\gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(j)}\right)\right)}{\exp (\gamma t)}\right) (9) \\ = \log \left(1 + \mathrm {e} ^ {- \gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(i)}\right) + \gamma t}\right) + \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \log \left(1 + \mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(j)}\right) - \gamma t}\right). (10) \\ \end{array}
$$
We define the Unified threshold integrated Sample-to-Sample (USS) loss  $L_{\mathrm{uss}}(\pmb{X}^{(i)})$  as
$$
L _ {\mathrm {u s s}} \left(\boldsymbol {X} ^ {(i)}\right) = \log \left(1 + \mathrm {e} ^ {- \gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(i)}\right) + b}\right) + \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \log \left(1 + \mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(j)}\right) - b}\right) \tag{11}
$$
$\sqrt[n]{\prod_{i=1}^{n} a_i} \leq \frac{1}{n} \sum_{i=1}^{n} a_i$  for  $a_i \geq 0$ .
where  $b = \gamma t$  is a constant to be learned (depicted in Fig. 1). The detailed derivations of the above inequalities are described in the supplementary (appendix).
We here analyze that the unified threshold  $t$  could be learned. Suppose that the model  $\mathcal{M}$  has been perfectly trained, and  $L_{\mathrm{uss}}$  has reached its minimum point after the training, which means (i) the positive sample-to-sample similarity  $g(\pmb{x}^{(i)},\pmb{x}_{*}^{(i)})$  tends to 1, and the negative ones  $g(\pmb{x}^{(i)},\pmb{x}_{*}^{(j)})$  tends to -1; and (ii)  $L_{\mathrm{uss}}$  reaches its stationary point in terms of variable  $b$ . From (ii), one can deduce that
$$
\begin{array}{l} 0 = \frac {\partial L _ {\mathrm {u s s}}}{\partial b} = \frac {\mathrm {e} ^ {- \gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(i)}\right) + b}}{1 + \mathrm {e} ^ {- \gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(i)}\right) + b}} - \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \frac {\mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(j)}\right) - b}}{1 + \mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(j)}\right) - b}} (12) \\ \stackrel {\text {(i)}} {=} \frac {\mathrm {e} ^ {- \gamma + b}}{1 + \mathrm {e} ^ {- \gamma + b}} - \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \frac {\mathrm {e} ^ {- \gamma - b}}{1 + \mathrm {e} ^ {- \gamma - b}} (13) \\ \Rightarrow b = \log \frac {(N - 2) e ^ {- \gamma} + \sqrt {(N - 2) ^ {2} e ^ {- 2 \gamma} + 4 (N - 1)}}{2}. (14) \\ \end{array}
$$
If  $N < \frac{\mathrm{e}^{2\gamma} + 3}{2}$ , the final learned threshold  $t = \frac{b}{\gamma}$  will locate between -1 and 1, which means now the leaned  $t$  has correctly separated the negative sample-to-sample similarities,  $g(\pmb{x}^{(i)},\pmb{x}_{*}^{(j)})$ , and the positive ones,  $g(\pmb{x}^{(i)},\pmb{x}_{*}^{(i)})$ . In the practice, following (30; 7), we set  $\gamma = 64$ , then, according to the above analysis, the unified threshold  $t$  could be leaned if  $N < 1.9\times 10^{55}$ .
3.2 Sample-to-Sample Based Softmax and BCE Losses
Sample-to-class based softmax and BCE losses are widely applied in the image classification. For the study of face verification, we here deduce the sample-to-sample based softmax and BCE losses from the naive loss  $L_{\mathrm{naive}}$ .
Similar to the deduction of USS loss, using the inequality of arithmetic and geometric means, we first present another two inequalities about  $L_{\mathrm{naive}}$ ,
$$
L _ {\text {n a i v e}} \left(\boldsymbol {X} ^ {(i)}\right) \leq - \frac {N}{N - 1} \log \frac {\mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(i)}\right)}}{\sum_ {j = 1} ^ {N} \mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(j)}\right)}} - \frac {N \log N}{N - 1}, \tag {15}
$$
$$
\sum_ {i = 1} ^ {N} L _ {\text {n a i v e}} \left(\boldsymbol {X} ^ {(i)}\right) \leq \frac {2}{N - 1} \sum_ {i = 1} ^ {N} \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \log \left(1 + \frac {\mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(j)}\right)}}{\mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(j)}, \boldsymbol {x} _ {*} ^ {(j)}\right)}}\right) - 2 N \log 2. \tag{16}
$$
Softmax Loss. For sample  $\mathbf{X}^{(i)} \in \mathcal{D}_i$  with  $\mathbf{x}^{(i)} = \mathcal{M}(\mathbf{X}^{(i)})$ , we define its sample-to-sample softmax loss as
$$
L _ {\text {s o f t}} \left(\boldsymbol {X} ^ {(i)}\right) = - \log \frac {\mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(i)}\right)}}{\sum_ {j = 1} ^ {N} \mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(j)}\right)}}. \tag {17}
$$
Then, according to Eq. (15), one can get
$$
L _ {\text {n a i v e}} \left(\boldsymbol {X} ^ {(i)}\right) \leq \frac {N}{N - 1} L _ {\text {s o f t}} \left(\boldsymbol {X} ^ {(i)}\right) - \frac {N}{N - 1} \log N. \tag {18}
$$
Similar to the naive loss  $L_{\mathrm{naive}}$ , the design of softmax loss  $L_{\mathrm{soft}}$  does not consider the unified threshold among the sample-to-sample pairs.
BCE Loss. For all samples captured from subject  $i$ , we assume the existence of a threshold  $t_i$  which could separate their all positive sample-to-sample pairs and negative ones, i.e.,
$$
\max  \left(\bigcup_ {\boldsymbol {X} ^ {(i)} \in \mathcal {D} _ {i}} \boldsymbol {s} _ {\boldsymbol {X} ^ {(i)}} ^ {(\text {n e g})}\right) <   t _ {i} \leq \min  \left(\bigcup_ {\boldsymbol {X} ^ {(i)} \in \mathcal {D} _ {i}} \boldsymbol {s} _ {\boldsymbol {X} ^ {(i)}} ^ {(\text {p o s})}\right), \quad \forall i, \tag {19}
$$
then, according to Eqs. (16) and (6),
$$
\begin{array}{l} \frac {N}{2} \sum_ {i = 1} ^ {N} L _ {\text {n a i v e}} \left(\boldsymbol {X} ^ {(i)}\right) + N ^ {2} \log 2 \\ \leq \sum_ {i = 1} ^ {N} \left[ \log \left(1 + \frac {\exp \left(\sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \frac {\gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(j)}\right)}{N - 1}\right)}{\exp \left(\gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(i)}\right)\right)}\right) + \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \log \left(1 + \frac {\exp \left(\gamma g \left(\boldsymbol {x} ^ {(i)} , \boldsymbol {x} _ {*} ^ {(j)}\right)\right)}{\exp \left(\gamma g \left(\boldsymbol {x} ^ {(j)}, \boldsymbol {x} _ {*} ^ {(j)}\right)\right)}\right) \right] (20) \\ \leq \sum_ {i = 1} ^ {N} \left[ \log \left(1 + \mathrm {e} ^ {- \gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(i)}\right) + \gamma t _ {i}}\right) + \sum_ {\substack {j = 1 \\ j \neq i}} ^ {N} \log \left(1 + \mathrm {e} ^ {\gamma g \left(\boldsymbol {x} ^ {(i)}, \boldsymbol {x} _ {*} ^ {(j)}\right) - \gamma t _ {j}}\right) \right]. (21) \\ \end{array}
$$
We define BCE loss for sample  $\mathbf{X}^{(i)}$  as