SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: Visualization of representation space for toy example. (a) Theoretically contrasting the feature formed by top-3 eigenvectors of  $\tilde{A}^{(u)}$  and  $\tilde{A}$  respectively. (b) UMAP visualization of the features learned without (left) and with labeled information (right). Details are in Appendix B (eigenvector calculation) and Appendix E.1 (visualization setting).
4.2 Main Theory
The toy example offers an important insight that the added labeled information is more helpful for the class with a stronger connection to the known class. In this section, we formalize this insight by extending the toy example to a more general setting. As a roadmap, we derive the result through three steps: (1) derive the closed-form solution of the learned representations; (2) define the clustering performance by the K-means measure; (3) contrast the resulting clustering performance before and after adding labels. We start by deriving the representations.
4.2.1 Learned Representations in Analytic Form
Representation without labels. To obtain the representations, one can train the neural network  $f: \mathcal{X} \mapsto \mathbb{R}^k$  using the spectral loss defined in Equation 6. We assume that the optimizer is capable to obtain the representation  $Z^{(u)} \in \mathbb{R}^{N \times k}$  that minimizes the loss, where each row vector  $\mathbf{z}_i = f(x_i)^\top$ . Recall that Theorem 3.1 allows us to derive a closed-form solution for the learned feature space by the spectral decomposition of the adjacency matrix, which is  $\tilde{A}^{(u)}$  in the case without labeling information. Specifically, we have  $F_k^{(u)} = \sqrt{D^{(u)}} Z^{(u)}$ , where  $F_k^{(u)}F_k^{(u)\top}$  contains the
3 When  $\tau_{1}\gg \tau_{c} > \tau_{s} > 0$  , the top-3 eigenvectors are almost equivalent to the feature embedding.
top- $k$  components of  $\tilde{A}^{(u)}$ 's SVD decomposition and  $D^{(u)}$  is the diagonal matrix defined based on the row sum of  $A^{(u)}$ . We further define the top- $k$  singular vectors of  $\tilde{A}^{(u)}$  as  $V_{k}^{(u)} \in \mathbb{R}^{N \times k}$ , so we have  $F_{k}^{(u)} = V_{k}^{(u)} \sqrt{\Sigma_{k}^{(u)}}$ , where  $\Sigma_{k}^{(u)}$  is a diagonal matrix of the top- $k$  singular values of  $\tilde{A}^{(u)}$ . By equalizing the two forms of  $F_{k}^{(u)}$ , the closed-formed solution of the learned feature space is given by  $Z^{(u)} = [D^{(u)}]^{-\frac{1}{2}} V_{k}^{(u)} \sqrt{\Sigma_{k}^{(u)}}$ .
Representation perturbation by adding labels. We now analyze how the representation is "perturbed" as a result of adding label information. We consider  $|\mathcal{V}_l| = 1^4$  to facilitate a better understanding of our key insight. We can rewrite  $A$  in Eq. 3 as:
$$
A (\delta) \triangleq \eta_ {u} A ^ {(u)} + \delta \mathfrak {U} ^ {\top},
$$
where we replace  $\eta_{l}$  to  $\delta$  to be more apparent in representing the perturbation and define  $\mathfrak{l} \in \mathbb{R}^{N}$ ,  $(\mathfrak{l})_x = \mathbb{E}_{\bar{x}_l \sim \mathcal{P}_{l_1}} \mathcal{T}(x|\bar{x}_l)$ . Note that  $\mathfrak{l}$  can be interpreted as the vector of "the semantic connection for sample  $x$  to the labeled data". One can easily extend to  $r$  classes by letting  $\mathfrak{l} \in \mathbb{R}^{N \times r}$ .
Here we treat the adjacency matrix as a function of the perturbation. In a similar manner as above, we can derive the normalized adjacency matrix  $\tilde{A} (\delta)$  and the feature representation  $Z(\delta)$  in closed form. The details are included in Appendix C.4.
4.2.2 Evaluation Target
With the learned representations, we can evaluate their quality by the clustering performance. Our theoretical analysis of the clustering performance can well connect to empirical evaluation strategy in the literature [75] using  $K$ -means clustering accuracy/errror. Formally, we define the ground-truth partition of clusters by  $\Pi = \{\pi_1,\pi_2,\dots,\pi_C\}$ , where  $\pi_{i}$  is the set of samples' indices with underlying label  $y_{i}$  and  $C$  is the total number of classes (including both known and novel). We further let  $\pmb{\mu}_{\pi} = \mathbb{E}_{i\in \pi}\mathbf{z}_i$  be the center of features in  $\pi$ , and the average of all feature vectors be  $\pmb{\mu}_{\Pi} = \mathbb{E}_{j\in [N]}\mathbf{z}_j$ .
The clustering performance of K-means depends on two measurements: Intra-class measure and Inter-class measure. Specifically, we let the intra-class measure be the average Euclidean distance from the samples' feature to the corresponding cluster center and we measure the inter-class separation as the distances between cluster centers:
$$
\mathcal {M} _ {\text {i n t r a - c l a s s}} (\Pi , Z) \triangleq \sum_ {\pi \in \Pi} \sum_ {i \in \pi} \| \mathbf {z} _ {i} - \boldsymbol {\mu} _ {\pi} \| ^ {2}, \mathcal {M} _ {\text {i n t e r - c l a s s}} (\Pi , Z) \triangleq \sum_ {\pi \in \Pi} | \pi | \| \boldsymbol {\mu} _ {\pi} - \boldsymbol {\mu} _ {\Pi} \| ^ {2}. \tag {8}
$$
Strong clustering results translate into low  $\mathcal{M}_{\mathrm{intra - class}}$  and high  $\mathcal{M}_{\mathrm{inter - class}}$ . Thus we define the K-means measure as:
$$
\mathcal {M} _ {k m s} (\Pi , Z) \triangleq \mathcal {M} _ {\text {i n t r a - c l a s s}} (\Pi , Z) / \mathcal {M} _ {\text {i n t e r - c l a s s}} (\Pi , Z). \tag {9}
$$
We also formally show in Theorem 4.1 that the K-means clustering error<sup>5</sup> is asymptotically equivalent to the K-means measure we defined above.
Theorem 4.1. (Relationship between the  $K$ -means measure and  $K$ -means error.) We define the  $\xi_{\pi \to \pi'}$  as the index set of samples that is from class division  $\pi$  however is closer to  $\pmb{\mu}_{\pi'}$  than  $\pmb{\mu}_{\pi}$ . In other words,  $\xi_{\pi \to \pi'} = \{i : i \in \pi, \| \mathbf{z}_i - \pmb{\mu}_{\pi} \|_2 \geq \| \mathbf{z}_i - \pmb{\mu}_{\pi'} \|_2\}$ . Assuming  $|\xi_{\pi \to \pi'}| > 0$ , we define below the clustering error ratio from  $\pi$  to  $\pi'$  as  $\mathcal{E}_{\pi \to \pi'}$  and the overall cluster error ratio  $\mathcal{E}_{\Pi, Z}$  as the Harmonic Mean of  $\mathcal{E}_{\pi \to \pi'}$  among all class pairs:
$$
\mathcal{E}_{\Pi ,Z} = C(C - 1) / \left(\sum_{\substack{\pi \neq \pi^{\prime}\\ \pi ,\pi^{\prime}\in \Pi}}\frac{1}{\mathcal{E}_{\pi\to\pi^{\prime}}}\right),where \mathcal{E}_{\pi \to \pi^{\prime}} = \frac{|\xi_{\pi\to\pi^{\prime}}|}{|\pi^{\prime}| + |\pi|}.
$$
The  $K$ -means measure  $\mathcal{M}_{kms}(\Pi, Z)$  has the same order of the Harmonic Mean of the cluster error ratio between all cluster pairs with proof in Appendix C.3.
$$
\mathcal {E} _ {\Pi , Z} = O \left(\mathcal {M} _ {k m s} (\Pi , Z)\right).
$$
4To understand the perturbation by adding labels from more than one class, one can take the summation of the perturbation by each class.
It is theoretically inconvenient to directly analyze the clustering error since it is a non-differentiable target.
The K-means measure  $\mathcal{M}_{kms}(\Pi, Z)$  have a nice matrix form as shown in Appendix C.2 which facilitates theoretical analysis. Our analysis revolves around contrasting the resulting clustering performance before and after adding labels as we will shown next.
4.2.3 Perturbation in Clustering Performance
With the evaluation target defined above, our main analysis will revolve around analyzing "how the extra label information help reduces  $\mathcal{M}_{kms}(\Pi, Z)$ ". Formally, we investigate the following error difference, as a result of added label information:
$$
\Delta_ {k m s} (\delta) = \mathcal {M} _ {k m s} (\Pi , Z) - \mathcal {M} _ {k m s} (\Pi , Z (\delta)),
$$
where the closed-form solution is given by the following theorem. Positive  $\Delta_{kms}(\delta)$  means improved clustering, as a result of adding labeling information.
Theorem 4.2. (Main result.) Denote  $V_{\emptyset}^{(u)} \in \mathbb{R}^{N \times (N - k)}$  as the null space of  $V_{k}^{(u)}$  and  $\tilde{A}_{k}^{(u)} = V_{k}^{(u)}\Sigma_{k}^{(u)}V_{k}^{(u)\top}$  as the rank- $k$  approximation for  $\tilde{A}^{(u)}$ . Given  $\delta, \eta_1 > 0$  and let  $\mathcal{G}_k$  as the spectral gap between  $k$ -th and  $k + 1$ -th singular values of  $\tilde{A}^{(u)}$ , we have:
$$
\Delta_ {k m s} (\delta) = \delta \eta_ {1} \mathrm {T r} \left(\Upsilon \left(V _ {k} ^ {(u)} V _ {k} ^ {(u) \top} \mathfrak {U} ^ {\top} (I + V _ {\emptyset} ^ {(u)} V _ {\emptyset} ^ {(u) \top}) - 2 \tilde {A} _ {k} ^ {(u)} d i a g (\mathfrak {l})\right)\right) + O (\frac {1}{\mathcal {G} _ {k}} + \delta^ {2}),
$$
where  $\text{diag}(\cdot)$  converts the vector to the corresponding diagonal matrix and  $\Upsilon \in \mathbb{R}^{N \times N}$  is a matrix encoding the ground-truth clustering structure in the way that  $\Upsilon_{xx'} > 0$  if  $x$  and  $x'$  has the same label and  $\Upsilon_{xx'} < 0$  otherwise. The concrete form and the proof are in Appendix C.4.
Theorem 4.2 is more general but less intuitive to understand. To gain a better insight, we introduce Theorem 4.3 which provides more direct implications. We provide the justification of the assumptions and the formal proof in Appendix C.5.
Theorem 4.3. (Intuitive result.) Assuming the spectral gap  $\mathcal{G}_k$  is sufficiently large and  $\mathfrak{l}$  lies in the linear span of  $V_k^{(u)}$ . We also assume  $\forall \pi_c \in \Pi, \forall i \in \pi_c, \mathfrak{l}_{(i)} =: \mathfrak{l}_{\pi_c}$  which represents the connection between class  $c$  to the labeled data. Given  $\delta, \eta_1, \eta_2 > 0$ , we have:
$$
\Delta_ {k m s} (\delta) \geq \delta \eta_ {1} \eta_ {2} \sum_ {\pi_ {c} \in \Pi} | \pi_ {c} | \mathfrak {l} _ {\pi_ {c}} \Delta_ {\pi_ {c}} (\delta),
$$
where Connection from class c to the labeled data.
$$
\Delta_ {\pi_ {c}} (\delta) = \stackrel {\downarrow} {\left(\mathfrak {l} _ {\pi_ {c}} - \frac {1}{N}\right)} - 2 \left(1 - \frac {\left| \pi_ {c} \right|}{N}\right) \left(\mathbb {E} _ {i \in \pi_ {c}} \mathbb {E} _ {j \in \pi_ {c}} \mathbf {z} _ {i} ^ {\top} \mathbf {z} _ {j} - \mathbb {E} _ {i \in \pi_ {c}} \mathbb {E} _ {j \notin \pi_ {c}} \mathbf {z} _ {i} ^ {\top} \mathbf {z} _ {j}\right).
$$
Intra-class similarity Inter-class similarity
Implications. In Theorem 4.3, we define the class-wise perturbation of the K-means measure as  $\Delta_{\pi_c}(\delta)$ . This way, we can interpret the effect of adding labels for a specific class  $c$ . If we desire  $\Delta_{\pi_c}(\delta)$  to be large, the sufficient condition is that
connection of class  $c$  to the labeled data  $>$  intra-class similarity - inter-class similarity.
We use examples in Figure 1 to epitomize the core idea. Specifically, our unlabeled samples consist of three underlying classes: traffic lights (known), apples (novel), and flowers (novel). (a) For unlabeled traffic lights from known classes which are strongly connected to the labeled data, adding labels to traffic lights can largely improve the clustering performance; (b) For novel classes like apples, it may also help when they have a strong connection to the traffic light, and their intra-class similarity is not as strong (due to different colors); (c) However, labeled data may offer little improvement in clustering the flower class, due to the minimal connection to the labeled data and that flowers' self-clusterability is already strong.
5 Empirical Validation of Theory
Beyond theoretical insights, we show empirically that SORL is effective on standard benchmark image classification datasets CIFAR-10/100 [35]. Following the seminal work ORCA [7], classes are divided into  $50\%$  known and  $50\%$  novel classes. We then use  $50\%$  of samples from the known classes as the labeled dataset, and the rest as the unlabeled set. We follow the evaluation strategy in [7] and report the following metrics: (1) classification accuracy on known classes, (2) clustering accuracy on the novel data, and (3) overall accuracy on all classes. More experiment details are in Appendix E.2.
Table 1: Main Results. Mean and std are estimated on five different runs. Baseline numbers are from [7, 63].
Method	CIFAR-10	CIFAR-100
All	Novel	Known	All	Novel	Known
FixMatch [37]	49.5	50.4	71.5	20.3	23.5	39.6
\( \mathbf{DS^{3}L}\) [21]	40.2	45.3	77.6	24.0	23.7	55.1
CGDL [62]	39.7	44.6	72.3	23.6	22.5	49.3
DTC [22]	38.3	39.5	53.9	18.3	22.9	31.3
RankStats [82]	82.9	81.0	86.6	23.1	28.4	36.4
SimCLR [11]	51.7	63.4	58.3	22.3	21.2	28.6
ORCA [7]	\( 88.3^{\pm 0.3} \)	\( 87.5^{\pm 0.2} \)	\( 89.9^{\pm 0.4} \)	\( 47.2^{\pm 0.7} \)	\( 41.0^{\pm 1.0} \)	\( 66.7^{\pm 0.2} \)
GCD [69]	\( 87.5^{\pm 0.5} \)	\( 86.7^{\pm 0.4} \)	\( 90.1^{\pm 0.3} \)	\( 46.8^{\pm 0.5} \)	\( 43.4^{\pm 0.7} \)	\( 69.7^{\pm 0.4} \)
OpenCon [63]	\( 90.4^{\pm 0.6} \)	\( 91.1^{\pm 0.1} \)	\( 89.3^{\pm 0.2} \)	\( 52.7^{\pm 0.6} \)	\( 47.8^{\pm 0.6} \)	\( 69.1^{\pm 0.3} \)
SORL (Ours)	\( 93.5^{\pm 1.0} \)	\( 92.5^{\pm 0.1} \)	\( 94.0^{\pm 0.2} \)	\( 56.1^{\pm 0.3} \)	\( 52.0^{\pm 0.2} \)	\( 68.2^{\pm 0.1} \)
SORL achieves competitive performance. Our proposed loss SORL is amenable to the theoretical understanding, which is our primary goal of this work. Beyond theory, we show that SORL is equally desirable in empirical performance. In particular, SORL displays competitive performance compared to existing methods, as evidenced in Table 1. Our comparison covers an extensive collection of very recent algorithms developed for this problem, including ORCA [7], GCD [69], and OpenCon [63]. We also compare methods in related problem domains: (1) Semi-Supervised Learning [21, 37, 62], (2) Novel Class Discovery [22, 82], (3) common representation learning method SimCLR [11]. In particular, on CIFAR-100, we improve upon the best baseline OpenCon by  $3.4\%$  in terms of overall accuracy. Our result further validates that putting analysis on SORL is appealing for both theoretical and empirical reasons.
6 Broader Impact
From a theoretical perspective, our graph-theoretic framework can facilitate and deepen the understanding of other representation learning methods that commonly involve the notion of positive/negative pairs. In Appendix D, we exemplify how our framework can be potentially generalized to other common contrastive loss functions [11, 34, 68], and baseline methods that are tailored for the open-world semi-supervised learning problem (e.g., GCD [69], OpenCon [63]). Hence, we believe our theoretical framework has a broader utility and significance.
From a practical perspective, our work can directly impact and benefit many real-world applications, where unlabeled data are produced at an incredible rate today. Major companies exhibit a strong need for making their machine learning systems and services amendable for the open-world setting but lack fundamental and systematic knowledge. Hence, our research advances the understanding of open-world machine learning and helps the industry improve ML systems by discovering insights and structures from unlabeled data.
7 Related Work
Semi-supervised learning. Semi-supervised learning (SSL) is a classic problem in machine learning. SSL typically assumes the same class space between labeled and unlabeled data, and hence remains closed-world. A rich line of empirical works [9, 13, 21, 29, 37, 38, 39, 42, 48, 50, 53, 54, 74, 76, 78] and theoretical efforts [3, 46, 47, 51, 60, 61, 73] have been made to address this problem. An important class of SSL methods is to represent data as graphs and predict labels by aggregating proximal nodes' labels [1, 18, 30, 71, 80, 84, 85]. Different from classic SSL, we allow its semantic space to cover both known and novel classes. Accordingly, we contribute a graph-theoretic framework tailored to the open-world setting, and reveal new insights on how the labeled data can benefit the clustering performance on both known and novel classes.
Open-world semi-supervised learning. The learning setting that considers both labeled and unlabeled data with a mixture of known and novel classes is first proposed in [7] and inspires a proliferation of follow-up works [49, 52, 63, 69, 81] advancing empirical success. Most works put emphasis on learning high-quality embeddings [49, 63, 69, 81]. In particular, Sun and Li [63] employ contrastive learning with both supervised and self-supervised signals, which aligns with our theoretical setup in Sec. 3.1. Different from prior works, our paper focuses on advancing theoretical understanding. To the best of our knowledge, we are the first to theoretically investigate the problem from a graph-theoretic perspective and provide a rigorous error bound.
Spectral graph theory. Spectral graph theory is a classic research problem [10, 14, 33, 40, 44, 70], which aims to partition the graph by studying the eigenspace of the adjacency matrix. The spectral graph theory is also widely applied in machine learning [1, 6, 45, 56, 58, 64, 86]. Recently, HaoChen et al. [23] derive a spectral contrastive loss from the factorization of the graph's adjacency matrix which facilitates theoretical study in unsupervised domain adaptation [24, 57]. In these works, the graph's formulation is exclusively based on unlabeled data. Sun et al. [64] later expanded this spectral contrastive loss approach to cater to learning environments that encompass both labeled data from known classes and unlabeled data from novel ones. In this paper, our adaptation of the loss function from [64] is tailored to address the open-world semi-supervised learning challenge, considering known class samples within unlabeled data.
Theory for self-supervised learning. A proliferation of works in self-supervised representation learning demonstrates the empirical success [5, 8, 11, 12, 23, 26, 68, 77] with the theoretical foundation by providing provable guarantees on the representations learned by contrastive learning for linear probing [4, 41, 55, 59, 66, 67]. From the graphic view, HaoChen et al. [23, 24], Shen et al. [57] model the pairwise relation by the augmentation probability and provided error analysis of the downstream tasks. The existing body of work has mostly focused on unsupervised learning. In this paper, we systematically investigate how the label information can change the representation manifold and affect the downstream clustering performance on both known and novel classes.
In this paper, we present a graph-theoretic framework for open-world semi-supervised learning. The framework facilitates the understanding of how representations change as a result of adding labeling information to the graph. Specifically, we learn representation through Spectral Open-world Representation Learning (SORL). Minimizing this objective is equivalent to factorizing the graph's adjacency matrix, which allows us to analyze the clustering error difference between having vs. excluding labeled data. Our main results suggest that the clustering error can be significantly reduced if the connectivity to the labeled data is stronger than their self-clusterability. Our framework is also empirically appealing to use since it achieves competitive performance on par with existing baselines. Nevertheless, we acknowledge two limitations to practical application within our theoretical construct:
- The augmentation graph serves as a potent theoretical tool for elucidating the success of modern representation learning methods. However, it is challenging to ensure that current augmentation strategies, such as cropping, color jittering, can transform two dissimilar images into identical ones.
- The utilization of Theorems 4.1 and 4.2 necessitates an explicit knowledge of the adjacency matrix of the augmentation graph, a requirement that can be intractable in practice.
In light of these limitations, we encourage further research to enhance the practicality of these theoretical findings. We also hope our framework and insights can inspire the broader representation learning community to understand the role of labeling prior.
Research is supported by the AFOSR Young Investigator Program under award number FA9550-23-1-0184, National Science Foundation (NSF) Award No. IIS-2237037 & IIS-2331669, Office of Naval Research under grant number N00014-23-1-2643, and faculty research awards/gifts from Google and Meta. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements either expressed or implied, of the sponsors. The authors would also like to thank Tengyu Ma, Xuefeng Du, and Yifei Ming for their helpful suggestions and feedback.
A Technical Details of Spectral Open-world Representation Learning
Theorem A.1. (Recap of Theorem 3.1) We define  $\mathbf{f}_x = \sqrt{w_x} f(x)$  for some function  $f$ . Recall  $\eta_u, \eta_l$  are two hyper-parameters defined in Eq. (1). Then minimizing the loss function  $\mathcal{L}_{\mathrm{mf}}(F, A)$  is equivalent to minimizing the following loss function for  $f$ , which we term Spectral Open-world Representation Learning (SORL):
$$
\begin{array}{l} \mathcal {L} _ {S O R L} (f) \triangleq - 2 \eta_ {l} \mathcal {L} _ {1} (f) - 2 \eta_ {u} \mathcal {L} _ {2} (f) \tag {10} \\ + \eta_ {l} ^ {2} \mathcal {L} _ {3} (f) + 2 \eta_ {l} \eta_ {u} \mathcal {L} _ {4} (f) + \eta_ {u} ^ {2} \mathcal {L} _ {5} (f), \\ \end{array}
$$
$$
\mathcal{L}_{1}(f) = \sum_{i\in \mathcal{Y}_{l}}\underset { \begin{array}{c}\bar{x}_{l}\sim \mathcal{P}_{l_{i}},\bar{x}_{l}^{\prime}\sim \mathcal{P}_{l_{i}},\\ x\sim \mathcal{T}(\cdot |\bar{x}_{l}),x^{+}\sim \mathcal{T}(\cdot |\bar{x}_{l}^{\prime}) \end{array} }{\mathbb{E}}\left[f(x)^{\top}f\left(x^{+}\right)\right],
$$
$$
\mathcal{L}_{2}(f) = \underset { \begin{array}{c}\bar{x}_{u}\sim \mathcal{P},\\ x\sim \mathcal{T}(\cdot |\bar{x}_{u}),x^{+}\sim \mathcal{T}(\cdot |\bar{x}_{u}) \end{array} }{\mathbb{E}}\left[f(x)^{\top}f\left(x^{+}\right)\right],
$$
$$
\mathcal{L}_{3}(f) = \sum_{i,j\in \mathcal{Y}_{l}}\underset { \begin{array}{c}\bar{x}_{l}\sim \mathcal{P}_{l_{i}},\bar{x}_{l}^{\prime}\sim \mathcal{P}_{l_{j}},\\ x\sim \mathcal{T}(\cdot |\bar{x}_{l}),x^{-}\sim \mathcal{T}(\cdot |\bar{x}_{l}^{\prime}) \end{array} }{\mathbb{E}}\left[  \left(f(x)^{\top}f\left(x^{-}\right)\right)^{2}\right],
$$
$$
\mathcal{L}_{4}(f) = \sum_{i\in \mathcal{Y}_{l}}\underset { \begin{array}{c}\bar{x}_{l}\sim \mathcal{P}_{l_{i}},\bar{x}_{u}\sim \mathcal{P},\\ x\sim \mathcal{T}(\cdot |\bar{x}_{l}),x^{-}\sim \mathcal{T}(\cdot |\bar{x}_{u}) \end{array} }{\mathbb{E}}\left[  \left(f(x)^{\top}f\left(x^{-}\right)\right)^{2}\right],
$$
$$
\mathcal{L}_{5}(f) = \underset { \begin{array}{c}\bar{x}_{u}\sim \mathcal{P},\bar{x}_{u}^{\prime}\sim \mathcal{P},\\ x\sim \mathcal{T}(\cdot |\bar{x}_{u}),x^{-}\sim \mathcal{T}(\cdot |\bar{x}_{u}^{\prime}) \end{array} }{\mathbb{E}}\left[  \left(f(x)^{\top}f\left(x^{-}\right)\right)^{2}\right].
$$
Proof. We can expand  $\mathcal{L}_{\mathrm{mf}}(F,A)$  and obtain
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {m f}} (F, A) = \sum_ {x, x ^ {\prime} \in \mathcal {X}} \left(\frac {w _ {x x ^ {\prime}}}{\sqrt {w _ {x} w _ {x ^ {\prime}}}} - \mathbf {f} _ {x} ^ {\top} \mathbf {f} _ {x ^ {\prime}}\right) ^ {2} \\ = \operatorname {c o n s t} + \sum_ {x, x ^ {\prime} \in \mathcal {X}} \left(- 2 w _ {x x ^ {\prime}} f (x) ^ {\top} f \left(x ^ {\prime}\right) + w _ {x} w _ {x ^ {\prime}} \left(f (x) ^ {\top} f \left(x ^ {\prime}\right)\right) ^ {2}\right), \\ \end{array}
$$
where  $\mathbf{f}_x = \sqrt{w_x} f(x)$  is a re-scaled version of  $f(x)$ . At a high level, we follow the proof in [23], while the specific form of loss varies with the different definitions of positive/negative pairs. The form of  $\mathcal{L}_{\mathrm{SORL}}(f)$  is derived from plugging  $w_{xx'}$  and  $w_x$ .
Recall that  $w_{xx'}$  is defined by
$$
w _ {x x ^ {\prime}} = \eta_ {l} \sum_ {i \in \mathcal {Y} _ {l}} \mathbb {E} _ {\bar {x} _ {l} \sim \mathcal {P} _ {l _ {i}}} \mathbb {E} _ {\bar {x} _ {l} ^ {\prime} \sim \mathcal {P} _ {l _ {i}}} \mathcal {T} (x | \bar {x} _ {l}) \mathcal {T} (x ^ {\prime} | \bar {x} _ {l} ^ {\prime}) + \eta_ {u} \mathbb {E} _ {\bar {x} _ {u} \sim \mathcal {P}} \mathcal {T} (x | \bar {x} _ {u}) \mathcal {T} (x ^ {\prime} | \bar {x} _ {u}),
$$
and  $w_{x}$  is given by