SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {M} _ {k m s} ^ {\prime} (\delta) = \frac {1}{\mathcal {M} _ {i n t e r} (\Pi , Z)} \mathcal {M} _ {i n t r a} ^ {\prime} (\delta) - \frac {\mathcal {M} _ {i n t r a} (\Pi , Z)}{\mathcal {M} _ {i n t e r} (\Pi , Z) ^ {2}} \mathcal {M} _ {i n t e r} ^ {\prime} (\delta) \\ = \eta_ {1} \mathcal {M} _ {i n t r a} ^ {\prime} (\delta) - \eta_ {1} \eta_ {2} \mathcal {M} _ {i n t e r} ^ {\prime} (\delta) \\ = \eta_ {1} \left(\mathrm {T r} ((I _ {\Pi} - H _ {\Pi}) [ Z (\delta) Z (\delta) ^ {\top} ] ^ {\prime}) - \eta_ {2} \mathrm {T r} ((H _ {\Pi} - \frac {1}{N} \mathbf {1} _ {N \times N}) [ Z (\delta) Z (\delta) ^ {\top} ] ^ {\prime})\right) \\ = \eta_ {1} \left(\operatorname {T r} \left(\left(I _ {\Pi} + \frac {\eta_ {2}}{N} \mathbf {1} _ {N \times N} - (\eta_ {2} + 1) H _ {\Pi}\right) [ Z (\delta) Z (\delta) ^ {\top} ] ^ {\prime}\right)\right) \\ = - \eta_ {1} \left(\operatorname {T r} \left(\Upsilon \left[ Z (\delta) Z (\delta) ^ {\top} \right] ^ {\prime}\right)\right) \\ = - \eta_ {1} \sum_ {j = 1} ^ {k} \operatorname {T r} (\Upsilon [ D (\delta) ^ {- \frac {1}{2}} \lambda_ {j} (\delta) \Phi_ {j} (\delta) D (\delta) ^ {- \frac {1}{2}} ] ^ {\prime}), \\ \end{array}
$$
where we let  $\eta_{1} = \frac{1}{\mathcal{M}_{inter}(\Pi,Z)}$ ,  $\eta_{2} = \frac{\mathcal{M}_{intra}(\Pi,Z)}{\mathcal{M}_{inter}(\Pi,Z)}$  and  $\Upsilon = (1 + \eta_2)H_{\Pi} - I_{\Pi} - \frac{\eta_2}{N}\mathbf{1}_N\mathbf{1}_N^\top$ . We proceed by showing the calculation of  $[D(\delta)^{-\frac{1}{2}}]^{\prime}$ ,  $[\lambda_j(\delta)]'$  and  $[\Phi_j(\delta)]'$ .
Since  $D(\delta) = I + \delta D_{l}$ , then  $[D(\delta)^{-\frac{1}{2}}]^{\prime}\bigg|_{\delta = 0} = -\frac{1}{2} D_{l}$ . To calculate  $[\lambda_j(\delta)]'$  and  $[\Phi_j(\delta)]'$ , we first need:
$$
\begin{array}{l} \left. \left[ \tilde {A} (\delta) \right] ^ {\prime} \right| _ {\delta = 0} = \left[ D (\delta) ^ {- \frac {1}{2}} A (\delta) D (\delta) ^ {- \frac {1}{2}} \right] ^ {\prime} \\ = \left[ D (\delta) ^ {- \frac {1}{2}} \right] ^ {\prime} \tilde {A} ^ {(u)} + \left[ A (\delta) \right] ^ {\prime} + \tilde {A} ^ {(u)} \left[ D (\delta) ^ {- \frac {1}{2}} \right] ^ {\prime} \\ = - \frac {1}{2} D _ {l} \tilde {A} ^ {(u)} + \mathfrak {U} ^ {\top} - \frac {1}{2} \tilde {A} ^ {(u)} D _ {l}. \\ \end{array}
$$
Then, according to Equation (3) in [20], we have:
$$
\begin{array}{l} \left. \left[ \lambda_ {j} (\delta) \right] ^ {\prime} \right| _ {\delta = 0} = \operatorname {T r} \left(\Phi_ {j} ^ {(u)} [ \tilde {A} (\delta) ] ^ {\prime}\right) \\ = \operatorname {T r} \left(\Phi_ {j} ^ {(u)} \left(- \frac {1}{2} D _ {l} \tilde {A} ^ {(u)} + \mathfrak {U} ^ {\top} - \frac {1}{2} \tilde {A} ^ {(u)} D _ {l}\right)\right) \\ = \operatorname {T r} \left(\left(- \frac {\lambda_ {j} ^ {(u)}}{2} D _ {l} \Phi_ {j} ^ {(u)} + \Phi_ {j} ^ {(u)} \mathbb {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)}}{2} \Phi_ {j} ^ {(u)} D _ {l}\right)\right) \\ = \operatorname {T r} \left(\Phi_ {j} ^ {(u)} \left(\mathfrak {U} ^ {\top} - \lambda_ {j} ^ {(u)} D _ {l}\right)\right). \\ \end{array}
$$
According to Equation (10) in [20], we have:
$$
\begin{array}{l} \left. \left[ \Phi_ {j} (\delta) \right] ^ {\prime} \right| _ {\delta = 0} = \left(\lambda_ {j} ^ {(u)} I _ {N} - \tilde {A} ^ {(u)}\right) ^ {\dagger} [ \tilde {A} (\delta) ] ^ {\prime} \Phi_ {j} ^ {(u)} + \Phi_ {j} ^ {(u)} [ \tilde {A} (\delta) ] ^ {\prime} \left(\lambda_ {j} ^ {(u)} I _ {N} - \tilde {A} ^ {(u)}\right) ^ {\dagger} \\ = \sum_ {i \neq j} ^ {N} \frac {1}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} \left(\Phi_ {i} ^ {(u)} [ \tilde {A} (\delta) ] ^ {\prime} \Phi_ {j} ^ {(u)} + \Phi_ {j} ^ {(u)} [ \tilde {A} (\delta) ] ^ {\prime} \Phi_ {i} ^ {(u)}\right) \\ = \sum_ {i \neq j} ^ {N} \frac {1}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} (\Phi_ {i} ^ {(u)} (- \frac {1}{2} D _ {l} \tilde {A} ^ {(u)} + \mathfrak {U} ^ {\top} - \frac {1}{2} \tilde {A} ^ {(u)} D _ {l}) \Phi_ {j} ^ {(u)} + \Phi_ {j} ^ {(u)} (... \Phi_ {i} ^ {(u)}) \\ = \sum_ {i \neq j} ^ {N} \frac {1}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} (\Phi_ {i} ^ {(u)} (\mathfrak {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l}) \Phi_ {j} ^ {(u)} + \Phi_ {j} ^ {(u)} (\mathfrak {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l}) \Phi_ {i} ^ {(u)}). \\ \end{array}
$$
Now we calculate the derivative of the  $K$ -means loss:
$$
\begin{array}{l} \frac {1}{\eta_ {1}} \left[ \mathcal {M} _ {k m s} (\delta) \right] ^ {\prime} \Big | _ {\delta = 0} = - \sum_ {j = 1} ^ {k} \left[ \operatorname {T r} \left(\Upsilon D (\delta) ^ {- \frac {1}{2}} \lambda_ {j} (\delta) \Phi_ {j} (\delta) D (\delta) ^ {- \frac {1}{2}}\right) \right] ^ {\prime} \Big | _ {\delta = 0} \\ = - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\Upsilon \left([ D (\delta) ^ {- \frac {1}{2}} ] ^ {\prime} \lambda_ {j} ^ {(u)} \Phi_ {j} ^ {(u)} + \lambda_ {j} ^ {(u)} \Phi_ {j} ^ {(u)} [ D (\delta) ^ {- \frac {1}{2}} ] ^ {\prime} + [ \lambda_ {j} (\delta) ] ^ {\prime} \Phi_ {j} ^ {(u)} + \lambda_ {j} ^ {(u)} [ \Phi_ {j} (\delta) ] ^ {\prime}\right)\right) \\ = \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\Upsilon \left(\frac {\lambda_ {j} ^ {(u)}}{2} D _ {l} \Phi_ {j} ^ {(u)} + \frac {\lambda_ {j} ^ {(u)}}{2} \Phi_ {j} ^ {(u)} D _ {l} - [ \lambda_ {j} (\delta) ] ^ {\prime} \Phi_ {j} ^ {(u)} - \lambda_ {j} ^ {(u)} [ \Phi_ {j} (\delta) ] ^ {\prime}\right)\right) \\ = \mathcal {M} _ {a} ^ {\prime} + \mathcal {M} _ {b} ^ {\prime} + \mathcal {M} _ {c} ^ {\prime}, \\ \end{array}
$$
$$
\mathcal {M} _ {a} ^ {\prime} = \sum_ {j = 1} ^ {k} \frac {\lambda_ {j} ^ {(u)}}{2} \mathrm {T r} \left(\Upsilon \left(D _ {l} \Phi_ {j} ^ {(u)} + \Phi_ {j} ^ {(u)} D _ {l}\right)\right),
$$
$$
\begin{array}{l} \mathcal {M} _ {b} ^ {\prime} = - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\Upsilon [ \lambda_ {j} (\delta) ] ^ {\prime} \Phi_ {j} ^ {(u)}\right) = - \sum_ {j = 1} ^ {k} \operatorname {T r} \left((\mathbb {U} ^ {\top} - \lambda_ {j} ^ {(u)} D _ {l}) \Phi_ {j} ^ {(u)}\right) \operatorname {T r} \left(\Upsilon \Phi_ {j} ^ {(u)}\right) \\ = - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\left(\mathbb {U} ^ {\top} - \lambda_ {j} ^ {(u)} D _ {l}\right) \Phi_ {j} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}\right), \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {M} _ {c} ^ {\prime} = - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\Upsilon \lambda_ {j} ^ {(u)} [ \Phi_ {j} (\delta) ] ^ {\prime}\right) \\ = - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i \neq j} ^ {N} \frac {\lambda_ {j} ^ {(u)}}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} \left(\Upsilon \Phi_ {i} ^ {(u)} \left(\mathfrak {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l}\right) \Phi_ {j} ^ {(u)} + \Upsilon \Phi_ {j} ^ {(u)} \left(\mathfrak {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l}\right) \Phi_ {i} ^ {(u)}\right)\right) \\ = - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i \neq j} ^ {N} \frac {\lambda_ {j} ^ {(u)}}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} \left((\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}) (\mathfrak {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l})\right)\right) \\ = - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i \neq j, i \leq k} \frac {\lambda_ {j} ^ {(u)}}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} \left(\left(\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}\right) \left(\mathbb {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l}\right)\right)\right) \\ \left. \right. - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i = k + 1} ^ {N} \frac {\lambda_ {j} ^ {(u)}}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} \left((\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}) (\mathfrak {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l})\right)\right) \\ = - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i <   j} \left(\frac {\lambda_ {j} ^ {(u)}}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} + \frac {\lambda_ {i} ^ {(u)}}{\lambda_ {i} ^ {(u)} - \lambda_ {j} ^ {(u)}}\right) \left(\left(\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}\right) \left(\mathbb {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l}\right)\right)\right) \\ - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i = k + 1} ^ {N} \frac {\lambda_ {j} ^ {(u)}}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} \left((\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}) (\mathbb {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l})\right)\right) \\ \end{array}
$$
$$
\begin{array}{l} = - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i <   j} \left(\left(\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}\right) \left(\mathbb {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l}\right)\right)\right) \\ \left. \right. - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i = k + 1} ^ {N} \frac {\lambda_ {j} ^ {(u)}}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} \left((\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}) (\mathbb {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l})\right)\right) \\ = - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i \neq j, i \leq k} \frac {1}{2} \left(\left(\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}\right) \left(\mathfrak {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l}\right)\right)\right) \\ - \sum_ {j = 1} ^ {k} \mathrm {T r} \left(\sum_ {i = k + 1} ^ {N} \frac {\lambda_ {j} ^ {(u)}}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} \left((\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}) (\mathfrak {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l})\right)\right). \\ \end{array}
$$
Thus, we have:
$$
\begin{array}{l} \mathcal {M} _ {b} ^ {\prime} + \mathcal {M} _ {c} ^ {\prime} = - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i = 1} ^ {k} \frac {1}{2} \left(\left(\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}\right) \left(\mathrm {I} \Uparrow^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l}\right)\right)\right) \\ - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i = k + 1} ^ {N} \frac {\lambda_ {j} ^ {(u)}}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} \left((\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}) (\mathfrak {U} ^ {\top} - \frac {\lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l})\right)\right), \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {M} _ {a} ^ {\prime} = \sum_ {j = 1} ^ {k} \frac {\lambda_ {j} ^ {(u)}}{2} \operatorname {T r} \left(\Upsilon \left(D _ {l} \Phi_ {j} ^ {(u)} + \Phi_ {j} ^ {(u)} D _ {l}\right)\right) \\ = \sum_ {j = 1} ^ {k} \frac {\lambda_ {j} ^ {(u)}}{2} \operatorname {T r} \left(\left(\Phi_ {j} ^ {(u)} \Upsilon + \Upsilon \Phi_ {j} ^ {(u)}\right) D _ {l}\right) \\ = \sum_ {j = 1} ^ {k} \frac {\lambda_ {j} ^ {(u)}}{2} \operatorname {T r} \left(\left(\Phi_ {j} ^ {(u)} \Upsilon \sum_ {i = 1} ^ {N} \Phi_ {i} ^ {(u)} + \sum_ {i = 1} ^ {N} \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}\right) D _ {l}\right) \\ = \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i = 1} ^ {N} \frac {\lambda_ {j} ^ {(u)}}{2} \left(\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}\right) D _ {l}\right). \\ \end{array}
$$
Then  $\left[\mathcal{M}_{kms - all}(\delta)\right]'\bigg|_{\delta = 0} / \eta_1$  is given by:
$$
\begin{array}{l} \mathcal {M} _ {a} ^ {\prime} + \mathcal {M} _ {b} ^ {\prime} + \mathcal {M} _ {c} ^ {\prime} = - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i = 1} ^ {k} \frac {1}{2} \left((\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}) (\mathfrak {U} ^ {\top} - \frac {3 \lambda_ {j} ^ {(u)} + \lambda_ {i} ^ {(u)}}{2} D _ {l})\right)\right) \\ - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i = k + 1} ^ {N} \frac {\lambda_ {j} ^ {(u)}}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} \left((\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}) (\mathfrak {U} ^ {\top} - \lambda_ {j} ^ {(u)} D _ {l})\right)\right) \\ = - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i = 1} ^ {k} \frac {1}{2} \left(\left(\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}\right) \left(\mathfrak {U} ^ {\top} - 2 \lambda_ {j} ^ {(u)} D _ {l}\right)\right)\right) \\ - \sum_ {j = 1} ^ {k} \operatorname {T r} \left(\sum_ {i = k + 1} ^ {N} \frac {\lambda_ {j} ^ {(u)}}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} \left((\Phi_ {j} ^ {(u)} \Upsilon \Phi_ {i} ^ {(u)} + \Phi_ {i} ^ {(u)} \Upsilon \Phi_ {j} ^ {(u)}) (\mathfrak {U} ^ {\top} - \lambda_ {j} ^ {(u)} D _ {l})\right)\right) \\ = - \sum_ {j = 1} ^ {k} \sum_ {i = 1} ^ {k} v _ {i} ^ {(u) \top} \Upsilon v _ {j} ^ {(u)} \cdot v _ {i} ^ {(u) \top} \left(\mathbb {U} ^ {\top} - 2 \lambda_ {j} ^ {(u)} D _ {l}\right) v _ {j} ^ {(u)} \\ - \sum_ {j = 1} ^ {k} \sum_ {i = k + 1} ^ {N} \frac {2 \lambda_ {j} ^ {(u)}}{\lambda_ {j} ^ {(u)} - \lambda_ {i} ^ {(u)}} v _ {i} ^ {(u) \top} \Upsilon v _ {j} ^ {(u)} \cdot v _ {i} ^ {(u) \top} (\mathfrak {U} ^ {\top} - \lambda_ {j} ^ {(u)} D _ {l}) v _ {j} ^ {(u)}. \\ \end{array}
$$
We can represent  $\frac{\lambda_j^{(u)}}{\lambda_j^{(u)} - \lambda_i^{(u)}} = 1 + \sum_{p=1}^{\infty} \left( \frac{\lambda_i^{(u)}}{\lambda_j^{(u)}} \right)^p$ . Denote the residual term as:
$$
\mathcal {M} _ {e} ^ {\prime} = - \sum_ {j = 1} ^ {k} \sum_ {i = k + 1} ^ {N} \sum_ {p = 1} ^ {\infty} 2 (\frac {\lambda_ {i} ^ {(u)}}{\lambda_ {j} ^ {(u)}}) ^ {p} v _ {i} ^ {(u) \top} \Upsilon v _ {j} ^ {(u)} \cdot v _ {i} ^ {(u) \top} (\mathbb {U} ^ {\top} - \lambda_ {j} ^ {(u)} D _ {l}) v _ {j} ^ {(u)} = O (\frac {1}{\mathcal {G} _ {k}}).
$$
We then have:
$$
\begin{array}{l} \left. \frac {1}{\eta_ {1}} \left[ \mathcal {M} _ {k m s - a l l} (\delta) \right] ^ {\prime} \right| _ {\delta = 0} \\ = - \operatorname {T r} \left(V _ {k} ^ {(u) \top} \Upsilon V _ {k} ^ {(u)} \cdot V _ {k} ^ {(u) \top} \mathfrak {U} ^ {\top} V _ {k} ^ {(u)}\right) + 2 \operatorname {T r} \left(V _ {k} ^ {(u) \top} \Upsilon V _ {k} ^ {(u)} \cdot \Sigma_ {k} ^ {(u)} V _ {k} ^ {(u) \top} D _ {l} V _ {k} ^ {(u)}\right) \\ - 2 \operatorname {T r} (V _ {\varnothing} ^ {(u) \top} \Upsilon V _ {k} ^ {(u)} \cdot V _ {k} ^ {(u) \top} \mathbb {1} ^ {\top} V _ {\varnothing} ^ {(u)}) + 2 \operatorname {T r} (V _ {\varnothing} ^ {(u) \top} \Upsilon V _ {k} ^ {(u)} \cdot \Sigma_ {k} ^ {(u)} V _ {k} ^ {(u) \top} D _ {l} V _ {\varnothing} ^ {(u)}) + \mathcal {M} _ {e} ^ {\prime} \\ = - \operatorname {T r} \left(\Upsilon V _ {k} ^ {(u)} V _ {k} ^ {(u) \top} \mathfrak {U} ^ {\top} V _ {k} ^ {(u)} V _ {k} ^ {(u) \top}\right) + 2 \operatorname {T r} \left(\Upsilon \tilde {A} _ {k} ^ {(u)} D _ {l} V _ {k} ^ {(u)} V _ {k} ^ {(u) \top}\right) \\ - 2 \operatorname {T r} (\Upsilon V _ {k} ^ {(u)} V _ {k} ^ {(u) \top} \mathfrak {U} ^ {\top} (I _ {N} - V _ {k} ^ {(u)} V _ {k} ^ {(u) \top})) + 2 \operatorname {T r} (\Upsilon \tilde {A} _ {k} ^ {(u)} D _ {l} (I _ {N} - V _ {k} ^ {(u)} V _ {k} ^ {(u) \top})) + \mathcal {M} _ {e} ^ {\prime} \\ = - 2 \operatorname {T r} \left(\Upsilon V _ {k} ^ {(u)} V _ {k} ^ {(u) \top} \mathfrak {U} ^ {\top}\right) + 2 \operatorname {T r} \left(\Upsilon \tilde {A} _ {k} ^ {(u)} D _ {l}\right) + \operatorname {T r} \left(\Upsilon V _ {k} ^ {(u)} V _ {k} ^ {(u) \top} \mathfrak {U} ^ {\top} V _ {k} ^ {(u)} V _ {k} ^ {(u) \top}\right) + \mathcal {M} _ {e} ^ {\prime} \\ = - 2 \operatorname {T r} \left(\Upsilon \left(V _ {k} ^ {(u)} V _ {k} ^ {(u) \top} \mathfrak {U} ^ {\top} - \tilde {A} _ {k} ^ {(u)} D _ {l} - \frac {1}{2} V _ {k} ^ {(u)} V _ {k} ^ {(u) \top} \mathfrak {U} ^ {\top} V _ {k} ^ {(u)} V _ {k} ^ {(u) \top}\right)\right) + \mathcal {M} _ {e} ^ {\prime} \\ = - \operatorname {T r} \left(\Upsilon \left(V _ {k} ^ {(u)} V _ {k} ^ {(u) ^ {\top}} \mathfrak {U} ^ {\top} - 2 \tilde {A} _ {k} ^ {(u)} D _ {l} + V _ {k} ^ {(u)} V _ {k} ^ {(u) ^ {\top}} \mathfrak {U} ^ {\top} V _ {\emptyset} V _ {\emptyset} ^ {\top}\right)\right) + O (\frac {1}{\mathcal {G} _ {k}}). \\ \end{array}
$$
D Analysis on Other Contrastive Losses
In this section, we discuss the extension of our graphic-theoretic analysis to one of the most common contrastive loss functions – SimCLR [11]. SimCLR loss is an extended version of InfoNCE loss [68] that achieves great empirical success and inspires a proliferation of follow-up works [5, 8, 12, 26, 34, 69, 77]. Specifically, SupCon [34] extends SimCLR to the supervised setting. GCD [69] and OpenCon [63] further leverage the SupCon and SimCLR losses, and are tailored to the open-world representation learning setting considering both labeled and unlabeled data.
At a high level, we consider a general form of the SimCLR and its extensions (including SupCon, GCD, OpenCon) as:
$$
\mathcal {L} _ {\mathrm {g n l}} (f; \mathcal {P} _ {+}) = - \frac {1}{\tau} \underset {(x, x ^ {+}) \sim \mathcal {P} _ {+}} {\mathbb {E}} [ f (x) ^ {\top} f (x ^ {+}) ] + \underset {x \sim \mathcal {P}} {\mathbb {E}} \left[ \log \left(\underset {x ^ {\prime} \sim \mathcal {P}} {\mathbb {E}} e ^ {f \left(x ^ {\prime}\right) ^ {\top} f (x) / \tau}\right) \right], \tag {15}
$$
where we let the  $\mathcal{P}_{+}$  as the distribution of positive pairs defined in Section 3.1. In SimCLR [11], the positive pairs are purely sampled in the unlabeled case  $(u)$  while SupCon [34] considers the labeled case  $(l)$ . With both labeled and unlabeled data, GCD [69] and OpenCon [63] sample positive pairs in both cases.
In this section, we investigate an alternative form that eases the theoretical analysis (also applied in [72]):
$$
\begin{array}{l} \widehat {\mathcal {L}} _ {\mathrm {g n l}} (f; \mathcal {P} _ {+}) = - \frac {1}{\tau (x , x ^ {+}) \sim \mathcal {P} _ {+}} \left[ f (x) ^ {\top} f \left(x ^ {+}\right) \right] + \log \left(\underset { \begin{array}{c} x, x ^ {\prime} \sim \mathcal {P} \\ x \neq x ^ {\prime} \end{array} } {\mathbb {E}} e ^ {f \left(x ^ {\prime}\right) ^ {\top} f (x) / \tau}\right) (16) \\ \geq \mathcal {L} _ {\mathrm {g n l}} (f; \mathcal {P} _ {+}), (17) \\ \end{array}
$$
which serves an upper bound of  $\mathcal{L}_{\mathrm{gnl}}(f)$  according to Jensen's Inequality.
A graph-theoretic view. Recall in Section 3.1, we define the graph  $G(\mathcal{X}, w)$  with vertex set  $\mathcal{X}$  and edge weights  $w$ . Each entry of adjacency matrix  $A$  is given by  $w_{xx'}$ , which denotes the marginal probability of generating the pair for any two augmented data  $x, x' \in \mathcal{X}$ :
$$
w _ {x x ^ {\prime}} = \eta_ {u} w _ {x x ^ {\prime}} ^ {(u)} + \eta_ {l} w _ {x x ^ {\prime}} ^ {(l)},
$$
and  $w_{x}$  measures the degree of node  $x$ :
$$
w _ {x} = \sum_ {x ^ {\prime}} w _ {x x ^ {\prime}}.
$$
One can view the difference between SimCLR and its variants in the following way: (1) SimCLR [11] corresponds to  $\eta_{l} = 0$  when there is no labeled case; (2) SupCon [34] corresponds to  $\eta_{u} = 0$  when only labeled case is considered. (3) GCD [69] and OpenCon [63] correspond to the cases when  $\eta_{u},\eta_{l}$  are both non-zero due to the availability of both labeled and unlabeled data.
With the define marginal probability of sampling positive pairs  $w_{xx'}$  and the marginal probability of sampling a single sample  $w_x$ , we have:
$$
\begin{array}{l} \widehat{\mathcal{L}}_{\mathrm{gnl}}\left(Z;G(\mathcal{X},w)\right) = -\frac{1}{\tau}\sum_{x,x^{\prime}\in \mathcal{X}}w_{xx^{\prime}}f(x)^{\top}f\left(x^{\prime}\right) + \log \left(\sum_{\substack{x,x^{\prime}\in \mathcal{X}\\ x\neq x^{\prime}}}w_{x}w_{x^{\prime}}e^{f(x^{\prime})^{\top}f(x) / \tau}\right) \\ = - \frac {1}{\tau} \operatorname {T r} (Z ^ {\top} A Z) + \log \operatorname {T r} \left((D \mathbf {1} _ {N} \mathbf {1} _ {N} ^ {\top} D - D ^ {2}) \exp (\frac {1}{\tau} Z Z ^ {T})\right). \\ \end{array}
$$
When  $\tau$  is large:
$$
\begin{array}{l} \widehat {\mathcal {L}} _ {\mathrm {s i m c l r}} \left(Z; G (\mathcal {X}, w)\right) \approx - \frac {1}{\tau} \operatorname {T r} (Z ^ {\top} A Z) + \log \operatorname {T r} \left((D \mathbf {1} _ {N} \mathbf {1} _ {N} ^ {\top} D - D ^ {2}) (\mathbf {1} _ {N} \mathbf {1} _ {N} ^ {\top} + \frac {1}{\tau} Z Z ^ {T})\right) \\ = - \frac {1}{\tau} \operatorname {T r} \left(Z ^ {\top} A Z\right) + \log \left(1 + \frac {\frac {1}{\tau} \operatorname {T r} \left(Z ^ {\top} \left(D \mathbf {1} _ {N} \mathbf {1} _ {N} ^ {\top} D - D ^ {2}\right) Z\right)}{\operatorname {T r} (D) ^ {2} - \operatorname {T r} \left(D ^ {2}\right)}\right) + \text {c o n s t} \\ \approx - \frac {1}{\tau} \operatorname {T r} \left(Z ^ {\top} A Z\right) + \frac {\frac {1}{\tau} \operatorname {T r} \left(Z ^ {\top} \left(D \mathbf {1} _ {N} \mathbf {1} _ {N} ^ {\top} D - D ^ {2}\right) Z\right)}{\operatorname {T r} (D) ^ {2} - \operatorname {T r} \left(D ^ {2}\right)} + \text {c o n s t} \\ = - \frac {1}{\tau} \operatorname {T r} \left(Z ^ {\top} \left(A - \frac {D \mathbf {1} _ {N} \mathbf {1} _ {N} ^ {\top} D - D ^ {2}}{\operatorname {T r} (D) ^ {2} - \operatorname {T r} (D ^ {2})}\right) Z\right) + \text {c o n s t .} \\ \end{array}
$$
If we further consider the constraint that the  $Z^{\top}Z = I$ , minimizing  $\widehat{\mathcal{L}}_{\mathrm{simcr}}(Z;G(\mathcal{X},w))$  boils down to the eigenvalue problem such that  $Z$  is formed by the top- $k$  eigenvectors of matrix  $(A - D1_N1_N^\top D - D^2)$ . Recall that our main analysis for Theorem 4.2 and Theorem 4.3 is based on the insight that the feature space is formed by the top- $k$  eigenvectors of the normalized adjacency matrix  $D^{-\frac{1}{2}}AD^{-\frac{1}{2}}$ . Viewed in this light, the same analysis could be applied to the SimCLR loss as well, which only differs in the concrete matrix form. We do not include the details in this paper but leave it as a future work.
E Additional Experiments Details
E.1 Experimental Details of Toy Example
Recap of set up. In Section 4.1 we consider a toy example that helps illustrate the core idea of our theoretical findings. Specifically, the example aims to cluster 3D objects of different colors and shapes, generated by a 3D rendering software [31] with user-defined properties including colors, shape, size, position, etc. Suppose the training samples come from three shapes,  $\mathcal{X}_{\square}, \mathcal{X}_{\bigcirc}, \mathcal{X}_{\bigodot}$ . Let  $\mathcal{X}_{\square}$  be the sample space with known class, and  $\mathcal{X}_{\bigcirc}, \mathcal{X}_{\bigodot}$  be the sample space with novel classes. Further, the two novel classes are constructed to have different relationships with the known class. Specifically, the toy dataset contains elements with 5 unique types:
$$
\mathcal {X} = \mathcal {X} _ {\perp} \cup \mathcal {X} _ {\bigcirc} \cup \mathcal {X} _ {\bigtriangledown},
$$
$$
\mathcal {X} _ {\square} = \{x _ {\square}, x _ {\square} \},
$$
$$
\mathcal {X} _ {\bigcirc} = \{x _ {\bigcirc}, x _ {\bigcirc} \},
$$
$$
\mathcal {X} _ {\oslash} = \{x _ {\oslash} \}.
$$
Experimental details for Figure 3(b). We rendered 2500 samples for each type of data. In total, we have 12500 samples. For known class  $\mathcal{X}$ $\square$ , we randomly select  $50\%$  as labeled data and treat the rest as unlabeled. For training, we use the same data augmentation strategy as in SimSiam [12]. We use ResNet18 and train the model for 40 epochs (sufficient for convergence) with a fixed learning rate of 0.005, using SORL defined in Eq. (6). We set  $\eta_l = 0.04$  and  $\eta_u = 1$ , respectively. Our visualization is by PyTorch implementation of UMAP [43], with parameters (n_neighbors=30, min_dist=1.5, spread=2, metric=euclidean).
E.2 Experimental Details for Benchmarks
Hardware and software. We run all experiments with Python 3.7 and PyTorch 1.7.1, using NVIDIA GeForce RTX 2080Ti and A6000 GPUs.