SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {P} (\hat {\beta} \neq \beta) = \mathbb {P} \left(\operatorname {s i g n} \left(\hat {\boldsymbol {\theta}} _ {\mathrm {S L}} ^ {\top} \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}\right) \neq \operatorname {s i g n} \left(\boldsymbol {\theta} ^ {* \top} \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}\right)\right), \text {w h e r e} \hat {\boldsymbol {\theta}} _ {\mathrm {S L}} \sim \mathcal {N} \left(\boldsymbol {\theta} ^ {*}, \frac {1}{n _ {l}} I _ {d}\right) \\ \leq \mathbb {P} \left(\operatorname {s i g n} (\tilde {Z}) \neq \operatorname {s i g n} \left(\boldsymbol {\theta} ^ {* ^ {\top}} \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}\right)\right), \text {w h e r e} \tilde {Z} \sim \mathcal {N} \left(\hat {\boldsymbol {\theta}} _ {\mathrm {U L}} ^ {\top} \boldsymbol {\theta} ^ {*}, \frac {1}{n _ {l}} \left(\hat {\boldsymbol {\theta}} _ {\mathrm {U L}} ^ {\top} \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}\right)\right) \\ \leq \mathbb {P} \left(Z ^ {\prime} \geq | \hat {\boldsymbol {\theta}} _ {\mathrm {U L}} ^ {\top} \boldsymbol {\theta} ^ {*} |\right), \text {w h e r e} Z ^ {\prime} \sim \mathcal {N} \left(0, \frac {1}{n _ {l}} \left(\hat {\boldsymbol {\theta}} _ {\mathrm {U L}} ^ {\top} \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}\right)\right) \\ = \mathbb {P} \Big (Z \geq \sqrt {n _ {l} s ^ {2}} S _ {C} (\hat {\boldsymbol {\theta}} _ {\mathrm {U L}}, \boldsymbol {\theta} ^ {*}) \Big), \\ \end{array}
$$
where  $Z\sim \mathcal{N}(0,1)$  and  $S_{C}(\hat{\pmb{\theta}}_{\mathrm{UL}},\pmb{\theta}^{*}) = \frac{|\hat{\pmb{\theta}}_{\mathrm{UL}}^{\top}\pmb{\theta}^{*}|}{\|\hat{\pmb{\theta}}_{\mathrm{UL}}||\pmb{\theta}^{*}||}$ . Therefore, for any  $A$  we have:
$$
\begin{array}{l} \mathbb {P} (\hat {\beta} \neq \beta) \leq \mathbb {P} (Z \geq \sqrt {n _ {l} s ^ {2}} (1 - A)) + \mathbb {P} \left(S _ {C} \left(\hat {\boldsymbol {\theta}} _ {\mathrm {U L}}, \boldsymbol {\theta} ^ {*}\right) \leq 1 - A\right) \\ \leq e ^ {- \frac {1}{2} n _ {l} s ^ {2} (1 - A) ^ {2}} + \mathbb {P} \Big (S _ {C} (\hat {\boldsymbol {\theta}} _ {\mathrm {U L}}, \boldsymbol {\theta} ^ {*}) \leq 1 - A \Big), \\ \end{array}
$$
where we used the Chernoff bound in the last step. Finally, setting  $A = c_0\sqrt{\frac{d\log(n_u)}{s^2n_u}}$  as a corollary of Proposition 6 in Azizyan et al. [2] for  $n_u \geq (160 / s)^2 d$  we have  $\mathbb{P}\Big(S_C(\hat{\pmb{\theta}}_{\mathrm{UL}},\pmb {\theta}^*)\leq 1 - A\Big)\leq \frac{d}{n_u}$ . Therefore, for  $n_u \geq (160 / s)^2 d$ , we have the following upper bound on estimating the sign incorrectly:
$$
\mathbb {P} (\hat {\beta} \neq \beta) \leq e ^ {- \frac {1}{2} n _ {l} s ^ {2} \left(1 - c _ {0} \sqrt {\frac {d \log (n _ {u})}{s ^ {2} n _ {u}}}\right) ^ {2}} + \frac {d}{n _ {u}}. \tag {9}
$$
Combining this result with Equation (8) finishes the proof of the estimation error bound in the proposition, as we obtain that for some  $C_1, C_2 > 0$ , the following holds:
$$
\mathbb {E} \left[ \| \hat {\pmb {\theta}} _ {\mathrm {U L} ^ {+}} - \pmb {\theta} ^ {*} \| \right] \leq C _ {1} \sqrt {\frac {d}{s ^ {2} n _ {u}}} + C _ {2} s e ^ {- \frac {1}{4} n _ {l} s ^ {2} (1 - c _ {0} \sqrt {\frac {d \log (n _ {u})}{s ^ {2} n _ {u}}}) ^ {2}}.
$$
Similarly, we can decompose the expected excess risk as follows:
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {E} (\hat {\boldsymbol {\theta}} _ {\mathrm {U L} +}) \right] = \mathbb {E} \left[ \mathcal {E} (\hat {\beta} \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}) \right] = \mathbb {E} \left[ \mathbb {1} _ {\{\hat {\beta} = \beta \}} \mathcal {E} (\hat {\beta} \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}) + \mathbb {1} _ {\{\hat {\beta} \neq \beta \}} \mathcal {E} (\hat {\beta} \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}) \right] \\ = \mathbb {E} \left[ \mathbb {1} _ {\{\hat {\beta} = \beta \}} \mathcal {E} (\beta \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}) + \mathbb {1} _ {\{\hat {\beta} \neq \beta \}} \mathcal {E} (- \beta \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}) \right] \\ \leq \mathbb {E} \left[ \mathcal {E} \left(\beta \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}\right) + \mathbb {1} _ {\{\hat {\beta} \neq \beta \}} \right] \tag {10} \\ = \mathbb {E} \left[ \mathcal {E} \left(\beta \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}\right) \right] + \mathbb {P} (\hat {\beta} \neq \beta), \\ \end{array}
$$
where in Equation (10) we upper bound the indicator function and the excess risk terms by 1. Finally, combining Equation (9) with the upper bound in Li et al. [24] for  $\mathbb{E}\left[\mathcal{E}(\beta \hat{\pmb{\theta}}_{\mathrm{UL}})\right]$ , we obtain the desired result. More concretely, we get that for some  $C_3, C_4 > 0$
$$
\mathbb {E} \left[ \mathcal {E} (\hat {\pmb {\theta}} _ {\mathrm {U L +}}) \right] \leq C _ {3} e ^ {- \frac {1}{2} s ^ {2}} \frac {d \log (d n _ {u})}{s ^ {3} n _ {u}} + C _ {4} e ^ {- \frac {1}{2} s ^ {2} n _ {l} \left(1 - c _ {0} \sqrt {\frac {d \log (n _ {u})}{s ^ {2} n _ {u}}}\right) ^ {2}}.
$$
B Proof of SSL Minimax Rate for Estimation Error in Theorem 1
In this section we provide the proofs for the lower and upper bounds on the estimation error presented in Theorem 1. The proof for the excess risk rates can be found in Appendix C.
B.1 Proof of estimation error lower bound
We first prove the estimation error lower bound in Theorem 1. As discussed in Section 2, consider the 2-GMM distributions from  $\mathcal{P}_{2\text{-GMM}}^{(s)}$ , with isotropic components and identical covariance matrices.
Consider an arbitrary set of predictors  $\mathcal{M} = \{\theta_i\}_{i=0}^M$ . We can apply Fano's method [11] to obtain that the following holds:
$$
\begin{array}{l} \inf  _ {\mathcal {A} _ {\mathrm {S S L}} \left\| \boldsymbol {\theta} ^ {*} \right\| = s} \sup  _ {\boldsymbol {\theta} ^ {*} \mid = s} \mathbb {E} _ {\mathcal {D} _ {l}, \mathcal {D} _ {u}} \left[ \mathcal {R} _ {\text {e s t i m}} \left(\mathcal {A} _ {\mathrm {S S L}} \left(\mathcal {D} _ {l}, \mathcal {D} _ {u}\right), P _ {X Y} ^ {\boldsymbol {\theta} ^ {*}}\right) \right] \\ \geq \frac{1}{2}\min_{\substack{i,j\in [M]\\ i\neq j}}\| \boldsymbol{\theta}_{i} - \boldsymbol{\theta}_{j}\| \left(1 - \frac{1 + \frac{1}{M}\sum_{i = 1}^{M}D\left(P_{XY}^{\boldsymbol{\theta}_{i}}{}^{n_{l}}P_{X}^{\boldsymbol{\theta}_{i}}{}^{n_{u}}\| P_{XY}^{\boldsymbol{\theta}_{0}}{}^{n_{l}}P_{X}^{\boldsymbol{\theta}_{0}}{}^{n_{u}}\right)}{\log(M)}\right) \\ = \frac {1}{2} \min  _ {\substack {i, j \in [ M ] \\ i \neq j}} \| \boldsymbol {\theta} _ {i} - \boldsymbol {\theta} _ {j} \| \left(1 - \frac {1 + \frac {1}{M} \sum_ {i = 1} ^ {M} n _ {l} D \left(P _ {X Y} ^ {\boldsymbol {\theta} _ {i}} \| P _ {X Y} ^ {\boldsymbol {\theta} _ {0}}\right) + n _ {u} D \left(P _ {X} ^ {\boldsymbol {\theta} _ {i}} \| P _ {X} ^ {\boldsymbol {\theta} _ {0}}\right)}{\log (M)}\right) (11) \\ \geq \frac {1}{2} \min  _ {\substack {i, j \in [ M ] \\ i \neq j}} \| \boldsymbol {\theta} _ {i} - \boldsymbol {\theta} _ {j} \| \left(1 - \frac {1 + n _ {l} \max  _ {i \in [ M ]} D \left(P _ {X Y} ^ {\boldsymbol {\theta} _ {i}} \| P _ {X Y} ^ {\boldsymbol {\theta} _ {0}}\right) + n _ {u} \max  _ {i \in [ M ]} D \left(P _ {X} ^ {\boldsymbol {\theta} _ {i}} \| P _ {X} ^ {\boldsymbol {\theta} _ {0}}\right)}{\log (M)}\right), (12) \\ \end{array}
$$
where  $D(\cdot ||\cdot)$  denotes the KL divergence. In Equation (11), we use the fact that the labeled and unlabeled samples are drawn i.i.d. from  $P_{X}$  and  $P_{XY}$  and in Equation (12) we upper bound the average with the maximum. The next step of the proof consists in choosing an appropriate packing  $\{\pmb {\theta}_i\}_{i = 1}^M$  and  $\pmb{\theta}_{0}$  on the sphere of radius  $s$ , i.e.  $\frac{1}{s}\pmb {\theta}_i\in S^{d - 1}$ , that optimizes the trade-off between the minimum and the maxima in Equation (12).
For the packing, we use the same construction that was employed by Wu and Zhou [39] for deriving adaptive bounds for unsupervised learning. This construction has the advantage that it also leads to a tight lower bound for the supervised setting. Let  $c_{0}$  and  $C_0$  be positive absolute constants and let  $\tilde{\mathcal{M}} = \{\psi_1,\dots,\psi_M\}$  be a  $c_{0}$ -net on the unit sphere  $S^{d - 2}$  such that we have  $|\tilde{\mathcal{M}} | = M\geq e^{C_0d}$ . For an absolute constant  $\alpha \in [0,1]$ , we construct the following packing of the sphere of radius  $s$  in  $\mathbb{R}^d$ :
$$
\mathcal {M} = \left\{\boldsymbol {\theta} _ {i} = s \left[ \begin{array}{c} \sqrt {1 - \alpha^ {2}} \\ \alpha \psi_ {i} \end{array} \right] \Bigg | \psi_ {i} \in \tilde {\mathcal {M}} \right\},
$$
and define  $\pmb{\theta}_0 = [s, 0, \dots, 0]$ . Note that, by definition,  $\| \pmb{\theta}_i - \pmb{\theta}_j \| \geq c_0 s \alpha$ , for any distinct  $i, j \in [M]$  which lower bounds the first term in (12). Furthermore,  $\| \pmb{\theta}_i - \pmb{\theta}_0 \| \leq \sqrt{2} \alpha s$ , for all  $i \in [M]$ .
In the next step, we upper bound the maxima in Equation (12). First, we write the KL divergence between two GMMs with identity covariance matrices:
$$
D \left(P _ {X Y} ^ {\boldsymbol {\theta} _ {i}} \mid \mid P _ {X Y} ^ {\boldsymbol {\theta} _ {0}}\right) = \frac {1}{2} \| \boldsymbol {\theta} _ {i} - \boldsymbol {\theta} _ {0} \| _ {2} ^ {2} \leq \alpha^ {2} s ^ {2}, \text {f o r a l l} i = [ M ]. \tag {13}
$$
Second, we can upper bound the KL divergence between marginal distributions, namely  $D\left(P_X^{\boldsymbol{\theta}_i}||P_X^{\boldsymbol{\theta}_0}\right)$ , using Lemma 27 in Wu and Zhou [39], which implies that:
$$
\max  _ {i \in [ M ]} D \left(P _ {X} ^ {\boldsymbol {\theta} _ {i}} | | P _ {X} ^ {\boldsymbol {\theta} _ {0}}\right) \leq C \max  _ {i \in [ M ]} \| \frac {1}{s} \boldsymbol {\theta} _ {i} - \frac {1}{s} \boldsymbol {\theta} _ {0} \| ^ {2} s ^ {4} \leq 2 C \alpha^ {2} s ^ {4}. \tag {14}
$$
Plugging Equations (13) and (14) into Equation (12) we obtain the following lower bound for the minimax error, which holds for any  $\alpha \leq 1$ :
$$
\inf  _ {\mathcal {A} _ {\mathrm {S L}}} \sup  _ {\| \boldsymbol {\theta} ^ {*} \| = s} \mathbb {E} _ {\mathcal {D} _ {l}, \mathcal {D} _ {u}} \left[ \mathcal {R} _ {\text {e s t i m}} \left(\mathcal {A} _ {\mathrm {S L}} \left(\mathcal {D} _ {l}, \mathcal {D} _ {u}\right), \boldsymbol {\theta} ^ {*}\right) \right] \geq \frac {1}{2} c _ {o} \alpha s \left(1 - \frac {1 + n _ {l} s ^ {2} \alpha^ {2} + n _ {u} C _ {1} s ^ {4} \alpha^ {2}}{C _ {0} d}\right).
$$
Minimizing over  $\alpha$  yields the optimum value  $\alpha = \min \left\{1, \sqrt{\frac{C_0d - 1}{3s^2n_l + 3C_1s^4n_u}}\right\}$ , where the minimum comes from how we have constructed the packing, which requires that  $\alpha \leq 1$ . Using this value for  $\alpha$  concludes the proof.
B.2 Proof of estimation error upper bound
We now prove the tightness of our lower bound by establishing the upper bound for the estimation error of the SSL Switching algorithm presented in Algorithm 2. We choose the following minimax optimal SL and UL+ estimators
$$
\hat {\boldsymbol {\theta}} _ {\mathrm {S L}} = \frac {1}{n _ {l}} \sum_ {i = 1} ^ {n _ {l}} Y _ {i} X _ {i} \tag {15}
$$
$$
\hat {\boldsymbol {\theta}} _ {\mathrm {U L} +} = \operatorname {s i g n} \left(\hat {\boldsymbol {\theta}} _ {\mathrm {S L}} ^ {\top} \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}\right) \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}, \text {w i t h} \hat {\boldsymbol {\theta}} _ {\mathrm {U L}} = \sqrt {(\hat {\lambda} - 1) _ {+}} \hat {v}, \tag {16}
$$
where  $(\hat{\lambda},\hat{v})$  is the leading eigenpair of the sample covariance matrix  $\hat{\Sigma} = \frac{1}{n_u}\sum_{j = 0}^{n_u}X_jX_j^T$  and we use the notation  $(x)_{+}\coloneqq \max (0,x)$ . It is known that this UL estimator is minimax optimal [39]. As unsupervised learning can only obtain classifiers up to a sign permutation, it is necessary to endow the UL  $^+$  algorithm with a means to discern between  $\hat{\theta}_{\mathrm{UL}}$  and  $-\hat{\theta}_{\mathrm{UL}}$ , as explained in Section 2.2. In
the case of the estimator introduced in Equation (16), the sign is selected so as to obtain an estimator that aligns better with the  $\hat{\theta}_{\mathrm{SL}}$ . The upper bound for the expected error incurred by the UL+ estimator is given in Proposition 2.
For the SL estimator  $\hat{\theta}_{\mathrm{SL}}$ , we apply standard results for Gaussian distributions to upper bound the estimation error that holds for any regime of  $n$  and  $d$ :
$$
\mathbb {E} _ {\mathcal {D} _ {l}} \left[ \| \hat {\boldsymbol {\theta}} _ {\mathrm {S L}} - \boldsymbol {\theta} ^ {*} \| \right] \leq \sqrt {\frac {d}{n _ {l}}}. \tag {17}
$$
Using Equation (17) and Proposition 2 and switching between  $\hat{\theta}_{\mathrm{SL}}$  and  $\hat{\theta}_{\mathrm{UL+}}$  according to the conditions in Algorithm 2 that pick the better performing of the two algorithms depending on the regime, we can show that there exist universal constants  $C, c_0 > 0$  such that for  $0 \leq s \leq 1$ ,  $d \geq 2$  and  $n_u \geq (160 / s)^2 d$ , it holds that
$$
\mathbb {E} \left[ \| \hat {\boldsymbol {\theta}} _ {\mathrm {S L - S}} - \boldsymbol {\theta} ^ {*} \| \right] \leq C \min  \left\{s, \sqrt {\frac {d}{n _ {l}}}, \sqrt {\frac {d}{s ^ {2} n _ {u}}} + s e ^ {- \frac {1}{2} n _ {l} s ^ {2} \left(1 - c _ {0} \sqrt {\frac {d \log (n _ {u})}{s ^ {2} n _ {u}}}\right) ^ {2}} \right\}, \tag {18}
$$
where the expectation is over  $\mathcal{D}_l\sim \left(P_{XY}^{\boldsymbol{\theta}^*}\right)^{n_l}$  and  $\mathcal{D}_u\sim \left(P_X^\boldsymbol {\theta}^*\right)^{n_u}$
Matching lower and upper bound. When  $n_l > O\left(\frac{\log(n_u)}{s^2}\right)$ , the exponential term becomes negligible and the first additive component dominates in the last term in the right-hand side of Equation (18). Basic calculations then yield that the expected error of the switching algorithm is upper bounded by  $C' \min \left\{ s, \sqrt{\frac{d}{n_l + s^2 n_u}} \right\}$  for some constant  $C'$ , which concludes the proof of the theorem.
C Proof of SSL Minimax Rate for Excess Risk in Theorem 1
In this section, we prove the minimax lower bound on excess risk for an algorithm that uses both labeled and unlabeled data and a matching (up to logarithmic factors) upper bound.
C.1 Proof of excess risk lower bound
We first prove the excess error minimax lower bound in Theorem 1, namely we aim to show that there exists a constant  $C_0 > 0$  such that for any  $s > 0$ ,  $n_u, n_l \geq 0$  and  $d \geq 4$ , we have
$$
\inf  _ {\mathcal {A} _ {\mathrm {S S L}} \left\| \boldsymbol {\theta} ^ {*} \right\| = s} \sup  _ {\boldsymbol {\theta} ^ {*}} \mathbb {E} \left[ \mathcal {E} \left(\mathcal {A} _ {\mathrm {S S L}} \left(\mathcal {D} _ {l}, \mathcal {D} _ {u}\right), \boldsymbol {\theta} ^ {*}\right) \right] \geq C _ {0} e ^ {- s ^ {2} / 2} \min  \left\{\frac {d}{s n _ {l} + s ^ {3} n _ {u}}, s \right\}, \tag {19}
$$
where the expectation is over  $\mathcal{D}_l\sim (P_{XY}^{\theta^*})^{n_l}$  and  $\mathcal{D}_u\sim (P_X^\theta)^{n_u}$ . Our approach to proving this lower bound is again to apply Fano's method [19] using the excess risk as the evaluation method. The reduction from estimation to testing usually hinges on the triangle inequality in a metric space. Since the excess risk does not satisfy the metric axioms, we employ a technique introduced in Azizyan et al. [2] to derive an alternative sufficient condition for applying Fano's inequality.
Let  $\pmb{\theta}_{1},\dots,\pmb{\theta}_{M}\in \Theta$ $M\geq 2$  , and  $\gamma >0$  . If for all  $1\leq i\neq j\leq M$  and  $\hat{\pmb{\theta}}$
$$
\mathcal {E} (\hat {\boldsymbol {\theta}}, \boldsymbol {\theta} _ {i}) <   \gamma \text {i m p l i e s} \mathcal {E} (\hat {\boldsymbol {\theta}}, \boldsymbol {\theta} _ {j}) \geq \gamma , \tag {20}
$$
$$
\begin{array}{l} \inf  _ {\mathcal {A} _ {\mathrm {S S L}}} \max  _ {i \in [ 0.. M ]} \mathbb {E} \left[ \mathcal {E} \left(\mathcal {A} _ {\mathrm {S S L}} \left(\mathcal {D} _ {l}, \mathcal {D} _ {u}\right), \boldsymbol {\theta} _ {i}\right) \right] \tag {21} \\ \geq \gamma \left(1 - \frac {1 + n _ {l} \max  _ {i \neq j} D \left(P _ {X Y} ^ {\boldsymbol {\theta} _ {i}} | | P _ {X Y} ^ {\boldsymbol {\theta} _ {j}}\right) + n _ {u} \max  _ {i \neq j} D \left(P _ {X} ^ {\boldsymbol {\theta} _ {i}} | | P _ {X} ^ {\boldsymbol {\theta} _ {j}}\right)}{\log (M)}\right), \\ \end{array}
$$
where the expectation is over  $\mathcal{D}_l\sim \left(P_{XY}^{\pmb {\theta}_i}\right)^{n_l}$  and  $\mathcal{D}_u\sim \left(P_X^{\pmb {\theta}_i}\right)^{n_u}$ .
In order to get a lower bound, we again pick  $\theta_{i},\ldots ,\theta_{M}$  to be an appropriate packing, so that the condition in Equation (20) can be satisfied. For this purpose, we can simply use the construction from Li et al. [24], which was previously used to obtain tight bounds for supervised and unsupervised settings. Let  $p = (d - 1) / 6$  . By Lemma 4.10 in Massart [26], there exists a set  $\tilde{\mathcal{M}} = \{\psi_1,\dots ,\psi_M\}$  such that  $\| \psi_i\| _0 = p$ $\psi_{i}\in \{0,1\}^{d - 1}$  , the Hamming distance  $\delta (\psi_i,\psi_j) > p / 2$  for all  $1\leq i <   j\leq$ $M = |\tilde{\mathcal{M}} |$  , and  $\log M\geq \frac{p}{5}\log \frac{d}{p}\geq d\log (6) / 60 = c_1d.$
$$
\mathcal {M} = \left\{\boldsymbol {\theta} _ {i} = \left[ \begin{array}{c} \sqrt {s ^ {2} - p \alpha^ {2}} \\ \alpha \psi_ {i} \end{array} \right] \Big | \psi_ {i} \in \tilde {\mathcal {M}} \right\}
$$
for some absolute constant  $\alpha$ . Note that since  $\| \pmb{\theta}_i\| = s$  and  $\| \pmb{\theta}_i - \pmb{\theta}_j\|^2 = \alpha^2\delta (\psi_i,\psi_j)$ , we have
$$
\frac {p \alpha^ {2}}{2} \leq \| \boldsymbol {\theta} _ {i} - \boldsymbol {\theta} _ {j} \| ^ {2} \leq 2 p \alpha^ {2} \tag {22}
$$
$$
s ^ {2} - p \alpha^ {2} \leq \boldsymbol {\theta} _ {i} ^ {\top} \boldsymbol {\theta} _ {j} \leq s ^ {2} - p \alpha^ {2} / 4. \tag {23}
$$
First, we show that the excess risk satisfies the condition in Equation (20). As in the proof of Theorem 1 in Li et al. [24], we have that for any  $\theta$ ,
$$
\mathcal {E} (\pmb {\theta}, \pmb {\theta} _ {i}) + \mathcal {E} (\pmb {\theta}, \pmb {\theta} _ {j}) \geq 2 c _ {0} e ^ {- s ^ {2} / 2} \frac {p \alpha^ {2}}{s},
$$
and thus, for all  $i$  and  $j \neq i$ , it holds that
$$
\mathcal {E} (\boldsymbol {\theta}, \boldsymbol {\theta} _ {i}) \leq c _ {0} e ^ {- s ^ {2} / 2} \frac {p \alpha^ {2}}{s} \Rightarrow \mathcal {E} (\boldsymbol {\theta}, \boldsymbol {\theta} _ {j}) \geq c _ {0} e ^ {- s ^ {2} / 2} \frac {p \alpha^ {2}}{s}. \tag {24}
$$
Then since the condition in Equation (20) is satisfied, we obtain
$$
\begin{array}{l} \inf  _ {\mathcal {A} _ {\mathrm {S S L}}} \sup  _ {\| \boldsymbol {\theta} ^ {*} \| = s} \mathbb {E} _ {\mathcal {D} _ {l}, \mathcal {D} _ {u}} \left[ \mathcal {E} \left(\mathcal {A} _ {\mathrm {S S L}} \left(\mathcal {D} _ {l}, \mathcal {D} _ {u}\right), \boldsymbol {\theta} ^ {*}\right) \right] \\ \geq \inf  _ {\mathcal {A} _ {\mathrm {S S L}}} \max  _ {i \in [ 0.. M ]} \mathbb {E} \left[ \mathcal {E} \left(\mathcal {A} _ {\mathrm {S S L}} \left(\mathcal {D} _ {l}, \mathcal {D} _ {u}\right), \boldsymbol {\theta} _ {i}\right) \right] (25) \\ \geq c _ {0} e ^ {- s ^ {2} / 2} \frac {p \alpha^ {2}}{s} \left(1 - \frac {1 + n _ {l} \max  _ {i \neq j} D \left(P _ {X Y} ^ {\boldsymbol {\theta} _ {i}} \mid \mid P _ {X Y} ^ {\boldsymbol {\theta} _ {j}}\right) + n _ {u} \max  _ {i \neq j} D \left(P _ {X} ^ {\boldsymbol {\theta} _ {i}} \mid \mid P _ {X} ^ {\boldsymbol {\theta} _ {j}}\right)}{\log (M)}\right). (25) \\ \end{array}
$$
Next, we bound the KL divergences between the two joint distributions and between the two marginals that appear in Equation (25). For the joint distributions we have that:
$$
D \left(P _ {X Y} ^ {\boldsymbol {\theta} _ {i}} \| P _ {X Y} ^ {\boldsymbol {\theta} _ {j}}\right) = \frac {1}{2} \| \boldsymbol {\theta} _ {i} - \boldsymbol {\theta} _ {j} \| _ {2} ^ {2} \leq p \alpha^ {2}, \tag {26}
$$
where the inequality follows from Equation (22). Using Proposition 24 in Azizyan et al. [2], we bound the KL divergence between the two marginals
$$
D \left(P _ {X} ^ {\boldsymbol {\theta} _ {i}} \| P _ {X} ^ {\boldsymbol {\theta} _ {j}}\right) \lesssim s ^ {4} \left(1 - \frac {\boldsymbol {\theta} _ {i} ^ {\top} \boldsymbol {\theta} _ {j}}{\| \boldsymbol {\theta} _ {i} \| \| \boldsymbol {\theta} _ {j} \|}\right) \leq p s ^ {2} \alpha^ {2}, \tag {27}
$$
where the inequality follows from (23). Plugging (26) and (27) into (25) and setting
$$
\alpha^ {2} = c _ {3} \min \left\{\frac {c _ {1} d - \log 2}{8 (p n _ {l} + s ^ {2} p n _ {u})}, \frac {s ^ {2}}{p} \right\},
$$
givesthe desired result
$$
\inf  _ {\mathcal {A} _ {\mathrm {S S L}}} \sup  _ {\| \boldsymbol {\theta} ^ {*} \| = s} \mathbb {E} _ {\mathcal {D} _ {l}, \mathcal {D} _ {u}} \left[ \mathcal {E} \left(\mathcal {A} _ {\mathrm {S S L}} \left(\mathcal {D} _ {l}, \mathcal {D} _ {u}\right), \boldsymbol {\theta} ^ {*}\right) \right] \gtrsim e ^ {- s ^ {2} / 2} \min  \left\{\frac {d}{s n _ {l} + s ^ {3} n _ {u}}, s \right\}.
$$
C.2 Proof of excess risk upper bound
Next, we prove the upper bound on the excess risk of the SSL switching estimator  $\hat{\theta}_{\mathrm{SSL - S}}$  output by Algorithm 2 with the supervised and unsupervised estimators defined in Appendix B.2 to show the tightness of the excess risk lower bound in Theorem 1. In particular, we show that there exist universal constants  $C, c_0 > 0$  such that for  $0 \leq s \leq 1$ ,  $d \geq 2$  and for sufficiently large  $n_u$  and  $n_l$ ,
$$
\mathbb {E} \left[ \mathcal {E} (\hat {\boldsymbol {\theta}} _ {\mathrm {S S L - S}}) \right] \leq C e ^ {- \frac {1}{2} s ^ {2}} \min  \left\{s, \frac {d \log (n _ {l})}{s n _ {l}}, \frac {d \log (d n _ {u})}{s ^ {3} n _ {u}} + e ^ {- \frac {1}{2} s ^ {2} \left(n _ {l} \left(1 - c _ {0} \sqrt {\frac {d \log (n _ {u})}{s ^ {2} n _ {u}}}\right) ^ {2} - 1\right)} \right\},
$$
The proof follows the same arguments as the proof presented in Appendix B.2 where we instead use excess risk upper bounds for SL and UL from Li et al. [24].
In addition, we also use a result that follows from Proposition 2 to choose the sign of the UL+ estimator.
Note that the upper bound on the excess risk of  $\hat{\theta}_{\mathrm{SSL - S}}$  is matching the lower bound in Equation (19), up to logarithmic factors. We conjecture that the logarithmic factors are an artifact of the analysis and can be removed. For instance, it may be possible to extend results in Ratsaby and Venkatesh [30] that bound the excess risk using the estimation error upper bound without incurring logarithmic factors. However, their results are not directly applicable here because they are only valid under the assumption that the estimation error is arbitrarily small and hence not enough for bounding the expectation.
D Theoretical motivation for the SSL Weighted Algorithm
In this section, we show theoretically that the SSL-W procedure introduced in Section 4 can achieve lower squared estimation error (up to sign permutation) compared to SSL-S. This result shows that it is possible to improve the error of the naive SSL-S algorithm by making effective use of all the data that is available.
For the purpose of the theoretical analysis, we consider a slightly different SSL-W estimator compared to the one introduced in Section 4. First, recall that for the classification problem we consider, unsupervised learning produces a set of two feasible predictors  $\{\hat{\theta}_{\mathrm{UL}}, - \hat{\theta}_{\mathrm{UL}}\}$  and cannot discern between them without access to a (small) labeled dataset. We denote by  $\hat{\theta}_{\mathrm{UL}}^{*}$  the UL estimator with correct sign, namely  $\hat{\theta}_{\mathrm{UL}}^{*}:= \arg \min_{\pmb {\theta}\in \{\hat{\theta}_{\mathrm{UL}}, - \hat{\theta}_{\mathrm{UL}}\}}\mathbb{E}\left[\| \pmb {\theta} - \pmb{\theta}^{*}\|^{2}\right]$ . Here, the hat symbol indicates that a finite sample is used to obtain the set of estimators  $\{\hat{\theta}_{\mathrm{UL}}, - \hat{\theta}_{\mathrm{UL}}\}$ , while the star symbol indicates that oracle information is employed to choose the sign of the estimator.