SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In what follows, we study theoretically the error of the SSL-W estimator constructed using  $\hat{\theta}_{\mathrm{UL}}^{*}$ , i.e.  $\hat{\theta}_{\mathrm{SSL - W}}^{*}(t)\coloneqq t\hat{\theta}_{\mathrm{SL}} + (1 - t)\hat{\theta}_{\mathrm{UL}}^{*}$ . Therefore, our result characterizes the error of the SSL-W estimator up to a sign permutation. To choose the correct sign, one needs only a small labeled dataset, similar in size to what is prescribed by Proposition 2. While this step is not captured by Proposition 3, SSL-S is unlikely to close the gap to SSL-W when provided with this small amount of additional labeled data. Moreover, for a fairer comparison we also give the switching estimator access to the UL estimator with correct sign and define  $\hat{\theta}_{\mathrm{SSL - S}}^{*}\coloneqq \arg \min_{\pmb {\theta}\in \{\hat{\theta}_{\mathrm{SL}},\hat{\theta}_{\mathrm{UL}}^{*}\}}\mathbb{E}\left[\| \pmb {\theta} - \pmb{\theta}^{*}\|^{2}\right]$
We can now state Proposition 3, which shows that there exists an optimal weight for which the SSL-W predictor achieves lower estimation error than the SSL Switching predictor,  $\hat{\theta}_{\mathrm{SSL - S}}^{*}$ .
Proposition 3. Consider a distribution  $P_{XY}^{\boldsymbol{\theta}^*} \in \mathcal{P}_{2-GMM}^{(s)}$  and let  $d \geq 2$ , and  $n_l, n_u > 0$ . Let  $\hat{\boldsymbol{\theta}}_{SSL-W}^*(t^*)$  be the SSL-W estimator introduced above. Then there exists a  $t^* \in (0,1)$  for which
$$
\mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {S S L - S} ^ {*} - \boldsymbol {\theta} ^ {*} \right\| ^ {2} \right] - \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {S S L - W} ^ {*} (t ^ {*}) - \boldsymbol {\theta} ^ {*} \right\| ^ {2} \right] = \min  \left\{r, \frac {1}{r} \right\} \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {S S L - W} ^ {*} (t ^ {*}) - \boldsymbol {\theta} ^ {*} \right\| ^ {2} \right], \tag {28}
$$
where  $r = \frac{\mathbb{E}\left[\left\|\hat{\pmb{\theta}}_{UL}^{*} - \pmb{\theta}^{*}\right\|^{2}\right]}{\mathbb{E}\left[\left\|\hat{\pmb{\theta}}_{SL} - \pmb{\theta}^{*}\right\|^{2}\right]}$ , and the expectations are over  $\mathcal{D}_l \sim \left(P_{XY}^{\pmb{\theta}^*}\right)^{n_l}, \mathcal{D}_u \sim \left(P_X^{\pmb{\theta}^*}\right)^{n_u}$ .
Figure 2: Estimation error gap between SSL-S and SSL-W as revealed by Proposition 3 for varying SNR and  $n_l$  ( $n_u = 10000$ ). The maximum gap is reached at the switching point, indicated by the vertical dashed lines.
Since the RHS of Equation (28) is always positive,  $\hat{\pmb{\theta}}_{\mathrm{SSL - W}}^{*}(t^{*})$  always outperforms  $\hat{\pmb{\theta}}_{\mathrm{SSL - S}}^{*}$  as long as the conditions of Proposition 3 are satisfied. The magnitude of the error gap between SSL-S and SSL-W depends on the gap between SL and UL+ (see Figure 2). The maximum gap is reached for  $\mathbb{E}\left[\left\| \hat{\pmb{\theta}}_{\mathrm{UL}}^{*} - \pmb{\theta}^{*}\right\|^{2}\right] \approx \mathbb{E}\left[\left\| \hat{\pmb{\theta}}_{\mathrm{SL}} - \pmb{\theta}^{*}\right\|^{2}\right]$  when SSL-W obtains half the error of SSL-S.
D.1 Proof of Proposition 3
The first step in proving Proposition 3 is to express the estimation error of  $\hat{\theta}_{\mathrm{SSL - W}}^{*}(t^{*})$  in terms of the estimation errors of  $\hat{\theta}_{\mathrm{SL}}$  and  $\hat{\theta}_{\mathrm{UL}}^{*}$  which is captured by Lemma 1.
Lemma 1. Let  $\hat{\theta}_{1}$  and  $\hat{\theta}_{2}$  be two statistically independent estimators of  $\pmb{\theta}^{*} \in \mathbb{R}^{d}$  and let  $\hat{\theta}_{1}$  be unbiased, i.e.  $\mathbb{E}\left[\hat{\theta}_{1}\right] = \pmb{\theta}^{*}$ . Then, the expected squared error of the weighted estimator  $\hat{\theta}_{t^{*}} = t^{*}\hat{\theta}_{1} + (1 - t^{*})\hat{\theta}_{2}$  with  $t^{*} = \frac{\mathbb{E}\left[\|\hat{\theta}_{2} - \pmb{\theta}^{*}\|^{2}\right]}{\mathbb{E}\left[\|\hat{\theta}_{1} - \pmb{\theta}^{*}\|^{2}\right] + \mathbb{E}\left[\|\hat{\pmb{\theta}}_{2} - \pmb{\theta}^{*}\|^{2}\right]}$  is given by
$$
\mathbb {E} \left[ \| \hat {\boldsymbol {\theta}} _ {t ^ {*}} - \boldsymbol {\theta} ^ {*} \| ^ {2} \right] = \left(\frac {1}{\mathbb {E} \left[ \| \hat {\boldsymbol {\theta}} _ {1} - \boldsymbol {\theta} ^ {*} \| ^ {2} \right]} + \frac {1}{\mathbb {E} \left[ \| \hat {\boldsymbol {\theta}} _ {2} - \boldsymbol {\theta} ^ {*} \| ^ {2} \right]}\right) ^ {- 1}.
$$
We can apply Lemma 1, since  $\hat{\theta}_{\mathrm{SL}}$  is unbiased and  $\hat{\theta}_{\mathrm{SL}}$  and  $\hat{\theta}_{\mathrm{UL}}^{*}$  are trained on  $\mathcal{D}_l$  and  $\mathcal{D}_u$  respectively, and hence, are independent. The proof then follows from calculating the difference between the harmonic mean and the minimum of estimation errors of  $\hat{\theta}_{\mathrm{SL}}$  and  $\hat{\theta}_{\mathrm{UL + }}$ . Let  $x,y\in \mathbb{R}_+$  and w.l.o.g. assume  $x\leq y$ . Then we have:
$$
x - \left(\frac {1}{x} + \frac {1}{y}\right) ^ {- 1} = x - \frac {x y}{x + y} = \frac {x ^ {2}}{x + y} = \frac {x}{y} \frac {x y}{x + y}.
$$
Applying this identity concludes the proof after choosing  $t^{*} = \frac{\mathbb{E}\left[\|\hat{\boldsymbol{\theta}}_{2} - \boldsymbol{\theta}^{*}\|^{2}\right]}{\mathbb{E}\left[\|\hat{\boldsymbol{\theta}}_{1} - \boldsymbol{\theta}^{*}\|^{2}\right] + \mathbb{E}\left[\|\hat{\boldsymbol{\theta}}_{2} - \boldsymbol{\theta}^{*}\|^{2}\right]}$  and for  $x = \min \left\{\mathbb{E}[\|\hat{\boldsymbol{\theta}}_{\mathrm{UL}}^{*} - \boldsymbol{\theta}^{*}\|^{2}],\mathbb{E}[\|\hat{\boldsymbol{\theta}}_{\mathrm{SL}} - \boldsymbol{\theta}^{*}\|^{2}]\right\}$  and  $y = \max \left\{\mathbb{E}[\|\hat{\boldsymbol{\theta}}_{\mathrm{UL}}^{*} - \boldsymbol{\theta}^{*}\|^{2}],\mathbb{E}[\|\hat{\boldsymbol{\theta}}_{\mathrm{SL}} - \boldsymbol{\theta}^{*}\|^{2}]\right\}$ .
Remark. Note that this lemma holds for arbitrary distributions and estimators as long as they are independent and one of them is unbiased. Therefore, future results that derive upper bounds for SL and  $\mathrm{UL + }$  for other distributional assumptions and estimators can seamlessly be plugged into Lemma 1. By the same argument,  $\hat{\theta}_{\mathrm{SSL - W}}$  obtained by other SL and  $\mathrm{UL + }$  estimators can also be expected to improve over the respective SL and  $\mathrm{UL + }$  estimators, given that one of them is unbiased.
D.2 Proof of Lemma 1
By definition of  $\hat{\theta}_{t^*}$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \| \hat {\boldsymbol {\theta}} _ {t ^ {*}} - \boldsymbol {\theta} ^ {*} \| ^ {2} \right] = \mathbb {E} \left[ \| t ^ {*} \hat {\boldsymbol {\theta}} _ {1} + (1 - t ^ {*}) \hat {\boldsymbol {\theta}} _ {2} - \boldsymbol {\theta} ^ {*} \| ^ {2} \right] \\ = \mathbb {E} \left[ t ^ {* 2} \| \hat {\boldsymbol {\theta}} _ {1} - \boldsymbol {\theta} ^ {*} \| ^ {2} + (1 - t ^ {*}) ^ {2} \| \hat {\boldsymbol {\theta}} _ {2} - \boldsymbol {\theta} ^ {*} \| ^ {2} + 2 t ^ {*} (1 - t ^ {*}) (\hat {\boldsymbol {\theta}} _ {1} - \boldsymbol {\theta} ^ {*}) ^ {\top} (\hat {\boldsymbol {\theta}} _ {2} - \boldsymbol {\theta} ^ {*}) \right] \\ = \mathbb {E} \left[ t ^ {* 2} \| \hat {\boldsymbol {\theta}} _ {1} - \boldsymbol {\theta} ^ {*} \| ^ {2} + (1 - t ^ {*}) ^ {2} \| \hat {\boldsymbol {\theta}} _ {2} - \boldsymbol {\theta} ^ {*} \| ^ {2} \right], \\ \end{array}
$$
where the last equality holds due to the independence of  $\hat{\theta}_{1}$  and  $\hat{\theta}_{2}$  and the unbiasedness of  $\hat{\theta}_{1}$ .
Plugging in  $t^* = \frac{\mathbb{E}\|\hat{\pmb{\theta}}_2 - \pmb{\theta}^*\|^2}{\mathbb{E}\|\hat{\pmb{\theta}}_1 - \pmb{\theta}^*\|^2 + \mathbb{E}\|\hat{\pmb{\theta}}_2 - \pmb{\theta}^*\|^2}$ , we get
$$
\begin{array}{l} \mathbb {E} \| \hat {\pmb {\theta}} _ {t ^ {*}} - \pmb {\theta} ^ {*} \| ^ {2} = \left(\frac {\mathbb {E} \| \hat {\pmb {\theta}} _ {2} - \pmb {\theta} ^ {*} \| ^ {2}}{\mathbb {E} \| \hat {\pmb {\theta}} _ {1} - \pmb {\theta} ^ {*} \| ^ {2} + \mathbb {E} \| \hat {\pmb {\theta}} _ {2} - \pmb {\theta} ^ {*} \| ^ {2}}\right) ^ {2} \mathbb {E} \| \hat {\pmb {\theta}} _ {1} - \pmb {\theta} ^ {*} \| ^ {2} \\ + \left(\frac {\mathbb {E} \| \hat {\pmb {\theta}} _ {1} - \pmb {\theta} ^ {*} \| ^ {2}}{\mathbb {E} \| \hat {\pmb {\theta}} _ {1} - \pmb {\theta} ^ {*} \| ^ {2} + \mathbb {E} \| \hat {\pmb {\theta}} _ {2} - \pmb {\theta} ^ {*} \| ^ {2}}\right) ^ {2} \mathbb {E} \| \hat {\pmb {\theta}} _ {2} - \pmb {\theta} ^ {*} \| ^ {2} \\ = \frac {\mathbb {E} \| \hat {\boldsymbol {\theta}} _ {1} - \boldsymbol {\theta} ^ {*} \| ^ {2} \mathbb {E} \| \hat {\boldsymbol {\theta}} _ {2} - \boldsymbol {\theta} ^ {*} \| ^ {2}}{\mathbb {E} \| \hat {\boldsymbol {\theta}} _ {1} - \boldsymbol {\theta} ^ {*} \| ^ {2} + \mathbb {E} \| \hat {\boldsymbol {\theta}} _ {2} - \boldsymbol {\theta} ^ {*} \| ^ {2}} \\ = \frac {1}{\frac {1}{\mathbb {E} \| \hat {\boldsymbol {\theta}} _ {1} - \boldsymbol {\theta} ^ {*} \| ^ {2}} + \frac {1}{\mathbb {E} \| \hat {\boldsymbol {\theta}} _ {2} - \boldsymbol {\theta} ^ {*} \| ^ {2}}}. \\ \end{array}
$$
E Simulation details
E.1 Methodology
We split each dataset in a test set, a validation set and a training set. The unlabeled set size is fixed to 5000 for the synthetic experiments and 4000 for the real-world datasets. The size of the labeled set  $n_l$  is varied in each experiment. For each dataset, we draw a different labeled subset 20 times and report the average and the standard deviation of the error gap (or the error) over these runs. The (unlabeled) validation set and the test set have 1000 labeled samples each.
We use logistic regression from Scikit-Learn [28] as the supervised algorithm. We use the validation set to select the ridge penalty for SL. For the unsupervised algorithm, we use an implementation of Expectation-Maximization from the Scikit-Learn library. We also use the self-training algorithm from Scikit-Learn with a logistic regression estimator. The best confidence threshold for the pseudolabels is selected using the validation set. Moreover, the optimal weight for SSL-W is also chosen with the help of the validation set. Since we use an unlabeled validation set, we need to employ an appropriate criterion for hyperparameter selection. Therefore, we select models that lead to a large (average) margin measured on the unlabeled validation set. We give SSL-S the benefit of choosing the optimal switching point between SL and  $\mathrm{UL + }$  by using the test set. Even with this important advantage, SSL-W (and sometimes self-training) still manage to outperform SSL-S.
E.2 Details about the real-world datasets
Tabular data. We select tabular datasets from the OpenML repository [37] according to a number of criteria. We focus on high-dimensional data with  $100 \leq d < 1000$ , where the two classes are not suffering from extreme class imbalance, i.e. the imbalance ratio between the majority and the minority class is at most 5. Moreover, we only consider datasets that have substantially more samples than the number of features, i.e.  $\frac{n}{d} > 10$ . In the end, we are left with 5 datasets, that span a broad range of application domains, from ecology to chemistry and finance.
To assess the difficulty of the datasets, we train logistic regression on the entire data that is available, and report the training error. Since we train on substantially more samples than the number of
dimensions, the predictor that we obtain is a good estimate of the linear Bayes classifier for each dataset.
Furthermore, we measure the extent to which the data follows a GMM distribution with spherical components. We fit a spherical Gaussian to data coming from each class and use linear discriminant analysis (LDA) for prediction. We record the training error (of the best permutation). Intuitively, this is a score of how much our assumption about the connection between the marginal  $P_{X}$  and the labeling function  $P(Y|X)$  is satisfied. For Figure 1 we take  $\theta_{\mathrm{Bayes}}^{*}$  to be the linear Bayes classifier and  $\theta_{UL}^{*}$  the LDA classifier described above. If the data is almost linearly separable (i.e.  $\mathcal{R}_{\mathrm{pred}}(\theta_{\mathrm{Bayes}}^{*}) \leq 0.01$ ), then we simply take the linear Bayes error as the compatibility.
Image data. In addition to the tabular data, we also consider a number of datasets that are subsets of the MNIST dataset [23]. More specifically, we create binary classification problems by selecting class pairs from MNIST. We choose 5 classification problems which vary in difficulty, as measured by the Bayes error, from easier (e.g. digit 0 vs digit 1) to more difficult (e.g. digit 5 vs digit 9). Table 3 presents the exact class pairs that we selected. To make the problem more amenable for linear classification, we consider as covariates the 20 principle components of the MNIST images.
Dataset name	d	Linear classif. training error	LDA w/ spherical GMM training error
mnist_0v1	784	0.001	0.009
mnist_lv7	784	0.006	0.036
madeline	259	0.344	0.381
philippine	308	0.240	0.318
vehicleNorm	100	0.141	0.177
mnist_5v9	784	0.024	0.045
mnist_5v6	784	0.024	0.042
a9a	123	0.150	0.216
mnist_3v8	784	0.042	0.105
musk	166	0.037	0.270
Table 3: Some characteristics of the datasets considered in our experimental study.
F More experiments
In this section we present further experiments that indicate that the SSL Weighted algorithm (SSL-W) can indeed outperform the minimax optimal Switching algorithm (SSL-S). Figures 3 and 4 show that there exists a positive error gap between SSL-S and SSL-W for a broad range of  $n_l$  values, for synthetic and real-world data, respectively. The extent of the error gap is determined by the  $\frac{n_u}{n_l}$  ratio as well as the signal-to-noise ratio that is specific to each data distribution. In addition, Figure 4 also indicates that self-training can outperform SSL-W in some scenarios. It remains an exciting direction for future work to provide a tight analysis of self-training that can indicate when it outperforms both SL and UL+ simultaneously.
Figure 3: Error gap between SSL-S and SSL-W on synthetic 2-GMM data with  $s = 0.1$ . The positive gap indicates that SSL-W outperforms SSL-S (and hence, also SL and UL+).
Note that we refer to the LDA estimator as  $UL$  since we use it as a proxy to assess how well unsupervised learning can perform on each dataset.
Figure 4: Error gap between SSL-S/self-training and SSL-W on real-world datasets. The positive gap indicates that SSL-W (and, in turn, self-training) outperforms SSL-S (and hence, also SL and UL+) for a broad range of  $n_l$  values.