SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alexandru Tifrea*
alexandru.tifrea@inf.ethz.ch
Amartya Sanyal
Max Planck Institute for Intelligent Systems, Tübingen
amsa@di.ku.dk
Gizem Yüce*
gizem.yuce@epfl.ch
Fanny Yang
fan.yan@inf.ethz.ch
Prior theoretical and empirical works have established that semi-supervised learning algorithms can leverage the unlabeled data to improve over the labeled sample complexity of supervised learning (SL) algorithms. However, existing theoretical work focuses on regimes where the unlabeled data is sufficient to learn a good decision boundary using unsupervised learning (UL) alone. This begs the question: Can SSL algorithms simultaneously improve upon both UL and SL? To this end, we derive a tight lower bound for 2-Gaussian mixture models that explicitly depends on the labeled and the unlabeled dataset size as well as the signal-to-noise ratio of the mixture distribution. Surprisingly, our result implies that no SSL algorithm improves upon the minimax-optimal statistical error rates of SL or UL algorithms for these distributions. Nevertheless, in our real-world experiments, SSL algorithms can often outperform UL and SL algorithms. In summary, our work suggests that while it is possible to prove the performance gains of SSL algorithms, this would require careful tracking of constants in the theoretical analysis. $^{1}$
Semi-Supervised Learning (SSL) has recently garnered significant attention, often surpassing traditional supervised learning (SL) methods in practical applications [5, 10, 22]. Within this framework, the learning algorithm leverages both labeled and unlabeled datasets sampled from the same distribution. Numerous empirical studies suggest that SSL can effectively harness the information from both datasets, outperforming both SL and unsupervised learning (UL) approaches [21, 42, 17, 25]. This observation prompts the question: how fundamental is the improvement of SSL over SL and UL?
Prior theoretical results do not provide a consensus on this topic. One line of work demonstrates that under certain settings, SSL is capable of lowering the labeled sample complexity. In these settings, the unlabeled data possesses a significant amount of information about the conditional distribution (up to permutation). Some examples of this setting include distributions where unlabeled samples are enough to obtain small estimation error in mixture models [30, 18] or when the data is clusterable [31]. Therefore, despite SSL performing provably better than SL in these settings, it is only as good as UL (up to permutation). Another line of work challenges the above literature by identifying scenarios where SSL achieves the same error rate as SL. In these scenarios, even oracle knowledge about the
<sup>1</sup> Please consult the arxiv version of the paper for an updated manuscript.
2By error of UL we mean the prediction error up to sign. We formalize this paradigm of using UL first and then identifying the correct sign as UL+ in Section 2.2.
marginal distribution fails to improve upon the error rates of SL algorithms since the marginal does not carry any information about the labeling i.e. the conditional distribution. Therefore, these settings do not allow SSL to improve upon SL but only upon UL.
In summary, the previous bounds do not provide a conclusive answer on the benefits of SSL; the positive and negative results consider different regimes dependent on sample sizes and the "compatibility" of the marginal distribution with the conditional distribution. In particular, they either improve upon UL or upon SL, but never both simultaneously. In this paper, we provide a first answer to the following question
Can semi-supervised classification algorithms simultaneously improve over the minimax rates of both SL and UL?
Specifically, in Sections 2 and 3, we study this question in the context of linear classification for symmetric 2-Gaussian mixture models (GMMs) as done in several works in this domain [30, 18, 2, 24, 39]. In this setting, we derive minimax error rates for semi-supervised learning that specifically depend on the "regime", characterized by three quantities: the amount of available unlabeled data  $n_u$ , amount of available labeled data  $n_l$ , and the inherent signal-to-noise ratio (SNR) that quantifies the amount of information the marginal input distribution has about the conditional label distribution (see 3 for more details). An SNR-dependent minimax rate allows us to analyze the whole spectrum of problem difficulties for 2-GMMs. By contrasting the SSL minimax rates with established minimax rates for SL and UL, we find that in no regime can SSL surpass the statistical rates of both SL and UL. In conclusion, the optimal algorithm is the one that can adeptly switch between SL and UL algorithms depending on the regime and hence, it never uses the available data fully.
Nevertheless, statistical rates may not offer a complete picture for explaining the practical benefits of SSL algorithms. Several prevalent SSL algorithms, such as self-training, are more sophisticated than UL approaches and use labeled data not only for determining the sign but also for learning the decision boundary. In Section 4, we show that an SSL ensembling method and self-training [41, 7] can indeed improve upon the best of SL and UL algorithms even in proof-of-concept experiments for linear classification tasks on both synthetic and real-world datasets. Since the improvements cannot be captured by statistical rates, our results highlight the significance of the constant factors in future theoretical work that analyzes the advantage of SSL algorithms.
2 Problem setting and background
Before providing our main results, in this section, we first define the problem setting for our theoretical analysis, the evaluation metrics and the types of learning algorithms that we compare. We then describe how we compare the rates between SSL and supervised and unsupervised learning
2.1 Linear classification for 2-GMM data
Data distribution. We consider linear binary classification problems where the data is drawn from a Gaussian Mixture Model consisting of two identical spherical Gaussians with identity covariance and uniform mixing weights. The means of the two components  $\pmb{\theta}^{*}$ ,  $-\pmb{\theta}^{*}$  are symmetric with respect to the origin but can have arbitrary non-zero norm. We denote this family of joint distributions as  $\mathcal{P}_{2\text{-GMM}} := \{P_{XY}^{\pmb{\theta}^{*}} : \pmb{\theta}^{*} \in \mathbb{R}^{d}\}$  that can be factorized so that the density reads  $p_{XY}^{\pmb{\theta}^{*}}(x,y) = p_{X|Y}^{\pmb{\theta}^{*}}(x|y)p_{Y}(y)$  with
$$
P _ {Y} = \operatorname {U n i f} \{- 1, 1 \} \text {a n d} P _ {X | Y} ^ {\boldsymbol {\theta} ^ {*}} = \mathcal {N} (Y \boldsymbol {\theta} ^ {*}, I _ {d}). \tag {1}
$$
This family of distributions has often been considered in the context of analysing both SSL [30, 18] and SL/UL [2, 24, 39] algorithms. For  $s \in (0,\infty)$ , we denote by  $\mathcal{P}_{2\text{-GMM}}^{(s)} \subset \mathcal{P}_{2\text{-GMM}}$  and  $\Theta^{(s)} \subset \mathbb{R}^d$  the set of distributions  $P_{XY}^{\theta^*}$  and the set of parameters with  $\| \pmb{\theta}^* \| = s$ , respectively. With this definition, we will be able to obtain refined bounds that depend explicitly on  $s$ . We consider algorithms  $\mathcal{A}$  that take as input a labeled dataset  $\mathcal{D}_l \sim (P_{XY}^{\pmb{\theta}^*})^{n_l}$  of size  $n_l$ , an unlabeled dataset  $\mathcal{D}_u \sim (P_X^\theta)^{n_u}$  of size  $n_u$ , or both, and output an estimator  $\hat{\pmb{\theta}} = \mathcal{A}(\mathcal{D}_l, \mathcal{D}_u) \in \mathbb{R}^d$ . The estimator is used to predict the label of a test point  $x$  as  $\hat{y} = \mathrm{sign}\left(\langle \hat{\pmb{\theta}}, x \rangle\right)$ .
Learning paradigm	Excess risk rate	Estimation error rate
SL	e-s2/2dsnl	√d/nl
UL(+)	e-s2/2dS3nu	√d/s2nu
Table 1: Known minimax rates of SL and UL for learning 2-GMMs [39, 24]. The minimax rates for UL are up to choosing the correct sign. This rate is the same as for UL+ if  $n_l \geq \log(n_u)$ . The notation  $f(x) \asymp g(x)$  is equivalent to  $f = \Theta(g)$ .
Evaluation metrics In this work, we consider two natural error metrics for this class of problems: prediction error and parameter estimation error<sup>3</sup>. For any vector  $\theta$ , we define its
Prediction error:  $\mathcal{R}_{\mathrm{pred}}(\pmb {\theta},\pmb{\theta}^{*})\coloneqq P_{XY}^{\pmb{\theta}^{*}}$  (sign  $(\langle \pmb {\theta},X\rangle)\neq Y)$  , and (2)
Estimation error:  $\mathcal{R}_{\mathrm{estim}}(\pmb {\theta},\pmb{\theta}^{*})\coloneqq \| \pmb {\theta} - \pmb{\theta}^{*}\|_{2}$  (3)
When the true  $\theta^{*}$  is clear from the context, we drop the second argument for simplicity. In our discussions, we focus on the prediction error, but include a minimax rate for the estimation error for completeness. In particular, we bound the excess risk of  $\theta$ , defined as the distance between the risk of  $\theta$  and the Bayes optimal risk
Excess prediction error:  $\mathcal{E}(\theta, \theta^{*}) := \mathcal{R}_{\mathrm{pred}}(\theta, \theta^{*}) - \inf_{\theta} \mathcal{R}_{\mathrm{pred}}(\theta, \theta^{*})$ .
where  $\inf_{\pmb{\theta}}\mathcal{R}_{\mathrm{pred}}(\pmb {\theta},\pmb{\theta}^{*})$  is achieved at  $\pmb{\theta}^{*}$  but can be non-zero.
For the set of all classification algorithms, we study the minimax expected error over a set of parameters  $\Theta$ . This worst-case error over  $\Theta$  indicates the limits of what is achievable with the algorithm class. For instance, the minimax optimal expected excess error of the algorithm class over  $\Theta$  takes the form:
Minimax excess error:  $\epsilon (n_l,n_u,\Theta)\coloneqq \inf_{\mathcal{A}}\sup_{\pmb {\theta}^*\in \Theta}\mathbb{E}\left[\mathcal{E}\left(\mathcal{A}(\mathcal{D}_l,\mathcal{D}_u),\pmb{\theta}^*\right)\right]$  4
2.2 Minimax optimal rates of supervised and unsupervised learning
We distinguish between three kinds of learning that can be used in the SSL setting to learn a decision boundary  $\hat{\theta}$  but are designed to leverage the available data differently. For simplification, our discussion is tailored towards learning  $\mathcal{P}_{2\text{-GMM}}$ , though the ideas hold more generally.
1) Semi-supervised learning (SSL) SSL algorithms, denoted as  $\mathcal{A}_{\mathrm{SSL}}$ , can utilize both labeled  $\mathcal{D}_l$  and unlabeled samples  $\mathcal{D}_u$  to learn the decision boundary and to produce an estimator  $\hat{\theta}_{\mathrm{SSL}} = \mathcal{A}_{\mathrm{SSL}}(\mathcal{D}_l,\mathcal{D}_u)$ . The promise of SSL is that by combining labeled and unlabeled data, SSL can reduce both the labeled and unlabeled sample complexities compared to algorithms solely dependent on either dataset.
2) Supervised learning (SL) SL algorithms, represented by  $\mathcal{A}_{\mathrm{SL}}$ , can only use the labeled dataset  $\mathcal{D}_l$  to yield an estimator  $\hat{\pmb{\theta}}_{\mathrm{SL}} = \mathcal{A}_{\mathrm{SL}}(\mathcal{D}_l,\emptyset)$ . The minimax rates of SL for distributions from  $\mathcal{P}_{2\text{-GMM}}^{(s)}$  (see Table 1) are achieved by the mean estimator  $\hat{\pmb{\theta}}_{\mathrm{SL}} = \frac{1}{n_l}\sum_{i = 1}^{n_l}Y_iX_i$ , for both excess risk and estimation error.
3) Unsupervised learning (UL) Traditionally, UL algorithms are tailored to learning the generative model for marginal distributions. For  $\mathcal{P}_{2\text{-GMM}}$  the marginal is governed by  $\pmb{\theta}^{*}$  and UL algorithms output a set of estimators  $\{\hat{\pmb{\theta}}_{\mathrm{UL}}, - \hat{\pmb{\theta}}_{\mathrm{UL}}\} = \mathcal{A}_{\mathrm{UL}}(\emptyset ,\mathcal{D}_u)$  one of which is guaranteed to be close to the true  $\pmb{\theta}^{*}$ . To evaluate prediction performance, we define the minimax rate of UL algorithms as the minimax rate for the closer (to the true  $\pmb{\theta}^{*}$ ) of the two estimators. This minimax rate of UL algorithms over  $\mathcal{P}_{2\text{-GMM}}^{(s)}$  is known for both the excess risk and the estimation error [24, 39] (see Table 1). These rates are achieved by the unsupervised estimator  $\hat{\pmb{\theta}}_{\mathrm{UL}} = \sqrt{(\hat{\lambda} - 1)_+}\hat{v}$ , where  $(\hat{\lambda},\hat{v})$  is the leading eigenpair of the sample covariance matrix  $\hat{\Sigma} = \frac{1}{n_u}\sum_{j = 0}^{n_u}X_jX_j^T$  and we use the notation  $(x)_{+}\coloneqq \max (0,x)$ .
For linear classification and a 2-GMM distribution from the family  $\mathcal{P}_{2\text{-GMM}}$ , a low estimation error implies not only good predictive performance, but also good calibration under a logistic model [29].
2.3 A "wasteful" type of SSL algorithm
Several SSL algorithms used in practice (e.g. SimCLR [9], SwAV [8]) follow a two-stage procedure: i) determine decision boundaries using only unlabeled data; and ii) label decision regions using only labeled data. We refer to this class of two-stage algorithms as  $\mathbf{U}\mathbf{L}+$  and denote them by  $\mathcal{A}_{\mathrm{UL + }}$  . Early analyses of semi-supervised learning focus, in fact, on algorithms that fit the description of  $\mathrm{UL + }$  [30, 31].
Algorithm 1: UL+ algorithms  $\mathcal{A}_{\mathrm{UL + }}$
Input:  $\mathcal{D}_l, \mathcal{D}_u$
$$
\begin{array}{l} \left\{\hat {\boldsymbol {\theta}} _ {\mathrm {U L}}, - \hat {\boldsymbol {\theta}} _ {\mathrm {U L}} \right\} \leftarrow \mathcal {A} _ {\mathrm {U L}} \left(\mathcal {D} _ {u}\right) \\ \hat {\theta} _ {\mathrm {U L} +} \leftarrow \text {S e l e c t} \left\{\hat {\theta} _ {\mathrm {U L}}, - \hat {\theta} _ {\mathrm {U L}} \right\} \text {u s i n g} D _ {l} \\ \text {r e t u r n} \theta_ {\mathrm {U L} +} \\ \end{array}
$$
For linear binary classification, the two-stage framework is depicted in Algorithm 1. The following proposition upper bounds the excess risk and estimation error incurred by the UL+ estimator given by
$$
\hat {\boldsymbol {\theta}} _ {\mathrm {U L} +} = \operatorname {s i g n} \left(\hat {\boldsymbol {\theta}} _ {\mathrm {S L}} ^ {\top} \hat {\boldsymbol {\theta}} _ {\mathrm {U L}}\right) \hat {\boldsymbol {\theta}} _ {\mathrm {U L}} \quad \text {w i t h} \hat {\boldsymbol {\theta}} _ {\mathrm {S L}} = \mathcal {A} _ {\mathrm {S L}} \left(\mathcal {D} _ {l}\right). \tag {5}
$$
Proposition 1 (Upper bounds for  $\hat{\theta}_{\mathrm{UL + }}$ ). Let  $\hat{\theta}_{\mathrm{UL + }}$  be the estimator defined in Equation (5). For any  $s \in (0,1]$  the following holds when  $n_u \geq (160 / s)^2 d$  and  $d \geq 2$ :
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {R} _ {p r e d} \left(\hat {\boldsymbol {\theta}} _ {U L +}, \boldsymbol {\theta} ^ {*}\right) \right] \lesssim \sqrt {\frac {d}{s ^ {2} n _ {u}}} + s e ^ {- \frac {1}{2} n _ {l} s ^ {2} \left(1 - c _ {0} \sqrt {\frac {d \log (n _ {u})}{s ^ {2} n _ {u}}}\right) ^ {2}}, \quad a n d \\ \mathbb {E} \left[ \mathcal {E} \left(\hat {\pmb {\theta}} _ {U L +}, \pmb {\theta} ^ {*}\right) \right] \lesssim e ^ {- \frac {1}{2} s ^ {2}} \frac {d}{s ^ {3} n _ {u}} + e ^ {- \frac {1}{2} s ^ {2} n _ {l} \left(1 - c _ {0} \sqrt {\frac {d \log (n _ {u})}{s ^ {2} n _ {u}}}\right) ^ {2}}. \\ \end{array}
$$
Appendix A contains the complete statement of the proposition including logarithmic factors and the proof. Note that for  $n_l = o\left(\frac{1}{s^2}\log (n_u)\right)$ , the first term in the upper bound dominates, thereby resulting in the same rate as the minimax rate of UL up to choosing the correct sign. We remark that, while Algorithm 1 defines UL+ algorithms as only using the unlabeled dataset  $\mathcal{D}_u$  for the unsupervised learning step, one can also use the labeled dataset  $\mathcal{D}_l$  without labels in that step. However, typically, in practice, UL+ style algorithms (e.g. SimCLR, SwAV) do not use the labeled data in this way, as they operate in a regime where unlabeled data is far more numerous than labeled data. Thus, we analyze Algorithm 1 in this work.
Why  $\mathrm{UL}+$  algorithms are "wasteful" As indicated in Algorithm 1,  $\mathrm{UL}+$  type algorithms follow a precise structure where labeled data is used solely to select from the set of estimators output by a UL algorithm. Intuitively, such algorithms do not take full advantage of the labeled data as it is not used to refine the decision boundary. In prior empirical and theoretical studies, this inefficiency has not been a problem since they have focused on the regime  $n_u = \omega (n_l)$ , where unlabeled data is often orders of magnitude more abundant than labeled data. When  $n_u = \Theta (n_l)$  or even  $n_u\ll n_l$ , however, Proposition 1 shows how this two-stage approach of  $\mathrm{UL}+$  can become strikingly ineffective. For simplicity, consider the extreme scenario where  $n_u$  is finite, but  $n_l\to \infty$ . The error of a UL+ algorithm will, at best, mirror the error of a UL algorithm with the correct sign (e.g.  $\Theta \left(d / n_u\right)$  for the excess risk). Thus, despite using both labeled and unlabeled data, UL+ algorithms bear a close resemblance to UL algorithms that only use unlabeled data.
2.4 Brief overview of prior error bounds for SSL
In this section, we discuss prior theoretical works that aim to show benefits and limitations of SSL.
Upper bounds There are numerous known upper bounds on the excess risk of SSL algorithms for  $\mathcal{P}_{2\text{-GMM}}$  distributions. However, despite showing better dependence on the labeled set size, these earlier bounds primarily match the UL+ rates [30, 31] or exhibit slower rates than UL+ [18]. That is, we cannot conclude from existing results that SSL algorithms can consistently outperform both SL and UL+, which is the question we aim to address in this paper.
Lower bounds In contrast to the upper bounds that aim to demonstrate benefits of SSL, three distinct minimax lower bounds for SSL have been proposed to show the limitations of SSL. Each proves, in different settings, that there exists a distribution  $P_{XY}$  where SSL cannot outperform the SL minimax rate. Ben-David et al. [6] substantiate this claim for learning thresholds from univariate data
sourced from a uniform distribution on  $[0,1]$ . Gopfert et al. [20] expand upon this by considering arbitrary marginal distributions  $P_{X}$  and a "rich" set of realizable labeling functions, such that no volume of unlabeled data can differentiate between possible hypotheses. Lastly, Tolstikhin and Lopez-Paz [36] set a lower bound for scenarios with no implied association between the labeling function and the marginal distribution, a condition recognized as being unfavorable for SSL improvements [32]. Each of the aforementioned results contends that a particular worst-case distribution  $P_{XY}$  exists, where the labeled sample complexity for SSL matches that of SL, even with limitless unlabeled data.
To summarize, the upper bounds show that SSL improves upon SL in settings where  $n_u$  is significantly larger than  $n_l$ . In fact, for such large unlabeled sample size even UL+ can achieve small error. On the other hand, the lower bounds prove the futility of SSL for distributions where even infinite unlabeled data cannot help i.e.  $P_X$  does not contain sufficient information about the conditional distribution. However these works fail to answer the question: does there exist a relatively moderate  $n_u$  regime where SSL is better than both SL and UL+?
In the family of  $\mathcal{P}_{2\text{-GMM}}$  distributions, the above lower bounds translate to the hard setting where  $n_u \ll 1 / s$ . We now aim to prove a minimax lower bound for a fixed difficulty  $s > 0$ , that allows us to answer the above question in different regimes in terms of  $n_l, n_u$ .
3 Minimax rates for SSL
In this section we provide tight minimax lower bounds for SSL algorithms and 2-GMM distributions in  $\mathcal{P}_{2\text{-GMM}}^{(s)}$ . Our results indicate that it is, in fact, not possible for SSL algorithms to simultaneously achieve faster minimax rates than both SL and UL+.
3.1 Minimax rate
We begin by introducing tight lower bounds on the excess risk (4) of a linear estimator obtained using both labeled and unlabeled data. In addition, we also present tight lower bounds on estimation error (3) for the means of class-conditional distributions obtained similarly using labelled and unlabeled data. This is especially relevant when addressing linear classification of symmetric and spherical GMMs. In this setting, a reduced estimation error points to not only a low excess risk but also suggests a small calibration error under the assumption of a logistic noise model [29]. Both of these results are presented in Theorem 1. We present short proof sketches here and relegate the formal conditions required by the theorem to hold as well as the full proofs to Appendices B and C (for the estimation error and excess risk, respectively).
Theorem 1 (SSL Minimax Rate for Excess Risk and Estimation Error). Assume the conditions in Proposition 1 and additionally let  $n_l > O\left(\frac{\log n_u}{s^2}\right)$ . Then for any  $s \in (0,1]$ , we have
$$
\inf  _ {\mathcal {A} _ {S S L}} \sup  _ {\| \boldsymbol {\theta} ^ {*} \| = s} \mathbb {E} \left[ \mathcal {E} \left(\mathcal {A} _ {S S L} \left(\mathcal {D} _ {l}, \mathcal {D} _ {u}\right), \boldsymbol {\theta} ^ {*}\right) \right] \asymp e ^ {- s ^ {2 / 2}} \min  \left\{s, \frac {d}{s n _ {l} + s ^ {3} n _ {u}} \right\}, \quad a n d
$$
$$
\inf  _ {\mathcal {A} _ {S S L}} \sup  _ {\| \boldsymbol {\theta} ^ {*} \| = s} \mathbb {E} \left[ \mathcal {R} _ {e s t i m} \left(\mathcal {A} _ {S S L} \left(\mathcal {D} _ {l}, \mathcal {D} _ {u}\right), \boldsymbol {\theta} ^ {*}\right) \right] \asymp \min  \left\{s, \sqrt {\frac {d}{n _ {l} + s ^ {2} n _ {u}}} \right\},
$$
where the infimum is over all the possible SSL algorithms that have access to both unlabeled and labeled data and the expectation is over  $\mathcal{D}_l\sim (P_{XY}^{\pmb{\theta}^*})^{n_l}$  and  $\mathcal{D}_u\sim (P_X^{\pmb{\theta}^*})^{n_u}$ .
In the rest of the section, we focus solely on the bound for excess risk; however, we note that the discussion here transfers to estimation error as well. In Section 3.2, we discuss the new insights that this bound provides. A direct implication of the result is that  $\epsilon_{\mathrm{SSL}}(n_l,n_u,\Theta^{(s)})\asymp$ $\min \left(\epsilon_{\mathrm{SL}}(n_l,0,\Theta^{(s)}),\epsilon_{\mathrm{UL}+}(n_l,n_u,\Theta^{(s)})\right)$ , that is, the minimax rate of SSL is the same as either that of SL or UL+, depending on the value of  $s$  and the rate of growth of  $n_u$  compared to  $n_l$ . Therefore, we can conclude that no SSL algorithm can simultaneously improve the rates of both SL and UL+ for  $\theta \in \Theta^{(s)}$ . We provide a full discussion of the rate improvements in more detail in Section 3.2.
Proof sketch The proof of the lower bound for excess risk is presented in Appendix C. For this proof, we adopt the packing construction in Li et al. [24] and apply Fano's method. Since the algorithms have access to both labeled and unlabeled datasets in the semi-supervised setting, KL
divergences between both the marginal and the joint distributions appear in the lower bound after the application of Fano's method, which is the key difference from its SL and UL counterparts.
We then show that the rate can be matched by the SSL Switching Algorithm (SSL-S) in Algorithm 2 - an oracle algorithm that switches between using a (minimax optimal) SL or UL+ algorithm based on the values of  $s, n_{l}$ , and  $n_{u}$ . The upper bound then follows as a corollary from Proposition 1 and the upper bounds for supervised learning.
For the parameter estimation error lower bound, we use Fano's method with the packing construction in Wu and Zhou [39], who have employed this method to derive lower bounds in the context of unsupervised learning. Similar to the excess risk analysis, the lower bound reveals that the SSL rate is either determined by the SL rate or the  $\mathrm{UL + }$  rate depending on  $s$  and the ratio of the sizes of the labeled and unlabeled samples. Once again, the minimax error rate is matched by the SSL Switching algorithm presented in Algorithm 2.
Algorithm 2: SSL-S algorithm
Input:  $\mathcal{D}_l, \mathcal{D}_u, s, \mathcal{A}_{\mathrm{SL}}, \mathcal{A}_{\mathrm{UL+}}$
Result:  $\theta_{\mathrm{SSL - S}}$