Title: Can semi-supervised learning use all the data effectively? A lower bound perspective
Venue: NeurIPS 2023
Accepted: yes
Keywords: semi-supervised learning

statistical lower bound
Abstract: Prior theoretical and empirical works have established that semi-supervised learning algorithms can leverage the unlabeled data to improve over the labeled sample complexity of supervised learning (SL) algorithms. However, existing theoretical work focuses on regimes where the unlabeled data is sufficient to learn a good decision boundary using unsupervised learning (UL) alone. This begs the question: Can SSL algorithms simultaneously improve upon both UL and SL? To this end, we derive a tight lower bound for 2-Gaussian mixture models that explicitly depends on the labeled and the unlabeled dataset size as well as the signal-to-noise ratio of the mixture distribution. Surprisingly, our result implies that no SSL algorithm improves upon the minimax-optimal statistical error rates of SL or UL algorithms for these distributions. Nevertheless, in our real-world experiments, SSL algorithms can often outperform UL and SL algorithms. In summary, our work suggests that while it is possible to prove the performance gains of SSL algorithms, this would require careful tracking of constants in the theoretical analysis.

Can semi-supervised learning use all the data effectively? A lower bound perspective

Abstract

Prior theoretical and empirical works have established that semi-supervised learning algorithms can leverage the unlabeled data to improve over the labeled sample complexity of supervised learning (SL) algorithms. However, existing theoretical work focuses on regimes where the unlabeled data is sufficient to learn a good decision boundary using unsupervised learning (UL) alone. This begs the question: Can SSL algorithms simultaneously improve upon both UL and SL? To this end, we derive a tight lower bound for 2-Gaussian mixture models that explicitly depends on the labeled and the unlabeled dataset size as well as the signal-to-noise ratio of the mixture distribution. Surprisingly, our result implies that no SSL algorithm improves upon the minimax-optimal statistical error rates of SL or UL algorithms for these distributions. Nevertheless, in our real-world experiments, SSL algorithms can often outperform UL and SL algorithms. In summary, our work suggests that while it is possible to prove the performance gains of SSL algorithms, this would require careful tracking of constants in the theoretical analysis. 1

1 Introduction

Semi-Supervised Learning (SSL) has recently garnered significant attention, often surpassing traditional supervised learning (SL) methods in practical applications [5, 10, 22]. Within this framework, the learning algorithm leverages both labeled and unlabeled datasets sampled from the same distribution. Numerous empirical studies suggest that SSL can effectively harness the information from both datasets, outperforming both SL and unsupervised learning (UL) approaches [21, 42, 17, 25]. This observation prompts the question: how fundamental is the improvement of SSL over SL and UL? 2

Prior theoretical results do not provide a consensus on this topic. One line of work demonstrates that under certain settings, SSL is capable of lowering the labeled sample complexity. In these settings, the unlabeled data possesses a significant amount of information about the conditional distribution (up to permutation). Some examples of this setting include distributions where unlabeled samples are enough to obtain small estimation error in mixture models [30, 18] or when the data is clusterable [31]. Therefore, despite SSL performing provably better than SL in these settings, it is only as good as UL (up to permutation). Another line of work challenges the above literature by identifying scenarios where SSL achieves the same error rate as SL. In these scenarios, even oracle knowledge about the

marginal distribution fails to improve upon the error rates of SL algorithms since the marginal does not carry any information about the labeling i.e. the conditional distribution. Therefore, these settings do not allow SSL to improve upon SL but only upon UL.

In summary, the previous bounds do not provide a conclusive answer on the benefits of SSL; the positive and negative results consider different regimes dependent on sample sizes and the 'compatibilibity' of the marginal distribution with the conditional distribution. In particular, they either improve upon UL or upon SL, but never both simultaneously. In this paper, we provide a first answer to the following question

Can semi-supervised classification algorithms simultaneously improve over the minimax rates of both SL and UL?

Specifically, in Sections 2 and 3, we study this question in the context of linear classification for symmetric 2-Gaussian mixture models (GMMs) as done in several works in this domain [30, 18, 2, 24, 39]. In this setting, we derive minimax error rates for semi-supervised learning that specifically depend on the "regime", characterized by three quantities: the amount of available unlabeled data n u , amount of available labeled data n l , and the inherent signal-to-noise ratio (SNR) that quantifies the amount of information the marginal input distribution has about the conditional label distribution (see 3 for more details). An SNR-dependent minimax rate allows us to analyze the whole spectrum of problem difficulties for 2-GMMs. By contrasting the SSL minimax rates with established minimax rates for SL and UL, we find that in no regime can SSL surpass the statistical rates of both SL and UL. In conclusion, the optimal algorithm is the one that can adeptly switch between SL and UL algorithms depending on the regime and hence, it never uses the available data fully.

Nevertheless, statistical rates may not offer a complete picture for explaining the practical benefits of SSL algorithms. Several prevalent SSL algorithms, such as self-training, are more sophisticated than UL approaches and use labeled data not only for determining the sign but also for learning the decision boundary. In Section 4, we show that an SSL ensembling method and self-training [41, 7] can indeed improve upon the best of SL and UL algorithms even in proof-of-concept experiments for linear classification tasks on both synthetic and real-world datasets. Since the improvements cannot be captured by statistical rates, our results highlight the significance of the constant factors in future theoretical work that analyzes the advantage of SSL algorithms.

2 Problem setting and background

Before providing our main results, in this section, we first define the problem setting for our theoretical analysis, the evaluation metrics and the types of learning algorithms that we compare. We then describe how we compare the rates between SSL and supervised and unsupervised learning

2.1 Linear classification for 2-GMM data

Data distribution. We consider linear binary classification problems where the data is drawn from a Gaussian Mixture Model consisting of two identical spherical Gaussians with identity covariance and uniform mixing weights. The means of the two components θ ∗ , -θ ∗ are symmetric with respect to the origin but can have arbitrary non-zero norm. We denote this family of joint distributions as P 2-GMM := { P θ ∗ XY : θ ∗ ∈ R d } that can be factorized so that the density reads p θ ∗ XY ( x, y ) = p θ ∗ X | Y ( x | y ) p Y ( y ) with

This family of distributions has often been considered in the context of analysing both SSL [30, 18] and SL/UL [2, 24, 39] algorithms. For s ∈ (0 , ∞ ) , we denote by P ( s ) 2-GMM ⊂ P 2-GMM and Θ ( s ) ⊂ R d the set of distributions P θ ∗ XY and the set of parameters with ∥ θ ∗ ∥ = s , respectively. With this definition, we will be able to obtain refined bounds that depend explicitly on s . We consider algorithms A that take as input a labeled dataset D l ∼ ( P θ ∗ XY ) n l of size n l , an unlabeled dataset D u ∼ ( P θ ∗ X ) n u of size n u , or both, and output an estimator ˆ θ = A ( D l , D u ) ∈ R d . The estimator is used to predict the label of a test point x as ˆ y = sign ( ⟨ ˆ θ , x ⟩ ) .

Evaluation metrics In this work, we consider two natural error metrics for this class of problems: prediction error and parameter estimation error 3 . For any vector θ , we define its

̸

When the true θ ∗ is clear from the context, we drop the second argument for simplicity. In our discussions, we focus on the prediction error, but include a minimax rate for the estimation error for completeness. In particular, we bound the excess risk of θ , defined as the distance between the risk of θ and the Bayes optimal risk

where inf θ R pred ( θ , θ ∗ ) is achieved at θ ∗ but can be non-zero.

For the set of all classification algorithms, we study the minimax expected error over a set of parameters Θ . This worst-case error over Θ indicates the limits of what is achievable with the algorithm class. For instance, the minimax optimal expected excess error of the algorithm class over Θ takes the form:

2.2 Minimax optimal rates of supervised and unsupervised learning

We distinguish between three kinds of learning that can be used in the SSL setting to learn a decision boundary ˆ θ but are designed to leverage the available data differently. For simplification, our discussion is tailored towards learning P 2-GMM, though the ideas hold more generally.

1) Semi-supervised learning (SSL) SSL algorithms, denoted as A SSL, can utilize both labeled D l and unlabeled samples D u to learn the decision boundary and to produce an estimator ˆ θ SSL = A SSL ( D l , D u ) . The promise of SSL is that by combining labeled and unlabeled data, SSL can reduce both the labeled and unlabeled sample complexities compared to algorithms solely dependent on either dataset.

2) Supervised learning (SL) SL algorithms, represented by A SL, can only use the labeled dataset D l to yield an estimator ˆ θ SL = A SL ( D l , ∅ ) . The minimax rates of SL for distributions from P ( s ) 2-GMM (see Table 1) are achieved by the mean estimator ˆ θ SL = 1 n l ∑ n l i =1 Y i X i , for both excess risk and estimation error.

3) Unsupervised learning (UL) Traditionally, UL algorithms are tailored to learning the generative model for marginal distributions. For P 2-GMM the marginal is governed by θ ∗ and UL algorithms output a set of estimators { ˆ θ UL , -ˆ θ UL } = A UL ( ∅ , D u ) one of which is guaranteed to be close to the true θ ∗ . To evaluate prediction performance, we define the minimax rate of UL algorithms as the minimax rate for the closer (to the true θ ∗ ) of the two estimators. This minimax rate of UL algorithms over P ( s ) 2-GMM is known for both the excess risk and the estimation error [24, 39]

(see Table 1). These rates are achieved by the unsupervised estimator ˆ θ UL = √ ( ˆ λ -1) + ˆ v , where ( ˆ λ, ˆ v ) is the leading eigenpair of the sample covariance matrix ˆ Σ = 1 n u ∑ n u j =0 X j X T j and we use the notation ( x ) + := max(0 , x ) .

2.3 A "wasteful" type of SSL algorithm

Several SSL algorithms used in practice (e.g. SimCLR [9], SwAV [8]) follow a two-stage procedure: i) determine decision boundaries using only unlabeled data; and ii) label decision regions using only labeled data. We refer to this class of two-stage algorithms as UL+ and denote them by A UL+. Early analyses of semi-supervised learning focus, in fact, on algorithms that fit the description of UL+ [30, 31].

Algorithm 1: UL+ algorithms A

For linear binary classification, the two-stage framework is depicted in Algorithm 1. The following proposition upper bounds the excess risk and estimation error incurred by the UL+ estimator given by

Proposition 1 (Upper bounds for ˆ θ UL+ ) . Let ˆ θ UL+ be the estimator defined in Equation (5) . For any s ∈ (0 , 1] the following holds when n u ≥ (160 /s ) 2 d and d ≥ 2 :

Appendix A contains the complete statement of the proposition including logarithmic factors and the proof. Note that for n l = o ( 1 s 2 log ( n u ) ) , the first term in the upper bound dominates, thereby resulting in the same rate as the minimax rate of UL up to choosing the correct sign. We remark that, while Algorithm 1 defines UL+ algorithms as only using the unlabeled dataset D u for the unsupervised learning step, one can also use the labeled dataset D l without labels in that step. However, typically, in practice, UL+ style algorithms (e.g. SimCLR, SwAV) do not use the labeled data in this way, as they operate in a regime where unlabeled data is far more numerous than labeled data. Thus, we analyze Algorithm 1 in this work.

Why UL+ algorithms are 'wasteful' As indicated in Algorithm 1, UL+ type algorithms follow a precise structure where labeled data is used solely to select from the set of estimators output by a UL algorithm. Intuitively, such algorithms do not take full advantage of the labeled data as it is not used to refine the decision boundary. In prior empirical and theoretical studies, this inefficiency has not been a problem since they have focused on the regime n u = ω ( n l ) , where unlabeled data is often orders of magnitude more abundant than labeled data. When n u = Θ( n l ) or even n u ≪ n l , however, Proposition 1 shows how this two-stage approach of UL+ can become strikingly ineffective. For simplicity, consider the extreme scenario where n u is finite, but n l → ∞ . The error of a UL+ algorithm will, at best, mirror the error of a UL algorithm with the correct sign (e.g. Θ( d / n u ) for the excess risk). Thus, despite using both labeled and unlabeled data, UL+ algorithms bear a close resemblance to UL algorithms that only use unlabeled data.

2.4 Brief overview of prior error bounds for SSL

In this section, we discuss prior theoretical works that aim to show benefits and limitations of SSL.

Upper bounds There are numerous known upper bounds on the excess risk of SSL algorithms for P 2-GMM distributions. However, despite showing better dependence on the labeled set size, these earlier bounds primarily match the UL+ rates [30, 31] or exhibit slower rates than UL+ [18]. That is, we cannot conclude from existing results that SSL algorithms can consistently outperform both SL and UL+, which is the question we aim to address in this paper.

Lower bounds In contrast to the upper bounds that aim to demonstrate benefits of SSL, three distinct minimax lower bounds for SSL have been proposed to show the limitations of SSL. Each proves, in different settings, that there exists a distribution P XY where SSL cannot outperform the SL minimax rate. Ben-David et al. [6] substantiate this claim for learning thresholds from univariate data

sourced from a uniform distribution on [0 , 1] . Göpfert et al. [20] expand upon this by considering arbitrary marginal distributions P X and a 'rich' set of realizable labeling functions, such that no volume of unlabeled data can differentiate between possible hypotheses. Lastly, Tolstikhin and LopezPaz [36] set a lower bound for scenarios with no implied association between the labeling function and the marginal distribution, a condition recognized as being unfavorable for SSL improvements [32]. Each of the aforementioned results contends that a particular worst-case distribution P XY exists, where the labeled sample complexity for SSL matches that of SL, even with limitless unlabeled data.

To summarize, the upper bounds show that SSL improves upon SL in settings where n u is significantly larger than n l . In fact, for such large unlabeled sample size even UL+ can achieve small error. On the other hand, the lower bounds prove the futility of SSL for distributions where even infinite unlabeled data cannot help i.e. P X does not contain sufficient information about the conditional distribution. However these works fail to answer the question: does there exist a relatively moderate n u regime where SSL is better than both SL and UL+ ?

In the family of P 2-GMM distributions, the above lower bounds translate to the hard setting where n u ≪ 1 / s . We now aim to prove a minimax lower bound for a fixed difficulty s > 0 , that allows us to answer the above question in different regimes in terms of n l , n u .

3 Minimax rates for SSL

In this section we provide tight minimax lower bounds for SSL algorithms and 2-GMM distributions in P ( s ) 2-GMM . Our results indicate that it is, in fact, not possible for SSL algorithms to simultaneously achieve faster minimax rates than both SL and UL+.

3.1 Minimax rate

We begin by introducing tight lower bounds on the excess risk (4) of a linear estimator obtained using both labeled and unlabeled data. In addition, we also present tight lower bounds on estimation error (3) for the means of class-conditional distributions obtained similarly using labelled and unlabelled data. This is especially relevant when addressing linear classification of symmetric and spherical GMMs. In this setting, a reduced estimation error points to not only a low excess risk but also suggests a small calibration error under the assumption of a logistic noise model [29]. Both of these results are presented in Theorem 1. We present short proof sketches here and relegate the formal conditions required by the theorem to hold as well as the full proofs to Appendices B and C (for the estimation error and excess risk, respectively).

Theorem 1 (SSL Minimax Rate for Excess Risk and Estimation Error) . Assume the conditions in Proposition 1 and additionally let n l > O ( log n u s 2 ) . Then for any s ∈ (0 , 1] , we have

where the infimum is over all the possible SSL algorithms that have access to both unlabeled and labeled data and the expectation is over D l ∼ ( P θ ∗ XY ) n l and D u ∼ ( P θ ∗ X ) n u .

In the rest of the section, we focus solely on the bound for excess risk; however, we note that the discussion here transfers to estimation error as well. In Section 3.2, we discuss the new insights that this bound provides. A direct implication of the result is that ϵ SSL ( n l , n u , Θ ( s ) ) ≍ min ( ϵ SL ( n l , 0 , Θ ( s ) ) , ϵ UL+ ( n l , n u , Θ ( s ) )) , that is, the minimax rate of SSL is the same as either that of SL or UL+, depending on the value of s and the rate of growth of n u compared to n l . Therefore, we can conclude that no SSL algorithm can simultaneously improve the rates of both SL and UL+ for θ ∈ Θ ( s ) . We provide a full discussion of the rate improvements in more detail in Section 3.2.

Proof sketch The proof of the lower bound for excess risk is presented in Appendix C. For this proof, we adopt the packing construction in Li et al. [24] and apply Fano's method. Since the algorithms have access to both labeled and unlabeled datasets in the semi-supervised setting, KL

divergences between both the marginal and the joint distributions appear in the lower bound after the application of Fano's method, which is the key difference from its SL and UL counterparts.

We then show that the rate can be matched by the SSL Switching Algorithm (SSL-S) in Algorithm 2 - an oracle algorithm that switches between using a (minimax optimal) SL or UL+ algorithm based on the values of s, n l , and n u . The upper bound then follows as a corollary from Proposition 1 and the upper bounds for supervised learning.

For the parameter estimation error lower bound, we use Fano's method with the packing construction in Wu and Zhou [39], who have employed this method to derive lower bounds in the context of unsupervised learning. Similar to the excess risk analysis, the lower bound reveals that the SSL rate is either determined by the SL rate or the UL+ rate depending on s and the ratio of the sizes of the labeled and unlabeled samples. Once again, the minimax error rate is matched by the SSL Switching algorithm presented in Algorithm 2.

Discussion of the details of the theorem We note that the SSL-S algorithm chooses to output the trivial estimator ˆ θ SSL-S = 0 if the SNR s is low. In the low-SNR regime, the trivial estimator ˆ θ = 0 achieves a small excess risk of E ( 0 , θ ∗ ) = Φ( s ) -Φ(0) ≃ e -s 2 / 2 s , where Φ is the CDF of the standard normal distribution.

Moreover, the technical condition s ∈ (0 , 1] is not overly restrictive as this range for the SNR is already sufficient for seeing all the relevant regimes, e.g. SSL-S switches from using SL to UL+ within this range. Finally, while the rates of either SL or UL+ cannot be improved further using SSL algorithms, it is nonetheless possible to improve the error by a constant factor, independent of n l and n u . To see this, in Section 4 we describe an algorithm that uses both D l and D u effectively and can hence achieve a provable improvement in error over both SL and UL+.

3.2 Comparison of SSL with UL+ and SL in different regimes

To understand whether an SSL algorithm is using the labeled and unlabeled data effectively, we compare the error rate of SSL algorithms to the minimax rates for SL and UL+ algorithms.

A gap in these rates for a certain SSL algorithm would indicate that the algorithm can obtain lower error than both SL and UL+ when presented with the same amount of (large enough) data. We study the improvement rates defined below, which capture the error ratio for the worst-case distributions between the minimax rate for SSL and the minimax rate for SL or UL+ .

Definition 1 (SSL improvement rates) . For a set of parameters Θ ⊆ R d , we define the improvement rates of SSL over SL and UL+ as h l and h u , respectively, where

where the expectations are over D l ∼ ( P θ ∗ XY ) n l and D u ∼ ( P θ ∗ X ) n u .

To simplify notation, we denote the improvement rates of SL and UL+ over Θ ( s ) as h l ( n l , n u , s ) and h u ( n l , n u , s ) , respectively. A straightforward upper bound for these rates is h l , h u ≤ 1 , achieved when utilizing an SL and UL+ algorithm, respectively. SSL demonstrates an enhanced error rate over SL and UL+, if both lim n l ,n u →∞ h l ( n l , n u , Θ) = 0 and lim n l ,n u →∞ h u ( n l , n u , Θ) = 0 . If h l or h u lies in (0 , 1) without converging to zero as n l , n u →∞ , then SSL surpasses SL or UL+, respectively, only by a constant factor.

Given Theorem 1, we can directly derive the improvement rates of SSL in the following corollary. Corollary 1. Assuming the setting of Theorem 1, the improvement rates of SSL can be written as:

Based on this corollary, we now argue how no SSL algorithm can simultaneously improve the rates of both SL and UL+ for any regime of s, n l and n u , as summarized in Table 2.

SSL rate is not faster than SL, but is faster than UL+ . For extremely low values of SNR (i.e. s → 0 faster than √ n u →∞ ), we have that lim n l ,n u →∞ h l ( n l , n u , s ) = c SL > 0 and hence SSL fails to improve the SL rates even with infinite unlabeled data. This setting has been the focus of previous worst-case analyses for SSL [6, 36, 20]. Alternatively, the SSL rate is also not better than the SL rate for a fixed SNR, when the labeled dataset is significantly larger than the unlabeled data, i.e. n u = o ( n l ) .

SSL rate is faster than SL, but not faster than UL+. As mentioned in Göpfert et al. [20], in order for SSL to lead to a rate improvement compared to SL, it is necessary that the unlabeled set size is at least a superlinear function of the labeled set size, i.e. n u = ω ( n l ) . Corollary 1 shows that this condition is, in fact, sufficient for 2-GMM distributions: for s > 0 , as long as n u grows superlinearly with respect to n l , h l ( n l , n u , s ) → 0 , and hence, SSL can achieve faster rates than SL. Despite the improvement over SL, for this setting, the asymptotic error ratio between SSL and UL+ does not vanish, i.e. lim n l ,n u →∞ h u ( n l , n u , s ) = c UL > 0 .

SSL rate is not faster than either SL or UL+. Finally, in the regime where the unlabeled dataset size depends linearly on the size of the labeled set i.e. lim n l ,n u →∞ n u n l → c for some constant c > 0 , neither of the improvement rates vanishes for n l , n u →∞ .

As explained in Section 2.2, prior UL+ algorithms that work well in practice do not use the labeled data for the unsupervised learning step. Interestingly, for the case when both the labeled and unlabeled data are used for the unsupervised step, the trends in Table 2 remain the same, with only one exception: the improvement rates in the last line of the table (i.e. n u n l → c ∈ (0 , ∞ ) ) would only change by a small constant. More importantly, the main takeaway from Table 2 remains the same: SSL cannot achieve better rates than both UL+ and SL at the same time since there is no regime for which h l and h u are simultaneously 0 .

4 Empirical improvements over the rate-optimal SSL-S algorithm

Section 3 shows that the SSL minimax rates can be achieved with a simple algorithm that switches between a minimax optimal SL and a minimax optimal UL+ algorithm. Despite being optimal in terms of statistical rates, this SSL Switching algorithm does not make the most effective use of the available data: SL algorithms solely uses the labeled data whereas UL+ learns the decision boundary using just the unlabeled data and the labeled data is used to only label the different prediction regions. Alternatively, one could use both labeled and unlabeled data to learn the decision boundary. In this section, we investigate the following question: Is it possible to improve over the error of SSL-S even though the rate would be, at best, as good as the 'wasteful' SSL-S algorithm as per Section 3?

In this section, we present experiments to show that, in fact, a remarkably simple algorithm (i.e. a weighted ensemble of the SL and UL+ classifiers) can outperform the minimax optimal SSL-S algorithm. We show that algorithms such as self-training, which have been shown to excel in practice [40, 34, 22], can also improve over SSL-S. Consequently, it remains an exciting avenue for future work to derive tight analyses that characterize the improvement of such algorithms over SL/UL+.

A simple algorithm more effective than SSL-S A natural approach to improve upon the naive SSL-S algorithm is to construct a weighted ensemble of an SL and a UL+ estimator, trained on D l and D u , respectively, with a controllable weighting hyperparameter t . We call this the SSL Weighted algorithm (SSL-W) shown in Algorithm 3. With an appropriate choice of t , it is possible to show that the SSL-W algorithm performs better (up to sign permutation) than SSL-S. The formal statement of this re-

Algorithm 3: SSL-W algorithm

Input :

D l , D u , t

Result:

ˆ

θ

SSL-W

ˆ θ SL ←A SL ( D l )

ˆ

θ

UL+

←A

UL+

(

D

l

,

D

u

)

ˆ

θ

SSL-W

(

t

) =

t

ˆ

θ

SL

+(1

-

t

) ˆ

θ

UL+

return ˆ θ SSL-W ( t )

sult together with the proof are deferred to Appendix D. In practice, one can fix the sign permutation of the ˆ θ SSL-W estimator using a small amount of labeled data. The intuition for this improvement is that the ensemble estimator ˆ θ SSL-W achieves lower error than the constituent estimators of the ensemble (i.e. ˆ θ SL and ˆ θ UL+), which, in turn, determine the error of the SSL-S algorithm.

4.1 Empirical improvements over the minimax optimal SSL-S algorithm

In this section we present linear classification experiments on synthetic and real-world data to show that there indeed exist SSL algorithms that can improve over the error of the (minimax optimal) SSL Switching Algorithm. For both synthetic and real-world data, we use ˆ θ SL = 1 n l ∑ n l i =1 Y i X i as the SL estimator and an Expectation-Maximization (EM) algorithm for the UL algorithm (see Appendix E for details). The optimal switching point for SSL-S and the optimal weight for SSL-W, as well as the optimal ℓ 2 penalty for logistic regression are chosen using a validation set.

Synthetic data. We consider data drawn from a symmetric and isotropic 2-GMM distribution P θ ∗ XY over R 2 . The labeled and unlabeled set sizes are set to 20 and 2000 , respectively. Figure 1a shows the gap between SSL-W and SL or UL+ as a function of the SNR s (Figure 3 in Appendix F shows the dependence of the error gap on n l ). There are two main takeaways. First, for varying SNR values s , SSL-W always outperforms SL and UL+, and hence, also SSL-S. Second, as argued in Section 3.2, SSL-W improves more over UL+ for small values of the SNR s , and it improves more over SL for large values of the SNR.

Real-world data. We consider 10 binary classification real-world datasets: five from the OpenML repository [37] and five 2-class subsets of the MNIST dataset [13]. For the MNIST subsets, we choose class pairs that have a linear Bayes error varying between 0 . 1% and 2 . 5% . 4 We choose from OpenML datasets that have a large enough number of samples compared to dimensionality (see Appendix E for details on how we choose the datasets). The OpenML datasets span a range of Bayes errors that varies between 3% and 34% .

In the absence of the exact data-generating process, we quantify the difficulty of SSL on real-world datasets using a notion of compatibility , reminiscent of Balcan and Blum [4]. Specifically, we consider the compatibility given by ρ -1 with ρ := 1 2 √ d ( ∆( θ ∗ UL + , θ ∗ Bayes ) + R pred ( θ ∗ Bayes ) ) , where

∆( θ ∗ UL + , θ ∗ Bayes ) := R pred ( θ UL + ) -R pred ( θ Bayes ) R pred ( θ ∗ Bayes ) , d is the dimension of the data, θ ∗ Bayes is obtained via SL on the entire dataset and θ ∗ UL+ determines the predictor with optimal sign obtained via UL on the entire dataset. Intuitively, this notion of compatibility captures both the Bayes error of a dataset, as well as how compatible the 2-GMM parametric assumption actually is for the given data.

In addition to SSL-S (Algorithm 2) and SSL-W (Algorithm 3) we also evaluate the performance of self-training, using a procedure similar to the one analyzed in Frei et al. [18]. We use a logistic regression estimator for the pseudolabeling, and train logistic regression with a ridge penalty in the second stage of the self-training procedure. Note that an ℓ 2 penalty corresponds to input consistency regularization [38] with respect to ℓ 2 perturbations.

Error gap

10

1

0

0.05

0.10

0.15

0.20

0.25

Err(SL)  -  Err(SSL-W)

Err(UL+) - Err(SSL-W)

0.30

0.35

0.40

SNR

(a) 2-GMM synthetic data.

musk

mnist_3v8

mnist_5v9

mnist_5v6

a9a

vehicleNorm

philippine

madeline

mnist_1v7

mnist_0v1

Datasets ordered by compatibility (low to large)

0.10

0.05

0.00

0.05

0.10

0.15

0.20

Error gap

Err(SL)  -  Err(SSL-W)

Err(UL+) - Err(SSL-W)

Err(SL)  -  Err(SelfT)

Err(UL+) - Err(SelfT)

(b) Real-world data.

Figure 1b shows the improvement in classification error of SSL algorithms (i.e. SSL-W and selftraining) compared to SL and UL+ . The positive values of the error gap indicate that SSL-W outperforms both SL and UL+ even on real-world datasets. This finding suggests that the intuition presented in this section carries over to more generic distributions beyond just 2-GMMs. Finally, Figure 4 in Appendix F shows that SSL-W improves over the error of the minimax optimal SSL-S algorithm, for varying values of n l . These findings provide some proof-of-concept evidence that existing SSL algorithms may already be obtaining lower error than the minimax optimal SSL-S algorithm. We hope this observation will encourage future research into characterizing this error gap for practically relevant algorithms such as self-training.

5 Related work

Other theoretical analyses of SSL algorithms. Beyond the theoretical studies highlighted in Section 2, there are a few others pertinent to our research. Specifically, Azizyan et al. [1], Singh et al. [33] present upper bounds for semi-supervised regression, which are contingent on the degree to which the marginal P X informs the labeling function. This is akin to the results we derive in this work. However, obtaining a minimax lower bound for semi-supervised regression remains an exciting direction for future work. We refer to [27] for an overview of prior theoretical results for SSL.

Balcan and Blum [4] introduced a compatibility score, denoted as χ ( f, P X ) ∈ [0 , 1] , which connects the space of marginal distributions to the space of labeling functions. While their findings hint that SSL may surpass the SL minimax rates, they offer no comparisons with UL/UL+. Moreover, the paper does not discuss minimax optimality of the proposed SSL algorithms.

On another note, even though SSL does not enhance the rates of UL, Sula and Zheng [35] demonstrate that labeled samples can bolster the convergence speed of Expectation-Maximization within the context of our study.

To conclude, Schölkopf et al. [32] leveraged a causality framework to pinpoint scenarios where SSL does not offer any advantage over SL. In essence, when the covariates, represented by X , act as causal ancestors to the labels Y , the independent causal mechanism assumption dictates that the marginal P X offers no insights about the labeling function.

Minimax rates for SL and UL. The proofs in this work rely on techniques used to derive minimax rates for SL and UL algorithms. Most of these prior results consider the same distributional assumptions as our paper. Wu and Zhou [39] show a tight minimax lower bound for estimation error for spherical 2-GMMs from P 2-GMM. Moreover, Azizyan et al. [2], Li et al. [24] derive minimax rates over P 2-GMM for classification and clustering (up to permutation).

In addition to the SL and UL algorithms considered in Section 3, Expectation-Maximization (EM) is another family of algorithms that is commonly analyzed for the same distributional setting considered in our paper. For instance, Wu and Zhou [39] rely on techniques from several previous seminal papers [12, 3, 14-16] to obtain upper bounds for EM-style algorithms.

6 Conclusions and limitations

In this paper, we establish a tight lower bound for semi-supervised classification within the class of 2-GMM distributions. Our findings demonstrate that SSL cannot simultaneously improve the error rates of both SL and UL across all signal-to-noise ratios. However, empirical evidence suggests that SSL can improve upon the error of minimax optimal SL or UL algorithms. This observation calls for careful analyses of the error of SSL algorithms that also track constant factors, not only rates.

Our theoretical analysis focuses exclusively on isotropic and symmetric GMMs due to limitations in the technical tools employed for the proofs. Similar constraints can be observed in recent analyses of SL or UL algorithms [24, 39]. However, it is worth noting that whenever the bounds for SL and UL can be extended to more general distributions in the future, these results can be seamlessly used to also extend Theorem 1 to these settings.

References

Martin Azizyan, Aarti Singh, and Larry Wasserman. Density-sensitive semisupervised inference. The Annals of Statistics , 2013.

Martin Azizyan, Aarti Singh, and Larry Wasserman. Minimax theory for high-dimensional gaussian mixtures with sparse mean separation. In Advances in Neural Information Processing Systems , 2013.

Sivaraman Balakrishnan, Martin J. Wainwright, and Bin Yu. Statistical guarantees for the EM algorithm: From population to sample-based analysis. The Annals of Statistics , 2017.

Maria-Florina Balcan and Avrim Blum. A discriminative model for semi-supervised learning. Journal of the ACM , 2010.

Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. A cookbook of self-supervised learning. arXiv:2304.12210 , 2023.

Shai Ben-David, Tyler Lu, and Dávid Pál. Does unlabeled data provably help? Worst-case analysis of the sample complexity of semi-supervised learning. In Annual Conference on Learning Theory (COLT) , 2008.

Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Annual Conference on Learning Theory (COLT) , 1998.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems 33 , 2020.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In 37th International Conference on Machine Learning , 2020.

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. In Advances in Neural Information Processing Systems 33 , 2020.

Thomas M. Cover and Joy A. Thomas. Elements of Information Theory . 2006.

Constantinos Daskalakis, Christos Tzamos, and Manolis Zampetakis. Ten steps of EM suffice for mixtures of two gaussians. In Annual Conference on Learning Theory (COLT) , 2017.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 2012.

Raaz Dwivedi, Nhat Ho, Koulik Khamaru, Michael I. Jordan, Martin J. Wainwright, and Bin Yu. Singularity, misspecification and the convergence rate of em. The Annals of Statistics , 2018.

Raaz Dwivedi, Nhat Ho, Koulik Khamaru, Martin J Wainwright, and Michael I Jordan. Theoretical guarantees for EM under misspecified gaussian mixture models. In Advances in Neural Information Processing Systems 31 , 2018.

Raaz Dwivedi, Nhat Ho, Koulik Khamaru, Martin Wainwright, Michael Jordan, and Bin Yu. Sharp analysis of expectation-maximization for weakly identifiable models. In International Conference on Artificial Intelligence and Statistics , 2020.

Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv:2112.10740 , 2021.

Spencer Frei, Difan Zou, Zixiang Chen, and Quanquan Gu. Self-training converts weak learners to strong learners in mixture models. In International Conference on Artificial Intelligence and Statistics , 2022.

Christophe Giraud. Introduction to high-dimensional statistics . CRC Press, 2021.

Christina Göpfert, Shai Ben-David, Olivier Bousquet, Sylvain Gelly, Ilya O. Tolstikhin, and Ruth Urner. When can unlabeled data improve the learning rate? In Annual Conference Computational Learning Theory , 2019.

Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self-supervised visual representation learning. In International Conference on Computer Vision , 2019.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems 33 , 2020.

Yann LeCun and Corinna Cortes. MNIST handwritten digit database, 2010. URL http: //yann.lecun.com/exdb/mnist/ .

Tianyang Li, Xinyang Yi, Constantine Carmanis, and Pradeep Ravikumar. Minimax Gaussian Classification & Clustering. In 20th International Conference on Artificial Intelligence and Statistics , 2017.

Thomas Lucas, Philippe Weinzaepfel, and Gregory Rogez. Barely-supervised learning: semisupervised learning with very few labeled images. In AAAI Conference on Artificial Intelligence , 2022.

Pascal Massart. Concentration inequalities and model selection: Ecole d'Eté de Probabilités de Saint-Flour XXXIII-2003 . Springer, 2007.

Alexander Mey and Marco Loog. Improvability through semi-supervised learning: A survey of theoretical results. arXiv:1908.09574 , 2020.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 2011.

John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers , 1999.

Joel Ratsaby and Santosh S. Venkatesh. Learning from a mixture of labeled and unlabeled examples with parametric side information. In Annual Conference on Learning Theory (COLT) , 1995.

Philippe Rigollet. Generalization error bounds in semi-supervised classification under the cluster assumption. Journal of Machine Learning Research , 2006.

Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In 29th International Conference on Machine Learning , 2012.

Aarti Singh, Robert D. Nowak, and Xiaojin Zhu. Unlabeled data: Now it helps, now it doesn't. In Advances in Neural Information Processing Systems 21 , 2008.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: Simplifying semisupervised learning with consistency and confidence. In Advances in Neural Information Processing Systems , 2020.

Erixhen Sula and Lizhong Zheng. On the semi-supervised expectation maximization. arXiv:2211.00537 , 2022.

Ilya O. Tolstikhin and David Lopez-Paz. Minimax lower bounds for realizable transductive classification. arXiv:1602.03027 , 2016.

Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. OpenML: networked science in machine learning. SIGKDD Explorations , 2013.

Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference on Learning Representations , 2021.

Yihong Wu and Harrison H. Zhou. Randomly initialized EM algorithm for two-component gaussian mixture achieves near optimality in O ( √ n ) iterations. Mathematical Statistics and Learning , 2021.

Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves ImageNet classification. June 2020.

David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting on Association for Computational Linguistics , 1995.

Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre-training and self-training. In Advances in Neural Information Processing Systems 33 , 2020.

A Proof of Proposition 1

In this section we provide the formal statement of Proposition 1 and its proof.

Proposition 2 (Fixing the sign of ˆ θ UL) . There exist universal constants c 0 , C 1 , C 2 , C 3 , C 4 > 0 such that for n u ≥ (160 /s ) 2 d , d ≥ 2 and s ∈ (0 , 1] , it holds that:

Proof. Recall that we consider the UL+ estimator ˆ θ UL+ = sign ( ˆ θ ⊤ SL ˆ θ UL ) ˆ θ UL and denote ˆ β := sign ( ˆ θ ⊤ SL ˆ θ UL ) . Now let β := sign( θ ∗⊤ ˆ θ UL ) = arg min ˜ β ∈{-1 , +1 } ∥ ˜ β ˆ θ UL -θ ∗ ∥ 2 .

Note that we can write the expected estimation error of ˆ θ UL+ as

̸

̸

Next, we recall a result by Wu and Zhou [39] established for this particular UL estimator, namely that E [ ∥ β ˆ θ UL -θ ∗ ∥ ] ≲ √ d s 2 n u . Moreover, the probability of incorrectly estimating the sign (permutation) can be written as

̸

̸

̸

where Z ∼ N (0 , 1) and S C ( ˆ θ UL , θ ∗ ) = | ˆ θ ⊤ UL θ ∗ | ∥ ˆ θ UL ∥∥ θ ∗ ∥ Therefore, for any A we have:

̸

where we used the Chernoff bound in the last step. Finally, setting A = c 0 √ d log( n u ) s 2 n u as a corollary of Proposition 6 in Azizyan et al. [2] for n u ≥ (160 /s ) 2 d we have P ( S C ( ˆ θ UL , θ ∗ ) ≤ 1 -A ) ≤ d n u . Therefore, for n u ≥ (160 /s ) 2 d , we have the following upper bound on estimating the sign incorrectly:

̸

̸

Combining this result with Equation (8) finishes the proof of the estimation error bound in the proposition, as we obtain that for some C 1 , C 2 > 0 , the following holds:

Similarly, we can decompose the expected excess risk as follows:

̸

̸

where in Equation (10) we upper bound the indicator function and the excess risk terms by 1. Finally, combining Equation (9) with the upper bound in Li et al. [24] for E [ E ( β ˆ θ UL ) ] , we obtain the desired result. More concretely, we get that for some C 3 , C 4 > 0 ,

B Proof of SSL Minimax Rate for Estimation Error in Theorem 1

In this section we provide the proofs for the lower and upper bounds on the estimation error presented in Theorem 1. The proof for the excess risk rates can be found in Appendix C.

B.1 Proof of estimation error lower bound

We first prove the estimation error lower bound in Theorem 1. As discussed in Section 2, consider the 2-GMM distributions from P ( s ) 2-GMM , with isotropic components and identical covariance matrices.

Consider an arbitrary set of predictors M = { θ i } i M =0 . . We can apply Fano's method [11] to obtain that the following holds:

̸

̸

̸

̸

̸

where D ( ·||· ) denotes the KL divergence. In Equation (11), we use the fact that the labeled and unlabeled samples are drawn i.i.d. from P X and P XY and in Equation (12) we upper bound the average with the maximum. The next step of the proof consists in choosing an appropriate packing { θ i } i M =1 and θ 0 on the sphere of radius s , i.e. 1 s θ i ∈ S d -1 , that optimizes the trade-off between the minimum and the maxima in Equation (12).

For the packing, we use the same construction that was employed by Wu and Zhou [39] for deriving adaptive bounds for unsupervised learning. This construction has the advantage that it also leads to a tight lower bound for the supervised setting. Let c 0 and C 0 be positive absolute constants and let ˜ M = { ψ 1 , ..., ψ M } be a c 0 -net on the unit sphere S d -2 such that we have | ˜ M| = M ≥ e C 0 d . For an absolute constant α ∈ [0 , 1] , we construct the following packing of the sphere of radius s in R d :

and define θ 0 = [ s, 0 , ..., 0] . Note that, by definition, ∥ θ i -θ j ∥ ≥ c 0 sα , for any distinct i, j ∈ [ M ] , which lower bounds the first term in (12). Furthermore, ∥ θ i -θ 0 ∥ ≤ √ 2 αs , for all i ∈ [ M ] .

In the next step, we upper bound the maxima in Equation (12). First, we write the KL divergence between two GMMs with identity covariance matrices:

Second, we can upper bound the KL divergence between marginal distributions, namely D ( P θ i X || P θ 0 X ) , using Lemma 27 in Wu and Zhou [39], which implies that:

Plugging Equations (13) and (14) into Equation (12) we obtain the following lower bound for the minimax error, which holds for any α ≤ 1 :

Minimizing over α yields the optimum value α = min { 1 , √ C 0 d -1 3 s 2 n l +3 C 1 s 4 n u } , where the minimum comes from how we have constructed the packing, which requires that α ≤ 1 . Using this value for α concludes the proof.

B.2 Proof of estimation error upper bound

We now prove the tightness of our lower bound by establishing the upper bound for the estimation error of the SSL Switching algorithm presented in Algorithm 2. We choose the following minimax optimal SL and UL+ estimators

where ( ˆ λ, ˆ v ) is the leading eigenpair of the sample covariance matrix ˆ Σ = 1 n u ∑ n u j =0 X j X T j and we use the notation ( x ) + := max(0 , x ) . It is known that this UL estimator is minimax optimal [39]. As unsupervised learning can only obtain classifiers up to a sign permutation, it is necessary to endow the UL+ algorithm with a means to discern between ˆ θ UL and -ˆ θ UL, as explained in Section 2.2. In

the case of the estimator introduced in Equation (16), the sign is selected so as to obtain an estimator that aligns better with the ˆ θ SL. The upper bound for the expected error incurred by the UL+ estimator is given in Proposition 2.

For the SL estimator ˆ θ SL, we apply standard results for Gaussian distributions to upper bound the estimation error that holds for any regime of n and d :

Using Equation (17) and Proposition 2 and switching between ˆ θ SL and ˆ θ UL+ according to the conditions in Algorithm 2 that pick the better performing of the two algorithms depending on the regime, we can show that there exist universal constants C, c 0 > 0 such that for 0 ≤ s ≤ 1 , d ≥ 2 and n u ≥ (160 /s ) 2 d , it holds that

where the expectation is over D l ∼ ( P θ ∗ XY ) n l and D u ∼ ( P θ ∗ X ) n u .

Matching lower and upper bound. When n l > O ( log( n u ) s 2 ) , the exponential term becomes negligible and the first additive component dominates in the last term in the right-hand side of Equation (18). Basic calculations then yield that the expected error of the switching algorithm is upper bounded by C ′ min { s, √ d n l + s 2 n u } for some constant C ′ , which concludes the proof of the theorem.

C Proof of SSL Minimax Rate for Excess Risk in Theorem 1

In this section, we prove the minimax lower bound on excess risk for an algorithm that uses both labeled and unlabeled data and a matching (up to logarithmic factors) upper bound.

C.1 Proof of excess risk lower bound

We first prove the excess error minimax lower bound in Theorem 1, namely we aim to show that there exists a constant C 0 > 0 such that for any s > 0 , n u , n l ≥ 0 and d ≥ 4 , we have

where the expectation is over D l ∼ ( P θ ∗ XY ) n l and D u ∼ ( P θ ∗ X ) n u . Our approach to proving this lower bound is again to apply Fano's method [19] using the excess risk as the evaluation method. The reduction from estimation to testing usually hinges on the triangle inequality in a metric space. Since the excess risk does not satisfy the metric axioms, we employ a technique introduced in Azizyan et al. [2] to derive an alternative sufficient condition for applying Fano's inequality.

̸

then

̸

̸

where the expectation is over D l ∼ ( P θ i XY ) n l and D u ∼ ( P θ i X ) n u .

In order to get a lower bound, we again pick θ i , . . . , θ M to be an appropriate packing, so that the condition in Equation (20) can be satisfied. For this purpose, we can simply use the construction from Li et al. [24], which was previously used to obtain tight bounds for supervised and unsupervised settings. Let p = ( d -1) / 6 . By Lemma 4.10 in Massart [26], there exists a set ˜ M = { ψ 1 , . . . , ψ M } , such that ∥ ψ i ∥ 0 = p , ψ i ∈ { 0 , 1 } d -1 , the Hamming distance δ ( ψ i , ψ j ) > p/ 2 for all 1 ≤ i < j ≤ M = | ˜ M| , and log M ≥ p 5 log d p ≥ d log(6) / 60 = c 1 d .

Define

for some absolute constant α . Note that since ∥ θ i ∥ = s and ∥ θ i -θ j ∥ 2 = α 2 δ ( ψ i , ψ j ) , we have

and

First, we show that the excess risk satisfies the condition in Equation (20). As in the proof of Theorem 1 in Li et al. [24], we have that for any θ ,

and thus, for all i and j = i , it holds that

̸

Then since the condition in Equation (20) is satisfied, we obtain

̸

̸

Next, we bound the KL divergences between the two joint distributions and between the two marginals that appear in Equation (25). For the joint distributions we have that:

where the inequality follows from Equation (22). Using Proposition 24 in Azizyan et al. [2], we bound the KL divergence between the two marginals

where the inequality follows from (23). Plugging (26) and (27) into (25) and setting

gives the desired result

C.2 Proof of excess risk upper bound

Next, we prove the upper bound on the excess risk of the SSL switching estimator ˆ θ SSL-S output by Algorithm 2 with the supervised and unsupervised estimators defined in Appendix B.2 to show the tightness of the excess risk lower bound in Theorem 1. In particular, we show that there exist universal constants C, c 0 > 0 such that for 0 ≤ s ≤ 1 , d ≥ 2 and for sufficiently large n u and n l ,

where the expectation is over D l ∼ ( P θ ∗ XY ) n l and D u ∼ ( P θ ∗ X ) n u .

The proof follows the same arguments as the proof presented in Appendix B.2 where we instead use excess risk upper bounds for SL and UL from Li et al. [24].

In addition, we also use a result that follows from Proposition 2 to choose the sign of the UL+ estimator.

Note that the upper bound on the excess risk of ˆ θ SSL-S is matching the lower bound in Equation (19), up to logarithmic factors. We conjecture that the logarithmic factors are an artifact of the analysis and can be removed. For instance, it may be possible to extend results in Ratsaby and Venkatesh [30] that bound the excess risk using the estimation error upper bound without incurring logarithmic factors. However, their results are not directly applicable here because they are only valid under the assumption that the estimation error is arbitrarily small and hence not enough for bounding the expectation.

D Theoretical motivation for the SSL Weighted Algorithm

In this section, we show theoretically that the SSL-W procedure introduced in Section 4 can achieve lower squared estimation error (up to sign permutation) compared to SSL-S. This result shows that it is possible to improve the error of the naïve SSL-S algorithm by making effective use of all the data that is available.

For the purpose of the theoretical analysis, we consider a slightly different SSL-W estimator compared to the one introduced in Section 4. First, recall that for the classification problem we consider, unsupervised learning produces a set of two feasible predictors { ˆ θ UL , -ˆ θ UL } and cannot discern between them without access to a (small) labeled dataset. We denote by ˆ θ ∗ UL the UL estimator with correct sign, namely ˆ θ ∗ UL := arg min θ ∈{ ˆ θ UL , -ˆ θ UL } E [ ∥ θ -θ ∗ ∥ 2 ] . Here, the hat symbol indicates that a finite sample is used to obtain the set of estimators { ˆ θ UL , -ˆ θ UL } , while the star symbol indicates that oracle information is employed to choose the sign of the estimator.

In what follows, we study theoretically the error of the SSL-W estimator constructed using ˆ θ ∗ UL , i.e. ˆ θ ∗ SSL-W ( t ) := t ˆ θ SL +(1 -t ) ˆ θ ∗ UL . Therefore, our result characterizes the error of the SSL-W estimator up to a sign permutation. To choose the correct sign, one needs only a small labeled dataset, similar in size to what is prescribed by Proposition 2. While this step is not captured by Proposition 3, SSL-S is unlikely to close the gap to SSL-W when provided with this small amount of additional labeled data. Moreover, for a fairer comparison we also give the switching estimator access to the UL estimator

We can now state Proposition 3, which shows that there exists an optimal weight for which the SSL-W predictor achieves lower estimation error than the SSL Switching predictor, ˆ θ ∗ SSL-S .

Proposition 3. Consider a distribution P θ ∗ XY ∈ P ( s ) 2-GMM and let d ≥ 2 , and n l , n u > 0 . Let ˆ θ ∗ SSL-W ( t ∗ ) be the SSL-W estimator introduced above. Then there exists a t ∗ ∈ (0 , 1) for which

0.0

0.2

0.4

0.6

0.8

1.0

SNR

s

10

10

10

8

10

6

10

4

10

2

Error gap

Err(SSL-S) - Err(SSL-W)

nl

=50,

nu

=10000

nl

=100,

nu

=10000

nl

=1000,

nu

=10000

Since the RHS of Equation (28) is always positive, ˆ θ ∗ SSL-W ( t ∗ ) always outperforms ˆ θ ∗ SSL-S as long as the conditions of Proposition 3 are satisfied. The magnitude of the error gap between SSL-S and SSL-W depends on the gap between SL and UL+ (see Figure 2). The maximum gap is reached for

D.1 Proof of Proposition 3

The first step in proving Proposition 3 is to express the estimation error of ˆ θ ∗ SSL-W ( t ∗ ) in terms of the estimation errors of ˆ θ SL and ˆ θ ∗ UL which is captured by Lemma 1.

Lemma 1. Let ˆ θ 1 and ˆ θ 2 be two statistically independent estimators of θ ∗ ∈ R d and let ˆ θ 1 be unbiased, i.e. E [ ˆ θ 1 ] = θ ∗ . Then, the expected squared error of the weighted estimator ˆ θ t ∗ =

We can apply Lemma 1, since ˆ θ SL is unbiased and ˆ θ SL and ˆ θ ∗ UL are trained on D l and D u respectively, and hence, are independent. The proof then follows from calculating the difference between the harmonic mean and the minimum of estimation errors of ˆ θ SL and ˆ θ UL+. Let x, y ∈ R + and w.l.o.g. assume x ≤ y . Then we have:

Applying this identity concludes the proof after choosing t ∗ = E [ ∥ ˆ θ 2 -θ ∗ ∥ 2 ] E [ ∥ ˆ θ 1 -θ ∗ ∥ 2 ] + E [ ∥ ˆ θ 2 -θ ∗ ∥ 2 ]

and for

Remark. Note that this lemma holds for arbitrary distributions and estimators as long as they are independent and one of them is unbiased. Therefore, future results that derive upper bounds for SL and UL+ for other distributional assumptions and estimators can seamlessly be plugged into Lemma 1. By the same argument, ˆ θ SSL-W obtained by other SL and UL+ estimators can also be expected to improve over the respective SL and UL+ estimators, given that one of them is unbiased.

D.2 Proof of Lemma 1

By definition of ˆ θ t ∗ , we have

where the last equality holds due to the independence of ˆ θ 1 and ˆ θ 2 and the unbiasedness of ˆ θ 1 .

E Simulation details

E.1 Methodology

We split each dataset in a test set, a validation set and a training set. The unlabeled set size is fixed to 5000 for the synthetic experiments and 4000 for the real-world datasets. The size of the labeled set n l is varied in each experiment. For each dataset, we draw a different labeled subset 20 times and report the average and the standard deviation of the error gap (or the error) over these runs. The (unlabeled) validation set and the test set have 1000 labeled samples each.

We use logistic regression from Scikit-Learn [28] as the supervised algorithm. We use the validation set to select the ridge penalty for SL. For the unsupervised algorithm, we use an implementation of Expectation-Maximization from the Scikit-Learn library. We also use the self-training algorithm from Scikit-Learn with a logistic regression estimator. The best confidence threshold for the pseudolabels is selected using the validation set. Moreover, the optimal weight for SSL-W is also chosen with the help of the validation set. Since we use an unlabeled validation set, we need to employ an appropriate criterion for hyperparameter selection. Therefore, we select models that lead to a large (average) margin measured on the unlabeled validation set. We give SSL-S the benefit of choosing the optimal switching point between SL and UL+ by using the test set. Even with this important advantage, SSL-W (and sometimes self-training) still manage to outperform SSL-S.

E.2 Details about the real-world datasets

Tabular data. We select tabular datasets from the OpenML repository [37] according to a number of criteria. We focus on high-dimensional data with 100 ≤ d < 1000 , where the two classes are not suffering from extreme class imbalance, i.e. the imbalance ratio between the majority and the minority class is at most 5 . Moreover, we only consider datasets that have substantially more samples than the number of features, i.e. n d > 10 . In the end, we are left with 5 datasets, that span a broad range of application domains, from ecology to chemistry and finance.

To assess the difficulty of the datasets, we train logistic regression on the entire data that is available, and report the training error. Since we train on substantially more samples than the number of

dimensions, the predictor that we obtain is a good estimate of the linear Bayes classifier for each dataset.

Furthermore, we measure the extent to which the data follows a GMM distribution with spherical components. We fit a spherical Gaussian to data coming from each class and use linear discriminant analysis (LDA) for prediction. We record the training error (of the best permutation). Intuitively, this is a score of how much our assumption about the connection between the marginal P X and the labeling function P ( Y | X ) is satisfied. For Figure 1 we take θ ∗ Bayes to be the linear Bayes classifier and θ ∗ UL the LDA classifier described above. 5 If the data is almost linearly separable (i.e. R pred ( θ ∗ Bayes ) ≤ 0 . 01 ), then we simply take the linear Bayes error as the compatibility.

Image data. In addition to the tabular data, we also consider a number of datasets that are subsets of the MNIST dataset [23]. More specifically, we create binary classification problems by selecting class pairs from MNIST. We choose 5 classification problems which vary in difficulty, as measured by the Bayes error, from easier (e.g. digit 0 vs digit 1) to more difficult (e.g. digit 5 vs digit 9). Table 3 presents the exact class pairs that we selected. To make the problem more amenable for linear classification, we consider as covariates the 20 principle components of the MNIST images.

F More experiments

In this section we present further experiments that indicate that the SSL Weighted algorithm (SSL-W) can indeed outperform the minimax optimal Switching algorithm (SSL-S). Figures 3 and 4 show that there exists a positive error gap between SSL-S and SSL-W for a broad range of n l values, for synthetic and real-world data, respectively. The extent of the error gap is determined by the n u n l ratio as well as the signal-to-noise ratio that is specific to each data distribution. In addition, Figure 4 also indicates that self-training can outperform SSL-W in some scenarios. It remains an exciting direction for future work to provide a tight analysis of self-training that can indicate when it outperforms both SL and UL+ simultaneously.

100

150

200

250

Labeled set size

nl

0.0025

0.0050

0.0075

0.0100

0.0125

0.0150

0.0175

0.0200

Error gap

Err(SSL-S) - Err(SSL-W)

10

20

30

40

Labeled set size

nl

0.03

0.02

0.01

0.00

0.01

0.02

Error gap

philippine

Err(SSL-S) - Err(SSL-W)

Err(SelfT) - Err(SSL-W)

10

20

30

40

Labeled set size

nl

0.00

0.02

0.04

0.06

Error gap

musk

Err(SSL-S) - Err(SSL-W)

Err(SelfT) - Err(SSL-W)

10

20

30

40

Labeled set size

nl

0.30

0.25

0.20

0.15

0.10

0.05

0.00

Error gap

mnist_5v9

Err(SSL-S) - Err(SSL-W)

Err(SelfT) - Err(SSL-W)

10

20

30

40

Labeled set size

nl

0.15

0.10

0.05

0.00

Error gap

mnist_3v8

Err(SSL-S) - Err(SSL-W)

Err(SelfT) - Err(SSL-W)

10

20

30

40

Labeled set size

nl

0.00

0.01

0.02

0.03

0.04

0.05

Error gap

a9a

Err(SSL-S) - Err(SSL-W)

Err(SelfT) - Err(SSL-W)

10

20

30

40

Labeled set size

nl

0.3

0.2

0.1

0.0

Error gap

mnist_5v6

Err(SSL-S) - Err(SSL-W)

Err(SelfT) - Err(SSL-W)

10

20

30

40

Labeled set size

nl

0.06

0.04

0.02

0.00

0.02

0.04

Error gap

vehicleNorm

Err(SSL-S) - Err(SSL-W)

Err(SelfT) - Err(SSL-W)

10

20

30

40

Labeled set size

nl

0.000

0.005

0.010

0.015

0.020

0.025

Error gap

madeline

Err(SSL-S) - Err(SSL-W)

Err(SelfT) - Err(SSL-W)

10

20

30

40

Labeled set size

nl

0.10

0.08

0.06

0.04

0.02

0.00

Error gap

mnist_1v7

Err(SSL-S) - Err(SSL-W)

Err(SelfT) - Err(SSL-W)

10

20

30

40

Labeled set size

nl

0.06

0.05

0.04

0.03

0.02

0.01

0.00

Error gap

mnist_0v1

Err(SSL-S) - Err(SSL-W)

Err(SelfT) - Err(SSL-W)

Peer reviews:
comments: SUMMARY:
this paper investigates whether improvements in the rates of sample complexities are possible for semi-supervised learning compared with supervised learning and unsupervised learning (up to a change in sign) for the specific case of a classification problem where each class is a symmetrical Gaussian. it is found that no improvement in the rate can be made, but it is possible to improve the constants.

STRENGTHS:
1 I think it is a very relevant question to investigate whether SSL can improve performance in theory over sup and unsup, and this can have a great impact

2 I like the deep theoretical analysis where minimax upper and lower bounds are derived 

3 the theory is complemented by some experiments to validate the theoretical findings

WEAKNESSES:
4 the work misses a very relevant recent survey concerning this question in general:
Mey, A., & Loog, M. (2019). Improvability through semi-supervised learning: A survey of theoretical results. arXiv preprint arXiv:1908.09574.

QUESTIONS:
5 for Algorithm 3, how does it make sense that $t$ is an input? should this not be learned also? similarly, I see the same problem with Theorem 6. This would imply the algorithm has knowledge of the distribution? Or is there a universal (distribution independent) t?

6 Similarly, in the experiments, these parameters seem to be tuned on a holdout labeled set. Why is that appropiate?

LIMITATIONS:
none
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
The paper presents a detailed analysis of semi-supervised learning (SSL) algorithms. Specifically, the authors establish lower bounds for 2-Gaussian mixture model distributions, revealing that no SSL algorithm can improve the sample complexities of optimal supervised or unsupervised learning. However, SSL can improve their error rates by a constant factor. The authors propose an algorithm that achieves lower error than both supervised and unsupervised learning, and conduct experiments on synthetic and real-world data.

STRENGTHS:
- The paper investigates the learning capacity of SSL algorithms and theoretically establishes a lower bound, which is an inherently interesting and meaningful endeavor.

- The paper is well-organized and easy to comprehend. The authors provide a clear overview of the problem and their proposed solution.

WEAKNESSES:
- Some details about the method and theory part should be further elaborated on, see Questions below.

QUESTIONS:
- The paper only considers Gaussian mixture distributions. Could more complex distributions be considered in future work to accommodate more real-world problems?

- The paper only considers the worst-case scenario. Is it possible to consider milder cases? For instance, could a data-dependent bound that relates to problem difficulty be derived?

- The statement on line 103 seems to be erroneous: when using the UL algorithm, why can't labeled data be considered as unlabeled data, so that UL can use $n_l + n_u$?

LIMITATIONS:
- As mentioned in the Questions section, the authors should provide further clarification on the method and theory parts.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper analyzes the effect of SSL methods to improve the error bound. The results suggest that SSL cannot improve over the statistical rates of both SL and UL at the same time, but it is possible to improve the errors by a constant factor. Simple experiments on synthetic and small-scale real-world data validate the theoretical findings.

STRENGTHS:
- This paper studies an interesting and important problem: whether semi-supervised learning can effectively use all the data.
- A detailed theoretical study is provided on the 2-GMM distribution, which sheds light on this problem. Modification:
- Though simple, the proposed SSL switching algorithm is interesting and insightful.

WEAKNESSES:
- The theoretical analysis is based on GMM, which introduces a strong assumption about the data distribution. This limits the scope of application of the theoretical analysis. 
- The proposed SSL switching algorithm introduces an extra hyperparameter s/t, and it seems that the algorithm is sensitive to the choice of the hyperparameter. Though it can be chosen by holdout labeled validation set, labeled data are relatively rare in the SSL setting. Is it possible to estimate the value of the hyperparameters?

QUESTIONS:
Please address the concerns in the weaknesses part.

LIMITATIONS:
The main limitation of this paper is that the analyses mainly focus on GMMs, which limits the scope of application.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The paper provides a tight lower bound for semi-supervised learning for the 2GMM model. It compares the minimax rate with supervised and unsupervised learning. The authors also provide supporting experiments on real-world and synthetic datasets."

STRENGTHS:
1. The paper is very well-written. The authors have effectively communicated their ideas, making it easy for readers to follow and understand the research.

2. The results presented in the paper go beyond existing prior work (when s > 0). 

3. The authors have conducted experiments to support their theoretical analysis. This empirical validation strengthens the credibility of their findings and enhances the practical relevance of the proposed approach.

WEAKNESSES:
The paper relies on strong assumptions regarding the 2GMM and linear model. This might limit the generalizability of the proposed method to other scenarios or data distributions.

QUESTIONS:
1. Line 65: There seems to be a typographical error in the notation "A(D_l, D_l)." Please clarify this notation.

2. Line 121: I assume that 'u' represents a function mapping the number of labeled data points to the corresponding number of unlabeled data points. Can you please confirm this understanding?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07