Prediction-Powered Semi-Supervised Learning with Online Power Tuning
Authors: Noa Shoham
Ron Dorfman
Shalev Shaer
Kfir Yehuda Levy
Yaniv Romano
Venue: NeurIPS 2025 poster ()
Keywords: Semi-Supervised Learning
Pseudo-Labeling
Online Learning
Prediction-Powered Inference
Debiasing
Primary area: general_machine_learning

Abstract: Prediction-Powered Inference (PPI) is a recently proposed statistical inference technique for parameter estimation that leverages pseudo-labels on both labeled and unlabeled data to construct an unbiased, low-variance estimator.
In this work, we extend its core idea to semi-supervised learning (SSL) for model training, introducing a novel unbiased gradient estimator. 
This extension addresses a key challenge in SSL: while unlabeled data can improve model performance, its benefit heavily depends on the quality of pseudo-labels. Inaccurate pseudo-labels can introduce bias, leading to suboptimal models.
To balance the contributions of labeled and pseudo-labeled data, we utilize an interpolation parameter and tune it on the fly, alongside the model parameters, using a one-dimensional online learning algorithm. 
We verify the practical advantage of our approach through experiments on both synthetic and real datasets, demonstrating improved performance over classic SSL baselines and PPI methods that tune the interpolation parameter offline.

Prediction-Powered Semi-Supervised Learning with Online Power Tuning
Noa Shoham' Ron Dorfman'
Shalev Shear' Kfir Y. Levy' Yaniv Romano 'Department of Electrical and Computer Engineering, Technion IIT DDeparteent of Computer Science, Technion IIT
Abstract
Prediction-Powenn Inference (PPI) recently proposed statistical inference technique for parameter estimation that leverages pseudo-labels on both labeled and unlabeled data to construct an unbiased, low-variance estimator. In this work, extend its core idea to semi- cuni-supereised learning (SSL) for model training, introducing a novel unbiased gradient estimator. This extension addresses a key challenge in SSL: while unlabeled data can improve model performance, its benefit heavily depends on the quality of pseudo-labels. Inaccurate pseudo-labels can introduce bias, leading to suboptimal models. To balance the contributions of labeled and pscudo-labeled data, we utilize an interpolation parameter and tune it on the fly, alongside the model parameters, using mee-iimeniionll online learning algorithm. We verify the practical advantage of our approach through experiments on both synthetic and real datasets, demonstrating improved performance over classic SSL baselines and PPI methods that tune the interpolation parameter offline.
Introduction
Semi-supervised learning (SSL) has garnered significant attention due to its success leveraging both labeled and unlabeled data to improve model performance [[33] A standard SSL approach augments pseudo-labeled data with trusted labeled data and then fits a new model-oo or fine-tunes a pre-trained one-by minimizing empirical risk (4/6 13.22]. A key limitation of this approach arises when the pseudo-labels are inaccurate. For instance, consider a minority subpopulation on which the pre-trained model performs poorly. In such cases, the resulting pseudo-labels introduce bias into the optimization objective. This issue is especially problematic when trusted labeled data is scarce and escudo-labeled data is abundant. Under these conditions, the SSL model becomes biased toward the erroncous pscudo-labela effectively ignoring the informative trusted labels from the minority group and failing to improve performance.
Recently, several approaches have been proposed to address the issue biased risk using pseudo- labels, with Dedfiction-Powered Inference (PPI) 223 26 emerging
notable example. PPI is designed to construct valid confidence intervals for parameters of interest by leveraging both labeled and pseudo-labeled data while correcting for potential biases in the latter, offering robustness in inference under minimal assumptions. Building on the principles introduced in PPI, subsequent works have extended these ideas to practical SSL settings
[14 27). incorporating corrections for
pseudo-label bias directly into the learning objective to improve robustness and generalization.
Nevertheless, existing PPI-based SSL methods Ieave two critical gaps: (i) Their analyses evaluate the effect of pseudo-labels only asymptotically- the population optimum, an idealized point never exactly reached during training and offer no guarantees about how pseudo-labels affect the actual convergence of the learning algorithm. (ii) Realizing the full benefit of pseudo-labele data hinges on an interpolation parameter that balances their quality against the variance of the labeled data. Prior work chooses this parameter offline, assuming knowledge of the pseudo-label quality and the labeled
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

data variance [14 24). Unfortunately, these quantities are unknown in practice, so an estimated fixed value may be markedly sub-optimal.
Contributions
We present Drediction-Poweree SSL (PP-SSL), novel framework for unbiased semi-supervised learning. We analyze PPI-inspired gradient estimates and demonstrate how their variance decreases as the pre-trained teacher model's aceuracy improves and the amount of unlabeled data increases. We further show how this reduced variance leads to faster convergence. This contrasts with prior work [14 27 which provides only asymptotic guarantees and finite-time convergence bounds.
Our work also guides us on how to choose the interpolation parameter E [0. 1] to optimally balance the effect of aseudo-labeles data against the variance of the labeled data. Unfortunately. the optimal choice of À relies on impractical prior knowledge of the labeled data variance and the teacher model's accuracy. To address this, we design an online learning approach that tunes À on the fly during training and achieves guarantees matching those of the optimal A. This result highlights the significance of moving from offline online analysis, enabling a rigorous account of training dynamics.
Numerical experiments on both real and synthetic data demonstrate the advantage of our online SSL framework regression and classification tasks. Our method outperforms both classic (biased) SSL and PPI-like training methods in scenarios where there a subgroup in the data on which the teacher model performs poorly.
2 Preliminaries and background
Consider n labeled data points, sampled i.idd. from some unknown distribution P Ix x Py|x- denoted by Dj = {(x',, ,,,,")) y') where (x', E X' x V. Also, assume we have N unlabeled data points sampled i.i.d. from the same Fx, denoted by Dunt = { In this work, we focus on the more common and interesting scenario where unlabeled data is abundant while labeled data is scarce and costly to obtain, i.e., > n.
Background and related work
Semi-supervised learning (SSL) is a machine learning paradigm that leverages both labeled and unlabeled data to improve model performance. In contrast, standard supervised learning relies solely on labeled data and typically resort to optimizing the empirical loss (a.k.a. empirical risk):
n
L. (w) = Σ
elw; x ,y,)),
(1)
where w E IRd denotes the trainable model parameters. A common SSL approach is pseudo- labeling [ o. 28 where a model f X' y is used to generate artificial labels for the unlabeled data to augment training. There are two main strategies for choosing this model. The first is self- training [7, where the same model being trained is also used to generate pseudo-labels that are continuously updated during training. In contrast, we adopt the teacher-student framework [18]22), which a fixed teacher model produces pseudo-lahels that remain constant while separate student model is trained. This setup offers greater stability and is better suited for theoretical analysis of pseudo-labes bias. The typical loss used in such settings
N Σ ( ((3'))
(ss)(w)= = Ln(w) + (l),,, where [k(w)) = N
However, pseudo-labeling may introduce confirmation bias. where erroneous pseudo-labels rein- force the model's mistakes [5]. Theoretical analyses of pseudo-labeling in SSL have primarily focused on offline settings. Previous works [5] study convergence and confirmation bias un- der confidence- based pseudo-labeling, but their analyses are limited to self-training with biased estimators. The authors of (22) analyze knowledge distillation (e.g., teacher-student) as a variance reduction mechanism but assume a self-distillation setup with identical teacher and student models
Software for reproducing the experiments is available hhttt::/ggthuubboo////ohhhaaaaap

---

and derive guarantees in an infeasible setting without ensuring unbiased risk estimation. In contrast, our theoretical contribution analyzes gradient variance in the online teacher-student setting and provides convergence guarantees under unbiased risk minimization.
In the context of statistical inference, Angelopoulos al. 23 proposed the Prediction-Powered Inference (PPI) framework, which mitigates the bias introduced by pseudo-labels through their use on both labeled and unlabeled data, employing the following loss:
(wr = Ln (u=) + E - where [((w)) - 1 Σ ((w; f(x)).
This formulation reduces the estimator's variance when f is accurate while remaining unbiased even when the teacher model f inaccurate. This is due to the fact that, in expectation, the pseudo-label loss on the unlabeled data equals the pseudo-label loss the labeled data, thereby canceling out the bias introduced by the teacher. Since when teacher predictions are inaccurate PPI can result in higher variance (larger confidence intervals), Angelopoulos et al. 24 proposed PPI++ introducing tuning parameter À E [0, 1] to interpolate the standard, supervised loss with the PPI loss:
LepI.. = (w) + X -L)(())) = (1 A) (w) + (w))
The above generalizes the PPI loss, interpolating between supervised learning (A 0) and vanilla PPI = 1). As the authors of [24] focus on parameter inference, they tune À offline to minimize the asymptotic variance of the parameter estimate. However, both PPI and PPI++ focus exclusively on offline confidence interval construction for parameter estimation (e.g., linear model weights) and do not address semi-superee superiupe training setting.
More recently. Houbly-Robuse Self-Traine Training [4] extended this line of work by applying PPI-like loss for training, using a pre-defined step function for the tunable weight À and providing asymptotic variance guarantees. Sifaou and Simeone (27 similarly build the PPI form, aiming improve robustness to teacher model errors, but still restrict their analysis and application to offline settings.
In this work, we aim to extend the PPI framework to semi-supervisee training by developing method that adaptively tunes the interpolation weight À during training using an online learning algorithm. Our approach dynamically balances the contributions of labeled and pseudo-labeled data minimize cumulative gradient variance, enabling more stable and effective learning than previous methods that rely fixed or offline-tuned weight.
3 Prediction-Poweree SuperSSue Learning
In this section, we describe our proposed Prediction-Poweree SSL (PP-SSL) framework, an unbiased semi-supervised learning algorithm. PP-SSL introduces a novel training paradigm that leverages prediction-powent gradient estimates. Formally, our objective is to minimize the expected loss function L: Rd >> IR, defined as,
min L(w) P(e,)) wERd
To this end, we consider iterative first-order optimization approach, where in each round t, we have access to a batch of n labeled samples, {(x] v ~ P, where P = Px x x a batch of N unlabeled samples, {
Px; and prediction model (i.e., teacher) f X' - y. Using the labeled data, the unlabeled data, and the teacher model, compute stochastic gradient estimate of C, which is used to update the current iterate This process is repeated over multiple rounds, ultimately producing an output Wout- Since we focus on non-convex optimization problems (c.g.. neural networks training). where global minimization is generally intractable, we aim to find an approximate stationary point for which ||V ((Wwu)))) I|2 is near-zero expectation.
Teacher's quality. The effectiveness of leveraging the teacher model depends on its ability to accurately predict the true labels. We quantify this through the teacher's prediction error, defined as
gt F Pa - f(x))²
(5)
As will show later, gd plays central role in the convergence guarantees of our approach.
3

---

Notations. Throughout, we use ||-I| to denote the L2 norm. We define Fe to be the filtration at iteration t, which encompasses all randomness up to that time. We use E and to denote expectation and variance, respectively, with E, and V, representing their conditional counterparts given Fr. Finally, we use standard big-O notation, where (-) hides numerical constants.
Prediction-Powered gradients. Our approach leverages stochastic gradient estimates inspired by the PPI++ loss (Equation Let us introduce the following notations (omitting iteration index t):
9"
1 n E 71
Vilwwz', 9²): gm n
n Vllw,r'
gN.
N Σ ff N
1
Here, 9" represents a standard mini-batch gradient estimator based on n labeled samples; g". replaces the true labels with predictions from the model f; and gN. is the gradient based on the unlabeled samples with pseudo-labes labels generated by f. We use these gradients to construct a prediction-power gradient, defined for some parameter À E [o, 1] follows:
ppp = g" ( gm
(6)
Since the labeled and unlabeled data are sampled from the same underlying feature distribution Px, it can be easily verified that dpp is an unbiased gradient estimator:
E
OpP
E [g"] + XE g" =[g" V[[[).
(7)
Notably, the convergence of (unbiased) gradient-based optimization methods, such as stochastic gradient descent (SGD), that utilize this estimator depends on its variance. Intuitively, when we set
À == 0, we resort to standard SGD (with mini-batch gradients); however, as we show, if the predictions of f are accurate, and with more unlabeled data, we can significantly reduce the gradient's 's variance, resulting in faster convergence for À > O.
We begin in Section 3.1 by analyzing the variance of the prediction-powent gradient and deriving the optimal tuning of A to minimize it. This tuning yields the tightest possible convergence bounds within our framework. However, the optimal value of \ depends on unknown quantities, making this approach infeasible in practice. Therefore, in Section 3.2, we treat À as a trainable parameter and co-optimize alongside the model parameters. Specifically, we employ a one-dimensional adaptive online learning algorithm to dynamically update À on the fly so as to minimize the cumulative variance of the gradients over time. We establish that this approach introduces only lower-order terms into the convergence rate.
Assumptions. For the purpose of formal analysis, we introduce the following assumptions. We assume the objective L is 3-smooth, i.c., L(u) < L(w) + V (u- w) + & ||u - w|l for all u, w E Rd. In addition, suppose that the constants o² and o² defined below are finite:
o² = sup Exyrr /V VL(w)]] <0,,
(8)
WERd
and
22 = sup Ex V (w; - vc( <0, weRd
(9)
where Ef (w: x. 3) = f(w; v. y) - fww; x (x) is the instance-dependent loss error function, which quantifies the difference between the losses when using the true label y and the teacher-provided pseudo-label f(x), and (w) = Ex.yr- [eí (:: x., y)] is the corresponding expected loss error function. Note that the assumption in Equation (8) is standard and establishes a bounded variance of the instance-dependen loss gradient. The assumption in Equation ② implies that the variance of the noisy loss error gradient is bounded as well. The next lemma establishes that of can be directly related to the teacher's 's expected prediction error El defined in Equation (5).
Lemma 3.1. Assume that the gradient V (((w; x, y) is Ly -ipsshitt in y, i.e., for any w E Rd x E X, and y1, 3/2 ), we have [[V xr yn) V V((u; v, ye)ll & Tyly - y2)- Then, of S T2. El.
In Appendix A we provide proof and show that the Lipschitzness condition applies to both the squared and the logistic losses, among others. In the next section, we show how the performance of our approach depends on o which is related to gl by the above incquality.

---

3.1 Prediction-Powers SSL with optimal tuning
Next, we analyze the variance the gradient estimates defined in Equation (6) and show that, with an optimal choice of X, the variance can be significantly lower than that of the standard gradient estimates based solely on labeled data.
Lemma 3.2. For every À E IR, the variance of gèp defined in Equation (6) is bounded as follows:
((ore) S (1 (1- + (ro² (1 r) o)))
where == n/N denotes the ratio of labeled to unlabeled samples.
From this result, with the proof in Appendix B.I we can optimally tune minimize the variance.
Corollary 3.3. The variance bound in Lemma 3.2m is minimized for
0²
x 1tr 02 0²
(10)
Substituting this value yields the following bound: J V(ppp) S 0². of402 tr
Let V*
2
(2)0"/- denote the optimal variance bound stated Corollary 3.3 When
1+r
the pseudo-labels are accurate (namely, o² < o²), V* approaches z² 4r
I+7 reflecting the benefit of incorporating high-quality unlabeled data, which results in substantial variance reduction (as < 1). On the other hand, when the pseudo-labels are highly unreliable (o? >> a²), the bound approaches 4o 4/²/n, effectively recovering the standard (labeled-onlyy gradient variance (up factor of 4, which is asymptotically equivalent); in this case, the optimal interpolation parameter X naturally down-weights the gradients from unlabeled data. This observation differs from conventional pseudo-labeling methods, which may reduce empirical variance but typically introduce bias when the pseudo-labels are incorrect. contrast, PP-SSL provides a principled mechanism for leveraging unlabeled data that improves statistical efficiency without compromising correctness. Additionally, in Appendix C we present a detailed comparison between the optimal parameter A derived in Corollary 3.3 and the one proposed in PPI++ (24), within the context of linear regression.
Convergence rate.
The unbiasedness property (Equation ). together with the variance bound (Corollary (.3), directly implies that using the prediction-powered gradient estimates within first- order methods such as SGD yields convergence rate of O O(VV*/T + + 1/"T) for smooth non-convex functions (29]. Here, V* can be substantially smaller than the variance of standard gradients estimates, as discussed above. Thus, the reduced variance directly translates to faster convergence.
3.2 Prediction-Poweett SSL with online tuning
As we have shown in the previous section, proper tuning of À can significantly reduce variance and, consequently, accelerate convergence. Unfortunately, the optimal choice X depends on o² and o2 (Equation (10)), which are typically unknown in practice. To address this, we propose an adaptive approach that dynamically adjusts À throughout the optimization process, while achieving performance similar X* Specifically, we employ the AdaGrad algorithm (30) to update online, alongside w, without requiring the knowledge of o² or o². We elaborate on this approach next.
Online learning and AdaGrad. Online learning is a decision-making framework in which an algorithm makes predictions and updates them based on information revealed over time. It is particularly well-suited for dynamically evolving environments. Online learning can be described as a sequential game, where in each round f, learner makes a decision u, E D, after which the environment reveals a (possibly arbitrary or even adversarially chosen) loss function h, and the learner incurs loss of hy (u+). The most common metric for evaluating the performance of an online learner is the regret, defined as the cumulative difference between the losses incurred by the learner's decisions and those of the best fixed decision in-hindsight,
R
min DED
hy(u).
5

---

Algorithm 1: PP-SSL with Online Tuning
1: Input: Prediction model X V. learning rate factor Y/o-
2: Initialize wi E RR >1 E (0,
3: for .,,., do
4: Receive n labeled samples {(z), v1) and N unlabeled samples {
5: Compute stochastic gradients:
n gt.
a
Σ Velwa vi), of N
N Vilue; H4
Velwi ((x()
71
6: Construct prediction-powweeed gradient estimate: gr t 9t + de df
-1/2
7: Update model parameter: W++1 & We nage", where 7) = no (Ei 11
8: Define (A) = llg. + Adl 21 and update d using AdaGrad (Equation 12
9: end for
One of the most powerful online learning methods is AdaGrad 30, which updates u, as follows 2
U++1 (1 7/ (ue( and Ttt == R& 2
V Vh.(w.))''
(11)
where RD := max ||u v|| is the diameter of DD and IT (v) == arg min ||u is the orthogonal projection onto D. AdaGrad enjoys the following guarantees; see, e.g.,
Lemma 3.4 (AdaGrad's Regret Bound). Let hı..... ho be any sequence of convex functions defined over a bounded, convex set with diameter Rd. Then, for any comparator u E D and any initial solution U1 E D, AdaGrad (1) ensures,
R²)x")= Σ (bu(u))-h(uu")) - S RD 2 Vho(w:)11
Importantly, AdaGrad only requires knowledge of the diameter, and otherwise automatically tunes its learning rate Ye- In addition, note that AdaGrad's regret bound depends on the sum of squared gradient norms, which will be crucial for the theoretical guarantees we derive.
Using AdaGrad dynamically tune A. Recall that our choice of X aims minimize the variance of the gradient estimates. Importantly, the performance of first-order methods like SGD as well as Adam 33 and AdaGrad is directly related to the cumulative variance of gradient estimates, i.e., to E. (g:),, where Sr is the gradient estimate used at time Since our prediction-powerte gradient estimates employ a parameter A, we can denote as g This suggests that we can cast the cumulative variance minimization problem
online learning problem with h; [0. 1] + R_ defined
(A) = 11 1|2 ||s + No..N - 9t )1
where we have used Equation (6) for g.¹ Importantly, is immediately apparent that h+(-) convex in X; this enables the use of AdaGrra dynamically tune X, in order to reduce the cumulative second moment. Since oi is always (conditionally) unbiased, then minimizing the cumulative second moment is equivalent to minimizing the cumulative variance.
Specifically, for our one-dimensional parameter [0, 1]. the AdaGrad update rule is given by:
1/2
Xe+1 clamp (Ae (h,(A.) 7
2
(12)
where clamp(:; a, b) == min(max :). denotes projection onto the interval (a, b).
This is in fact the scalar version of AdaGrad, also known AdaGrad-Nomn 3 32

---

Dataset. We design synthetic linear regression experiment with two groups to simulate scenario in which the Teacher model exhibits different performance across groups. Each input has m 10 features: the first nine are drawn from standard normal distribution, and the last is a binary indicator denoting group membership. For group A, labels are generated using linear model with additive zero-mean Gaussian noise. Group A comprises the 80% of samples with the smallest clean labels, ie.. the outputs of the true model before adding noise. The remaining samples form group B, where Overall dynamic approach. Our dynamic PPI-inspired training approach is depicted in Algo- rithm[]] At each round, we employ n labeled samples, N unlabeled samples, and the teacher model I to yield a prediction-powers
gradient 9; as in Equation (6). The latter is then used to update the model weights w1+11 as well as to craft a second moment estimate hy (A). Then, we update X using the AdaGrad update rule on he(.), as given Equation 12 Note that our update rule for we also employs an AdaGrad stepsize of the form 7j = 7o (oo== 11) (/13)-1/;
this is crucial for properly adapting to o² and o² without prior knowledge of these quantities. While we concretely employ AdaGrad for simultaneously tuning and X, one alternatively use any stochastic optimization method ge.g., SGD) for training w, and any online learning approach for tuning X. As Theorem 3.5 below shows, our specific choice of using AdaGrad for both parameters allows us to implicitly and optimally adapt to the unknown o² and o.. This provides a substantial advantage over alternative approaches that require hyperparameter tuning achieve comparable performance.
The next theorem establishes the convergence of PP-SSL (Algorithm D and shows that it performs similarly to employing the optimal (yet practically unknown) X We provide proof in Appendix B
Theorem 3.5. Suppose, in addition to the previous assumptions, that C(-) is M-bounded, ie., wax [[(w]] S M, and that the stochastic gradients are G-bounded, (VE(w;,,,)) G for all w E Rd, x E X, E V. Then, Algorithm with a learning rate parameter of Tlo = /2M/3 ensures the following convergence rate:
E
1 T
Σ
So
MBV T
MB
VMG
T
where V is the variance bound obtuined for the optimal choice of X (see Corollary 3.3
The convergence rate in Theorem 3.5 of three terms. The first terms correspond the rate we would obtain upon using X throughout all updates. However, since we do not know X in advance and instead learn it during training, incur additional term of order /MBG/T /T. Fortunately, this extra term decays substantially faster than the leading term, which decays as O( ((vV=// /T). Therefore, the overall convergence rate is of order (VV+/TT) matching the rate with the optimal choice of X*.
4 Experiments
In this section, we demonstrate the benefits of our proposed method through a series of experiments. We evaluate performance on both a synthetic dataset (Section 4.1) and three real-world datasets (Sections 4.2 4.3. 4.3.2). Our experiments encompass both regression and classification tasks. For regression tasks, we report Mean Absolute Error (MAE), Mean Squared Error (MSE), and the coefficient of determination (R²). ). For classification tasks, we report accuracy. Additional experiments analyzing the choice and dynamics of À are provided in Appendix H
Baseline methods. We compare our proposed PP-SSL method to 4 baseline training methods: (1) Teacher model f used for pseudo-labeling; (2) Only Labeled model that is trained only on limited labeled data (n samples) by minimizing I (3) SSL model trained both on labeled and pseudo-labeled data (n + N samples) by minimizing (2); and (4) PPI++ model trained both on labeled and pseudo-labelee data (n +N N samples) minimizing the debiased SSL loss (4) with A = App from [4 Importantly, our PP-SSL model is also trained both on the same limited labeled data and aseudo-labeled data, minimizing (4) but with À obtained by Algorithm I
4.1 Experiments on synthetic data with a linear model
Vanilla PPI is omitted due to its inferior performance compared to PPI++ with tunable X.
7

---

50
50
50
25
25
25
10
10
10
Teather
Oaly Labeled.
MSE
MSE
MSE
SSL
+P
PP-SSL (Ours)
0.1 3.0
7.0
0.1
3.0
7.0
0.1
3.0 5.0 7.0
Noise blas (1)
Noise bios (1)
Noise bias (w)
(a) All test set
(b) Group
(c) Group
Figure 1: MSE on synthetic data function of noise bias μ. Results shown for: (a) the full test set; (b) Group A, where the teacher model is accurate (oracle-like); and (c) Group B, which includes additive biased noise. Results evaluated on 100 independent experiments.
labels follow the same linear model but include an additional biased Gaussian noise term with mean fr, which controls the magnitude of the bias. This induces distributional shift between the two groups. The Teacher is a fixed linear regressor that serves as an oracle for group by using the true model weights for prediction.
Experimental setup. We set n 20, N 1,000, and Thest 1,000. We vary the noise bias E 1, to control the intensity of the distributional shift. All methods (except the Teacher) implemented by fitting linear regression model using ADAM optimizer with the same hyper- parrr-prraeeters and batch size. See Appendix D for more details on data generation process, training schemes, additional experimental setups, and results that include the R metric.
Main Results. Figure D presents group-wise MSE obtained by different models across varying group B's noise bias levels μr. As be seen, the average MSE across the groups (left panel) increases as l grows, but our PP-SSL shows hetter performance, demonstrating the advantage of using debiased SSL. risk with adaptive À when the pseudo-labels are biased. In group A (middle), the Teacher model provides oracle predictions by design, and thus the SSL method has strong performance. However, in group B (right) the SSL model performs even worse than the Only Labeled model as the Teacher performs poorly. This is in striking contrast with our PP-SSL that tends to achieve the best MSE for that group, especially in the high bias regimes. Crucially, PP-SSL outperforms PPI++ although the two use a debiased risk but with different strategies to tune X.
In the above experiment, all models have access to the group indicator during both training and testing. In Appendix [ we report similar results for the setting where the group indicator is not provided to the models.
Adaptive tuning of X. To evaluate the benefit adaptively tuning X, we conduct targeted exper- iment comparing PP-SSL with a PPI++-inspired baseline, that employs prediction-powered gradi- ents (see Eq. (6)) but uses a fixed À value through- training. In Figure 2 show the final MSE on the test set for PP-SSL (dashed line) and the baseline for different À values. Our adaptive method consistently achieves performance com- parable to the baseline with the best fixed À value. The optimal error depends on the teacher's quality (bias), with smaller values preferred for highly biased teachers, which aligns with our theory. These results demonstrate that our method auto- matically adjusts to the teacher's 's quality, achieving near-optimal performance.
5
4
MSE
2
0.2
0.4
0.6
1.0
μ =
7
PPI
PP-SSI.
Figure 2: Final test MSE for PP-SSL and PPI baseline with constant À for varying À values.
4.2 Experiments on real tabular data with a linear model
Dataset. We evaluate our method on the "California Housing" dataset 34 which contains numeric features and target variable representing house prices, across 20,640 samples. To demonstrate the

---

Teather
0.8
0.8
0.8
Labeled.
MSB
0.6
0.6
MSB
0.6
SSL
PP-SSL. (Ours)
0.0
1.0
0.0
0.0
1.0
Group protrain traction
protrain traction
Group
pretram traction
(a) All set
(b) Group
(c) Group
Figure 3: Results for the California Housing dataset: MSE of various methods as function of NB /Na the fraction of samples from each group used to train the Teacher model; e.g., NB/NA= 0.5 means twice many group A samples as group B. Results correspond to 100 data splits.
30
29
20
12
12
Teacher
Only Labeled
MAE
MAE
MAE
P2[++
(Ouns)
Group pretizain (Ng/NN)
Group pretieiin fraction (Mg/NAa
Group pretrain (Ni/MA)
(a) All test set
(b) Group
(c) Group
Figure 4: Results for age estimation: MAE function of NB/NA, across 5 data splits.
importance of the debiased SSL risk we split the data into two groups. Group A includes the (af, y²) with the 40% lowest target y"; otherwise the samples are assigned to group B.
Experimental setup and results. We split the data by setting
100, N 18,000, 7hva 1,000 for validation, and Yhest R 1,000 for testing. For all methods, we train a linear model. The Teacher model is trained NA z 50 disjoint labeled samples from group A, and varied number of samples from group B, denoted by Ng. in the range of to about 50. See Appendix [ for more details. Following Figure 3] we can see that
N increases and the Teacher's performance on group B improves, and all methods show reduced MSE values. PPI++ and PP-SSL have comparable MSE across the board, where both methods tend to outperform the other baselines as the teacher becomes less accurate due to the limited data from Group B. Additional experiments in Appendix further support these observations.
4.3 Experiments real visual data with deep neural network
4.3.1 UTKFaac data
We use the facial age estimation UTKFaae dataset 335 face images with age annotations ranging from to 116 years. We select the aligned and cropped version of the dataset, and resize the images to 224 x 224 pixels.
Dataset.
This dataset contains about 20,000
Experimental setup and results. We repeat similar setup to that from Section 4.2 Specifically, we split the data by setting n 2J 700, N s 16,500, Teaal RJ 2,000, and Rest R 2,000. For all methods, we train a ResNet50 model (36]. The teacher model trained on NA z 800 disjoint labeled samples from group A (ages over 30) and varied number of samples from group B (ages 0-29), with Ng ranging from 10 to about 800. As portrayed in Figure 4 the SSL method highly sensitive to the performance of the Teacher model: observe how SSL performs worse than the Only Labeled approach. By contrast, PPI++ and PP-SSL achieve lower MSE values, where our method has a noticeable advantage. This experiment shows that the debiasing approach can offer performance gains over the Only Labeled baseline, even when the Teacher model is relatively weak. In Appendix F we provide additional implementation details and results based on MSE and R² metrics.

---

80
190s
92
80
Pas
Accurasy
Accuraug
Aceuancy
00-
to
Teather
Labeled.
SSL
Sammatt
Spatter
Spoddde
84
Satarddo
Spatter
SpenkksNotee
Satandte
spetter
SpeokdeMotte
PP-SSL. (Ours)
Commuption type
Cocruption
Corrup
(a) All test set
(b) Group
(c) Group B
Figure 5: Accuracy CIFAR-10 data function of corruption type. Results are shown for: (a) the full test set; (b) Group A, clean images; and (c) Group B, corrupted images. Results evaluated on different seeds.
4.3.2 CIFAR-10 data
Dataset. evaluate our method on the CIFAR-10 dataset [37] and its corrupted variant, CIFAR- 10- C [38. CIFAR-10 contains 60,000 images of size 32 32 from 10 classes, split into 50,000 training images and 10,000 test images.
Experimental setup and results. To simulate a biased teacher, we split the dataset into two groups: Group consists of the dog, cat, and deer classes from CIFAR-10-C (corrupted samples), while Group A consists of the remaining clean CIFAR-10 classes. Corruptions such as Saturate, Spatter, and Speckle Noise from the CIFAR-10-C benchmark were applied to both training and test samples in Group B. We followed a standard SSL protocol, where features are extracted using a ResNet50 pretrained on ImageNet, and a linear classifier was trained on top. The teacher model a linear classifier trained small clean subset. See Appendix for additional details. As shown in Figure PP-SSL consistently outperforms baselines terms both total accuracy and Group (corrupted) accuracy, while maintaining comparable performance Group A (clean). The performance gap Group varies with the corruption type, reflecting the differing degrees of pseudo- -abell inaccuracy under different distortions. Additional corruption types and severity levels yielded similar trends, which we omit here for brevity.
Discussion
We introduce PP-SSL, semi-supervised online learning framework that incorporates a debiased empirical risk account for errors in pseudo-labels generated by teacher model. We demonstrate that our approach to dynamically update the interpolation parameter, which governs the reliance on pseudo-labeled data during training, consistently enhances performance compared baseline methods, especially when the teacher model exhibits degraded performance on specific subgroup.
A key limitation of method, shared by many SSL methods, is the assumption that labeled and unlabeled data are drawn from the same underlying distribution. In practice, unlabeled examples may come from different domains or be produced by generative models, introducing distributional shifts and bias. Extending our framework to handle such shifts is natural direction for future work. For instance, importance weighting can be introduced correct for covariate shift, either by reweighting the gradient estimator used for X adaptation or by modifying the loss itself. More complex shifts, such those induced by synthetic or generative data, would likely require adaptations both the pseudo-labeling mechanism and the gradient- based weighting strategy.
Our theoretical analysis provides the first gradient variance bounds for PPI-based methods, yet assumes a fixed teacher for tractability. Extending these guarantees to self self-training scenarios with evolving teachers remains challenging due to non-stationarity and requires regret analysis. While not currently covered by our theory, PP-SSL remains practically applicable to self-training, where the model iteratively generates laeudo-labels the PPI loss remains unbiased.
nuanced dynamic
Overall, our work offers a principled step toward more adaptive and reliable SSL. with both theoretical and practical implications. Similar to works in this field, this research has potential social implications.
10

---

Acknowledgments and Disclosure of Funding
NS, SS, and YR supported by the European Union (ERC, SafetyBounds, 101 101163414). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency, Neither the European Union nor the granting authority can be held responsible for them. NS, SS, and YR were also partially supported by the Israel Science Foundation (ISF grant 729/21). YR acknowledges additional support from the Career Advancement Fellowship at the Technion. KYL and RD were partially supported by Israel PBC-VATAT by the Technion Artificial Intelligence Hub (Tech.AI), and by the Israel Science Foundation (grant No. 3109/24).
References
[1] Xiaojin Zhu and Andrew Goldberg. Introduction to semi-supervised learning. Morgan & Claypool Publishers, 2009.
[2] Xiaojin Zhu. Semi-supervised learning literature survey. Technical Report 1530, Computer Sciences, University of Aisconsin-Madison 2005.
[3] Olivier Chapelle. Berhhard Scholkopf, and Alexander Zien. Semi-supervised learning. IEEE Transactions on Neural Networks, 2009.
[4] Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, 2013.
[5] Eric Arazo, Diego Ortego, Paul Albert, Noel E O' O'Connor, and Kevin McGuinness. Pseudo- labeling and confirmation bias in deep semi-supervised learning. In 2020 International joint conference neural networks (IJCN)), pages 1-8. IEEE, 2020.
[6] Isaac Triguero, Salvador García, and Francisco Herrera. Self-labeled techniques for semi- supervised learning: taxonomy, software and empirical study. Knowledge and Information systems, 2015.
[7] Anru Zhang, Lawrence D Brown, and T Tony Cai. Semi-supervised inference: General theory and estimation means. 2019.
[8] Jesper E Van Engelen and Holger H Hoos. A survey semi-supervised learning. Machine learning, 2020.
[9] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[10] David Azriel, Lawrence Brown, Michael Sklar, Richard Berk, Andreas Buja, and Linda Zhao. Semi-supervised linear regression. Journal of the American Statistical Association, 2022.
[11] Zixing Song, Xiangli Yang. Zenglin Xu, and Irwin King. Graph-based supersupe learning: comprehensive review. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[12] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in neural information processing systems, 2019.
[13] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty- aware pseudo-label selection framework for semi semi-supervised learning. arXiv preprini 1xXXi:2000.06229 2021.
[14] Banghua Zhu, Mingyu Ding, Philip Jacobson, Ming Wu, Wei Zhan, Michael Jordan, and Jiantao Jiao. Doubly-Robel Self- Training. Advances in Neural Information Processing Systems, 2024.
[15] Haiyun He, Gholamali Aminian, Yuheng Bu, Miguel Rodrigues, and Vincent YF Tan. How does pseudo-labeling affect the generalization error of the semi-supervised gibbs algorithm? In International Conference on Artificial Intelligence and Statistics, pages 8494-8520. PMLR, 2023.
11

---

[16] Spencer Frei, Difan Zou, Zixiang Chen, and Quanquan Gu. Self-trainigg converts weak learners strong learners mixture models. In International Conference on Artificial Intelligence and Statistics, pages 8003-8021. PMLR, 2022.
[17] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self- with deep networks on unlabeled data. arXiv preprint arXiv:2010.03622, 2020.
[18] Vasilis Kontonis, Fotis Ilopoulos, Khoa Trinh, Cenk Baykal, Gaurav Menghani, and Erik Vee. Slam: Student-label mixing for distillation with unlabeled examples. Advances in Neural Information Processing Systems, 2023.
[19] Hunter Lang, David Sontag, and Aravindan Vijayaraghavan. Theoretical analysis of weak-to- strong generalization. Advances in neural information processing systems, 2024.
[20] M Emrullah Iidiz, Halil Alperen Gozeten, Ege Onur Taga, Marco Mondelli, and Samet Oymak. High-dimensional analysis of knowledge distillation: Weak-to-strong generalization and scaling laws. arXiv preprint arXiv:2410.1880 18837, 2024.
[21] Vaishnavh Nagarajan, Aditya K Menon, Srinadh Bhojanapalli, Hossein Mobahi, and Sanjiv Kumar. On student-teacher deviations in distillation: does it pay to disobey? Advances in Neural Information Processing Systems, 2023.
[22] Mher Safaryan, Alexandra Peste, and Dan Alistarh. Knowledge distillation performs partial variance reduction. Advances in Neural Information Processing Systems, 2023.
[23] Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael Jordan, and Tijana Zrnic. Prediction-powered inference. Science, 2023.
[24] Anastasios N Angelopoulos, John C Duchi, and Tijana Zmnic PPI++: Efficient prediction- powered inference. arXiv preprint arX1v:2311.01453, 2023.
[25] Tijana Zrnic and Emmanuel Candès. pross-rredicioonppoweedd inference. Proceedings of the National Academy of Sciences, 2024.
[26] Wenlong Ji, Lihua Lei, and Tijana Zmic. Predictions as surrogates: Revisiting surrogate outcomes in the age of ai. arXiv preprint arXiv:2501.09731, 2025.
[27] Houssem Sifaou and Osvaldo Simeone. Semi-supervised learning via cross ceeess-reeicioonpppee powwered inference for wireless systems. IEEE Transactions on Machine Learning in Communications and Networking, 2024.
[28] Youngtaek Oh, Dong-Jin Kim, and In So Kweon. Daso: Distribution-awarc semantics-oriented pseudo-label for imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9786-9796, 2022.
[29] Saced Ghadimi and Guanghui Lan. Stochastic first-and zeroth- order methods for nonconvex stochastic programming. SIAM journal on optimization, 2013.
[30] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 2011.
[31] Kfir Levy. Online Offline Conversions, Universality and Adaptive Minibatch Sizes. In Advances in Neural Information Processing Systems, 2017.
[32] Rachel Ward, Xiaoxia Wu, and Leon Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes. Journal of Machine Learning Research, 2020.
[33] Diederik Kingma. Adam: method for stochastic optimization. arXiv preprint
arXiv: 1412.6980, 2014.
[34] California Housing Data Set. housing. html Accessed: January, 2025.
12

---

[35] Zhifei Zhang, Yang Song. and Hairong Qi. Age progression/regrssion by conditional adver- sarial autoencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.
[36] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[37] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[38] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint vrXiv:1903.12266 2019.
[39] Ron Dorfman, Naseem Amin Yehya, and Kfrr Yehuda Levy. Dynamic byzantine-rooust learning: Adapting to switching byzantine workers. In Forty-first International Conference on Machine Learning, 2024.
[40] Ebimobowei Sampson. Facial age estimation in pytorch. https://github. com/ cotttbbuu..co//biiss/// Ebimsv/ Facial_ Bacall__Ae_eeestmmmeeeeeeeeeesee astimation_PyTorchh 2021. Accessed: 2025-05-19.
13

---

A Relating the teacher's prediction error to the gradient variance
In this section, we prove Lemma 3.1 which establishes connection between the teacher model's prediction error & - En P(U (x)))² and the variance bound of the error loss gradient of. defined in Equation (9 This result holds for any loss function whose gradient Vf(w; x, y) is
Ly Li-Lscchiitz y. We then show that both the squared loss and the logistic loss satisfy this condition.
For case of reference, we state Lemma 3.1 from the main manuscript.
Lemma 3.1 Assume that the gradient Vl(w;ar y) is Ly -ypssshciitz in y, i.e., for any o E Rd E X, and y 1/2 E y. we have [[V[(w;; 21, V1) Vf(w;; 3/)ll < Ly w V2l. Then, oe - L Ef. Proof of Lemma 3.1 Recall that & (w; v) = (ww,,y) - f, [(x)). Since Vl(w; x, y) is
Ly-Lipschitz in we have:
E Vi! (w; w) (V((u; [())) EEL (v- (())) 12
Since the variance is always upper bounded by the second moment, we get:
E V (w; v) (w)]l = E V
s L
Finally, as this bound holds for any R., it also holds for the supremum, and thus of S 13 D Next, we consider the squared and logistic losses and show that their gradients are Lipschitz with respect to the y input. Let &ie X - y denote model parameterized by weights and assume that vw. < for all w Rd and E X. For instance, for a linear model & (x) = æ, this condition holds when the imput features are bounded, i.c., ||x||| R.
Rd
Squared loss. The squared L2 loss is defined LLz (,;,,)) (dw((x) v)² with gradient Vlr, (w; x, = (ow. (2) - y) Am Therefore, for any U1. V/2 E3, we have:
(Vl vn) V (w x, 22)l (1 v1) V Vw (x)l 1y1 V2 Ve)
S R 11 - 3/2l-
Therefore, the squared loss gradient is Lipschitz with coefficient R.
Logistic loss.
The logistic loss is defined as log (w:x,y)
yog (o(o((z))) (1
log (1 a (x))), where (u)) 1/(1 + e=*) ) is the sigmoid function. Its gradient is given by V Vlogg (w: x, y) = (a( (x)) - v) V V (x). Thus, for any y1, 322 V, we have:
Ving (w: xr, v1) V flog. (w:; v. M2)l (1 vi) Va Pn (z)]| |n/ y2l || dh (x)]
S 1//1
V2l
Similarly to the squared loss, the logistic loss gradient is also Lipschitz with coefficient R. For both cases, Lemma 3.1 implies that of bounded by g2
B Theoretical analysis of prediction-powered gradients and PP-SSL
In this section, begin by establishing the variance bound of the prediction-powertt gradient in Appendix B.I Then, in Appendix B.2 we prove the convergence result for our PP-SSL method, as stated in Theorem 3.5
B.1 Proof of Lemma 3.2
Proof. Let us define n. = t M Vi(w; 5t w where ÿ is the true (unknown) label āf i.e.,
ÿ' ~ Prix((1+*) Thus, we can decompose gpp == g" + N...J n. as follows:
GM) - SPp = N. NJ
(VC'(w) VC(w)) + (9" n (V[(w)- (CC(w)-VC(( VV((w))
14

---

where ('(w) = Ex-Px x [e(w; 2, f(x))]. Note that gèp (1 A)g" + Appp- Considering the variance of gpp and substituting this expression, we obtain:
V(gpp) = Ellgèp V
= E (1(1 - X) (0" - VL(w)) + A (ohp Ve(w))]l
- 4(1 - 1) I1g" -
110 v
4x (G@ - - (w)) 1 + 4x22 (10" gm (w)
4(1 - ()")((") + 4X (V(g*) + V(g - g/. -
where the first inequality follows from IE:=1 u:| - n Ei= | || and we also used L! (w)
((w) - C (w). Dividing by 4 and plugging the variance bounds from Equations and (accounting for mini-batch gradients) yields:
V(rp))
4 - (1 ()- o²
2 N
22
(1 -1- X) 02 + X2 (ro² 2 + r))
N
where 1/N r/n. Finally, multiplying by concludes the proof.
B.2 Proof of Theorem 3.5
Proof. Recall that we use the AdaGrad stepsize to update wy. i.c., for some parameter 17o > 0, we
update wa follows:
7/0
u++1 = Wi - nege" where 1)
(AdaGrad-Norm)
E
Therefore, we can employ the following lemma (see, e.g., Lemma G.3 in 39
Lemma B.1. For 9-smooth and M-bounded function (i.e., wax [f(w)] M), the iterates
W1, wr defined by the above AdaGirad-Normy update rule satisfy:
[V(ww] <
2M
+ ToB
Σ
T + Σ (VL(ww) - gt VL(w:)
T
(((())
1/0
Now, recalling that g²² is an unbiased estimator of V V(:,),, in the sense that 14,1 = VL(w:), taking expectation of the bound above yields:
2M
[(:)
moB
E.
10
170
2M
no.3
E.
Σ (xee))
170
2M
T
n/.3
E.
R$
Σ he()*),
(13)
7o
where the equality follows from the definition he (A) = lo and the last incquality uses the definition of regret, R4 = = (>.) Σt= he()))
Since we update D using AdaGrad well (see Equation (12)), we can bound the regret R4 (X*). as formalized in the following lemma (proved in Appendix B.3}.
Lemma B.2. Assume the stochastic gradients are G-bounded; that is, for any E Rd and (z, v) E X x Y, we have [V x. y) S G. Then, the Ad-AGrad algorithm used Equation 12
yields
the following regret bound:
RA (A*) & 128G + 8GG
2
h h₂*))
1=1
15

---

Plugging this regret bound into Equation 13 gives:
T B[ &
2M + 7oB
T he(A*) +
T Σ he(A)).
E
128G + 8G
(14)
1=1
Observe that the last two terms under the square root take the form 8GV2A + A. where A Σ h,()),, which we can bound using the helper Lemma B.3 below, which we prove Ap- pendix B.4
Lemma B.3. For any A, C > 0, it holds that CVA + 2 +2A
Hence, from Equation 14 it follows that:
T
2M
+ n0/3
E,
128G + 256G2 + Σ
(x()+)
970
1=1
2MM
2M
+ 1o.3
SVGG+
+ +m3
2 hx(A*)
1/0
2.M
2M
+
1033
SVGG+
mB
Σ Ehg (A*).
(15)
Y/o
where the second inequality holds because va+b inequality follows from Jensen's inequality applied to the concave function H = Vz. Next, focusing on (A*), we apply the law of total expectation obtain:
Va + Vb for any a, b 2 0, and the last
Eh, Bllge" ||2 EE,- 11o;" 1|2
E E 1E |1
VVV )] [(
where (t) follows from the definition of conditional variance and (#) is due to the (conditional) unbiasedness of o²" Note that the conditional variance with the optimal X", ( ), bounded by V Ako? /n according Corollary 3.3 where (1402%02) Therefore,
1tr
Eh₂(2*) S V + E|IV
Substituting this back into Equation 15
we get:
2M
2M
E
V
((w))l
nnB
8V6G+
n0,3
2V >
EE V Lww))l
170
170
22M
22M
noB
8V6G +
+
n0,3
V2V-T +
2
E V L L((we) (wx)
170
170
(16)
We can now solve this inequality for ΣT in Appendix B.5
E| VL(x)) using the following lemma, which we prove
Lemma B.4. Let A,B.C B C, then A < 4B² 2C.
Applying Lemma
B.4 the inequality in Equation 16 we get:
T
2M + mo3 (8VBG+ v2V-T) ((V+)+ +
2M
2
E ([ww))1
5
+ n0
100
7/0
16

---

Setting 170 /2M/3 gives the following bound:
E [[(()) 5 64MB.
Finally, dividing by T concludes the proof.
B.3 Proof of Lemma B.2
Proof. By the definition of h, (A) = 1| l1.. + 1l2 the gradient (derivative) of h4 is given
by Vh+(A) = 2(4/ g²). Applying the Cauchy-Schward Schwarz inequality, we can bound this gradient norm as follows:
[A,)(),)) < 4 d' 19 I|2 = (1²,, & 1,G*h,(A.)
(17)
where the last inequality follows from the bounded gradients assumption and the triangle inequality:
||
Vilug f(
N
Ve(we Ty. f(i)))
Vi(we x/, f(x))) 2G
Substituting Equation 117 into the AdaGrad regret bound from Lemma 3.4 and noting that for our case D = [0, 1] with diameter Ro = 1, we obtain:
R2())s 2 Σ
Who(WW)1
32G Σ he(Ae)
1 22²(1*+ +2 Σ hx()*)
4G / (x)( + 4G, 2 Σ he(d*).
where the last inequality is due to Va +b Va + Vb. Solving this inequality for (4())) (using
Lemma B.4 yields the result:
R4(A2) S 128 8G, 2 2 ha(A')
B.4 Proof of Lemma BK
Proof. We consider two cases:
Case 1: 4 2 CVA. In this case, CVA+ A 5
Case 2: A < CVA.
Dividing both sides by VA, we obtain VĀ < C. Hence:
CVA+ A < 20 202
Combining both cases, have CvA+ A max 22A, 2C < 2C + 2A, which concludes the proof.
B.5 Proof of Lemma B.4
Proof. Consider two cases. First, if BVA 2 C, then A < 2BVA, which implies A < 4B2, Second,
if B 5 C, then A < 2C. In both cases, we conclude that A < max {4B2,20 _,2 < 4 O
17

---

C A comparative analysis of optimal À tuning for PPI++ and PP-SSL
Next, we theoretically compare the optimal weighting parameter in our method (PP-SSL) and in Prediction-Powwe Inference (PPI++). This parameter plays a crucial role in balancing the influence of labeled and unlabeled data in both approaches, directly affecting the variance. While both methods aim to reduce variance in parameter estimation, they derive the optimal À from different optimization objectives. We analyze the derivations of the original PPI++ expression, and our variance bound minimization from Corollary 3.3 For clarity and concrete interpretation, we focus on linear regression with mean squared error loss throughout our analysis.
C.1 +PI++ expression
In this section, we analyze the optimal expression defined 24 Proposition 2]:
Xpp Tr H (Cov(Vem', VeLy ) Cov V int Vln H)
(V
2(1 + H Cov(VE, )H
where Hw is the Hessian of loss (w-) with the optimal parameter w².
For linear regression with MSE loss, we consider model ğ x: Tw with loss f(w, (x.y)) ( TTuu w)2 and prediction loss e (w, f(x))) ff((x) xTww w) This yields the following gradients:
Vew = x(x T w v), Vel = x(x Tw w - f(x)),
with Hessian H., xxx which at w" becomes Hw- = B[xx"' = Eg.
We define the true residual y u and pseudo-label residual Ef f(x) x w", allowing us to express the gradients as Vl.. -re and v = rr Since we use the true regression coefficient vector w", it is reasonable to assume independence between features x and residuals e, E, Under this simplified assumption, the covariance terms become Cov ve E[xx" EEyl = Eg Cov(e, ey) and Cov(Ve, w. = F[xx" o31 == Ex Var(ey).
We now substitute into the optimal À formula:
x Tr(Ez 1 (Ez Cov(e, + E., (Cov(e,e;)))
2(1 + Tr Eg Var(e,)
Tr(E, E. E
coN is symmerric (1 + 1 Var(E) 1) This results in the simplified optimal value for Appr.. the linear case:
Cov(E,Es) Var(E,)
AppL--
(18)
1
This demonstrates that for the general linear case with MSE loss, the optimal À depends on the covariance between true e and pseudo-labeled e residuals, the variance of the pseudo-labeled residuals Var(e)) and the labeled-unlabeled data ratio
C.2 PP-SSL expression
From Corollary
3.3
the optimal X that minimizes the variance bound is:
1 o² 1
x² + 0
where n/N is the labeled-unlabeled ratio and o² and o. are constants from Assumption g
For linear regression with MSE loss, by assuming independence between features and residuals we have:
2 sup Ey /aaarP///uuuee
VV(w; VL(w)] Ex. ||0|| Var(e)Tr(EE.),
wekd
18

---

and
oe -E Bay-r ply x x- -| Var(e,) Tr(Ex))
Substituting the two expressions into APp.SLL yields
1
Var(s)
App.SSI. I+r
Var(e) Var(ey)
The above expression shows that the optimal ssp.ssI. depends on the variances of true and pseudo- labeled residuals, and the ratio between the number of labeled and unlabeled points.
While both PPI++ and PP-SSL down-weight the contribution of the pseudo-labeles labeled data based on the accuracy of the teacher, they differ in key ways. When pseudo-labels are perfect (i.e., E E). both PPI++ and PP-SSL assign high weights to the pseudo-label term: App-SSL- 2(1+)) and Xpp T4y with + placing more trust in the pseudo-labels. When pseudo-labell laudll are noisy uncorrelated with the true residuals, both methods reduce the contribution the pseudo-labeled data by assigning smaller X values.
The analysis of PP-SSL under the simple linear regression model with MSE loss can conducted not only through minimizing bound on the gradient variance, but also by directly minimizing the gradient variance itself. In fact, this direct optimization yields the same analytical solution for X as in PPI++ see the derivation below. However, the two methods differ in practice: PPI++ computes gradients and Hessians at the optimal point using all data offline, while PP-SSL adapts À dynamically during training by minimizing the variance of the gradient estimate on-the-fly. Consequently, the actual values of used in training often differ between the two approaches, despite the similar underlying theory. This is reflected in performance differences, shown in Figure S3
For completeness, we conclude this discussion by presenting the analytical solution for our X that minimizes the prediction-powert -poeeeed gradient variance under linear model. Recall gflll (w) from Equation its Cov(o (w) can be reformulated as
(ov(gX"((w)) (V) X(Cov(VL, V + Cov(Vé, V()) + A2 + Cov(Vet)
For linear regression with MSE loss, assuming independence between features and residuals, as detailed before, the covariance matrices share common factor Σx, resulting in
Σn (Var(e) X(2Cov(e, ey)) + A² + 1-11 1)
Cov(gX (w)) (()))))))
By taking the gradient with respect to X, equating it to zero, and re-arranging the terms we get: Cov(s,ey)
Aincar ++7 Var(E,)
Observe the above Xinear coincides with the one from Equation 18
D Synthetic regression experiments with two-groups: supplementary details
In this section, provide additional experimental details related the experiments described in Section[4.] This includes the data generation process and hyperparameter configuration (Section D.I). extended results for the same experimental setup (Section D.2), and new experiment where the group indicator is assumed to unknown (Section D.3) Lastly, Section D.4 presents a comparison between models trained with and without access to the group indicator.
D.1 Experimental details
Metrics We performed tests of solving regression problems, therefore metrics are expected for those experiments:
MSE (Mean Squared Error), defined as 1 Eney (f(x*) - - where f(x:) is regression model. (lower is better)
E (f(2"))=' E. (5-W')=
R²a defined as
where z is the empirical mean of y values in the test
set. (higher is better)
19

---

Setup and environment The experiments were conducted on system running Ubuntu 20.04.6 LTS, each experimnt (single seed) with 2 CPU cores of Intel(R) Xeon(R) Gold CPUs 2.40 GHz, 32 GB of RAM. The software environment used Python 3.1 1.3 and PyTorch 2.5.1.
Data generation process We design a controlled synthetic regression experiment to evaluate the performance of our method in setting with group-wise label shift. The dataset consists of samples r' R where each row is independently drawn from a standard normal distribution A (0, I. The weight vector E _ is also sampled from (0, mm). The target labels are computed as y' w x' + where b E _ is a bias term drawn from In) To create two distinct groups, define split threshold r 0.2, which corresponds to the 7-th percentile of the linear projection w x (excluding the bias term). For samples in the second group (i.e., those above the threshold), we modify the target by adding noise: y = T x + + c, where c ~ N(,,, 1). By varying μ., we control the shift in label distribution between groups and evaluate the robustness of each method. In the experiments from Section 4.1 which assume access to the group indicator, we augment the generated features with an additional feature representing the group identity. This results in m + 1 input features. The ground truth labels are generated using a weight vector wo E Im++1, where the last component-correst corresponding to the group feature- canture-istn zero. This setup allows us to train model with an expanded input space, enabling it to capture potential group-dependent variations in the data.
In).
Model details We train linear model in PyTorch with weights of size (or m + when the group indicator feature is included) and a bias term, using the ADAM optimizer.
Hyperparameters Table ST summarizes the hyperparameters used in the experiments described in Sections D.3 D.2 and
Table S1: Experimental settings for synthetic regression and two-groups regression tasks
Synthetic regression
Synthetic regression with group indicator
Parameter
without group indicator
Total number of samples Number of features (m)
2000
2000
10
10 (+1 group indicator)
Split value (r)
0.2
0.2
Optimizer
Adam
Adam
Batch size
256
256
Learning rate
0.001
0,001
Epochs
3000
3000
Number of labeled samples (n)
20
20
Number of unlabeled samples (N)
990
990
Number of test samples (r2
790
790
Number of validation samples (Ava)
200
200
Labeled fraction
1 %
1
Unlabeled fraction
50%
50%
Validation fraction
10%
10%
Test fraction
39%
39%
Number of seeds Early stopping
100
100
Enabled
Enabled
D.2 Training with access to the group indicator
This section provides a detailed report of the experiments summarized in Section 4.1 performance across a range of noise means μ E [0.1. 7]. which are applied as a bias to group B. Models are assessed using mean squared error (MSE) and R² scores, both overall and per group. Figure ST presents the MSE for all methods across the extended range of noise biases μu, expanding upon the results shown Figure
We evaluate
rep the corresponding R²2 scores on the full test set. Note that higher R22 values indicate better performance, while R² < 0 implies that a naive predictor (i.e., always predicting the mean y) performs better. Following that figure, we can see that our method matches outperforms all baselines across the full range of noise values. The performance gap
20

---

increases as the bias in group B increases, demonstrating our method's robustness to inaccurate pseudo-labels and its ability effectively leverage the group indicator.
30
20
10
8
6
Teacher
Only Labeled
MSF
SSL
4
PPI++
PP-SSL (Ours)
2
0.1 1.0 2.0 3.0 4.0 5.0 7.0 Noise bias (μ)
Figure S1: MSE on synthetic data as a function of noise bias μ.. Results are shown for the entire test set. Results evaluated on 100 independent experiments.
1.0
0.8
Teacher
0.6
Only Labeled
SSL
0.4
PPI++
PP- PP-SSL (Ours)
0.2
0.0
0.1
3.0
5.0
7.0
Noise bias (u)
Figure S2: R on synthetic data function of noise bias μ. Results the entire test set when training with access to the group indicator feature. Results evaluated on 100 independent experiments.
D.3 Training without access to the group indicator
This experiment extends the setup from Section 4.1 but differs a key aspect: does not assume access to the group indicator for each sample.
The data generated as described in Section D.I using the hyperparameters specified Table ST Figure S3 shows the group-wise MSF. For group A, where the teacher is accurate, all methods perform similarly well, with the Teacher model outperforming others. However, group B, which is affected by biased noise, we can see the advantage of our approach expecially as the noise mean increases. This highlights our method's 's robustness even the absence explicit group information. As can be seen in Figure SS4 our method also achieves the lowest MSE and the highest ft2 under high noise-bias conditions across both groups, while maintaining competitive results when noise-bias is low.
Figure S5 presents the MSE curves evaluated on the training and test data. As can be seen, our PSL method converges faster than the Only Labeled baseline, which is in line with the analysis from Section Observe also that PPI++ exhibits similar convergence rate.
D.4 Effect of group indicator: a comparative analysis
We now provide side-by-side comparison of the performance of the models trained with and without access the group indicator. Figure S6 shows that including group indicator significantly improves
21

---

50
50
25
25
15
15
Teacher
SSL
+P
10
10
PP-SS1 (Ours)
MSE
MSE
5
5
1.0
2.0
5.0
7.0
1.0
2.0
5.0
7.0
Noise bias (1)
Noise bias (μ4
(a) Group A
(b) Group
Figure S3: MSE on synthetic data without group indicator function of noise bias μr. Results are shown for: (a) the full test set; (b) Group A, where the teacher model is accurate (oracle-like); and (c) Group B, which includes additive biased noise. Results evaluated on 100 independent experiments.
50
1.0
25
15
0.8
Teacher SSL
0.6
PP
10
PR-SS1, (Ours)
MSE
0.4
5
0.2
0.0
1.0
2.0
5.0
7.0
1.0
2.0
5.0
7.0
Noise bias (μ4
Noise bias (μI)
(b) R
(a) MSE
Figure S4: Overall MSE and R² on synthetic data without group indicator, across all test samples versus noise bias μ added to group B.
model performance, expected. These results highlight the value of leveraging group information, particularly for methods like ours that can effectively utilize this knowledge.
E
Tabular real data experiments: supplementary details
In this section, we provide additional experimental details related the experiments described in Section 4.2 This includes the data split process and hyperparameter configuration (Section E.1) and extended results for the same experimental setup (Section E.2
E.1
Experimental details
Metrics We report both MSE and Rt2 metrics to evaluate model performance.
Setup and environment All experiments were run on an Ubuntu 20.04.6 LTS system. In order to run single seed experiment, hardware included CPU cores from Intel(R) Xeon(R) Gold processor at 2.40 GHz and 32 GB of RAM. The software environment used Python 1.3.
Data preparation We use the California Housing dataset 34 which consists of 20,640 samples and numerical features: median income, house age, average rooms household, average bedrooms per household, population, average household size, latitude, and longitude. The target variable median house value. To simulate natural distribution shift, we partitioned the data into groups
22

---

Only
12
Only
PPSSL (Ours)
PP-SSL (Ours)
Train Loss
8
+P[++
Test Loss
10
+
6-
6-
4-
2
5
10
15
20
25
30
0
5
10
15
20
25
30
Epoch (x100)
Epoch (x
(a) Train MSE
(b) Test MSE
Figure S5: experiment.
Train and test MSE (loss) during model training using 0.1 for synthetic data
50
50
25
25
15
15
Teacher
Only Labeled
PT+++
10
10
(Ouzs)
MSE
MSE
5
5
0.1
5.0 Noise bias (μ))
1.0
2.0
7.0
0.1
1.0
2.0
5.0
7.0
Noise bias (μ4)
(a) Without group indicator
(b) With group indicator
Figure S6: Synthetic experiments comparison of MSE across different methods as function of noise
level on the entire test set, with and without access the group indicator feature. Right: Models are trained with an additional input feature indicating group membership. Left: Models are trained without access to group membership information.
based on the 7-percentile of the target variable y. Group A consists of samples with lower target values (bottom 7 percent), and group B contains higher- samples. We chose 0.4, meaning the bottom 40% of the samples belong to group A. Figure(S). S7 shows the sorted target values used for this grouping.
Model details We use linear model of the form w x + b, trained for 1,000 epochs on each configuration. For consistency and fair comparison, all methods (including the Only Labeled baseline) were implemented using numpy code, as PPI++ and our method could not be supported directly by Scikit-learn.
Table S2 summarizes the key hyperparameters and dataset statistics used in the California Housing experiment.
Hyperparameters
E.2 Additional results
In what follows, we provide additional results for the experiment from Section 4.2 Figure S9
reports the overall R² score on the full test set, covering both groups and Figure 58 shows the MSE across the two groups with broader range of N₂ values that were omitted in Section 4.2 for readability. Overall, one can see that the results follow the same trends as those presented in Section 4.2
23

---

split: 1.57
2000
Frequency
1500
1000
500
2
3
4
5
Target (Median House Value)
Figure S7: Ordered target house prices illustrating the 40% threshold for group splitting.
Table S2: California Housing Experiment Parameters
Parameter
Value
Number of labeled samples (n)
102 (0.5%)
Number of unlabeled samples (N)
16,327 (79%)
Pre-train sample count (NA + max NB)
102 (0.5%
Pre-train samples from group A (Na)
51
Pre-train samples from group B (NB)
(51, 38, 25, 12, 10, 5, 2, 4,108 (20%)
Number of test samples (Res)
Number of features (m)
Split threshold (r)
0.4
Number of seeds
100
Epochs
1000
Optimizer
SGD
Batch size
Full dataset
Learning rate
0.01
Early stopping
Enabled (patience: 10 steps)
Visual real data experiments: supplementary details
F1 Experimental details
Metrics For evaluation, we report the Mean Average Error (MAE) on the full test set well as
separately on group A (ages 0-30) and group (ages 31+). MAE is defined as > i= [f(xr) y"
where fxx is regression model. (lower is better) We report also MSE and R²
Setup and environment All experiments were conducted on a high--erformann computing cluster running Ubuntu 20.04.6 LTS. The hardware configuration includes 98 CPU cores (Intel(R) Xeon(R) Gold 2.40GHz), 256 GB of RAM, and 8 NVIDIA A40 GPUs. The software stack consists of Python 3.11.3, PyTorch 2.5.1, and CUDA 12.4.
Data preparation We use the UTKFace dataset [35]. which comprises over 20,000 face images labeled with age, gender, and ethnicity. Images are preprocessed by applying standard face alignment followed by resizing 128x128 pixels. For data augmentation during training. apply random horizontal flipping, random rotations, and color jitter to improve generalization. The age labels in UTKFaaee range from to 116 years, which treat regression problem to predict the exact age.
Model details All models are based on the ResNet-50 architecture pretrained on ImageNet, down- loaded from "htttss///domload pytorch, wrg./model//reenett---a-aa .at The fi- nal classification head is replaced with fully connected module composed of: Dropout(0.2)

---

4
0.
00
2
Teacher
1.2
Only Labeled
000
SSL
PPI++
1.0
MSE
1000 0 0p
0
PP-SSL (Ours)
0.8
A
0.6
0.4
1.0
0.75
0.49
0.24
0.2
0.1
0.04
0.02
Group pretrain fraction (Na/Na)
Figure S8: Results on California Housing dataset: MSE as a function of NB/N, AA on the entire test Other details are as Figure
0.8
Teacher
0.6
Only Labeled
SSL
R2
0.4
d 03000
PPI++
PP-SSL (Ours)
0.2
0O00
0.0
1.0
0.24
0.1
0.04
Group B pretrain fraction (Ns/Na)
Figure S9: Results Califoria Housing dataset: R2 as function of Na on the entire test set. Other
details in Figure 3
L2n4ar(2048, 256) - ReLU + Linear(256, 1), as implemented in the open- source codebase in 40 This architecture is used across all methods.
Hyperparameters We use the same training setting for all methods, as summarized in Table S3
F.2 Additional results
In Section 4.3. we present experiments using the MAE performance metric. Here, we complement those by reporting additional results using the MSE and R² metrics, evaluated both per group and on the entire test set. Figure presents the results based on the MSE metric, while Figure S11 displays the corresponding R² scores.
Notably, our proposed PP-SSL method consistently achieves the best performance among all semi- supervised approaches. It demonstrates noticeable improvements, with particularly strong gains in group B where the Teacher model in relatively inaccurate. Additionally, the low variability of the results highlights the stability of PP-SSL across different training runs.
G CIFAR-10 real data experiments: supplementary details
G.1 Experimental details
Metrics For evaluation, we report the top-1 Accuracy on the full test set as well as separately on Group (clean classes) and Group B (corrupted classes). Accuracy is defined as correct-olassffiatiio by all classifications.
25

---

Table S3: Training and experimental configuration for the UTKFace age estimation experiments.
Parameter
Value
Backbone architecture
ResNet-50 pretrained on ImageNet
Model final layers
Dropout(0.2) - Linear(2048,256) - ReLU - Linear(256,1)
Loss function
L1
Optimizer
SGD
Momentum
0,9
Weight decay
0,001
Initial learning rate
0.001
Batch size
512
Epochs
100
Early stopping
Patience = 10 (based on validation loss)
Number of seeds
5
Pretrain set fraction
7% (1,628 samples)
Labeled set fraction
3% (698 samples)
Unlabeled set fraction
70% (16,288 samples)
Validation set fraction
10% (2,326 samples)
Test set fraction
10% (2.062 samples)
Group size (ages 0-29)
813 samples
Group B sizes (ages 30+)
{ 16. 33, 81, 163, 325, 488, 816}
1000
1000
1000
500
200
200
200
Teacher
150
1.50
150
Laboled
MSE
SSE. PP-SSL (Ours))
100
100
100
1.0 0.6
0.4
0.2
0.02
1.0
0.6
0.4
0.2
0.02
1.0
0.6
0.4
0.2
0.02
Group pretrain fraction (Ng)N,)
Group pretzain fraction (Ng)Ma)
Groug
peetrais fraction (Ng)Ma)
(a) All test set
(b) Group A
(c) Group B
Figure S10: Results for age estimation: MAE a function of NB/NA, A. across data splits. Other details are as in Figure
Setup and environment All experiments were conducted on a high-performance computing cluster running Ubuntu 20.04.6 LTS. The hardware configuration includes 98 CPU cores (Intel(R) Xeon(R) Gold 2.40GHz), 256 GB of RAM, and 8 NVIDIA A40 GPUs. The software stack consists of Python 3.11.3, PyTorch 2.5.1, and CUDA 12.4.
Data preparation We use the CIFAR- 10 dataset 337. standard benchmark comprising 60,000 color images in 10 classes (airplane, automobile, bird, cat, deer, dog, frog. horse, ship, and truck), with 6,000 images per class. The dataset is split into 50,000 training images and 10,000 test images. For our experiment, we reserve 5,000 images from the training set to pretrain a teacher model, and the remaining 45,000 images used for student training.
All images are resized to 256 x 256 and center-cropped to 224 x 224 to match the input requirements of pretrained ResNet-50 models. We apply standard data augmentation for training, including random resized crops and horizontal flipping. To evaluate model robustness, we optionally add visual corruptions (e.g., Gaussian noise, saturation, brightness) selected classes such cat, dog, and deer using a class- aware corruption transform prior to normalization.
Model details We extract features using ResNet-50 model pretrained on ImageNet, ob- tained from the official PyTorch model zoo at 1osnet50-11ad3fal pth The ResNet encoder outputs 2048-dimensional feature vector for each image. We replace the final classification head with a single linear layer of shape (2048, 10) to classify among the 10 CIFAR- 10 classes.
poooomooad.yyttrchh...oo org/models/
26

---

1.0
0.8
0.6
B
R2
0.4
0.2
leacher
0.0
Only Labeled
1.0
0.6
0.4
0.2
0.02
SSL.
Group B pretrain fraction (Na/NA)
PP-SLL (Ours)
Figure S11: Results for age estimation: R as function of NB/ NB/NA, on the entire test set across data splits. Other details in Figure 4
In knowledge distillation settings, we initialize the teacher with pretrained linear model and load the checkpoint into duplicate of the student architecture. The provides soft targets for distillation-han supervision.
teacher is frozen during training and
Hyperparameters Table S4 summarizes the key hyperparameters and dataset statistics used in the California Housing experiment.
Table $4: California Housing Experiment Parameters
Parameter
Value
Number of labeled samples (n)
4,500 (9%)
Number of unlabeled samples (N)
40,500 (81%)
Number of pretrain samples (N
5,000 (10%)
Number of test samples (Rea)
10,000
Number of features in classifier
2048
Number of seeds
5
Epochs
100
Optimizer
SGD
Batch size
256
Learning rate
0.001
Labeled Loss
CE
Unlabeled Loss
KL divergence
G.2 Experimental Setup
In this section, we present an empirical study on semi-supervised learning under class-conditional corruption, using the CIFAR-10 10 dataset controlled testbed. Our goal is to investigate how the presence of visual corruption in a subset of classes affects learning, and how distillation-basee methods corruption into the dataset in the image space (c.g., saturation, brightness, Gaussian noise). We split the training data into "noisy" and "clean" subsets based class labels, and study how different training strategies perform this mixed-quality regime.
mitigate these effects. To that end, inject various types of
Following recent semi- cemi-supervised learning pipelines, we first extract 1048-dimensional features from all training and test images using a fixed ResNet50 backbone pretrained on ImageNet. To enable teacher-student framework, we first split the training set into two parts. A disjoint subset of 5000 clean training samples is used train a teacher model. This model trained in a supervised manner using the standard cross-entropy loss. Once trained, the teacher model is saved and used to guide
27

---

downstream learning via knowledge distillation. We evaluate performance separately corrupted and non-corrupted classes to understand the strengths and limitations of each method under real-world noise scenarios.
To simulate semi-supervised learning scenario, split the feature-extracted training data such that only 10% of samples have labels, and the remaining 90% are treated as unlabeled. A linear classifier is trained on this data, with supervision provided via hard labels and, when available, soft predictions from the teacher model. The loss function combines cross-entropy for labeled data with Bellback-Leibblee divergence term for distillation (pseudo-labee labeled data):
LSSI. LCB A LKI.,
Lpp.SSI. LCR A (C L.,.K)::
where for sample with label vector y E {0,13 and predicted probabilities p E [0, 1]c, the cross-entropy loss is defined as:
LCR(D, v)
Σ Yi log(p;),
where C' is the number of classes and is the predicted probability for class i. When applying knowl- edge distillation, we further incorporate a KL divergence loss between the probability distribution produced by the teacher model and that of the student model. The KL divergence is given by:
(xl(gllp)
Ti log
f
where is the distribution from the teacher, and is the student's predicted distribution.
G.3 Additional Results
Table S5: Results for corruption severity
Labeled
SSL
PPI++
PP-SSL
Teacher
Brightness
Total
81.42+0.03
81.48+0.06
77.92+0.02
81.23+0.11
85.68+0.01
Group A
85.83+0.03
85.97+0.05
81.9940.04
84.93+0.06
88.5020.03
Group
3.72+0.74
4.92+1.41
56.93+0.12
60.5520.26
4.81+0.30
Saturate
Total
74.8940.09
74.65+0.30
70.82+0.14
74.97+0.65
79.1520.06
Group A
79.11
78,88
81.14
84.12+0.21
75.46=0.09
Group
88.6240.02
88.7020.03
85.49+0.03
88.24+0.04
88.53+0.03
Shot Noise
Total Group
84.22+0.07
84.41+0.19
80. 80.96+0.03
83.91+0.18
88.48+0.03
57.44+1.12
56.38+3.20
70.99+0.10
74.41+0.55
45.31+0.26
Group B
78.2140.33
78.45+0.45
80.58+0.03
83.5620.09
74.45+0.07
Spatter
Total
82.31+0.03
82.34+0.05
80. 1640.03
83.1020.06
82.86+0.02
Group A
88.4040.04
88.5220.06
85.48+0.03
88.28+0.10
88.50+0.03
Group B
54.36+1.10
54.95+1.52
19.1840.09
72.57+0.21
42.06+0.23
Speckle Noise
Total 74.08+0.11
73.89+0.17
75.88+0.06
78.82+0.09
69.81+0.10
Group A
63.1540.22
63.57=0.42
76.92+0.03
79.93=0.09
63.19=0.07
Group
88.43+0.03
88.5340.02
85.46+0.03
88.27+0.04
88.53+0.03
28

---

Table S6: Results for corruption severity
Labeled
SSL
PPI++
PP-SSL
Teacher
Brightness
Total
82.89=0.07
82.87=0.19
80.97+0.1 17
84.2620.11
81.1440.06
Group A
72.10=0.28
71.63+0.75
72.65+0.63
77.59=0.38
64.15+0.17
Group B
87.52+0.03
87.68+0.09
84.54+0.14 14
87.12+0.04
88.48+0.06
Saturate
Total
80.73+0.09
80.67=0.13
79.95+0.06
83.32+0.02
77.15+0.15
Group A
62.94=0.28
62.61+0.41
67.92+0.
72.94+0.10
51.07=0.52
Group B
88.35+0.02
88.41+0.02
85.11+0.07
87.77+0.03
88.48=0.04
Shot Noise
Total
79.29=0.20
79.31+0.45
78. 78.92+0.09
81.9920.16
77.34+0.14 14
Group A
57.93+0.65
63.57+0.19 19 57.67=1.44
67.31+0.39
51.87+0.44 44
Group B
88.45+0.03
88.5840.06
85.49+0.09
88.28+0.10
88.50+0.05
Spatter
Total
66.09+0.42
66. 1320.58
76.75+0.05
79.20=0.11
65.4640.26
Group A
13.92+1.39
13.77=1.95
56.37+0.23
58.0920.34
12.63+0.91
Group B
88.45+0.03
88.5810.03
85.4920.05
88.25+0.04
88.67#0.06
Speckle Noise
Total
77.69=0.23
77.86+0.31
78.52+0.08
81.53+0.11
76.30+0.15
Group A
52.62+0.74
52.95+1.02
62.33+0.21
65.7940.31
48. 13+0.45
Group B
88.44+0.03
88.54+0.02
85.46+0.06
88.27+0.04
88.65+0.05
Only Labeled
SSL
PPI++
PP-SSL (Ours)
Training Loss
0.9
0.8
0.7
0.6
100
Epoch
Figure S12: Convergence graph training loss as function of epochs during CIFAR-10 training
Analysis. Tables $5 S6 shows that method improves the accuracy the corrupted classes across different corruption severities, with the adaptive choice of leading to improved performance compared to baselines. The results confirm the robustness of PP-SSL varying levels of noise.
In addition to accuracy, Figure S12 we also evaluated convergence time. PP-SSL converges faster than the only-labeled baseline, standard SSL, and PPI++ which require more training epochs to stabilize. This faster convergence further highlights the efficiency our adaptive weighting strategy in practice.
29

---

H
Additional Experiments on the Choice of
In this section, we provide additional experiments that validate our theory regarding the optimal weight parameter A. These results complement the insights derived in Corollary 3.3
H.1 Synthetic Regression: Theoretical vs. Empirical X
We first conduct a controlled synthetic regression experiment to compare the theoretical X from Corollary 3.3 (i.e., in Eq. [0P with the empirical value that minimizes the gradient variance. Since the theoretical X (denotes as Athorry below) is derived by minimizing an upper bound on the variance rather than the variance itself, need not coincide exactly with the empirical variance minimizer (denoted as X practicc)
Experimental setup. Synthetic data is generated according to the linear noisy model y x" w +e, with x ~ (0, Ia) where d = 10, ~ A0 o²)) where of = 0.01, and w ~ ~O, Lg). We sample 50 labeled examples and N E {200, 2000] unlabeled examples from this model. synthetic teacher provides pseudo-labels of the form f(x) x x'(w"" be,) Ç, where b R is bias parameter and ( ~ (O, of) introduces noise. Both b and of control the teacher's quality. Since directly mapping them to o² and of is non-trivial, we set b E {0.1, of {0.01, 1.5] and empirically estimate o² and o². We consider the MSE loss E(w; x. y) }(x" w y)² and estimate its gradient at randomly sampled test point T ~ (0, Ta).
Results. Table S7 reveals that Aiory consistently approaches the performance (in terms of measured variance) of practice across different settings. The theoretical value yields variance reductions within narrow margin of the empirical optimum.
Table S7: Comparison between theoretical and empirical A Theoretical values match empirical ones closely in terms of minimizing gradient variance.
r
C
a²
22
theory
practice
Var Var(Aiiheor))
Var (> practice
0.025
0.01
0.1
10.4078
0,0793
0.9682
0,99
0,0050
0,0046
0.025
1.5
14.1715
7.6479
0.6336
0.70
0.1733
0.1669
0.25
0.01
0.1
11.5082
0.0440
0.7969
0.77
0,0749
0,0709
0.25
1.5
1
11.9349
10.1486
0.4324
0.35
0.1034
0.1004
H.2
Adaptive Dynamics of X
Next, we investigate how the adaptive algorithm adjusts À over training epochs in relation to teacher quality. We track A, when initialized at Àg across 1000 epochs for teachers of varying pseudo- label mean squared error (MSE).
Results. Table S8 shows that X converges automatically toward the theoretical X Moreover, the limiting value increases as the teacher error decreases, which matches intuition: better teachers justify higher weight on pseudo-labels.
Table $8: Adaptive dynamics of À across epochs. X converges toward the theoretical X*, with larger for higher-quality teachers.
Teacher MSE
Aı00
Ago0
Ag00
Auoo
Asoo
A600
Ayco
Asap
A00o
A1000
Xieoryy
0.01
1.000
0.966
0.946
0.934
0.929
0.926
0.925
0,924
0.924
0.924
0.924
0.968
0.04
1.000
0.945
0.902
0.866
0.836
0.811
0.791
0.775
0.761
0.751
0.742
0.795
0.05
1.000
0.930
0.867
0.812
0.766
0.727
0.696
0.671
0.651
0.635
0.622
0.634
30

---

H.3 Comparison to the Infeasible Best Fixed À
Finally, we benchmark our adaptive method (PP-SSL) against an oracle baseline that uses the infeasible "best" fixed À minimizing test enror. We employ the subgroup bias synthetic regression setup from Section 4.1
Results. Table S9 shows that PP-SSL matches the performance of the infeasible oracle across all bias levels. This demonstrates that adaptivity is sufficient recover the performance of the best fixed X, without prior knowledge of the optimal value.
Table $9: Comparison of PP-SSL the (infeasible) best X. Our adaptive algorithm matches the oracle performance across all bias settings.
Bias p
Inf. X"
Inf. MSE (Total/A/B)
PP-SSL À
PP-SSL MSE (Total/A/B)
0.1
0.50
1.93 1.76/ 2.59
0.46
1.93/1777 2.58 1.85 1.69 2.50
0.5
0.50
1.85 1.69 2.50
0.59
3.0
0.35
2.76 1.95 6.05
0.33
2.76 1.96 6.04
5.0
0.35
2.45 2.07 4.05
0.36
2.47 2.06 4.12
Summary. Across all three settings, observe strong alignment between the theoretical prescrip- tion, the adaptive algorithm, and the empirical optimum. This provides further evidence that the proposed choice of À is both theoretically sound and practically effective.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims presented in the abstract and introduction are substantiated by the theoretical results in Section as well as the empirical results in Section and in the Appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our method in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
32

---

Answer: [Yes]
Justification: The theoretical results and assumptions are presented in Section with complete proofs provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section outlines the experimental setup, with full details provided in the appendix. Code is included in public GitHub repository https://ithub.com/ooshhhhaaa
SSL
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
33

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The paper includes the complete code in the supplementary materials, along with detailed experimental settings and dataset descriptions in the Appendix. The "California Housing" and "UTTKFace" datasets used in the paper are publicly available and properly cited. A public GitHub repository will be released upon publication ensure full reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
oliccy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https 7/nips.
Code/Sumeeeeeeeee sobbbmssste aassonnooicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details regarding the experiments described in Section appear at the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yss]
Justification: All plots shown as boxplots capture variability across runs, and tables report means along with standard errors. Corresponding figures can be found in Section and in the Appendix and tables can be found in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
34

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information about the computational resources used in this work in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in this paper follows the NeurIPS Code of Ethics in all
aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
35

---

Justification: In Section we discuss the potential positive and negative impacts of our method.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper does not involve the release of high-risk models datasets. All data used are standard, publicly available benchmarks, and no additional safeguards are necessary.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
All datasets and external sources utilized in this work are appropriately cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
36

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release any new assets. However, we provide a well- documented software package.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
37

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The core methodology in this research does involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
38


Reviews:
[{"review_id": "mVJ0BId36d", "reviewer": "Reviewer_NZKV", "summary": "The paper proposes PPI-SSL, which extends the Prediction-Powered Inference (PPI) framework to the semi-supervised learning (SSL) setting. It introduces a way to balance the contributions of labeled and pseudo-labeled data by adaptively tuning a weighting parameter $\\lambda$, updated via AdaGrad.", "questions": "- Please clarify what is truly novel in the paper beyond the AdaGrad update of \\lambda.\n- How realistic is the Lipschitz assumption in Lemma 3.1, and how is it enforced in practice?\n- Could you verify Corollary 3.3 empirically on a synthetic setting, and is it used to initialize \\lambda in real experiments?", "limitations": "Limitations are discussed in the paper, and there are no obvious societal concerns.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strength\n- The paper is clearly written and well organized.\n- The proposed adaptive mechanism for $\\lambda$ is intuitive and practically motivated.\n\n## Weakness\n- I am not an expert in this area and not a theory person, but as far as I understand, the formulation of Eq. (3) has already appeared in prior works like PPI and Doubly-Robust Self-Training. So it is not accurate to say this paper newly “extends its core idea to semi-supervised learning (SSL) for model training.” At least that part is not new. The relatively novel contribution seems to be the AdaGrad-based update of $\\lambda$, which deserves clearer emphasis.\n- Lemma 3.1 assumes the gradient function is Lipschitz, which seems like a very strong assumption. I’m wondering how realistic this is for modern networks, and whether any regularization is applied to enforce it. (In practice, it’s more common to assume the model is Lipschitz rather than the gradient itself.)\n- It would be helpful to empirically verify Corollary 3.3 in a toy experiment, e.g., using a simple MLP where the bound involving $\\sigma$ can be computed. Also, is this bound used in practice to initialize $\\lambda$? If so, please clarify.", "was_revised": true, "final_justification": "I thank the authors for the extra clarification and synthetic experiment. I am raising my score as my concerns have been addressed.", "rebuttal": "Thank you for your thoughtful review and constructive feedback. We appreciate your careful evaluation of our work and the opportunity to address your concerns.\n\nWe understand that your major concern is with the novelty of our work, so we start by clarifying this issue and then proceed by responding to the rest of your comments.\n\n## C1: Novel Contributions Beyond AdaGrad Update;  the formulation of Eq. (3)\n\nYou correctly note that Eq. (3), which we refer to as the PPI loss in the paper, has appeared in prior works. It was introduced in PPI [1], and used in PPI++ [2], and Doubly-Robust Self-Training [3]. However, our work introduces several key distinctions that extend beyond simply applying AdaGrad to update the tuning parameter λ of the PPI loss that weighs the influence of the unlabeled data.\n\nFirst, to clarify, PPI and PPI++ focus on constructing prediction intervals for parameters of interest, such as the mean or quantile of a parametric distribution. These two papers do not utilize the unbiased loss for *model training*. As such, they differ from the focus of this paper, which centers on the formulation of an unbiased risk for semi-supervised learning. The Doubly Robust Self-Training [3] method does involve model training; however, our work introduces key novelties in relation to [3], as well as to [1,2], which we outline in lines 32-39 and summarize here: \n\n*Shift from Asymptotic to Finite-time Analysis:* While PPI, PPI++, and Doubly-Robust Self-Training analyze only the asymptotic behavior of Eqs. (3)-(4) at the **population optimum**, our work focuses on finite-time analysis during training. Specifically, we explicitly quantify the benefit of the teacher depending on its quality (teacher error), and show how and when it enables better generalization guarantees compared to the standard case. Such a finite-time, dynamic analysis of the optimization process does not exist in prior work.\n\n*Adaptive Training Mechanism:* Our second main contribution is the adaptive adjustment of λ during training, which automatically balances labeled and pseudo-labeled data contributions without requiring prior knowledge of variance parameters. This eliminates manual tuning and provides greater robustness compared to fixed-weight approaches. Such an adaptive adjustment of λ does not exist in prior work.\n\n*Performance Gain:* Our method demonstrates superior performance under group imbalance, where the teacher model performs well on one group but underperforms on the other. In such cases, traditional SSL methods fail, and existing PPI-based methods underperform compared to our method, despite using the same loss function but with a fixed lambda.\n\n## C2:  How realistic is the Lipschitz assumption in Lemma 3.1, and how is it enforced in practice?\n\nThe Lipschitz assumption in Lemma 3.1 applies to the *loss function*, not the model itself. This is an important distinction that makes the assumption more realistic in practice.\n\n*Why is it realistic?* We show in Appendix A that this assumption naturally holds for the squared loss (regression) and the logistic loss (classification). The assumption provides the theoretical foundation for relating the loss error gradient variance σₑ² to the teacher's prediction error $\\mathcal{E}^f$.\n\n*Importantly, we do not need to enforce this assumption.*  Our analysis holds under the variance conditions specified in Eqs. (8) and (9): bounded gradient variance σ² and bounded loss error gradient variance σₑ². The Lipschitz assumption serves only to provide intuition and theoretical interpretation of how the teacher's prediction error relates to the loss error gradient variance. In other words, our theory relies on the assumptions in Eqs. (8) and (9) and holds even if the loss gradient $\\nabla\\ell(w;x,y)$ is not Lipschitz in $y$. Therefore, there is no need to enforce the Lipschitz assumption in practice. \n\n\n## C3: Verifying Corollary 3.3 and λ initialization in our experiments\n\nThank you for raising this important point. **No**, the theoretical λ* from Corollary 3.3 is not used for initialization since it depends on the unknown quantities σ² and σₑ². Instead, we simply initialize it to $\\lambda_0=1$. Our adaptive mechanism updates λ during training without requiring these parameters. We also provide a new experiment, showing that our algorithm for online learning λ aligns with the optimal $\\lambda^*$; we kindly refer the reviewer to response C3 to Reviewer sxXE.\n\nTurning to the empirical validation of Corollary 3.3, we have conducted a new synthetic regression experiment to validate the theoretical λ* from Corollary 3.3 against the empirical λ that minimizes the gradient variance. We emphasize that $\\lambda^*$ from Eq. (10) is the minimizer of the variance's **upper bound**, which might differ from the minimizer of the variance itself, depending on how loose the upper bound is for a specific problem setting. See results and full experimental setup below.\n\n*Results:* The table below shows that $\\lambda^\\*$ from Corollary 3.3 (denoted as $\\lambda^\\*\\_{\\text{theory}}$) attains nearly the same performance, in terms of gradient variance (**Var($\\cdot$)**), as the empirical $\\lambda^*\\_{\\text{practice}}$, for different values of $r,\\sigma^2,\\sigma_e^2$.\n\n|      r   |      σ² |     σₑ² |   $\\lambda^*\\_{\\text{theory}}$ (Cor. 3.3) |  $\\lambda^*\\_{\\text{practice}}$ (Empirical) |  Var($\\lambda^*\\_{\\text{theory}}$) |   Var($\\lambda^*\\_{\\text{practice}}$) |\n|:--------:|:--------:|:--------:|:------------------------:|:------------------------:|:-----------------:|:-------------------:|\n| 0.025 | 10.4078 |  0.0793 | 0.9682 |      0.99 |           0.0050 |             0.0046 |\n| 0.025 | 14.1715 |  7.6479 | 0.6336 |     0.70 |           0.1733 |             0.1669 |\n| 0.25  | 11.5082 |  0.0440  | 0.7969 |     0.77 |           0.0749 |             0.0709 |\n| 0.25  | 11.9349 | 10.1486 | 0.4324 |      0.35 |           0.1034 |             0.1004 |\n\n\n*Experimental setup:*  We design a synthetic experiment to analyze the behavior of prediction-powered gradients in a controlled linear regression setting. The data is generated according to the model $y = \\mathbf{x}^\\top \\mathbf{w}^* + \\epsilon$, where $\\mathbf{x} \\sim \\mathcal{N}(0, I_d)$, $\\epsilon \\sim \\mathcal{N}(0, \\sigma_*^2)$ with $\\sigma_*^2=0.01$, and $\\mathbf{w}^* \\in \\mathbb{R}^d$ is a parameter vector sampled once per configuration from $\\mathcal{N}(0,I_d)$. We sample $n=50$ labeled samples from this model and $N\\in$ {$200, 2000$} unlabeled samples from the same feature distribution $\\mathcal{N}(0, I_d)$. A synthetic teacher provides pseudo-labels of the form $f(\\mathbf{x}) = \\mathbf{x}^\\top (\\mathbf{w}^* + b \\mathbf{e}_1) + \\zeta$, where $\\mathbf{e}_1 = [1,0,0,...,0]$, $b\\in\\mathbb{R}$ controls the teacher bias, and $\\zeta\\sim\\mathcal{N}(0, \\sigma\\_{\\zeta}^2)$. This setup gives control over $\\sigma_e^2$ through the parameters $\\sigma\\_{\\zeta}^2$ and $b$. The results in the table correspond to a randomly sampled test point $\\tilde{w}\\sim\\mathcal{N}(0,I_d)$.\n\n## References:\n[1] Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I Jordan, and Tijana \nZrnic. Prediction-powered inference. Science, 2023.\n\n[2] Anastasios N Angelopoulos, John C Duchi, and Tijana Zrnic. PPI++: Efficient prediction-\npowered inference. arXiv preprint arXiv:2311.01453, 2023.\n\n[3] Banghua Zhu, Mingyu Ding, Philip Jacobson, Ming Wu, Wei Zhan, Michael Jordan, and Jiantao Jiao. Doubly-Robust Self-Training. Advances in Neural Information Processing Systems, 2024"}, {"review_id": "4BySpWYLui", "reviewer": "Reviewer_sxXE", "summary": "This paper proposes Prediction-Powered Semi-Supervised Learning (PP-SSL), which incorporates the bias-correction principle of Prediction-Powered Inference (PPI) into the training process of semi-supervised learning (SSL). The method addresses the bias introduced by inaccurate pseudo-labels using an unbiased gradient estimator, and dynamically integrates it into training via an online learning algorithm. Theoretical analysis establishes finite-time convergence guarantees, and empirical results on synthetic, tabular, and image-based regression tasks demonstrate that PP-SSL outperforms both classical SSL methods and previous PPI-style approaches under distributional shift.", "questions": "1. The explanation of the teacher model is not clear. If I understand correctly, the paper focuses exclusively on fixed teacher models. It is unclear how the method would adapt to self-training or evolving pseudo-label generators, which are more common in modern SSL pipelines. I would appreciate it if the authors could clarify this point.\n\n2. While the online λ tuning is theoretically well-justified, there is limited qualitative analysis or visualization. Can the authors provide qualitative visualizations of λ values during training and how they correlate with pseudo-label error over time?\n\n3. Are there comparisons between online λ and fixed λ values to better illustrate the benefits of adaptive tuning?\n\n4. Have the authors considered extending PP-SSL to image classification tasks like CIFAR-10 or STL-10? The absence of popular benchmark datasets may limit direct comparison with standard SSL baselines. It would be helpful to demonstrate how PP-SSL behaves in more common classification settings.", "limitations": "The authors assume that labeled and unlabeled data come from the same distribution, which can be a practical limitation. It would be helpful if the paper included a more discussion of this limitation.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The paper presents an insightful extension of the Prediction-Powered Inference (PPI) framework from inference to optimization, specifically tailored for semi-supervised learning. Unlike prior work, it directly addresses the practical issue of pseudo-label bias during training and introduces a principled mechanism to mitigate it. The paper is generally well-organized, with clear motivation, formulation, and theoretical analysis.\n\nNevertheless, several questions remain regarding the generality of the datasets, the explanation of the teacher model, and the analysis of the learned λ. Please refer to the questions.", "was_revised": true, "final_justification": "The authors' rebuttal provides clear and thoughtful responses, supported by additional experiments on image classification and the behavior of the adaptive λ. These results sufficiently address the concerns raised in my initial review. I agree that establishing a theoretical foundation for unbiased semi-supervised learning via the PPI loss is a meaningful and worthwhile contribution. My overall evaluation remains positive, and I have kept my original score.", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. We appreciate the recognition of our work as a principled extension of Prediction-Powered Inference (PPI) to semi-supervised learning (SSL), and for acknowledging the clarity of our motivation, theoretical contributions, and empirical results. Below we address the reviewer’s main concerns:\n\n\n## C1: How the method would adapt to self-training or evolving pseudo-label generators \nYou are correct that our current theoretical analysis focuses on a fixed teacher model. It is a widely used learning strategy, and so we believe it is important to build a clear theoretical foundation for unbiased semi-supervised learning via the PPI loss. We agree that extending our method to the case of self-supervised learning with evolving pseudo-label generators is an important direction. Below, we explain what makes such an extension challenging.\n\n\nOur method assumes a pre-trained, fixed teacher model that generates pseudo-labels throughout training. This design choice allows us to (i) theoretically analyze the convergence and dynamics of the PPI loss as the pseudo-label error variance σₑ² remains constant, and (ii) leverage regret bounds for the optimal λ.\n\n\n*From a theoretical perspective:* \nIf we assume the teacher evolves over time, $f_1,\\ldots,f_t$, and assuming the teacher is independent of the training data (e.g., using held-out data), our results directly generalize with σₑ² as a bound over all teachers. However, this would yield loose bounds. A more refined theory  requires deriving new gradient variance bounds that depend on the evolving sequence of the teacher’s error gradient variance σₑₜ²; naturally, this is a more complex theoretical setup than the one considered in our paper. \nThe common self-training setup, where the teacher evolves based on the training data (e.g., without using held-out data), is even more challenging because (i) it is unclear how to bound the gradient variance (due to the dependence on past data), and (ii) it requires moving from static to dynamic regret bounds as the optimal λ* changes over time (i.e., λₜ*); clearly, this transition is challenging and not trivial.\n\n\n*From a practical perspective:* While our theoretical guarantees are established for fixed teachers, our algorithm can be practically applied to self-training settings. In such cases, the same model serves as both student and teacher, with pseudo-labels being iteratively updated, and the PPI loss would still have the unbiasedness effect. Of course, this idea requires more exploration and validation that is beyond the scope of our current paper.\n\n## C2 : Qualitative visualizations of λ values during training and how they correlate with pseudo-label error over time\nWe appreciate this request for deeper insight into λ dynamics. It's important to note that in our fixed teacher setting, pseudo-label error remains constant over time since the teacher model is fixed. What could be correlated are the final λ values and the pseudo-label error, and so we provide such an analysis below.\nWe conducted a synthetic linear regression experiment where both the data generation process and the performance of the teacher model are controlled (see Reviewer NKZV, Comment C3, for experimental details). We will add the visualizations to the revised version.\n\nThe following table shows the value of $\\lambda_t$ for different epochs for teachers with varying pseudo-label errors (MSE). We initialize the learning with $\\lambda_0 = 1$. Observe how $\\lambda_t$ adjusts automatically, reaching the theoretical $\\lambda^\\*$ from Corollary 3.3 across different runs. Observe also that $\\lambda^\\*$ is increasing as the teacher’s MSE decreases, in line with the intuition.  \n\n| Teacher model MSE|\\| $\\lambda\\_{0}$ | $\\lambda\\_{100}$ | $\\lambda\\_{200}$ | $\\lambda\\_{300}$ | $\\lambda\\_{400}$ | $\\lambda\\_{500}$ | $\\lambda\\_{600}$ | $\\lambda\\_{700}$ | $\\lambda\\_{800}$ | $\\lambda\\_{900}$ | $\\lambda\\_{1000}$ |\\| $\\lambda^*\\_{\\text{theory}}$ |\n|-|- |- |- |- |- |- |- |- |- |- |- |- |\n|0.01|\\|  1.000 | 0.966 | 0.946 | 0.934 | 0.929 | 0.926 | 0.925 | 0.924 | 0.924 | 0.924| 0.924|\\| 0.968 |\n|0.04|\\| 1.000 | 0.945 | 0.902 | 0.866 | 0.836 | 0.811 | 0.791 | 0.775 | 0.761 | 0.751| 0.742 |\\| 0.795 |\n|0.05|\\| 1.000 | 0.930 | 0.867| 0.812 | 0.766 | 0.727 | 0.696 | 0.671 | 0.651 | 0.635| 0.622 |\\| 0.634 |\n\n\n## C3: Are there comparisons between adaptive λ and fixed λ?\n\n\nYes, in the paper we compare our adaptive method to PPI++, which uses a fixed, asymptotically optimal value of λ. (We also compared in our experiments to PPI with λ=1, but we omitted the latter as we found its performance to be inferior to PPI++.) In nearly all experiments, our adaptive method outperforms PPI++. \n\n\nTo fully address the reviewer's concern, we now include a new experiment that compares our method, PP-SSL, to PPI with an infeasible “best” λ that minimizes the error on the test data. We use the synthetic regression setup with subgroup bias, as described in Section 4.1. The results (see below) show that our adaptive algorithm matches the performance of the model trained using the best fixed λ across all bias settings. \n\n| Bias $\\mu$ |\\| Infeasible “best” λ | Infeasible “best” MSE (Total / A / B) |\\| PP-SSL λ | PP-SSL MSE (Total / A / B) |\n|:--|--|--|-|--|\n| 0.1 |\\| 0.50| 1.93 / 1.76 / 2.59 |\\| 0.46| 1.93 / 1.77 / 2.58  |\n| 0.5 |\\| 0.50| 1.85 / 1.69 / 2.50|\\| 0.59 | 1.85 / 1.69 / 2.50  |\n| 3.0 |\\| 0.35| 2.76 / 1.95 /6.05|\\| 0.33| 2.76/ 1.96 / 6.04  |\n| 5.0 |\\|0.35 | 2.45 / 2.07 / 4.05 |\\| 0.36| 2.47/ 2.06 / 4.12|\n\n## C4: Evaluation on image classification task\nFollowing your suggestion, we now include new results of PP-SSL for a semi-supervised image classification task. To illustrate the importance of the unbiased loss, we designed a controlled experiment on CIFAR-10 and its corrupted variant CIFAR-10-C [1]: we divided the data into 2 groups based on the classes, simulating a setting where the teacher is less accurate for some classes (Group A: dog, cat, and deer images from CIFAR-10-C) and more accurate for the other classes (Group B: images from the clean CIFAR-10). Image corruptions—Saturate, Spatter, and SpeckleNoise—were drawn from the CIFAR-10-C benchmark and applied to both training and test images of Group A.\n\nWe followed a standard SSL protocol: a ResNet50 pretrained on ImageNet was used to extract features from all images, and a linear classifier was trained on top using 10% labeled data and 90% unlabeled data. Specifically, the benchmark methods are as follows.\n- Teacher: a model trained on a clean, disjoint labeled subset using cross-entropy loss.\n- Standard SSL: Student model trained using CE loss for labeled data and KL loss for unlabeled data (classic knowledge distillation setting).\n- OnlyLabeled: Student model trained using CE loss, evaluated on labeled data.\n- PPI++: Student model trained using CE loss for labeled data and KL loss for pseudo-labeled data (both unlabeled and labeled as in Eq. (4)), with a fixed weight λ for the PPI loss. \n- Doubly-Robust Fixed-Teacher (DR-FT): Variant of Doubly-Robust Self-Training [2], that follows PPI++ loss but with heuristic update of λ, which is set to 0 throughout training and switched to 1 only during the final epoch.\n- Ours PP-SSL: Same as PPI++ but using our adaptive λ algorithm rather than a fixed one (PPI++) or a heuristic update (DR-FT).\n\nFollowing the table below, we can see that PP-SSL consistently outperforms all baselines in terms of total accuracy and Group A (corrupted) accuracy, while maintaining comparable performance on Group B (clean) samples. Notably, the performance gap on Group A varies with the corruption type, reflecting the varying inaccuracies of the pseudo-labeling under different distortions. To ensure the generality of these findings, we also ran the same experiment across other corruption types and severity levels. Since the results exhibited similar trends, we omit them for clarity and focus here only on severity level 5.\n| Method||Saturate|||Spatter||| Speckle Noise ||\n|:--|:-|-|-|:-|-|-|:-|-|-|\n| *Accuracy*| *Total*|*Group A* | *Group B* | *Total*|*Group A*| *Group B* | *Total*|*Group A*  | *Group B*| \n| DR-FT | 79.83±0.17 | 67.65±0.35 | 85.05±0.18 | 76.50±0.09 | 55.67±0.32 | 85.43±0.16 | 78.47±0.17 | 62.26±0.34 | 85.42±0.16 |\n| Labeled | 80.76±0.13 | 63.05±0.42 | 88.35±0.03 | 66.06±0.71 | 13.81±2.38 | 88.46±0.04 | 77.82±0.38 | 53.02±1.25 | 88.44±0.05 |\n| SSL | 80.67±0.15| 62.61±0.47| 88.41±0.02 | 66.25±0.70 | 14.15±2.36 | 88.57±0.03 | 77.95±0.37 | 53.25±1.23 | 88.53±0.03|\n| PP-SSL (Ours)| **83.31±0.02**| **72.94±0.11** | 87.76±0.04| **79.20±0.13** | **58.07±0.40** | 88.26±0.05 | **81.52±0.13** | **65.76±0.36** | 88.27±0.05 |\n| PPI++ |79.96±0.08 | 68.01±0.2 | 85.08±0.09 | 76.75±0.08 | 56.37±0.38 |85.49±0.09 | 78.51±0.12 | 62.36±0.34 | 85.44±0.09 |\n| Teacher| 76.92±0.12| 49.88±0.35 | **88.51±0.05** | 65.00±0.18 |10.01±0.67 | **88.57±0.04** | 76.12±0.11 | 47.16±0.35 | **88.54±0.05**|\n## C5: Assumption that labeled and unlabeled data come from the same distribution.\nThank you for raising this important point, which was also noted by Reviewer Gh11 (see Comment C2). As discussed there, while PP-SSL assumes a shared distribution between labeled and unlabeled data, we believe the framework can be extended to address distribution shift—for example, through importance weighting of loss or gradients, as in prior work (e.g., PPI, Zhu et al., 2023). These ideas could be incorporated into our adaptive λ scheme. We also note that in more complex cases—such as using generative models to produce unlabeled data—additional adaptations to the pseudo-labeling and gradient-weighting components of PP-SSL may be necessary. We will expand on these directions in the final version of the discussion section.\n\n## References:\n[1] Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. Hendrycks et al., 2019.\n\n[2] Doubly-Robust Self-Training. Zhu et al.,NeurIPS 2024"}, {"review_id": "UoEP5eJxsq", "reviewer": "Reviewer_yHkN", "summary": "The paper proposes using prediction-powered inference (PPI) with online power tuning as a method for training models in a semi-supervised fashion. The basic idea is to optimize the PPI++ loss while running an online learning algorithm to set the power tuning parameter. This method is shown to outperform natural baselines on several datasets.", "questions": "- In line 4: What do you mean by a factor of 4 being asymptotically equivalent? Having the same dependence on n? Also, where does the factor 4 come from?\n\n- In line 180: \"In contrast, PP-SSL provides a principled mechanism for leveraging unlabeled data that improves statistical efficiency without compromising correctness.\" What is correctness here? Don't we just care about optimizing the loss? If so, why do we care about not introducing bias?\n\n- Why did you decide to use AdaGrad (as opposed to a different stochastic optimization method)? Did you try any others?\n\n- I don't fully understand what the baseline methods are. What is \"teacher model f used for pseudo-labeling\"? Is this using f to label all unlabeled data and training a model on everything? Also, what is $\\lambda_{\\mathrm{PP}}^*$, seeing that your gradients depend on the current iterate (suggesting the tuning parameter should change at every step), unlike in PPI++ where there is a single distribution?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Online power tuning is a very reasonable idea and finding ways to leverage it in a learning context (in combination with PPI debiasing techniques) is a great direction.\n\nMy main concern is about whether the empirical evaluation is sufficient. Semi-supervised learning is a very vast literature, with many methods and tricks developed over the years. The paper doesn't compare to any baselines developed for semi-supervised learning. Also, two of three experiments are with a simple linear model, so I'm guessing the pseudo-labels can't be that good. It would be convincing to see a comparison with state-of-the-art semi-supervised learning methods, with state-of-the-art predictive models, and with challenging datasets. I was also wondering why there was no comparison to Zhu et al. [14], since that paper operates in the same setting?\n\nSee \"questions\" for further comments.", "was_revised": true, "final_justification": "This paper feels borderline. I would feel okay seeing it accepted, but I still think it is lacking empirical evaluation. Its core claim is that it offers a strong baseline for semi-supervised learning. This is the positioning in the abstract and throughout. But we are not seeing non-trivial baselines for semi-supervised learning in the comparisons.\n\nIt seems that the authors feel that the theory is an important part of the contribution. I'm sympathetic to that, but then the core messaging should be revised. Overall I think online power tuning is a good idea.", "rebuttal": "We appreciate the reviewer’s constructive feedback. It appears that the reviewer’s primary concern is a lack of experimental results.\n> My main concern is about whether the empirical evaluation is sufficient\n\nThe main manuscript includes synthetic regression experiments, and two real data regression experiments—tabular and visual. While we believe the paper offers a reasonable balance between theory and empirical evaluation, we agree that additional classification experiments could strengthen our work, a comment that was also raised by reviewers sxXE and Gh11.\n\n*New image classification experiments:* In response, we have added new image classification experiments and a new baseline method: the approach of Zhu et al. [4] (Doubly-Robust Self-Training, DR-ST). We kindly refer the reviewer to response C4 to Reviewer sxXE for the details and empirical results. In short, to demonstrate the importance of the unbiased loss, we study the performance of teacher-student learning on CIFAR-10 under image corruption, termed CIFAR-10-C, which allows us to control the accuracy of the teacher model for the minority group. These experiments are in line with all other experiments presented in the paper; our PP-SSL method outperforms the standard baseline methods to which we compare—Teacher, OnlyLabeled, Standard SSL, PPI++, and DR-ST [4]. Below, we elaborate on all baseline methods we used in paper.\n\n*New experiments validating our key theoretical results:* \n- Specifically, we empirically verify Corollary 3.3 by showing that our theoretical $\\lambda^*$ (the weight of the pseudo-labeled losses) aligns with the minimum of the gradient variance; kindly refer to response C3 to Reviewer NZKV.\n- We also provided a new experiment, showing that our algorithm for online learning λ successfully reaches the optimal λ; kindly refer to response C3 to Reviewer sxXE.\n## C1: Paper doesn't compare to any baselines developed for SSL; what the baseline methods are and what is the teacher model? \n\nThank you for the opportunity to clarify this point. We compared our method with common and simple SSL baselines as well as their PPI variants as follows.\n\n- **Teacher:** The teacher model $f$ is used for pseudo-labeling or for distillation, as widely used in the literature. The teacher is used to generate pseudo labels for the different teacher-student schemes, as we detail below.\n- **Standard SSL:** A model $g(x)$ trained on the following loss function: $\\mathcal{L}_{Labeled}(g(X),y) + \\mathcal{L}\\_{Pseudo}(g(\\tilde{X}),f(\\tilde{X}))$, where $(X,y)$ are the labeled data, $\\tilde{X}$ are the unlabeled data, and $f(\\tilde{X})$ are the pseudo labels for the unlabeled data. For classification, $\\mathcal{L}\\_{Labeled}$ is the cross-entropy loss, and $\\mathcal{L}\\_{Pseudo}$ is the KL divergence between the class probabilities (softmax) of the teacher $f(\\cdot)$ and student $g(\\cdot)$ models. For regression, the two losses are the standard squared loss. As such, the Standard SSL model acts as the most widely used approach for semi-supervised learning.\n- **PPI++:**  A model $g(x)$ is trained using loss based on the PPI loss (Eq. (4)): $ \\mathcal{L}\\_{Labeled}(g(X),y) + \\lambda \\cdot (\\mathcal{L}\\_{Pseudo}(g(\\tilde{X}),f(\\tilde{X})) - \\mathcal{L}\\_{Pseudo}(g(X),f(X)))$. Notice that we use pseudo labels $f(X)$ for the labeled data as well. The value of λ is fixed and determined by the asymptotic variance minimization presented in PPI++[3]. We refer to this fixed λ as $\\lambda\\_{PP}^*$.\n- **Doubly-Robust Fixed-Teacher:** Variant of DR-ST [4] where the model $g(x)$ is also trained using the same loss function as PPI++ but has a heuristic for choosing the value of λ. Instead of a fixed value (as in PPI and PPI++), λ is set to 0 throughout training and switch it to 1 only during the final epoch as done in [4]. \n - **PP-SSL (Ours):** A model $g(X)$ based on the PPI++ loss but with an adaptive, online learning of λ. That is, the tunable parameter $\\lambda_t$ is rigorously updated throughout the optimization steps, following our theory and the corresponding algorithm we proposed. The loss for step t becomes $ \\mathcal{L}\\_{Labeled}(g(X),y) + \\lambda_t \\cdot (\\mathcal{L}\\_{Pseudo}(g(\\tilde{X}),f(\\tilde{X})) - \\mathcal{L}\\_{Pseudo}(g(X),f(X)))$. \n- **OnlyLabeled:**  Model $g(x)$ trained only on the labeled data, for example, using cross-entropy loss (classification) or squared loss (regression). This is an important baseline as we want to show that using unlabeled data outperforms the OnlyLabeled model. \n## C2:  What is $\\lambda\\_{PP}^*$, seeing that your gradients depend on the current iterate, unlike in PPI++ where there is a single distribution? \n\nAs detailed above, $\\lambda\\_{PP}^*$ is a fixed λ, derived in PPI++ [3], which minimizes the asymptotic variance of the parameters of a linear model (more details are in [3, Section 6.1, Proposition 2]). In contrast, our theoretically-grounded method derives (i) the optimal λ for gradient variance minimization; and (ii) an online learning approach that automatically balances labeled and pseudo-labeled data contributions without requiring prior knowledge of variance parameters. Such an adaptive adjustment of λ does not exist in prior work. \n## C3: Comparison to state-of-the-art baseline methods.\nWe emphasize that our approach is orthogonal to many common SSL techniques, and they can be used in conjunction with our method. The goal of the paper is not to provide a SOTA SSL method; rather, we provide a theoretically-grounded framework for unbiased SSL.\n\nWith that said, SOTA SSL methods can be modified to use our proposed unbiased loss, which provides a safeguard against undesired bias for minority groups for which teacher pseudo labels are less accurate, as explained below.\n \nMost state-of-the-art semi-supervised learning losses that use pseudo labeling can be written in a format of $\\mathcal{L} = \\mathcal{L}\\_{Labeled}(g(X),y) + \\lambda\\cdot\\mathcal{L}\\_{Pseudo}(g(\\tilde{X}),f(\\tilde{X}))$. Therefore, to use both the SOTA SSL loss and our PP-SSL for unbiasedness, one can just update the loss to the form of: $\\mathcal{L}\\_{PP-SSL} = \\mathcal{L}\\_{Labeled}(g(X),y) + \\lambda\\cdot(\\mathcal{L}\\_{Pseudo}(g(\\tilde{X}),f(\\tilde{X})) - \\mathcal{L}\\_{Pseudo}(g(X),f(X)) )$.\n\nIn our new experiment of CIFAR-10 classification (see Reviewer sxXE (C4)), we used the exact mechanism of combining SOTA SSL loss and our PP-SSL method. In that case, the pseudo loss, which is the KL divergence loss, is more complex than the standard loss functions used in previous experiments provided in the paper. \n## C4: Why was there no comparison to Zhu et al., since that paper operates in the same setting?\nDR-ST [4] operates in a different setting; they keep updating the teacher as the training progresses, while we assume the teacher is fixed. Note that (1) the setting we consider is just as important as the self-training one, and (2) it is unclear how to provide in-depth theory for self-training for the reasons we share with reviewer sxXE, see our reply C1 (we target for theoretically-grounded algorithm for SSL).\n\nIndeed, DR-ST [4] do not provide any theory that justifies their choice for the update of λ, which they set to be 0 for all epochs and 1 for the last epoch. Also, they do not provide theoretical analysis for cases where the teacher is continuously updated.\nFollowing your question, we added an experiment with DR-FT variant of DR-ST where the teacher is fixed as done in all other baselines, (see Reviewer sxXE(C4)).  Using DR-ST heuristic of choosing λ. The results of this experiment show that DR-FT is comparable to PPI++ method, and our PP-SSL achieves higher accuracy (both overall and per group). \n## C5: Technical questions\n> Factor of 4: \n\nYes, by being asymptotically equivalent we mean the same dependence on n (and $\\sigma^2$). In other words, it can be written as $\\mathcal{O}(\\sigma^2/n)$. The 4 factor comes from the inequality: $\\lVert a+b+c+d\\rVert^2 \\leq 4\\lVert a\\rVert^2 + 4\\lVert b\\rVert^2 + 4\\lVert c\\rVert^2 + 4\\lVert d\\rVert^2$, which is used to prove Lemma 3.2 in Appendix B.1. \n> Correctness and bias:\n\n By “correctness” we mean unbiasedness; we will revise the text to avoid any ambiguity. We care about unbiasedness because the loss in Eq. (4) is an empirical loss defined over a **finite sample** and our goal is to provide convergence guarantees for the **expected loss** over the distribution, as defined between lines 110 and 111. When optimizing with a biased gradient estimate:\nEmpirically, we get worse performance where the teacher’s labels are incorrect (see the Standard-SSL results in our experiments, Section 4 and Appendices D,E,F.), and\nTheoretically, we cannot guarantee convergence to a stationary point, but only to its neighborhood, whose size depends on the bias (see Theorem 4 in [1]). In other words, the Standard-SSL baseline performance is highly affected by the teacher’s errors.\n> Using AdaGrad: \n\nAs we mentioned in lines 223-227, our theoretical analysis relies on AdaGrad for both the model weights $w$ and the tuning parameter λ to properly adapt to $\\sigma^2$ and $\\sigma_e^2$, which we do not know in advance. AdaGrad provides a clear and simple second-order regret bound (Lemma 3.4), and it was commonly analyzed in previous work [2].\nImportantly, these second-order regret bounds (i.e., where regret depends on the sum of square gradients rather than their bounds) is crucial for our analysis. Nevertheless, In practice, any stochastic optimization method for $w$ and online learning method for λ can be used, such as SGD and Adam, which we also employ in our experiments. We will clarify this point in the text. \n## References:\n[1] On the Convergence of SGD with Biased Gradients. Ajalloeian et al., 2020. \n\n[2] AdaGrad stepsizes: Sharp convergence over nonconvex landscapes. Ward et al., 2018.\n\n[3] PPI++: Efficient prediction-powered inference. Angelopoulos et al., 2023.\n\n[4] Doubly-Robust Self-Training. Zhu et al., 2024"}, {"review_id": "RbPQgTQSW8", "reviewer": "Reviewer_Gh11", "summary": "This paper proposes Prediction-Powered Semi-Supervised Learning (PP-SSL), which is a fresh direction for dealing with the classic challenge in semi-supervised learning: how to get the most out of many unlabeled samples when the labeled ones are scarce and expensive. Traditional approaches, for example simple pseudo-labeling or teacher-student frameworks, often run into trouble when the teacher model makes mistakes—especially for minority groups. Such errors in pseudo-labels can bias the learning process, causing the trained model to perform worse than one would expect, or even worse than using only the labeled data.\n\nThe authors build on the Prediction-Powered Inference (PPI) principle, where pseudo-labels are used both for labeled and unlabeled data in a way that the expected bias cancels out. In earlier work, the weighting between real and pseudo-labels—this λ parameter—was set offline, based on quantities like the variance of the labeled data and the accuracy of the teacher model. But these are usually unknown in practice.\n\nWhat makes PP-SSL special is the automatic, online adjustment of λ during training, using the AdaGrad algorithm. With this, the method finds a good balance between using real labels and pseudo-labels on the fly, without needing prior knowledge or manual tuning. The paper provides a theoretical analysis, showing that the new gradient estimator is unbiased and can achieve lower variance than the standard supervised gradients, leading to faster and more stable training, especially as the accuracy of the teacher improves or more unlabeled data becomes available.\n\nThe authors give thorough experiments, both on synthetic datasets (with carefully designed group imbalances) and on real-world datasets like California Housing and UTKFace for age estimation. The results show that PP-SSL consistently matches or outperforms standard SSL and previous PPI-based methods, especially in difficult cases where the teacher model is weak for certain subgroups. The approach also proves more robust in situations with distribution shifts between groups.\n\nFinally, the paper openly discusses limitations, such as the assumption that labeled and unlabeled samples are from the same distribution, and the current focus on fixed (not self-updating) teacher models. There is also attention to broader societal impacts, including fairness and robustness, and the need for further work to handle distributional shifts or to apply the method to more task types.\n\nIn summary, PP-SSL gives a principled, practical improvement for semi-supervised learning, making better use of unlabeled data while protecting against the classic pitfall of pseudo-label bias, especially in real-world, unbalanced datasets.", "questions": "1. The authors briefly mention self-training as an alternative to the teacher-student approach in SSL. Given that they propose an *online* learning approach within the teacher-student paradigm, onw wonders whether they can transfer their ideas to self-training (which is online by conceptualizatoin)? In particular, could you use the same model for both labeled and pseudo-labeled data, but iteratively update pseudo-labeled data only, while dynamically tuning λ as in PP-SSL to continuously adapt how much weight to give to pseudo-labels. In this case, could some of the convergence results be translated to self-training?\n\n2. \"A key limitation of our method, shared by many SSL methods, is the assumption that labeled and unlabeled data are drawn from the same distribution. ..... Extending our framework to handle such shifts is a natural direction for future work.\" Can the authros share some ideas of how thery are aiming to do that in the PP-SSL framework? https://www.jmlr.org/papers/v23/22-0541.html or (more general) https://arxiv.org/pdf/2505.07367 might be starting points for the iterative/online case.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The method reduces bias from bad pseudo-labels by tuning the λ parameter automatically during training, so you do not have to guess the right value beforehand. It works well even if the teacher model is not perfect, especially for smaller groups that often get ignored. The approach is backed by clear theoretical results, and also shows strong improvements in practical experiments with real and synthetic data. The algorithm is simple to use with existing optimizers like AdaGrad, so implementation is straightforward.\n\nThe main weakness, as the auhtors themselves note, is that the method assumes the labeled and unlabeled data are coming from the same distribution. As a minor weakness imo, most experiments are in regression settings. Considering classification tasks too would make the results stronger.", "was_revised": true, "final_justification": "see reply to rebuttal", "rebuttal": "Thank you for your thoughtful and constructive review. We appreciate your detailed summary and recognition of both the theoretical contributions and practical strengths of our work. Your comments about the novelty of dynamically tuning λ, the robustness to teacher bias, and the simplicity of integrating PP-SSL into existing optimization pipelines are especially encouraging.\n\nSeveral of your questions overlap with those raised by other reviewers (e.g., Reviewer sxXE), and for convenience, we restate and adapt the relevant parts of our responses here, while providing additional clarifications specific to your suggestions.\n\n## C1: Could the method be extended to self-training?\nThis insightful question was also raised by Reviewer sxXE (C1), and we summarize our response here for your convenience.\n\nIn principle, our method can indeed be applied in self-training settings, where the same model is used to generate pseudo-labels. However, this introduces theoretical challenges: the pseudo-label generator is no longer fixed and becomes dependent on the training data and dynamics. As a result, the independence assumptions in our theoretical analysis no longer hold.\n\nNevertheless, if the teacher is updated using data disjoint from the training set (e.g., a held-out validation set), then the PP-SSL framework and its guarantees can still be extended. In this case, $\\sigma_e^2$ can be replaced with a worst-case bound across teacher updates. A more refined treatment would require analyzing a time-varying $\\lambda_t^*$ corresponding to each teacher, shifting from static to dynamic regret analysis. This is an exciting direction for future research, and our work provides a principled foundation towards this goal.\n\n## C2 : How can the method handle distribution shift between labeled and unlabeled data?\nThis important point aligns with our discussion of limitations in the paper and was also mentioned by Reviewer sxXE (C5).\n\nWhile PP-SSL currently assumes a shared distribution between labeled and unlabeled data, we believe the framework can be extended. For example, as done in PPI and Doubly-robust self-training (Zhu et al., 2023), importance weighting can be introduced to correct for covariate shift. This involves estimating the density ratio between labeled and unlabeled covariates and adjusting the loss accordingly. In our context, this could also be done by reweighting the gradient estimator used for λ adaptation, or by modifying the loss function itself to account for distribution mismatch.\n\nExtending our work to handle distribution shifts unlocks the opportunity to use generative models to produce large amounts of unlabeled data from a different distribution. While this may increase the quantity and diversity of available data, it introduces shifts that go beyond simple covariate differences. In such cases, standard importance weighting may not be sufficient to correct for the resulting shift. Addressing this setting would likely require adapting both the pseudo-labeling mechanism and the gradient-based weighting strategy in PP-SSL to account for more complex forms of distribution shift. \n\nWe appreciate the suggestion to consider iterative or online extensions of our PP-SSL framework. As the reviewer notes, iterative SSL methods such as the one in [1] offer a natural way to incorporate distributional shifts by sequentially adapting the model on new chunks of unlabeled data. We believe this direction can be complementary to our current work. One potential extension is designing a double-loop algorithm inspired by [1]: the unlabeled data is split into $K$ chunks; for each outer iteration $k = 0,\\ldots,K$, we would run our PP-SSL algorithm using the labeled data and the $k$-th chunk of unlabeled data (with a suitable teacher); the resulting model is then used as a teacher to label the next chunk. This setup introduces a time-varying error variance $\\sigma\\_{e,t}$, which presents new challenges for analysis, particularly in tracking how the optimization dynamics evolve over outer iterations. Extending our convergence results to this setting is a promising direction for future work.\n\nWe also reviewed the more recent reciprocal learning framework [2], which introduces an alternating scheme between sample adaptation and ERM updates. However, both [1] and [2] abstract away the optimization process, assuming access to an exact ERM solution or working under strong convexity and Lipschitz assumptions. In contrast, our work explicitly models the training dynamics and optimization errors in non-convex settings, which we believe is crucial for practical SSL scenarios involving deep models.\n\nWe agree that incorporating ideas from these works into our framework—while preserving our focus on optimization dynamics—would be a meaningful direction. In future work, we plan to explore how concepts like sample adaptation and staged learning can be integrated into PP-SSL in a way that remains faithful to the stochastic, non-convex nature of modern training procedures.\n\nWe will elaborate on these ideas in the final version of the paper’s discussion section.\n\n## C3: Considering classification tasks would strengthen the results.\n This suggestion is shared with Reviewers sxXE (C4) and yHkN, and we thank you all for encouraging this extension. \nAs part of our ongoing experiments, we applied PP-SSL to a semi-supervised classification task using CIFAR-10 with class-conditional corruption (based on CIFAR-10-C). Our method consistently outperformed strong baselines (SSL, PPI++, DR-ST) in total accuracy and in the corrupted subset (Group A), while maintaining comparable performance on clean classes (Group B). The performance gap on Group A depends on the corruption type and aligns with our theory regarding the pseudo-label error $\\sigma_e^2$. We conducted additional tests across multiple corruption types and severity levels and observed the same trends, though we present only severity 5 in the main paper for clarity.\n\n## C4: Reference formatting\n\nThank you for catching this. We will correct the missing publisher/journal information for Reference [2] in the camera-ready version.\n\nOnce again, thank you for your positive evaluation and for highlighting both the novelty and potential impact of our method. Your feedback has helped us clarify our contributions and better communicate the broader applicability of PP-SSL.\n\n## References:\n[1] Information-Theoretic Characterization of the Generalization Error for Iterative Semi-Supervised Learning. He et al., JMLR, 2022. \n\n[2] Generalization Bounds and Stopping Rules for Learning with Self-Selected Data.  Rodemann et al.,  2025."}]

Author rebuttal:

Meta-review: This paper introduces PP-SSL, which brings the prediction-powered inference (PPI) debiasing principle into semi-supervised training by using an unbiased gradient estimator combined with online tuning of the interpolation weight lambda (via an AdaGrad-style update), yielding finite-time convergence and variance-reduction guarantees. Main commendations include principled extension of PPI to SSL training with theory (sxXE, Gh11, and yHkN), the idea of adaptive online tuning of lambda is novel and sensible (yHkN and Gh11), empirical strength and added image experiments (sxXE and Gh11, post-rebuttal). Main concerns include assumption on distributions (sxXE and Gh11). Overall, I think this is a nice paper that deserves to be accepted.
Decision: Accept (poster)