Semi-Supervised Regression with Heteroscedastic Pseudo-Labels
Authors: Xueqing Sun
Renzhen Wang
Quanziang Wang
Yichen Wu
Xixi Jia
Deyu Meng
Venue: NeurIPS 2025 poster ()
Keywords: Semi-supervised regression
Pseudo-labels
Heteroscedasticity
Primary area: general_machine_learning
TL;DR: We propose an uncertainty-aware pseudo-labeling framework that dynamically adjusts pseudo-label influence from a bi-level optimization perspective.

Abstract: Pseudo-labeling is a commonly used paradigm in semi-supervised learning, yet its application to semi-supervised regression (SSR) remains relatively under-explored. Unlike classification, where pseudo-labels are discrete and confidence-based filtering is effective, SSR involves continuous outputs with heteroscedastic noise, making it challenging to assess pseudo-label reliability. As a result, naive pseudo-labeling can lead to error accumulation and overfitting to incorrect labels. To address this, we propose an uncertainty-aware pseudo-labeling framework that dynamically adjusts pseudo-label influence from a bi-level optimization perspective. By jointly minimizing empirical risk over all data and optimizing uncertainty estimates to enhance generalization on labeled data, our method effectively mitigates the impact of unreliable pseudo-labels. We provide theoretical insights and extensive experiments to validate our approach across various benchmark SSR datasets, and the results demonstrate superior robustness and performance compared to existing methods.

Semi-Supervised Regression with Heteroscedastic Pseudo-Labels
Xueqing Sun', Renzhen Wang'; Quanziang Wang', Yichen Wu Xixi Jia', Deyu Mengl.
Xi' an Jiaotong University City University of Hong Kong Harvard University Xidian University Pazhou Laboratory (Huangpu)
xqsunestu.c xjtu, edu. en, frzwang, (ewwang,ymenng.a.. Exjeenn)exjuuu... edu. (quanziangwang, wuyichen am97, hsijiaxidian] 0gmaaii.aaammmmm.mmommmmmm. com
Abstract
Pseudo-labeling is commonly used paradigm in semi-supervised learning, yet its application to semi-supervised regression (SSR) remains relatively under-explored. Unlike classification,, where pseudo-labels are discrete and confidence- fil- tering is effective, SSR involves continuous outputs with heteroscedastic noise, making it challenging to assess pseudo-label reliability. As result, naive pseudo- labeling can lead to error accumulation and overfitting to incorrect labels. To address this, propose uncertainty-awve aware pseudo-labeling framework that dy- namically adjusts pseudo-label influence from a bi-level optimization perspective. By jointly minimizing empirical risk over all data and optimizing uncertainty esti- mates to enhance generalization on labeled data, our method effectively mitigates the impact of unreliable pseudo-labels. We provide theoretical insights and exten- sive experiments to validate our approach across various benchmark SSR datasets, and the results demonstrate superior robustness and performance compared existing methods. Our code is available at ///pg://gghhbbbooomss//eeecccaae Pseudo-Labels:
1 Introduction
Deep learning has achieved superior performance on various scenarios, particularly when large amounts of labeled data are available (28]. However, acquiring large-scale datasets with accurate labels is often costly impractical in many real-world scenarios, such medical imaging where labeled data typically scarce and expensive or video analysis that requires labor-intensive frame-by- frame annotations [10 (1]. To address this challenge, Semi-Supervise
Learning (SSL) has emerged as potent paradigm that leverages vast amounts of unlabeled data to enhance learning performance, thereby reducing the dependency expensive labeled annotations 44 52].
and consistency regularization 40 43 I 49 have been widely adopted, achieving impressive results across various domains. However, supervuped regression (SSR) presents unique challenges distinct from its classification counterpart. Unlike classification, where pseudo-labels are discrete and can be sharpened to encourage high-confidence predictions, regression inherently deals with continuous outputs, making it difficult to establish reliable pseudo-labeling mechanism [18). Furthermore, the lack of well-defined decision boundaries in regression complicates uncertainty quantification. increasing the risk of propagating incorrect pseudo-labels.
In classification tasks, SSL techniques such as pseudo-labeling 29
Due these challenges, the pseudo-labeling methods commonly employed in semi-supervised classification are difficult to directly extend to semi-supervised regression due to the continuous
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

250
Sample Count
Predicted Age
50
200
150
40
100
30
50
0
21
26
31
36
41
46
51
56
21
26
31
36
41
46
51
56"
Age
Age
Figure 1: Experiment of UCVME [10] on UTKPaace [58] with 10% labeled data. Left: Histogram of
true labels for the unlaheled data. Right: Box-plot of pseudo-labels generated by UCVME.
nature of regression outputs. Consequently, existing SSR approaches focus primarily on leveraging consistency regularization to improve the utilization of unlabeled data. For example, TNNR [47] introduces a loop consistency that enforces consistency on the prediction differences between pairs of inputs. UCVME [O] proposes a novel uncertainty consistency loss for co-trained models to better account for model uncertainty. RankUp
[8] reformulates the regression task as ranking problem and imposes consistency of the rank between sample pairs.
These methods aim to enforce smoothness between predictions for labeled and unlabeled data, ensuring that the model learns coherent mapping from input features continuous outputs. However, the reliance on consistency regularization alone does not fully address the difficulties in handling uncertainty and the risk of overfitting to potentially incorrect pseudo-labels. To investigate this, we visualize the distribution of estimated pseudo-labels predicted by UCVME on the UTKFace dataset [88 with 10% labeled data. As shown in Fig. [ despite the application of uncertainty consistency constraints, UCVME shows significant variance in aseudo-labels (right subfigure), even for age groups with large sample sizes (left subfigure). To address the aforementioned challenge, this paper introduces an uncertainty-awee aware lauul-ig method that dynamically adjusts the influence of pseudo-labelee samples based on calibrated uncertainty.
Our core motivation is that the uncertainty or noise in pseudo-labels varies with input features, leading to heteroscedasticity Therefore, it is crucial to develop an effective stralegy for assigning appropriate uncertainty values to psendo-labels, reflecting their varying degrees of error during training. To achieve this, we propose a bi-level learning framework that not only minimizes empirical risk over both labeled and unlabeled data but also explicitly optimizes uncertainty estimates to improve generalization on labeled data during training. We theoretically and experimentally demonstrate that our method significantly enhances pseudo-label robustness and overall performance across various benchmark SSR tasks. In summary, our main contributions are: (1) We propose a bi-level SSR framework that explicitly optimizes the uncertainty of pseudo-labeled samples to improve the regression model's generalization when learning from potentially incorrect pseudo-labels. (2) We offer theoretical analysis of the proposed method through gradient alignment, providing insights into why it enables the regression model to effectively mitigate the risk of overfitting to incorrect pseudo-labels. (3) We empirically validate approach on various benchmark datasets and SSR settings, demonstrating its ability to improve both performance and robustness in SSR tasks.
2 Related work
2.1 SemiiSupervised Learning
Semi-Superrised Learning (SSL) [44 (2) trains models with both labeled and unlabeled data, which significantly reduces the cost of data annotation in real-world applications. SSL methods have been applied on wild scenarios, such image classification [3 2), segmentation [59 6 [6]. and other tasks [54] 51 33.. Current works on SSL can be categorized as follows: 1) exeudo-labellin 46 I7 train the model on both labeled data and unlabeled data methods: These models 40 43 2 49 apply prediction invariance loss on unlabeled data across perturbations. 3) 30 aim to model the real data distribution from the labeled and unlabeled data and generate new samples.
methods: These works 29 50
57
with pseudo- labels,, which are generated by the model itself. 2) Consistency-basee
Generative methods: These methods

---

2.2 Semi-Supervised Regression
Semi-Supervised Regression (SSR), has attracted significant research interest in recent years, which includes 1) Consistency-based methods: Similar to the aforementioned methods in SSL., these methods designed different constraints to ensure prediction consistency in SSR. On the one hand, methods developed for semi-supervised classification can be readily extended to SSR, such as Mean Teacher [3] and Temporal Ensembling [26]. On the other hand, methods specifically designed for SSR have emerged. For example, TNNR [47] enforces loop consistency on prediction differences, UCVME [10] introduces uncertainty consistency loss for co-trained models, and RankUp [18] enforces rank consistency between sample pairs. Additionally, CLSS [ employs contrastive learning to encourage sample features with the same labels to be closer. 2) Uicertainty-basee methods: In SSR, uncertainty estimation provides a potential way to improve the quality of pseudo-labels for unlabeled data. Specifically, SSDKL. (20] proposes minimize the prediction variance of the posterior regularization, and Sitt [21 employs dropout to compute multiple predictions and use their variance quantify the uncertainty of pseudo-labels. In related manner, PIM 7 performs weakly supervised regression by weighting candidate labels according to their incurred losses, giving higher weights to labels with smaller losses. Unlike these methods, we propose an uncertainty-aww aware aseudo-labeling method from a bi-level optimization perspective in this work.
2.3 Uncertainty Estimation
Uncertainty estimation has been explored extensively in fully-supervised learning 27 22 19.. From Bayesian viewpoint, Gal and Ghahramani [13] reinterpret Monte Carlo Dropout as variational inference for epistemic uncertainty estimation, which has been extended to semi-supervised classifica- tion methods (36] and semi-supervised segmentation tasks [56 48 to filter unreliable pseudo-labels.
Additionally, some works [55] 31 quantified uncertainty through prediction discrepancies between co-trained models. In the SSR task, UCVME [TO], SSDKL [20]. and SimRegMatch [21] have em- ployed the uncertainty estimation methodology to improve the accuracy of pseudo-labels. Concretely, UCVME models pscudo-label noise under the heteroscedastic assumption, using shared encoder with dual heads to predict both the mean and variance of the noise distribution. SSDKL combines neural network encoder with Gaussian Process in the latent space, enabling posterior inference to obtain predictive variance for unlabeled samples. SimRegMatch estimates the uncertainty of pseudo-label by performing multiple forward passes with dropout, and selects lower uncertainty pseudo-labels through a thresholding strategy. Different from these methods, we propose a bi-level optimization framework and estimate pseudo-label uncertainty guided by labeled samples.
3 Method
3.1 Notations and Problem Setups
Semi-supervised regression (SSR) involves a labeled dataset Dj = [x), 1/ and an unlabeled dataset D,, { where of E X and z e X' represent training examples from the input space X. and y, E R denotes the label associated with the labeled example a/. The objective of SSR is to train model fo that generalizes well on the test dataset.
In SSR, widely used strategy involves employing pseudo-labeling techniques to augment the training dataset by assigning pseudo-labels to unlabeled data. In this strategy, each unlabeled example is assigned a pscudo-label based on the predictions of the model itself or its variants. The model is then trained on both labeled and unlabeled examples by optimizing objective function L = Li + ACu- The loss function consists of two terms: the supervised loss L calculated on the labeled data, and the unsupervised loss L, calculated on the pseudo-labeles data. The contributions between the supervised and unsupervised losses.
À balances the
One of the key challenges in aseudo-labeling based SSR is to effectively leverage pseudo-labeled data while mitigating the negative impact of incorrect pseudo-labels on model training. In this work, we propose bi-level learning framework, which explicitly learns pseudo-label uncertainty to enhance the reliability and utility of pseudo-labeled data. By modeling the uncertainty associated with pseudo- labels, the method enables more robust training in SSR settings. In the following section, first
3

---

(vi - fo (x()))
x
f8
24
exp(2,)
- (o((5())))
jj
g6
($)
copy
Compute L iiner
x
(e)())
(yx (orc
Update 0 (T)
Compute L outer
Update Φ
Figure 2: Method Overview. The proposed bi-level optimization framework consists of main
steps: (1) Inner-loop update, which updates the regression model using _ as defined in eq. (5). where 2j log o²: (2) Outer-loop update, which updates the uncertainty-learne using Louter as
defined in eq. 6 Note that we assume batch size of I for better visualization.
introduce a probabilistic formulation to model the uncertainty in pseudo-labeling. We then describe how to effectively learn and incorporate pscudo-label uncertainty within our bi-level framework.
3.2 Heteroscedastic Pseudo-Labels
The objective of SSR is to learn the true label function fa- (x) using both labeled and unlabeled data. However, when applying pseudo-labeling are usually imperfect. Moreover, the level of uncertainty or noise in these pseudo-labels varies across samples, particularly when the label estimation process involves an external source, such as weak supervision, self- soff-training, or noisy teacher model.
SSR, the pseudo-labels assigned the unlabeled data
To model this relationship, treat the sseudo-labels for each sample heteroscedastic. Especially, for each unlabeled sample x. and its corresponding pseudo-label Új. we assume that the variance of each pseudo-label depends on the input sample itself [3] This can be formulated as:
fo(a)) Cj Cj ~~~N(0,07
where Sa(a") is the true label prediction for input æ",, and is a noise term whose variance of varies with the input s... In maximum likelihood inference, we can write the negative log-likelihood (NLL) of the observation variable úū as:
logg(ús x3)) x (ūs So(x)))² log(o)))
of
As such, for batch of unlabeled data B,, we can define the unsupervised loss as the sum of the negative log-likellie liceliidd over all these samples:
L.
Σ 25 EB.
7 (ū: So(x))))
Σ log(o)).
Notably, if all pseudo-labels are assumed be homoscedastic and fixed at the above loss degrades to the standard mean squared error (MSE) loss, which is commonly employed in various SSR methods.
In eq. (3), when an incorrect pseudo-label ūز is assigned to x during training, the model can mitigate the issue of error accumulation in by increasing the uncertainty of the pseudo-label, i.e., adjusting oj to downweight its contribution to the overall loss. In contrast, the standard MSE loss enforces rigid fitting of incorrect pseudo-labels, thereby exacerbating confirmation bias training progresses. Thus, a key challenge is to devise an effective strategy for dynamically assigning appropriate uncertainty values pseudo-labels with varying degrees of error during training.
3.3 Learning Uncertainty via Bi-level Optimization
To dynamically assign uncertainty values to pseudo-labels, a natural strategy is to introduce an auxiliary network go parameterized by o to produce o, for each unlabeled sample xy.. and then

---

jointly optimize the parameters {0, Φ} of fa and in an end-to-end manner. However, this strategy suffers from a fundamental limitation, fails to distinguish between two distinct scenarios:
Hard but correct samples: The pseudo-label D, is correct, but the model prediction fo(z)) is inaccurate, leading to large squared error (ūj - [o(())))
Easy but incorrect samples: The pseudo- ū is incorrect, but the model prediction fo (zz) is already close the true label, also resulting in a large squared error.
The two scenarios result in large squared error in the numerator of the first term in eq. (3) leading to large of during minimization. However, the increased of would suppress the learning for hard but correct samples, which is crucial to improve the performance of SSR models.
Ideally, we seek a formulation where oj selectively suppresses unreliable pseudo-labels while still enforcing learning on difficult but valid samples. To this end, we propose a novel bi-level optimization framework that introduces a supportive model, referred as the uncertainty- uncertainty associated with pseudo-labels. Specifically, we parameterize the uncertainty-leamee as lightweight network Io that maps each pseudo-labeled sample z"; (or certain conditional information derived from the regression model, as elaborated in the next subsection) to its corresponding log- variance. This mapping can be formally expressed as:
to learn the
2j log of 96 (2)),
where & denotes the parameters of ge. Note that we predict the log-variance rather than the variance directly, as this approach enhances numerical stability. Directly predicting of can lead instability due to potential division by zero in the loss function 223 22).
Our proposed bi-level optimization framework, which jointly optimizes the regression network fo and uncertainty-learne Jg. is illustrated in Fig. 2 and can be described in the following two steps:
Inner-loop for optimizing fe: The regression network fo is trained on both labeled and pseudo- labeled data by considering the uncertainty of pseudo-labels as follows:
0º (3) arg min LainnCiaa Lianer == (r( + (0,
(5)
where (1) = (y:- - 2z()))) represents the supervised loss computed over a batch of labeled data, and L. (0, () EE exp(2;) (ū)-fo(x;)) )) < 2j denotes the unsupervised loss computed over a batch of pseudo-labeled data, with 2j obtained from eq. (4). The hyper-
parameter À controls the trade-off between La and Lu. In eq. ( á essentially acts as hyper- parameters for 6, and in this step, we just express 0* as a function of Φ.
Outer-loop for optimizing I,: Our ultimate objective is to generate well-calibrated uncertainty estimates that protect the regression model from incorrect pseudo-labels while ensuring good generalization on labeled data. To achieve this, we optimize & by continuously tracking the performance of the updated model (e-(0) (0) on labeled data, which can be formulated as:
arg min Couter
Σ fo- c
(6)
where B, is another batch of labeled data from D; that differs from Bi used in eq. i... Br + B,. This formulation aims to find the optimal hyper-parameter Ø² such that the regression model, obtained by optimizing eq. (5) should also have good performance over any unbiased and reliable training batch.
In fact, eq. (5) and (6) formulate a bi-level learning framework. In the inner loop eq. 5). the regression model fe is updated using both labeled data and pseudo-labeled data, with the latter calibrated based on the uncertainty estimates generated by gø- In the outer loop eq. (6), the parameters of go optimized to ensure that the refined model fe-(0) achieves more reliable and robust performance.
3.4 Practical Implementations
Uncertainty-learner'' architecture. In practice, the uncertainty-learrer de does not directly map each unlabeled sample to its log-variance 2j. as presented in eq. (4). Instead, to improve computational
5

---

efficiency, it takes the prediction of the regres- sion model rj = Fo (x") and the correspond- ing pseudo-label fú as its input. This mapping
Algorithm 1 Mini-batch Training Algorithm of the
Method
can be formally formulated as: 2, = 90 1/))
1: Input: labeled / unlabeled data Dj / Du, labeled /
unlabeled batch size n/m, max iterations T
The architecture of 96 is implemented as multi- layer perceptron (MLP) with one single hidden layer. Despite its simplicity, according to the uni- versal approximation theorem, such a network is theoretically capable of approximating any continuous function defined on a compact set with arbitrary precision [4]. Note that although
2: Output: regression network parameter 8"
3: Initialize the parameters 0 of the regression net-
work and those of the uncertainty- to T do
4: for
5:
B. = {( V/ j==1 t- GetBatch (D n).
6:
B. {a fj=1 + GetBatch (D., m).
7:
B. { vie M==1 t GetBatchi (i, m).
both and ûj are derived from the regression
8:
Compute inner loss Linner by eq.
model, they are strictly identical in this work.
9:
Update regression network 0 by eq.
This discrepancy arises from the use of different 10:
Compute outer loss L by eq. 6
random augmentations during the training pro-
11: Update umcertainty-leamer by Φ by eq.
cess. Inspired by FixMatch [41], we generate
12:
end for
the pseudo-label 0j from a weakly augmented version of o.y, while rj is obtained from a strongly augmented version of the same input.
Training Algorithm. The objective function of the regression network fe in eq. (5) and that of the uncertainty-learner 96 in eq. (6) formulate a bi-level optimization problem, where the solution of 8"()) and 6* are nested with each other. Approximately, update 0 and ø in gradient-based
method following 32 12 35
(1) Update 0. At iteration f, we fix the eecertainty-eerree parameter o' and conduct one-step gradient descent w.r.t regression network parameter 0 in the inner optimization problem eq. 5 as follows:
( 6 VoLinner (0', ,d'),
where a: > 0 is the learning rate of the regression network.
(2) Update ø. Based on the updated regression network parameter 0++1 (d), we optimize o' in the outer optimization problem eq. (6) by gradient descent:
0+ V Couter po (0+)))9))
(8)
where B > 0 is the learning rate of the mucertaintyleaame
Since CoLoute in eq. (8) introduces second-order derivative, it requires unrolling the second- order differentiation of the entire regression network. This process is computationally expensive and inefficient for deep models. To alleviate this computational burden, we introduce an efficient approximation algorithm. Specifically, we assume that Φ is only correlated with the parameters of the regression head (i.e., a single fully connected layer) of fo- This allows second-order derivation of the regression head
only unroll the in eq. (8). Since the regression head contains far fewer parameters than the entire regression network, our proposed algorithm is considerably more efficient than conventional bi-level optimization algorithms
32 12 35).
3.5 Theoretical Analysis
We herein theoretically analyze the proposed bi-level optimization framework and reveal how method helps the regression model mitigate the risk of overfitting to incorrect -seudo-labels.
Theorem Let VaLinner (8, Φ) and Lo (8) be the gradients of the inner and outer loss w.r.t. 6, respectively. The bi-level optimization in eq. (5) and eq. (6) is equivalent to the following optimization problem:
min (VoLinner (0, () V Colane (0))
The proof is presented in Appendix A.I This theorem demonstrates that optimizing the parameter in our bi-level framework fundamentally aligns the gradients of the inner and outer losses w.r.t. 6. Based on eq. (5) and eq. (6). the gradient-matching formulation in eq. ( suggests that our method implicitly regularizes the gradients from labeled and unlabeled data under the guidance of the outer data, and the discrepancy between these gradients modulated by á through its influence on the inner loss. Specifically, inaccurate uncertainty estimated by 96 would interfere with the inner loss

---

Table I: Main results on FTKeaace The performance is reported in the form of 'mean + std' across six random runs, Bold means the best results and our method are shown in gray cells
= 5%
7 10%
7= 20%
Method
MAE J
R²1
MAE L
R²T
MAE 4
R² 1
Fully-Supervised
4,713 = 0.039
0.652 = 0.006
4,713 = 0.039
0.652 0.006
4.713 + 0.039
0.652 0.006
Supervised
6.135 + 0.046
0.454 + 0.008
5.618 + 0.026
0.533
0.005
5.278 4: 0.048
0.581 0.005
Mean Teacher
F8]
5.912 =: 0.071
0,476 + 0.011
5.474 0.052
0.537 + 0.009
5.126 : 0.040
0.384 0.007
Temporal Ensembhing
26
5,963 1: 0.060
0.478 1: 0.008
5,437 1: 0.022
0.557
0.001
5.123 1 0.032
0.604 0.002
SSDKI. 20
5.855 +: 0.080
0,469 +: 0.010
5.391 +: 0.040
0,551
0.007
5.173 4: 0.039
0.589 0.007
TNNR (47
5,817 07 0.057
0.505 +: 0.008
5,372 + 0.052
0.563 + 0.007
5,104 + 0.025
0.602 + 0.005
MimcegMatch
5.794 = 0.073
0,512
5.338 0.065
0.584
5.164 0.108
0.612
UCVME 10
5.862 + 0.036
0,495 0.006
5.227 1 0.045
0,585 1: 0.006
4,870 1: 0.038
0.634 1: 0.005
CLSS 9
6.063 0.145
0.018
5.621
0.077
0,514
5,429 0.078
0.546
RankUp
18
5.719 + 0.112
0.495 + 0.012
5.252 + 0.040
0.576 + 0.004
4.914 + 0.015
0.621 + 0.002
Ours
5.639 = 0.035
0.523 0.009
5.143 - 0.075
0.597 0.009
4,929 + 0.038
0.624 0.004
gradient w.r.t. 6, thus increasing the empirical risk in eq. (9) In contrast, when IS estimates accurate uncertainty in the inner loss, helps the regression model perform gradient decent in an appropriate direction consistent with the outer data, ensuring stable and effective semi-supervised learning.
4 Experiments
In this section, we experiment with three benchmarks to evaluate our method. The experimental setups are described in Section 4.1] and the main results under varying label ratios are presented in Section 4.2 Section 4.3 provides an ablation study to assess the contribution of each key component, along with further discussion to gain a deeper understanding of the proposed method.
4.1 Experimental Setups
Datasets. We evaluate the effectiveness of our algorithm on three benchmark datasets: UTKFaace (58) an image-based age estimation dataset; IMDB-WII (39], a large-scale dataset for age estimation; and STS-B [5)45], benchmark for assessing semantic similarity between sentence pairs. Please refer to Appendix B.I for more detailed information about these datasets. These datasets are commonly used in supervised and semi-supervised regression tasks [10 18 53
34] In this work, conduct experiments with varying label ratios % where 5%, 10%, and 20% of the training data are labeled, and the remaining data are used unlabeled samples.
Evaluation Metrics. For the image datasets UTKFaae and IMDB-WIKI we follow UCVME [10 in using Mean Absolute Error (MAE, J) and the coefficient of determination (It2, 1) for evaluation. For the text dataset STS-B, we adopt Mean Squared Error (MSE, 4) and Rt, following DIR (33. Note that J and indicate that lower and higher values are better, respectively. Detailed definitions of these metries are provided in Appendix B.2 All experiments are conducted six times using fixed random seeds (0 to 5), and we report the mean and standard deviation for each metric.
Training Details. We conduct all experiments within unified codebase to ensure fair comparisons. Specifically, following [10), we use ResNet-50 [15] pretrained on ImageNet as the backbone for the UTKFace and MMDB-WIKS datasets. For the STS-B dataset, we adopt a BiLSTMM
regression For more details on the implementation
model with Ge word embeddings, in line with 45 53 details, please refer to Appendix [B.3
Comparison Methods. We compare our method with various state-of-the-art SSR methods, which can be roughly divided into two categories. 1) Consistency-based methods: These methods leverage consistency constraints in model predictions, including some adapted from classification methods such as Mean Teacher [4 and Temporal Ensembling (26). as well as methods specifically designed
for SSR, including TNNR 47 UCVME [1O. CLSS 100 and RankUp [18. 2) Uncertainty-buut
methods: These approaches explicitly model uncertainty in data or predictions, including SSDKL 20 and SimRegMatch (21) Please refer to Appendix B.4 for more details about these methods.
Moreover, we construct two benchmarks to validate the effectiveness of SSR methods: 1) Supervised: A lower bound for SSR methods that is trained on the labeled data without incorporating any unlabeled data. 2) Supervsdpe An upper bound trained on all labeled and unlabeled data while assuming ground truth labels of unlabeled data are known.
7

---

Table 2: Main results HIDB-WIKL. The performance is reported in the form of 'mean + std'
across six random runs. Bold means the best results and our method are shown in gray cells
5%
7 10%
7 20%
Method
MAE J
R'
M
R'
MAE.
R²
Fully- Supey-SSpee
7.974 0.043
0,724 + 0.002
7,974 + 0.043
0.724 = 0.002
7.974 + 0.043
0.724 + 0.002
Supervised
10.172 1 0.077
0.610 1: 0.004
9.248 1: 0.052
0.657 1 0.002
8.647 1: 0.099
0.690 1: 0.005
Mcan Teacher
45]
9.492 0.081
0.647 0.002
8.633
0.689
0.002
8.191 0.066
0.711 0.002
Temporal Ensembling
2d
11.335 = 0.114
0,532 0.007
9.517 + 0.064
0.639 = 0.004
9.577 + 0.126
0.639 + 0.007
SSDKL 200
10.116 = 0.073
0.611 + 0.004
9.488 1 0.031
0,641 = 0.002
9.056 = 0.043
0.656 0.003
TNNR 47
10,069 1. 0.088
0.612.10.00
9.309 1 0.052
0.654 1 0.003
8.640 1 0.033
0.688 0.001
SimRegMMath
21
9.908 + 0.097
0.628 + 0.004
9.110 0.16
0.665 + 0.007
8.587 + 0.094
0.693 0.006
UCVME [0
9.730 0.156
0.007
8,920
0.039
0.673 = 0.004
8,309 0.117
0.698 0.003
CS 9
9.906 0.058
0.621 + 0.007
9.251
+: 0.107
0.656 + 0.006
8.781 + 0.070
0.681 + 0.003
RankUp
18
10.251 = 0.072
0.599 0.005
8.836
0.047
0.676 0.003
8,216 + 0.022
0.703 0.001
Ours
9.177 + 0.061
0.664 = 0.003
8,539 1 0.065
0.695 = 0.003
8,166 = 0.071
0.712 = 0.002
Table 3: Main results on STS-B. The performance is reported in the form of 'mean + std' across six random runs. Bold means the best results and our method are shown in gray cells
5%
10%
20%
Method
MSE 4
R²1 f
MSE 4
R²
MSE 4
R² f
Fully Sully-Supervieed
0,986 1: 0.024
0.533 1: 0.012
0,986 1: 0.024
0.533 1: 0.012
0,986 1: 0.024
0.533 0.012
Supervised
1.746 + 0.016
0,173
0.007
1.524 + 0.010
0.278
0.005
1.332 4 0.012
0.369
0.005
Mcan Teacher
43
1,751 1: 0.010
0.170 1 0.005
1.607 1 0.017
0,240 1: 0.007
1.409 1 0.009
0.332 0.004
Temporal Ensemhling
26
1.683 4: 0.007
0.203 + 0.004
1.554 0.009
0.264 + 0.004
1.374 + 0.015
0.349 0.007
SSDKL 200
1.606 0.057
0,239
0.027
1.407
0.029
0,333
0.033
1,211 + 0.031
0.426 0,015
TNNR (47
1,746 - 0.007
0,166 1: 0.004
1.534 - 0.003
: 0.001 0,266
1.333 + 0.020
0.362 : 0.010
SinReeyMatch
1.714 0.032
1,559 0.024
1.437 0.015
UCVME lm
1,713 + 0.009
0,188 0.004
1.577 + 0.016
0.253 4- 0.008
1.373 + 0.020
0.349 0.009
CLSS 9
1.622 + 0.046
0,231
0.022
1.421 + 0.030
0.327
0:27-0000 0.014
1,286 + 0.032
0
RankUp 18
1.844 + 0.063
0,126 + 0.030
1.454
+ 0.016
0,311 - 0.008
1.279 + 0.018
0.394 + 0.008
Ours
1,540 + 0.006
0.270 + 0.003
1,403 + 0.013
0.335 0.006
1 1,246 + 0.015
0.409 0.008
4.2 Main Results
Results on UTKFace. Table summarizes the results of our method and other comparison methods on UTKFace dataset with various ratios. It can be observed that: 1) All SSR methods perform better than the benchmark Supervised, which the lower bound trained by labeled samples, demonstrating that incorporating unlabeled data is beneficial for the SSR problem and effectively reduces regression error. For example, under the ratio 5%, our method reduces the MAE by up to 8. %% and increases RR by up to 15.2% compared to Supervised. 2) Compared to other SSR methods,
method
consistently achieves the best or second best results across all ratios. In particular, when the labeled data is scarce (7 5%), our method outperforms the second best RankUp [8] by 1.4% for MAE and 5.7% for R² 3) As the proportion of labeled data increases, the performance gains of our method slightly lower than those of UCVME and RankUp We attribute this stronger supervision from labeled data, which likely improves the quality and stability of pseudo-labed demonstrate that the well-calibrated uncertainty generated by our method plays essential role in improving SSR performance, especially in scenarios where the labeled data is scarce.
Such results further
Results on IMDB-WIKL As shown in Table 2 our method consistently outperforms all comparison SSR methods significantly. For instance, under 7 = 5%, compared to the second best method Mean Teacher, method reduces MAE by about 3.3% and increases R by 2.6%. which achieves new state-of-the-art results. Note that the performance of our method with 20% labeled data is closer to Sullyy-Suprriseed, served as upper bound in the SSR problem. These results further validate the strength of our method. Furthermore, compared with other uncertainty estimation SSR methods like UCVME, method exhibits significant improvement, demonstrating the effectiveness of our predicted uncertainty by 9a in our bi-level optimization framework.
Results on STS-B. The results are listed in Table 3 It can be observed that our method achieves the best or second-bes -hett performance across all ratios. On the one hand, if the labeled data is limited, such as y 5% or 10%, method improves all these SSR methods on both MSE and R2a with large
8

---

150
UCVME
UCVME
100-
MSR
100
50
50
2122
25
23522
33.30
37.40
69.52
53.58
57.00
21:22
25
25-40
49-12
33.56
37-60
Aboolite Prediction Emor
30 26 20
Figure 4: Correlation analysis be- tween estimated uncertainty o² and absolute prediction error on IMDB- WIKI with y 10% un- der age 40.
Age Range
Age Range
(a) Predictions unlabeled data
(b) Predictions test data
Figure 3: Subgroup performance comparison between ours and
the second-best (UCVME) UTKFaae with 5%.
Table 4:
Ablation study results on the IMDB-WIKI dataset with label ratios of 5% and 10%. BL. UL, and BLO refer to Baseline, Uncertainty-leare and Bi-level Optimization, respectively.
Table 5: Computational cost (average training time per iteration) and memory complexity on UTKKaae dataset with 10%.
Components
5%
7
10%
Method
Time (ms)
GPU (MB)
BL
UL
BLO
MAE 4
R²a f
MAE +
R²a f
Bascline
83.0
5099
SinReg 21
548.1
7419
x
9.512
0.651
8.864
0.683
UCVME 110
257.2
10057
x
9.914
0.630
9.562
0.651
RankUp[18
108.6
7433
9.177
0.664
8,539
0.695
Ours
91.6
5116
margin. For example, when we have only 5% labeled data, MSE of our method lower than the second-best method SSDKL by 4.1% 1% and R²2 is higher by 13.0%. These results demonstrate that method does not rely on too much labeled data and performs better under such extreme scenarios with scarce labeled samples. On the other hand, our method achieves comparable results with SSDKL when we have more labeled data, i.e., 20%. This suggests that our method is not only effective data-scarce scenarios but also highly adaptable to different levels of labeled data availability.
4.3 Discussion and A Ablation Study
Performance analysis across subgroups. As aforementioned, our method consistently achieves sound performance across three different datasets with various ratios. To further validate the effective- ness of our method, in Fig. we compare the MSE between the ground truth labels and pseudo-labels estimated by UCVME and method across all ages UTKFace. We can that method can globally produce more accurate pscudo-labels than UCVME across different subgroups, especially on the elder age ranges whose samples are significantly fewer than those of other age ranges, as shown in Fig. 1 These results further demonstrate that our method can consistently improve the accuracy across different subgroups.
How our algorithm adjusts uncertainty. To answer this question, we visualize the correlation between the uncertainty estimated by the proposed uncertainty-leamer Is and the prediction error of the backbone regression model fo- Specifically, for all unlabeled training samples IMDB-W WIDI at age 40, we visualize their corresponding estimated uncertainties o² and absolute prediction errors in Fig. 4 It can observed that the estimated uncertainties increase as the prediction errors increase, indicating that the uncertainty-eaaren tends to assign larger uncertainty for the samples with larger prediction errors and actively reduces the contribution of these samples to the regression model training. Furthermore, the correlation between estimated uncertainties and absolute prediction errors not a simple linearity, suggesting it essential to use the uncertainty-learue to estimate the unstable uncertainties of different samples. Additional visualization results provided in Appendix C.1
Ablation study. We conduct an ablation study to analyze the contribution of each component of our method, with the results presented in Table We begin with the Baseline (BL) method, which omits the uncertainty-learee (UL.) and applies the loss L: XL,, with fixed uncertainty oz = for all unlabeled samples. Compared to method, the absence of uncertainty estimation leads

---

to poor performance, confirming its effectiveness in SSR. Next, we assess the roles of the UL and bi-level optimization (BLO) in our method. Interestingly, the model with UL alone (Baseline + UI.) performs worse than the Baseline, likely due to joint training of the UL. and the regression model, which leads to inaccurate uncertainty estimation and suppress learning from hard but correct samples, as discussed in Section 3.3 In contrast, incorporating the proposed bi-level optimization framework substantially improves performance, suggesting that it enables more accurate uncertainty estimation and, consequently, more effective training of the regression model. This complies with the conclusion of Theorem [ which theoretically supports the benefit of bi-level optimization in improving uncertainty estimation.
Computational cost analysis. We evaluate the computational cost on single NVIDIA GeForce RTX 4090 for fair comparison. As shown in Table 5 our method incurs only - -17MB additional GPU memory and less than 9ms extra training time per iteration compared the Baseline, as unrolls gradients only through the linear regression head to update a small number of parameters the outer-loop optimization. Compared to recent SSR methods, our method achieves superior performance on all datasets with less time and resource consumption. Further analysis of the computational cost of other recent SSR methods provided in Appendix C.2
5 Conclusion
We propose an uncertainty-aware pseudo-labeling framework for SSR tasks, which addresses the challenge of heteroscedastic noise in pseudo-labels. Different from the existing methods, our approach dynamically calibrates pseudo-label uncertainty from a bi-level optimization perspective. Such learning paradigm ensures reliable uncertainty estimation that improves generalization of the regression model. Theoretical analysis via gradient alignment and empirical results on benchmark SSR tasks demonstrate the effectiveness of our method, significantly enhancing robustness and accuracy over existing approaches.
Limitations and Broader Impact. Despite the promising results achieved by our method in semi-supervised regression, particularly in enhancing the robustness of pseudo-labels through het- eroscedastic uncertainty modeling, several limitations remain to be addressed. Notably, our approach does not explicitly consider potential systematic biases present in the labeled or unlabeled data, such as demographic imbalances domain-specific disparities. If left unmitigated, these biases may be implicitly propagated or even amplified through the pseudo-labeling process. Addressing these issues will be one of the key focuses of future work. Moreover, our current framework is built under the assumption that both labeled and unlabeled samples can be accessed simultaneously during training. Such requirement may raise privacy concerns in real-world scenarios involving sensitive user data. Incorporating fairess--aarr objectives and priaacy-preeerving mechanisms into future versions of our framework could further enhance its practical applicability and ethical reliability.
Acknowledgement
We would like to thank all anonymous reviewers for their constructive suggestions for improving this paper. This work supported in part by the Major Key Project of PCL under Grant PCL2024A06. Tianyuan Fund for Mathematics the National Natural Science Foundation of China under Grant 12426105, the China NSFC projects under contracts 62306233, 62476214 and 62372359, and the China Postdoctoral Science Foundation (2024M752550).
References
[1] D. Berthelot, N. Carlini, E. D. Cubuk, A. Kurakin, K. Sohn, H. Zhang, and C. Raffel. Remix- match: Semi-supervised learning with distribution matching and augmentation anchoring. In International Conference on Learning Representations, 2020.
[2] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel. Mixmatch: holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems, 32, 2019.
[3] C. M. Bishop and N. M. Nasrabadi. Pattern recognition and machine learning, volume Springer, 2006.
10

---

[4] N. Bodla, G. Hua, and R. Chellappa. Semi-supervised fusedgan for conditional image generation. In Proceedings of the European Conference on Computer Vision, pages 669-683 683, 2018.
[5] D. Cer, M. Diab, E. Agirre, Lopez-Gazpio, and L. Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-200) 20177, Vancouver, Canada, Aug. 2017. Association for Computational Linguistics.
[6] X. Chen, Y. Yuan, G. Zeng, and Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2613-2622, 2021.
[7] X. Cheng, D.-B. Wang, L. Feng, M.-L. Zhang, and B. An. Partial-label regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7140-7147, 2023.
[8] E. D. Cubuk, B. Zoph, J. Shlens, and Q Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition workshops, pages 702-703, 2020.
[9] W. Dai, Y. Du, H. Bai, K.-T. Cheng, and X. Li. Semi-supervised contrastive learning for deep regression with ordinal rankings from spectral seriation. Advances in Neural Information Processing Systems, 36:57087-57098, 2023.
[10] W. Dai, X. Li, and K.-T. T. Cheng. Semi-supervised deep regression with uncertainty consistency and variational model ensembling via bayesian neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7304-7313, 2023.
[11] R. Denton, S. Gross, and R. Fergus. Semi-supervised learning with context-conditional genera- tive adversarial networks. arXiv preprint arXiv:1611.06430, 2016.
[12] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pages 1126-1 135. PMLR, 2017.
[13] Y. Gal and Z. Ghahramani. Dropout bayesian approximation: Representing model uncer- tainty in deep learning. In International Conference on Machine Learning, pages 1050-1059, PMLR, 2016.
[14] S. Haykin. Neural networks: comprehensive foundation. Prentice hall PTR, 1994.
[15] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 770-778, 2016.
[16] L. Hoyer, D. J. Tan, M. F. Naeem, L. Van Gool, and F. Tombari. SemiVL: Semi-supervised semantic segmentation with vision-language guidance. In Proceedings of the European Confer- ence on Computer Vision, pages 257-275. Springer, 2024.
[17] X. Hu, Y. Niu, C. Miao. X.-S Hua, and H. Zhang. On non-random missing labels in semi- supervised learning. In International Conference on Learning Representations, 2022.
[18] P.-Y. Huang, S.-W. Fu, and Y. Tsao. Rankup: Boosting semi-supervised regression with an auxiliary ranking classifier. Advances in Neural Information Processing Systems, 37:107444- 107468, 2024.
[19] A. Immer, E. Palumbo, A. Marx, and Vogt. Effective bayesian heteroscedastic regression with deep neural networks. Advances in Neural Information Processing Systems, 36, 2024.
[20] N. Jean, M. Xie, and S. Ermon. Semi-supervised deep kernel learning: Regression with unlabeled data by minimizing predictive variance. Advances in Neural Information Processing Systems, 31, 2018
[21] Y. Jo, H. Kahng, and B. Kim. Deep semi-supervised regression via pseudo-label filtering and calibration. Applied Soft Computing, 161:1116700 2024.

---

[22] A. Kendall and Y. Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in Neural Information Processing Systems. 30, 2017.
[23] A. Kendall, Y. Gal, and R. Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7482-7491, 2018.
[24] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[25] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks In International Conference on Learning Representations, 2017.
[26] Laine and Aila. Temporal ensembling for semi-supervised learning. In International Conference Learning Representations, 2017. 2017..
[27] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
[28) Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521 2015.
[29] D.-H. Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Proceedings of the Workshop on Challenges Representation Learning, ICML, volume 3, page 896. Atlanta, 2013.
[30] Z. Li, C. Deng, E. Yang. and D. Tao. Staged sketch-to-ing torecce-----ageeee synthesis via semi-supervised generative adversarial networks. JEEE Transactions on Multimedia, 23:2694-2700 2020.
[31] Y. Lin, Yao, Z. Li, G. Zheng, and X. Li. Calibrating label distribution for class-imbalanced barely-supervised knee segmentation. In International Conference on Medical Image Computing and Computer-Assistel Intervention, pages 109-118. Springer, 2022.
[32] H. Liu, K. Simonyan, and Y. Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations, 2019.
[33] T. Mahmud, C.-H. Liu, B. Yaman, and D. Marculescu. Ssvod: Semi-supervised video object detection with sparse annotations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6773-6782, 2024.
[34] S. L. Pintea, Y. Lin, J. Dijkstra, and J. C. van Gemert. A step towards understanding why classification helps regression. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 19972-19981, 2023.
[35] M. Ren, W. Zeng, B. Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, pages 4334-4343. PMLR, 2018.
[36] M. N. Rizve, K. Duarte, Y. S. Rawat, and M. Shah. In defense of pseudo-labeling: An uncertainty-awant pseudo-label selection framework for semi-supervised learning. In Interna- lional Conference on Learning Representations, 2021.
[37] J. Rodemann, J. Goschenhofer, E. Dorigatti, T. Nagler, and T. Augustin. Approximately Bayes- optimal pseudo-label selection. In Proceedings of the Thitt Conference on Uncertainty in Artificial Intelligence, pages 1762-1773. PMLR, 2023.
[38] Rodemann, C. Jansen, G. Schollmeyer, and T. Augustin. In all likelihoods: Robust selection of pseudo-labeled data. In Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications, pages 412-425. PMLR, 2023.
[39] R. Rothe, R. Timofte, and L. Van Gool. Deep expectation of real and apparent age from single image without facial landmarks. International Journal of Computer Vision, 126(2):144-157, 2018.
12

---

[40] M. Sajjadi, M. Javanmardi, and T. Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. Advances in Neural Information Processing Systems, 29, 2016.
[41] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li. Fixmatch: Simplifying remi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33:596-608, 2020.
[42] J. T. Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv preprint arXiv:1511.0633/ 2015.
[43] A. Tarvainen and H. Valpola. Mean teachers better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. Advances in Neural Information Processing Systems, 30, 2017.
[44] J. E. Van Engelen and H. H. Hoos. A survey semi-supervised learning. Machine Learning. 109(2):373-440, 2020.
[45] A. Wang, A. Singh. J. Michael, Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task bench- mark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019.
[46] R. Wang, X. Jia, Q. Wang. Y. Wu, and D. Meng. Imbalanced semi superviue learning with bias adaptive classifier. In International Conference Learning Representations, 2023.
[47] S. J. Wetzel, R. G. Melko, and I. Tamblyn. Twin neural network regression is semi-supervised regression algorithm. Muchine Learning: Science and Technology, 3(4)0455077, 2022.
[48] Y. Xia, F. Liu, D. Yang, J. Cai, L. Yu, Z. Zhu, Xu, Yuille, and H. Roth. 3d sumi-supervised learning with uncertainty aware -iet co-training. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3646-3655, 2020.
[49] Q. Xie, Z. Dai, Hovy, T. Luong, and Q Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 33:6256-622 2020.
[50] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10687-10698, 2020.
[51] Z. Xing, Q Dai, H. Hu, J. Chen, Z. Wu, and Y.-G. Jiang. Svformer: Semi-supervised video transformer for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18816-18826 2023.
[52] X. Yang, Z. Song, I. King, and Z. Xu. A survey on deep semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering, (((9):8934-8954 2023.
[53] Y. Yang, K. Zha, Y. Chen, H. Wang, and D. Katabi. Delving into deep imbalanced regression. In International Conference on Machine Learning, pages 1882-11851. PMLR, 2021.
[54] Y. Yang, D.-C Zhan, Y.-F Wu, Z.-B. Liu, H. Xiong, and Y. Jiang. Semi-supervised multi-modal clustering and classification with incomplete modalities. IEEE Transactions Knowledge and Data Engineering. 33(2):682 695, 2019.
[55] H. Yao. X. Hu, and X. Li. Enhancing pseudo label quality for semi-supervised domain- generalized medical image segmentation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 3099-3107, 2022.
[56] I.. Yu, Wang, X. Li, C.-W. Fu, and P.-A. Heng. Uncertainty-aware self-ensembling model for semi-supervised 3d left atrium segmentation. In Medical Image Computing and Computer- Assisted Intervention MICCA/ 2019, volume 1 11765 of Lecture Notes in Computer Science, pages 605-613 Springer, Cham, 2019.
[57] B. Zhang, Y. Wang, W. Hou, H. Wa, J. Wang. M. Okumura, and T. Shinozaki. Flexmatch: Boosting supervsee learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34: 18448444881819,2222
13

---

[58] Z. Zhang, Y. Song, and H. Qi. Age progression/regresionn by conditional adversarial au- toencoder. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 5810-8811 2017.
[59] Y. Zhou, X. He, L. Huang, L. Liu, F. Zhu, S. Cui, and L. Shao. Collaborative learning of semi-superye supervsper segmentation and classification for medical images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2079-20888 -088, 2019.
14

---

Supplementary Material
A Theoretical Analysis
A.1 Proof of Theorem
In this section, we first review the proposed bi-level learning framework optimize the parameters of the regression model fo and the uncertainty-learner Jr. The bi-level optimization framework can be formulated follows:
Φ arg min Couter (0" (o))
(10)
s.t. (Φ) arg min Linner (0, Φ"),
where the specific inner and outer losses can be represented as
Couter (0(0))
Σ (yk- for (o)((e))
# EB:
Linner (0, Li( + (0,
(11)
Σ z
(U: - So(x))
+
Σ z EB,
ÜUg Sa(a))) + 22
exp(2z)
where 2, = 96(2)) denotes the uncertainty estimated by the uncertainty- 9a. optimize this iiliicntoottmizattion based method to approxi- bi-level optimization framework, we use a nested uncertainty-leamee parameter Φ. Specifically, mately update the regression model parameter 9 and the in the inner loop, the updating formulation of f at iteration i can be expressed as:
(e'' 0 Vo (0", Φ²),
(12)
where or is the learning rate of the inner loss.
8+41 0 Couter (0(++(0'))
(13)
where 3 is the learning rate of the outer loss.
For simplicity of notation, we denote V Connee((e. Φ) and V Co (0) be the gradients of the inner and outer loss w.r.t. respectively. Then we provide the proof of Theorem in the following.
min
(V Linner (0.0), Louter (0))
(14)
Proof. For the outer optimization problem, we expand this loss function by the first-order Taylor expansion around 0¹ as follows:
ooter (0++1 (++)) ~ Couter + (0'+ 8¹) Lotter (8*)
(15)
Substitute eq.
12 into eq. 15 we obtain
Couter (0++ (6)) Couter (0') + (-a Linner (0', ('))) Louter (0*).
(16)
Note that the first term on the right side of eq. 16) is irrelevant to the optimization of Φ', which can be omitted. Thus, minimizing the outer loop of eq. (6) is equivalent minimizing the second term in eq. (66). That is,
min Cotter (0'+) + min (inninn (0', 00,00), V VoL (0')).
(17)
o¹
We thus finish the proof.
15

---

A.2 Connection to Bayesian Decision Theory
We make an effort connect our method Bayesian decision theory 37 38 to further interpret our bi-level framework. To make the theoretical analysis more tractable, we restate our bi-level learning formulation with slight simplification. Specifically, we let the predictive variance oj go rather than using its logarithm as in the main text. This modification only involves a log-transform and does not alter the essential behavior of our model. The inner-loop optimization becomes:
1
0* arg min
U - fo(x:))
(Ú) - so(a;))² lo Io g6( 1=1
(18)
and the outer-loop optimization is:
arg min
Σ (E
(yx - fo-)
(19)
which seeks optimal uncertainty weights gx(x,) guide the model fo-(Ø) toward better generaliza- tion.
To link our bi-level learning framework to a Bayesian decision problem, we formulate the following decision problem. Concretely, the uncertainty estimator Io X -> IR defines randomized (mixed)
decision rule by assigning weights to unlabeled samples zjj E Du. The model parameter 0 is treated as the latent state, and we define the utility function as:
u u(gs, 0) = log P(D; UD = log II N(u.lfo(zi). )
II ((00)f((())) (20) myEDe
Z,ED;
where ūن denotes the pseudo-label for zy.- Under this formulation, the Bayes-optimal uncertainty function 9 is the one that maximizes the expected utility w.r.t the posterior distribution over A, i.e., d² arg max, Egg (D. [u(g6: 0)].
We now show an intuition connection between our bi-level optimization and this Bayesian formulation. Specifically, the inner-loop optimization is equivalent to minimizing the negative log- log-likelihood of the data under the generative model defined earlier. Therefore, 0* (Ø) can be interpreted as a MAP estimate of model parameters 6, conditioned the fixed uncertainty function Js- The outer-loop optimization aims to find the optimal uncertainty weights that lead to a model fo-(s) with best generalization performance. While this not a direct maximization of expected utility Eg (u(ss. 0)). argue the following: (i) If the posterior P() is sharply peaked (posterior concentration), and (ii) if the outer-loop loss over B3 is reliable proxy for risk under p(æ, y), then minimizing the outer-loop loss approximately maximizes the expected utility, and thus o² approximates the Bayes-optimal mixed-action policy.
A.3 Pseudo-Label Smoothness Property
Our framework implicitly enforces a smoothness property on the generated pseudo-labels, as sup- ported by both theoretical analysis and empirical evidence:
Theoretical smoothness guarantee. The Lipschitz continuity of both the regression model fe and the uncertainty estimator 96 ensures that nearby unlabeled inputs yield similar pseudo- labels and uncertainty estimates. Specifically, for any two unlabeled samples x and z we have: I01 d2l S L x²} and lo? oz < LyIgfaæ æ²1. where L, and Lg are the Lipschitz constants of fo and gG, respectively. These constants can be effectively controlled by architectural design. For example, we can apply spectral normalization to fo to limit its Lipschitz constant, and we design 9o shallow MLP with ReLU activations and small- initialized weights to promote smoothness.
Empirical evidence. To further validate the smoothness of pseudo-labels in practice, we conducted a variance analysis across age groups on the IMDB-WIKI dataset. Specifically, we computed the standard deviation of predicted pseudo-labels among samples that share the same ground-trute truth age and compared the results with UCVME [0, which explicitly enforces local consistency via consistency loss. As reported in the table 6 our method consistently yields lower or comparable standard deviations than UCVME in most age groups. This suggests that our aseudo-label are more stable and consistent across similar inputs, thereby indirectly capturing the desired smoothness prior.
16

---

Table 6: Performance comparison across age groups.
Method
Age=10
20
40
50
60
70
80
90
UCVME 10
10.623
6.522
5.009
5.878
7.772
10.666
12.936
13.085
5.017
Ours
3.290
6.389
6.157
6.971
8.558
10.231
10,301
6,364
4.011
B Detailed Experimental Setup
B.1
Dataset Details
In this work, we experiment with three benchmark datasets: UTKFaae
58 IMDB WIBI and
STS-B 45, which are detailed as follows:
UTKF [58) The dataset is image age estimation dataset, where the objective is predict an individual's 's age based on facial image. The dataset consists of 23,705 face images with the labels ranging from 116 years old. Following the protocols used in UCVME [1O], this paper considers only samples aged 21 to 60. The resulting modified dataset includes 10,518 18 training samples, 3,287 testing samples, and 2,629 validation samples.
IMDB-WIKI [39]. The dataset a larger-scale age estimation dataset which collected over 523K facial images and their corresponding age. Following (53), we utilize curated datasets consisting of 191.5K images for training and 1 11.0K images for validation and testing. The ages range from 0 to 186 years old and the number of images per age varies from 1 to 7149.
STS-B 5 45). Semantic Textual Similarity Benchmark (STS-B) dataset, which consists of 7.2K sentence pairs collected from real worlds, such as news, image and video captions, and natural language inference data. The target of each pair is a continuous similarity score ranged from 0 to 5. Following DIR [53), construct the training set containing 5.2K pairs, and both the balanced validation set and the test set containing IK pairs each.
B.2 Evaluation Metrics
To assess the performance of comparison algorithms, we adopt Mean Absolute Error (MAE () and the coefficient of determination (R²2 t) for image datasets following UCVME 10, while employing Mean Squared Error (MSE J) and R² metrics for text datasets following DIR
53
Mean Absolute Error (MAE 4). MAE measures the average absolute difference between predictions and ground truths, offering simple and robust metric for evaluating prediction accuracy. i.e.
MAE
ffo (x:) Vel
(21)
where fo- (x:) denotes the prediction for the sample vi, and yi is the corresponding ground truth. Coefficient of Determination (R²2 1). R² reflects the goodness of fit of the model the data, i.e.,
R² En (fo- (ar:) y1)²
(22)
Ei 7)
where y is the mean of y:-
Mean Squared Error (MSE 1). MSE measures the average squared difference between predictions and ground truths, emphasizing larger errors due to the squaring operation. i.e.,
MSE 1 Σ (fo- 2
(23)
B.3 Training Details
Face and IMDB-WIKI Following the training protocol in UCVME [10), we utilize ResNet- 50 [15] pretrained on the ImageNet dataset as the backbone regression model. The model is trained
by Adam optimizer 24 for 30 epochs, with a learning rate of 10-4 for the feature extractor and
10 for the regression head. Additionally, conduct random cropping and horizontal flipping as
17

---

weak augmentation, and RandAugment g as strong augmentation in the data augmentation process of SSL. As for 9æ. optimized by Adam optimizer with learning rate of 10
STS-B. Following 45 5.3
we adopt a BiLSTM architecture with GloVe word embeddings as the backbone regression model, which is trained by Adam optimizer for 200 epochs, with a learning rate of 10 for the feature extractor and 10-3 for the regression head. For data augmentation, no augmentation is applied to the labeled dataset, while synonym replacement and insertion operations are used as strong augmentation for the unlabeled dataset.
B.4 Comparison Methods
To comprchensively evaluate the proposed method, we compare with some state-of-the-art semi- supervised regression methods, including: Consistency-basee methods: Mean Teacher [3]. Temporal
Ensembling [26.. TNNR 47). UCVME [0)] CLSS RankUp [8]; Uncertainty- based methods:
SSDKL 20). MatRegMatch [21]
In this section, we provide brief introduction to comparison algorithms.
averages model weights to form a farget-generating teacher model via exponential moving average (EMA), and computes the consistency loss between the teacher's predictions and the student's outputs.
Mean Teacher 4
Temporal Ensembling 26 maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target.
TNNR 47 is trained to predict differences between the labels of the input pair and follow the principle that the loop of predicted differences should sum to zero.
UCVME [0] improves training by generating high-quality pseudo-labels and uncertainty estimates for heteroscedastic regression.
CLSS [9] uses the recovered ordinal relationship for contrastive learning unlabeled samples to allow more data to used for feature representation learning.
RankUp [8] converts the original regression task into a ranking problem and training components: the Auxiliary
concurrently with the original regression objective. It introduces Ranking Classifier (ARC) and Regression Distribution Alignment (RDA).
SSDKL
non-parametric kernel learning methods based on minimizing predictive variance in the posterior regularization framework. It combines the hierarchical representation learning neural networks with the probabilistic modeling capabilities of Gaussian processes.
SitRegMatch [1 semi-supervised regression framework with two primary modules: uncertainty-baat based filtering and Limilarity-base pseudo-label calibration. It filters reliable pseudo-labels using uncertainty and refines them by incorporating labeled data through similarity- based weighting.
C Additional Experiment Results
C.1 More Visualization Results
To further analyze the correlation between estimated uncertainty o² and absolute prediction error, present additional age-wise visualizations on the MDB-WIKS dataset. Consistent with observations Fig. 5, the uncertainty o² increases alongside the absolute prediction error across age groups. highlighting the role and effectiveness of the uncertainty-leamer.
C.2 Computational Resources
Compared other SSR methods, our proposed algorithm achieves superior performance while maintaining significantly lower computational and memory costs. Among the algorithms compared table 3 MittegMatch (2]] estimates pseudo-label uncertainty through Monte Carlo sampling, which requires multiple forward passes, and stores feature vectors of samples to compute the similarity matrix. UCVME [0] relies on multi-model architecture and aggregates predictions from repeated forward passes compute umcertainty-basee consistency loss. RankUp [I8] not only
18

---

1.2
1.2
1.1
1.0
1.00
1.0
0.9
0.8
10
20
30
10
20
30
Erao
Absolute Prediction
(a) age=40
(b) age=50
(c) age=60
Figure 5: Correlation analysis between estimated uncertainty 0² and absolute prediction error on IMDB-WiL WIKI with = 10% under different age.
outputs regression predictions, but also generates ranking scores for each sample, necessitating the computation of all pairwise ranking losses. Furthermore, RDA module involves frequent lookups from a pseudo-label table during training, which adds additional time overhead. Our method does not require multiple forward passes or the storage of additional intermediate variables during computation. is lightweight network; in the optimization process,
Meanwhile, the proposed uncertainty-learen we assume that the parameters of the uncertainty-lcarme are only related to the regression head, which significantly reduces computational time and resource consumption.
C.3 Additional Experimental Comparisons
To further assess the generality and robustness of our method, we additionally include methods from weakly supervised learning (PIM 77,, cemi-supervised classification (UPS (36 and CADR 17)
We provide a brief introduction to these methods below.
PIM 7 computes its loss as weighted sum of the predictive losses over candidate labels, where labels with smaller losses are assigned higher weights through a softmax-based weighting function.
uses Monte Carlo Dropout to estimate uncertainty via the variance of multiple forward passes, and applies threshold to identify cow-confidence predictions negative labels, thereby enabling negative learning.
UPS 36
CADR (17) leverages the prior distribution of labeled samples to perform inverse probability weighting (CAP module) and adjusting per-class selection thresholds for unlabeled samples (CAI module).
Adapting existing methods for SSR. Several of the compared methods cannot be directly applied to regression tasks, we propose reasonable adaptations enable fair comparison under the SSR setting. Specifically, (1) PIM is originally designed for partial-label regression, where each unlabeled instance is associated with candidate label set. Since such candidate sets absent in suprrisuer regression, we construct pseudo-label candidates via Monte Carlo Dropout and compute the PIM loss using softmax-base defining negative targets in a discrete label space, which is not directly applicable to continuous regression outputs. To adapt UPS fairly, retain only the uncertainty-baet -based pseudo-label estimation via Monte Carlo Dropout and remove the negative labeling mechanism, yielding the adapted variant UPS UPS_Reg. (3) CADR builds on class-specific confidence modeling, which presupposes discrete label categories. To make it applicable to regression, we discretize the continuous target into age bins and treat each bin as surrogate class for confidence estimation.
weighting over these candidates. (2) UPS relies
Results Analysis. As shown in table our proposed method significantly outperforms methods across multiple settings. Under the ratio of 20%, method outperforms the second best method PIM by 3.3% for MAE and 1.1 1 for R².
all comparison
Comparison with weighting-buse based methods. Our uncertainty- method and PIM both share the common idea of assigning different loss contributions to samples. However, there is fundamental difference in how the weights generated. PIM assigns weights solely based the prediction
19

---

Table 7: Experimental comparison with methods the IMDB-WIKI dataset. Bold means the best results and our method are shown in gray cells
5%
10%
20%
Method
MAE
R 1
MAE 4
R f
MAE L
Rº
PIM
9.345
0.100
0.659 + 0.005
8,691 +: 0.021
0.691 = 0.001
8.441 1 0.014
0.704 =: 0.001
UPS Reg 30
9.430 + 0.087
0.647 = 0.006
8.845 = 0.055
0.676 = 0.003
8,415 + 0.002
0.699 = 0.001
CADR 17
10.392 + 0.091
0.562 + 0.009
9.531 + 0.064
0.622 = 0.007
8.536 + 0.056
0.668
+ 0.008
Ours
9.177 0.061
0.664 0.003
8.539 + 0.065
0.695 0.003
8.166 + 0.071
0.712 0.002
emor of pseudo-labels within each batch, making it prone to reinforcing inaccurate pseudo-labels and amplifying noise. In contrast, our method infers per-sample uncertainty and adaptively adjusts each sample's contribution in a continuous manner, resulting in more reliable weighting.
Comparison with classificatonn-drrieee SSL methods. The inferior performance of UPS and CADR highlights the challenge of directly adapting classification-basee methods to regression tasks. Morevoer, compared with UPS @PS_Reg, while both methods incorporate uncertainty, our method uses uncertainty- soft weighting within bi-level optimization framework, yielding more stable and accurate uncertainty estimation.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims reflect the paper's contributions, which is proposing an uncertainty-aware pseudo-labeling framework that dynamically adjusts pseudo-label influ- ence from a bi-level optimization perspective. Our empirical results and theoretical insights demonstrate superior robustness and performance compared to existing methods.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our proposed method in Section 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
21

---

Answer: [Yes]
Justification: We have provide full set of assumptions and a complete (and correct) proof in Appendix A.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided a detailed implementation of our proposed framework in Section 3 and provided some experiments details in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
22

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All the datasets public illustrated in Appendix B.1 We will release the full code and commands once the paper gets accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic/guudees/Codeebbbmsss wwt.ssiiinoooolylo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppppgguuuaaaeoeaec
wtw.eeeeeeoSSSeeesSSSeoeee.. www.ssssiicgcyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We claim the experiments setting and detainls Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All of our experimental results were conducted six runs using fixed random seeds (0, 2, 3, 4, and 5), and we report both the mean and standard deviation of each metric.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: For more detailed information about the computer resources used for conduct- ing our experiments, please refer Section 4.3
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes]
Justification:
We have verified that all requirements of the ethical guidelines have been met.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impacts of our proposed method in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not release any high-risk data models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited all code, data, and models used in this work to ensure transparency and proper acknowledgment.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
datasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code used in this paper, which is well documented. The documentation includes detailed descriptions of the code's purpose, functionality, usage examples, and license details. The documentation will be publicly available.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve human subjects or crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
This study does involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
The paper did not describe the usage of LLMs, as they were only used for writing, editing, formatting purposes.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neuri)) [ttp:e/r/eearrrgg.ccc///ooeeeeer / for what should or should not be described.
27


Reviews:
[{"review_id": "OHqYTni0RX", "reviewer": "Reviewer_c7Z8", "summary": "This paper is on semi-supervised regression (SSR), a scenario where labeled data is limited but unlabeled data is abundant. While pseudo-labeling is widely adopted in semi-supervised classification, it is less explored for regression tasks. The main difficulty stems from the continuous nature of regression targets. Unlike classification, where pseudo-labels are discrete and confidence is easier to quantify, regression must contend with heteroscedastic noise, which makes pseudo-label reliability much harder to assess.\n\nThe authors propose an uncertainty-aware pseudo-labeling framework for SSR. The central idea is to dynamically adjust the influence of pseudo-labeled samples based on their estimated uncertainty. The method employs a bi-level optimization approach. In the inner loop, the regression model is trained on both labeled and pseudo-labeled data, where each pseudo-label is weighted according to its predicted uncertainty. In the outer loop, the parameters that determine the uncertainty are optimized to ensure better generalization on the labeled data.\n\nA lightweight neural network, referred to as the uncertainty-learner, is introduced to estimate the uncertainty for each pseudo-labeled example. This network takes as input both the model's prediction and the pseudo-label for a given sample and produces a log-variance value representing the estimated uncertainty. This enables the model to down-weight unreliable pseudo-labels during training, thereby reducing the risk of error accumulation and overfitting to noisy signals.\n\nThe paper provides a superficial theoretical justification for this approach: The bi-level optimization implicitly aligns the learning gradients from labeled and pseudo-labeled data. MOreover, when the uncertainty estimation is accurate, the model's learning direction is guided toward solutions that generalize well, rather than merely fitting the pseudo-labeled data. This contributes to greater stability and robustness in training.\n\nComprehensive experiments are conducted on three benchmark datasets: UTKFace and IMDB-WIKI for age estimation from images, and STS-B for semantic similarity regression on text pairs. The method is evaluated under different proportions of labeled data, specifically at 5%, 10%, and 20%. Across all settings, the proposed approach consistently outperforms several strong baselines, including both consistency-based and uncertainty-based SSR methods. The gains are especially pronounced when labeled data is scarce.\n\nThe ablation studies further validate the design choices. Removing the uncertainty-learner or not using bi-level optimization leads to a noticeable drop in performance. The experiments also show that the uncertainty estimates produced by the model are well correlated with the actual prediction errors. Samples with higher prediction errors are assigned higher uncertainty, and their influence on model updates is reduced.Regarding efficiency, the proposed method introduces only minimal computational and memory overhead compared to standard baselines. This makes it practical for real-world applications.\n\nIn concusion, the paper presents a principled and effective solution to semi-supervised regression with noisy pseudo-labels. By modeling heteroscedastic uncertainty and optimizing it in a bi-level manner, the approach significantly enhances robustness and accuracy, particularly in data-scarce regimes. The results suggest that careful handling of uncertainty is crucial for the success of SSR, and the framework established here sets a new standard for future research in the field.", "questions": "see above", "limitations": "see above", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I think this is a prime example of a borderline paper. The methodology, in particular the bi-level optimization for learning uncertainty in pseudo-labeling for heteroscedastic SSR, is novel and the experiments speak for themselves. However, the theoretical analysis is pretty slim and not that insightful. Theorem 1 states a mere re-write of the optimizaiton problem. Directly seeing the implicit regularization of the gradients in eq.9 is nice, but also already apparent to the educated reader by eq. 5 and 6. I think the paper would greatly benefit from a deeper theoretical underpinning. (That being said, I want to re-iterate that I really like the methodological ideas and find the experimental results very convincing.) Here is an actionable suggestion of how to add some theoretical motivation: The authors could easily prove (approximate) Bayes-optimality of \\theta^* and \\phi^* by framing the bi-level optimization problem as decision problem with mixed decision rules. Building on related work on uncertainty-aware pseudo-label selection [1-4], in particular [2] which embeds pseudo-label selection in decision theory, it is easy to see that the calibration via $\\sigma^2$ corresponds to a probability weight vector of all actions (=pseudo-labeled data). Let me maybe add some further explanations: the decision-theoretic embedding and Bayes-optimality proves in [2] are concerned with pseudo-label *selection\", where a only a subset of pseudo-labeled data is added to labeled data iteratively (\"self-training\"). I am aware the setup is different here. However, the decision-theoretic embedding still holds by considering mixed-action decision theory, also known as randomized decision theory, which involves making decisions by using a weighting vector (probability distribution) over multiple actions, rather than choosing a single action. The calibration via the $\\sigma^2$ corresponds to exactly such a weighting over all actions (=pseudo-labels). By framing as an adequate decision problem, it can be seen that the proposed method corresponds to the best strategy given all available information and uncertainties (i.e., Bayes action).\n\n[1] Rizve, Mamshad Nayeem, et al. \"In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning.\" International Conference on Learning Representations. 2021.\n\n[2] Rodemann, Julian, et al. \"Approximately Bayes-optimal pseudo-label selection.\" Uncertainty in Artificial Intelligence. 2023.\n\n[3] Rodemann, Julian, et al. \"In all likelihoods: Robust selection of pseudo-labeled data.\" International Symposium on Imprecise Probability: Theories and Applications. 2023.\n\n[4] Dietrich, Stefan, et al. \"Semi-supervised learning guided by the generalized Bayes rule under soft revision.\" International Conference on Soft Methods in Probability and Statistics. 2024\n\nMinor remarks: \n\n- Index in second summand of eq.3 should be j rather than x_j^u, as the latter is not used in the sum. Please also update the index set accordingly to $\\|\\mathcal B_u \\|$\n- redundant use of \"optimization\" in proof of theorem 1: \"irrelevant to optimization of $\\Phi_t$, which can beomitted in optimization\"", "was_revised": true, "final_justification": "See summary of strenghts and weaknesses", "rebuttal": "We sincerely thank you for the encouraging feedback on the novelty of our bi-level optimization framework for learning uncertainty and the strength of our experimental results, as well as the constructive suggestions for strengthening the theoretical aspect from a Bayesian decision perspective. Below, we respond to each point in detail. Please let us know if any concerns remain.\n\n**W1.**  The theoretical analysis is pretty slim and not that insightful. Theorem 1 states a mere re-write of the optimization problem.\n\n> We agree that Theorem 1 may appear to be a direct derivation from the optimization procedure.  However, we respectfully point out that this  theoretical result plays an important role in clarifying the mechanism behind our method, and contributes to the broader understanding of uncertainty-based learning in two respects:  \n>\n> + **Theoretical insight into the mechanism.** This theorem provides a formal explanation of how uncertainty-aware weighting affects the training process.  In particular, it shows that our method automatically focuses on pseudo-labeled samples whose gradients are similar to those from labeled data, and reduces the impact of samples with higher predictive uncertainty. This insight helps explain why our method remains robust under incorrect or uncertain supervision. By selectively down-weighting the influence of noisy gradients,  it promotes more stable training and better generalization. In this way,   we think the theorem not only reflects the optimization procedure, but also helps explain more clearly why the method works in practice. \n> + **Theoretical bridge to gradient matching.** The theorem bridges our bi-level optimization framework with the emerging principle of gradient matching, a concept that has proven effective in tasks such as domain adaptation [5], dataset distillation [6], and few-shot learning [7]. This connection not only grounds our approach in a broader theoretical context but also suggests its potential applicability to other settings.   By showing that our uncertainty-based weighting tends to select pseudo-labels whose gradients align with those from labeled data, the theorem places our method within a broader family of alignment-based strategies.  This underscores its potential to serve as a foundation for semi-supervised learning methods that leverage gradient matching.\n\n**Q1.** Here is an actionable suggestion of how to add some theoretical motivation: The authors could easily prove (approximate) Bayes-optimality of $ \\theta^* $ and $ \\phi^* $ by framing the bi-level optimization problem as decision problem with mixed decision rules.\n\n> Following your valuable suggestion, we made an effort to connect our method to Bayesian decision theory to further interpret our bi-level framework.\n>\n> + To make the theoretical analysis more tractable, we restate our bi-level learning formulation with a slight simplification. Specifically, we let the predictive variance $\\sigma_j^2 = g_\\phi(x_j)$, rather than using its logarithm as in the main text. This modification only involves a log-transform and does not alter the essential behavior of our model. The **inner-loop optimization** becomes: $\\theta^{\\*}(\\phi) = \\arg\\min_\\theta \\sum_{i=1}^n (y_i - f_\\theta(x_i))^2 + \\lambda \\sum_{j=1}^m \\left[ \\frac{1}{g_\\phi(x_j)} (\\hat y_j - f_\\theta(x_j))^2 + \\log g_\\phi(x_j) \\right],$ and the **outer-loop optimization** is: $\\phi^* = \\arg\\min_\\phi \\sum_{(x_k, y_k) \\in \\hat{\\mathcal B}\\_l} \\left( y_k - f_{\\theta^{\\*}(\\phi)} (x_k) \\right)^2,$ which seeks optimal uncertainty weights $g_\\phi(x_j)$ to guide the model $f_{\\theta^*(\\phi)}$ toward better generalization.\n>\n> + To link our bi-level learning framework to a Bayesian decision problem, we formulate the following decision problem. Concretely, the uncertainty estimator $ g_\\phi: \\mathcal{X} \\to \\mathbb{R}^+ $ defines a **randomized (mixed) decision rule** by assigning weights to unlabeled samples $ x_j \\in \\mathcal D_u $. The model parameter $ \\theta $ is treated as the latent state, and we define the utility function as: $u(g_\\phi, \\theta) = \\log P(\\mathcal D_l\\cup D_u|\\theta)=\\log \\prod_{x_i\\in\\mathcal D_l} \\mathcal N (y_i \\mid f_\\theta(x_i), \\sigma^2) \\prod_{x_j\\in\\mathcal D_u}\\mathcal N (\\hat y_j \\mid f_\\theta(x_j), g_\\phi(x_j)), $ where $ \\hat y_j $ denotes the pseudo-label for $ x_j $. Under this formulation, the **Bayes-optimal uncertainty function** $ g_\\phi^* $ is the one that maximizes the expected utility with respect to the posterior distribution over $\\theta$, i.e., $\\phi^* = \\arg\\max_{\\phi} \\mathbb E_{\\theta \\mid \\mathcal D_l}[u(g_\\phi, \\theta)].$\n> + We now show an intuition connection between our bi-level optimization and this Bayesian formulation. Specifically, the inner-loop optimization is equivalent to minimizing the negative log-likelihood of the data under the generative model  defined earlier. Therefore, $\\theta^\\*(\\phi)$ can be interpreted as a MAP estimate of model parameters $\\theta$, conditioned on the fixed uncertainty function $g_\\phi$. The outer-loop optimization aims to find the optimal uncertainty weights that lead to a model $f_{\\theta^\\*(\\phi)}$ with best generalization performance. While this is not a direct maximization of expected utility $\\mathbb E_{\\theta}[u(g_\\phi, \\theta)]$, we argue the following: (i) If the posterior $p(\\theta \\mid \\mathcal{D}_l)$ is sharply peaked (posterior concentration), and (ii) if the outer-loop loss over $\\hat{\\mathcal{B}}_l$ is a reliable proxy for risk under $p(x, y)$, then minimizing the outer-loop loss approximately maximizes the expected utility, and thus $\\phi^*$ approximates the Bayes-optimal mixed-action policy. \n>\n>  Although we were unable to provide a full theoretical equivalence between our optimization procedure and the Bayes-optimal policy under the mixed-action decision framework due to the limited time available during the rebuttal phase,  we believe our preliminary formulation offers a promising starting point. In particular, the Bayesian framework in [2] on learning under noisy supervision may offer useful insights for further formalizing our uncertainty modeling.  \n>\n> We sincerely thank the reviewer again for this valuable suggestion. It opens a new door for our future research to rigorously bridge Bayesian decision theory and optimization-driven uncertainty learning, as well as to address confirmation bias in semi-supervised regression from a principled Bayesian perspective.\n\n**M1.** Index in second summand of eq.3 should be $j$ rather than $x_j^u$.   \n> Thank you for pointing out this minor but important detail! We will fix it in the revised version.\n\n**M2.** Redundant use of \"optimization\" in proof of theorem 1: \"irrelevant to optimization of $\\Phi_t$, which can beomitted in optimization\".\n> Thank you for this helpful remark! We will remove the redundant use of \"optimization\" in the proof of Theorem 1 and improve the wording accordingly in the revised version.\n\n---\n> **Reference:**    \n[1] In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning. In ICLR, 2021.  \n[2] Approximately Bayes-optimal pseudo-label selection. In UAI, 2023.     \n[3] In all likelihoods: Robust selection of pseudo-labeled data. In ISIPTA, 2023.   \n[4] Semi-supervised learning guided by the generalized Bayes rule under soft revision. In ICSMPS, 2024.   \n[5] Gradient Matching for Domain Generalization. In ICLR, 2022.   \n[6] Dataset Condensation with Gradient Matching. In ICLR, 2021.   \n[7] Gradient matching generative networks for zero-shot learning. In CVPR, 2019."}, {"review_id": "yurVKbX5Kd", "reviewer": "Reviewer_5vPR", "summary": "This paper investigates how to leverage pseudo-labels for semi-supervised regression (SSR). The proposed method is an uncertainty-aware framework that dynamically adjusts the influence of pseudo-labels from a bi-level optimization perspective. Extensive experiments validate the effectiveness of the proposed approach.", "questions": "1. Since pseudo-labels are the predictions of the current model, is the proposed method similar to the progressive identification method in [1]?\n\n2. How efficient is the bi-level optimization method?\n\n3. The proposed method assigns pseudo-labels to all unlabeled samples. Could it be further improved by assigning pseudo-labels only to confident samples?\n\n[1] X. Cheng et al. Partial-Label Regression. In AAAI 2023.", "limitations": "N/A.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-organized and easy to follow.\n2. Investigating how to leverage pseudo-labels in semi-supervised regression is an interesting problem.\n\nWeaknesses:\n1. The bi-level optimization mechanism is inefficient.\n2. The use of pseudo-labels has already been extensively studied in related fields, such as weakly supervised learning.\n3. The experiments only consider the cases where $\\gamma$ = 5%, 10%, and 20%, lacking results for larger a values.", "was_revised": true, "final_justification": "My main concerns about this work lie in the efficiency of the proposed method and its potential for further improvement. The additional experiments provided by the authors have addressed my doubts. Therefore, I give a positive score.", "rebuttal": "We sincerely thank the reviewer for providing valuable feedback. Below, we respond to each point in detail. Please let us know if any concerns remain.\n\n**W1. & Q2.** The bi-level optimization mechanism is inefficient. \n> Although traditional bi-level methods are often inefficient due to the need for computing second-order derivatives, we respectfully point out that our bi-level learning framework reduces such inefficiency through algorithmic improvements, as analyzed in **Lines 217–222 of Section 3.4**. Moreover, our algorithm shows competitive computational efficiency compared to recent SSR methods, as remarked in **Table 5 of the main text**. We detail the reasons as follows:  \n> + **The efficiency is mainly from a significant simplification in computing second-order derivatives.**  Unlike traditional bi-level methods that unroll the entire network to compute second-order gradients [6,7], our method simplifies this by assuming the uncertainty-learner parameters $\\phi$ are associated only to the regression head. This allows gradient unrolling solely through the regression linear layer, enabling efficient outer-loop updates of $\\phi$ in Eq. (8). Since the regression head contains far fewer parameters than the full regression network, this design leads to a substantial reduction in computational cost and results in significantly improved efficiency for our algorithm.\n> + **Our approach demonstrates highly competitive efficiency compared to recent SSR methods.** To verify this, we have quantified the training cost in **Table 5 of the main text (also shown below)**, which reports the average training time per iteration and GPU memory consumption on UTKFace, measured using an NVIDIA GeForce RTX 4090. As shown, our method **incurs only ~17MB additional GPU memory and less than 9ms extra training time per iteration compared to the Baseline**.   Moreover, our method outperforms recent SSR approaches [2,4,5] in both accuracy (Tables 1–3) and efficiency, demonstrating a favorable trade-off between performance and cost.\n> + Besides, **in the test stage our method requires no extra overhead compared with the Baseline model.**\n| Method | Time (ms) | GPU (MB) |\n|-|-|-|\n| Baseline | 83.0 | 5099 |\n| SimRegMatch [5] | 548.1 | 7419 |\n| UCVME [2] | 257.2  | 10057 |\n| RankUp [4] | 108.6 | 7433  |\n| Ours| **91.6** | **5116** |\n|\n\n**W2.** The use of pseudo-labels has already been extensively studied in related fields, such as weakly supervised learning.\n> We agree that pseudo-labeling has been widely studied in related areas, such as weakly supervised learning (WSL). However, our work differs from these prior studies in several important aspects:\n>  + **Application.** Most existing pseudo-labeling approaches focus on classification tasks. In contrast, our method tackles the more challenging SSR problem, where pseudo-labeling introduces additional complexity due to the continuous output space and the lack of natural thresholds to assess their reliability. To the best of our knowledge, uncertainty-aware pseudo-labeling for regression remains underexplored.\n>  + **Methodology.** Our method propose a bi-level SSR framework that explicitly optimizes the uncertainty of pseudo-labeled samples to improve the regression model's generalization when learning from potentially incorrect pseudo-labels.  In contrast to WSL methods such as Partial-Label Regression (PLR) [1],  which directly compute the prediction loss using the pseudo-label and a candidate label set,  our approach focuses on modeling pseudo-label uncertainty and dynamically adjusts the influence of pseudo-labels through a bi-level optimization process.  Furthermore, many pseudo-labeling techniques developed for SSL classification, such as the classic FixMatch, rely on fixed confidence thresholds and therefore cannot be directly applied to SSR settings due to the absence of a natural thresholding mechanism in regression tasks.  \n> + **Performance.** In our comparison experiments (Table 1-3 of the main text), we include several representative pseudo-labeling methods such as SimRegMatch [5] and UCVME [2]. Our method consistently outperforms these baselines on both image and text datasets. For example, on the IMDB_WIKI dataset (Table 2), our method outperforms UCVME method by 5.7% for MAE and 4.9\\% for $R^2$.  These results confirm the effectiveness and robustness of our proposed uncertainty-based pseudo-labeling technique.  \n>\n> We appreciate your suggestion and will include a more comprehensive discussion of pseudo-labeling methods, including [1], in the Related Work section of the revised manuscript. \n\n**W3.** The experiments only consider the cases where $\\gamma$= 5%, 10% and 20%, lacking results for larger a values.\n> + **In semi-supervised learning, a core assumption is that the amount of labeled data is significantly smaller than that of unlabeled data.** Accordingly, our experiments focus on low-label regimes with $\\gamma$= 5%, 10% and 20%, which are commonly adopted in prior SSR studies [2, 3, 4] to reflect realistic scenarios. These settings allow us to evaluate the effectiveness of pseudo-labeling and uncertainty modeling under challenging conditions where they are most needed. Following this paradigm, we align our experimental setup with the commonly adopted settings in SSR [2, 3, 4].\n> +  **In response to the concern, we have added experiments with $\\gamma=$40\\% on the IMDB_WIKI dataset.** The results are presented in the table below and show that our method remains effective even under a higher labeled data ratio. Moreover, the relatively smaller performance gains in this setting are expected, as the performance of SSR models naturally approaches that of the fully supervised model (the upper bound model), when a large portion of the training data is labeled.  \n\n| Method | Fully-Supervised| UCVME [2] | Rankup [4] | CLSS [3] | Ours |\n|-|-|-|-|-|-|\n| MAE | 7.974 ± 0.043 | 8.123 ± 0.086 | 8.072 ± 0.030 | 8.602 ± 0.066 | **8.037 ± 0.051** |\n| $R^2$| 0.724 ± 0.002 | 0.713 ± 0.002| 0.714 ± 0.003 | 0.693 ± 0.003| **0.717 ± 0.001**|\n|\n\n**Q1.** Since pseudo-labels are the predictions of the current model, is the proposed method similar to the progressive identification method in [1]？\n>  We respectfully argue that the progressive identification method (PIM) significantly differs from ours in both problem setting and methodology as follows.\n>  + **Problem Setting.** The PIM method is designed to the Partial-Label Regression, which allows each training example to be annotated with a set of candidate labels. In contrast, in the SSR setting considered in our work, each unlabeled sample is assigned a single pseudo-label predicted by the model, resulting in a one-to-one correspondence between sample and pseudo-label.\n> + **Methodology.** While both methods adopt a form of loss weighting, the underlying mechanisms differ. PIM computes the predictive loss as a weighted sum over candidate labels, where labels with smaller losses receive higher weights via a softmax function. **In contrast, our approach employs an uncertainty learner to estimate the uncertainty of each pseudo-label, and down-weights samples with higher uncertainty during training.**\n> + **Performance.**  To further address your concern, we attempted to adapt the PIM method to the SSR setting. Since PIM cannot be directly applied to SSR problems due to the absence of candidate label sets for unlabeled data, we constructed a set of pseudo-label candidates for each unlabeled sample using Monte Carlo Dropout and applied a softmax-based weighting to compute the PIM loss. As shown in the table below, **our method outperforms the adapted PIM.**   \n>\n> We will include a detailed discussion and comparison of PIM in the revised version. \n\n| Method | 5% |  | 10% | | 20% | |\n|-|-|-|-|-|-|-|\n| | MAE | $R^2$ | MAE | $R^2$| MAE | $R^2$|\n| PIM [1] | 9.345 ± 0.100 | 0.659 ± 0.005 | 8.691 ± 0.021 | 0.691 ± 0.001 | 8.441 ± 0.014 | 0.704 ± 0.001 |\n| Ours  |**9.177 ± 0.061** | **0.664 ± 0.003**  | **8.539 ± 0.065**  | **0.695 ± 0.003**  |**8.166 ± 0.071**| **0.712 ± 0.002** |\n|\n\n**Q3.** Could it be further improved by assigning pseudo-labels only to confident samples?\n> We agree that assigning pseudo-labels only to confident samples can further improve performance. **To verify this, we conducted a fine-tuning experiment using only high-confidence pseudo-labeled samples on IMDB_WIKI under $\\gamma$=10%.**\n> + Specifically, we defined a threshold at the $K\\%$-th percentile of the uncertainty distribution over all unlabeled samples. Samples with uncertainty above this threshold were discarded. \n> + The results in the table below shows that performance gradually improves as $K$ increases from 20 to 60, and surpasses the results reported in the paper. **This supports our hypothesis that fine-tuning on confident pseudo-labeled data can refine the model and improve generalization.**\n> + However, to ensure fair comparison with other baseline methods, we did not apply fine-tuning in the main experiments.\n\n| K=| 20 | 30 | 40 | 50 | 60 | 70 | Ours |\n|-|-|-|-|-|-|-|-|\n| MAE | 8.734 | 8.593 | 8.570 | 8.530 | **8.451** | 8.503 | 8.539 |\n| $R^2$ | 0.692 | 0.696 | 0.698 | 0.697 | **0.700** | 0.697 | 0.695 |\n---\n**Reference:**  \n[1] Partial-Label Regression. In AAAI 2023.  \n[2] Semi-supervised deep regression with uncertainty consistency and variational model ensembling via bayesian neural networks. In AAAI, 2023.  \n[3] Semi-supervised contrastive learning for deep regression with ordinal rankings from spectral seriation. In NeurIPS, 2023.  \n[4] Rankup: Boosting semi-supervised regression with an auxiliary ranking classifier. In NeurIPS, 2024.  \n[5] Deep semi-supervised regression via pseudo-label filtering and calibration. In Applied Soft Computing, 2024.  \n[6] Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.  \n[7] Learning to reweight examples for robust deep learning. In ICML, 2018."}, {"review_id": "X6yJIngJMl", "reviewer": "Reviewer_DUVZ", "summary": "This paper tackles the problem of learning from noisy pseudo-labels in semi-supervised regression. The authors observe that incorrect pseudo-labels with high loss can lead to overestimated uncertainty, which suppresses both noisy and difficult-but-correct samples. To address this, they introduce a bi-level optimization framework. In the inner loop, the regression model is trained using a heteroscedastic loss where the uncertainty for each pseudo-label is predicted by a lightweight auxiliary network. In the outer loop, this uncertainty learner is updated by minimizing the supervised loss on a separate batch of labeled data, effectively treating it as a hyperparameter. The goal is to adjust uncertainty estimates in a way that selectively downweights incorrect pseudo-labels without penalizing informative ones. Empirically, the method achieves consistent improvements across three semi-supervised regression benchmarks (UTKFace, IMDB-WIKI, and STS-B), particularly under low-label regimes (5% - 10%).", "questions": "Could strategies such as thresholding, warm-up, or moving average be incorporated to prevent the early suppression of reliable pseudo-labels? Have the authors considered or tested such techniques?\n\nIs the bi-level structure essential, or could similar effects be achieved with dynamic reweighting or teacher-student frameworks? A justification or ablation would help validate the design choice.\n\nHow does the proposed method compare with approaches like UPS or CADR in terms of reweighting or selection mechanisms? A clearer conceptual or empirical comparison would strengthen the contribution.\n\nIf these issues are carefully addressed, I would consider raising my score.", "limitations": "yss", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The paper presents a clear motivation and formulates the challenge of noisy pseudo-label suppression in semi-supervised regression with solid theoretical grounding.\n\n2. The proposed bi-level uncertainty optimization framework is modular, lightweight, and integrates well with existing regression pipelines.\n\n3. Experimental results show consistent improvements on three benchmarks under low-label settings, which demonstrates the general effectiveness of the method.\n\n**Weaknesses**\n1. The uncertainty learner (UL) performs worse than the base regression model in ablation studies and the explanation given is that UL cannot effectively identify hard but correct samples due to the lack of outer-loop supervision. However, this reasoning is not fully explored. The paper does not consider whether dynamic thresholding strategies such as moving thresholds or soft reweighting could reduce the risk of suppressing informative pseudo-labels during early training.\n\n2. Following 1, while the method relies on uncertainty-aware weighting, it remains unclear whether any thresholding, warm-up, or moving average mechanisms can be used to prevent the early elimination of correct pseudo-labels. This leaves open the question of whether the bi-level design is necessary or if similar benefits could be achieved through simpler mechanisms such as dynamic reweighting or teacher-student frameworks.\n\n3. The comparison with related methods such as UPS[1] and CADR[2] is insufficient. UPS uses Monte Carlo Dropout to estimate uncertainty and filters pseudo-labels through thresholding. CADR models sample-specific prediction errors and reweights training loss to address missing-not-at-random biases. Both approaches reflect the idea that error arises from the output but is structured by the input. Although the proposed method differs in structure by using bi-level optimization, the paper would benefit from a more detailed comparison of how different methods reweight or suppress pseudo-labels.\n\n[1] IN DEFENSE OF PSEUDO-LABELING: AN UNCERTAINTY-AWARE PSEUDO-LABEL SELECTION FRAMEWORK FOR SEMI-SUPERVISED LEARNING\n[2] ON NON-RANDOM MISSING LABELS IN SEMI-SUPERVISED LEARNING", "was_revised": true, "final_justification": "The authors provided extensive clarifications and additional experiments that convincingly addressed my key concerns regarding dynamic thresholding, soft reweighting, warm-up strategies, and comparisons with closely related methods (UPS and CADR). Given these comprehensive improvements and additional insights provided during the rebuttal, I confidently raise my rating from 3 to 4, recognizing the enhanced clarity, strengthened methodology, and improved empirical validation.", "rebuttal": "We appreciate very much your constructive comments on our paper. Please kindly find our response to your comments below. We hope that our response satisfactorily addresses the issues you raised. Please feel free to let us know if you have any additional concerns or questions.\n\n**W1. & Q2.** The paper does not consider whether dynamic thresholding strategies such as moving thresholds or soft reweighting could reduce the risk of suppressing informative pseudo-labels during early training.\n> Thanks for pointing this out. As recommended, we conducted additional experiments using both **moving threshold and soft reweighting strategies**, with the corresponding results are presented below.\n>  - (**Implementation Details**). **As for Moving Threshold Strategy**, we have compared with a dynamic thresholding approach SimRegMatch [4] in Table 2 of the main text (also shown below), where the threshold is updated after each training iteration and set to the 𝐾-th percentile of uncertainty values computed from the current batch of unlabeled data; **For Soft Reweighting**, we follow the method in [5],  treating per-sample uncertainty $\\sigma_j^2$ as a learnable parameter to weight the regression loss; The term \"Baseline\" in the table refers to omitting the uncertainty learner and using a fixed uncertainty $\\sigma^2_j = 1$ for all unlabeled samples, as discussed in Lines 325–327 of Section 4.3.\n>  - (**Results Analysis**). As shown in the table, the **moving threshold method underperforms compared to both the baseline and our proposed approach**. We attribute this to **its inherently hard-thresholding nature**: although it dynamically updates the threshold and uses a Top-K selection strategy, it tends to discard many potentially informative pseudo-labels. **Soft reweighting performs better than the moving threshold method but is still inferior to our approach.** Our uncertainty-aware method can actually be viewed as a form of soft reweighting, since it continuously adjusts each sample’s loss contribution based on estimated uncertainty. However, there is a fundamental difference in how the weights are generated. Soft reweighting directly learns sample-wise weights, which may lead to instability and overfitting, especially during early training, whereas our approach adaptively determines the weights, leading to more stable and robust performance.\n>\n| Method| 5%  |        | 10%       |        | 20%       |        |\n|--------------------|----------|--------|----------|--------|----------|--------|\n|                    | MAE      | $R^2$     | MAE      | $R^2$     | MAE      | $R^2$     |\n| Baseline            | 9.512 ± 0.076     | 0.651 ± 0.005 | 8.864  ±  0.083 | 0.683 ± 0.004 | 8.605 ± 0.060  | 0.696 ±  0.002 |\n| SimRegMatch [4] | 9.908 ± 0.097  | 0.628 ± 0.004 | 9.110 ± 0.166  | 0.665 ± 0.007 | 8.587 ± 0.094  | 0.693 ± 0.006|\n| Soft Reweighting [5]   | 9.475 ±  0.036  | 0.652 ± 0.001| 8.779 ± 0.029  | 0.686 ± 0.002| 8.399 ± 0.058 | 0.706 ± 0.003 |\n| Ours               | **9.177 ± 0.061**  | **0.664 ± 0.003**| **8.539 ±  0.065** | **0.695 ±  0.003**| **8.166 ± 0.071** | **0.712 ± 0.002** |\n|\n\n**W2. & Q1.** Following 1, while the method relies on uncertainty-aware weighting, it remains unclear whether any thresholding, warm-up, or moving average mechanisms can be used to prevent the early elimination of correct pseudo-labels. This leaves open the question of whether the bi-level design is necessary or if similar benefits could be achieved through simpler mechanisms such as dynamic reweighting or teacher-student frameworks.\n\n> Thank you for your thoughtful feedback. To address your concerns, we conducted additional experiments incorporating **warm-up, and moving average mechanisms**, in order to further assess the necessity of our bi-level design.\n> - (**Implementation Details**). For the warm-up strategy, following [3], **we trained the model exclusively on labeled data during the initial epochs** to mitigate premature filtering of pseudo-labels. For the moving average mechanism, we **smooth model parameters over training iterations** [6].\n> - (**Results Analysis**). From the following table, we can observe that **our proposed method significantly outperforms both the warm-up and moving average methods** across multiple settings, highlighting the effectiveness and necessity of the bi-level design.\n>\n| Method             | 5%              |        | 10%             |        | 20%             |        |\n|--------------------|----------------|--------|----------------|--------|----------------|--------|\n|                    | MAE            | $R^2$     | MAE            | $R^2$     | MAE            | $R^2$     |\n| Warm-up [3]           | 9.426 ± 0.006  | 0.656 ± 0.003 | 8.904 ± 0.017  | 0.682 ± 0.004 | 8.531 ± 0.025  | 0.700 ± 0.002 |\n| Moving Average [6]      | 10.057 ± 0.056 |0.614 ± 0.003 |9.105 ± 0.049| 0.662 ± 0.003|8.502 ± 0.029 | 0.691 ± 0.002 |\n| Ours               | **9.177 ± 0.061**  | **0.664 ± 0.003** | **8.539 ± 0.065**  | **0.695 ± 0.003** | **8.166 ± 0.071**  | **0.712 ± 0.002** |\n|\n\n**W3. & Q3.** The comparison with related methods such as UPS[1] and CADR[2] is insufficient. UPS uses Monte Carlo Dropout to estimate uncertainty and filters pseudo-labels through thresholding. CADR models sample-specific prediction errors and reweights training loss to address missing-not-at-random biases. Both approaches reflect the idea that error arises from the output but is structured by the input. Although the proposed method differs in structure by using bi-level optimization, the paper would benefit from a more detailed comparison of how different methods reweight or suppress pseudo-labels.\n\n> Thanks for your valuable comment. UPS [1] and CADR [2] were originally proposed for **semi-supervised classification tasks and are not directly applicable to regression problems**. Therefore, we did not include them in our initial comparison.\nFollowing your suggestion, **we have now implemented and adapted these methods to our semi-supervised regression setting**.\n>\n> - **UPS** uses Monte Carlo Dropout to estimate uncertainty via the variance of multiple forward passes, and applies a threshold to identify low-confidence predictions as negative labels, thereby enabling negative learning. In contrast, regression inherently deals with continuous outputs. and defining negative targets is nontrivial. **To fairly adapt UPS to regression, we only use Monte Carlo Dropout to estimate pseudo-label uncertainty, but without negative labeling (denoted as UPS_Reg).**\n>\n> - **CADR** was designed to address the challenge of non-random (i.e., biased) label missingness in classification tasks. It achieves this by leveraging the prior distribution of labeled samples to perform inverse probability weighting (CAP module) and  adjusting per-class selection thresholds for unlabeled samples (CAI module). **Since such class-specific confidence modeling cannot be directly applied to continuous targets, we approximate this by discretizing the regression target into age bins, treating each bin as a class.**\n>\n> Based on these adaptations, we have conducted direct comparisons with both UPS and CADR in the following table. **Our method consistently achieves better performance across various settings.** \n>   - Compared with UPS_Reg, while both methods incorporate uncertainty, our method uses uncertainty-aware soft weighting within a bi-level optimization framework, yielding more stable and accurate uncertainty estimation. \n>   - The inferior performance of CADR highlights the challenge of directly adapting classification-based methods to regression tasks. \n>   - Thanks again for this suggestion, we will include the results and discussions of these comparison baselines in the revised manuscript. \n\n| Method     | 5%               |        | 10%              |        | 20%              |        |\n|------------|-----------------|--------|------------------|--------|------------------|--------|\n|            | MAE             | $R^2$    | MAE              | $R^2$     | MAE              | $R^2$    |\n| UPS_Reg [1]    | 9.430 ± 0.087   | 0.647 ± 0.006 | 8.845 ± 0.055   | 0.676 ± 0.003 | 8.415 ± 0.002   | 0.699 ± 0.001 |\n| CADR [2]      | 10.592 ± 0.091  | 0.562 ± 0.009 | 9.531 ± 0.064   | 0.622 ± 0.007 | 8.536 ± 0.056   | 0.668 ± 0.008 |\n| Ours       | **9.177 ± 0.061**   | **0.664 ± 0.003** | **8.539 ± 0.065**   | **0.695 ± 0.003** | **8.166 ± 0.071**   | **0.712 ± 0.002** |\n\n---\n**Reference:**  \n[1] Rizve, Mamshad Nayeem, et al. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In ICLR, 2021.  \n[2] On non-random missing labels in semi-supervised learning. In ICLR, 2021.  \n[3] FixMatch: Simplifying semi-supervised learning with consistency and confidence. In NeurIPS, 2020.  \n[4] Deep semi-supervised regression via pseudo-label filtering and calibration. In Applied Soft Computing, 2024.  \n[5] What uncertainties do we need in bayesian deep learning for computer vision? In NeurIPS, 2017.  \n[6] Acceleration of Stochastic Approximation by Averaging. In SIAM J. Control Optim, 1992."}, {"review_id": "Ahkkt2bSwW", "reviewer": "Reviewer_LePr", "summary": "This paper studies pseudo-labeling in the semi-supervised regression (SSR) setting.\nIn this context, the pseudo-labels are modeled as heteroscedastic noisy observations of the true ones.\nAs the authors point out, unlike in classification, naive training with noisy pseudo-labels in SSR may lead to poor results due to the continuous nature of the outputs.\nTo address this, the authors propose a bi-level optimization framework that: (i) estimates the noise level of each pseudo-labeled sample via a neural network $g_\\phi$, and (ii) construct an accurate predictor $f_\\theta$ using both the labeled and pseudo-labeled samples with appropriate weighting.\nThey also present numerical experiments demonstrating the advantages of their approach compared to existing methods.", "questions": "* Is the noisy-label framework introduced in Equation (1) used in prior works? If so, it would be helpful to mention and cite them.\n \n* Learning with noisy labels is a well-studied area. In particular, in some cases, training directly on noisy samples can still yield good generalization (e.g., linear regression [1]). It would be interesting to better understand whether the proposed scheme achieves improved generalization compared to naive training with noisy labels, even for simple model classes such as linear predictors.\n\n\n[1] Bartlett, P.L., Long, P.M., Lugosi, G. and Tsigler, A., 2020. Benign overfitting in linear regression.\n\n\n## Minor comments:\n* Line 121: Did you nean *noise level*? \n* Algorithm 1, line 7: replace $i$ with $k$\n* Experimental setups 4.1: it would be better to define $\\gamma$ explicitly in the text\n* Line 119: \"The objective is to learn the *true* label function $f_\\theta (x)$..\" . Does it mean that $y \\stackrel{\\text{a.s.}}{=}f_\\theta(x)$?  If so, it may be better to denote the ground truth function by $f*$ (or $f_{\\theta^*}$) to avoid confusion with the learned predictor.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths:\n* Pseudo labeling in SSR setting is very intesrting research area.\n* The authors suggest an intersting idea as bi-level optimization approach to deal with the uncertienty of the pseudo labels\n* The paper includes extensive experiments.\n\n# Weaknesses:\n* Noisy model: The authors modeld the psuedo-labels as noisy observations. However, in practice, pseudo-labels are typically generated by model (either the one trained over the labeled data or an auxiliary weaker model). Therefore, the proposed framework may not capture this case. In particular, when two unlabeled samples $x_1,x_2$ are close, we might expect their pseudo-labels to also be close, a property not directly reflected in the proposed noise model.\n\n\n* Theoretical results: Although this paper is primarily methodological, the theoretical results are rather limited and do not provide sufficient insight. It would be interesting to understand whether the proposed method reduces the sample complexity in terms of labeled data, or to characterize the noise regimes in which the method is expected to perform well, even for linear regression.\n\n* Uncertainty Learner: A key component of the method is estimating the noise level of each sample by $g_\\phi$. However, it only mentioned that $g_\\phi$ is an auxiliary network (L. 143).  I encourage the authors to expand this discussion. For example, what are classical approaches for estimating noise levels in noisy labels? What architectural choices are typically used for networks estimating label noise?", "was_revised": true, "final_justification": "The authors addressed my main concerns.", "rebuttal": "We sincerely thank you for yur valuable feedback. We hope that our response satisfactorily addresses the issues you raised. Please feel free to let us know if you have any additional concerns or questions.\n\n**W1.** Noisy model: The authors model the pseudo-labels as noisy observations. However, in practice, pseudo-labels are typically generated by model (either the one trained over the labeled data or an auxiliary weaker model). Therefore, the proposed framework may not capture this case. In particular, when two unlabeled samples $x_1$, $x_2$ are close, we might expect their pseudo-labels to also be close, a property not directly reflected in the proposed noise model.\n> We thank the reviewer for raising this insightful concern. While it is true that pseudo-labels are generated by a model rather than directly observed, we respectfully argue that our framework implicitly enforces a smoothness property on the generated pseudo-labels. We support this claim with both theoretical and empirical evidence:\n> + **Theoretical smoothness guarantee.** The Lipschitz continuity of both the regression model $ f_\\theta $ and the uncertainty estimator $ g_\\phi $ ensures that nearby unlabeled inputs yield similar pseudo-labels and uncertainty estimates. Specifically, for any two unlabeled samples $x_1^u $ and $ x_2^u$, we have: $|\\hat y_1 - \\hat y_2| \\leq L_f \\|x_1^u - x_2^u\\|$ and $|\\sigma_1^2 - \\sigma_2^2| \\leq L_g L_f \\|x_1^u - x_2^u\\|, $ where $L_f$ and $L_g$ are the Lipschitz constants of $f_\\theta$ and $g_\\phi$, respectively. These constants can be effectively controlled by architectural design. For example, we can apply spectral normalization to $f_\\theta$ to limit its Lipschitz constant, and we design $g_\\phi$ as a shallow MLP with ReLU activations and small-initialized weights to promote smoothness.\n> + **Empirical evidence.** To further validate the smoothness of pseudo-labels in practice, we conducted a variance analysis across age groups on the IMDB_WIKI dataset. Specifically, we computed the standard deviation of predicted pseudo-labels among samples that share the same ground-truth age and compared the results with UCVME [1], which explicitly enforces local consistency via consistency loss. As reported in the table below, our method consistently yields lower or comparable standard deviations than UCVME in most age groups. This suggests that our pseudo-labels are more stable and consistent across similar inputs, thereby indirectly capturing the desired smoothness prior.\n>\n> We hope this analysis clarifies that our framework, despite modeling pseudo-labels as noisy observations, inherently incorporates the smoothness property expected from realistic model-generated pseudo-labels.  We will add this discussion in the revised version.\n\n| Method   | Age=10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 |\n|-|-|-|-|-|-|-|-|-|-|\n| UCVME [1] | 10.623| 6.522 | 5.009 | 5.878 | 7.772 |10.666 |12.936 |13.085 | 5.017 |\n| Ours  | 3.290 | 6.389 | 6.157 | 6.971 | 8.558 |10.231 |10.301 | 6.364 | 4.011 |\n|\n\n**W2. & Q2.** It would be interesting to understand whether the proposed method reduces the sample complexity in terms of labeled data, or to characterize the noise regimes in which the method is expected to perform well, even for linear regression.\n> We thank the reviewer for raising this important point. While our work is primarily methodological, we agree that offering theoretical insights is valuable. Below, we outline a preliminary theoretical result that helps characterize how our uncertainty-aware pseudo-labeling affects generalization.\n> + **Notations.** We consider the standard regression setting with true risk defined as $ R(f) = \\mathbb E_{(x, y)}[\\ell(f(x), y)],$ and empirical risk $\\widehat R(f) = \\widehat R_l(f) + \\widehat R_u(f),$ where $\\widehat R_l(f) = \\frac{1}{N} \\sum_{i=1}^{N} \\ell(f(x_i^l), y_i^l)$ and $ R_u(f) = \\frac{1}{M} \\sum_{j=1}^{M} \\ell(f(x_j^u), y_j^u).$ Since unlabeled targets $y_j^u$ are inaccessible, we train with pseudo-labels $\\hat y_j^u$ by minimizing $\\widetilde R_u(f) = \\frac{1}{M} \\sum_{j=1}^{M} \\ell(f(x_j^u), \\hat y_j^u).$ Let $\\mathcal F$ be the hypothesis space, and $\\mathcal R_{N+M}$ be its expected Rademacher complexity over $N+M$ training samples. Let $\\hat f = \\arg\\min_{f \\in \\mathcal F} \\widehat R(f)$ be the empirical risk minimizer, and $f^* = \\arg\\min_{f \\in \\mathcal{F}} R(f)$ be the true minimizer. Our inner-loop loss can be denoted as $\\mathcal L^{inner}=\\frac{1}{N}\\sum_{i=1}^{N}\\ell(f(x_i), y_i)+\\frac{1}{M}\\sum_{j=1}^M\\frac{1}{\\sigma_j^2}\\ell(f(x_j),\\hat y_j)$.\n>+ **Theoretical Understanding.** Under the assumptions that the loss $\\ell(f(x), y)$ is $\\mu$-Lipschitz and bounded by $B$, and for unlabeled sample $x_j^u$, that the pseudo-label follows $\\hat y_j \\sim \\mathcal{N}(f(x_j), \\sigma_j^2)$ and the model prediction of $f(x_j)$ lies between its true label $y_j$ and pseudo-label $\\hat y_j$ as a weighted average, we can show that, for any $\\delta>0$, with probability at least $1 - \\delta$, it satisfies: $|R(\\hat f) - R(f^*)| \\leq \\frac{1}{M}\\sum_{j=1}^{M} \\frac{|1 - \\sigma_j|}{1 + \\sigma_j^2}(y_j - \\hat y_j)^2 + 2\\mu \\mathcal{R}_{N+M} + B \\sqrt{\\frac{\\log(1/\\delta)}{2(N+M)}}.$ The first term illustrates that the generalization error is partially governed by the pseudo-label quality and the associated uncertainty $\\sigma_j^2$. Specifically, larger uncertainties leads to down-weighting the contribution of high-noise pseudo-labels, while reliable pseudo-labels with lower uncertainty receive higher weight. Hence, learning uncertainty-aware weights provides a theoretically grounded mechanism to mitigate the effect of noisy pseudo-labels and better align the empirical risk with the true target loss.\n>\n> We believe this lays the foundation for deeper investigations into sample complexity and robustness under different noise regimes, which we plan to pursue in future work. \n\n**W3.**  Expand the discussion about uncertainty-learner. For example, what are classical approaches for estimating noise levels in noisy labels? What architectural choices are typically used for networks estimating label noise?\n\n> As suggestion, we will add a more detailed discussion of the noise levels estimation module would as follows. \n> + Estimating sample-wise noise or uncertainty is a well-studied problem. Classical methods can be broadly categorized as follows:\n>    + **Bayesian-inspired methods.** These methods treat outputs as distributions rather than point estimates, often using dropout as a variational Bayesian approximation. Notable works applying dropout to estimate noise or uncertainty include SimRegMatch [4], the method proposed in reference [1], as well as UA-MT [5] and UMCT [6], which use Monte Carlo dropout to quantify pseudo-label uncertainty.\n>    + **Co-train methods.** These methods quantified uncertainty through prediction differences between co-trained models. For example, the method [7] and [8] estimate uncertainty based on prediction discrepancies.\n>    + **Non-parametric kernel methods.** Such methods do not assume a parametric form of the predictive distribution. SSDKL [3] minimizes predictive variance within a posterior regularization framework, combining neural networks for representation learning with Gaussian processes for probabilistic regression.\n>    + **Heteroscedastic regression.** These methods assume that the noise level varies with input samples. For example, UCVME [1] employs a consistency loss under the heteroscedastic assumption to directly learn the noise in data labels through the model.\n> + Other architectural choices for networks estimating label noise include: UCVME [1] adopts a shared encoder with dual heads to predict the mean and variance of the noise distribution, while SSDKL leverages Gaussian Processes to estimate the predictive variance for unlabeled samples.\n> \n> We will strengthen the corresponding analysis and discussion in the Related Work section.\n\n**Q1.** Is the noisy-label framework introduced in Equation (1) used in prior works? If so, it would be helpful to mention and cite them.\n> We thank the reviewer for this helpful question. The formulation in Equation (1), where pseudo-labels are modeled as noisy observations with sample-dependent variance, is inspired by classical heteroscedastic regression modeling [2]. Our contribution lies in extending this well-established idea to the semi-supervised setting, where the variance is learned jointly with the pseudo-labels and plays a central role in guiding uncertainty-aware learning on unlabeled data. We will revise the manuscript to explicitly cite this related work and clarify the connection to prior literature.\n\n**M1.** Line 121: Did you mean noise level?\n> Many thanks for pointing out this unclear description. Line 121 mainly refers to the level of uncertainty or noise in the pseudo-labels varies across samples. We will provide a clearer explanation of this issue in the revised version.\n\n**M2. & M3. & M4.**\n> Manys thanks for the careful check of details. We will fix these errors in the revised version.\n\n---\n**Reference:**  \n[1] Semi-supervised deep regression with uncertainty consistency and variational model ensembling via bayesian neural networks. In  AAAI, 2023.  \n[2] Pattern recognition and machine learning. In New York: springer, 2006.  \n[3] Semi-supervised deep kernel learning: Regression with unlabeled data by minimizing predictive variance.  In NeurIPS, 2018.  \n[4] Deep semi-supervised regression via pseudo-label filtering and calibration. In Applied Soft Computing, 2024.  \n[5] Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation. In MICCAI, 2019.  \n[6] 3d semi-supervised learning with uncertainty-aware multi-view co-training. In CVPR, 2020.  \n[7] Enhancing pseudo label quality for semi-supervised domain-generalized medical image segmentation. In AAAI, 2022.  \n[8] Calibrating label distribution for class-imbalanced barely-supervised knee segmentation. In ICML, 2022."}]

Author rebuttal:

Meta-review: The paper introduces an uncertainty-aware pseudo-labeling framework for semi-supervised regression. It tackles the challenge of unreliable continuous pseudo-labels under heteroscedastic noise. Using a bi-level optimization strategy, it dynamically adjusts pseudo-label influence by jointly minimizing empirical risk and optimizing uncertainty estimates for better generalization. After the rebuttal and discussions between the reviewers and authors, the paper received the final scores of A, BA, BA, BA. The reviewers largely agreed that the paper makes a substantial and novel methodological contribution, with strong empirical validation and useful theoretical work. The strengths include a clear motivation, robust bi-level pseudo-labeling design, and strong results on several benchmarks. The remaining weaknesses are that the theoretical analysis is still limited, offering only preliminary insights rather than a deep characterization of when and why the method works best; the noise modeling assumption may not fully capture real pseudo-label generation; and some reviewers felt the role and design of the uncertainty learner could have been discussed in more depth. However, these weaknesses were not considered significant.
Decision: Accept (poster)