Title: Semi-Supervised Regression with Heteroscedastic Pseudo-Labels
Venue: NeurIPS 2025
Accepted: yes
Keywords: Semi-supervised regression

Pseudo-labels

Heteroscedasticity
Abstract: Pseudo-labeling is a commonly used paradigm in semi-supervised learning, yet its application to semi-supervised regression (SSR) remains relatively under-explored. Unlike classification, where pseudo-labels are discrete and confidence-based filtering is effective, SSR involves continuous outputs with heteroscedastic noise, making it challenging to assess pseudo-label reliability. As a result, naive pseudo-labeling can lead to error accumulation and overfitting to incorrect labels. To address this, we propose an uncertainty-aware pseudo-labeling framework that dynamically adjusts pseudo-label influence from a bi-level optimization perspective. By jointly minimizing empirical risk over all data and optimizing uncertainty estimates to enhance generalization on labeled data, our method effectively mitigates the impact of unreliable pseudo-labels. We provide theoretical insights and extensive experiments to validate our approach across various benchmark SSR datasets, and the results demonstrate superior robustness and performance compared to existing methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies pseudo-labeling in the semi-supervised regression (SSR) setting.\nIn this context, the pseudo-labels are modeled as heteroscedastic noisy observations of the true ones.\nAs the authors point out, unlike in classification, naive training with noisy pseudo-labels in SSR may lead to poor results due to the continuous nature of the outputs.\nTo address this, the authors propose a bi-level optimization framework that: (i) estimates the noise level of each pseudo-labeled sample via a neural network $g_\\phi$, and (ii) construct an accurate predictor $f_\\theta$ using both the labeled and pseudo-labeled samples with appropriate weighting.\nThey also present numerical experiments demonstrating the advantages of their approach compared to existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n* Pseudo labeling in SSR setting is very intesrting research area.\n* The authors suggest an intersting idea as bi-level optimization approach to deal with the uncertienty of the pseudo labels\n* The paper includes extensive experiments.\n\n# Weaknesses:\n* Noisy model: The authors modeld the psuedo-labels as noisy observations. However, in practice, pseudo-labels are typically generated by model (either the one trained over the labeled data or an auxiliary weaker model). Therefore, the proposed framework may not capture this case. In particular, when two unlabeled samples $x_1,x_2$ are close, we might expect their pseudo-labels to also be close, a property not directly reflected in the proposed noise model.\n\n\n* Theoretical results: Although this paper is primarily methodological, the theoretical results are rather limited and do not provide sufficient insight. It would be interesting to understand whether the proposed method reduces the sample complexity in terms of labeled data, or to characterize the noise regimes in which the method is expected to perform well, even for linear regression.\n\n* Uncertainty Learner: A key component of the method is estimating the noise level of each sample by $g_\\phi$. However, it only mentioned that $g_\\phi$ is an auxiliary network (L. 143).  I encourage the authors to expand this discussion. For example, what are classical approaches for estimating noise levels in noisy labels? What architectural choices are typically used for networks estimating label noise?\n\nQUESTIONS:\n* Is the noisy-label framework introduced in Equation (1) used in prior works? If so, it would be helpful to mention and cite them.\n \n* Learning with noisy labels is a well-studied area. In particular, in some cases, training directly on noisy samples can still yield good generalization (e.g., linear regression [1]). It would be interesting to better understand whether the proposed scheme achieves improved generalization compared to naive training with noisy labels, even for simple model classes such as linear predictors.\n\n\n[1] Bartlett, P.L., Long, P.M., Lugosi, G. and Tsigler, A., 2020. Benign overfitting in linear regression.\n\n\n## Minor comments:\n* Line 121: Did you nean *noise level*? \n* Algorithm 1, line 7: replace $i$ with $k$\n* Experimental setups 4.1: it would be better to define $\\gamma$ explicitly in the text\n* Line 119: \"The objective is to learn the *true* label function $f_\\theta (x)$..\" . Does it mean that $y \\stackrel{\\text{a.s.}}{=}f_\\theta(x)$?  If so, it may be better to denote the ground truth function by $f*$ (or $f_{\\theta^*}$) to avoid confusion with the learned predictor.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of learning from noisy pseudo-labels in semi-supervised regression. The authors observe that incorrect pseudo-labels with high loss can lead to overestimated uncertainty, which suppresses both noisy and difficult-but-correct samples. To address this, they introduce a bi-level optimization framework. In the inner loop, the regression model is trained using a heteroscedastic loss where the uncertainty for each pseudo-label is predicted by a lightweight auxiliary network. In the outer loop, this uncertainty learner is updated by minimizing the supervised loss on a separate batch of labeled data, effectively treating it as a hyperparameter. The goal is to adjust uncertainty estimates in a way that selectively downweights incorrect pseudo-labels without penalizing informative ones. Empirically, the method achieves consistent improvements across three semi-supervised regression benchmarks (UTKFace, IMDB-WIKI, and STS-B), particularly under low-label regimes (5% - 10%).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper presents a clear motivation and formulates the challenge of noisy pseudo-label suppression in semi-supervised regression with solid theoretical grounding.\n\n2. The proposed bi-level uncertainty optimization framework is modular, lightweight, and integrates well with existing regression pipelines.\n\n3. Experimental results show consistent improvements on three benchmarks under low-label settings, which demonstrates the general effectiveness of the method.\n\n**Weaknesses**\n1. The uncertainty learner (UL) performs worse than the base regression model in ablation studies and the explanation given is that UL cannot effectively identify hard but correct samples due to the lack of outer-loop supervision. However, this reasoning is not fully explored. The paper does not consider whether dynamic thresholding strategies such as moving thresholds or soft reweighting could reduce the risk of suppressing informative pseudo-labels during early training.\n\n2. Following 1, while the method relies on uncertainty-aware weighting, it remains unclear whether any thresholding, warm-up, or moving average mechanisms can be used to prevent the early elimination of correct pseudo-labels. This leaves open the question of whether the bi-level design is necessary or if similar benefits could be achieved through simpler mechanisms such as dynamic reweighting or teacher-student frameworks.\n\n3. The comparison with related methods such as UPS[1] and CADR[2] is insufficient. UPS uses Monte Carlo Dropout to estimate uncertainty and filters pseudo-labels through thresholding. CADR models sample-specific prediction errors and reweights training loss to address missing-not-at-random biases. Both approaches reflect the idea that error arises from the output but is structured by the input. Although the proposed method differs in structure by using bi-level optimization, the paper would benefit from a more detailed comparison of how different methods reweight or suppress pseudo-labels.\n\n[1] IN DEFENSE OF PSEUDO-LABELING: AN UNCERTAINTY-AWARE PSEUDO-LABEL SELECTION FRAMEWORK FOR SEMI-SUPERVISED LEARNING\n[2] ON NON-RANDOM MISSING LABELS IN SEMI-SUPERVISED LEARNING\n\nQUESTIONS:\nCould strategies such as thresholding, warm-up, or moving average be incorporated to prevent the early suppression of reliable pseudo-labels? Have the authors considered or tested such techniques?\n\nIs the bi-level structure essential, or could similar effects be achieved with dynamic reweighting or teacher-student frameworks? A justification or ablation would help validate the design choice.\n\nHow does the proposed method compare with approaches like UPS or CADR in terms of reweighting or selection mechanisms? A clearer conceptual or empirical comparison would strengthen the contribution.\n\nIf these issues are carefully addressed, I would consider raising my score.\n\nLIMITATIONS:\nyss", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

comments: SUMMARY:
This paper investigates how to leverage pseudo-labels for semi-supervised regression (SSR). The proposed method is an uncertainty-aware framework that dynamically adjusts the influence of pseudo-labels from a bi-level optimization perspective. Extensive experiments validate the effectiveness of the proposed approach.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well-organized and easy to follow.
2. Investigating how to leverage pseudo-labels in semi-supervised regression is an interesting problem.

Weaknesses:
1. The bi-level optimization mechanism is inefficient.
2. The use of pseudo-labels has already been extensively studied in related fields, such as weakly supervised learning.
3. The experiments only consider the cases where $\gamma$ = 5%, 10%, and 20%, lacking results for larger a values.

QUESTIONS:
1. Since pseudo-labels are the predictions of the current model, is the proposed method similar to the progressive identification method in [1]?

2. How efficient is the bi-level optimization method?

3. The proposed method assigns pseudo-labels to all unlabeled samples. Could it be further improved by assigning pseudo-labels only to confident samples?

[1] X. Cheng et al. Partial-Label Regression. In AAAI 2023.

LIMITATIONS:
N/A.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is on semi-supervised regression (SSR), a scenario where labeled data is limited but unlabeled data is abundant. While pseudo-labeling is widely adopted in semi-supervised classification, it is less explored for regression tasks. The main difficulty stems from the continuous nature of regression targets. Unlike classification, where pseudo-labels are discrete and confidence is easier to quantify, regression must contend with heteroscedastic noise, which makes pseudo-label reliability much harder to assess.\n\nThe authors propose an uncertainty-aware pseudo-labeling framework for SSR. The central idea is to dynamically adjust the influence of pseudo-labeled samples based on their estimated uncertainty. The method employs a bi-level optimization approach. In the inner loop, the regression model is trained on both labeled and pseudo-labeled data, where each pseudo-label is weighted according to its predicted uncertainty. In the outer loop, the parameters that determine the uncertainty are optimized to ensure better generalization on the labeled data.\n\nA lightweight neural network, referred to as the uncertainty-learner, is introduced to estimate the uncertainty for each pseudo-labeled example. This network takes as input both the model's prediction and the pseudo-label for a given sample and produces a log-variance value representing the estimated uncertainty. This enables the model to down-weight unreliable pseudo-labels during training, thereby reducing the risk of error accumulation and overfitting to noisy signals.\n\nThe paper provides a superficial theoretical justification for this approach: The bi-level optimization implicitly aligns the learning gradients from labeled and pseudo-labeled data. MOreover, when the uncertainty estimation is accurate, the model's learning direction is guided toward solutions that generalize well, rather than merely fitting the pseudo-labeled data. This contributes to greater stability and robustness in training.\n\nComprehensive experiments are conducted on three benchmark datasets: UTKFace and IMDB-WIKI for age estimation from images, and STS-B for semantic similarity regression on text pairs. The method is evaluated under different proportions of labeled data, specifically at 5%, 10%, and 20%. Across all settings, the proposed approach consistently outperforms several strong baselines, including both consistency-based and uncertainty-based SSR methods. The gains are especially pronounced when labeled data is scarce.\n\nThe ablation studies further validate the design choices. Removing the uncertainty-learner or not using bi-level optimization leads to a noticeable drop in performance. The experiments also show that the uncertainty estimates produced by the model are well correlated with the actual prediction errors. Samples with higher prediction errors are assigned higher uncertainty, and their influence on model updates is reduced.Regarding efficiency, the proposed method introduces only minimal computational and memory overhead compared to standard baselines. This makes it practical for real-world applications.\n\nIn concusion, the paper presents a principled and effective solution to semi-supervised regression with noisy pseudo-labels. By modeling heteroscedastic uncertainty and optimizing it in a bi-level manner, the approach significantly enhances robustness and accuracy, particularly in data-scarce regimes. The results suggest that careful handling of uncertainty is crucial for the success of SSR, and the framework established here sets a new standard for future research in the field.\n\nSTRENGTHS AND WEAKNESSES:\nI think this is a prime example of a borderline paper. The methodology, in particular the bi-level optimization for learning uncertainty in pseudo-labeling for heteroscedastic SSR, is novel and the experiments speak for themselves. However, the theoretical analysis is pretty slim and not that insightful. Theorem 1 states a mere re-write of the optimizaiton problem. Directly seeing the implicit regularization of the gradients in eq.9 is nice, but also already apparent to the educated reader by eq. 5 and 6. I think the paper would greatly benefit from a deeper theoretical underpinning. (That being said, I want to re-iterate that I really like the methodological ideas and find the experimental results very convincing.) Here is an actionable suggestion of how to add some theoretical motivation: The authors could easily prove (approximate) Bayes-optimality of \\theta^* and \\phi^* by framing the bi-level optimization problem as decision problem with mixed decision rules. Building on related work on uncertainty-aware pseudo-label selection [1-4], in particular [2] which embeds pseudo-label selection in decision theory, it is easy to see that the calibration via $\\sigma^2$ corresponds to a probability weight vector of all actions (=pseudo-labeled data). Let me maybe add some further explanations: the decision-theoretic embedding and Bayes-optimality proves in [2] are concerned with pseudo-label *selection\", where a only a subset of pseudo-labeled data is added to labeled data iteratively (\"self-training\"). I am aware the setup is different here. However, the decision-theoretic embedding still holds by considering mixed-action decision theory, also known as randomized decision theory, which involves making decisions by using a weighting vector (probability distribution) over multiple actions, rather than choosing a single action. The calibration via the $\\sigma^2$ corresponds to exactly such a weighting over all actions (=pseudo-labels). By framing as an adequate decision problem, it can be seen that the proposed method corresponds to the best strategy given all available information and uncertainties (i.e., Bayes action).\n\n[1] Rizve, Mamshad Nayeem, et al. \"In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning.\" International Conference on Learning Representations. 2021.\n\n[2] Rodemann, Julian, et al. \"Approximately Bayes-optimal pseudo-label selection.\" Uncertainty in Artificial Intelligence. 2023.\n\n[3] Rodemann, Julian, et al. \"In all likelihoods: Robust selection of pseudo-labeled data.\" International Symposium on Imprecise Probability: Theories and Applications. 2023.\n\n[4] Dietrich, Stefan, et al. \"Semi-supervised learning guided by the generalized Bayes rule under soft revision.\" International Conference on Soft Methods in Probability and Statistics. 2024\n\nMinor remarks: \n\n- Index in second summand of eq.3 should be j rather than x_j^u, as the latter is not used in the sum. Please also update the index set accordingly to $\\|\\mathcal B_u \\|$\n- redundant use of \"optimization\" in proof of theorem 1: \"irrelevant to optimization of $\\Phi_t$, which can beomitted in optimization\"\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}