Rethinking Confidence Scores and Thresholds in Pseudolabeling-based SSL
Authors: Harit Vishwakarma
Yi Chen
Satya Sai Srinath Namburi GNVV
Sui Jiet Tay
Ramya Korlakai Vinayak
Frederic Sala
Venue: ICML 2025 poster (2025)
Keywords: Semi-supervised Learning
Pseudolabeling
Self-Training
Confidence Functions
Primary area: general_machine_learning->unsupervised_and_semisupervised_learning
TL;DR: We address overconfidence and miscalibration in pseudolabeling-based SSL with a framework for learning scores and thresholds with explicit error control. This boosts pseudolabel quality and quantity, enhancing accuracy and training efficiency in SSL.

Abstract: Modern semi-supervised learning (SSL) methods rely on pseudolabeling and consistency regularization.  Pseudolabeling is typically performed by comparing the model's confidence scores and a predefined threshold. While several heuristics have been proposed to improve threshold selection, the underlying issues of overconfidence and miscalibration in confidence scores remain largely unaddressed, leading to inaccurate pseudolabels, degraded test accuracy, and prolonged training. We take a first-principles approach to learn confidence scores and thresholds with an explicit knob for error. This flexible framework addresses the fundamental question of optimal scores and threshold selection in pseudolabeling. Moreover, it gives practitioners a principled way to control the quality and quantity of pseudolabels. Such control is vital in SSL, where balancing pseudolabel quality and quantity directly affects model performance and training efficiency. Our experiments show that, by integrating this framework with modern SSL methods, we achieve significant improvements in accuracy and training efficiency. In addition, we provide novel insights on the trade-offs between the choices of the error parameter and the end model's performance.

Rethinking Confidence Scores and Thresholds in Pseudolabeling-based SSL
Harit Vishwakarma Yi Chen Satya Sai Srinath Namburi GNVV3 Jiet Tay*
Ramya Korlakai Vinayak 2 Frederic Sala'
Abstract
& Hoos. 2019). While SSL dates back decades and in- cludes a wide variety of approaches, modern SSL methods frequently rely on a pair of ideas: self-training or pseu- dolabeling where model generated labels are assigned to unlabeled data for further training (McLachlan, 1975; Amini et al., 2023; Rosenberg et al., 2005; Lee, 2013; Rizve et al., 2021) and consistency regularization enforce stability in predictions across perturbed inputs (Laine & Aila, 2017; Bachman et al., 2014; Sajjadi et al., 2016; Fan et al., 2021; Kukačka et al., 2017). SSL techniques with these ideas have strongly performed on several benchmark datasets.
Modern semi-supervised learning (SSL) methods rely on pseudolabeling and consistency regular- ization. Pseudolabeling is typically performed by comparing the model's confidence scores and a predefined threshold. While several heuristics have been proposed improve threshold selec- tion, the underlying issues of overconfidence and miscalibration in confidence scores remain largely unaddressed, leading to inaccurate pseudolabels, degraded test accuracy, and prolonged training. We take a first-principles approach to learn con- fidence scores and thresholds with an explicit knob for error. This flexible framework addresses the fundamental question of optimal scores and threshold selection in pscudolabeling. Moreover, it gives practitioners a principled way to control the quality and quantity of pseudolabels. Such control is vital in SSL, where balancing pseudola- bel quality and quantity directly affects model performance and training efficiency. Our exper- iments show that, by integrating this framework with modern SSL methods, achieve significant improvements in accuracy and training efficiency. In addition, we provide novel insights on the trade- offs between the choices of the error parameter and the end model's performance.
While pseudolabeling is a powerful technique, its effective- ness hinges on a fundamental question: which points should be labeled using the model's 's predictions? Since pseudola- bels are derived from a model being trained, they can be highly unreliable. A naive approach that assigns pseudola- bels too liberally risks injecting noisy labels into training, amplifying the model's existing errors a phenomenon known as confirmation bias. Conversely, an overly con- servative approach that selects only the correct predictions severely limits the amount of useful training data. Both ex- tremes can degrade SSL performance, leading to either poor generalization or slow convergence. To fully harness the potential of pseudolabeling, we need principled approach for pseudolabeling with explicit control on these trade-offs.
A widely used strategy pseudolabels points on which the model's confidence score exceeds threshold. This ap- proach provides simple mechanism for selecting points while controlling the quality and quantity of pscudola- bels via thresholds. However, the prior works based on this approach suffer from two key limitations that restrict their effectiveness. First, thresholding techniques are often heuristic-driven, lacking precise control for a target error level (Sohn et al., 2020; Zhang et al., 2021; Wang al., 2023; Xu al., 2021: Chen al., 2023; Li al., 2023). Second, commonly used scores, such the model's softmax outputs, tend to be unreliable. Recent studies in this vein (Loh et al.. 2022; Mishra et al., 2024; Rizve et al., 2021) have highlighted issues of overconfidence and miscalibration in these scores, leading to inefficiencies in pseudolabeling.
1. Introduction
The lack of high-quality labeled data is a major bottleneck in training high-accuracy models. The semi-supervised learning (SSL) paradigm tackles this problem by leveraging abundant unlabeled data alongside a limited set of labeled examples (Chapelle al.. 2006: Zhu, 2005; van Engelen
"Equal contribution 'Department of Computer Sciences, Uni- versity of Misconsin-Maddion WI, USA 'Department of Electrical and Computer Engineering. University of Maddisn, WI, USA 'GE HealthCare "NYU Courant Institute. Correspondence to: Harit Vishwakarma <hvishwakarma @ es. edu>, Yi Chen <yi.chen@ wisc.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In addition to these problems with the choices of scores and thresholds, an equally important question remains: how

---

Rethinking Confidence Scores and Thresholds in Pscudolabeling-basee SSL
much error should be allowed in pseudolabeling? As dis- cussed earlier, a very low tolerance may pseudolahel too few points, and conversely, a high tolerance may allow for large errors, hurting the efficiency of pseudolabeling in both cases. To solve these issues, we seek a principled solution to get confidence scores and thresholds for pseudolabeling al a given target level of error tolerance.
among dynamic schedules, it is better to use a decreas- ing schedule of error tolerances.
2. Related Work
Semi-supervised learning (SSL), There is a rich literature on SSL (Zhu, 2005; Chapelle et al., 2006; Singh et al., 2008; Oliver et al., 2018). This literature comprises of a wide vari- ety of approaches. Among these, significant focus has been placed on self self-training (also called pseudolabeling) (Scud- der, 1965; Blum & Mitchell, 1998; Rosenberg et al., 2005; Lee, 2013; Oymak & Gulcu, 2020; Amini et al., 2023), gen- erative models (Nigam et al., 2000; Adams & Ghahramani, 2009; Kingma et al., 2014). graph-based strategies (Blum & Chawla, 2001; Niyogi, 2013; Subramanya & Talukdar, 2022), and transductive approaches (Vapnik et al., 1998; Joachims, 1999). Due to their simplicity, pseudolabeling- based approaches have gained prominence and are widely used in application areas such as NLP (Karamanolakis et al., 2021), speech recognition (Kahn et al., 2020), and protein prediction JEI-Manzalawy et al., 2016). Our paper focuses on recent variants of this, discussed next.
We take a first principles approach to formalize the pseudola- beling objective: maximize the number of pseudolabeled points while adhering to the given error tolerance. We for- mulate this objective as an optimization problem over a flexible space of confidence functions and thresholds. By solving this optimization problem, we obtain confidence scores tailored to our objective of pseudolaheling. To ensure the pseudolabeling error constraint strictly followed, we use separate procedure to estimate thresholds these scores using part of the validation data. Pseudolabeling with these scores and thresholds ensures we are pseudolabeling maximal set of points that can be pseudolabeled at the given error tolerance level.
We integrate this approach into popular pseudolabeling- based SSL methods, providing two benefits. First, it pro- vides principled way to derive confidence scores and thresholds for any given error tolerance. Second, by en- abling more precise pseudolabeling. it improves the utiliza- tion of the unlabeled data and is expected to yield an end model with higher test accuracy compared to the ad-hoc choices of scores and thresholds.
Pseudolabeling based SSL. These methods generate artifi- cial labels for unlabeled data and use them for training the model. A crucial challenge here is the issue of confirmation bias (Arazo et al., 2020), i.e., when a model starts to rein- force its own mistakes. To overcome this and to maintain a high quality of pseudolabels, confidence-basee thresholding is applied. Here, only the unlabeled data with confidence higher than a particular threshold is used (Sohn et al., 2020). Due the limitations of fixed thresholds, adaptive thresh- olds based on the classifier's 's learning status have been intro- duced to improve performance (Xu et al., 2021; Zhang et al., 2021; Wang et al., 2023). Nearly all of these methods also use some form of consistency regularization (Laine & Aila,
Our main contributions are summarized as follows,
1. Our work settles the question of the right choices of confidence scores and thresholds in pseudolabeling- based SSL methods by introducing a framework for learning confidence scores and thresholds. Departing from heuristic-driven or ad-hoc and unreliable choices of scores and thresholds, this framework provides prin- cipled choices of scores and thresholds for pseudolabel- ing with any target error tolerance.
2017: Bachman et al., 2014; Sajjadi et al., 2016; Fan et al.,
2021: Kukačka al., 2017) where the core idea is that the model should produce similar prediction when presented with different versions (perturbations) of inputs and all the present SSL methods (Xie et al., 2020; Wang et al., 2023; Sohn et al., 2020; Zhang et al., 2021; Chen et al., 2023; Zheng et al., 2022; Xu et al., 2021; Roelofs et al., 2022).
2. We show how this framework for learning confidence scores and thresholds can work in concert with popu- lar SSL methods such as Fixmatch (Sohn et al., 2020), Freematch (Wang et al., 2023), etc., and conduct an extensive empirical evaluation demonstrating that by pseudolabeling using confidence scores and thresholds learned from our method can yield significant improve- ments in the test accuracy.
Confidence functions and calibration. Miscalibration (overconfidence) in neural networks plagues various ap- plications (Nguyen et al., 2015; Hendrycks & 20mpel,,011, Guo et al., 2017), including SSL. To mitigate this in gen- eral, range of solutions have been proposed, including training-time methods (Moon et al., 2020; Kumar et al., 2018; Hui et al., 2023; Corbière et al., 2019; Foret et al.,
3. Leveraging our framework's ' ability to pseudolabel at any target error level, we study the impact of varying pseudolabeling error levels-from fixed to dynamic tolerance throughout training. Our results confirm the intuition that lower pseudolabeling errors lead to bet- ter end models compared to higher errors. Moreover,
2021) and post-hoc methods (Guo et al., 2017: Kumar et al.,
2019; Gupta & Ramdas, 2022: Kull et al., 2019; Zadrozny & Elkan, 2002). In pseudolaheling based SSL, recent works (Rizve et al., 2021; Loh et al., 2023; Mishra et al., 2024)
2

---

Rethinking Confidence Scores and Thresholds in Pseudolabeling-basee SSL
noted the issue of miscalibration. To promote calibration, Loh et al. (2023) use Bayesian neural nets by replacing the model'ss final layer with Bayesian layer. Rizve et al. (2021) pseudola-
Supervised loss. Given a batch of labeled data, Dj. the supervised loss is computed as follows, C, | D/) = [Di E(x,))DD Hly, h, x). Here H(,, hi x) is the standard cross-entropy loss between the 1-hot representation of y and the softmax output of h on input x.
utilize negative labels and an uncertainty-awe bel selection technique. Mishra et al. (2024) incorporate a regularizer to encourage calibration.
Unsupervised loss and consistency regularization. For the unlabeled batch X pseudolabels y = h(x) are com- puted for each x E X Then, a pseudolabeling mask, S(x, g. h) (g(()) M Z t(ū]) is computed. It is 1 for points having confidence score g(x) [o], bigger than pre- determined threshold tlø), corresponding to the predicted class j. Recent methods, couple this loss and consistency regularization together by doing pseudolabeling on weakly augmented data using weak transform w : X + X' and then defining the cross-entropy loss on the strongly augmented data using strong transformation Q : X H X. The loss is,
While calibration is a reasonable goal in general, it may not be sufficient to address the overconfidence problem in SSL and other applications. In pseudolabeling, we seek the use of scores that can easily segregate the model's ' correct and incorrect predictions, which is closely related to the ordinal ranking criterion (Hendrycks & Gimpel, 2017; Moon et al., 2020; Foret et al., 2021; Corbière et al., 2019). Rather than experimenting with several such choices, ideally, we would have a flexible framework that can learn confidence functions explicitly optimizing pseudolabeling objectives.
1 E. == D
Σ S(w(x), g.thh) t/h) H H(g h, ((x)).
3. Background and Problem Setup
(..
We begin with notation, then provide useful background and a statement of our goal.
Regularization. A regularization term (or a sum of mul- tiple regularizers) is often included along with the above two losses to encourage desired behavior(s). For instance, Freematch (Wang et al., 2023) adds a self-adaptive class fairness regularizer to encourage diverse predictions during the initial training phase. Similarly, a regularizer is intro- duced in (Mishra et al., 2024) to encourage calibration in the model's 's confidence scores. Including such regularizers has been fruitful in pseudolabeling- based SSL..
Notation. Consider a feature space X' and label space y {1....,k) }} in a k-class classification task. As usual in semi- supervised learning, we have access to a set X., = {x,} J === of unlabeled data drawn from the distribution PN over X We also have access to D, = {(x1, 1/ set of la- beled data points drawn from the joint distribution Pay. with N₁ < Thu Let h : X' - y denote a model and g : X Th S I be an associated confidence function giving score g(x) indicating the confidence of h on its prediction for any data point x. For any x the hard label prediction is Φ := (xx.. When the prediction ÿ is used as a pseudolabel, we denote it as ÿ. In general, for a vector v E Rdd víi denotes its i-th component. The vector denotes thresholds over the scores k-classes, and tly] is its y th entry, i.e., the threshold for class y.
3.2. Quality and Quantity of Pseudolabels
Given classifier h. the quality and quantity of the pseu- dolabels w.r.t. to score function g and thresholds t, are:
Pseudolabeling coverage (quantity). Given a set of points X, the pseudolabeling coverage is the fraction of points that are pseudolabeled using h g and t. This measurement captures the quantity of pseudolabels and is defined as
3.1. Pseudolabeling- Semi-Supervised Learning
Given, as above, large collection of unlabeled data X,, and a small set of labeled points D;, inductive sumi-supervised learning (SSL) seeks to learn a classifier /i from the model class H. The promise of SSL is that by effectively using X,, in the learning process, it can learn a better classifier than its supervised counterpart, which learns only from Di-
Plg,t h X)
Σ (x)EX
S(x, gtx.gtth))
(I)
P(g. t h)) = Ex [S g. t [i]..
(2)
Pseudolabeling error (quality). This is the fraction of pseudolabeled points that got incorrect labels. This metric captures the quality of pscudolabels:
In many recent pseudolabeling- based SSL techniques, in each iteration of training, a batch of labeled and unla- beled data is obtained, then the sum of the losses Σ = E. + X, + X,E, is minimized with respect to the model h. Here L, is the supervised loss, E, unsupervised loss, and E, is (the sum of) regularization term(s). The positive con- stants Xu, are hyperparameters controlling the relative importance of the corresponding terms.
Elg, t hh D) = E(xwDe S(x, g, h) 1(h(x) A Aw)
ExxuDE S(x.g, t h)
(3)
E(g. t (1) Ex [S (x1,xx9ttt t h1) 11((x)))x))))))))))))(()) + b)] P(g. t(h)
(4)
3

---

Rethinking Confidence Scores and Thresholds in Pseudolabeling-bssee SSL
3.3. Our Goals
thresholding heuristics. Instead, solving the optimization problem over flexible enough space will subsume specific strategies. Next, we discuss how to make this framework tractable to obtain scores and thresholds in practice.
Pseudolabeling-basee SSL aims to learn classifier Tia that generalizes well the unseen data, i.e., has high test ac- curacy. This is typically achieved by pscudolabeling points using confidence scores and thresholds and incorporating them into training. However, existing choices of scores and thresholding strategies are often ad hoc and unreliable, lim- iting their effectiveness. Departing from these unreliable approaches, our goal is to:
Practical version. The optimization problem discussed ear- lier involves population-level quantities which are usually not accessible in practice. Thus we have to use their finite sample estimates and smooth variations to make the opti- mization problem tractable. Specifically, the coverage and error are estimated using small amount of held-out labeled data (called calibration data Deal) curated from the valida- tion data. Then differentiable surrogates for the 0-1 vari- ables are introduced. Let o(a, 2) = 1/(1 + exp(-az)) denote the sigmoid function on TR with scale parameter a E R. The surrogates are as follows,
(i) Design principled solutions for confidence scores and thresholding to maximize the number of pscudolabeled points while ensuring the pseudolabeling error is at most e.
(ii) Incorporate these in the existing pseudolaheling-basee SSL methods and assess whether this gives a better end model Tisst-
P(g.t Deal) = Deaul (x....))
1
Σ g0, - t(Øl).
(iii) Study the sensitivity of the SSL pipeline to pseudola- beling errors by leveraging the ability of our approach to explicitly ensure the pseudolabeling error remains below
(6)
E(g. Deal): Eie. 1 (vtw) a ((.x(()
Eier (a.g(x))[) [3] t(p
4. Methodology
Using these surrogates, the following practical optimization problem is obtained. It is also converted into unconstrained formulation by introducing the penalty term À E g+ con- trolling the relative importance of the error and coverage.
In this section, we discuss our principled framework to learn scores and thresholds with explicit control of the pseudola- SSL..
beling errors and use them in pseudolaheling- This framework is an adaptation of a recent solution de- signed to improve confidence functions for auto-labeling (Vishwakarma et al., 2024). The detailed steps are outlined in Algorithm 3 in the Appendix.
G. ? E arg min -P(g, t h. Deal) + +XE(g,t) t h. Deal) gEG,tETA
(P1) We use 2-layer neural nets as choice of g and T 1 The optimization problem (PI) is non-convex but differentiable and we solve it using Stochastic Gradient Descent (SGD). See Appendix C for more details on our choice of G and training details and hyperparameters.
4.1. Pseudolabeling Optimization Framework
Given the current model Tú in the ith iteration, can we obtain confidence scores and thresholds using which we can identify a maximal set of points that can be pseudolabeled using 142 with at most error? We begin with theoretical formulation to learn such scores and thresholds, and then introduce its practical version.
2
4.2. Threshold Estimation
While we can obtain both the confidence scores and thresh- olds by solving (PI), we propose to estimate thresholds separately on held-out part of the validation data avoid potential generalization issues due learning them simulta- neously from the same data Deal and ensure that the pseu- dolabeling error constraint is strictly adhered to.
Theoretical framework. Instead of improving calibration or heuristics for thresholding, we propose to express the ob- jective of pseudolabeling as an optimization problem over the space of confidence functions and thresholds. The ob- jective is to maximize the quantity, i.e., the pseudolabeling coverage (eq. (2)) while keeping the pscudolabeling error (eq. (4)) below a tolerance level e E (0. 1). More specifi- cally, given the classifier hy in any iteration of SSL,
When dealing with datasets containing many classes there may not be enough samples per class to estimate reliable thresholds. Thus, to accommodate these possibilities we consider two variations of the threshold estimation proce- dure, (i) estimate a common (joint) threshold for all classes and (ii) estimate separate (classwise) thresholds for each class. The procedures are outlined in Algorithm 2 and Algo- rithm in the Appendix. We discuss them briefly here.
g7, t E arg Plg, t R.)) s.t. E(g,t t h.)) S, (5)
are the optimal confidence functions and thresholds for pseu- dolabeling using h, 's predictions such that the pseudolabel- ing error is bounded by e. This frees us from arbitrary choices of confidence scores, calibration techniques, and
The procedure takes in a confidence function Φ. and part of the held-out validation data referred to as Deh- The idea
4

---

Rethinking Confidence Scores and Thresholds in Pscudolabeling-base based SSL
(A)
(B)
Train Model
Minimize loss on groundtruth and pseudolabeled data
Learn new confidence scores and thresholds
Data
Groundtruth Labeled data
Trained Model
Pseudolabeled
(D)
(C)
data
Labeled
Pseudolabel accumulation (optional)
Pseudolabel points with confidence >= threshold
Unlabeled
Unlabeled data
Psoudolabeled
Figure 1. Workflow of pseudolabefing-hanl based SSL with PabLO (A): Train model with standard supervised loss, consistency regularization, and other regularizers (B): Learn new confidence scores and thresholds (C): Pseudolabel points with scores greater than the estimated thresholds (D): An optional pscudolabeling accumulation round. Note, that the pscudolabels can be noisy (incorrect). The training and pscudolabeling loop continues until a pre-specified number of iterations. In the end, it outputs model has that is expected to have higher test accuracy than the model trained only the given groundtruth training data. Note that in the end, there might be points left unlabeled, and the pseudolabels might be noisy.
previous pscudolabels for points that not pscudolabeled in current
to estimate eπrоrs at several thresholds on this data and then pick the smallest threshold. This can be done separately for each class to obtain classwise thresholds or a common threshold for all classes. We discuss classwise thresholding here. First the data Du, is partitioned into & subsets ( corresponding to each class E y. Here, we slightly abuse notation: instead of t E T we use t E T in the estimate of pseudolabeling error at threshold for class y. To obtain threshold tly] for class y. the procedure finds the smallest t E T such that E h, ( + Ciō(E) S e. Here C1 is a constant (we use C1 0.25) and (2) = V2 (1-2) and E is used for brevity in place of (ūi;, h, D( The same process is used for joint threshold estimation, where a single threshold is estimated using the entire Di and the same is used for all classes. Using the thresholds found using these procedures ensures pseudolabeling error remains below (or close to) the tolerance level e.
after each iteration. Moreover, it is not guaranteed that a previously pseudolabeled point will get pseudolabled the current iteration as well. Given the quality of pseudolabels is high, appealing to reuse the past pseudolabel for a point that did not get pseudolabeled in the current iteration. We propose to do so for techniques where the quality of pseudolabels is assured. We refer to this as "pseudolabel accumulation".
Mathematically, let ((-1) - -Y(--1 [s] and y, be the previous and current (fresh) pseudolabels for jth unlabeled point. Let the corresponding masks (indicating whether the score is above the threshold) for these psuedola- bels be S 3 and s(0 Then with accumulation,
7(0 7:00 b]]
[j]
1 ( (x, Z (
y []] - S +V [ (11++11-- (1- - Gル [
2 - 5 S(
Remarks. Departing from fixed thresholds as in (Sohn et al., 2020), prior works have proposed adaptive and class-wise heuristic thresholding schemes based on the model's learn- ing status, such as (Djurisic et al., 2023; Zhang et al., 2021; Wang et al., 2023) and others. In contrast, our ap- proach is principled way estimate adaptive and class- wise pseudolabeling thresholds while providing strict con- trol over the quality of pscudolabels. Similar procedures have been used in the context of creating reliable datasets and are backed by theoretical guarantees for the quality of pseudolabels produced (Vishwakarma al., 2023).
Here V is the boolean or operation and the steps are exe- cuted in the order. In words, if the point is pseudolabeled in the current iteration (i.e., its current mask is 1), then it will use the current pseudolabel otherwise, if the point was pseu- dolabeled in earlier iteration(s) it will use the pseudolabel from that iteration and mark the point as pseudolabeled. In case the point is not pseudolabeled in this iteration or any other iteration the past, it will remain unlabeled. While is appealing to use this trick, its use is only warranted in settings ensuring high-quality pseudolabels. We try to understand the consequences of the inclusion and exclusion of this trick in pseudolabeling- beabbbilig--- SSL via experiments discussed in the next section.
4.3. Pseudolabeling and Accumulation
In the usual secudolabeling-based SSL setups, the pseudola- bels inferred by the model for mini-batch are discarded
We put together the steps for learning scores, thresholds, and performing (optional) accumulation in a common template
5

---

Rethinking Confidence Scores and Thresholds in Pseudolabellngg based SSL
Dataset
Backbone Model h
k:
Ty
Nyal
Niest
N₁
Neal
Nih
Augmentation
CIFAR-10
WRN-28-2
10
50K
6K
4K
40 or 250
1K
1K
Weak, Strong
CIFAR-100
WRN-28-
100
50K
6K
4K
400 or 2500
3K
3K
Weak, Strong
SVHN
WRN-28-2
10
604,388
15,620
10,412
40 250
3K
3K
Weak, Strong
Table 1. Details of the dataset we use in our experiments. k is the number of classes. Nr is the number of labeled data points used for training the backbone model h. nh is the number of unlabelled data points used for consistency regularization and pscudolabeling for all the methods. Noal is the number of points used for model selection in all methods. Niess is the number of test data points. Neal is the number of points used for learning the 3 function, Nu. is the number of data points used for threshold estimation.
of gecudolabeling-basee SSL algorithms. We refer to this adapted method (Algorithm 3 in Appendix B) as PabLo. The high-level steps are also illustrated in Figure 1. Next, we discuss the empirical evaluation of PabLO and baselines.
CIFAR-100 (Krizhevsky et al., 2009) is an extended version of CIFAR-10 with 100 classes. SVHN (Netzer et al., 201 is 10-class image dataset of digits from Google Street View. More details are summarized in Table We use a portion of the validation data (Dval) for our method, split into Deal. used to learn the function g, and Dea, used to estimate the threshold. Unless otherwise mentioned, we use N₁ as 250 for CIFAR-10 and SVHN and 2500 for CIFAR- 100 in our experiments.
5. Experiments
We conduct empirical evaluation over several settings to,
C1. Verify that the adaptations of popular pseudolabeling- based SSL methods with P PabLo output models with better test accuracy.
Adjusted iterations for baselines. Empirically, our method requires more time to run compared to base SSL techniques. Therefore, we adopt the following strategy ensure fair comparison between the baselines and our method: First, we train our method for 25K iterations and obtain the average per iteration time, denoted as Qg.- Then, we train each baseline method b for 5K iterations and obtain the average per iteration time, denoted as ag- Using these two values, we obtain the adjusted number of iterations, as x 25000, for the baseline method b. Coincidentally, baselines under the same dataset have similar runtimes. We, therefore, set the adjusted number of iterations on a dataset level. For CIFAR- 10, the adjusted number of iterations for baselines is 37, .00. For 10FAR-100, the adjusted number of iterations for baselines is 70,000. For SVHN, it is 145,000.
C2. Study the effects of choice of error tolerance e on test accuracy of the final model.
C3. Understand the role of pseudolabel accumulation in our method and baselines.
5.1. Experimental Setup
First, we briefly describe the experimental setup, with details deferred to Appendix C. The code is available on GitHub
Methods. We use two simple base methods that capture the core ideas of pseudolabeling (PL) and consistency reg- ularization (CR). The first is Fixmatch (Sohn et al., 2020), which uses fixed thresholds on (maximum softmax proba- bility) MSP scores for PL and CR. Freematch (Wang et al., 2023) improves upon it by using adaptive, class-wise thresh- olds and class fairness regularization (CFR) along with CR, and is a promising method among others using dynamic thresholds for PL. We include their combinations with re- cently proposed Bayesian Model Averaging (BAM) (Loh et al., 2023) and Margin Regularization (MR)* 2 (Mishra et al., 2024) to improve calibration in SSL. We replace the pseudolabeling component by our method PabLO to ob- tain Fixmatch + Ours (a combination of PabLO and CR) and Freematch + Ours (a combination of PabLo, CR, and CFR). We provide implementations of these in the code submitted along with the paper.
Models and training. The backbone encoder is a Wide ResNet-28-2 for all the datasets. We use the default hy- perparameters and dataset-specific settings (learning rates, batch size, optimizers, and schedulers) following previous baseline recommendations (Wang et al., 2022). For confi- dence functions class G, we use a class of 2-layer neural nets and provide the last two layers' representations from h as input. We train it using SGD. The hyperparameters are deferred to Appendix C. Unless otherwise specified, our method uses pseudolabeling error tolerance = 5%.
5.2. Results and Discussion
To verify our main claims, we compare the basclines, their combinations with our method, and methods that induce calibrated scores in SSL. We run all methods with three random seeds and report (in Table 2) the mean and standard deviation of accuracy across three runs.
Datasets. We experiment with three datasets: CIFAR-10 (Krizhevsky et al., 2009) is an image dataset with 10 classes.
'https:// http:::://iiaiie thub. com/ homm/arrtt77 Taarrt7/Paablle /P///P/aLLLsss SSL We assign this name for convenience.
6

---

Rethinking Confidence Scores and Thresholds in Pscudolabeling-base based SSL
CIFAR-10
CIFAR-100
SVHN
Top-1 Accuracy
100
100
100
75
75
75
50
50
50
25
25
25
0
5K
15K
25K
35K
10K
30K
50K
70K
50K
100K
150K
Fixmatch
Fixmatch + Ours
Fixmatch BAM
Fixmatch + MR
Top-1 Accuracy
100
100
100
75
75
75
50
50
25
25
25
0
5K
15K
25K
35K
10K
30K
50K
70K
50K
100K
150K
Iterations
Iterations
Iterations
Freematch
Freematch + Ours
Freematch BAM
Freematch + MR
Figure Top-1 accuracy of our method and baselines on CIFAR-IO, CIFAR-100, and SVHN. We plot values for every steps.
C1. Test accuracy improvements. Since our method maxi-
Fixmatch Ours
Freematch Ours
mizes the pseudolabeling coverage and accuracy, provides more accurate pseudolabels for model training. Therefore, expect it to yield a model with better test accuracy than the baselines. We report the test accuracies the end of 25K iterations in Table 2 for our methods. For the baselines, we report the test accuracies at the end of the corresponding adjusted number of iterations (well above 25K). Figure 2 illustrates how the top-1 accuracy evolves during the SSL. Similarly, Figure and show how batch pseudolabeling accuracy and batch pseudolabeling coverage change.
Tap-1 Accuracy
100
100
75
25
0.01 0.10 0.20
0.40
0.01 0.10 0,20
0.40
Error tolerance (E) CIFAR-10
Error tolerance (E) (b) CIFAR-100
Figure 3. Top-1 accuracy of our method with different error toler- ance on (a) CIFAR-10 10 and (b) CIFAR-100 dataset.
First, as expected, integrating method into the base methods improves test accuracy across all settings. For CIFAR-10, using it with Fixmatch provides almost 2% im- provement over Fixmatch alone, and using it with Freematch yields 1% improvement over Freematch. Much more signif- icant improvements are observed in the much harder setting of CIFAR-100: nearly 10% improvement in top-1 curacy over Fixmatch and around 5% improvement over Freematch. SVHN is an easier setting; here, the improve- ments are marginal. With Fixmatch, our performance is similar to that of the baselines. But, using P PabLO with Freematch improves performance by 3%.
and CIFAR- 100 settings. each with three random seeds, and report the results in Figure 3. The results are as expected higher values of e lead to degraded test accuracy due to high noise in the pseudolabels and with decreasing leads to improved accuracy. These results also suggest that priori- quantity
tizing the quality (accuracy) of pseudolabels is better choice in pseudolabeling, The results are also summarized in Table and Table 9 in the Appendix.
To investigate the error tolerance further, we designed error tolerance scheduling using different error tolerances during various stages SSL training. Table summarizes the error tolerance set at difffernt iterations of SSL training and the corresponding top-1 accuracy for the CIFAR-10 10 and CIFAR-100 datasets. As we see, starting SSL with a small eгтor tolerance and ending with a large tolerance severely impacts the performance on both CIFAR-10 and CIFAR- datasets. While our findings suggest lower error tolerance is preferable, this may not hold general. Nevertheless, our
C2. Error tolerance affects performance. In our method, the error tolerance parameter is knob to control the amount of noise in pseudolabels. A common wisdom in pseudolaheling is that higher noise will lead to worse per- formance, which is our expectation. To see this, we run our method with e E (0.01, 0.05, 0.1, 0.2, 0.43 in CIFAR-10

---

Rethinking Confidence Scores and Thresholds in Pscudolabeling- based SSL
Dataset
CIFAR-10
CIFAR-100
SVHN
# Labels
40
250
400
2500
40
250
Fixmatch
57.16 + 12.12
90.8 0.78
30.38 + 0.68
59.09 1.10
62.27 + 5.57
97.57 0.08
Fixmatch MR
54.76
5.92
90.41 + 0.83
16.31 + 1.44
54.16 0.18
74.41
16.61
97.35 0.08
Fixmatch BaM
57.72 1.44
90.67 1 0.90
31.45 1: 1.29
56,60 2.45
76.18 18.61
97.51 0.13
Fixmatch Ours
89.72 5.31
92.69 1. 0.74
67.44 1. 1.14
69.10 0.45
96,58 1 0.14
96.54 0.13
Freematch
79.61 7.24
92.26 + 0.18
43.88 + 3.60
63.13 0.46
86.87 + 4.15
92.90 2.76
Freematch MR
84.19 6.19
92.17 + 0.36
44.20 + 2.93
62.03 + 0.82
84.89 + 7.82
93.26 2.36
Freematch BaM
88.33 0.33
92.32 + 0.25
44.49 + 3.05
62.13 + 2.93
81.43 + 14.80
91.08 3.72
Freematch Ours
90.19 5.37
93.10 = 0.28
65.37 +: 4.05
68.76 +: 1.38
96,55 = 0.15
96,65 0.26
Softmatch
83,60 7.09
91.74 1 0.78
40.73 1. 1,46
61,43 1. 0.34
59.52
12.47
96.93 0.23
Softmatch + Ours
89.96 4.74
93.14 +: 0.33
67.84 + 0.33
68.74 1: 0.72
96,55 + 0.10
96.51 0.37
Adamatch
75.00 + 1.10
91.35 + 0.66
31.61 + 1.92
58.08 + 0.44
80.83 + 7.09
96.99
0.08
Adamatch + Ours
86.62 1 10.54
93.06 1. 0.19
67.00 1. 1.02
68.12 1. 0.48
96,55 1. 0.09
96.44 0.20
Table Top-1 Accuracy for CIFAR-10, CIFAR-100 and SVHN averaged across 3 random seeds. We report the average accuracy + std. deviation. Here, Ni (40, 250} for CIFAR-IO, SVHN and Nr {400, 2500} for CIFAR-100. The best accuracy is bolded.
CIFAR-10 Top-1 Accuracy
CIFAR-100
Method
Acc-True
Acc-False
Top-1 Accuracy
Fixmarch
67.62 +: 2.10
90.08 + 0.78
Schedule Fixmatch Ours
91.11 + 1.31
65.78 1.36
Fixmatch MR
64.78 4: 4.64
90.41 + 0.83
Fixmateh BaM
68.10 + 2.02
90.67 : 0.90
Schedule
Freematch Ours
91.09 1.01
66.13 0.48
Freematch
85.40 +: 1.36
92.26 :: 0.18
Schedule
Freematch
4:
92.17 d:
35.38 + 29.27
19.56 3.07
Fixmatch Ours
Freematch BaM
85.48 +: 3.02
92.32 d:
Schedule 30.48 + 10.52
Fixmatch Ours
92.69 +: 0.74
92.80 1: 0.56
Freematch + Ours
24.80 30 3.49
Freematch Ours
4: 0.28
1.08
Table 3. Top-1 accuracy for the two error tolerance (e) schedul- ing. The table reports the use between each iteration inter- val and the top-1 accuracy yielded by the corresponding sched- ule. For schedule 1, we the number of iterations is in the following interval, respec- tively: [o, 5K), [5K, 10K), [10K, 15K), [15K, 20K), [20K, 25K). For schedule we sct 0.01, 0.05, 0.1, 0.2, 0.4, for the same intervals, respectively.
Table 4. Results on CIFAR-10 with and without pseudolabel accu- mulation (Acc) for all the methods.
0.4, 0.2, 0.1, 0.05, 0.01 when
in Figure 6 in Appendix C. These results are unsurprising since our method ensures high quality of pseudolabels while maximizing coverage; it can eventually catch up with the version using accumulation, leading to similar final test ac- curacies. On the other hand, having accumulation hurts the performance of baseline models. This might be because the pseudolabels generated by the baseline models inaccu- rate, especially in the earlier iterations, thus degrading the overall performance. Overall, we believe accumulation will be helpful when we have pseudolabels with high accuracy. The plots for pseudolabeling coverage and accuracy over the entire run are in Figures in Appendix C.
framework provides the flexibility control this explicitly and thus can be tuned by practitioners for the setting at hand.
C3. Is pseudolabel accumulation helpful? Accumulation allows the methods to use old pseudolabels for points that couldn't get pseudolabeled in the current iteration. Thus, we expect accumulation help improve the utilization of un- labeled data and lead better test accuracy cases where the pseudolabel quality is assured to be high in all itera- tions. We run two variations of our method and baselines with and without accumulation and report the results in Table We observe that our method has similar test accu- racy irrespective of accumulation. However, accumulation achieves better coverage in early iterations, as observed
Experiment with baselines using calibration data for training. We run the baselines where the amount of training data for the baseline is increased by the amount of calibra- tion data used in our method. For the CIFAR- 10 setting with 250 labels, we run the baseline now with 250 + 1000 = 1250 labeled points for training. Similarly, for CIFAR-100 with 2500 labels setting, we run with 2500 + 3000 5500
8

---

Rethinking Confidence Scores and Thresholds in Pscudolabeling- SSL
6. Conclusion
Method
CIFAR-10
CIFAR-100
Common semi-superrvied learning (SSL) methods rely on pseudolabeling, but their effectiveness limited by unreli- able confidence scores and heuristic thresholding strategies. We address these issues by introducing a principled frame- work for learning confidence scores and thresholds with explicit control over pseudolabeling error. We adapt ex- isting SSL methods with this framework and empirically show that the adapted methods achieve a higher test accu- racy compared to their standard versions. Additionally, we introduce pseudolabel accumulation and analyze its impact, showing that it benefits methods with reliable pseudolabels, such those using our framework. In sum, by provid- ing principled, data-driven approach to obtaining scores and thresholds for pseudolabeling, our work enhances SSL methods and opens the door to more reliable and efficient pseudolabeling-bant based SSL.
FixMatch (train + cal)
92.68 + 0.31
64.77 = 0.10
FixMatch + Ours
92.69 + 0.74
69.10 + 0.45
FreeMatch (train cal)
93.03 + 0.03
67.69 + 0.12
FreeMatch + Ours
93.10 + 0.28
68.76 + 1.38
Table 5. Comparison of FixMatch and FreeMatch with and without our method on CIFAR-I0 10 and CIFAR-100. Here Ni = 250 for CIFAR-10 10 and 2500 for CIFAR-100. The overall training data from methods annotated as (train + cal) includes these N₁ points and the calibration data Neal (from Table 1) used in our method,
labels for training. The results are reported in the Table 5. We can that even with more labeled data in training. the baselines still fall short significantly in the CIFAR-100 setting, while the performance gap in the CIFAR-10 (casier) setting narrows down.
Impact Statement
This research improves semi-supervised learning, enabling more accurate and efficient machine learning in settings where labeled data is hard to obtain by following first princi- ples in designing thresholds and confidence functions. Our work has various potential societal implications, with no specific concerns that require special attention in this con- text.
Validation data usage, limitations, and future work. In the experiments, our method used parts of the validation data differently in comparison to the baselines. We comment on the role of validation data in the baselines, in method, and potential limitations due to it. Recall, our goal is to address the problem of ad hoc choices of confidence scores and thresholds in pseudolabeling baaseeeeinn-ddddddddee SSL. To this end, we introduced a principled solution to learn scores and thresh- olds that can directly achieve any specified pseudolabeling error while maximizing the number of pseudolabeled points. Given the focus of our paper, our experiments are designed to study whether using our learnable scores and thresholds can benefit the baselines. To keep our solution statistically sound, we used part of the validation data to learn the scores and thresholds. The lack (or cost) of a sufficient amount of validation data could be a limitation in practice. Future work can explore ways reduce the amount of validation data needed for learning confidence scores and thresholds while preserving the soundness of our method. For instance, generative models or data augmentation techniques could be employed to this end.
A Acknowledgments
This work was partly supported by funding from the Ameri- can Family Data Science Institute and the Institute for Foun- dations of Data Science (IFDS). We thank the anonymous reviewers for their valuable feedback.
References
Adams, R. P. and Ghahramani, Z. Archipelago: nonparamer- ric bayesian semi-supervised learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 1 1-8, 2009.
Amini, M.-R., Feofanov, V., Pauletto, L.,, Hadjadj, L., Devi- jver, E., and Maximov, Y. Self-training: A survey, 2023.
More broadly, in semi-supervised learning (SSL) research, the cost of validation data is often not explicitly accounted for, partly due the widespread use of standard benchmark datasets where hyperparameters have been tuned extensively over time. However, in real-world applications, new datasets typically require a substantial amount of validation data for model selection and hyperparameter tuning both of which are essential parts of the modern training process. Developing novel benchmarks that explicitly incorporate validation data into the overall labeled data budget could help better reflect practical deployment settings and would be a valuable direction for future SSL research.
Arazo, E., Ortego, D., Albert, P., O'Connor, N. E., and McGuinness, K. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International joint conference on neural networks (IJCNN, pp. 1-8. IEEE, 2020.
Bachman, P., Alsharif, O., and Precup, D. Learning with In Advances in Neural Information Processing Systems, volume 27, 2014.

---

Rethinking Confidence Scores and Thresholds in Secudolabeling-basee SSL
Blum, A. and Chawla, S. Learning from labeled and unla- beled data using graph mincuts. 2001.
Hui, L.,, Belkin, M., and Wright, S. Cut your losses with squentropy. In Proceedings of the 40th International Con- ference on Machine Learning, pp. 14114-14131, 2023.
Blum, A. and Mitchell, T. Combining labeled and unlabeled data with co-training. In Proc. of the eleventh annual conference on Computational learning theory, pp. 92- 100. ACM, 1998.
Joachims, T. Transductive inference for text classification using support vector machines. In Bratko, I. and Dzeroski, S. (eds.), Proceedings of ICML-99, 16th International Conference on Machine Learning, pp. 200-209, 1999.
Chapelle, O., Schölkopf. B., and Zien, A. (eds.). Semi- Supervised Learning. The MIT Press, 2006. ISBN 9780262033589.
Kahn, J., Lee, A., and Hannun, A. Self-training for end- to-end speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), pp. 7084-7088. IEEE, 2020.
Chen, H., Tao, R., Fan. Y.. Wang, Y., Wang, J., Schiele, B., Xie, X., Raj. B., and Savvides, M. Sofumatch: Addressing the shantity-qualliyy tradeoff in semi- supervise learning. In The Eleventh International Conference on Learning Representations, 2023, URL https: / /pppneeeee.eeeeeeeeeeeeee..eeeee net/forund i yd= 20XBDiF.
Karamanolakis, G., Mukherjee, S., Zheng, G., and Awadal- lah, A. H. Self-training with weak supervision. arXiv preprint arXiv:2104.05514 2021.
Kingma, D. P., Mohamed, S., Jimenez Rezende, D., and Welling, M. Semi-supervised learning with deep genera- tive models. In Advances in Neural Information Process- ing Systems, volume 27, 2014.
Corbière, C., THOME, N., Bar-Hen, A., Cord, M., and Pérez, P. Addressing failure prediction by learning model confidence. In Advances in Neural Information Process- ing Systems 32, pp. 2902-2913, 2019.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Djurisic, A., Bozanic, N., Ashok, A., and Liu, Extremely simple activation shaping for -ut-of-distributtio detection. In The Eleventh International Conference Learning Representations, 2023.
Kukačka, J., Golkov, V., and Cremers, D. Regularization for deep learning: A taxonomy, 2017.
Kull, M., Perello Nieto, M., Kängsepp, M., Silva Filho, T., Song, H., and Flach, P. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In Advances in Neural Information Processing Systems, volume 32, 2019.
El-Manzalawy, Y., Munoz, E. E., Lindner, S. E., and Honavar, Plasmosep: Predicting surface-exposed pro- teins on the malaria parasite using semisupervised self- training and expert-annotated data. Proteomics, 16(23): 2967-2976, 2016.
Kumar, A., Sarawagi, S., and Jain, U. Trainable calibration measures for neural networks from kernel mean embed- dings. In Proceedings of the 35th International Confer- ence on Machine Learning, volume 80 of Proceedings of Machine Learning Reseurch, pp. 2805-2814, PMLR, 10-15 Jul 2018.
Fan, Y., Kukleva, A., and Schiele, B. Revisiting consistency regularization for semi-supervised learning, 2021.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
Kumar, A., Liang, P. S., and Ma, T. Verified uncertainty calibration. Advances in Neural Information Processing Systems, 32, 2019.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Laine, S. and Aila, T. Temporal ensembling for semi- supervised learning. Fifth International Conference on Learning Representations, 2017.
Gupta, C. and Ramdas, A. Top-label calibration and multiclass-to-binaryy reductions. In International Confer- ence on Learning Representations, 2022. URL https: ///pen/eview.neeeee.e..eeeeeeeee wet.oeuuww.ee//uumtt..... forum 2i- WqoBaaPsS-
Lee, D.HH. Pseudo-label:
The simple and efficient semi- supervised learning method for deep neural networks. In ICML Workshop on Challenges in Representation Learn- ing, 2013.
Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and dutto--distrbbutinn examples in neural networks. In International Conference on Learning Rep- resentations, 2017
Li, M., Wu, R., Liu, H., Yu, J., Yang, X., Han, B., and Liu, T. Instant: Semi-supervised learning with instance- dependent thresholds. In Seirth- Conference on Neural Information Processing Systems, 2023.
10

---

Rethinking Confidence Scores and Thresholds in Pscudolabcling-basee SSL
Loh, C., Dangovski, R., Sudalairaj, S., Han, S., Han, L., Karlinsky, I.., Soljacic, M., and Srivastava, A. On the importance of calibration in semi-supervised learning, 2022.
Roelofs, B., Berthelot, D., Sohn, K., Carlini, N., and Kurakin, A. Adamatch: A unified approach to semi- supervised learning and domain adaptation. In The Tenth International Conference on Learning Representations, 2022.
Loh, C., Dangovski, R., Sudalairaj. S., Han, S., Han, L., Karlinsky, L., Soljacic, M., and Srivastava, A. Mitigating confirmation bias in semi-supervised learning via efficient bayesian model averaging. Transactions on Machine Learning Research, 2023.
Rosenberg, C., Hebert, M., and Schneiderman, H. Semi- supervised self- self-training of object detection models. In Seventh IEEE Workshops on Applications of Computer Vision (WACVMMTTIN''5)] Volume 1, volume 1, pp. 29-36, 2005. doi: 10.1 109/AC/VO1.22055
McLachlan, G. J. Iterative reclassification procedure for con- structing an asymptotically optimal rule of allocation in discriminant analysis. Journal of the American Statistical Association, 70(350):365-365 369. 1975.
Sajjadi, M., Javanmardi, M., and Tasdizen, T. Regulariza- tion with stochastic transformations and perturbations for deep semi-supervised learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 1171-1179, 2016.
Mishra, S., Murugesan, B., Ayed, I. B., Pedersoli, M., and Dolz, J. Do not trust what you trust: Miscalibration in semi semi-superre supervised learning, 2024.
Scudder, H. Probability of error of some adaptive pattern- recognition machines. JEEE Transactions on Information Theory, 11(3):363-371, 1965.
Moon, J., Kim, J., Shin, Y., and Hwang, S. Confidence- aware learning for deep neural networks. In Proceedings of the 37th International Conference on Machine Learn- ing, volume 119, pp. 7034-7044, 2020.
Singh, A., Nowak, R., and Zhu, J. Unlabeled data: Now it helps, now it doesn't. In Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng. A. Y., et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 7. Granada, Spain, 2011.
Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C. A., Cubuk, E. D., Kurakin, A., and Li, C.-L. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural informa- tion processing systems, 33:596 608, 2020.
Nguyen, A., Yosinski, J., and Clune, Deep neural net- works are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 427-436, 2015.
Subramanya, A. and Talukdar, P. P. Graph-based semi- supervised learning. Springer Nature, 2022.
van Engelen, E. and Hoos, H. H. A survey on semi- supervised learning. Machine Learning, 109:373 440, 2019.
Nigam, K., McCallum, A. K., Thrun, S., and Mitchell, T. Text classification from labeled and unlabeled documents using em. Machine learning, 39:103-134, 2000.
Niyogi, P. Manifold regularization and semi- superviee learning: Some theoretical analyses. Journal of Machine Learning Research, 14(5), 2013.
Vapnik, V.N Vapnik, V., et al. Statistical learning theory. 1998.
Vishwakarma, H., Lin, H., Sala, F., and Vinayak, R. K. Promises and pitfalls of threshold-based auto-labeling. In seirth- Conference on Neural Information Pro- cessing Systems, 2023.
Oliver, A., Odena, A., Raffel, C. A., Cubuk, F. D., and Good- fellow, Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, volume 31, 2018.
Vishwakarma, H., Tay, S. J., Namburi, S. S. S., Sala, F. Vinayak, R. K., et al. Pearls from pebbles: Improved confidence functions for auto-labeling. arXiv preprint arXiv:2404. 16188, 2024.
Oymak, S. and Guleu, T. C. Statistical and algorithmic insights for semi- seni-super learning with self-training. arXiv preprint arXiv:2006. 1006, 2020.
Rizve, M. N., Duarte, K., Rawat, Y. S., and Shah, M. In de- fense of pseudo-labeling: An uncertainty- pseudo- label selection framework for semi-supervised learning. In International Conference on Learning Representations, 2021.
Wang, Y., Chen, H., Fan, Y., Sun, W., Tao, R., Hou, W., Wang, R., Yang, L., Zhou, Z., Guo, L.-Z., Qi, H., Wu, Z., Li, Y.-F, Nakamura, S., Ye, W., Savvides, M., Raj, B., Shinozaki, T., Schiele, B.. Wang, J., Xie, X., and Zhang, Y. Usb: A unified supi-supee learning benchmark

---

Rethinking Confidence Scores and Thresholds in Pscudolabeling-base SSL
for classification. In Thirty-sixth Conference on Neural In- formation Processing Systems, Datasets and Benchmarks Track, 2022.
Wang, Y., Chen, H., Heng, Q. Hou, W., Fan, Y., Wu, Z.,, Wang, J., Savvides, M., Shinozaki, T., Raj, B., Schiele, B., and Xie, X. Freematch: Self-adaptive threshold- ing for semi-supervised learning. In The Eleventh In- ternational Conference on Learning Representations, 2023. URL www.h:tttttt....m http:///eeeeeeeeeeeeeeeeeeeeeeee id-PDrUPTXJI A.
Xie, Q. Dai. Z.. Hovy, E., Luong, T., and Le, Q. Un- supervised data augmentation for consistency training. In Advances in Neural Information Processing Systems, volume 33, 2020.
Xu, Y., Shang, L.. Ye. J., Qian, Q., Li, Y.-F., Sun, B., Li, H., and Jin, R. Dash: Semi-supervised learning with dynamic thresholding. In International Conference on Machine Learning, pp. 11525-15330 PMLR, 2021.
Zadrozny, B. and Elkan, C. Transforming classifier scores into accurate multiclass probability estimates. In Proceed- ings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 694 2002.
Zhang, B., Wang, Y., Hou, W., Wu, H., Wang, J., Oku- mura, M.. and Shinozaki, T. Flexmatch: Boosting semi- supervised learning with curriculum pseudo labeling. Ad- vances in Neural Information Processing Systems, 34: 18408-18419, 2021.
Zheng, M., You, S., Huang, L., Wang, F., Qian, C., and Xu, C. Simmatch: Semi-supervised learning with similarity matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14471- 14481, 2022.
Zhu, X. Semi-supervised learning literature survey. In Uni- versity of Wisconsin-Madison Department of Computer Sciences, 2005.
12

---

Rethinking Confidence Scores and Thresholds in Pscudolabeling-basee SSL
Supplementary Material
The supplementary material organized as follows, First, we summarize the notations in Table 6 in Appendix A, then we provide formal algorithms in Appendix B and additional experimental results and details are provided Appendix
A. Glossary
The notations are summarized in Table 6 below.
B. Detailed Algorithms
Algorithm Estimate Pseudolabeling Thresholds Classwise
Input: Confidence function D. classifier h4, Part of validation data ( for threshold estimation, pseudolabeling error tolerance c, space of thresholds T. label space V.
Output: Pseudolabeling thresholds t, where t, [u] is the threshold for class y-
for y E y do
Extract the set of points ( for which the groundtruth class is y.
D - (x',x')) E Dih y =
T (ooo).
# Estimate pseudolabeling error at each threshold on class specific data (v) Pick the smallest threshold with the sum of the estimated error and C1 times the std. deviation is below Here C1 is set to 0.25 and @(z) /=(1 (2. [y] + minft E T EGG., t h4, ) + Ciō(E(0,, t hi, (() S e}. end for return E
Algorithm 2 Estimate Pseudolabeling Threshold Jointly for All Classes
Input:
Confidence function D. classifier 7e, Part of validation data D for threshold estimation, pseudolabeling error tolerance e, space of thresholds T. label space V.
Output: Pseudolabeling thresholds t,, where t; [y] is the threshold for class y.
T' - TUT(Os }
Estimate pscudolabeling error at each threshold on the entire set Din Pick the smallest threshold with the sum of the estimated error and C₁ times is below e. Here C₁ is set to 0.25 and ((z) 1x(1-1))
min E T' E Tu, Dus) + Cia(E(O,, Thi Daa)) s e).
for y E y do
E, +- L end for return t;
13

---

Rethinking Confidence Scores and Thresholds in Pscudolabeling-base SSL
Algorithm 3 Pseudolabeling Based SSL with PabLO
Input: Labeled data for training D;, validation data Dval, unlabeled pool X., enror tolerance e, use-accumulation flag, num_iters, batch size B, replication factor y,, weak and strong Ω2 augmentations, number of calibration points Neal- num, of threshold estimation points Nuh frequency of invoking PabLO F, space of thresholds T, label space y.
Output: heal. model with the best validation accuracy.
# Set initial pseudolabels and masks to O.
Y, [0. .o.., Oj. Sh [0, o..... 0], <
Draw calibration and threshold estimation sets from Dval-
Deat, Ds + DrawRandomly (Dal, Neat, Nth).
Training loop with pseudolabeling.
while v num .iters do
Draw batches Dj, X of labeled and unlabeled points, Ta denotes the indices corresponding to points in X Dj, X u. Iu + DrawRandomBat
p.x,; B)
Create weak and strong augmentations of x
X uw: X
(X S2(X))
s Begin Pseudolabeling Block **/
# Perform pseudolabeling using PabLO.
if 1%F 0 then
# Get D: by solving optimization (P1).
Is. F f SolveOptProbllet
(h, Deal)
# Estimate pseudolabeling thresholds.
if estimate threshold classwise then
# Use Algorithm 1.
t; + Classwi Seaasssssthrreshhlllf.... hu Den, e, T,))
else
# Use Algorithm 2.
E + Joint hu, Dth, T,))
end if
# Compute fresh psuedolabels y and pseudolabeling masks S( for all points in X..
( + (X,,)))) s + 11 ( (w.,))) 2
if usc-accumulation then
# Apply pseudolabel accumulation if enabled. (11-(())) )()))((())) ((--1
()) S (1)) +1
((()) + s(0 v ((i-))
end if
else
Ir. i, -:- E
end if
** End Pseudolabeling Block #*/
Extract pseudolabels and masks for the current unlabeled batch. Then compute supervised and unsupervised losses.
yb. Sh +- Y_(Th), S.(72]
(h.) + supervised.loss(M D.)
(ha unsupervised loss Xb xb x Vb st))
Z(h.)) + (.) + (h.
# Perform gradient descent step to get new model h441-
7et SGD- -GD.updatte....e...emppp tpdate ([(())))) i i- i+1
Evaluate model on Dval to keep track of the best model. if 1%eval.freq = 0 then
eval acc +- eva 1 uate model 1 (ha, Dvat) If eval_ace is best so far then hast = h; end if end while
14

---

Rethinking Confidence Scores and Thresholds in Pscudolabeling-base based SSL
CIFAR-10
CIFAR-100
SVHN
Batch Pl. Accuracy
100
100
100
75
75
75
50
50
50
25
25
25
0
0
0
5K
15K
25K
35K
10K
30K
50K
70K
50K
100K
150K
Fixmatch
Fixmatch Ours
Fixmatch + BAM
Fixmatch + MR
Batch Pl. Accuracy
100
100
100
75
75
75
50
50
50
25
25
25
0
0
5K
15K
25K
35K
10K
30K
50K
70K
50K
100K
150K
Iterations
Iterations
Iterations
Freematch
Freematch + Ours
Freematch + BAM
Freematch + MR
Figure 4. Batch pscudolabel accuracy of our method and baselines on CIFAR-I0, CIFAR- 100, and SVHN. We plot the values for every 200 steps.
C. Additional Experiments and Details
Compute. We ran all of our experiments on a segh-throughput system with various GPUs. Therefore, each individual experiment task may be scheduled among NVIDIA A100 SXM4-40GB, NVIDIA A100 SXM4-80GB, NVIDIA L40, and NVIDIA H100 80GB HBM3. We measured the runtime of our algorithm on a desktop with a single NVIDIA RTX 4090. CIFAR-10, 10, it took about 0.203 seconds for each iteration for our method and around 140 seconds for baselines. CIFAR- 100, it took about 0.396 seconds for each iteration for our method and around 0.143 seconds for the baselines. On SVHN, it took about 1.275 seconds for each iteration for method and around 0.225 seconds for the baselines.
Hyperparameters. For the baselines, we have used their default settings. To maintain consistency and experiment the efficiency of method, we used WRN-28-2 which is 1.4M parameter model for all the datasets. We summarize the main hyperparameters we have used in our method in Table 10.
Note that the number of epochs we used to train the function and to estimate is dynamic. That is, its actual value depends on and is proportional the current number of iterations of the SSL training. More concretely, iteration of SSL training, use minn(1/20 max epoch) number of epochs to find g and t.
We additionally conduct the following ablation study to study our technique's dependence the amount of data used in learning and thresholds.
A2. How much data is needed to learn the g and t? We take Nea and Nu. from the validation data to learn the confidence function y and estimate the thresholds respectively. Intuitively larger values of these should lead to good g and that can extract the expected level pseudolabeling coverage and accuracy from the classifier hand. However, the task of learning good g and estimating thresholds this better run method with Nea and No. in (250, 500, 750, 1000} CIFAR- 10 setting for random seeds and report the result in Fig 8. We observe that our method can achieve desired performance with just 500 labeled points (i.e 50 labels per class), This is interesting because we can achieve 90% accuracy by just using 250 points (N₂) for training / and a total of 1K for learning g. Refer to Table for more details.
super hard and expect it will take fewer samples be successful. To understand
15

---

Rethinking Confidence Scores and Thresholds in Pscudolabeling-base based SSL
CIFAR-10 10
CIFAR-100
SVHN
Batch Pl. Coverage
100
100
100
75
75
75
50
50
50
25
25
25
0
0
5K
15K
25K
35K
10K
30K
50K
70K
50K
100K
150K
Fixmatch
Fixmatch + Ours
Fixmatch + BAM
Fixmatch + MR
Batch Pl. Coverage
100
100
100
75
75
75
50
50
50
25
25
25
0
0
0
5K
15K
25K
35K
10K
30K
50K
70K
0
50K
100K
150K
Iterations
Iterations
Iterations
Freematch
Freematch Ours
Freematch + BAM
Freematch MR
Figure Batch pscudolabel coverage of our method and baselines on CIFAR-IO, 10, CIFAR- 100, and SVHN, We plot the values for every 200 steps.
Top-1 Accuracy
Batch Pl, Accuracy
Batch Pl. Coverage
100
100
100
80
80
80
60
60
40
20
20
20
10K
20K
30K
10K
20K
30K
10K
20K
30K
Iterations
Iterations
Iterations
Fixmatch (no accu.)
Fixmatch + MR (no accu.)
Fixmatch + BAM (accu.)
Fixmatch + BAM (no accu.)
Fixmatch (accu.)
Fixmatch + MR (accu.)
Figure 6. (A1.) Left to Right: Top-1 Accuracy, Batched pseudolabeling Accuracy, and batched pseudolabeling coverage of Fixmatch with and without pscudolaheling accumulation enabled on CIFAR-10. It can be seen that enabling pscudolaheling accumulation worsen the nerformance of baseline methods in terms of accuracy and coverage.
16

---

Rethinking Confidence Scores and Thresholds in Pscudolabeling-base based SSL
Symbol
Definition
X'
feature space.
V
label space i.c. 1, ..
hypothesis space (model class for the classifiers).
g
space of confidence functions.
k
number of classes.
x, y
is datapoint in X' and y is its true label (if available).
h
a model h X' + y
confidence function X'' + T R hard label prediction. y is used as pseudolabel. a best learned model using SSL.
y
hast
pseudolabeling eтror tolerance.
g:
optimal confidence function at i iteration.
t?
optimal threshold at iteration.
X
available unlabeled data drawn from the distribution Pa over X. batch of unlabeled data.
X
Dr
set of labeled data points drawn from the distribution Pzg.
D
batch of labeled data.
Dval
validation data.
Deal
calibration data; part of validation data used to optimize surrogate functions. part validation data estimate threshold t.
Da
k dimensional vector of thresholds representing for k classes
tly
yth entry of t i.e. the threshold for class y.
number of unlabeled points, i.e. size of X, used for consistency regularization and pseudolabeling.
Ni
number of labeled points, i.e. size of D,. Usual SSL setting has, N₁ < fuu
Nva
number of points used for model selection.
Nustt
number of test data points.
Neal
number of points used for learning the g function. number of data points used for threshold estimation. supervised loss.
No
E.
C.
unsupervised loss with weighted importance Au- sum of regularization terms for supervised and unsupervised loss with weighted importance Xy- standard cross-entropy loss.
C,
Hly, h, x)
S(x,9, g, t h))
pseudolabeleing mask.
w
weak transformation, w X'
Ω
strong transformation, Q X' X.
ao. ab
average time taken by our method and baseline methods.
P(g.t h,
estimated pseudolabeling coverage, see eq. (1).
P(g,t h)
population level pseudolabeling coverage, eq. (2).
E(g. t h D)
estimated pseudolabeling error, see eq. (3).
E(g,t] h)
population level pseudolabeling error, see eq. (4).
P(g.t h D)
surrogate estimated pseudolabeling coverage, see eq. (6).
E(g.t t h, D)
surrogate estimated pseudolabeling error, see eq. (7).
X
hyperparamter controlling the importance of pseudolabeleing coverage and error in (P1).
Table Glossary variables and symbols used in this paper.
17

---

Rethinking Confidence Scores and Thresholds in Pscudolabeling-basee SSL
Top-1 Accuracy
Batch Pl. Accuracy
Batch Pl. Coverage
100
80
80
80
40
40
40
20
10K
20K Iterations
30K
10K
20K Iterations
20K
30K
Iterations
Freematch (no accu.)
Freematch + MR (no accu.)
Freematch + BAM (accu.)
Freematch + BAM (no accu.)
Freematch (accu.)
Freematch + MR (accu.)
Figure 7. (A1.) Left Right: Top-1 Accuracy, Batched pseudolabeling Accuracy, and batched pscudolabeling coverage of Frcematch with and without pscudolabeling accumulation enabled on CIFAR-10. It can be seen that enabling pscudolabeling accumulation worsen the performance baseline methods terms accuracy and coverage.
Top-1 Accuracy
90%
Fixmatch + Ours
80%
Freematch + Ours
400
600
800
1000
Nth and Ncal
Figure 8. Top-1 accuracy of our method with different No. and Neat-
Method
Nea1 = No., = 250
Neal = No. 500
Neal = No. = 750
Fixmatch Ours
82.67 + 7.08
91.74 +: 0.41
91.66 +:2 2.11
Freematch Ours
82.13 + 7.93
92.33 + 0.49
93.20 + 0.53
Table Results on CIFAR-10 with varying Neat and Nu.
Method
0.01
0.1
0.2
0.4
Fixmatch + Ours
93.05 + 0.54
91.54 + 0.95
88.35 + 2.90
56.72 + 22.25
Freematch Ours
92.11 + 1.18
92.31 + 0.16
83.89 + 10.36
52.17 + 25.36
Table Results CIFAR-10 with varying
18

---

Rethinking Confidence Scores and Thresholds in Pscudolabeling-basee based SSL
Method
0.01
0.1
0.2
0.4
Fixmatch + Ours
69.19 + 1.13
65.01 + 0.34
53.88 + 8.15
23.58 = 18.21
Freematch + Ours
70.13 + 0.67
64.95 + 1.41
59.83 + 1.32
24.09 = 17.22
Table 9. Results CIFAR-100 with varying
Method
Hyperparameter
Values
optimizer
SGD
learning rate
0.01
batch size
64
Learning function
max epoch
500
weight decay
0.01
momentum
0.9
optimizer
SGD
learning rate
0.01
batch size
64
Estimating t
max epoch
500
weight decay
0.01
momentum
0.9
Table 10. Hyperparameters used for our method.
19


Reviews:
[{"review_id": "2LDoM9KnYx", "reviewer": "Reviewer_cbpA", "summary": "The paper proposes PabLO, a framework for improving pseudolabeling-based semi-supervised learning (SSL) by learning confidence scores and thresholds with explicit control over pseudolabeling error tolerance. The core idea is to formulate pseudolabeling as an optimization problem that maximizes coverage while bounding error, replacing heuristic thresholding strategies. PabLO integrates with existing SSL methods and introduces pseudolabel accumulation to reuse high-confidence labels. Experiments on CIFAR-10/100 and SVHN demonstrate significant accuracy improvements over baselines.\n\n## update after rebuttal\n\nThe author's response has partially addressed my concerns. After reviewing the other reviewers' comments and the author's rebuttal, I believe this paper still requires further improvements. Therefore, I maintain my rating.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the feedback and the noted strengths of our paper. Our work is well-positioned in the literature on SSL and confidence calibration. Our principled methods to learn confidence scores and thresholds with error bounds replace the heuristic-based choices and enhance the prior SSL methods. Our response to the queries is as follows,\n\n**Generalization/transfer to unlabeled data.** Yes, we assume that the validation data and the unlabeled data are independent and identically distributed (i.i.d). Under this assumption, it is easy to show using standard uniform convergence results that the optimal solution on the validation set will transfer (generalize) to the unlabeled set. More specifically, our method uses parts of the validation data:  $N_{cal}$ samples for learning the confidence scores and $N_{th}$ samples for estimating thresholds. Using the uniform convergence results, we can show that the excess pseudolabeling error when transferring the solution to the unalabeled data will be $O(1/{\\sqrt{N_{cal}}} + 1/{\\sqrt{N_{th}}})$. Thus, as long as  $N_{cal}$ and $N_{th}$ are sufficiently large and the i.i.d assumption is satisfied, the solution will generalize to the unlabeled data.\n\nThe setting where the distributions of unlabeled and validation data are not aligned is interesting, and adapting our methods to these settings would be a fruitful direction for future work.\n\n\n**On pseudolabel accumulation.** For the accumulation procedure, our hypothesis is that it may help the high-precision pseudolabeling methods, but it may not be useful for methods with noisy pseudolabels. We do not expect it to help baselines, and the results in Table 4 support this. The accumulation method is generally helpful when used in concert with our confidence scores and thresholds that can ensure high-precision pseudolabels. \n\n**More baselines and datasets.** Our focus is on principled choices for confidence scores and thresholds for pseudolabeling based SSL, and we proposed solutions to this end. The goal of our experiments is to show that modern and commonly used methods, such as Fixmatch, Freematch, can be adapted to use our scores and thresholds, and the performance of the resulting method is better in comparison to using their standard versions. Since our focus is on confidence functions, we included baselines Margin Regularization (MR) and Bayesian Model Averaging (BAM) that are aimed at improving the calibration of confidence scores for pseudolabeling. \n\nWe used common benchmark datasets in SSL, that are sufficient for our empirical analysis. Introducing stronger benchmarks for SSL would be interesting future work. \n\n\n**Fixmatch + SVHN Case.** We point the reviewer to Figure 2 (top, right), where we plot the test accuracy over iterations for this case. This plot suggests that all the methods for the Fixmatch + SVHN case have similar performance. \n\n\n**Related works.** We have updated the related works with a discussion on the shared references. \n\n**Discussion on runtime.** It is correct that additional training for confidence functions and calculating thresholds increases runtime. For this reason, we adjust the training iterations of the baselines so that all the methods are run for the same amount of time. We have included the details on runtime in Appendix C. We are happy to provide any specific details the reviewer is interested in. \n\nOn running the CIFAR-100 experiment longer. For fair comparison, we fixed our experiment protocol, i.e., run the variations with our method for 25K iterations and the baselines for the equivalent number of iterations in the same amount of time. Thus, the evaluation is fair. Prior works run till 1 million iterations to reach near convergence. Note that this takes an enormous amount of time, and one of the advantages of using our method is that it can achieve high accuracy earlier and does not require running very long.\n\n**Observed pseudolabeling errors.** We plot the pseudolabeling error and coverage for the Cifar-10 setting. **[Please see the plot on this link](https://tinyurl.com/j2hb77fs)**.  We see that the observed pseudolabeling error is very close to the target $\\epsilon$ when it starts to give non-trivial pseudolabeling coverage, and it approaches $\\epsilon$ as the training progresses.\n\n\n**Impact of confidence function architecture and its alternatives.** Our framework to learn confidence scores is flexible to work with any choice of $\\mathcal{G}$. While the choice of the function class $\\mathcal{G}$ is up to the user, in general, it makes sense to use a flexible non-linear function class. A multi-layer neural network with any activation function could be a good fit here. We chose a 2-layer NN since the classification model is doing the heavy lifting of learning the features, so for $g$ we do not need a highly complex network.\n\n\nWe hope our response resolves the queries. We are happy to answer any more questions you may have.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "1. The authors' method essentially uses the validation set to identify confidence functions and thresholds that can be transferred to the unlabeled training set. It seems to implicitly assume a strong alignment between the distributions of these two sets. Is there any theoretical proof that guarantees the optimal solution on the validation set can be transferred to the unlabeled training set? Moreover, have the authors considered whether the method would still be effective in cases where the distributions are not aligned?\n2. The pseudolabel accumulation method proposed by the authors appears to be immature. In Table 4, it fails to consistently improve the results across different methods, which significantly limits the generalizability of this approach.\n3. In the evaluation, the authors used three datasets, among which CIFAR-10 and SVHN are relatively simple. It is recommended that the authors introduce stronger benchmarks. Furthermore, the authors only conducted experiments on two SSL methods (Fixmatch, Freematch), which is too limited to demonstrate the generalizability of their approach. It would be beneficial to include more baselines to enhance the persuasiveness of their method.", "theoretical_claims": "The paper does not provide formal theoretical guarantees.", "experimental_designs_or_analyses": "1. In the paragraph titled \"Adjusted iterations for baselines,\" the authors limited the number of iterations for the methods. Have the authors considered the convergence of the method? In the CIFAR100 experiment plots, it appears that the methods have not yet reached the convergence iterations.\n2. Regarding the failure of the method in the Fixmatch + SVHN experiment, the authors did not provide a detailed explanation for the cause.\n3. The impact of the confidence function architecture (2-layer NN) is unexplored.", "relation_to_broader_scientific_literature": "The work builds on SSL and confidence calibration. It extends prior SSL methods by replacing heuristic thresholds with learned, error-bounded ones.", "essential_references_not_discussed": "There are many SSL methods that the authors should ideally introduce and compare experimentally, such as SimMatch (CVPR 2022), SoftMatch (ICLR 2023), AdaMatch (ICLR 2022), and so on.", "other_strengths_and_weaknesses": "PabLO integrates seamlessly with existing SSL frameworks, but training confidence functions and calculating thresholds increase runtime, and the discussion regarding runtime is not sufficiently comprehensive.", "other_comments_or_suggestions": "Nothing.", "questions_for_authors": "1. How does the actual pseudolabeling error during training compare to the target ϵ? Could you provide per-iteration error measurements?\n2. Why use a 2-layer NN? Have you explored alternatives?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WIFtSQ4fnC", "reviewer": "Reviewer_S5pP", "summary": "This paper proposes PabLo, a novel method for semi-supervised learning. The authors conceive their approach through noting that the threshold for selecting pseudolabels from the teacher model should be both permissive enough to allow for a large degree of supervision, while not being so permissive as to introduce low-quality labels. PabLo proposes solving a surrogate optimization problem during each SSL training iteration, in which pseudolabeling thresholds are chosen to maximize pseudolabel coverage while constrained by a maximum error bound. Additionally, PabLo introduces a pseudolabel accumulation step, in which pseudolabels from previous training iterations can be brought into the current update if it is otherwise missed by the current teacher model. The authors validate their approach on three different image classification datasets, demonstrating an improvement when used in conjunction with common SSL approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and the noted strengths — *a lightweight, intuitive, and theoretical framework to learn scores and thresholds that can be integrated with existing SSL approaches to improve their performance*. Our response to the queries is as follows,\n\n**More labeled data.**  We have evaluated the methods on settings with smaller $N_l$. Specifically, using $N_l=$ 40 for Cifar-10 and SVHN and 400 for Cifar-100. **[The results are available here](https://anonymous.4open.science/r/icml-rebuttal-2024-anon-E247/cifar10_cifar100_svhn_low_label.png)**. The results are consistent with claims in the main paper and even more pronounced than those in Table 2.\n\n\n**On pseudolabel accumulation.** For the accumulation procedure our hypothesis is that it may help the high-precision pseudolabeling methods but it may not be useful for methods with noisy pseudolabels. We do not expect it to help baselines and the results in Table 4 are for completion — to show the performance of baselines with accumulation as well. The accumulation method is generally helpful when used in concert with our confidence scores and thresholds that can ensure high-precision pseudolabels. More sophisticated accumulation strategies accounting for the staleness of pseudolabels might be interesting to explore for future works.\n\n\n**On hyperparameters.** While several of the SSL baselines also require choosing hyperparameters, we provide guidance on selecting the hyperparameters $N_{cal}$, $N_{th}$, $\\epsilon$. The results in Table 6, suggest setting $N_{cal}, N_{th}$ to as high value as possible will be favorable. For $\\epsilon$ a small value ( $\\le$ 1%) is favorable when $N_l$ is not too small (i.e. when the initial models are not expected to be too bad); when we expect the initial models to be bad (or have $N_l$ too small), then using slightly higher epsilon around 5% is favorable. We will include a detailed discussion on these recommendations in the paper.\n\n\n\n**Class-wise vs global thresholds.** Class-wise threshold estimation is suitable when there are less number of classes and global threshold estimation is suitable when the number of classes is large. We use class-wise for CIFAR-10 and SVHN settings and global for CIFAR-100 cases. The rationale behind this is, that when the number of classes is large it is hard to have sufficient samples for each class to learn meaningful thresholds. Thus in such settings, it is more useful to learn a global threshold.\n\n**Moving related works section earlier.**  We will move this in the camera-ready version.\n\n**Thresholds over iterations.** Since we learn new scores in each pseudolabeling iteration, the scale of scores is not consistent across the iterations and thus we do not see a pattern in thresholds over iterations. However, Figures 4 and 5, provide insights into how the quantity and quality of pseudolabels evolve over time.\n\nWe hope our response resolves the queries. We are happy to answer any further questions you may have.", "claims_and_evidence": "The claims presented in the paper are all valid.", "methods_and_evaluation_criteria": "The proposed experimental methods are valid.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design is sound.", "relation_to_broader_scientific_literature": "This paper investigates thresholding in the context of pseudolabeling beyond the level of simply using heuristic techniques, as has been tried previously in work such as FixMatch. The more principled approach of choosing thresholds improves upon these approaches and can be integrated directly with them.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- The method is lightweight and can easily be integrated with currently existing SSL approaches.\n- Improvement over baselines in the provided experiments is convincing.\n- The theoretical framework which motivates PabLo is intuitive, and the careful analysis of the pseudolabeling threshold is principled.\n\nWeaknesses:\n- The experiments are limited to a single labeled fraction for each dataset; the results would be more convincing if multiple labeled data fractions were evaluated for each dataset.\n- The psuedolabel accumulation step, although introduced as a possible improvement, does not meaningfully improve the test accuracy.\n- PabLO includes several hyperparameters chosen heuristically (e.g. $N_{cal}$, $N_{th}$, $\\epsilon$). This complexity of selecting all of these limits the practical usability of the method.\n- The authors introduce two algorithms for selecting thresholds, either selecting class-wise thresholds or global thresholds. However, it is not made clear which is adopted in the final algorithm, and no comparison between the two is included in the ablation studies.", "other_comments_or_suggestions": "I believe the paper will flow better if the related work section was moved to the beginning of the paper, after the introduction.", "questions_for_authors": "- See weaknesses: does the final algorithm use class-wise or global thresholds (e.g. algorithm 1 or 2)?\n- Have the authors made plots visualizing the pseudolabel threshold as a function of training iteration (similar to Fig 5)? It would be interesting to see how the value of the threshold evolves during training, as this is subtly different from the pseudolabel coverage percent plotted in Fig 5.\n- What is the authors intention of the analysis on pseudolabel accumulation, as the approach ultimately does not improve PabLo's performance? Have the authors investigated more sophisticated strategies for pseudolabel accumulation (e.g. downweighting the contribution of pseudolabels accumulated from past training iterations)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iEMIgv5QKY", "reviewer": "Reviewer_koUF", "summary": "This paper introduces a principled framework for improving pseudolabeling-based semi-supervised learning (SSL) by explicitly controlling confidence scores and thresholds to manage pseudolabel quality and quantity. The approach addresses limitations of heuristic-driven methods, offering a systematic way to balance pseudolabel accuracy and coverage. Extensive experiments including ablation studies confirm the effectiveness of the optimization framework and threshold estimation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the careful review and positive feedback.  We appreciate the recognition of the strengths of our work — *a flexible and principled approach for learning confidence scores and thresholds for pseudolabeling and its empirical effectiveness*. Our response to the queries is as follows,\n\n**Differences in insights and techniques compared to [1]**\n\nThe prior work [1] studies a procedure to create labeled datasets with accuracy guarantees. Thus, [1] is only concerned with labeling the data correctly and not with how good the end model is. In contrast, our work is on semi-supervised learning, where **our goal is to learn a classifier with good generalization error**. In this procedure, the pseudolabels are never \"committed\", i.e., in each iteration, the assigned pseudolabel to a point can change. In contrast, in [1], the label assigned to a point is never changed. Furthermore, our approach and insights are novel within the area of pseudolabeling-based semi-supervised learning (SSL) [4,5]. We provide two points on this:\n\nFirst, our work settles the question of the right choices of confidence functions for SSL. Recent works [2,3] have highlighted the problem of miscalibrated confidence functions, leading to inefficiencies in pseudolabeling-based SSL. Although these works offer solutions to this problem, they still fall short of addressing the core issue: the confidence function is not specifically tailored to the needs of SSL. To provide a principled solution to this problem, we adapt the framework for learning confidence functions in the TBAL setting from [1] to the SSL setting. \n\nSecond, we show how this framework for learning confidence functions can work in concert with popular SSL methods such as Fixmatch [4], Freematch [5], etc., and conduct an extensive empirical evaluation demonstrating that using confidence function learned from our method can yield significant improvements in the test accuracy. The flexibility to be integrated into a variety of techniques is novel to this work; works like [1] did not need to offer it. As a result, our work provides a flexible solution to practitioners using SSL, freeing them from the trial and error of selecting various confidence functions or hand-crafting one.\n\n[1] Vishwakarma, H., Lin, H., Sala, F., and Vinayak, R. K. Promises and pitfalls of threshold-based auto-labeling. NeurIPS, 2023\n\n[2] Loh et al., Mitigating confirmation bias in semi-supervised learning via efficient bayesian model averaging. TMLR, 2023.\n\n[3] Mishra et al., Do not trust what you trust: Miscalibration in semi-supervised learning, arXiv, 2024.\n\n[4] Sohn et al., Fixmatch: Simplifying semi-supervised learning with consistency and confidence, NeurIPS, 2020.\n\n[5] Wang et al., Freematch: Self-adaptive thresholding for semi-supervised learning, ICLR, 2023.\n\n\nWe hope our response resolves your queries. We are happy to answer any further questions.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Not applicable, as this paper does not put forward any theoretical claims or provide corresponding proofs.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The key contributions of the paper are closely related to the fields of semi-supervised learning and pseudolabeling. To leverage unlabeled data to train the predictive model under semi-supervised learning, the authors proposes a approach to learn confidence scores and thresholds via an optimization problem in pseudolabeling unlabeled data that maximizes pseudolabel coverage while ensuring error rates remain below a target tolerance.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper proposes a approach to learn confidence scores and thresholds via an optimization problem that maximizes pseudolabel coverage while ensuring error rates remain below a target tolerance (ϵ), which addresses limitations of heuristic-driven methods, offering a systematic way to balance pseudolabel accuracy and coverage.\n2. The framework could be flexibility combined with popular SSL methods (e.g., Fixmatch, Freematch), enhancing their performance by leveraging high-quality pseudolabels. \n3. Ablation studies confirm the effectiveness of the optimization framework and threshold estimation.\n\nWeaknesses:\n1. My main concern is that the core technique used in this paper, which estimates thresholds on the test set to filter pseudolabels, closely resembles that of [1], despite the paper citing [1] and acknowledging this in Line 231 “Similar procedures have been used in the context of creating reliable datasets and are backed by theoretical guarantees for the quality of pseudolabels produced”. Since the auto-labeling method proposed in [1] can also be interpreted as a form of pseudolabeling, the contributions of this work appear incremental. The paper should further discuss the differences in insights and techniques compared to [1], such as in their objectives, algorithms, or validation strategies.\n[1] Vishwakarma, H., Lin, H., Sala, F., and Vinayak, R. K. Promises and pitfalls of threshold-based auto-labeling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.", "other_comments_or_suggestions": "Please refer to the weaknesses above.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dlcC8Pt9QC", "reviewer": "Reviewer_sFAy", "summary": "This paper proposes a method for selection of points to be pseudolabeled in pseudolabeling-based semi-supervised learning idea. Contrasting previous works which use confidence-based thresholding, PaBlo trains a selector function with an optimization objective which balances coverage with pseudolabeling error, using a subset of the held-out validation data.\n\nUpdate after rebuttal:\nApologies for the delay. Thanks for the additional results and new experiments. I'm raising my score to a Weak accept (3).\nHowever, I think the paper should incorporate an extensive discussion on the data availability assumptions that this method makes (validation data).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the feedback and questions. Our response is as follows,\n\n**On baselines that do not rely on confidence scores and thresholds.**\nWhile this would be interesting, our paper's focus is on pseudolabeling methods based on confidence scores and thresholds. For this reason, we chose baselines necessary to evaluate our claim — using our scores and thresholds in the common methods (FixMatch, FreeMatch) can improve the performance significantly, in contrast to using them with ad-hoc choices of scores and thresholding techniques. We also compared the variations of these with recent methods (BAM, MR) designed to encourage calibrated scores in SSL settings. \n\n\n**On validation data requirements.** We clarify as follows. \n\n* *it is common practice in SSL methods to use validation data to find the best model checkpoint by evaluating validation accuracy*. The baselines use all $N_{val}$ validation samples for this purpose. \n\n* In our method, we split this $N_{val}$ data into three parts, $N_{val}'$, $N_{cal}$, $N_{th}$ and use $N_{val}'$ for the usual model checkpoint evaluation and use $N_{cal}, N_{th}$ for learning confidence function and estimating thresholds respectively. **Note $N_{cal}$ and $N_{th}$ are much smaller than $N_{val}$**.\n\n* The idea of using the validation data for training is natural. Common benchmarks have been extensively studied in the literature for which there is a reasonable understanding of hyperparameters and expected performance. However, this overlooks the generality of SSL methods. In general, most of the SSL methods need validation data for model selection, tuning hyperparameters, etc. This cost is often overlooked in the prior works. Our work mentions it transparently. \n\n**Clarification on Table 6 results.** These results are to study the effect of sizes of calibration $N_{cal}$ and threshold estimation sets $N_{th}$. When these are too small, it will result in high variance in learning scores and thresholds, resulting in unreliable pseudolabeling (high excess pseudolabeling error). Using uniform convergence results it can be shown that the excess pseudolabeling error will scale as $O(\\frac{1}{\\sqrt{N_{cal}}} +\\frac{1}{\\sqrt{N_{th}}})$. Thus we see as $N_{cal}, N_{th}$ increases the performance gets better. \n\n**Inverse U Shape curve for error tolerance.**  In general, we expect an inverse U shape for test accuracy vs epsilon curve. In the Fig. 3. setting, we did not see this. We have run this experiment in one more setting where we see the inverse U shape curve. We summarize our findings in two points,\n\n* We note that zero error tolerance is not equivalent to classical supervised learning. We see sizeable pseudolabeling taking place even with 0 error tolerance in certain settings, e.g., as in Figure 3, CIFAR-10 with 250 labeled points for training. **[Please see the new results here](https://anonymous.4open.science/r/icml-rebuttal-2024-anon-E247/cifar10_low_high_epsilon.png)** . This means reasonable models are found early in the training that are 100% accurate on part of the space and our scores, and thresholds are able to find that space.\n\n* Second, in settings with an even smaller amount of labeled data for training, we start to see the expected inverse U shape curve. We ran the procedure in the CIFAR-10 setting with 40 labeled points for training with various choices of $\\epsilon \\in$  { 0.1%, 1%, 5%, 20%, 40% }. **[See new results](https://anonymous.4open.science/r/icml-rebuttal-2024-anon-E247/cifar10_epsilon.png)** for the plot of test accuracy corresponding to each $\\epsilon$. The test accuracies are 74.6%, 91.5%, and 83.8% corresponding to $\\epsilon$ 0.1%, 1%, and 5% respectively.  \n\n\nWe hope our response resolves your queries. We are happy to answer any further questions.", "claims_and_evidence": "Claim 1: the adaptations of popular pseudolabelingbased SSL methods with PabLO output models with better test accuracy.\n\nThis claim is verified on 3 datasets (CIFAR 10, CIFAR 100 and SVHN) with baselines Freematch and FixMatch.\nIt would be interesting to include additional baselines which do not solely rely on simple confidence thresholds, such as uncertainty-based methods (e.g. [1], [2]) \n\n\n[1] Rizve, M. N., Duarte, K., Rawat, Y. S., & Shah, M. (2021). In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arXiv preprint arXiv:2101.06329.\n\n[2] Nguyen, V., Husain, H., Farfade, S., & Hengel, A. V. D. (2022). Confident sinkhorn allocation for pseudo-labeling. arXiv preprint arXiv:2206.05880.\n\nMy main question about this experimental section is about the data splits. Given the statistics in Table 1, it seems that the validation set (which is labeled) used for the training of the selector function is much bigger than the labeled dataset used for training (6K samples, while the labeled data is only 250 samples for CIFAR 10), which contradicts the main purpose of semi-supervised learning, where labeled data is scarce. This brings a natural question: what happens if you use the validation data as labeled training data, and just use one of the baseline methods? I expect the performance of the baselines to increase significantly by using this validation data naively.  Furthermore, the results in Table 6, for N_cal = N_th = 250 (approximately 82% top-1 accuracy) seem much lower than the results of the baselines Fixmatch and Freematch reported in Table 2 for CIFAR 10 (90.8% and 92.26%).  \n\nClaim 2: a lower error tolerance is preferable\n\nIt would be interesting to report the number of points which are pseudolabeled in Figure 3, and smaller values of epsilon. Indeed, with 0 error tolerance, we should recover classical supervised learning, which should be worse than pseudo-labeling. Hence it would be more intuitive to see an \"inverse\" U shape in Figure 3. \n\nClaim 3: Accumulation is not always helpful\n\nThis claim is supported with results on CIFAR-10.", "methods_and_evaluation_criteria": "The datasets are commonly used in the PL litterature.\nOther baselines could be included to compare with Pablo.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "See above:  My main question about this experimental section is about the data splits. Given the statistics in Table 1, it seems that the validation set (which is labeled) used for the training of the selector function is much bigger than the labeled dataset used for training (6K samples, while the labeled data is only 250 samples for CIFAR 10), which contradicts the main purpose of semi-supervised learning, where labeled data is scarce. This brings a natural question: what happens if you use the validation data as labeled training data, and just use one of the baseline methods? I expect the performance of the baselines to increase significantly by using this validation data naively.  Furthermore, the results in Table 6, for N_cal = N_th = 250 (approximately 82% top-1 accuracy) seem much lower than the results of the baselines Fixmatch and Freematch reported in Table 2 for CIFAR 10 (90.8% and 92.26%).", "relation_to_broader_scientific_literature": "This paper positions itself as an improvement over the confidence-based PL methods which use thresholding.", "essential_references_not_discussed": "The related work section covers the main strands of prior works related to this paper.", "other_strengths_and_weaknesses": "The paper is well-written and easy to follow.", "other_comments_or_suggestions": "NA", "questions_for_authors": "See above on the data splits + the results in Table 6.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a new method to optimise thresholds for pseudo-label techniques in semi-supervised learning. The idea is to maximise pseudo-label coverage while maintaining a good accuracy.

The method is sensible, the paper is nicely narrated, and the experimental results are very good (the method can be added as an additional brick to most popular semi-supervised algorithms).

The authors added nice experiments during the discussion phase, and clarified some points. While none of them was really willing to champion the paper, I still recommend acceptance. I strongly encourage the authors to add the additional experimental results to their paper, and also leverage some of the points that were discussed with reviewers. In particular, the use of validation data should be discussed more.
Decision: Accept (poster)