Pearls from Pebbles: Improved Confidence Functions for Auto-labeling
Authors: Harit Vishwakarma
Yi Chen
Sui Jiet Tay
Satya Sai Srinath Namburi GNVV
Frederic Sala
Ramya Korlakai Vinayak
Venue: NeurIPS 2024 poster (2024)
Keywords: Auto-labeling
Confidence Calibration
Failure Prediction
Selective Classification
Primary area: active_learning
TL;DR: We introduce Colander, a novel post-hoc method addressing overconfidence in threshold-based auto-labeling (TBAL). Colander achieves up to 60% better coverage than baselines.

Abstract: Auto-labeling is an important family of techniques that produce labeled training sets with minimum manual annotation. A prominent variant, threshold-based auto-labeling (TBAL), works by finding thresholds on a model's confidence scores above which it can accurately automatically label unlabeled data. However, many models are known to produce overconfident scores, leading to poor TBAL  performance. While a natural idea is to apply off-the-shelf calibration methods to alleviate the overconfidence issue, we show that such methods fall short. Rather than experimenting with ad-hoc choices of confidence functions, we propose a framework for studying the optimal TBAL confidence function. We develop a tractable version of the framework to obtain Colander (Confidence functions for Efficient and Reliable Auto-labeling), a new post-hoc method specifically designed to maximize performance in TBAL systems. We perform an extensive empirical evaluation of Colander and compare it against methods designed for calibration. Colander achieves up to 60% improvement on coverage over the baselines while maintaining error level below 5% and using the same amount of labeled data.

Pearls from Pebbles: Improved Confidence Functions for Auto-labeling
Harit Vishwakarma avishwakarma@cs. sasc.edu University Wisconsin-Madise
Yi Chen
Sui Jiet Tay st54948nyu. edu NYU Courant Institute
yi. chen@wisc. edu
University of Wisconsin-
Satya Sai Srinath Namburi
Frederic Sala
www..nnbbbrrrgegeaaaaaaee com GE HealthCare
fredsala@cs, wisc edu University of Wisconsin-Madison
Ramya Korlakai Vinayak
ramyaßece wisc.edu University of Wisconsin-Madison
Abstract
Auto-labeling is an important family of techniques that produce labeled training sets with minimum manual annotation. A prominent variant, threshold-basee auto-labeling (TBAL), works by finding thresholds on a model's s confidence scores above which it can accurately automatically label unlabeled data. However, many models are known to produce overconfident scores, leading to poor TBAL per- formance. While natural idea is to apply off-the-shelf calibration methods alleviate the overconfidence issue, we show that such methods fall short. Rather than experimenting with ad-hoc choices of confidence functions, we propose framework for studying the optimal TBAL confidence function. We develop a tractable version of the framework obtain Colander (Confidence functions for Efficient and Reliable Auto-labeling), a new post-hoc method specifically designed maximize performance TBAL systems. We perform an extensive empirical evaluation of Col ander and compare it against methods designed for calibration. Colander achieves up 60% improvement on coverage over the baselines while maintaining eтror level below 5% and using the same amount of labeled data.
Introduction
The demand for labeled data in machine learning (ML) perpetual. Obtaining it is expensive and time-consuming, creating bottleneck in ML workflows. Threshold- based auto-labeling (TBAL) is promising solution obtain high-quality labeled data at low cost [47, 42, 56]. A TBAL system (Fig. 1) takes unlabeled data as input and outputs labeled dataset. It works iteratively: in each iteration, acquires human labels for a small chunk of data to train a model, then auto-labels points using the model's predictions where its confidence scores are above certain threshold. The threshold is determined using validation data so that auto-labeled points meet desired accurucy criteria. The goal is to maximize coverage the fraction of points automatically labeled (out of the total)-whin maintaining accuracy. TBAL powers industry products like Amazon SageMaker Ground Truth [47].
The confidence function is critical to the TBAL workflow (Figure 1). Existing TBAL systems rely on common choices like softmax outputs from neural networks [42, 56]. These functions are not well aligned with the objective of the auto-labeling system. Using them results in substantially suboptimal coverage (Figure 2(a)). For this reason, we ask:
Work done while at University of Wisconsin-Madison
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

What the right choices of confidence functions for TBAL and how obtain them?
An ideal confidence function for auto- labeling will achieve the maximum coverage given auto-labeling er- ror tolerance and thus will bring down the labeling cost significantly. Finding such an ideal function, however, is dif- ficult because of the inherent tension between accuracy and coverage. The models used in auto-labeling are often highly inaccurate so achieving certain error guarantee is easier when being conservative in terms of confidence- but this reduces coverage. Conversely, high coverage may appear to require lowering the requirements in confi- dence, but this may easily lead to overshooting the desired error level. This is compounded by the fact that TBAL is iterative, so even small deviations in error levels can cascade in future iterations.
Threshold-baseer Auto-labeling System
Superfevel
the confidonoe scores
Unlabeled Data
Unlabeled
Dval Get Human-isbeled Datts
Dirain
Darsin Train Model (ERM)
Daal Auto-labulerg
Human labeled labeled
h Auto-laheled
$0000
Confidence Scores 8
Labeled
Auto-label peints with confidinnce 2
Figure 1: High-level diagram of TBAL system.
Overconfiencce may further stymie hopes of balancing accuracy and coverage. While overconfidence a challenge in general, is exacerbated TBAL: since models are trained on small amount of labeled data, they are often inaccurate, making the problem of designing confidence functions even more challenging. Common choices produce overconfident scores, i.e., high scores for both correct and incorrect predictions [51, 38, 17, 16, 3J. Fig. 2(a) shows that softmax scores are overconfident, resulting in poor auto-labeling performance.
Several methods have been introduced to address overconfidence, including a variety of calibration techniques [12]. Applying these can nevertheless miss out on significant performance (Figure 2(b)) since the calibration goal differs from auto-labeling. From the auto-labeling standpoint, we seek minimum overlap between the correct and incorrect model prediction scores. Other approaches [6, 35] bake the objective of separating scores into model training or use different optimization procedures [64] that encourage separation. We observe that these do not help TBAL either, since, after some point, the model is correct almost all the training points, making it hard to train it to discriminate between its own correct and incorrect predictions.
We tackle these challenges by proposing framework to learn suitable confidence functions for TBAL. In particular, we express the auto-labeling objective as an optimization problem over the space of confidence functions and thresholds. Our framework subsumes existing methods. i.c., they are points in the space of solutions. The resulting method, Colander (Confidence functions for Efficient and Reliable Auto-labeling). relies practical surrogate to the framework that can be used to learn optimal confidence functions for auto-labeling. Using these learned functions in TBAL can achieve up to 60% improvement in coverage versus baselines like softmax, temperature scaling [12], CRL [35] and FMFP [64]. We summarize our contributions as follows,
1. We propose principled framework to study the choices of confidence functions suitable for nuto-labeling and provide a practical method (Colander) to learn confidence functions for efficient and reliable auto-labeling.
2. We systematically study commonly used choices of scoring functions and calibration methods and demonstrate that they lead to poor auto-labeling performance.
3. Through extensive empirical evaluation real-world datasets, show that using the confidence scores obtained using our procedure boosts auto-labeling performance significantly in comparison common choices of confidence functions and calibration methods.
2 Background and Motivation
provide notation, background on TBAL and its relationship to other methods, and describe the importance of confidence functions.

---

Inservet
Iromet
Incernet
Dunelyy
Dernaty
0 g
C
0.55
Sccens
Scores
Scare
50hme
(a) Softmax
(b) Temp. Scaling
(c) Colander (Ours)
(d) Coverage
(e) Auto-labeling Err.
Figure 2: Scores distributions (Kernel Density Estimates) of a CNN model trained on CIFAR-10 data. (a) softmax scores of the vanilla training procedure (SGD) (b) scores after post-hoc calibration using temperature scaling and (c) scores from our Colander procedure applied on the same model. For training the CNN model we use 4000 points drawn randomly and 1000 validation points (of which 500 are used for Temp. Scaling and Colander) ). The test accuracy of the model is 55%. Figures (d) and (e) show the coverage and auto-labeling error of these methods. The dotted-red line corresponds to user-given error tolerance of 5%.
Notation. Let [m]] :=
m for any natural number m. Let Xu be a set of unlabeled points drawn from some instance space Xt Let y {L....,k k} be the label space. There an unknown ground truth labeling function f" X - Y. Let (9 be noiseless oracle that provides the true label for any point x .X. Denote the model (hypothesis) class by H. where each f H is a function f X + Y. Each classifier h also has an associated confidence, function
.C - A that quantifies the confidence the prediction by model ft E H on any data point x E X.. Here, A is a (k- -)diimensional probability simplex. Let v[i] denote the 7th component for any vector v E Rd_ For any point x E X the prediction is 9 == (xx and the associated confidence is (xx)[) The vector denotes scores over k-classes, and tly] denotes its yth entry, i.e., score for class y. Table 3 (in Appendix B.4) contains summary of the notation.
Threshold- -based auto-labeling. seeks to obtain labeled datasets while reducing the labeling burden on humans (Figure 1). The input is a pool of unlabeled data X.. It outputs, for each x E X.. a label y E y. The output label could be either y, from the oracle (human). or ÿ. from the model. Let Nu be the number of unlabeled points, A S [Nu] the set of indices of auto-labeled points, and (A.) be these points. Let N₂ be the size of the luto-labeled set A. The auto- labelln error, denoted by (().(A)). and the coverage, denoted by (Xx
(A)), are defined as follows:
1 (A). = No iEA
and ((X.(A)) = LAI/N, = Na/Na.
(1)
The goal of an auto-labeling algorithm is to label the dataset so that ((),(A)) S ca while maximizing coverage P( ((..(A) for user-given error tolerance parameter ca E [0. 1]. As depicted in Figure 1, the TBAL algorithm proceeds iteratively. In each iteration, it queries labels for a subset of unlabeled points from the oracle. It trains classifier from the model class H on the oracle-labeled data acquired till that iteration. then uses the model's confidence scores on the validation data to identify the region in the instance space, where the current classifier is confidently accurate and automatically labels the points in this region. The auto-labeled points are removed from the unlabeled pool. Similarly, to maintain parity between the validation and unlabeled data in the next round, the validation points in the auto-labeling region are removed as well. These steps are executed in a loop until all the data is labeled the budget to query oracle labels is exhausted.
Fundamental differences between TBAL, self-training and active learning. At first glance, TBAL may appear similar active learning (AL) [46], self-training (ST) [2], and selective classifica- tion (SC) [10]. However, as described in [56], it is fundamentally different technique designed with different goals. Perhaps the most substantial difference is that TBAL's aim is to create accurately labeled datasets, while the goal in AL and ST is to learn the best (in terms of generalization error) possible classifier in a given model class with limited ground truth labels. This difference is most substantial in the settings where AL converges to a bad classifier, e.g., due to incorrect choice of the model class, sampling bias, etc. [56] illustrates this notion with a scenario where TBAL coverage above 95% while the other techniques average around 20%. See Appendix A.I for details.
3

---

Threshold-baser Auto-labeling System Colander
Colander
Prediciin
Prsticke
D
D
Continere
du 1
min f(g.t h4 D SeçRer
Unlabeled Data
X
X X t h. (
Uriabeled Human labeled
Get Human- labeled Data
Train Model
Learn confidence function for auto-labeling
Autoo labeted
Estimate eirors superievel sets of the confidence scores
Thouphald Uneabeed Parts
ti Autoa Mubollbelra Error
R. <f.,, D((
O 00
Extinated
Labeled Data
Confidence Boores
Convidence Sconee
Auto-label points with confidence 2
Estimate Threshoids
Threstolds
Figure 3: Threshold-bare based Auto-labeling with Colander: takes unlabeled data as input, selects a small subset of data points, and obtains human labels for them to create D(4) and D( for the ith iteration. Trains model h; on (
train" In contrast to the standard TBAL procedure, here we randomly split ( into two parts, ( and (( Colander kicks in, takes h, and D( input and learns coverage maximizing confidence function 9; for ht Using ( and ÿ: auto-labeling thresholds t; are determined to ensure the auto-labeled data has error at most ca (a user-given parameter). After obtaining the thresholds the rest of the steps are the same as standard TBAL. The whole workflow
th
runs until all the data labeled or another stopping criterion is met.
Problems with confidence functions in TBAL. The success of TBAL hinges on the ability of the classifier's 's confidence scores to distinguish between correct and incorrect labels. Prior works on TBAL [56, 42] train the model with stochastic gradient descent (SGD) and use the softmax output of the model as confidence scores, which are known to be overconfident [38]. A natural choice to mitigate this problem to use post-hoc calibration techniques, e.g.. temperature scaling [12]. We
evaluate these choices by running TBAL for a single round on the CIFAR-10 [24] dataset with SimpleCNN model with 5.8M parameters [20] with error threshold 5%. Details are in Appendix A.2.
In Figures 2(d) and 2(e) we observe that using softmax scores from the classifier only produces 2.9% coverage while the error threshold is violated with 10% error. Using temperature scaling only increases the coverage marginally to 4.9% and still violates the threshold with error 14%. Looking closer the scores for correct versus incorrect examples on validation data, we observe a large overlap for softmax (Figure 2(a)) and a marginal shift with considerable overlap for temperature scaling (Figure 2(b).. To overcome this challenge, we propose a novel framework (Section to learn such confidence functions in principled way. Our method in this example can achieve 50% coverage with an error of 3.4% within the desired threshold (Figure 2(c)).
3 Proposed Method (Colander)
The observations in Figure 2(a) and 2(b) suggest that fixed choices of confidence functions can leave significant coverage on the table. To find a better choice in a principled manner, develop framework based on auto-labeling objectives-maximiznng coverage while having bounded auto- labeling error. We instantiate by using empirical estimates and casy-to-optimin surrogates. We use the overall TBAL workflow from [56] and introduce our method to replace the confidence (scoring) function after training the classifier.
3.1 Auto-labeling optimization framework
In any iteration of TBAL, we have model fi trained on a subset of data labeled by the oracle. This model may not highly accurate. However, could be accurate some regions the instance space, and with the help of confidence function g, we want to identify the points where the model is correct and auto-label them. As we saw earlier, arbitrary choices of g perform poorly on this task. Instead, we propose framework to find the right function from a sufficiently rich family.

---

Optimal confidence function. To find confidence function aligned with our objective, we consider space thresholds T (e.g. [0. 1)) and functions G I - T where T is the k dimmenssional product space of T. We express the auto-labeling objective an optimization problem (P1):
arg max PEC.ETт*
P(g.t t ()) s.t. 8(g,t t A) I Ca.
(P1)
Here (/)) and E(g, | At) are the population level coverage and auto-labeling error which defined as follows,
P(g,t At) == Px (s(x) 2 (D)
and 8(g, t A) = (x + (x) 2 t(o])
(2)
The optimal g and t* that achieve the maximum coverage while satisfying the auto-labeling error constraint belong to the solution(s) this optimization problem.
3.2 Practical method learn confidence functions
The framework provides a theoretical characterization of the optimal confidence functions and thresholds for TBAL. However, it is impractical since the distributions and f* are unknown. Next, give practical method based on the above framework to learn confidence functions for TBAL.
Empirical optimization problem. Since we do not know the distributions of and /*, we use estimates of coverage and auto-labeling errors on a fraction validation data to solve the optimization problem. Let D be some finite number of labeled samples, and then the empirical coverage and auto-labeling error are defined as follows,
Egg.t h D) == DI
Σ (X.))ED
(g(()) 2 t()).
(3)
Egg t D) EDEAED 1 + ( 2 t()
(4)
EiEYED (g
We randomly split the validation data into two parts Deal and Duh and use Deal to compute ((g. h, Deat) and E(g,t fi, Deat). Using these estimates, we now seek to solve the following problem,
arg max .ET
Ögg,t t h, Dea)) ,(g,t t h, Deal) S ca-
(P2)
Nevertheless, the presence of 0-1 variables means the problem remains challenging.
Surrogate optimization problem. To make the optimization (P2) tractable using gradient-
based methods, we introduce differentiable surrogates for the 0-1 variables. Let (a, 1/(1 + exp(-az)) denote the sigmoid function on R with scale parameter a E R. It is easy see that, for any g. y and t. (xx) [y] 2 t[y] = aaa, [xx)[v] t(y)) 2 1/2. Using this fact, define the following surrogates of the auto-labeling error and coverage:
(gg, t/h,, Deat) = Deal
Σ (x.y)E
(a, -t(l),
(5)
Exxeye 1 + D) (a, (x) +())
Elg.t 1, Deal)
(6)
E(xa)e (x)]]] t(β))
and the surrogate optimization problem as follows.
arg min pE
Pgg ft, Deat) X8 t. h, Deat)
(P3)
Here, À E R+ is the penalty term controlling the relative importance of the auto-labeling error and coverage. We tune with the procedure discussed in Section 4.3. The gap between the surrogate and actual coverage diminishes + We discuss this in the Appendix.
5

---

Choice of G. Our framework is flexible with respect to the choice of function class G. In this work, we use neural networks with at least two layers on model class H. We use representations from the last two layers as imput for the functions in C (Figure 4). Let z (x;
I and (x; It) Rdd be the outputs of the last and the second-last layer of the net h for input x and let n(x; h)) (z(1) (x; fi), 2(2)( (x: fi]] denote the concatenation. This input is passed network Gung IR d A : it outputs confidence scores for the & classes. Specifically g is defined as ( g(x) == softmax (W.tanb(W1z(xx/ [Iz(x:())) Here W1 E I (R(+td)))(((eedd) and (2(++a) x the
learnable weight matrices. As usual, for v Rd, sof tofttax(/ (v) == exp(vli)/(E exp(v(i)) and tanh(v)(i == (cxp(2v(d]) (cxp((222( + 1).
We emphasize, Colander can use any function class for G X Th Here, chose 2-layer nets and successfully used the same across all experiments, thus we may not need an exhaustive architecture search. Intuitively, we do not need a large network for g since f already performs the heavier representation learning work. As result, simple models are preferable to avoid overfitting and to reduce training time since post-hoc hoc methods should be fast.
classification model.
confidence function,
Solving the surrogate optimization. The optimiza- tion problem (P3) is nonconvex. Nevertheless, it is apply gradient-based meth- ods. We solve for g and t simultancously using Adam [23]. Training details, including hyperparameters, are deferred to the Appendix.
differentiable and
Figure 4: Our choice of g function.
3.3 TBAL procedure with Colander
We take the workflow of TBAL and plugin our method Colander learn the new confidence function and threshold. We discuss the updated workflow below and place the detailed Algorithms and in the Appendix B due space constraints.
1. Initialization. First, select ns points randomly from X₂ and obtain human labels for them to create initial training data ((1) train- This is written as RANDOMQUERY(XX) (Xu. n,) in Algorithm 1. The procedure RANDOMQUERY(X.. nta)) selects n, points randomly from X,, and obtains human labels
for them to create ((1) train The steps of this procedure are detailed in Algorithm 4 in the Appendix B.
2. Train classification model. After obtaining human-labeled training data ((()) Drain for the current round i, the procedure TRAINMODEL D( trains model from model class H on the training data (1)) Any training procedure can used here. We use methods listed in Section ?? for model
training. This step outputs model hn trained on D ((i) train Note, this model h, may not be highly accurate due to several factors such as the small amount of training data, the choice of H. the training algorithm, and its hyperparameters. Indeed, TBAL does not expect the model to have high accuracy but it aims to identify and auto-label points where the model's accuracy is at least 1 - ca-
3. Learn new confidence function using Colander. The model h, obtained in the previous step also produces softmax scores that can be used for auto-labeling. However, as saw earlier in Section 2, using these scores may lead to poor auto-labeling performance. Thus, we plug in our procedure Colander to learn scores designed to auto-label many points possible with the current model hi while respecting the error constraint. We first randomly splits the validation data ( val into ( and ( using procedure RANDOMSPLIT(D val: v). The part ( has a fraction v of the points from (()). Then we consider problem P3 with fi; and ((6) We solve it obtain the post-hoc confidence function ÿ. which is expected to provide the most appropriate scores for TBAL.
We get thresholds L as output from Colander, and tempting to use these along with 9 for luto-labeling. However, these thresholds may violate the auto-labeling error constraint as they

---

are obtained by solving the relaxed optimization problem. Thus. it is crucial to estimate reliable thresholds &, from the held-out data D to ensure the auto-labeling error constraint not violated.
4. Threshold estimation. The scores from the new confidence function 9; on D are used to estimate auto-labeling thresholds in Algorithm 2. This procedure finds thresholds for each class separately. It first splits the points in D according to the ground truth class into subsets D(h,"). Then, for each class y, it finds the auto-labeling threshold tly] by selecting the minimum threshold t such that the estimate of auto-labeling error for class y. 6y (āt, tlfi,, Dth ((t))) plus a confidence interval [(6y(á, (/s,, D th
(1.21) )) estimated on points in (8:2) having scores above i, is at most the given error th
tolerance Co. Here {(z) = Ci /=(11 - 2) for E [0, 1] and C1 2 is a hyperparameter.
5. Auto-labeling. This is a simple step. We compute the scores on the remaining unlabeled data
x using the function Å and any point E x having score above {[ğ] is assigned auto-label V A, (x), and the points that did not meet this criterion remain unlabeled.
6. Remove auto-labeled points. The points that got auto-labeled in the previous steps are removed from the unlabeled pool. To make the validation data consistent with this unlabeled pool for the next round, the points in the validation data that fall into the auto-labeling region are also removed.
7. Get more human-labeled data. Lastly, it calls the procedure ACTIVEQUERY (h,, x10 ma)) to select ne points from the remaining unlabeled pool using an active learning strategy. This newly acquired human-labeled data is added to the training data (19) in Algorithm 3 Appendix B. Note, that the TBAL procedure flexible to work with any choice of active querying strategy. We pick a simple strategy based on random sampling from the points where the classifier is most uncertain. To avoid confounding in TBAL with other scores we use the softmax scores from the classifier to determine uncertainty here.
irain: The details of the querying strategy
The procedure then moves to step 2 and runs the loop until there are more unlabeled points left or it has queried the stipulated number of human labels N₁.
4 Empirical Evaluation
We validate the following claims through extensive empirical evaluation,
C1. Colander learns better confidence functions for auto-labeling compared to standard training and common post-hoc methods that seek to mitigate the overconfidence problem. Using it in TBAL can boost the coverage significantly while keeping the auto-labeling error low.
C2. Colander is independent of any particular train-time method and thus should help improve the performance when coupled with different train-time methods.
4.1 Baselines
We examine several train-time and post-hoc methods that improve confidence functions from the calibration and ordinal ranking perspectives. Details of these methods are in the Appendix C.7.
Train-time methods. We use the following methods for training the model h. Vanilla neural networks are trained with the cross- cross-entropy loss using stochastic gradient descent (SGD) [1, 4, 12]. Squentropy [19] adds the average square loss over the incorrect classes to the gross-entropy loss to improve the calibration and accuracy of the model. Correciness Ranking Loss (CRL) [35] aligns the confidence scores of the model with the ordinal rankings criterion via regularization. FMFP [64] aligns confidence scores with the ordinal rankings criterion by using sharpness-awar minimization (SAM) [7] in licu of SGD.
Post-hoc methods. We use the following methods for learning (or updating) the confidence function ÿ after learning h. Temperature scaling [12] is a variant of Platt scaling [41]. It rescales the logits by a learnable scalar parameter. Top-Label Eestogram-Binniine
[14] builds on the histogram-binning
method [62] and focuses on calibrating the scores of the predicted label assigned to unlabeled points. Scaling-Bimiing [28] applies temperature scaling and then bins the confidence function values. Dirichlet Calibration [26] models the distribution of predicted probability vectors separately
7

---

Dataset
Model
N
Nu
K
Nr
Ne
Nuvp
Modality
Preprocess
Dimension
MNIST
LeNet-5
70K
500
500
500
Image
None
CIFAR-10
CNN
50K
10k
Image
None
32
Tiny- -magenet
ML.
110k
200
10k
Image
CL.IP
512
20 Newsgroup
MLP
11.3k
9k
2k
Text
Elbb
1,024
Table I: Details of the dataset and model we used to evaluate the performance our method and other calibration methods. For the Tiny-Imagenet and 20 Newsgroup datasets, use CLIP and FlagEmbedding, respectively, to obtain the embeddings of these datasets and conduct auto-labeling on the embedding space. For Iny-Imagenet, neurons on each layer as model ft; for 20 Newsgroup, we use a 3-layer perceptron with 1,000, 500, 30 neurons on each layer model
3-layer perceptron with 1,000, 500, 300
instances of each class and assumes Dirichlet class conditional distributions. Adaptive Temperature Scaling [21] builds on top of temperature scaling and considers that different samples contribute to the calibration error differently. Each train-time method is piped with a post-hoc method, yielding total x 6 = methods.
4.2 Datasets and models
We evaluate the performance of auto-labeling on four datasets. Each is paired with model for auto-labeling: MNIST [30] is hand-written digits dataset. We use the LeNet [31] for auto-labeling. CIFAR-1O [24] is an image dataset with 10 classes. We use a CNN with approximately 5.8M parameters [20] for auto-labeling. Tmyg-mmageNer [29] is an image dataset comprising 100K images across 200 classes. We use CLIP [43] to derive embeddings for the images in the dataset and use an MLP model. 20 Newsgroups [34] is a natural language dataset comprising around 18K news posts across 20 topics. We use the FlagEmbedding [58] to obtain text embeddings and use MLP model.
4.3 Hyperparameter search and evaluation
The complexity of TBAL workflow and lack of labeled data make hyperparameter search and evaluation challenging. Similar challenges have been observed in active learning [32]. We discuss our practical approach and defer the details to Appendix C.10 and code?.
Hyperparameter search. We run only the first round of TBAL with each method using a hyper- parameter combination times and measure the mean auto-labeling error and mean coverage on Dhyp, which represents a small part of the held-out human-labeled data. We pick the combination that yields the lowest average auto-labeling error while maximizing the coverage. We first find the best hyperparameters for each train-time method, fix those, and then search the hyperparameters for the post-hoe methods. Note that the best hyperparameter for a post-hoc method depends on the training-ting time method that it pipes to. The hyperparameter search spaces are in the Appendix C; and the selected values used for each setting are the supplementary material.
Performance evaluation. After fixing the apper-parameters we run TBAL with each combination of train-time and post-hoc method full X,, of size N, with fixed budget of N. labeled training samples and N, validation samples. The details of these values for each dataset are in Table in Appendix C. Here, know the ground truth labels for the points in X,,, so measure the auto-labeling етrоr and coverage as defined in (1) and report them in Table 2.
4.4 Results and discussion
Our findings, shown in Table 2, are:
Cl: Co Colander improves TBAL performance. Our approach aims optimize the confidence function to maximize coverage while minimizing emros.. When applied to TBAL, we expect it to yield substantial coverage enhancement and eтror reduction compared to vanilla training and softmax scores. Indeed, the results in Table corresponding to the vanilla training match our expectations. We see across all data settings, our method achieves significantly higher coverage while keeping auto- labeling error below the tolerance level of 5%. The improvements are even more pronounced when
chttps://gitte http:///gttaaaaaaaaaaa/////////e
Neur IPS-24

---

MNIST
CIFAR-10
20 Newsgroups
Tiny-ImageNee
Train-time
Post-hoc
Err (1)
Cov (t)
Err (1)
Cov (?)
Err (1)
Cov (1)
Err (
Cov (t)
Softmax
4.1
85.0.25
4.8.0.2
14.0.2.1
6.0,0.6
48.2a1.6
11.1:03
32.6.0.5
TS
7.8
94.2+0.5
7.3
23.2:0.7
9.720.6
60.7=2.3
16.3:0,5
37.4:15
Vanilla
Dirichlet
7.9.0.7
93.2
7.7.0.5
22.4.1.2
9.4.0.9
1..4
17.1:04
33.3.20
6.7
92.6+1.5
6.1
18.6=1.1
8. 8.1
18.1
15.7:0.
35.4:12
Top-HB
7.4
93.1+3.6
6.0=0.7
15.6=1.
9.2s1.0
59.0=2.0
16.6.0.5
37.6:22
AdaTS
7.5 0.9
92.8.20
8.6
16.9.
9.6
61.8
15.9
36.7:19
Ours
4.2
95.61114
3.0
78.5
2.5
80.610.7
1.4:2.1
59.2.0.8
Softmax
4.7.0.4
86.0.4.5
5.2
15.9.0.8
5.8.0.5
48.3.0.3
10.4:04
32,5:06
TS
8.0
94.8=0.8
6.8
20,3=1.1
9,5+1.
61.7a166
15.8:0.6
37.4:17
CRL
Dirichlet
8.6:0.6
93.1
7.7
20.9
8.7.0.9
18.0.1.4
16.3.0.4
33.1a19
7.4.0.8
93.1+2.7
5.9.0.9
17.9=1.5
8.9=1.1
57.9=3.9
15.0.0.4
35.5
Top-HB 7.7.0.8
1.5
4.4
12.3:04
8.8
58.8
16.5:0.
38.9
AdaTS
7.8=0.7
94.3+1.2
8,8=0.4
17.1=1.
9.1
60.8=1.9
16.2=0.4
38.9:12
Ours
4,5
95.6+1.3
2.2=0.6
77.9=0.2
1,8
81.3
2.8+2.1
61.2+1.4
Softmax
4,8.0.8
84.2+111
4.9.0.1
15.6=1.7
5,4+0.7
15.4
10.5+03
32.4:14
TS
8.0.0.6
95.3s1.6
6.5
21.0=1.5
9.5
57.7=2.2
16.2.1.1
37.7.18
FMFP
Dirichlet
8.2
94.0,2.2
6.9.0.4
21.7,1.2
8.9.1.0
56.6.2.4
17.4:08
33.0-18
7.2=1.1
93.1+23
6.1
19.5=100
8.640.1
55.8=1.3
15.5+0.6
36.
Top-IIB
7.1
93.3+4.9
5.2+0.5
14.2+24
9. 9.0+0.7
57.9=2.4
16.2:0.4
37.4:1.1
AdaTS
7.6=0.4
94.1=1.0
7.2
27.5=1.5
8.7.0.9
56.7=2.7
16.3=0.6
37.6:1.7
Ours
4.6.0.8
95.7.10.2
3.0
77.410.2
2.5
80.8
1.8
60.811.4
Softmax
3.721.0
88.2439
5.2.0.5
21.2=1.8
4.6.0.4
$2.0a1.2
7.8:03
36,2:08
TS
6.2
95.6.09
6.9.0.6
28.2:2.5
8.3
66.6:1.4
13.3.0.1
4..9.
Squentropy
Dirichlet
6,5
95.940.8
7,3,0,3
29.4=1.1
7,840.6
64.0=1.3
14.1
42.5+07
6.0.0.8
95.3+1.2
6.2.0.4
23.8
7.8.0.7
63.0=2.9
13.0.0.5
45,2:20
Top-HB
5.3:0.4
96.4:09
4.3.0.5
15.8
8.2.08
66.5.2.2
13.7.0.1
45.9:14
AdaTS
6.6=1.0
95.7=1.0
7.6=0.3
22.6=1.2
7.4s0.6
64.7=2.6
14.0=0.3
1+0.
Ours
4. 4..8
97.2.0.5
2.3
79.0.0.3
3.3.0.8
82.920.4
0.6.0.2
66.5
Table 2: In every round the error was enforced to be below 5%; 'TS' stands for Temperature Scaling, 'SB' stands for Scaling Binning, 'Top-HB' stands for Top-Label Histogram Binning. 'AdaTS' stands for Adaptive Temperature Scaling. The column Err stands for auto-labeling error and Cov stands for coverage. Each cell value is mean + std. deviation on 5 repeated runs with different random seeds.
the datasets are more complex than MNIST. Also consistent with our expectation and observations in Figure 2(b), the post-hoc calibration methods improve the coverage over using softmax scores but at the cost of slightly higher error. While they are reasonable choices to apply in the TBAL pipeline, they fall short of maximally improving TBAL performance due to the misalignment of goals.
C2: Colander is compatible with and improves over other train-time methods. Our method is compatible with various choices of train-time methods, and if train-time method (Squentropy here) provides a better model relative to another train-time method (e.g.. Vanilla), then our method exploits this gain and pushes the performance even further. Across different train-time methods, we do not see significant differences in the performance, except for Squentropy. Using Squentropy with softmax improves the coverage by as high as 6-7% while dropping the auto-labeling error in contrast to using softmax scores obtained with other train-time methods for the ImageNet setting. This is unexpected: Squentropy adds the average square loss over the incorrect classes as a regularizer, and has offered better accuracy and calibration compared to training with cross-entropy loss.
Train-time methods designed for ordinal ranking objective perform poorly auto-labeling. CRL and FMFP are state-of-the- methods designed to produce scores aligned with the ordinal ranking criteria. Ideally, if the scores satisfy this criterion, TBAL's performance would improve. However, we do not any significant difference from the Vanilla method, Similar to the other basclines, their

---

evaluation is focused on models trained on large amounts of data. But, in TBAL, we have less data for training. The training error goes to zero after some rounds, and no information is left for the CRL loss to distinguish between correct and incorrect predictions (i.e., count SGD mistakes). On the other hand, FMFP is based on a hypothesis that training models using Sharpness Aware Minimizer (SAM) could lead to scores satisfying the ordinal ranking criteria. However, this phenomenon is still not well understood, especially in settings like ours with limited training data.
Related Work
Data labeling. We briefly discuss prominent methods for labeling. Crowdsourcing [45, 50] uses a crowd of non-experts complete of labeling tasks. Works in this domain focus on denoising the obtained information, modeling label errors, and designing effective labeling tasks [11, 22, 33, 53, 52, 54, 5]. Weak supervision (WS), in contrast, emphasizes labeling through multiple inexpensive but noisy sources, not necessarily human [44, 48, 55, 18, 49, 60, 633. Works such as [44, 8] concentrate on binary or multi-class labeling, while [48, 55] extend WS to structured prediction tasks.
Auto-labeling occupies an intermediate position between weak supervision and crowdsourcing in terms of human dependency. It aims to minimize costs to obtain human labels while generating high-quality labeled data using a specific model. [42] use a TBAL-like algorithm and explore the cost of training for auto-labeling with large-scale model classes. Recent work [56] theoretically analyzes the sample complexity of validation data required to guarantee the quality of auto-labeled data.
Overcontidence and calibration. The issue of overconfidence [51, 38, 16, 3] is detrimental in several applications (such as robustness to out-of-distribtiion points [59, 57D, including ours. Many solutions have emerged to mitigate the overconfidence and miscalibration problems. Gawlikowski et al. [9] provide a comprehensive survey on uncertainty quantification and calibration techniques for neural networks. Guo et al. [12] evaluated a variety of solutions ranging from the choice of network architecture, model capacity, weight decay regularization [25], histogram-
and isotonic regression [61, 62] and temperature scaling [41, 39] which they found to be the most promising solution. The solutions fall into two broad categories: train-time and post-hoc. Train-time solutions modify the loss function, include additional regularization terms, or use different training procedures [27, 37, 36, 19]. On the other hand, post-hoc methods such as top-label histogram binaiin [13], scaling binning [28], Dirichlet calibration [26] calibrate the scores directly learn a model that corrects miscalibrated confidence scores.
Beyond calibration. While calibration aims to match the confidence scores with probability of correctness, it is not the precise solution to the overconfidence problem in many applications, including our setting. The desirable criteria for scores for TBAL are closely related to the ordinal ranking criterion [17]. To get such scores, Corbière et al. [6] add a module in the net for failure prediction, Zhu et al. [64] switch to sharpness aware minimization [7] to learn the model; CRL [35] regularizes the loss.
6 Conclusion
We studied issues with confidence scoring functions used in threshold-based auto-labeling (TBAL). We showed that the commonly used confidence functions and calibration methods can often be bottleneck, leading to poor performance. We proposed Col to learn confidence functions that are aligned with the TBAL objective. We evaluated our method extensively against common baselines on several real-world datasets and found that it improves the performance of TBAL significantly in comparison the several common choices of confidence function. Our method is compatible with several choices of methods used for training the classifier in TBAL and using it in conjunction with them improves TBAL performance further. A limitation of Colander is that, similar to other post-hoe methods it also requires validation data to learn the confidence function. Reducing (or eliminating) this dependence on validation data could be an interesting future work.
7 Acknowledgments
This work was partly supported by funding from the American Family Data Science Institute. We thank Heguang Lin, Changho Shin, Dyah Adila, Tzu-Heng Huang, John Cooper, Aniket Rege, Daiwei Chen and Albert Ge for their valuable inputs. We thank the anonymous reviewers for their valuable comments and constructive feedback our work.
10

---

References
[1] S.-i. Amari. Backpropagation and stochastic gradient descent method. Neurocomputing, 5(4-5),
1993.
[2] M.-R. Amini, V. Feofanov, L. Pauletto, L. Hadjadj, E. Devijver, and Y. Maximov. Self-training:
A survey, 2023.
[3] Y. Bai, S. Mci, H. Wang, and C. Xiong. Don't just blame over parametrizationn for over- confidence: Theoretical analysis of calibration in binary classification. In Proceedings of the 38th International Conference Machine Learning, volume 139, pages 566-576, 2021.
[4] L. Bottou. Stochastic Gradient Descent Tricks. Springer Berlin Heidelberg, 2012. ISBN
978-3-642-35289-8
[5] Y. Chen, R. K. Vinayak, and B. Hassibi. Crowdsoureed clustering via active querying: Practical algorithm with theoretical guarantees. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 11, pages 27-37, 2023.
[6] C. Corbière, N. THOME, A. Bar-Hen, M. Cord, and P. Pérez. Addressing failure prediction by learning model confidence. In Advances in Neural Information Processing Systems 32, pages 2002-2913.22 2019.
[7] Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
[8] D. Y. Fu, M. F. Chen, F. Sala, S. M. Hooper, K. Fatahalian, and C. Ré. Fast and three-rious: Speeding up weak supervision with triplet methods. Proceedings of the 37th International Conference on Machine Learning (ICML 2020), 2020.
[9] J. Gawlikowski, C. R. N. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. Kruspe, R. Triebel,
P. Jung, R. Roscher, et al. A survey of uncertainty in deep neural networks. arXiv preprini arXiv:2107.03342 2021.
[10] Y. Geifiman and R. Elniv..v. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems, volume 30, 2017.
[11] R. G. Gomes, P. Welinder, A. Krause, and P. Perona. Crowdclustering. In Advances in Neural Information Processing Systems 24, pages 558-566. 2011.
[12] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017.
[13] C. Gupta and A. Ramdas. Distribution- free calibration guarantees for histogram binning without sample splitting, In International Conference on Machine Learning, pages 3942-3952. PMLR, 2021.
[14] C. Gupta and A. Ramdas. Top-label calibration and multiclass-to-binay reductions. In International Conference on Learning Representations, 2022.
[15] S. Hanson and L. Pratt. Comparing biases for minimal network construction with back- propagation. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 1. Morgan-Kann Kaufmann, 1988.
[16] M. Hein, M. Andriushchenko, and J. Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. 2018.
[17] D. Hendrycks and K. Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference Learning Representations, 2017.
[18] T.-H. Huang, C. Cao, Schoenberg, H. Vishwakarma, N. Roberts, and F. Sala. Scriptoriumws:
A code generation assistant for weak supervision. In ICLR Deep Learning for Code Workshop. 2023.
[19] I.. Hui, M. Belkin, and Wright. Cut your losses with squentropy. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[20] S. Hussain. Cifar 10- cnn using pytorch, Jul 2021. URL htttps://www.kgggle www.kaawwgg..aaaaaaaooooa......o code/
shadabhussain/ cit///ifaarrree 10-1000nnn-- con-ccnnnniiie using pytorch.
[21] T. Joy, F. Pinto, S.-N. Lim, P. H. Torr, and P. K. Dokania. Sample-dependent adaptive temper- ature scaling for improved calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11919-14922 2023,

---

[22] D. R. Karger, S. Oh, and D. Shah. Budget-optimal crowdsourcing using low-rank matrix approximations. In 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages
IEEE, 2011.
[23] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 1:Xiv:/412.6996 2014.
[24] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[25] A. Krogh and J. A. Hertz. A simple weight decay can improve generalization. In Proceedings of the 4th International Conference on Neural Information Processing Systems, NIPS' 91, page 950-957, San Francisco, CA, USA, 1991. Morgan Kaufmann Publishers Inc.
[26] M. Kull, M. Perello Nieto, M. Kängsepp. Silva Filho, H. Song, and P. Flach. Beyond temper- ature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In Advances in Neural Information Processing Systems, volume 32, 2019.
[27] A. Kumar, S. Sarawagi, and U. Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In Proceedings of the 35th International Conference on Machine Learning. PMLR, 2018.
[28] A. Kumar, P. Liang, and T. Ma. Verified uncertainty calibration. Advances in Neural Information Processing Systems, 32, 2019.
[29] Y. Le and X. Yang. Tiny imagenet visual recognition challenge. CS 231N. 7(7):3, 2015.
[30] Y. LeCun. The mnist database of handwritten digits. http://yann. lecun, com/exlb/mnist// 1998.
[31] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):2277 2222222278223 1998.
[32] D. Lowell, Z. C. Lipton, and B. C. Wallace. Practical obstacles to deploying active learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 21-30. Association for Computational Linguisties, 2019.
[33] A. Mazumdar and B. Saha. Clustering with noisy queries. In Advances in Neural Information Processing Systems, volume 30, 2017.
[34] T. Mitchell. Twenty Newsgroups. UCI Machine Learning Repository, 1999.
[35] J. Moon, J. Kim, Y. Shin, and S. Hwang. Confidence-auee learning for deep neural networks. In Proceedings of the 37th International Conference on Machine Learning, volume 119, 2020.
[36] J. Mukhoti, V. Kulharia, A. Sanyal, S. Golodetz, P. Tor,, and P. Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 33: 15288-15299, 2020.
[37] R. Müller, S. Kornblith, and G. E. Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
[38] A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 427-436, 2015.
[39] A. Niculescu-Mizil and R. Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning, 2005.
[40] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapea,, M. Brucher, M. Perrot, and Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research,
2011.
[41] J. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances large margin classifiers, 10(3):61-74, 1999.
[42] H. Qiu, K. Chintalapudi, and R. Govindan. MCAL: Minimum cost human-machine active labeling. In The Eleventh International Conference on Learning Representutions, 2023.
[43] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et Learning transferable visual models from natural language supervision. In International conference on machine learning, pages
PMLR, 2021.
12

---

[44] A. J. Ratner, M. D. Sa, S. Wu, D. Selsam, and C. Ré. Data programming: Creating large training sets, quickly. In Proceedings of the 29th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, 2016.
[45] V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. Journal of Machine Learning Research, 11(43):297-13322, 2010.
[46] B. Settles. Active learning literature survey. 2009.
[47] SGT. Aws sagemaker ground truth.
https://aws amazon. www.aa.coo//aageaaaaaa.aa.. con/sagemaker/ data-labeling/. 2022. Accessed: 2024-05-22.
[48] C. Shin, W. Li, H. Vishwakarma, N. C. Roberts, and F. Sala. Universalizing weak supervision.
In International Conference on Learning Representations, 2022.
[49] C. Shin, S. Cromp, D. Adila, and F. Sala. Mitigating source bias for fairer weak supervision. In Advances in Neural Information Processing Systems, volume 36, pages
2023. [50] A. Sorokin and D. Forsyth. Utility data annotation with amazon mechanical turk. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pages
1-8, 2008. doi: 10. 111.////R...000845050000
[51] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intrigu- ing properties of neural networks. In International Conference on Learning Representations, 2014.
[52] R. K. Vinayak and B. Hassibi. Crowdsourced clustering: Querying edges vs triangles. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, 2016.
[53] R. K. Vinayak, S. Oymak, and B. Hassibi. Graph clustering with missing data: Convex algorithms and analysis. In Advances in Neural Information Processing Systems, 2014.
[54] R. K. Vinayak, T. Zrnic, and Hassibi. Tensor-based crowdsourced clustering via triangle queries. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2322-2326 IEEE, 2017.
[55] H. Vishwakarma and F. Sala. Lifting weak supervision to structured prediction. In Advances in Neural Information Processing Systems, volume 35, 2022.
[56] H. Vishwakarma, H. Lin, F. Sala, and R. K. Vinayak. Promises and pitfalls of threshold-based auto-labeling. In Seirth-seventh Conference on Neural Information Processing Systems, 2023.
[57] H. Vishwakarma, H. Lin, and R. K. Vinayak. Taming false positives in dit-o--disti detection with human feedback. In International Conference on Artificial Intelligence and Statistics. PMLR, 2024.
[58] S. Xiao, Z. Liu, P. Zhang, and N. Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
[59] Yang, Wang, D. Zou, Z. Zhou, K. Ding, W. Peng, H. Wang, G. Chen, Li, Sun, X. Du,
K. Zhou, W. Zhang, D. Hendrycks, Y. Li, and Z. Liu. Openood: Benchmarking generalized duttrf-distribution detection, 2022.
[60] Y. Yu, S. Zuo, H. Jiang, W. Ren, T. Zhao, and C. Zhang. Fine-tuning pre-trained language model with weak supervision: A contrastive-rogulare
self-training approach. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1063-1077, 2021.
[61] B. Zadrozny and Elkan. Learning and making decisions when costs and probabilities are both unknown. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pages 204-213, 2001.
[62] B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694-699, 2002.
[63] D. Zhu, X. Shen, M. Mosbach, A. Stephan, and D. Klakow. Weaker than you think: A critical look at weakly supervised learning. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14229-14253, 2023.
[64] Zhu, Z. Cheng, X.-Y Zhang and C.-I.. Liu. Rethinking confidence calibration for failure prediction. In European Conference Computer Vision, pages 518-536. Springer, 2022.
13

---

Supplementary Material Organization
The supplementary material is organized as follows. We provide deferred details of the background and motivation section in Appendix A of the method in Appendix B. Then, in Appendix C, we provide additional experimental results and details of the experiment protocol and hyperparameters used for the experiments. Our code with instructions to run, is uploaded along with the paper.
A Appendix to the Background and Motivation Section
A.1 Detailed comparison with active learning and self-training
To illustrate the differences between TBAL and the combination of active learning (AL) and self- training for the task of data labeling, we run an experiment on the concentric circles data setting as used in [56]. The details are as follows:
Data setting. We generate two concentric circles with points in the outer circle belonging one class and the inner circle belonging to the other class. The total number of points generated is 10,000 of which we use 2000 for validation.
Methods. We run TBAL, AL+Self-
Training, and AL+SSef--rriining- using logistic regression. The com-
Auto- Labolaggiinggggo. Enoor (N)
100
bination of AL+Self-Training means, in each iteration, the algorithm
Covernge (56)
80
20
60
queries human-labeled data points and
pscudo-labels the points in the un-
20-
labeled data using self-training and adds both the human-labeled and pseudo-labeled points in the training pool. With this procedure, AL+Self- Training first learns the best classifier (hal-st) with the given budget max- imum training points (N₂) that can be queried from humans. Then auto-labels all the remaining unlabeled points with this classifier's pre- dictions. For A --afff-TraiinngSSC we do selective auto-labeling using hat-st. i.e., only auto-label the points where the classifier will have error most <a- We use ca =1% here.
200
400
600
800
1000
200
400
600
800
1000
Max Training Points (N6)
Max Training Points (86)
AL Sell Training
AL + Self + SC
+ TBAL
Figure 5: Results of experiment on 2-concentric circles to show the differences between TBAL, AL and ST.
Results and discussion. The Figure shows auto-labeling error and coverage achieved by these methods when run with different choices of human-labeled data budget for training. First, that even with linear classifiers TBAL is able to auto-label a huge chunk the data (high coverage) while maintaining auto-labeling error below the tolerance level of 1% On the other hand, methods like AL+Self- Training (+SC) that try to first learn the optimal classifier the given function class either have high auto-labeling error very low coverage. These results are also consistent with the observations in [56] on the comparison between TBAL and AL, AL+SC. While such findings least in some settings, advantages-for the TBAL approach compared to other techniques, we reiterate that our goal is to understand and improve the role of the confidence function within TBAL, rather than comparing TBAL. other techniques.
confirm the notion that there are differences-and
A.2 Details of the motivating experiment section 2
We run TBAL for a single round on the CIFAR-10 10 dataset with a SimpleCNN classification model with around 5.8M parameters [20]. We randomly sampled 4,000 points for training the classifier and randomly sampled 1,000 points as validation data. We train the model zero training error using minibatch SGD with learning rate le-3, weight decay le-3 [15, 25], momentum 0.9, and batch size 32. The trained model has validation accuracy around 55%, implying we could hope to get coverage around 55%. We run the luto-labeling procedure with an error tolerance of 5%.
14

---

B
Additional Details on the Method
B.1 Detailed algorithms
See Algorithms 1, 2 and 3.
B.2 Tightness of surrogates.
The surrogate lato-labeling error and coverage introduced to relax the optimization problem (P2) is indeed a good approximation of the actual auto-labeling error and coverage. To see this, we use
toy data setting of Uniform(0,, 1) with d-dimensional threshold classifier fi)(2)) 1 (x: 2 0). For any x; let true labels (a)) and consider the confidence function (x)(2) = Ioe - x Let
25 (x) and consider the points on the side where ý We plot actual and surrogate errors in Figure 6(a) and the surrogate and actual coverage in Figure 6(a).
for three choices of As expected, the gap between the surrogates and the actual functions diminishes increase the a.
0.25, 1.0
(- 0.125 & 10.0
01125, A 50.0
Anto Labeling
82
B.3 Active querying strategy.
We employ the margin-random query approach to select the next batch of training data. This method involves sorting points based on their margin (uncertainty) scores and selecting the top Cn,, points, from which n, points randomly chosen. This strategy provides straightforward and com- putationally efficient way balance the apploration-exploitaaai trade-off. It's important to acknowledge the ex- istence of alternative active-querying strategies; however, we adopt the margin-ranee mandomm approach as our stan- dard to maintain focus on evalu- ating various choices of confidence functions for auto-labeling. Note that while we use the new confidence
0.5
0.5 (pammmmeer of gt)
(parammter g1)
(panameter of
g3)
(a) Auto-labeling error and surrogate error at various a.
Acto Labeligg Congage
0,125,
0.125
10,0 Adaatinn
0.15,
50,0
0.5
Adbelle
0.8
0.5
06
0.6
1.0
0.0
0.5
1.0
0.0
0.5
1.0
(parameter g)
(parameter ol
[panameser
(b) Auto-labeling coverage and surrogate coverage various
Figure 6: Illustration of the tightness of surrogate error and coverage functions based on the choice of a.
scores computed using post-hoc methods for auto-labeling, we do not use these scores in active querying. Instead, we use the softmax scores from the model for this. We do this to avoid conflating study with study active querying strategies. We use for experiments.
15

---

B.4 Glossary
The notation summarized in Table 3 below.
Symbol
Definition
1 (E)
indicator function of event E. It is if E happens and 0 otherwise.
OC
feature space.
y
label space i.e. .. k.
H
hypothesis space (model class for the classifiers).
class of confidence functions.
number of classes.
x,y
element in . and y is its true label. unknown groundtruth labeling function. hypothesis (model) in H.
f
h
g
confidence function g -
Ca
auto-labeling error tolerance.
Xa
given pool of unlabeled data points.
X
unlabeled data left the beginning of ith round. indices of points that are auto-labeled. subset of points in X,, with indices in A, i.e. the of auto-labeled points.
A
(A)
200
ERM solution and auto-labeling thresholds respectively ith round. & dimensional vector of thresholds.
t
tly
yth entry i.e. the threshold for class y-
(x)[el
the confidence score for class y output by confidence function g data point number of unlabeled points, i.c. size of X,.
No.
N,
number of manually labeled points that can be used for training ft.
Na
Total auto-labeled points Dout.
fraction of Dval that can be used for training post-hoc calibrator.
yi
label assigned to the ith point by the algorithm. could be either yi or Ú..
y
groundtruth label for the ith point. predicted label for the ith point by classifier.
D
labeled data queried from oracle (human) in the ith round.
D (s)
training data to learn (0) in the ith round.
train
D
validation data the ith round.
val
D (
cal
calibration data in the ith round to learn post-hoc g.
D
part of validation data in the ith round to estimate threshold
D
part of x-(9) that got auto-labeled in the ith round.
auto
Dout.
Output labeled data, including auto-labeled and human labeled data.
8(g,t| (t)
population level auto-labeling error, see eq. (2).
p(g,t hi)
population level auto-labeling coverage, see eq. (2).
8(g,t|
estimated auto-labeling error, eq. (4).
estimated auto-labeling coverage, see eq. (3).
E(g, D)
surrogate estimated auto-labeling error, see eq. (6).
Øgg,t t h D)
surrogate estimated auto-labeling coverage, see eq. (5).
Table 3: Glossary variables and symbols used in this paper.
16

---

Algorithm Threshold-based Auto-Labeling (TBAL)
Input:
Unlabeled data Xu, labeled validation data Vaal. auto labeling error tolerance fas N₁ training data query budget, seed data size na. batch size for active query nts calibration data fraction v, space of thresholds 7', coverage lower bound po. label space y.
Output: Auto-labeled dataset Dout.
1: procedure TBAL(X Dval: Co. No, ngy nb. Po. No.T.U/ T
2: 1#8* Initialization. #*8/
3: D(1) RANDOMQUERY (X,,, n,).
Qquery-
Randomly select n, points and get human labels. >Remove these points from the unlabeled pool.
X + X \ fx: (x,8) E )
4:
5:
D t vall; (00) < 0.
D Validation data for the first round is full D
6:
Dout f DQuery: The (1) naj +
pAdd human-labeled data to the output Dout
7:
/*0* Run the auto-labeling loop. ***/
8:
1st Until no more unlabeled points are left or the budget for training data is exhausted. */
9:
while X
# 0 and ni (6)
S N2 do Dquery. >Include human -abeled points the training data. >Train a classification model.
D train
+ D(inl)
11:
h4
- (RA ( (.
12:
D D(
+ RANDOMSPLIT(
val v).
>Randomly split current validation data.
cal th
13:
Di /** Colander block, to learn the new confidence function 9:- **s/
14:
9., t
+ arg min (g, h, ((d) + XE(g, t Ri, (()) Colander.
> /*** Estimate auto-labeling thresholds using å: and ( See Algorithm 2. ***/
15:
f
+ -ESTTHRESHOOLDD h, ((6) fas Po, Po, T,Y)).
17:
>/ /*** Auto-label the points having scores above the thresholds. ***/
((i) + ((xxh((x)) x x(()) J.
18:
D
[(xx 0) D [x) 2 t.
19:
auto
20:
X!!
x!!") {x: (x, E ( R-Rmove auto-labeled points from unlabeled set.
( + ((xx (x)) (x. v) D D
21:
val
22:
D(++1) {(x, 0) E D( (x) M.
Remmov validation points from the
auto-labeling region.
jaa* Get the next batch of manually labeled data using an active querying strategy. **ay
Diitet <- ACTIVEQUERY( x-( na). ((++1) (((
24:
25:
(x: (x, v) D D
PRemove human-labelle data from the
unlabeled pool.
26:
Dout f- Dout U (t) in the output data.
auto U Queery- bAdd the auto-labeled and manually labeled points
The ((+1)
(6)
27:
n4
28:
i.
29:
end while
30:
return Dout-
31:
end procedure
17

---

Algorithm Estimate Auto-Labeling Threshold
Input:
Confidence function åi classifier fi. Part of validation data D for threshold estimation, auto labeling error tolerance ca. space thresholds T, coverage lower bound po. label space y/ Output:
Auto-labeling thresholds t,, where f, [y] is the threshold for class y-
1: procedure ESTTHRESHOLDOR, hi, (( Cor Po: T, y)
2: 1 Estimate thresholds for each class. #
3:
for y E y do (i.2)
4:
+ [(xxx v'') D y =y)
DGroup points class-wise.
5: 184* Only evaluate thresholds with est. coverage least Pu- **w/
6: Ty + f T: (gr, h, D(i po) U (oo).
-aa Estimate auto-labeling error each threshold. Pick the smallest threshold with the sum of estimated error and C1 times the std. deviation is below ea. C1 is set to 0.25 here. #*8/
8: tg/ t min Ty Eg, thh,, t(,,, + CiÊ(E, (āi, the, ((ia))) S ca). end for 10: return t,.
9:
11: end procedure
Algorithm 3 Active Querying Strategy to Acquire Human- Samples for Training
Input: Classifier he unlabeled data x("), batch size nd, constant C' 2 1.
Output: Diitel, a subset of x of size at most ny with human (groundtruth) labels.
procedure ACTIVEQUERY(Ú,, x!10, m1))
2: s!i)) Softmax scores from h; for all points x!
3: X' +- Top
T C n, points from x-(1) sorted in ascending order on the scores s())
4: Dquery +- RANDOMQUERY(X (X', n4).
5: return ((tt))
6: end procedure
Algorithm 4 Select Subset Points Randomly and Obtain Human Labels
Input: X, n.
Output: D, subset of X of size most with human (groundtruth) labels. procedure RANDOMQUERY(X,,n n)
2:
if then X| > n
3:
X" randomly select points from X.
4: else
5:
X" X. end
6:
7:
D - {(x, human haman_label(x)) X")
8:
return D.
9:
end procedure
18

---

Additional Experiments and Details
C.1 Experiments on No, N, and
We need to understand the effect of training data query budget i.e. N₁, the total validation data No. and the data that can used for calibrating the model i.e. the calibration data fraction v on auto-labeling objective. As varying these hyperparameters on each train-time method is expensive, we experimented with only Squentropy as was the best-performing method across settings for various datascts.
When we vary the budget for training data Nr, observe from Figure that our method does not require a lot of data to train the base model, i.e. achieving low auto-labeling error and high coverage with low budget. While other methods benefit from having more training data for auto-labeling objectives, it comes at the expense of reducing the available data for validation.
From Figure 8, observe that, while the coverage of method remains the across different Ne. it reduces for other methods. The cause of this phenomenon can be attributed to the fact that we borrowing the data from the training budget it limits the performance of the base model, which in turn limits the auto-labeling objective.
As increase the percentage of data that can be used to calibrate the model, i.e., V, we note from Figure 9 that other methods improve the coverage, which can be understood from the fact that when more data is available for calibrating the model, the model becomes better in terms of the labe-lanling objective. But it's interesting to note that even with a low calibration fraction, our method achieves superior coverage compared other methods. It is also important note that the auto-labeling error increases as we increase v. This is because when v increases, the number of data points used to estimate the threshold decreases, leading less granular and precise threshold.
Feature
Model
Error
Coverage
Pre-logits
Two Layer
4.6 + 0.3
82.8 0.5
Logits
Two Layer
3.2 1.3
82.8 + 0.3
Concat
Two Layer
3.3 0.8
82.9 + 0.4
Table 4: Auto-labeling error and coverage for the feature representations we could use for 20 Newsgroup. As we can see, the feature representation does not lead to significant difference in auto-labeling error and coverage.
Feature
Model
Error
Coverage
Pre-logits
Two Layer
2.1 0.5
79.0 0.2
Logits
Two Layer
3.1 : 0.4
76.5 + 0.9
Concat
Two Layer
0.5
79.0 0.3
Table 5: Auto-labeling error and coverage for the 3 feature representations could use for CIFAR10 SimpleCNN. As we can see, the feature representation does not lead to a significant difference in auto-labeling error and coverage.
C.2 Experiments on Colander input
Figure 4 illustrates that we could use logits (last layer's representations). pre-logits (second last layer's representations), or the concatenation of these two as the input to g. To help us decide which one we should use, we conduct hyperparameter search for input features on the CIFAR- 10 and 20 Newsgroup dataset using the Squentropy train-time method. Table 4 and present the auto- labeling emorr and coverage of using the 3 types of feature representations. As we can see, all feature representation leads similar auto-labeling error and coverage, and in some cases, it is better to include pre-logits as well. Thus, concatenated representation (Concat), for more flexibility.
19

---

C.3 Experiments on
We rum TBAL with five values of e, {0.01, 0.025, 0.05, 0.075, 0.1] and report the results in Table 6. As expected the auto-labeling error is high with larger values of and smaller with small Ca.
C,
(x)
(t)
(4)
(1)
Cov
(1)
Err
Cov (1)
Softmax
161
TS
8.19 0.8
19.44 1.66
8.19 1.88
19.44 a 1.16
7.26 a 0.29
23.15 1.07
9.24 0.75
22.49 0.24
11.63 1.511
25.79 1.00
Dirichlet
8.22
16.94 1.2
8.22 4.4
16,94 #.2
=.48
22.36
9.68 0.82
18.65 057
11.26 1.36
24.91 200
6.13
11.74
6.00
0.65
+
=0.52
Top-HB
5,76 a 8.42
9.89 0.5
5,76 0.42
9.89 1.035
5.95
0.7
15.58
1.02
7:45
0.8
13.84
0.78
8:71 1:33
17.9 0.56
Ours
1.3 8.18
78.33 9.76
1,32 0.21
78.75 104
2.96
0.2
78.48
0.17
4.3
1.033
78.94 0.42
6.29 0.5
78.97 0.46
Table eo variation. Dataset: CIFAR-10, Train-time method: Vanilla.
C.4 Single vs multi-round TBAL
We further demonstrate that the performance gains are due to the use of Colander, even if methods use mul-
80
Nuto Lablliing Accunicy
98
60
tiple rounds. To do so,
96
Coverage
we show the evolution of coverage and error over multiple rounds in Figure 16. The effects of using Colander are visible from the first round itself, and the following rounds im- prove performance further. We also single round (passive) variant of TBAL where we sample all the human-labeled points for training (N,) randomly at once, train classifier, do auto-labeling, and then stop. This setting avoids confounding due to multiple rounds. We observe that using Colandes yields significantly higher coverage in comparison the baselines (see Table 7). This reinforces the fact that the gains in the multi-round TBAL are directly due to Colander, while multiple rounds of data selection, training, and auto-labeling are superior this single round version.
98
20-
92-
20
11
Epoch
Epuch
Softmax
Ours
Dirichlet
75
SB
Top-1IB
Figure 16: Per-epoch metrics for all post-hoc methods for CIFARIO setting. (left) Auto-labeling accuracy (right) Coverage. Train-time method is vanilla.
C.5 Experiments on different architectures
In TBAL it is priori clear what model the practi- tioner should use. The overall system is flexible enough to work with any chosen model class. Our focus on evaluating the effect of various training time and post-hoc methods designed improve the confidence functions for any given model. To answer the query, we ran experi- ments with Resnet18 and ViT models in the CIFAR- 10 setting (see Table 8). As we expected there are variations in the results in the baselines due to model choices but our method maintains high performance irrespective of the classification model used. This due to its ability to learn confidence scores tailored for TBAL.
Post-hoc
Err (L)
Cov (t)
Softmax
2.7 + 0.54
11.06 1.46
TS
3.04 + 0.49
12.03 1.98
Dirichlet
2.98 + 0.32
11.22 = 2.1
SB
2.72 0.34
9.75
1 1.33
Top-HB Ours
1.83 1 0.61
5.50
1 1.08
2.02 + 0.28
49.62 0.69
Table 7: Results with single round of auto-labeling. Dataset and model: CIFAR-10 setting in the paper.
20

---

Post-hoc
Err (1)
Cov (f)
Post-hoc
Err
Cov (1)
Softmax
14.02
1.83
1.0.3
Softmax
4.48 1.233
33.24 1.14
19.32 2.51
2.54
0.33
6.38 0.47
39.14 1.96
Dirichlet
17.27 3.26
2.87
0.55
Dirichlet
6.30 +01
37.99 1.47
9.22
= 10.1
0.46
1: 0.51
SB
5.16 0.23
35.32 + 1.36
Top-HB
0.00
0.00
+ 0.000
Top-HB
4.46 0.40
29.66 + 0.74
Ours
0.32
75.56
+ 0.15
Ours
2.85 0.25
78.56 0.54
Table 8: Model variation. CIFAR-10 dataset with ViT (Left) and ResNet18 (Right), Train-time method Vanilla.
Method
Hyperparameter
Values
optimizer
SGD
learning rate
0.001, 0.01, 0.1
batch size
32, 256
Common
max epoch
50, 100
weight decay
0.001, 0.01, 0.1
momentum
0.9
rank target
softmax
CRL
rank weight
0.7, 0.8, 0.9
FMFP
optimizer
SAM
Table 9: Hyperparameters swept over for train-time methods. Those listed next to Common are the hyperparameters for the four train-time methods: Vanilla, CRL, FMFP, and Squentropy. Therefore, we do not list those again for each method. Note that for FMFP, used SAM optimizer instead of SGD. For each method, we swept through all possible combinations of the possible values for each hyperparameter. Underlined values only used on TinylmageNet since it is complicated dataset containing 200 classes.
C.6 Hyperparameters
The hyperparameters and their values swept are listed Table and 10 for train-time and post-hoc methods, respectively.
C.7 Train-time and post-hoc methods
C.7.1 Train-time methods
1. Vanilla: Neural networks are commonly trained by minimizing the cross entropy loss using stochastic gradient descent (SGD) with momentum [1, 4]. We refer to this the Vanilla training method. We also include weight decay to mitigate the overconfidence issue associated with this method [12].
2. Squentropy [19]: This method adds the average square loss the incorrect classes to the cross-entropy loss. This simple modification to the Vanilla method leads to the end model with better test accuracy and calibration.
3. Correctness Ranking Loss (CRL) [35]: This method includes a term in the loss function of the vanilla training method so that the contidence scores of the model are aligned with the ordinal rankings criterion [17, 6]. The confidence functions satisfying this criterion produce high scores on points where the probability of correctness is high and low scores on points with low probabilities of being correct.
4. FMFP [64] aims to align confidence scores with the ordinal rankings criterion. It uses Sharpness Aware Minimizer (SAM) [7] train the model, with the expectation that the flat minima would benefit the ordinal rankings objective of the confidence function.
21

---

C.7.2 Post-hoc methods
1. Temperature scaling [12]: This is variant of Platt scaling [12], classic and one of the easiest parametric methods for post-hoc calibration. It rescales the logits by a learnable scalar parameter and has been shown work well for neural networks.
2. Top-Label cesstorram-iiniinn
[14]: Since TBAL assigns the top labels (predicted labels) to the selected unlabeled points, it is appealing to only calibrate the scores of the predicted label. Building upon a rich line of histogram-binning methods -on-paaaattrii) for post-hoc calibration [62]. this method focuses calibrating the scores of predicted labels.
3. Scaling-Binning [28]: This method combines parametric and non-parametric methods, first applies temperature scaling and then bins the confidence function values to ensure calibration.
4. Dirichlet Calibration [26]: This method models the distribution of predicted probability vectors separately on instances of each class and assumes the class conditional distributions are Dirichlet distributions with different parameters. It uses linear parameterization for the distributions, which allows easy implementation in neural networks as additional layers and softmax output.
Note: For binning methods, uniform mass binning [62] has been a better choice over uniform width binning. Hence, we use uniform mass binning as well.
C.8 Compute resources and time
Our experiments were conducted on machines equipped with the NVIDIA RTX A6000 and NVIDIA GeForce RTX 4090 GPUs. The wall clock time of our method is similar to other post-hoc methods. For instance, single run on the CIFAR-10 setting on NVIDIA RTX A6000 takes around 1.5 hours with post-hoc methods and roughly 1 hour without post-hoc methods. The additional time taken by our method over the baselines not doing any post-hoe calibration traded-off the by the quality and quantity of the auto-labeled data it outputs. We leave a thorough benchmarking of wall clock time and its optimization for future work.
C.9 Detailed dataset and model
1. The MNIST dataset [30] consists of 28 x 28 grayscale images of trin-written digits across 10 classes. It was used alongside the LeNet5 [31], convolutional neural network, for auto-labeling.
2. The CIFAR- 10 dataset [24] contains 3 32 x 32 color images across 10 classes. We utilized its raw pixel matrix in conjunction with SimpleCNN [20]. a convolutional neural network with approximately 5.8M parameters, for auto-labeling.
[29] is a color image dataset that consists of 100K images across 200 classes. 3. Instead of using the x 64 x 64 raw pixel matrices input, utilized CLIP [43] to derive embeddings within the R 512 vector space. We used a 3-layer perceptron (1.000-500-300) as the auto-labeling model.
4. Newsgroups [34, 40] is natural language dataset comprising around 18,000 news posts across 20 topics. We used the FlagEmbedding [58] to map the textual data into R R1021 embed- dings. We used a 3-layer perceptron (1, (1,000-500-30) as the auto-labeling model.
C.10 Detailed experiments protocol
We predefined TBAL hyperparameters for each dataset-model pair and the hyperparameters we will sweep for each train-time and post-hoe method in Table 9 and Table 10 respectively. For dataset- model pair, initially, we perform a hyperparameter search for the train-time method. Subsequently, we optimize the hyperparameters for post-hoc methods while keeping the train-time method fixed with the previously found optimum hyperparameter for that dataset-model pair.
We fix the hyperparameters for the train-time method while searching hyperparameters for the post- hoc method alleviate computational budget throttle. We effectively reduce the search space to the sum of the cardinalities of unique hyper-parameter combinations across the two methods instead of larger multiplicative product. Furthermore, due to the independent nature of these hyper-parameter combinations, TBAL runs can be highly parallelized to expedite the search process.
22

---

Since TBAL operates iteratively to acquire human labels for model training, selecting hyper- parameters at each round of TBAL could quickly become intractable and lose practical significance. To better align with its practical usage, we only conducted a hyperparameter search for the initial TBAL round. The specific set of hyperparameters used for the search are reported in Table 10.
After completing the hyperparameter search for train-time and post-hoc methods, the determined hyperparameter combinations are subjected to a full evaluation across all iterations of TBAL. At the end of each iteration, the auto-labeled points are evaluated against their ground truth labels to determine their auto-labeling error. These points are then added to the auto-labeled set, where their ratio to the total amount of unlabeled data determines the coverage. This iterative process continues until all unlabeled data are exhaustively labeled by either the oracle or through auto-labeling in the final iteration. The auto-labeling error and coverage at the final iteration of TBAL are then recorded.
Since TBAL incorporates randomized components detailed in Algorithm 1, times, each with a unique random seed while maintaining the same hyperparameter combination. We then recorded the results from the final iteration of these runs and calculated the mean and standard deviation of both auto-labeling error and coverage. These figures are reported in Table 2.
the algorithm
A limitation of the grid search approach in hyper-parameter optimization becomes apparent when our predefined typer-parameter choices result in sub-optimal coverage and auto-labeling errors. Using these sub sub-optimal Syper-paramette prompting the need for repetitive searches to find more effective syper-parametent When encounter- ing such scenarios, TBAL users should explore additional hyper-parameter options until satisfactory performance is achieved in the initial round. However, we opted for a more straightforward approach to hyper-parameter selection, mindful of the computational demands of repeatedly optimizing mul- tiple hyper- parameters across different methods. In scenarios expressed conditionally, we retained the top-1 nlyper-parmmeter combination for any given method if it achieved the highest coverage while adhering to the specified error margin (e.). If no hyper-parameter combinations yielded an auto-labeling error most equal to the error margin (ea). then chose the hyper-parameter combi- nation with the lowest auto-labeling error, regardless of its coverage. In the case of ties, we resolved them through random selection. This process results in obtaining singular values for each choice of hyper-paramen parameter after completing each method's 's hyper-parameter search.
adversely affect the multi-round iterative process in TBAL,
23

---

D
Broader Impact
This paper contributes to the advancement of the practice of creating labeled datasets in machine learning. While our work has various possible societal implications, we do not identify any specific concerns that require special attention in this context.
E
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims are backed by our novel technique in Section C and thorough empirical evaluation in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contri- butions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss them briefly.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tent system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important

---

role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: It is empirical paper, it does not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the necessary details are provided in Section 4 and in the Appendix C. We have also uploaded the code along with the submission.
Guidelines:
The answer NA means that the paper does include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice. or if the contribution is a specific model and empirical evaluation, it may be necessary either make possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
example, if the contribution
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
25

---

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We use publicly available datasets and uploaded the code as supplementary
material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: for more details.
/ //tips.cc/publleeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparame- ters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: These details are provided in the Section 4 and Appendix C.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We run each setting with multiple random seeds and report the mean, standard deviations of the evaluation metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
26

---

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative error rates). eтror bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer:
[Yes]
Justification: Provided in the Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn' 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethies https::/neurt// wctpublic/Ethiicccooooco htwp:/eeeiiccuiieeeeeeleeelell lins?
Answer: [Yes]
Justification:
We have followed the NeurlPS Code of Ethics to the best of our knowledge.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper has a brief discussion on the broader impacts.
Guidelines:
The answer NA means that there is societal impact of the work performed.
27

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point out. For example, is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g.. gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper does not release such data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have appropriately credited them along with citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code is well documented along with instructions to run.
Guidelines:
The answer NA means that the paper does not release assets. Researchers should communicate the details of the dataset/codemmodel part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did not use crowdsourcing or human subjects the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements your country or institution) were obtained?
Answer: [NA]
Justification: We did use crowdsoureing or human subjects the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29

---

10
80
Auto Labeling Error
Coverage
40
20
5000
10000
15000
20000
25000
5000
10000
15000
20000
25000
Softmax
Ours
Dirichlet
Top-HB
SB
IS
Figure 7: Autolabeling error and coverage of different post-hoc methods on CIFAR-10 for various N
80
Auto Labeling Error
Coverage
10.
40
20
1000
2500
5000
7500
10000
1000
2300
5000
7500
10000
N.
Softmax
Ours
Dirichlet
Top-HB
SB
TS
Figure 8: Autolabeling error and coverage of different post-hoc methods on CIFAR- 10 for various
20
Auto Labeling Error
15
60
Coverage
10
40
1.10
2025
0.50
0.75
0,90
0.95
0.05
25
0.50
0.75
0.90
0.95
Ours
Dirichlet
Top-HB
SB
TS
Figure 9; Autolabeling error and coverage of different post-hoc methods on CIFAR-10 for various
30

---

80
Auto Labeling Emror
60
Coverage
40
&
Percentage of data used for training (,,N,)
Percentage of data used for training (N/NN,)
Softmax
Ours
Dirichlet
Top-HB
SB
TS
Figure 10: Auto-labeling and coverage for different post-hoc methods on CIFAR- 10 while
vary N₁. N₂, 40, 000 is the size of the given unlabeled pool.
70
Auto Labeling labeling& Emor
Coverage
40
30-
20
10
D
2
Percentage of data used training (N,N,,
Percentage data used for training (N/N,,
Softmax
Ours
Dirichlet
Top-HB
SB
TS
Figure 11: Auto-labeling error and coverage for different post-hoc methods on Tiny-ImageNe while
we vary N, N₂ 90, 000 is the size of the given unlabeled pool.
12
80
Auto Labeling Ervor
10
Coverage
40
2
2
6
2
2
Percentage of data used for training (N,NN.) N.)
Pencentage of data used for training (N/N.)
Softmax
Ours
Dirichlet
Top-HB
SB
TS
Figure 12: Auto-labeling error and coverage for different post-hoc methods on Newsgroups while vary N₁. Nº 9, 052 is the size of the given unlabeled pool.
31

---

17.5
15.0
Auto Labeling Ertor
60
12.5
Coverage
10.0
7.5
2.5
94
100
12
31
22
94
100
Percentage of validation data used (N,/ Novua)
Percentage of validation data used Noma)
Softmax
Ours
Dirichlet
Top-HB
SB
TS
Figure 13: Auto-labeling and coverage for different post-hoc methods CIFAR- 10 while
vary N,. N = 8, 000 is the maximum number points available for validation.
30
60
Labeling Emor
25
20
Coverage
40
20
2
Percentage of validation data used (N,/ Nemm))
Percentage validation used (No/
Softmax
Ours
Dirichlet
Top-HB
SB
TS
Figure 14: Auto-labeling error and coverage for different post-hoc methods on Tiny- ling-ImageNet while
we vary N,. No 18, 000 is the maximum number of points available for validation.
Labeling Error
12
10
Coverage
60
50
40
100
100
Percentage of validation data used (N,/ Nyam
Percentage of validation data used (N./ Nemo)
Softmax
Ours
Dirichlet
Top-HB
SB
TS
Figure 15: Auto-labeling error and coverage for different post-hoc methods on 20 Newsgroups while vary N,. N Vimae 1, 600 is the maximum number of points available for validation.
32

---

Method
Hyperparameter
Values
Temperature scaling
optimizer
Adam
learning rate batch size
0.001, 0.01, 0.1
64
max epoch
500
weight decay
0.01, 0.1, 1
Top-label histogram binning
points per bin
20,
Scaling-binniigg
number of bins
15,
learning rate
0.001, 0.01, 0.1
batch size
64
max epoch
500
weight decay
0.01, 0.1, 1
Dirichlet calibration
regularization parameter
0.001, 0.01, 0.
Ours
10, 100
features key
concat independent Adam
class-wise optimizer
learning rate
0.01, 0.1
max epoch weight decay batch size regularize
500
0.01, 0.1, 1
64
false
0.01, 0.1,
Table 10: Hyperparamters swept over for post-hoc methods. For each method, swept through all possible combinations of the possible values for each hyperparameter.
33


Reviews:
[{"review_id": "mcUhTu5Y99", "reviewer": "Reviewer_YvdP", "summary": "This paper introduces a Colander for threshold-based auto-labeling (TBAL) pipeline. Colander trains a flexible confidence function using latent information of the classifier model, instead of fixed confidence functions. The optimization problem of Colander is based on the objective of TBAL, that is, maximizing coverage while minimizing auto-labeling errors. The authors consider differentiable surrogates for the 0-1 variables using the sigmoid function to make this problem tractable using gradient-based methods.", "questions": "Active Learning Vs. TBAL\n- Active learning and TBAL fundamentally play the role of labelers for exploiting unlabeled raw data. Lines 100-104 include these differences, but it is hard to get the point. Please explain these differences in detail.\n- Although the paper includes a performance comparison with active learning (and self-training), an algorithmic setup of active learning is not provided. Please provide experimental settings for understanding whether it is fair or not.\n- Why does the TBAL process require a human labeler at every iteration? Is it impossible to use labeled and unlabeled data completely separately in advance?\n\nColander (proposed solution)\n- In Algorithm 1 (line 14), Colander extracts a confidence function  g ̂_i and t ̂_i^' through an optimization problem. As the reviewer’s knowledge, t ̂_i^' is different from  t ̂_i obtained estimation threshold process. Where is t ̂_i^' used?\n- The optimization problem (P1,2,3) is solved using a given classifier. It means that this optimization process only works when the classifier's performance is guaranteed. Is it right and is this a correct assumption?\n- The reviewer understood that the proposed solution configures the confidence function as a neural network, so it is flexible. But what is the meaning of 'choice of confidence function.' Does Colander learn multiple confidence functions and choose one of them?\n- Why did the authors select exactly two layers of the classifier as inputs to the confidence function? There are various options: single layer, three layers, and inclusion of output information.\n- Is it right to learn a new classifier/confidence function every round? If so, how long does the entire process require? If not, some expressions in the algorithm (pseudocode) and Section 3.3 need to be changed.\n\nPost-hoc method\n- Most baseline methods were proposed before 2020, except one algorithm. Please check following methods.\n  -\tR. Verma, et al. \"Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles.\" AISTATS, 2023.\n  -\tL. Tao, et al. \"Calibrating a deep neural network with its predecessors.\" IJCAI, 2023.\n  -\tT. Joy, et al. \"Sample-dependent adaptive temperature scaling for improved calibration.\" AAAI, 2023.\n  -\tThe reviewer would appreciate it if the authors could provide some insight/intuition into the problems with previous methods. The experimental sections just show empirical results and lack discussion.\n\nEmpirical results\n- It is important to provide accurate settings when performing partial improvements in the overall pipeline. Have you checked the performance of various procedure method changes such as ActiveQuery?\n- Are there any performance analysis results regarding changes in confidence scores?\n- The experimental results clearly show that the algorithm has high performance in the provided experimental settings. What about time efficiency? Have you checked the wall-time?\n- In the optimization problem (P3), how did the authors select the upper-bound error parameter ϵ? How do experimental results change depending on parameter changes?\n- The value of C_1 in line 8 of Algorithm 2 seems to be fixed at 0.25. What results can see when it changes?", "limitations": "In this paper, there was not enough discussion about the limitations.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "This paper is well-motivated. The proposed method is explained with simplicity and clarity.", "weaknesses": "Contribution: Although this paper proposes the Colander for the TBAL pipeline, there is no theoretical analysis and understanding of mathematical properties.\n\nCompleteness:\n- The paper includes unknown notation, which is used without pre-defined. \n- In Section 3.3, an undefined procedure exists: RandomQuery, RandomSplit, and ActiveQuery. In the case of Random method, it is possible to guess by readers, but in the case of ActiveQuery, a detailed explanation is needed.\n- There are a few expressions that make it difficult to figure out the intent. Additionally, some keywords used in the paper need to be unified. \n   - What is the difference between the score (or scoring) function and the confidence function?\n   - What is the difference between ‘inherent tension’ (in line 33) and trade-off?\n   - What is the difference between post-hoc method and calibration method?\n- The reviewer is unsure if Figure 1 is necessary for the reader: it contains acronyms that are not defined in advance, such as ERM, and certain formulas (h) are left undefined in the introduction.\n- Line 174, line 186: The reader does not know which Appendix to look at because Appendix information is omitted.\n- Appendix, Algorithm 2, line 8: The formula where C_1  weight is considered is not defined in the text.\n- Appendix, Table 3: there are typos.\n- This paper is considered to have no significant contributions compared to previous research about TBAL[1].\n[1] Vishwakarma, Harit, et al. \"Promises and pitfalls of threshold-based auto-labeling,\" Neural Information Processing Systems, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed review. We have updated our work to account for points on notation and writing. Our response:\n### Contributions\nWe believe the reviewer has missed our paper's key contribution, specifically **with respect to prior work on TBAL [1]**. Our work **does not reiterate [1]**. Instead, *the weaknesses of the TBAL technique in [1] inspired our work*. That is, [1] uses a simple, fixed confidence function that leads to poor performance in terms of error rate and coverage. Our goal is to understand the impact of this choice and to produce **innovations that resolve these weaknesses**, as illustrated in our draft's Figure 2. This is important given wide industry adoption of TBAL [2]. We: \n- demonstrate the importance of confidence functions in TBAL, evaluate several choices of functions, and find them to be **severely limited---as they are not well-aligned with TBAL objectives**.\n- propose a framework to learn the optimal confidence function and provide a practical version of it.\n- provide extensive empirical evaluations, showing that using confidence functions learned via Colander in TBAL improves auto-labeling coverage significantly.\n### Theoretical Analysis\nOur main contribution lies in demonstrating the issues with common choices of confidence functions in TBAL, proposing a principled solution to learn the optimal confidence functions for TBAL, and showing its effectiveness empirically. A rigorous theoretical analysis of our method is left as future work.\n### Completeness\n* We have fixed notation and typos.\n* ActiveQuery is discussed in Appendix B.3 (lines 583-603). We have added a detailed algorithm.\n* We use scoring and confidence functions synonymously.\n* The terms inherent tension and trade-off indeed refer to similar behaviors.\n* Train-time and post-hoc calibration methods are standard terms: Train-time methods modify the training procedure while post-hoc methods operate on trained classifiers.\n* Fig. 1 shows a simplified workflow of the standard procedure highlighting the role of confidence functions. We have set out to make the figure readable without the notations as well; we kept the notations for consistency with later figures.\n* It is the Appendix B.2 (line 571 onwards)\n* $\\hat{\\sigma}$  is the standard deviation of the error estimate. We have included the formula.\n### Active Learning vs. TBAL\nOur answer:\n1. The details of the AL setup and experiment are in Appendix A.1. We added further detail. \n2. The aim of the AL experiment is to **highlight the fundamental difference between TBAL and procedures (such as AL)** that seek to first learn the best possible classifier from the given function class and then do auto-labeling. These methods are severely limited by the choice of function class. On the other hand, TBAL succeeds since it iteratively auto-labels and eliminates the auto-labeled space. The comparison is fair as the methods use same amount of labeled data, same function classes, and training to learn the model.\n3. We adhered to the workflow of TBAL from [1]. While other ways to involve the human labeler are possible, these would not permit a comparison to [1].\n### On Colander\n1. Because we solve a relaxed version of the optimization, $\\hat{\\mathbf{t}}'_i$ is not guaranteed to ensure auto-labeling error below $\\epsilon_a$. We estimate the thresholds again in line 16, to ensure the auto-labeling error constraint is strictly followed by the estimated threshold.\n2. The optimization process does not require any specific performance assumptions on the classifier. It has to be better than a random classifier to get any meaningful output from the procedure; it is not a strong requirement.\n3. Colander does not learn multiple functions to choose from. In each round, it takes the current classifier and learns a confidence function by solving P3.\n4. Colander can use any function class for $g$. In experiments, we chose 2-layer nets and successfully used the same across all datasets, thus we may not need an exhaustive architecture search. Intuitively, we do not need a large network for $g$ since $h$ already performs the heavier representation learning work. As a result, simple models are preferable to avoid overfitting and to reduce training time (since post-hoc methods should be fast).\n5. TBAL learns a new classifier in every round, thus the confidence function also needs to be relearned. We comment on the time below.\n### Baselines\n We use prominent and recent post-hoc and train-time baselines. Thank you for the references. Of these [3] fits our setting. We provide results in the attached pdf (Table 1); these are consistent with the draft. The key insight is that baselines are not tailored to the TBAL objective. Colander is designed to maximize TBAL performance. Figure 2 and its discussion make this point.\n\n###  Empirical results\n1. Our primary focus is on confidence functions, and to avoid confounding, we have chosen not to compare various active querying strategies within this work. Including multiple variables would obscure the effects of our method.\n2. Could you please clarify this question?\n3. The wall clock time of our method is similar to other post-hoc methods: a single CIFAR-10 run on an NVIDIA RTX A6000 is roughly 1.5 hours (post-hoc) and 1 hour (no post-hoc).\n4. Our framework is flexible to work with any choice of $\\epsilon_a \\in (0,1)$. In experiments, we used a fixed $\\epsilon_a=0.05\\$. We provide additional results in Table 2 in the pdf with varying $\\epsilon_a$ to demonstrate its effect.\n5. Results with $C_1 \\in ${$0.0, 0.25$} in 20-newsgroup setting with vanilla train-time and all post-hoc methods are in Table 3. $C_1=0.0$ leads to higher variance in auto-labeling error, consistent with previous works [1].\n\n[1] Vishwakarma et al., Promises and Pitfalls of Threshold-based Auto-labeling, NeurIPS 2023.\n\n[2] https://aws.amazon.com/sagemaker/data-labeling\n\n[3] Joy et al., Sample-dependent Adaptive Temperature Scaling for Improved Calibration, AAAI 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZiVXpScZNV", "reviewer": "Reviewer_7Dad", "summary": "This paper addresses the challenges of overconfident scores in threshold-based auto-labeling (TBAL) systems. It critiques existing confidence scoring and calibration methods and introduces Colander, a new post-hoc method tailored to optimize TBAL performance.", "questions": "-\tNot a question but authors can consider following related paper for comparison. It also works with confidence scoring: https://aclanthology.org/2021.naacl-main.84.pdf", "limitations": "Yes.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "-\tThe paper has a good identification of the problem which is over confidence of existing TBAL confidence functions, and it proposes a novel framework to find the optimal confidence function. \n-\tThe paper conducts extensive experiments on both image and text data. \n-\tThe paper compares various post-hoc functions. \n-\tThe paper is well written with great details in Appendix", "weaknesses": "-\tThe authors rightly mention limitations (“A limitation of Colander is that similar to other post-hoc methods it also requires validation data to learn the confidence function. Reducing (eliminating) this dependence on validation data could be an interesting future work.”) in the conclusion. So, does this mean that they use Gold data for validation? If that is the case, that might be the weakness of TBAL in general. The following paper talks about it in detail: https://aclanthology.org/2023.acl-long.796/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and encouraging review. We are delighted with your assessment and recognition of our work’s contributions. Our response:\n\n**Dependence on validation data.**  Yes,  Colander, like other post-hoc methods, relies on validation data to learn the confidence function. In fact, this is in general a requirement in TBAL systems as noted in [4] and your reference [1]. Reducing or eliminating this dependence is a useful direction for future work. It is also related to the general problem of model evaluation in machine learning where solutions based on active testing have emerged [3].\n\n**Question on a related work.**  We appreciate the suggestion of the related paper [2]. This work uses softmax scores in a similar pipeline augmented with external knowledge sources. In our experiments, we have a comparison with the softmax scores. On the other hand, we believe using scores from Colander could be helpful in [2]. We have included a discussion on this idea in the draft.\n\n[1] https://aclanthology.org/2023.acl-long.796.pdf \n\n[2] https://aclanthology.org/2021.naacl-main.84.pdf\n\n[3] http://proceedings.mlr.press/v139/kossen21a/kossen21a.pdf\n\n[4] https://openreview.net/pdf?id=RUCFAKNDb2", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XlzIz0ZjSY", "reviewer": "Reviewer_Nkxd", "summary": "This paper discusses threshold-based auto-labeling functions aimed at identifying a large subset of unlabeled instances where the auto-labeling error remains below a specified threshold. The authors observed that standard temperature-scaled softmax scores are inadequate for effectively thresholding labeled instances. To address this, they propose learning a confidence scoring function and a threshold based on a subset of the validation data. This confidence function is a two-layer neural network trained on the logits from the last two layers of the base model. Extensive experiments were conducted on both image and text datasets using various training-time strategies to optimize the base model, demonstrating the efficacy of their approach.", "questions": "n/a", "limitations": "The authors acknowledge that the limitation of their work is the requirement of validation data.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well written, with a variety of experiments conducted to demonstrate the applicability of their proposed method. The choices of strategies and hyperparameters are documented well. \n2. The performance of the proposed method is quite significant, achieving lower error rate and higher coverage than the other methods, especially on the harder datasets.", "weaknesses": "1. The whole TBAL procedure is a hybrid mixture of iterative training/self training and active learning. While the authors explained the differences between TBAL and self-training + active learning (ST+AL), the difference seems small (mostly, TBAL aims to identify a low-error auto-labelled dataset and ST+AL aims for a good classifier). These two goals do not seem fundamentally different, and can be easily translated to each other. Given the similarity of the framework, I feel it is still necessary to include some ST/AL works in the experiments. \n2. A lesser weakness is that due to iterative training, the proposed method and compared methods all went through rounds of data selection, which makes the comparison indirect. For example, it would be hard to tell the exact improvement on thresholding quality the method brings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback and for noting the strengths of our paper. Our responses are the following:\n\n **On ST/AL works in the experiments**\n\n Our response involves the following 3 points:\n \n 1. The focus of our paper is to study confidence functions for existing TBAL techniques [1,2,3]. Our main contribution is a method to learn the **optimal confidence function for TBAL**. Given this focus, we have used various natural and widely used choices of confidence functions as baselines. In particular, we have considered both post-hoc methods and train-time methods used for calibrating confidence functions and thoroughly evaluated our proposed method for learning confidence function vs the combination of train-time and post-hoc methods. ST/AL is not a method for learning or calibrating confidence functions, and hence cannot be used as a baseline in our setting.\n \n\n 2. However, given the overlapping components, we addressed the fundamental differences between TBAL and ST/AL in our paper. Our experiments demonstrate that methods like ST/AL, which first aim to learn the best possible classifier from a given function class before performing auto-labeling, **can be severely constrained by the choice of function class**, especially when it does not contain a high accuracy classifier. In contrast, **for TBAL it is not necessary to learn a classifier with high accuracy**. Indeed, TBAL can iteratively auto-label and cover much of a dataset with classifiers that are moderately accurate. \n \n  3. Our aim is **not to establish TBAL as the state-of-the-art method** for data labeling or learning a good model with less data. For this reason, the comparison with ST/AL and other methods is not useful in our work: **what we seek to know is whether TBAL itself can be improved with our new confidence funtion approach**. An extensive evaluation of TBAL, ST/AL, and other similar techniques would be an interesting  work. The differences in the goal -- labeling data with reliability guarantees versus learning the best model in the class with fewer labels are important distinctions and these are discussed in detail along with thorough empirical comparisons in [1].\n \n\n**On indirect comparison due to multiple rounds**\n\nWe agree that multiple rounds of auto-labeling and data selection make the comparison indirect. However, it is evident that the gains are due to using Colander since we keep all the other components as it is and only change the confidence functions. To make this more clear, we have run experiments in the single round (passive learning) setting as well, and our results are consistent with the multi-round setting. Reinforcing the fact that the performance improvements are due to Colander, Table 4 in the pdf shows results in single round setting. The evolution of coverage and error over multiple rounds in the multi-round setting is shown in Figure 1.\n\n\n[1] Promises and Pitfalls of Threshold-based Auto-labeling, Vishwakarma et al., NeurIPS, 2023, \n\n[2] MCAL: Minimum Cost Human-Machine Active Labeling, Qiu et al. ICLR, 2023.\n\n[3] AWS Sagemaker Ground Truth https://aws.amazon.com/sagemaker/data-labeling/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C2emxzTHSG", "reviewer": "Reviewer_oXkX", "summary": "This paper proposes a novel auto-labeling method, called Colander. In contrast to existing works, Colander models the objective of finding an optimal confidence function as a constrained optimization problem (the confidence function should have maximum coverage whilst obtaining a sufficiently low error rate; both components are controlled with a penalty term). Colander is evaluated and compared to baselines on four datasets covering vision and language tasks: MNIST, CIFAR-10, Tiny-ImageNet and 20 Newsgroups. The obtained results clearly indicate that the method improves upon existing baselines, both in terms of coverage and error rate.", "questions": "* Did you conduct additional experiments evaluating how different architectures affect auto-labeling performances for different datasets (i.e., how does changing model architectures for each of the four analyzed datasets affect auto-labeling performance)?", "limitations": "As discussed in the Weaknesses, the paper mentions a limitation in the conclusion section but I'd encourage the authors to further elaborate on how and where their approach is limited, and how such limitations can potentially be addressed in future work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper proposes a novel approach to identifying confidence functions for auto-labeling. As indicated in the paper, finding confidence functions for auto-labeling is challenging, and this paper tackles this problem elegantly via constrained optimization.\n* The paper’s results are promising, showing that the introduced method improves upon existing baselines.\n* Overall, I believe that this paper provides a solid contribution to the research area of auto-labeling.", "weaknesses": "* The paper focuses heavily on formally introducing Colander, and the experimental content is comparatively thin. To provide the reader with a better notion of robustness, it would for example have been useful to provide additional details on the hyperparameter search in the main paper, and how selecting those differently affects both the coverage and error rate. Likewise, it would have been useful to provide additional details on the impact of the introduced penalty term.\n* It would also have been interesting to better understand the relationship between chosen model architecture for a dataset and auto-labeling performance.\n* The paper scarcely discusses limitations in the conclusion section, yet I would have expected a more detailed discussion of where and how this approach is limited.\n* Related to that, the paper does not touch upon future work / research questions that the obtained results create.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and for recognizing the key strengths of our paper. Our responses:\n\n\n**Details and effects of hyperparameter selection**\n\nDue to space constraints, we deferred a detailed discussion on hyperparameter search to Appendices C.3 to C.7. We plan to incorporate more details in the main paper given the additional space in the camera-ready version. In the rebuttal pdf (Table 5) we provide results with various choices of the penalty term $\\lambda$. We observe our method is robust to a wide range for  $\\lambda$, except around extreme values.\n\n\n**Effect of model architecture choice**\n\nIn TBAL it is not a priori clear what model the practitioner should use. The overall system is **flexible enough to work with any chosen model class**. Our focus is on evaluating the effect of various training time and post-hoc methods designed to improve the confidence functions for any given model. To answer the query, **we ran experiments with Resnet18 and ViT models in the CIFAR-10 setting** (see Table 6 in the rebuttal pdf). As we expected there are variations in the results in the baselines due to model choices but **our method maintains high performance irrespective of the classification model used**. This is due to its ability to learn confidence scores tailored for TBAL.\n\n\n**Limitations and future work**\n\nWe discussed these in the conclusions sections and briefly reiterate them here. Colander, like other post-hoc methods, relies on validation data to learn the confidence function. The use of validation data is in general a requirement in TBAL systems as noted in [1]. Reducing or eliminating this dependence is an important direction for future work. It is also related to the general problem of model evaluation in machine learning where solutions based on active testing have emerged [2]. We have updated the draft with additional discussions.\n\n[1] Vishwakarma et al., Promises and Pitfalls of Threshold-based Auto-labeling, NeurIPS, 2023.\n\n[2] Kossen et al., Active Testing: Sample–Efficient Model Evaluation, ICML, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dr7CznZuwN", "reviewer": "Reviewer_ffq7", "summary": "This paper introduces a threshold-based auto-labeling (TBAL) method called Colander to maximize TBAL performance by finding the optimal labling confidence function and thresholds. In order to find the optimal confidence function, Colander treats the auto-labeling objective as an optimization problem that maximizes the coverage under label error constraint. They use a neural network as the confidence function, and design a surgate for the optimization problem which can then be optimized using gradient-based methods.", "questions": "Q1: How do you determine the threshold set of _T_? Is it a grid of _k_ classes * _n_ steps in [0,1]?\n\nQ2: Why not use the thresholds produced by Colander (line 14 of Algorithm 1)?\n\nQ3: My understanding is the that confidence function is searched by a neural network using gradient descent, and the thresholds are searched by iterating the training process over the threshold set T. But can we also make the thresholds a neural network (meaning that thresholds would be sample-specific)?", "limitations": "The computation cost of Colander increases linearly with the size of the set of confidence thresholds _T_, but it seems the process requires finer granularity in _T_ in order to get a better estimate of the optimal confidence function and thresholds.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The paper turns auto-labeling into an optimization problem, and proposes a disciplined solution that can be solved by gradient methods.\n\nS2: The proposed optimization surrogate may be adopted by other auto-labeling methods.\n\nS3: The paper presents extensive experiments to compare with existing methods, and the results are promising.", "weaknesses": "W1: the paper is overall well-written, but the discussion of the thresholds is confusing or missing details. Specifically:\n- In line 89, it says \"the vector _**t**_ denotes scores over _k_ classes\", and I suppose score means the predicted scores (or probabilities) of the _k_ class, but later _**t**_ is defined as a vector of thresholds.\n- In line 144 (P1), it is unclear what _T^k_ stands for, and how the set of threshold, _T_, is determined. \n- In Algorithm 1, the Colander produces the estimated confidence function and thresholds hat _ti'_ (line 14), but the threshold is not used, and it relies on Algorithm 2 to estimate the threshold for each class. There is no discussion what the difference between these two thresholds.\n\nW2: There is no discussion of the computation overhead of Colander.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and acknowledgment of the paper's strengths. Our response:\n\n**Clarification on thresholds**\n\nThe vector $\\mathbf{t}$ denotes the thresholds over $k$ classes. $T^{k}$ stands for the space of threshold vectors $\\mathbf{t}$. Because we solve a relaxed version of the optimization, using  $\\hat{\\mathbf{t}}'_i$ does not guarantee auto-labeling error below $\\epsilon_a$. We estimate the thresholds again in line 16 to ensure the auto-labeling error constraint is strictly followed by the estimated threshold.  We have updated the draft to further clarify this point. \n\n \n**Discussion on computation overhead** \n \nThe wall clock time of our method is similar to other post-hoc methods. For instance, a single run on CIFAR-10 setting on NVIDIA RTX A6000 takes roughly 1.5 hours with post-hoc methods and roughly 1 hour without post-hoc methods. We have included a discussion on the computation overhead in the paper.\n \n**Answers**\n\n **A1.** Yes, our current implementation uses a grid in $[0,1]$ as the set $T$. We emphasize that this is not a necessity. We chose it to favor a simple implementation. It boils down to evaluating the number of thresholds equal to the number of validation points given in the round, so the effective size of $T$ is the number of validation points $N_v$. Moreover, a binary search based implementation can reduce the effective size to $\\log(N_v)$ for each class. \n \n**A2.**  See the above discussion on thresholds.\n\n\n**A3.** It is an interesting idea for future works to make thresholds instance-specific. We chose the current thresholding technique as it is backed by the theoretical guarantees on auto-labeling error and coverage [1].\n\n\n**Limitation.** Please see our response A1 above.\n\n\n[1] Vishwakarma et al.,  Promises and Pitfalls of Threshold-based Auto-labeling,  NeurIPS 2023,", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all of the reviewers for their insightful and positive feedback. We have used their suggestions to improve our draft, added experiments, and improved the clarity of our work. Before providing individual responses, we (1) summarize the strengths highlighted by reviewers, (2) provide a pair of common responses, and (3) describe some new experiments that strengthen our work. 


## 1. Strengths

**Motivation and novelty of our method (7Dad, Nkxd, oXkX, ffq7, YvdP):** Reviewers appreciated our **novel and well-motivated approach** to addressing the issue of overconfidence in auto-labeling tasks. The reviewers appreciated our mathematically grounded framework to learn the optimal confidence function for TBAL. They also noted our optimization framework could be useful for a variety of auto-labeling methods.

**Significant performance improvements over the vanilla TBAL (Nkxd, oXkX, ffq7):** Integrating our method into TBAL provides significant performance improvements over the vanilla version used in the previous works on TBAL [1].

**Thorough empirical evaluation (7Dad, Nkxd, ffq7):** The evaluations covered diverse datasets, including text and image data, showcasing the proposed method's applicability across different domains. We provide comparisons against several train-time and post-hoc methods commonly used to reduce the overconfidence issue.

**Clarity (7Dad, Nkxd, oXkX):** Most reviewers found our paper well-written and easy to follow, and they appreciated the illustrations. They also liked our step-by-step discussion for designing the objective function, which leads to a tractable optimization problem.

## 2. Response to common questions
* **On contributions and practical significance**. We are motivated by the fact that TBAL is widely used in practice to create labeled datasets [2], including by industry giants like Amazon (in its SageMaker Ground Truth product). TBAL has also recently been the subject of theoretical study in exciting works such as [1,3]. Given this interest---both practical and theoretical---our goal is to understand the role of **confidence functions, a key component in TBAL that has not been previously studied**. As our work shows, TBAL systems are heavily affected by the choice of confidence function. This led us to develop a new approach for these that **leads to substantial improvements in TBAL systems**. Given its strengths, we anticipate that our new method will become the standard for TBAL systems. 

* **On active learning/self-training and TBAL**: The focus of our work is on understanding and improving confidence functions for TBAL. A prior work [1] studies TBAL theoretically and shows the fundamental differences between active learning and TBAL. It also provides extensive experiments illustrating this difference. Since there are some overlaps in these methods, we have a brief discussion and a simulation in our paper to clarify the fundamental differences between them.


## 3. Additional Experiments
We provide a brief summary of the additional experiments and results here. The tables and figures are in the attached rebuttal pdf. The details are deferred to individual reviewer responses.

1. (YvdP) We run TBAL with the additional calibration baseline Adaptive Temperature Scaling (AdaTS) [4] and report the results in Table 1. The results are **consistent with the main paper and our expectations**. 
2. (YvdP) We run TBAL with five values of $\epsilon_a \in$ { $0.01,0.025,0.05,0.075, 0.1 $ } and report the results in Table 2. **As expected** the auto-labeling error is high with larger values of $\epsilon_a$ and smaller with small $\epsilon_a$.

3. (YvdP) The results with $C_1 \in$ { $0.0, 0.25$ } are in Table 3. Using $C_1=0.0$ leads to higher variance in the auto-labeling error. This is consistent with prior work [1].

4. (Nkxd) We further demonstrate that the **performance gains are due to the use of Colander**, even if methods use multiple rounds. To do so, we show the evolution of coverage and error over multiple rounds in Figure 1 in the rebuttal pdf. The effects of using Colander are visible from the first round itself, and the following rounds improve performance further. We also run a single round (passive) variant of TBAL where we sample all the human-labeled points for training ($N_t$) randomly at once, train a classifier, do auto-labeling, and then stop. This setting avoids confounding due to multiple rounds. We observe that using Colander yields significantly higher coverage in comparison to the baselines (see Table 4 in the pdf). This reinforces the fact that the gains in the multi-round TBAL are directly due to Colander, while multiple rounds of data selection, training, and auto-labeling are superior to doing everything in a single round.
 

5. (oXkX) We run TBAL with ViT and ResNet18 models. The results are in Table 6. We see the choice of classification model affects baseline performance but TBAL with Colander remains robust to the model choice.

6. (oXkX) The results with $\lambda$ variation are in Table 5. We see Colander is robust to a wide range of values of $\lambda$, except extreme values.


**References**

[1] Vishwakarma et al., Promises and Pitfalls of Threshold-based Auto-labeling, NeurIPS 2023.

[2] https://aws.amazon.com/sagemaker/data-labeling/

[3] Qiu et al., MCAL: Minimum Cost Human-Machine Active Labeling, ICLR, 2023

[4] Joy et al., Sample-dependent Adaptive Temperature Scaling for Improved Calibration, AAAI 2023.

Meta-review: The submission proposed a principled approach to optimize for confidence thresholds for TBAL, and showed its effectiveness empirically. Reviewers pointed out that there is no rigorous theoretical analysis or results, but generally agreed that the approach is principled and empirical results are sound. 

There were considerable discussions on the difference between TBAL and active learning, and I think the authors have done sufficient work in trying to clarify the differences between the two approaches, and added sufficient details to the submission. TBAL is a relatively new and interesting framework with some theoretical properties analyzed by previous work, e.g. [1]. I think this paper proposed an interesting approach to improve empirical TBAL performance.

[1] Vishwakarma et al., Promises and Pitfalls of Threshold-based Auto-labeling, NeurIPS 2023.
Decision: Accept (poster)