Pearls from Pebbles: Improved Confidence Functions for Auto-labeling

Authors: harit  vishwakarma
yi  chen
jiet  sui
  tay
satya  sai
srinath  namburi
frederic  sala
ramya  korlakai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-03 13:13:25

Abstract: Auto-labeling is an important family of techniques that produce labeled training sets with minimum manual annotation. A prominent variant, threshold-based auto-labeling (TBAL), works by finding thresholds on a model's confidence scores above which it can accurately automatically label unlabeled data. However, many models are known to produce overconfident scores, leading to poor TBAL  performance. While a natural idea is to apply off-the-shelf calibration methods to alleviate the overconfidence issue, we show that such methods fall short. Rather than experimenting with ad-hoc choices of confidence functions, we propose a framework for studying the optimal TBAL confidence function. We develop a tractable version of the framework to obtain Colander (Confidence functions for Efficient and Reliable Auto-labeling), a new post-hoc method specifically designed to maximize performance in TBAL systems. We perform an extensive empirical evaluation of Colander and compare it against methods designed for calibration. Colander achieves up to 60% improvement on coverage over the baselines while maintaining error level below 5% and using the same amount of labeled data.

Main paper: 
 introduction: 
 The demand for labeled data in machine learning (ML) is perpetual. Obtaining it is expensive and time-consuming, creating a bottleneck in ML workflows. Threshold-based auto-labeling (TBAL) is a promising solution to obtain high-quality labeled data at low cost [47, 42, 56] . A TBAL system (Fig. 1 ) takes unlabeled data as input and outputs a labeled dataset. It works iteratively: in each iteration, it acquires human labels for a small chunk of data to train a model, then auto-labels points using the model's predictions where its confidence scores are above a certain threshold. The threshold is determined using validation data so that the auto-labeled points meet a desired accuracy criteria. The goal is to maximize coverage-the fraction of points automatically labeled (out of the total)-while maintaining accuracy. TBAL powers industry products like Amazon SageMaker Ground Truth [47] .The confidence function is critical to the TBAL workflow (Figure 1 ). Existing TBAL systems rely on common choices like softmax outputs from neural networks [42, 56] . These functions are not well aligned with the objective of the auto-labeling system. Using them results in substantially suboptimal coverage (Figure 2(a) ). For this reason, we ask: Notation. Let [m] := {1, 2, . . . , m} for any natural number m. Let X u be a set of unlabeled points drawn from some instance space X. Let Y = {1, . . . , k} be the label space. There is an unknown ground truth labeling function f * : X → Y. Let O be a noiseless oracle that provides the true label for any point x ∈ X. Denote the model (hypothesis) class by H, where each h ∈ H is a function h : X → Y. Each classifier h also has an associated confidence function g : X → ∆ k that quantifies the confidence of the prediction by model h ∈ H on any data point x ∈ X. Here, ∆ k is a (k -1)-dimensional probability simplex. Let v[i] denote the i th component for any vector v ∈ R d . For any point x ∈ X the prediction is ŷ := h(x) and the associated confidence is g(x) [ŷ] .The vector t denotes scores over k-classes, and t[y] denotes its y th entry, i.e., score for class y. Table 3 (in Appendix B.4) contains a summary of the notation.Threshold-based auto-labeling. It seeks to obtain labeled datasets while reducing the labeling burden on humans (Figure 1 ). The input is a pool of unlabeled data X u . It outputs, for each x ∈ X u , a label ỹ ∈ Y. The output label could be either y, from the oracle (human), or ŷ, from the model. Let N u be the number of unlabeled points, A ⊆ [N u ] the set of indices of auto-labeled points, and X u (A) be these points. Let N a be the size of the auto-labeled set A. The auto-labeling error, denoted by E(X u (A)), and the coverage, denoted by P(X u (A)), are defined as follows:E(X u (A)) := 1 N a i∈A 1(ỹ i ̸ = f * (x i )), and P(X u (A)) := |A|/N u = N a /N u .The goal of an auto-labeling algorithm is to label the dataset so that E(X u (A)) ≤ ϵ a while maximizing coverage P(X u (A)) for a user-given error tolerance parameter ϵ a ∈ [0, 1]. As depicted in Figure 1 , the TBAL algorithm proceeds iteratively. In each iteration, it queries labels for a subset of unlabeled points from the oracle. It trains a classifier from the model class H on the oracle-labeled data acquired till that iteration. It then uses the model's confidence scores on the validation data to identify the region in the instance space, where the current classifier is confidently accurate and automatically labels the points in this region. The auto-labeled points are removed from the unlabeled pool. Similarly, to maintain parity between the validation and unlabeled data in the next round, the validation points in the auto-labeling region are removed as well. These steps are executed in a loop until all the data is labeled or the budget to query oracle labels is exhausted.Fundamental differences between TBAL, self-training and active learning. At first glance, TBAL may appear similar to active learning (AL) [46] , self-training (ST) [2] , and selective classification (SC) [10] . However, as described in [56] , it is a fundamentally different technique designed with different goals. Perhaps the most substantial difference is that TBAL's aim is to create accurately labeled datasets, while the goal in AL and ST is to learn the best (in terms of generalization error) possible classifier in a given model class with limited ground truth labels. This difference is most substantial in the settings where AL converges to a bad classifier, e.g., due to incorrect choice of the model class, sampling bias, etc. [56] illustrates this notion with a scenario where TBAL coverage is above 95% while the other techniques average around 20%. See Appendix A.1 for details. th and ĝi auto-labeling thresholds ti are determined to ensure the auto-labeled data has error at most ϵ a (a user-given parameter). After obtaining the thresholds the rest of the steps are the same as standard TBAL. The whole workflow runs until all the data is labeled or another stopping criterion is met.Problems with confidence functions in TBAL. The success of TBAL hinges on the ability of the classifier's confidence scores to distinguish between correct and incorrect labels. Prior works on TBAL [56, 42] train the model with stochastic gradient descent (SGD) and use the softmax output of the model as confidence scores, which are known to be overconfident [38] . A natural choice to mitigate this problem is to use post-hoc calibration techniques, e.g., temperature scaling [12] . We evaluate these choices by running TBAL for a single round on the CIFAR-10 [24] dataset with a SimpleCNN model with 5.8M parameters [20] with error threshold 5%. Details are in Appendix A.2.In Figures 2(d ) and 2(e) we observe that using softmax scores from the classifier only produces 2.9% coverage while the error threshold is violated with 10% error. Using temperature scaling only increases the coverage marginally to 4.9% and still violates the threshold with error 14%. Looking closer at the scores for correct versus incorrect examples on validation data, we observe a large overlap for softmax (Figure 2 (a)) and a marginal shift with considerable overlap for temperature scaling (Figure 2 (b)). To overcome this challenge, we propose a novel framework (Section 3) to learn such confidence functions in a principled way. Our method in this example can achieve 50% coverage with an error of 3.4% within the desired threshold (Figure 2 (c)).To illustrate the differences between TBAL and the combination of active learning (AL) and selftraining for the task of data labeling, we run an experiment on the 2 concentric circles data setting as used in [56] . The details are as follows:Data setting. We generate two concentric circles with points in the outer circle belonging to one class and the inner circle belonging to the other class. The total number of points generated is 10,000 of which we use 2000 for validation. Then it auto-labels all the remaining unlabeled points with this classifier's predictions. For AL+Self-Training+SC, we do selective auto-labeling using ĥal-st , i.e., only auto-label the points where the classifier will have an error at most ϵ a . We use ϵ a =1% here.Results and discussion. The Figure 5 shows auto-labeling error and coverage achieved by these methods when run with different choices of human-labeled data budget for training. First, we can see that even with linear classifiers TBAL is able to auto-label a huge chunk of the data (high coverage) while maintaining auto-labeling error below the tolerance level of 1% On the other hand, methods like AL+Self-Training (+SC) that try to first learn the optimal classifier in the given function class either have high auto-labeling error or very low coverage. These results are also consistent with the observations in [56] on the comparison between TBAL and AL, AL+SC. While such findings confirm the notion that there are differences-and, at least in some settings, advantages-for the TBAL approach compared to other techniques, we reiterate that our goal is to understand and improve the role of the confidence function within TBAL, rather than comparing TBAL to other techniques. 
squentropy [19]:: 
 This method adds the average square loss over the incorrect classes to the cross-entropy loss. This simple modification to the Vanilla method leads to the end model with better test accuracy and calibration.3. Correctness Ranking Loss (CRL) [35] : This method includes a term in the loss function of the vanilla training method so that the confidence scores of the model are aligned with the ordinal rankings criterion [17, 6] . The confidence functions satisfying this criterion produce high scores on points where the probability of correctness is high and low scores on points with low probabilities of being correct.4. FMFP [64] aims to align confidence scores with the ordinal rankings criterion. It uses Sharpness Aware Minimizer (SAM) [7] to train the model, with the expectation that the flat minima would benefit the ordinal rankings objective of the confidence function.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss them briefly.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
proposed method (colander): 
 The observations in Figure 2 (a) and 2(b) suggest that fixed choices of confidence functions can leave significant coverage on the table. To find a better choice in a principled manner, we develop a framework based on auto-labeling objectives-maximizing coverage while having bounded autolabeling error. We instantiate it by using empirical estimates and easy-to-optimize surrogates. We use the overall TBAL workflow from [56] and introduce our method to replace the confidence (scoring) function after training the classifier.In any iteration of TBAL, we have a model h trained on a subset of data labeled by the oracle. This model may not be highly accurate. However, it could be accurate in some regions of the instance space, and with the help of a confidence function g, we want to identify the points where the model is correct and auto-label them. As we saw earlier, arbitrary choices of g perform poorly on this task. Instead, we propose a framework to find the right function from a sufficiently rich family.Optimal confidence function. To find a confidence function aligned with our objective, we consider a space of thresholds T (e.g. [0, 1]) and functions G : X → T k , where T k is the k-dimensional product space of T . We express the auto-labeling objective as an optimization problem (P1):arg max g∈G,t∈T k P(g, t | h) s.t. E(g, t | h) ≤ ϵ a .(P1)Here P(g, t|h) and E(g, t | h) are the population level coverage and auto-labeling error which are defined as follows,EQUATIONThe optimal g ⋆ and t ⋆ that achieve the maximum coverage while satisfying the auto-labeling error constraint belong to the solution(s) of this optimization problem.The framework provides a theoretical characterization of the optimal confidence functions and thresholds for TBAL. However, it is impractical since the distributions and f ⋆ are unknown. Next, we give a practical method based on the above framework to learn confidence functions for TBAL.Empirical optimization problem. Since we do not know the distributions of x and f ⋆ , we use estimates of coverage and auto-labeling errors on a fraction of validation data to solve the optimization problem. Let D be some finite number of labeled samples, and then the empirical coverage and auto-labeling error are defined as follows,EQUATIONEQUATIONWe randomly split the validation data into two parts D cal and D th and use D cal to compute P(g, t | h, D cal ) and E(g, t | h, D cal ). Using these estimates, we now seek to solve the following problem,arg max g∈G,t∈T k P(g, t | h, D cal ) s.t. E(g, t | h, D cal ) ≤ ϵ a . (P2)Nevertheless, the presence of 0-1 variables means the problem remains challenging.To make the optimization (P2) tractable using gradientbased methods, we introduce differentiable surrogates for the 0-1 variables. Let σ(α, z) := 1/(1 + exp(-αz)) denote the sigmoid function on R with scale parameter α ∈ R. It is easy to see that, for any g, y and t, g(x)[y] ≥ t[y] ⇐⇒ σ(α, g(x)[y] -t[y]) ≥ 1/2.Using this fact, we define the following surrogates of the auto-labeling error and coverage:EQUATIONEQUATIONand the surrogate optimization problem as follows,arg min g∈G,t∈T k -P(g, t | h, D cal ) + λ E(g, t | h, D cal ) (P3)Here, λ ∈ R + is the penalty term controlling the relative importance of the auto-labeling error and coverage. We tune it with the procedure discussed in Section 4.3. The gap between the surrogate and actual coverage diminishes as α → ∞. We discuss this in the Appendix.Choice of G. Our framework is flexible with respect to the choice of function class G. In this work, we use neural networks with at least two layers on model class H. We use representations from the last two layers as input for the functions in G (Figure 4 ). Let z (1) (x; h) ∈ R k and z (2) (x; h) ∈ R d2 be the outputs of the last and the second-last layer of the net h for input x and let z(x; h) := [z (1) (x; h), z (2) (x; h)] denote the concatenation. This input is passed to network G nn2 : R k+d2 → ∆ k ; it outputs confidence scores for the k classes. Specifically g is defined as g(x) := softmax W 2 tanh(W 1 z(x; h)) . Here W 1 ∈ R (k+d2)×2(k+d2) and R Solving the surrogate optimization. The optimization problem (P3) is nonconvex. Nevertheless, it is differentiable and we can apply gradient-based methods. We solve for g and t simultaneously using Adam [23] . Training details, including hyperparameters, are deferred to the Appendix.We take the workflow of TBAL and plugin our method Colander to learn the new confidence function and threshold. We discuss the updated workflow below and place the detailed Algorithms 1 and 2 in the Appendix B due to space constraints.1. Initialization. First, select n s points randomly from X u and obtain human labels for them to create initial training data Dtrain . This is written as RANDOMQUERY(X u , n s ) in Algorithm 1. The procedure RANDOMQUERY(X u , n s ) selects n s points randomly from X u and obtains human labels for them to create D train . Note, this model ĥi may not be highly accurate due to several factors such as the small amount of training data, the choice of H, the training algorithm, and its hyperparameters. Indeed, TBAL does not expect the model to have high accuracy but it aims to identify and auto-label points where the model's accuracy is at least 1 -ϵ a .3. Learn new confidence function using Colander . The model ĥi obtained in the previous step also produces softmax scores that can be used for auto-labeling. However, as we saw earlier in Section 2, using these scores may lead to poor auto-labeling performance. Thus, we plug in our procedure Colander to learn new scores designed to auto-label as many points as possible with the current model ĥi while respecting the error constraint. We first randomly splits the validation data D cal . We solve it to obtain the post-hoc confidence function ĝi , which is expected to provide the most appropriate scores for TBAL.We get thresholds t′ i as output from Colander, and it is tempting to use these along with ĝi for auto-labeling. However, these thresholds may violate the auto-labeling error constraint as they are obtained by solving the relaxed optimization problem. Thus, it is crucial to estimate reliable thresholds ti from the held-out data D 5. Auto-labeling. This is a simple step. We compute the scores on the remaining unlabeled data X (i) u using the function ĝi and any point x ∈ X (i) u having score above t[ŷ] is assigned auto-label ŷ = ĥi (x), and the points that did not meet this criterion remain unlabeled.6. Remove auto-labeled points. The points that got auto-labeled in the previous steps are removed from the unlabeled pool. To make the validation data consistent with this unlabeled pool for the next round, the points in the validation data that fall into the auto-labeling region are also removed.Get more human-labeled data. Lastly, it calls the procedure ACTIVEQUERY( ĥi , X (i) u , n b ) to select n b points from the remaining unlabeled pool using an active learning strategy. This newly acquired human-labeled data is added to the training dataD (i)train . The details of the querying strategy are in Algorithm 3 Appendix B. Note, that the TBAL procedure is flexible to work with any choice of active querying strategy. We pick a simple strategy based on random sampling from the points where the classifier is most uncertain. To avoid confounding in TBAL with other scores we use the softmax scores from the classifier to determine uncertainty here.The procedure then moves to step 2 and runs the loop until there are no more unlabeled points left or it has queried the stipulated number of human labels N t .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: It is an empirical paper, it does not have theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
empirical evaluation: 
 We validate the following claims through extensive empirical evaluation, C1. Colander learns better confidence functions for auto-labeling compared to standard training and common post-hoc methods that seek to mitigate the overconfidence problem. Using it in TBAL can boost the coverage significantly while keeping the auto-labeling error low.C2. Colander is independent of any particular train-time method and thus should help improve the performance when coupled with different train-time methods.We examine several train-time and post-hoc methods that improve confidence functions from the calibration and ordinal ranking perspectives. Details of these methods are in the Appendix C.7.Train-time methods. We use the following methods for training the model ĥ. Vanilla neural networks are trained with the cross-entropy loss using stochastic gradient descent (SGD) [1, 4, 12] . Squentropy [19] adds the average square loss over the incorrect classes to the cross-entropy loss to improve the calibration and accuracy of the model. Correctness Ranking Loss (CRL) [35] aligns the confidence scores of the model with the ordinal rankings criterion via regularization. FMFP [64] aligns confidence scores with the ordinal rankings criterion by using sharpness-aware minimization (SAM) [7] in lieu of SGD.Post-hoc methods. We use the following methods for learning (or updating) the confidence function ĝ after learning ĥ. Temperature scaling [12] is a variant of Platt scaling [41] . It rescales the logits by a learnable scalar parameter. Top-Label Histogram-Binning [14] builds on the histogram-binning method [62] and focuses on calibrating the scores of the predicted label assigned to unlabeled points. Scaling-Binning [28] applies temperature scaling and then bins the confidence function values. 1 : Details of the dataset and model we used to evaluate the performance of our method and other calibration methods. For the Tiny-Imagenet and 20 Newsgroup datasets, we use CLIP and FlagEmbedding, respectively, to obtain the embeddings of these datasets and conduct auto-labeling on the embedding space. For Tiny-Imagenet, we use a 3-layer perceptron with 1,000, 500, 300 neurons on each layer as model h; for 20 Newsgroup, we use a 3-layer perceptron with 1,000, 500, 30 neurons on each layer as model h.instances of each class and assumes Dirichlet class conditional distributions. Adaptive Temperature Scaling [21] builds on top of temperature scaling and considers that different samples contribute to the calibration error differently. Each train-time method is piped with a post-hoc method, yielding a total of 4 × 6 = 24 methods.We evaluate the performance of auto-labeling on four datasets. Each is paired with a model for auto-labeling: MNIST [30] is a hand-written digits dataset. We use the LeNet [31] for auto-labeling. CIFAR-10 [24] is an image dataset with 10 classes. We use a CNN with approximately 5.8M parameters [20] for auto-labeling. Tiny-ImageNet [29] is an image dataset comprising 100K images across 200 classes. We use CLIP [43] to derive embeddings for the images in the dataset and use an MLP model. 20 Newsgroups [34] is a natural language dataset comprising around 18K news posts across 20 topics. We use the FlagEmbedding [58] to obtain text embeddings and use an MLP model.The complexity of TBAL workflow and lack of labeled data make hyperparameter search and evaluation challenging. Similar challenges have been observed in active learning [32] . We discuss our practical approach and defer the details to Appendix C.10 and code2 .Hyperparameter search. We run only the first round of TBAL with each method using a hyperparameter combination 5 times and measure the mean auto-labeling error and mean coverage on D hyp , which represents a small part of the held-out human-labeled data. We pick the combination that yields the lowest average auto-labeling error while maximizing the coverage. We first find the best hyperparameters for each train-time method, fix those, and then search the hyperparameters for the post-hoc methods. Note that the best hyperparameter for a post-hoc method depends on the training-time method that it pipes to. The hyperparameter search spaces are in the Appendix C; and the selected values used for each setting are in the supplementary material.Performance evaluation. After fixing the hyper-parameters, we run TBAL with each combination of train-time and post-hoc method on full X u of size N , with a fixed budget of N t labeled training samples and N v validation samples. The details of these values for each dataset are in Table 1 in Appendix C. Here, we know the ground truth labels for the points in X u , so we measure the auto-labeling error and coverage as defined in (1) and report them in Table 2 .Our findings, shown in Table 2 , are:C1: Colander improves TBAL performance. Our approach aims to optimize the confidence function to maximize coverage while minimizing errors. When applied to TBAL, we expect it to yield substantial coverage enhancement and error reduction compared to vanilla training and softmax scores. Indeed, the results in Table 2 corresponding to the vanilla training match our expectations. We see across all data settings, our method achieves significantly higher coverage while keeping autolabeling error below the tolerance level of 5%. The improvements are even more pronounced when the datasets are more complex than MNIST. Also consistent with our expectation and observations in Figure 2 (b), the post-hoc calibration methods improve the coverage over using softmax scores but at the cost of slightly higher error. While they are reasonable choices to apply in the TBAL pipeline, they fall short of maximally improving TBAL performance due to the misalignment of goals.C2: Colander is compatible with and improves over other train-time methods. Our method is compatible with various choices of train-time methods, and if a train-time method (Squentropy here) provides a better model relative to another train-time method (e.g., Vanilla), then our method exploits this gain and pushes the performance even further. Across different train-time methods, we do not see significant differences in the performance, except for Squentropy. Using Squentropy with softmax improves the coverage by as high as 6-7% while dropping the auto-labeling error in contrast to using softmax scores obtained with other train-time methods for the Tiny-ImageNet setting. This is unexpected: Squentropy adds the average square loss over the incorrect classes as a regularizer, and it has offered better accuracy and calibration compared to training with cross-entropy loss.Train-time methods designed for ordinal ranking objective perform poorly in auto-labeling. CRL and FMFP are state-of-the-art methods designed to produce scores aligned with the ordinal ranking criteria. Ideally, if the scores satisfy this criterion, TBAL's performance would improve. However, we do not see any significant difference from the Vanilla method. Similar to the other baselines, their evaluation is focused on models trained on large amounts of data. But, in TBAL, we have less data for training. The training error goes to zero after some rounds, and no information is left for the CRL loss to distinguish between correct and incorrect predictions (i.e., count SGD mistakes). On the other hand, FMFP is based on a hypothesis that training models using Sharpness Aware Minimizer (SAM) could lead to scores satisfying the ordinal ranking criteria. However, this phenomenon is still not well understood, especially in settings like ours with limited training data.: This method models the distribution of predicted probability vectors separately on instances of each class and assumes the class conditional distributions are Dirichlet distributions with different parameters. It uses linear parameterization for the distributions, which allows easy implementation in neural networks as additional layers and softmax output.Note: For binning methods, uniform mass binning [62] has been a better choice over uniform width binning. Hence, we use uniform mass binning as well.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: All the necessary details are provided in Section 4 and in the Appendix C. We have also uploaded the code along with the submission. 
related work: 
 Data labeling. We briefly discuss prominent methods for labeling. Crowdsourcing [45, 50] uses a crowd of non-experts to complete a set of labeling tasks. Works in this domain focus on denoising the obtained information, modeling label errors, and designing effective labeling tasks [11, 22, 33, 53, 52, 54, 5] . Weak supervision (WS), in contrast, emphasizes labeling through multiple inexpensive but noisy sources, not necessarily human [44, 48, 55, 18, 49, 60, 63] . Works such as [44, 8] concentrate on binary or multi-class labeling, while [48, 55] extend WS to structured prediction tasks.Auto-labeling occupies an intermediate position between weak supervision and crowdsourcing in terms of human dependency. It aims to minimize costs to obtain human labels while generating high-quality labeled data using a specific model. [42] use a TBAL-like algorithm and explore the cost of training for auto-labeling with large-scale model classes. Recent work [56] theoretically analyzes the sample complexity of validation data required to guarantee the quality of auto-labeled data.Overconfidence and calibration. The issue of overconfidence [51, 38, 16, 3] is detrimental in several applications (such as robustness to out-of-distribution points [59, 57] ), including ours. Many solutions have emerged to mitigate the overconfidence and miscalibration problems. Gawlikowski et al. [9] provide a comprehensive survey on uncertainty quantification and calibration techniques for neural networks. Guo et al. [12] evaluated a variety of solutions ranging from the choice of network architecture, model capacity, weight decay regularization [25] , histogram-binning and isotonic regression [61, 62] and temperature scaling [41, 39] which they found to be the most promising solution. The solutions fall into two broad categories: train-time and post-hoc. Train-time solutions modify the loss function, include additional regularization terms, or use different training procedures [27, 37, 36, 19] . On the other hand, post-hoc methods such as top-label histogram-binning [13] ,scaling binning [28] , Dirichlet calibration [26] calibrate the scores directly or learn a model that corrects miscalibrated confidence scores.Beyond calibration. While calibration aims to match the confidence scores with a probability of correctness, it is not the precise solution to the overconfidence problem in many applications, including our setting. The desirable criteria for scores for TBAL are closely related to the ordinal ranking criterion [17] . To get such scores, Corbière et al. [6] add a module in the net for failure prediction, Zhu et al. [64] switch to sharpness aware minimization [7] to learn the model; CRL [35] regularizes the loss.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]Justification: We use publicly available datasets and uploaded the code as supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 We studied issues with confidence scoring functions used in threshold-based auto-labeling (TBAL). We showed that the commonly used confidence functions and calibration methods can often be a bottleneck, leading to poor performance. We proposed Colander to learn confidence functions that are aligned with the TBAL objective. We evaluated our method extensively against common baselines on several real-world datasets and found that it improves the performance of TBAL significantly in comparison to the several common choices of confidence function. Our method is compatible with several choices of methods used for training the classifier in TBAL and using it in conjunction with them improves TBAL performance further. A limitation of Colander is that, similar to other post-hoc methods it also requires validation data to learn the confidence function. Reducing (or eliminating) this dependence on validation data could be an interesting future work.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: These details are provided in the Section 4 and Appendix C.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We run each setting with multiple random seeds and report the mean, standard deviations of the evaluation metrics.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: The paper has a brief discussion on the broader impacts. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper does not release such data or models that have high risk for misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have appropriately credited them along with citations. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.  
 
 Appendix: The supplementary material is organized as follows. We provide deferred details of the background and motivation section in Appendix A of the method in Appendix B. Then, in Appendix C, we provide additional experimental results and details of the experiment protocol and hyperparameters used for the experiments. Our code with instructions to run, is uploaded along with the paper.A.We run TBAL for a single round on the CIFAR-10 dataset with a SimpleCNN classification model with around 5.8M parameters [20] . We randomly sampled 4,000 points for training the classifier and randomly sampled 1,000 points as validation data. We train the model to zero training error using minibatch SGD with learning rate 1e-3, weight decay 1e-3 [15, 25] , momentum 0.9, and batch size 32. The trained model has validation accuracy around 55%, implying we could hope to get coverage around 55%. We run the auto-labeling procedure with an error tolerance of 5%.See Algorithms 1, 2 and 3.B.2 Tightness of surrogates.The surrogate auto-labeling error and coverage introduced to relax the optimization problem (P2) is indeed a good approximation of the actual auto-labeling error and coverage. To see this, we use a toy data setting of x ∼ Uniform(0, 1) with 1-dimensional threshold classifier h θ (x) = 1(x ≥ θ).For any x, let true labels y = h 0.5 (x) and consider the confidence function g w (x) = |w -x|. Let ŷ = h 0.25 (x) and consider the points on the side where ŷ = 1. We plot actual and surrogate errors in Figure 6 (a) and the surrogate and actual coverage in Figure 6 (a). for three choices of α. As expected, the gap between the surrogates and the actual functions diminishes as we increase the α.We employ the margin-random query approach to select the next batch of training data. This method involves sorting points based on their margin (uncertainty) scores and selecting the top Cn b points, from which n b points are randomly chosen. This strategy provides a straightforward and computationally efficient way to balance the exploration-exploitation trade-off. It's important to acknowledge the existence of alternative active-querying strategies; however, we adopt the margin-random approach as our standard to maintain a focus on evaluating various choices of confidence functions for auto-labeling. Note that while we use the new confidence scores computed using post-hoc methods for auto-labeling, we do not use these scores in active querying. Instead, we use the softmax scores from the model for this. We do this to avoid conflating the study with the study of active querying strategies. We use C = 2 for all experiments.The notation is summarized in Table 3 training data to learn ĥ(i) in the ith round.validation data in the ith round.calibration data in the ith round to learn a post-hoc g.part of validation data in the ith round to estimate threshold t.D (i) auto part of X (i)u that got auto-labeled in the ith round.Output labeled data, including auto-labeled and human labeled data.population level auto-labeling error, see eq. ( 2).population level auto-labeling coverage, see eq. ( 2). E(g, t | h, D) estimated auto-labeling error, see eq. ( 4). P(g, t | h, D) estimated auto-labeling coverage, see eq. ( 3). E(g, t | h, D) surrogate estimated auto-labeling error, see eq. ( 6). P(g, t | h, D) surrogate estimated auto-labeling coverage, see eq. ( 5).Table 3 : Glossary of variables and symbols used in this paper.Input: Unlabeled data X u , labeled validation data D val , auto labeling error tolerance ϵ a , N t training data query budget, seed data size n s , batch size for active query n b , calibration data fraction ν, space of thresholds T , coverage lower bound ρ 0 , label space Y. Output: Auto-labeled dataset D out .1: procedure TBAL(X u , D val , ϵ a , N t , n s , n b , ν, ρ 0 , T, Y) query ← RANDOMQUERY(X u , n s ). ▷ Randomly select n s points and get human labels. 4:X (1) u ← X u \ {x : (x, y) ∈ D (1) query }.▷Remove these points from the unlabeled pool. EQUATIONth ← RANDOMSPLIT(Dval , ν). ▷Randomly split current validation data. EQUATIONcal ). ▷ Colander .▷ /*** Estimate auto-labeling thresholds using ĝi and D th , ϵ a , ρ 0 , T, Y).▷ /*** Auto-label the points having scores above the thresholds. ***/ 18:D (i) u ← {(x, ĥi (x)) : x ∈ X (i) u }.19:D (i) auto ← {(x, ŷ) ∈ D(i) u : ĝi (x)[ ŷ] ≥ ti [ ŷ] }.20:X (i) u ← X (i) u \ {x : (x, ŷ) ∈ D (i)auto }. ▷Remove auto-labeled points from unlabeled set.21:D (i) val ← {(x, ĥi (x)) : (x, y) ∈ D (i) val }. 22: D (i+1) val ← {(x, ŷ) ∈ D(i) val : ĝi (x)[ŷ] < ti [ŷ]}.▷Remove validation points from the auto-labeling region. D (i+1) query ← ACTIVEQUERY( ĥi , X (i) u , n b ). 25: X (i+1) u ← X (i) u \ {x : (x, y) ∈ D (i+1) query }.▷Remove human-labeled data from the unlabeled pool. D out ← D out ∪ D (i) auto ∪ D (i+1)query . ▷Add the auto-labeled and manually labeled points in the output data. n (i+1) t ← n (i) t + n b . 28: i ← i + 1.D (i,y) th ← {(x ′ , y ′ ) ∈ D (i) th : y ′ = y}.▷Group points class-wise. u ← Softmax scores from ĥi for all points in X (i) u .3:X ′ ← Top C × n b points from X (i)u sorted in ascending order on the scores S if then|X| > n 3:X ′′ ← randomly select n points from X. X ′′ ← X. D ← {(x, human_label(x) : x ∈ X ′′ }.return D. 9: end procedureWe run TBAL with five values of ϵ a ∈ {0.01, 0.025, 0.05, 0.075, 0.1} and report the results in Table 6 . As expected the auto-labeling error is high with larger values of and smaller with small ϵ a .Post-hoc ϵa = 0.01 ϵa = 0.025 ϵa = 0.05 ϵa = 0.075 ϵa = 0. We further demonstrate that the performance gains are due to the use of Colander, even if methods use multiple rounds. To do so, we show the evolution of coverage and error over multiple rounds in Figure 16 . The effects of using Colander are visible from the first round itself, and the following rounds improve performance further. We also run a single round (passive) variant of TBAL where we sample all the human-labeled points for training (N t ) randomly at once, train a classifier, do auto-labeling, and then stop. This setting avoids confounding due to multiple rounds. We observe that using Colander yields significantly higher coverage in comparison to the baselines (see Table 7 ). This reinforces the fact that the gains in the multi-round TBAL are directly due to Colander, while multiple rounds of data selection, training, and auto-labeling are superior to this single round version.Post In TBAL it is not a priori clear what model the practitioner should use. The overall system is flexible enough to work with any chosen model class. Our focus is on evaluating the effect of various training time and post-hoc methods designed to improve the confidence functions for any given model. To answer the query, we ran experiments with Resnet18 and ViT models in the CIFAR-10 setting (see Table 8 ). As we expected there are variations in the results in the baselines due to model choices but our method maintains high performance irrespective of the classification model used. This is due to its ability to learn confidence scores tailored for TBAL. The hyperparameters and their values we swept over are listed in Table 9 and  Vanilla: Neural networks are commonly trained by minimizing the cross entropy loss using stochastic gradient descent (SGD) with momentum [1, 4] . We refer to this as the Vanilla training method. We also include weight decay to mitigate the overconfidence issue associated with this method [12] .1. Temperature scaling [12] : This is a variant of Platt scaling [12] , a classic and one of the easiest parametric methods for post-hoc calibration. It rescales the logits by a learnable scalar parameter and has been shown to work well for neural networks.2. Top-Label Histogram-Binning [14] : Since TBAL assigns the top labels (predicted labels) to the selected unlabeled points, it is appealing to only calibrate the scores of the predicted label. Building upon a rich line of histogram-binning methods (non-parametric) for post-hoc calibration [62] , this method focuses on calibrating the scores of predicted labels.3. Scaling-Binning [28] : This method combines parametric and non-parametric methods. It first applies temperature scaling and then bins the confidence function values to ensure calibration.Our experiments were conducted on machines equipped with the NVIDIA RTX A6000 and NVIDIA GeForce RTX 4090 GPUs. The wall clock time of our method is similar to other post-hoc methods.For instance, a single run on the CIFAR-10 setting on NVIDIA RTX A6000 takes around 1.5 hours with post-hoc methods and roughly 1 hour without post-hoc methods. The additional time taken by our method over the baselines not doing any post-hoc calibration is traded-off the by the quality and quantity of the auto-labeled data it outputs. We leave a thorough benchmarking of wall clock time and its optimization for future work.C.9 Detailed dataset and model 1. The MNIST dataset [30] consists of 28 × 28 grayscale images of hand-written digits across 10 classes. It was used alongside the LeNet5 [31] , a convolutional neural network, for auto-labeling.2. The CIFAR-10 dataset [24] contains 3 × 32 × 32 color images across 10 classes. We utilized its raw pixel matrix in conjunction with SimpleCNN [20] , a convolutional neural network with approximately 5.8M parameters, for auto-labeling.3. Tiny-ImageNet [29] is a color image dataset that consists of 100K images across 200 classes. Instead of using the 3 × 64 × 64 raw pixel matrices as input, we utilized CLIP [43] to derive embeddings within the R 512 vector space. We used a 3-layer perceptron (1,000-500-300) as the auto-labeling model.4. 20 Newsgroups [34, 40] is a natural language dataset comprising around 18,000 news posts across 20 topics. We used the FlagEmbedding [58] to map the textual data into R 1024 embeddings. We used a 3-layer perceptron (1,000-500-30) as the auto-labeling model.We predefined TBAL hyperparameters for each dataset-model pair and the hyperparameters we will sweep for each train-time and post-hoc method in Table 9 and Table 10 respectively. For a datasetmodel pair, initially, we perform a hyperparameter search for the train-time method. Subsequently, we optimize the hyperparameters for post-hoc methods while keeping the train-time method fixed with the previously found optimum hyperparameter for that dataset-model pair.We fix the hyperparameters for the train-time method while searching hyperparameters for the posthoc method to alleviate computational budget throttle. We effectively reduce the search space to the sum of the cardinalities of unique hyper-parameter combinations across the two methods instead of a larger multiplicative product. Furthermore, due to the independent nature of these hyper-parameter combinations, TBAL runs can be highly parallelized to expedite the search process.Since TBAL operates iteratively to acquire human labels for model training, selecting hyperparameters at each round of TBAL could quickly become intractable and lose its practical significance.To better align with its practical usage, we only conducted a hyperparameter search for the initial TBAL round. The specific set of hyperparameters used for the search are reported in Table 10 .After completing the hyperparameter search for train-time and post-hoc methods, the determined hyperparameter combinations are subjected to a full evaluation across all iterations of TBAL. At the end of each iteration, the auto-labeled points are evaluated against their ground truth labels to determine their auto-labeling error. These points are then added to the auto-labeled set, where their ratio to the total amount of unlabeled data determines the coverage. This iterative process continues until all unlabeled data are exhaustively labeled by either the oracle or through auto-labeling in the final iteration. The auto-labeling error and coverage at the final iteration of TBAL are then recorded.Since TBAL incorporates randomized components as detailed in Algorithm 1, we ran the algorithm 5 times, each with a unique random seed while maintaining the same hyperparameter combination. We then recorded the results from the final iteration of these runs and calculated the mean and standard deviation of both auto-labeling error and coverage. These figures are reported in Table 2 .A limitation of the grid search approach in hyper-parameter optimization becomes apparent when our predefined hyper-parameter choices result in sub-optimal coverage and auto-labeling errors. Using these sub-optimal hyper-parameters can adversely affect the multi-round iterative process in TBAL, prompting the need for repetitive searches to find more effective hyper-parameters. When encountering such scenarios, TBAL users should explore additional hyper-parameter options until satisfactory performance is achieved in the initial round. However, we opted for a more straightforward approach to hyper-parameter selection, mindful of the computational demands of repeatedly optimizing multiple hyper-parameters across different methods. In scenarios expressed conditionally, we retained the top-1 hyper-parameter combination for any given method if it achieved the highest coverage while adhering to the specified error margin (ϵ a ). If no hyper-parameter combinations yielded an auto-labeling error at most equal to the error margin (ϵ a ), we then chose the hyper-parameter combination with the lowest auto-labeling error, regardless of its coverage. In the case of ties, we resolved them through random selection. This process results in obtaining singular values for each choice of hyper-parameter after completing each method's hyper-parameter search.This paper contributes to the advancement of the practice of creating labeled datasets in machine learning. While our work has various possible societal implications, we do not identify any specific concerns that require special attention in this context. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.https://github.com/harit7/TBAL-Colander-NeurIPS-24 


Decision: The submission proposed a principled approach to optimize for confidence thresholds for TBAL, and showed its effectiveness empirically. Reviewers pointed out that there is no rigorous theoretical analysis or results, but generally agreed that the approach is principled and empirical results are sound. 

There were considerable discussions on the difference between TBAL and active learning, and I think the authors have done sufficient work in trying to clarify the differences between the two approaches, and added sufficient details to the submission. TBAL is a relatively new and interesting framework with some theoretical properties analyzed by previous work, e.g. [1]. I think this paper proposed an interesting approach to improve empirical TBAL performance.

[1] Vishwakarma et al., Promises and Pitfalls of Threshold-based Auto-labeling, NeurIPS 2023.

Reviews: review: paper_summary: paper_summary: This paper introduces a Colander for threshold-based auto-labeling (TBAL) pipeline. Colander trains a flexible confidence function using latent information of the classifier model, instead of fixed confidence functions. The optimization problem of Colander is based on the objective of TBAL, that is, maximizing coverage while minimizing auto-labeling errors. The authors consider differentiable surrogates for the 0-1 variables using the sigmoid function to make this problem tractable using gradient-based methods.
strength_weakness: strength_weakness: Contribution: Although this paper proposes the Colander for the TBAL pipeline, there is no theoretical analysis and understanding of mathematical properties.

Completeness:
- The paper includes unknown notation, which is used without pre-defined. 
- In Section 3.3, an undefined procedure exists: RandomQuery, RandomSplit, and ActiveQuery. In the case of Random method, it is possible to guess by readers, but in the case of ActiveQuery, a detailed explanation is needed.
- There are a few expressions that make it difficult to figure out the intent. Additionally, some keywords used in the paper need to be unified. 
   - What is the difference between the score (or scoring) function and the confidence function?
   - What is the difference between ‘inherent tension’ (in line 33) and trade-off?
   - What is the difference between post-hoc method and calibration method?
- The reviewer is unsure if Figure 1 is necessary for the reader: it contains acronyms that are not defined in advance, such as ERM, and certain formulas (h) are left undefined in the introduction.
- Line 174, line 186: The reader does not know which Appendix to look at because Appendix information is omitted.
- Appendix, Algorithm 2, line 8: The formula where C_1  weight is considered is not defined in the text.
- Appendix, Table 3: there are typos.
- This paper is considered to have no significant contributions compared to previous research about TBAL[1].
[1] Vishwakarma, Harit, et al. "Promises and pitfalls of threshold-based auto-labeling," Neural Information Processing Systems, 2023.
questions: questions: Active Learning Vs. TBAL
- Active learning and TBAL fundamentally play the role of labelers for exploiting unlabeled raw data. Lines 100-104 include these differences, but it is hard to get the point. Please explain these differences in detail.
- Although the paper includes a performance comparison with active learning (and self-training), an algorithmic setup of active learning is not provided. Please provide experimental settings for understanding whether it is fair or not.
- Why does the TBAL process require a human labeler at every iteration? Is it impossible to use labeled and unlabeled data completely separately in advance?

Colander (proposed solution)
- In Algorithm 1 (line 14), Colander extracts a confidence function  g ̂_i and t ̂_i^' through an optimization problem. As the reviewer’s knowledge, t ̂_i^' is different from  t ̂_i obtained estimation threshold process. Where is t ̂_i^' used?
- The optimization problem (P1,2,3) is solved using a given classifier. It means that this optimization process only works when the classifier's performance is guaranteed. Is it right and is this a correct assumption?
- The reviewer understood that the proposed solution configures the confidence function as a neural network, so it is flexible. But what is the meaning of 'choice of confidence function.' Does Colander learn multiple confidence functions and choose one of them?
- Why did the authors select exactly two layers of the classifier as inputs to the confidence function? There are various options: single layer, three layers, and inclusion of output information.
- Is it right to learn a new classifier/confidence function every round? If so, how long does the entire process require? If not, some expressions in the algorithm (pseudocode) and Section 3.3 need to be changed.

Post-hoc method
- Most baseline methods were proposed before 2020, except one algorithm. Please check following methods.
  -	R. Verma, et al. "Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles." AISTATS, 2023.
  -	L. Tao, et al. "Calibrating a deep neural network with its predecessors." IJCAI, 2023.
  -	T. Joy, et al. "Sample-dependent adaptive temperature scaling for improved calibration." AAAI, 2023.
  -	The reviewer would appreciate it if the authors could provide some insight/intuition into the problems with previous methods. The experimental sections just show empirical results and lack discussion.

Empirical results
- It is important to provide accurate settings when performing partial improvements in the overall pipeline. Have you checked the performance of various procedure method changes such as ActiveQuery?
- Are there any performance analysis results regarding changes in confidence scores?
- The experimental results clearly show that the algorithm has high performance in the provided experimental settings. What about time efficiency? Have you checked the wall-time?
- In the optimization problem (P3), how did the authors select the upper-bound error parameter ϵ? How do experimental results change depending on parameter changes?
- The value of C_1 in line 8 of Algorithm 2 seems to be fixed at 0.25. What results can see when it changes?
limitations: limitations: In this paper, there was not enough discussion about the limitations.
paper_summary: paper_summary: This paper addresses the challenges of overconfident scores in threshold-based auto-labeling (TBAL) systems. It critiques existing confidence scoring and calibration methods and introduces Colander, a new post-hoc method tailored to optimize TBAL performance.
strength_weakness: strength_weakness: -	The authors rightly mention limitations (“A limitation of Colander is that similar to other post-hoc methods it also requires validation data to learn the confidence function. Reducing (eliminating) this dependence on validation data could be an interesting future work.”) in the conclusion. So, does this mean that they use Gold data for validation? If that is the case, that might be the weakness of TBAL in general. The following paper talks about it in detail: https://aclanthology.org/2023.acl-long.796/
questions: questions: -	Not a question but authors can consider following related paper for comparison. It also works with confidence scoring: https://aclanthology.org/2021.naacl-main.84.pdf
limitations: limitations: Yes.
paper_summary: paper_summary: This paper discusses threshold-based auto-labeling functions aimed at identifying a large subset of unlabeled instances where the auto-labeling error remains below a specified threshold. The authors observed that standard temperature-scaled softmax scores are inadequate for effectively thresholding labeled instances. To address this, they propose learning a confidence scoring function and a threshold based on a subset of the validation data. This confidence function is a two-layer neural network trained on the logits from the last two layers of the base model. Extensive experiments were conducted on both image and text datasets using various training-time strategies to optimize the base model, demonstrating the efficacy of their approach.
strength_weakness: strength_weakness: 1. The whole TBAL procedure is a hybrid mixture of iterative training/self training and active learning. While the authors explained the differences between TBAL and self-training + active learning (ST+AL), the difference seems small (mostly, TBAL aims to identify a low-error auto-labelled dataset and ST+AL aims for a good classifier). These two goals do not seem fundamentally different, and can be easily translated to each other. Given the similarity of the framework, I feel it is still necessary to include some ST/AL works in the experiments. 
2. A lesser weakness is that due to iterative training, the proposed method and compared methods all went through rounds of data selection, which makes the comparison indirect. For example, it would be hard to tell the exact improvement on thresholding quality the method brings.
questions: questions: n/a
limitations: limitations: The authors acknowledge that the limitation of their work is the requirement of validation data.
paper_summary: paper_summary: This paper proposes a novel auto-labeling method, called Colander. In contrast to existing works, Colander models the objective of finding an optimal confidence function as a constrained optimization problem (the confidence function should have maximum coverage whilst obtaining a sufficiently low error rate; both components are controlled with a penalty term). Colander is evaluated and compared to baselines on four datasets covering vision and language tasks: MNIST, CIFAR-10, Tiny-ImageNet and 20 Newsgroups. The obtained results clearly indicate that the method improves upon existing baselines, both in terms of coverage and error rate.
strength_weakness: strength_weakness: * The paper focuses heavily on formally introducing Colander, and the experimental content is comparatively thin. To provide the reader with a better notion of robustness, it would for example have been useful to provide additional details on the hyperparameter search in the main paper, and how selecting those differently affects both the coverage and error rate. Likewise, it would have been useful to provide additional details on the impact of the introduced penalty term.
* It would also have been interesting to better understand the relationship between chosen model architecture for a dataset and auto-labeling performance.
* The paper scarcely discusses limitations in the conclusion section, yet I would have expected a more detailed discussion of where and how this approach is limited.
* Related to that, the paper does not touch upon future work / research questions that the obtained results create.
questions: questions: * Did you conduct additional experiments evaluating how different architectures affect auto-labeling performances for different datasets (i.e., how does changing model architectures for each of the four analyzed datasets affect auto-labeling performance)?
limitations: limitations: As discussed in the Weaknesses, the paper mentions a limitation in the conclusion section but I'd encourage the authors to further elaborate on how and where their approach is limited, and how such limitations can potentially be addressed in future work.
paper_summary: paper_summary: This paper introduces a threshold-based auto-labeling (TBAL) method called Colander to maximize TBAL performance by finding the optimal labling confidence function and thresholds. In order to find the optimal confidence function, Colander treats the auto-labeling objective as an optimization problem that maximizes the coverage under label error constraint. They use a neural network as the confidence function, and design a surgate for the optimization problem which can then be optimized using gradient-based methods.
strength_weakness: strength_weakness: W1: the paper is overall well-written, but the discussion of the thresholds is confusing or missing details. Specifically:
- In line 89, it says "the vector _**t**_ denotes scores over _k_ classes", and I suppose score means the predicted scores (or probabilities) of the _k_ class, but later _**t**_ is defined as a vector of thresholds.
- In line 144 (P1), it is unclear what _T^k_ stands for, and how the set of threshold, _T_, is determined. 
- In Algorithm 1, the Colander produces the estimated confidence function and thresholds hat _ti'_ (line 14), but the threshold is not used, and it relies on Algorithm 2 to estimate the threshold for each class. There is no discussion what the difference between these two thresholds.

W2: There is no discussion of the computation overhead of Colander.
questions: questions: Q1: How do you determine the threshold set of _T_? Is it a grid of _k_ classes * _n_ steps in [0,1]?

Q2: Why not use the thresholds produced by Colander (line 14 of Algorithm 1)?

Q3: My understanding is the that confidence function is searched by a neural network using gradient descent, and the thresholds are searched by iterating the training process over the threshold set T. But can we also make the thresholds a neural network (meaning that thresholds would be sample-specific)?
limitations: limitations: The computation cost of Colander increases linearly with the size of the set of confidence thresholds _T_, but it seems the process requires finer granularity in _T_ in order to get a better estimate of the optimal confidence function and thresholds.

Comments: title: Please be polite during the discussion
Thanks for the detailed response!
Thank You!
comment: Dear reviewers,

Thanks for responding to the authors' rebuttals! Please try and be polite during discussions. Everyone has a right to have their own opinions on the submission, and nobody should be pressured to change the review or rating. 

Thanks!
Thanks for the clarification on the thresholds and the additional experiments! My questions and concerns regarding the proposed method have been addressed and I decided to raise the presentation score.
Thank the authors for their hard work! After an additional explanation, some of my concerns were addressed. So, I have raised my score. 

Clarification regarding my score. 
+) This work is well-motivated. I believe the community would appreciate this work, which can be very practical. Additional results provided in the rebuttal period strengthen the claim. 
-) The manuscript is not ready to be published. Notations can be revised, and the figure can be better described. The text size in the figure is too tiny.  The critical limitation of this work is outdated baselines. The authors added a new baseline in the rebuttal period, but overall evaluations are quite limited. 

So, I think this work is around the borderline, but I lean to the reject side. In terms of presentation and evaluation, I strongly believe this manuscript is not ready to be published.
Thank you for providing such a detailed response and the additional results and insights. I will maintain my score and recommend acceptance as indicated in my initial review.
Thank you for your response. I enjoyed reading your paper and learned a lot. 

A few things regarding the reviewer YvdP. As I differ a lot in scores, I felt I should comment on a few of them. 
 - The presentation of the paper is really good. I do believe that Figure 1 was useful while I was reading the paper and two of my colleagues also agree with that. 
 - Many comments are unreasonable for example "There is no theoretical analysis and understanding of mathematical properties"

Good luck with the other reviewers!