Title: Promises and Pitfalls of Threshold-based Auto-labeling
Venue: NeurIPS 2023
Accepted: yes
Keywords: Auto Labeling

Active Learning

Selective Classification
Abstract: Creating large-scale high-quality labeled datasets is a major bottleneck in supervised machine learning workflows. Threshold-based auto-labeling (TBAL), where validation data obtained from humans is used to find a confidence threshold above which the data is machine-labeled, reduces reliance on manual annotation. TBAL is emerging as a widely-used solution in practice. Given the long shelf-life and diverse usage of the resulting datasets, understanding when the data obtained by such auto-labeling systems can be relied on is crucial. This is the first work to analyze TBAL systems and derive sample complexity bounds on the amount of human-labeled validation data required for guaranteeing the quality of machine-labeled data. Our results provide two crucial insights. First, reasonable chunks of unlabeled data can be automatically and accurately labeled by seemingly bad models. Second, a hidden downside of TBAL systems is potentially prohibitive validation data usage. Together, these insights describe the promise and pitfalls of using such systems. 
We validate our theoretical guarantees with extensive experiments on synthetic and real datasets.

Promises and Pitfalls of Threshold-based Auto-labeling

Abstract

Creating large-scale high-quality labeled datasets is a major bottleneck in supervised machine learning workflows. Threshold-based auto-labeling (TBAL), where validation data obtained from humans is used to find a confidence threshold above which the data is machine-labeled, reduces reliance on manual annotation. TBAL is emerging as a widely-used solution in practice. Given the long shelf-life and diverse usage of the resulting datasets, understanding when the data obtained by such auto-labeling systems can be relied on is crucial. This is the first work to analyze TBAL systems and derive sample complexity bounds on the amount of human-labeled validation data required for guaranteeing the quality of machine-labeled data. Our results provide two crucial insights. First, reasonable chunks of unlabeled data can be automatically and accurately labeled by seemingly bad models. Second, a hidden downside of TBAL systems is potentially prohibitive validation data usage. Together, these insights describe the promise and pitfalls of using such systems. We validate our theoretical guarantees with extensive experiments on synthetic and real datasets 1 .

1 Introduction

Machine learning (ML) models with millions or even billions of parameters are used to obtain state-of-the-art performance in many applications, e.g., object identification [53], machine translation [63], and fraud detection [71]. Such large-scale models require training on large-scale labeled datasets. As an outcome, the typical supervised ML workflow begins with the construction of a large-scale high quality dataset. Datasets with up to millions of labeled data points have played a pivotal role in the advancement of computer vision. However, collecting labeled data is an expensive and time consuming process. A common approach is to rely on the services of crowdsourcing platforms such as Amazon Mechanical Turk (AMT) to get groundtruth labels.

Even with crowdsourcing, obtaining labels for the entire dataset is expensive. To reduce costs, data labeling systems that partially rely on using a model's predictions as labels have been developed. Such systems date back to teacher-less training [21]. Modern examples include Amazon Sagemaker Ground Truth [58] and others [61, 55, 2, 64]. These approaches can be broadly termed auto-labeling .

Auto-labeling systems aim to label unlabeled data using predictions from ML models that are often trained on small amounts of human labeled data which can produce incorrect labels. The shelf life of datasets is longer than those of models, e.g., ImageNet continues to be a benchmark for many computer vision tasks [14] fifteen years after its initial development. As a result, to reliably train new models on auto-labeled datasets and deploy them, we need a thorough understanding of how reliable the datasets output by these auto-labeling systems are. Unfortunately, many widely

Threshold Based Auto Labeling System

Bootstrap

Select Points for

Labeling Region

Validation Data

(A)

(B)

(C)

(D)

used commercial auto-labeling systems [58, 55] are largely opaque with limited public information on their functionality. It is therefore unclear whether the quality of the datasets obtained can be trusted. To address this, we study the high level workflow of a popular threshold-based auto-labeling (TBAL) system (see Figure 1). We emphasize that our goal is to understand such systems and their performance-not to promote them as a superior alternative to other approaches. Our goal is:

Goal. Develop a fundamental understanding of TBAL systems. This is crucial since there is a lack of theoretical understanding of the reliability of these systems despite their wide adoption.

The TBAL systems we study (Figure 1) work iteratively. At a high level, in each iteration, the system trains a model on currently available human labeled data and decides to label certain parts of unlabeled data using the trained model by finding high-accuracy regions using validation data. It then collects human labels on a small portion of unlabeled data that is deemed helpful for training the current model in the next iteration. The validation data is created by sampling i.i.d. points from the unlabeled pool and querying human labels for them. In addition to training data, the validation data is a major driver of the cost and accuracy of auto-labeling and will be a key component in our study.

Our Contributions. We study TBAL systems (Figure 1) and make the following contributions:

Provide the first theoretical characterization of TBAL systems , developing tradeoffs between the quantity of manually labeled data and the quantity and quality of auto-labeled data (Section 3).

Empirical results validating our theoretical understanding on real and synthetic data (Section 4).

Our results reveal two important insights . Promisingly, even poor quality models are capable of reliably labeling at least some data when we have access to sufficient validation data and a good confidence function that can accurately quantify the confidence of a given model on any data point. On the downside, in certain scenarios, the quantity of the validation data required to reach a certain quantity and quality of auto-labeled data can be high.

2 Threshold-Based Auto-Labeling Algorithm

We begin with the problem setup and describe the TBAL algorithm that is inspired by the commercial systems [58]. Then we provide experiments and theoretical analysis shedding light on the pros and cons of TBAL. We emphasize that TBAL is not our proposal and our goal is to understand the effectiveness of such an auto-labeling system.

2.1 Problem Setup

Notation. Let the instance and label spaces be X and Y = { 1 , . . . , k } . We assume that there is some deterministic but unknown function f ∗ : X ↦→ Y that assigns true label y = f ∗ ( x ) to any x ∈ X . We also assume that there is a noiseless oracle O that can provide the true label y ∈ Y for any given x ∈ X . Let X pool ⊆ X denote a sufficiently large pool of unlabeled data to be labeled.

The goal of an auto-labeling algorithm is to produce accurate labels ˜ y i ∈ Y for points x i ∈ X pool while minimizing the number of queries to the oracle. Let [ m ] := { 1 , 2 , . . . , m } , A ⊆ [ N ] be the set of indices of auto-labeled points, and X pool ( A ) be these points. The auto-labeling error ̂ E ( X pool ( A ))

and the coverage ̂ P ( X pool ( A )) are defined as

̸

where N a denotes the size of auto-labeled set A . TBAL algorithm aims to auto-label the dataset so that ̂ E ( X pool ( A )) ≤ ϵ a while maximizing coverage ̂ P ( X pool ( A )) for any given ϵ a ∈ (0 , 1) .

Hypothesis Class and Confidence Function. A TBAL algorithm is given a fixed hypothesis space H and a confidence function g : H×X ↦→ T ⊆ R + that quantifies the confidence of h ∈ H on any data point x ∈ X . Confidence functions include prediction probabilities and margin scores. For example, when H is a set of unit-norm homogeneous linear classifiers, i.e. h w ( x ) = sign ( w T x ) with w ∈ { w ∈ R d : || w || 2 = 1 } , a reasonable confidence function is g ( h w , x ) = | w T x | .

Note that the target f ∗ might not be in the hypothesis space H . Our analysis (Section 3) shows that the TBAL algorithm can work well, i.e., accurately label a reasonable fraction of unlabeled data with simpler hypothesis classes that do not contain the target hypothesis f ∗ . We illustrate this with a simple example in Section 2.3 and Figure 2. Note as well that the features x could be raw features or representations from self-supervised techniques, pre-trained models, etc. We analyze TBAL in settings (i) with no assumptions on the features and (ii) when the features are linearly separable.

2.2 Description of the algorithm

The TBAL algorithm is given in Algorithm 1. It starts with an unlabeled pool X pool and an auto-labeling error threshold ϵ . For ease of exposition, the algorithm is given the labeled validation set D val of size N v separately. In practice, it is created by selecting points

Algorithm 1 Threshold-based Auto-Labeling (TBAL)

Input: Unlabeled pool X pool , auto labeling error threshold ϵ a , seed data size n s , batch size for active query n b , labeled validation data pool D val .

1: X (1) u = X pool ; D (1) val = D val

2: D (1) query = randomly_query_batch ( X (1) u , n s )

3: Remove queried points from X (1) u

̸

5: while X ( i ) u = ϕ do

6: D ( i ) train = D ( i -1) train ∪ D ( i ) query

7: ˆ h i = empirical_risk_min ( H , D ( i ) train )

8: ˆ t i = Estimate Threshold ( X ( i ) u , D ( i ) val , ϵ a , ˆ h i , n 0 )

10:

X

u

=

{

x

:

x

∈

X

u

, g

h

i

,

x

)

<

t

i

}

11: D ( i +1) val = { ( x , y ): ( x , y ) ∈ D ( i ) val , g ( ˆ h i , x ) < ˆ t i }

12: D ( i +1) query = active_query_batch ( ˆ h i , X ( i +1) u , n b )

13: Remove queried points from X ( i u

15: i = i +1

16: end while

Algorithm 2 Estimate Threshold

Input:

X ( i ) u , D ( i ) val , ϵ a , ˆ h i , n 0

Output: Threshold ˆ t i

1: T i = { g ( ˆ h i , x ) : ( x , y ) ∈ D ( i ) val }

2: for t ∈ T i do

3: N t = |{ x ∈ X ( i ) v : g ( ˆ h i , x ) ≥ t }|

4: end for

5: ˜ T i = { t : t ∈ T i , N t ≥ n 0 } ∪ {∞}

6:

ˆ

t

i

= min

{

t

∈

˜

T

i

:

̂

E

a

( ˆ

h

i

, t

|

X

(

i

)

v

) +

C

1

ˆ

σ

i

ϵ

a

}

at random from X pool . The algorithm starts with an initial batch of n s random data points and obtains oracle labels for these. The algorithm works in an iterative manner using the following steps.

Oracle labeled data obtained in each iteration i is added to the training pool D ( i ) train . It is used to train a model ˆ h i by performing empirical risk minimization (ERM).

Find the region where ˆ h i can auto-label accurately. The algorithm estimates a threshold ˆ t i on the confidence score above which it can auto-label with the desired auto-labeling accuracy on the validation data (see Algorithm 2). Thresholds that have too little validation data are discarded, since their estimates are unreliable. The minimum threshold is found such that the sum of the estimated error ̂ E a ( ˆ h i , t | X ( i ) v ) (see eq. (2)) and an upper confidence bound using the standard deviation of the estimated error, is at most the given auto-labeling error threshold.

Auto-label the points in the pool, X ( i ) u , which have confidence g ( ˆ h i , x ) > ˆ t i . These are added to the set D out and removed from the unlabeled pool. The validation points that fall in the auto-labeled region are also removed from the validation set so that in the next round the validation

+1)

TBAL Output

AL+SC Output

Queried +ve

Queried -ve

Auto-Labeled +ve

Auto-Labeled -ve

Unlabeled

200

400

600

800

1000

1200

0

20

40

60

80

100

Coverage (%)

Auto-Labeling Error (%)

20

40

60

80

100

200

400

600

800

1000

1200

0

2

4

TBAL

AL

AL+SC

PL+SC

PL

a

(a) TBAL and AL+SC on Circles dataset

(b) Auto labeling performance of various methods

set and the unlabeled pool are from the same region and the same distribution. Removing the auto-labeled points from X pool is a crucial step in the TBAL algorithm that enables it to focus only on the remaining unlabeled regions in the next iteration.

If there are points left in X pool, the algorithm selects points using some active querying strategy [57], obtains human labels for them, and adds them to the training pool. Note that the auto-labeled data is not added into the training set.

This process continues until there are no data points left to be labeled. The algorithm then outputs the labeled dataset, which is a mixture of human- and machine-labeled points.

2.3 Comparison between Auto-Labeling, Active Learning and Selective Classification

What is the difference between TBAL and methods such as active learning and selective classification?

Active learning. The goal of active learning [57] (AL) is to find the best model in hypothesis class H by training with less labeled data compared to passive learning. This is usually achieved by obtaining labels for the most informative points. Note that the end goal is to output a model from the function class whose predictions on new data as good as the best model in the function class could.

Selective Classification. The goal of selective classification (SC) [17] is to find the best combination of the hypothesis and selection functions to minimize error and maximize coverage of selection regions.

Auto-Labeling. The output of an auto-labeling procedure is a labeled dataset (not a model). When the hypothesis class is of lower complexity, it is often not possible to find a good classifier. The goal of an auto-labeling system is to label as much of the unlabeled data as accurately as possible with a given function class and with limited labeled data from humans.

Is active learning alone enough to auto-label data? AL has been found to be effective in reducing the number of labels needed to learn versus passive learning, particularly in low-noise cases [28]. Doing auto-labeling using AL followed by SC may be effective in such settings. However, in real-world scenarios, noise levels may be higher and the hypothesis class could be misspecified. In these instances, using the model learned through active learning to automatically label all data may result in a high number of errors.

We illustrate this difference between AL, SC, and auto-labeling through an example. Suppose the data consists of two concentric circles, one for each class, with the same number of points per class (Figure 2(a)). This setting is not linearly separable. We run TBAL, AL, and AL followed by SC with an error tolerance of ϵ a = 1% and linear classifiers and confidence functions. The results are shown in Figure 2. Note that the multiple optimal linear classifiers will all incur an error of 50% . AL algorithms can only output models that make at least 50% error. If we naively use the output model for auto-labeling, we can obtain near full coverage but incur around 50% auto-labeling error. If we use the model output by AL with threshold-based SC, labeling error is reduced. However, it can only label ≈ 25% of the unlabeled data. In contrast, TBAL can label almost all of the data accurately (close to 100% coverage) with less than 1% auto-labeling error.

3 Theoretical Analysis

The performance of the TBAL (Algorithm 1) depends on many factors including the hypothesis class, the accuracy of the confidence function, the data sampling strategy, and the size of the training and validation data. In particular, the amount of validation data plays a critical role in determining the accuracy of the confidence function, which in turn affects the accuracy and coverage.

We derive bounds on the auto-labeling error and the coverage for Algorithm 1 in terms of the size of the validation data, the number of auto-labeled points N ( k ) a , and the Rademacher complexity of the extended hypothesis class H T,g induced by the confidence function g . Our first result, Theorem (3.2), applies to general settings and makes no assumptions on the particular form of the hypothesis class, the data distribution, and the confidence function. We then instantiate and specialize the results for a specific setting in Section 3.1. We introduce some notation to aid in stating our results,

Definition 3.1. (Hypothesis Class with Abstain) The function g , set T and H induce an extended hypothesis class H T,g := H × T . Any function ( h, t ) ∈ H T,g is defined as ( h, t )( x ) = h ( x ) if g ( h, x ) ≥ t and ⊥ otherwise. Here ( h, t )( x ) = ⊥ means ( h, t ) abstains in classifying the point x .

̸

Error Definitions. Let S ⊆ X denote a non-empty sub-region of X and S ⊆ S be a finite set of i.i.d. samples from S . The subset S ( h, t ) ⊆ S denotes the regions where ( h, t ) does not abstain i.e. S ( h, t ) := { x ∈ S : ( h, t )( x ) = ⊥} , and the conditional probability mass associated with it is P ( h, t |S ) := P ( S ( h, t ) |S ) , and its empirical counterpart ̂ P ( h, t | S ) := | S ( h, t ) | / | S | . We use P ( S ) to denote the probability mass of set S and P ( S ′ |S ) for the conditional probability of subset S ′ ⊆ S given S . The population level and empirical auto-labeling errors are defined as follows:

̸

Here ℓ ⊥ 0 -1 ( h, t, x , y ) := ℓ 0 -1 ( h, x , y ) · ℓ ⊥ ( h, t, x ) with ℓ 0 -1 ( h, x , y ) := 1 ( h ( x ) = y ) , and ℓ ⊥ ( h, t, x ) := 1 ( g ( h, x ) ≥ t ) .

Rademacher Complexity. The Rademacher complexities for the function classes induced by the H , T, g and the loss functions are defined as R n ( H T,g ) := R n ( H , ℓ 0 -1 ) + R n ( H T,g , ℓ ⊥ ) . Let ˆ h i and ˆ t i be the ERM solution and the auto-labeling threshold at epoch i . Let p 0 ∈ (0 , 1) be a constant such that P ( ˆ h i , ˆ t i |X ( i ) ) ≥ p 0 for all i . Let X ( i ) v denote the validation set, and n ( i ) v and n ( i ) a the number of validation and auto-labeled points at epoch i . Let ̂ E a ( ˆ h i , ˆ t i | X ( i ) v ) be the empirical conditional risk of ˆ h i in the region where g ( ˆ h i , x ) ≥ ˆ t i evaluated on the validation data X ( i ) v .

We provide the following guarantees on the auto-labeling error and the coverage achieved by TBAL.

Theorem 3.2. (Overall Auto-Labeling Error and Coverage) Let k denote the number of rounds of the TBAL Algorithm 1. Let n ( i ) v , n ( i ) a denote the number of validation and auto-labeled points at epoch i and n ( i ) = | X ( i ) | . Let X pool ( A k ) be the set of auto-labeled points at the end of round k . N ( k ) a = ∑ k i =1 n ( i ) a be the total number of auto-labeled points. Then, for any δ ∈ (0 , 1) , with probability at least 1 -δ ,

Discussion. We interpret this result, starting with the auto-labeling error term ̂ E ( X pool ( A k )) . The term (a) ̂ E a ( ˆ h i , ˆ t i | X ( i ) v ) is the empirical conditional error in the auto-labeled region computed on the validation data in i -th round, which is at most ϵ a . Thus, summing term (a) over all the rounds is at most ϵ a . The term (b) provides an upper bound on the excess error over the empirical estimate term

(a) as a function of the Rademacher complexity of H T,g and the validation data used in each round. The last term (c) captures the variance in the overall estimate as a function of the total number of auto-labeled points and the Rademacher complexity of H T,g . If we let n ( i ) v ≥ n v i.e. the minimum validation points ensured in each round, then we can see the second term is O ( R n v ( H T,g )) and the third term is O ( √ 1 /n v ) .

Therefore, validation data of size O ( 1 /ϵ 2 a ) in each round is sufficient to get a O ( ϵ a ) bound on the excess auto-labeling error. The terms with Rademacher complexities suggest that it is better to use a hypothesis class and confidence function such that the induced hypothesis class has low Rademacher complexity. While such a hypothesis class might not be rich enough to include the target function, it would still be helpful for efficient and accurate auto-labeling of the dataset which can then be used for training richer models in the downstream task. The coverage term provides a lower bound on the empirical coverage ̂ P ( X pool ( A k )) in terms of the true coverage of the sequence of estimated hypotheses ˆ h i and threshold ˆ t i .

We note that the size of the validation data needed to guarantee the auto-labeling error in each round by Algorithm 1 is optimal up to log factors. This follows by applying a result on the tail probability of the sum of independent random variables due to Feller [20]:

Lemma 3.3. Let c 1 , c 2 and σ > 0 . Let x i ∈ X be a set of n i.i.d. points from X with corresponding true labels y i . Given ( h, t ) ∈ H T,g , let E [( ℓ ⊥ 0 -1 ( h, t, x i , y i ) -E ( h, t |X ) ) 2 ] = σ 2 i > σ 2 for every x i for σ i > 0 and let ∑ n i σ 2 i ≥ c 1 then for every ϵ ∈ [0 , ( ∑ n i =1 σ 2 i ) / √ c 1 ] with n v < 12 σ 2 log(4 c 2 ) /ϵ 2 , the following holds w.p. at least 1 / 4 , E a ( h, t |X ) > ̂ E a ( h, t | X ) + ϵ.

Therefore, if a sufficiently large validation set is not used in each round, there is a constant probability of erroneously deciding on a threshold for auto-labeling. Such a requirement on validation data also applies to active learning if we seek to validate the output model. Bypassing this requirement demands the use of approaches that are different from threshold-based auto-labeling and traditional validation techniques. We note the possibility of using recently proposed active testing techniques [33], a nascent approach to reducing validation data usage.

3.1 Linear Classifier Setting

Next, we consider a simple setting where active learning is known to be optimal to see if TBAL can offer similar performance guarantees. To do so, we instantiate results from 3.2 to homogeneous linear separators under the uniform distribution in the realizable setting. Let P x be supported on the unit ball in R d , X = { x ∈ R d : || x || ≤ 1 } . Let W = { w ∈ R d : || w || 2 = 1 } = S d , H = { x ↦→ sign ( ⟨ w , x ⟩ ) ∀ w ∈ W} , the score function be given by g ( h, x ) = g ( w , x ) = |⟨ w , x ⟩| , and set T = [0 , 1] . For simplicity, we will use W in place of H .

Corollary 3.4. (Overall Auto-Labeling Error and Coverage) Let ˆ w i , ˆ t i be the ERM solution and the auto-labeling threshold respectively at epoch i . Let n ( i ) v , n ( i ) a denote the number of validation and auto-labeled points at epoch i . Let the TBAL algorithm run for k -epochs. Then, for any δ ∈ (0 , 1) , w.p. at least 1 -δ ,

These results imply that by ensuring the sum of the empirical validation error term (a) and the upper confidence interval to be less than ϵ a in each round of the algorithm we can ensure that the overall auto-labeling error remains below ϵ a . Furthermore, by applying standard VC theory to the first round, we obtain that ˆ t 1 ≤ 1 / 2 . Therefore, right after the first round, we are guaranteed to label at least half of the unlabeled pool. We empirically observe that TBAL has coverage at par with active learning while respecting the auto-labeling error constraint (See Figure 4(a)).

Tightness of the Bounds. We study this in the setting of the Unit-Ball experiment. The upper bound on excess risk in this setting is given in Corollary 3.4 which is an instantiation of our general results to this specific setting. We consider a simplified form of the upper bound by ignoring the constants to get a sense of the rate in terms of the validation data size. We compute this simplified upper bound for different amounts of validation data. We compare these with the maximum auto-labeling error observed over 25 runs of auto-labeling in the Unit-Ball setting with different random seeds

0

1000

2000

3000

4000

No. of validation points

0.4

0.5

0.6

0.7

Auto-labeling upper bound

0

1000

2000

3000

4000

No. of validation points

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Max auto-labeling error

(Observed)

for each validation data size. The results are in Figure 3. As expected, we see that the worst-case error rate follows a similar rate as our upper bound but the upper bound is conservative. Next, we explain why this is the case.

Our upper bound is slightly conservative, as it is based on a uniform bound over all hypotheses in a given hypothesis class. Since the individual hypotheses whose excess auto-labeling error we need to bound are not known a priori we need to derive a bound on the number of validation samples using which we can guarantee that the excess auto-labeling error of any hypothesis (model) is small with high probability. Note that this is a conservative (worst-case) analysis to get an upper bound on the validation sample complexity. The upper bound has two parts: a) the Rademacher complexity of the hypothesis class and b) a term with the number of validation samples. We note that on the validation samples, it matches lower bounds order-wise (see Lemma 3.3). This is the first analysis to provide these bounds based on uniform convergence without making any assumptions about the data distributions or hypothesis class. We provide further discussion on the role of Rademacher complexity in the Appendix C.1.

4 Experiments

We study the effectiveness of TBAL on synthetic and real datasets. We validate our theoretical results and aim to understand the amount of labeled validation and training data required to achieve a certain auto-labeling error and coverage. We also seek to understand whether our findings apply to real data-where labels may be noisy-along with how TBAL performs compared to common baselines.

Baselines. We compare TBAL to the following methods:,

Passive Learning (PL) queries a subset of the points randomly to train a model from a given model class and then uses it to predict the labels for the remaining unlabeled pool.

Active Learning (AL) (using margin-random query strategy, described below) trains a model from a model class and uses it to predict the labels for the remaining unlabeled pool.

Passive Labeling + Selective Classification (PL+SC) first performs passive learning to train a model from a given model class. Then it performs auto-labeling on the unlabeled data using threshold-based selective classification with the model output by passive learning. Only those unlabeled points that are deemed as fit to be labeled by the selection function are auto-labeled.

Active Learning + Selective Classification (AL+SC) first performs active learning (using margin-random query strategy) to train a model from a given model class. It then performs auto-labeling using threshold-based selective classification with the model output by AL.

For selective classification in the above methods, we use Algorithm 2 to estimate the threshold and use it to perform auto-labeling. In experiments, we modify Algorithm 2 slightly-instead of estimating a single threshold for all classes, we estimate thresholds for each class separately.

Active Querying Strategy. We use the margin-random query strategy for querying the next batch of training data. In this strategy, the algorithm first sorts the points based on the margin (uncertainty) score and then selects the top Cn b ( C > 1 ) points from which n b points are picked at random. This is a simple and computationally efficient method that balances the exploration and exploitation trade-off. We note that other active-querying strategies exist; we use margin-random as our standard querying strategy to keep the focus on comparing auto-labeling-not active learning approaches.

N

v

100

400

800

1200

1600

2000

Error (%)

TBAL

AL+SC

3.10

±

1.80

1.97

1.64

1.39

1.33

1.28

±

±

±

±

±

0.76

0.50

0.39

0.30

0.34

0.68

±

0.81

0.59

0.66

0.67

0.70

0.71

±

±

±

±

±

0.18

0.19

0.19

0.19

0.21

Coverage (%)

TBAL

AL+SC

71.43

±

8.86

93.99

96.26

96.67

97.13

97.15

±

±

±

±

±

2.39

1.33

0.84

0.45

0.54

96.95

±

1.01

97.89

98.06

98.10

98.16

98.20

±

0.50

±

0.53

±

0.45

±

0.44

±

0.34

Datasets. We use the following synthetic and real datasets. We also provide empirical results on MNIST and another synthetic dataset in the Appendix. For each dataset, we split the data into two sufficiently large pools. One is used as X pool on which auto-labeling algorithms are run and the other is used as X val from which the algorithms subsample validation data.

Unit-Ball is a synthetic dataset of uniformly sampled points from the d -dimensional unit ball. The true labels are generated using a homogeneous linear separator with w = [1 / √ d, . . . , 1 / √ d ] . We use d = 30 and generate N = 20 K samples, out of which 16 K are in X pool and 4 K are in X val . The dataset has just two classes but there is no margin between them.

Tiny-ImageNet [1] is a subset of the larger ImageNet [14] dataset, designed for image classification tasks. It consists of 200 classes , each with 500 training images and 50 validation and test images. With a total of 100 K images, Tiny ImageNet provides a diverse and challenging dataset. We use pre-computed embeddings of the images using CLIP [50].

IMDB Reviews [41] is a comprehensive collection of movie reviews, consisting of 50 K individual reviews. It is a balanced dataset of positive and negative labels. We use the standard train set of size 25 K and split it into X pool and X val of sizes 20 K and 5 K respectively. We compute embeddings of reviews using a pre-trained model bge-large-en [70] from the Massive Text Embedding Benchmark (MTEB) [44, 19].

CIFAR-10 [36] is an image dataset with 10 classes. We randomly split the standard training set into X pool of size 40 K and the validation pool of size 10 K . We use the raw features for training.

Models and Training. For the linear models, we use SVM with the usual hinge loss and train it to loss tolerance 10 -5 . To train a multi-layer perceptron (MLP) on the pre-computed embeddings of IMDB and Tiny-ImageNet we use SGD with a learning rate of 0 . 05 , 0 . 1 respectively, and batch size of 64. To train the medium CNN we use SGD with a learning rate of 10 -2 , batch size 256, and momentum of 0.9. More details on model training are in the Appendix.

The score function g . For SVMs we use the standard implementations of [69, 46] in sklearn to get the prediction probabilities and use them as the score function. Neural networks use softmax output.

4.1 Role of Validation Data

The TBAL algorithm uses validation data to estimate the auto-labeling errors at various thresholds to determine the threshold for automatically labeling points accurately. Thus, it is crucial to have

200

400

600

800

1000

0

2

4

6

Auto-Labeling Error (%)

200

400

600

800

1000

0

25

50

75

100

Coverage (%)

TBAL

AL+SC

AL

PL+SC

PL

a

10K

15K

20K

25K

30K

35K

0

20

40

60

80

Auto-Labeling Error (%)

10K

15K

20K

25K

30K

35K

0

20

40

60

80

Coverage (%)

TBAL

AL+SC

AL

PL+SC

PL

a

(a) Unit-Ball: varying training samples size, validation

4

samples size=

K

,

ϵ

a

= 1%

, C

1

= 0

.

25

.

(b) CIFAR-10: varying training samples size,validation size= 10 K , ϵ a = 10% , C 1 = 0 . 25 .

accurate estimates of the auto-labeling errors. Our analysis shows that to get such good estimates, large amounts of validation data are needed. In this section, we study the effect of varying the amount of validation data on auto-labeling performance.

Setup. We fix the maximum training data size N q and run the algorithm with different amounts of validation data. We also consider the two cases where the algorithm uses an upper confidence bound on the error estimate and where it does not. We use the Unit-Ball, IMDB, Tiny-ImageNet, and datasets for this study with N q = 500 , 500 , and 10 K respectively, and the auto-labeling error thresholds ϵ a = 1%, 5%, 10%, respectively. Initial seed data of size n s is 20% of N q and query batch size n b is 5% of N q ; C = 2 for both AL and TBAL for both datasets. We give the same initial seed samples of size n s to all the methods to ensure they have the same starting point.

Results. Tables 1,2 and 3 demonstrate the impact of validation data on the performance of TBAL and other algorithms. The auto-labeling error and coverage of TBAL and other methods are affected by the amount of validation data provided. When the validation data is insufficient, the auto-labeling error of TBAL increases. However, as more validation data is used, the auto-labeling error and coverage of TBAL improves. Providing too little X val can lead to incorrect estimates of the auto-labeling error, which in turn results in poor auto-labeling performance. This is further highlighted in our theoretical analysis as seen in Theorem 3.2. We also take a more nuanced look at the performance when the algorithm uses an upper confidence bound (with C 1 = 0 . 25 ) on the estimates and when it does not. We see the effects of not using any upper confidence bound (i.e. C 1 = 0 ). The left side tables in Tables 1,2 and 3 show the results when C 1 = 0 and the right side tables show the results when C 1 = 0 . 25 . These show that not using UCB leads to high auto-labeling error (i.e. not meeting the guarantees) even when there is a sufficient amount of validation data. This can happen with high coverage as well-yielding a dataset with large errors. On the other hand using UCB, i.e. C 1 = 0 . 25 , the algorithm can keep the auto-labeling error below the given threshold but suffers in coverage.

In the above Tables, we omitted AL, PL, and PL+SC for clarity. We provide full results with all baselines in Appendix D.3.

4.2 Role of Training Data Size

The labels queried for model training also play an important role in the process while incurring costs to obtain. The next experiment focuses on the impact of training data on auto-labeling.

Setup. We limit the amount of training data the algorithm can use and record the resulting auto-labeling error and coverage. We ensure all algorithms have sufficiently large but equal amounts of validation data. We run on Unit-Ball, IMDB, Tiny-Imagenet, and CIFAR-10 datasets with the same values of n s , n b , and C as in previous experiments.

Results. Figures 4(a), 4(b), and 2(b) indicate that TBAL and methods utilizing selective classification (AL+SC, PL+SC) maintain a high level of accuracy, even in scenarios where minimal training samples are used. This is expected as the threshold estimation method (when used with sufficient validation data) will find auto-labeling thresholds such that the auto-labeling error does not exceed ϵ a . The impact of training data size can be seen clearly in the coverage achieved by the algorithms. As expected, with fewer training samples the model has low accuracy leading to low coverage. However, as more samples are acquired, a more accurate model within the function class is learned, resulting in increased coverage. The Appendix D has additional discussion and results.

5 Related Work

We briefly review related work, deferring a more detailed discussion to the Appendix A.

There is a rich body of work on active learning (AL) [57, 12, 30, 28, 8] focused on learning the best model in a function class with less labeled data than passive learning. Various AL algorithms have been developed and analyzed, e.g., uncertainty sampling [62, 45], disagreement region based [9, 27], margin based [4] and abstention based methods that minimize the Chow's excess risk [72].

Selective classification (SC) equips a given classifier with the option to abstain from prediction in order to guarantee prediction quality. The foundations for SC are laid down in [17, 67, 18, 68] where results on the error rate in the prediction region and the coverage of the given classifier are provided. However, these works lack practical algorithms to find the prediction region. A recent work [26] gives a disagreement-based active learning strategy to learn a selective classifier.

A recent paper [49] studies a TBAL-like algorithm for auto-labeling. It focuses on the cost of training incurred when these systems use large-scale model classes for auto-labeling. It proposes an algorithm to predict the training set size that minimizes the overall cost and provides an empirical evaluation.

Weak supervision is another line of work aimed at auto-labeling that does not rely on obtaining human labels but instead uses potentially noisy but cheaply available sources to infer labels [51, 22]. In contrast, we are focused specifically on analyzing the performance of TBAL algorithms [58].

6 Conclusion and Future Work

In this work, we analyzed threshold-based auto-labeling systems and derived sample complexity bounds on the amount of human-labeled validation data required to guarantee the quality of machine-labeled data. Our study shows that these methods can accurately label a reasonable size of data using seemingly bad models when good confidence functions are available. Our analysis points to the hidden downside of these systems in terms of a large amount of validation data usage and calls for more sample-efficient methods including active testing.

7 Acknowledgments

This work was partly supported by funding from the American Family Data Science Institute. We thank Stephen Mussmann, Changho Shin, Albert Ge, Yi Chen, Kendall Park, and Nick Roberts for their valuable inputs. We thank the anonymous reviewers for their valuable comments and constructive feedback on our work.

References

Tiny imagenet dataset. http://cs231n.stanford.edu/tiny-imagenet-200.zip . Accessed: May 16, 2023.

Airbus. Airbus active labeling blog. https://acubed.airbus.com/blog/wayfinder/ automatic-data-labeling-strategies-for-vision-based-machine-learning-and-ai/ , 2022. Accessed: 2022-11-18.

Maria-Florina Balcan, Andrei Z. Broder, and Tong Zhang. Margin based active learning. In COLT , 2007.

Maria-Florina Balcan and Phil Long. Active and passive learning of linear separators under log-concave distributions. In Conference on Learning Theory , pages 288-316. PMLR, 2013.

Alina Beygelzimer, Sanjoy Dasgupta, and John Langford. Importance weighted active learning. In Proceedings of the 26th annual international conference on machine learning , pages 49-56, 2009.

Kamalika Chaudhuri, Sham M. Kakade, Praneeth Netrapalli, and Sujay Sanghavi. Convergence rates of active learning for maximum likelihood estimation. In Proceedings of the 28th International Conference on Neural Information Processing Systems , 2015.

C Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory , 16(1):41-46, 1970.

Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Rostamizadeh, and Sanjiv Kumar. Batch active learning at scale. In Advances in Neural Information Processing Systems , volume 34, pages 11933-11944, 2021.

David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine Learning , 15(2):201-221, 1994.

Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In International Conference on Algorithmic Learning Theory , pages 67-82. Springer, 2016.

Sanjoy Dasgupta. Coarse sample complexity bounds for active learning. In Y. Weiss, B. Schölkopf, and J. Platt, editors, Advances in Neural Information Processing Systems , volume 18. MIT Press, 2006.

Sanjoy Dasgupta. Two faces of active learning. Theoretical Computer Science , 412(19):1767-1781, 2011. Algorithmic Learning Theory (ALT 2009).

Sanjoy Dasgupta, Adam Tauman Kalai, and Claire Monteleoni. Analysis of perceptron-based active learning. In International conference on computational learning theory , pages 249-263. Springer, 2005.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Giulia DeSalvo, Mehryar Mohri, and Umar Syed. Learning with deep cascades. In Proceedings of the Twenty-Sixth International Conference on Algorithmic Learning Theory (ALT 2015) , 2015.

Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. JMLR , 11:1605-1641, aug 2010.

Ran El-Yaniv and Yair Wiener. Active learning via perfect selective classification. Journal of Machine Learning Research , 13(2), 2012.

Hugging Face. Hugging face mteb leaderboard. https://huggingface.co/spaces/mteb/ leaderboard , 2023. Accessed: 2023-08-07.

William Feller. Generalization of a probability limit theorem of cramér. In Transactions of the American Mathematical Society , pages 361-372' 1943.

S. Fralick. Learning to recognize patterns without a teacher. IEEE Transactions on Information Theory , 13(1):57-64, 1967.

Daniel Y. Fu, Mayee F. Chen, Frederic Sala, Sarah M. Hooper, Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020) , 2020.

Olivier Gascuel and Gilles Caraux. Distribution-free performance bounds with the resubstitution error estimate. Pattern Recognition Letters , 13(11):757-764, 1992.

Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. A survey of uncertainty in deep neural networks. arXiv preprint arXiv:2107.03342 , 2021.

Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems , volume 30, 2017.

Roei Gelbhart and Ran El-Yaniv. The relationship between agnostic selective classification, active learning and the disagreement coefficient. The Journal of Machine Learning Research , 20(1):1136-1173, 2019.

Steve Hanneke. A bound on the label complexity of agnostic active learning. ICML, 2007.

Steve Hanneke. Theory of disagreement-based active learning. Found. Trends Mach. Learn. , 7(2-3):131-309, jun 2014.

Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. 2018.

Daniel Joseph Hsu. Algorithms for active learning . PhD thesis, UC San Diego, 2010.

Matti Kääriäinen. Active learning in the non-realizable case. In International Conference on Algorithmic Learning Theory , pages 63-77. Springer, 2006.

Julian Katz-Samuels, Jifan Zhang, Lalit Jain, and Kevin Jamieson. Improved algorithms for agnostic pool-based active classification. In International Conference on Machine Learning , pages 5334-5344. PMLR, 2021.

Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample-efficient model evaluation. International Conference on Machine Learning , 2021.

Akshay Krishnamurthy, Alekh Agarwal, Tzu-Kuo Huang, Hal Daumé III, and John Langford. Active learning for cost-sensitive classification. In International Conference on Machine Learning , pages 1915-1924. PMLR, 2017.

Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus uncertainty optimization. Advances in Neural Information Processing Systems , 33:18237-18248, 2020.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy-based learning. Predicting structured data , 1(0), 2006.

Michel Ledoux and Michel Talagrand. Probability in banach spaces. 1991.

Chunwei Ma, Ziyun Huang, Jiayi Xian, Mingchen Gao, and Jinhui Xu. Improving uncertainty calibration of deep neural networks via truth discovery and geometric optimization. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence , volume 161 of Proceedings of Machine Learning Research , pages 75-85. PMLR, 27-30 Jul 2021.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.

Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In Advances in Neural Information Processing Systems , volume 34, pages 15682-15694, 2021.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning . The MIT Press, 2012.

Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316 , 2022.

Stephen Mussmann and Percy Liang. On the relationship between data efficiency and error for uncertainty sampling. In Proceedings of the 35th International Conference on Machine Learning, ICML , 2018.

John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers , 10(3):61-74, 1999.

Nikita Puchkin and Nikita Zhivotovskiy. Exponential savings in agnostic active learning through abstention. Proceedings of Machine Learning Research, pages 3806-3832. PMLR, 2021.

PyTorch. Cifar-10 pytorch tutorial. https://pytorch.org/tutorials/beginner/blitz/ cifar10_tutorial.html , 2022. Accessed: 2022-11-18.

Hang Qiu, Krishna Chintalapudi, and Ramesh Govindan. Minimum cost active labeling. arXiv preprint arXiv:2006.13999 , 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.

A. J. Ratner, Christopher M. De Sa, Sen Wu, Daniel Selsam, and C. Ré. Data programming: Creating large training sets, quickly. In Proceedings of the 29th Conference on Neural Information Processing Systems (NIPS 2016) , Barcelona, Spain, 2016.

Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the 44th International Conference on Very Large Data Bases (VLDB) , Rio de Janeiro, Brazil, 2018.

Joseph Redmon and Ali Farhadi. YOLO9000: better, faster, stronger. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6517-6525, 2017.

Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B. Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning, 2020.

Samsung-SDS. Samsung sds auto-labeling service. https://www.samsungsds.com/en/ insights/\TechToolkit_2021_Auto_Labeling.html , 2022. Accessed: 2022-11-18.

Nabeel Seedat and Christopher Kanan. Towards calibrated and scalable uncertainty representations for neural networks. arXiv preprint arXiv:1911.00104 , 2019.

Burr Settles. Active learning literature survey. 2009.

SGT. Aws sagemaker ground truth. https://aws.amazon.com/sagemaker/ data-labeling/ , 2022. Accessed: 2022-11-18.

Kulin Shah and Naresh Manwani. Online active learning of reject option classifiers. Proceedings of the AAAI Conference on Artificial Intelligence , 34(04):5652-5659, 2020.

Shubhanshu Shekhar, Mohammad Ghavamzadeh, and Tara Javidi. Active learning for classification with abstention. IEEE Journal on Selected Areas in Information Theory , 2(2):705-719, 2021.

Superb-AI. Superb ai automated data labeling service. https://www.superb-ai.com/ product/automate , 2022. Accessed: 2022-11-18.

Simon Tong and Daphne Koller. Support vector machine active learning with applications to text classification. Journal of machine learning research , 2(Nov):45-66, 2001.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 30th Conference on Neural Information Processing Systems (NIPS 2017) , 2017.

Rudy Venguswamy, Mike Levy, Anirudh Koul, Satyarth Praveen, Tarun Narayanan, Ajay Krishnan, Jenessa Peterson, Siddha Ganju, and Meher Kasam. Curator: A No-Code Self-Supervised Learning and Active Labeling Tool to Create Labeled Image Datasets from Petabyte-Scale Imagery. In EGU General Assembly Conference Abstracts , pages EGU21-6853, April 2021.

Deng-Bao Wang, Lei Feng, and Min-Ling Zhang. Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. In Advances in Neural Information Processing Systems , volume 34, pages 11809-11820, 2021.

Yingfan Wang, Haiyang Huang, Cynthia Rudin, and Yaron Shaposhnik. Understanding how dimension reduction tools work: An empirical approach to deciphering t-sne, umap, trimap, and pacmap for data visualization. Journal of Machine Learning Research , 22(201):1-73, 2021.

Yair Wiener and Ran El-Yaniv. Agnostic selective classification. Advances in neural information processing systems , 24, 2011.

Yair Wiener and Ran El-Yaniv. Agnostic pointwise-competitive selective classification. Journal of Artificial Intelligence Research , 52:171-201, 2015.

Ting-Fan Wu, Chih-Jen Lin, and Ruby Weng. Probability estimates for multi-class classification by pairwise coupling. Advances in Neural Information Processing Systems , 16, 2003.

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.

Yufan Zeng and Jiashan Tang. Rlc-gnn: An improved deep architecture for spatial-based graph neural network with application to fraud detection. Applied Sciences , 11, 06 2021.

Yinglun Zhu and Robert Nowak. Efficient active learning with abstention. arXiv preprint arXiv:2204.00043 , 2022.

Supplementary Material

The supplementary material is organized as follows. We provide a detailed discussion on related works in Section A. We give details of the definitions and notation in Section B.1. The notations are also summarized in the Table 5 in section B. Then we give the proof of the main theorem (Theorem 3.2) followed by proofs of supporting lemmas in section C. We provide details of its instantiation for finite VC-dimension hypothesis classes and the homogeneous linear separators case in Section C.3. Then, we provide the technical details of the lower bound (Lemma 3.3). Then we provide details of additional experiments in Section D. In Section D.4 we provide insights into auto-labeling using PaCMAP [66] visualizations of auto-labeled regions in each round.

A Extended Related Work

There is a rich body of work on active learning on empirical and theoretical fronts [57, 12, 30, 28, 8, 54]. In active learning, the goal is to learn the best model in the given function class with fewer labeled data than in classical passive learning. To this end, various active learning algorithms have been developed and analyzed, e.g., uncertainty sampling [62, 45], disagreement region based [9, 27], margin based [3, 4], importance sampling based [5] and others [6]. Active learning has been shown to achieve exponentially smaller label complexity than passive learning in noiseless and low-noise settings [13, 3, 27, 28, 4, 11, 30, 6, 34, 32]. This suggests, in these settings auto-labeling using active learning followed by selective classification is expected to work well. However, in practice we do not have favorable noise conditions and the hypothesis class could be misspecified i.e. it may not contain the Bayes optimal classifier. In such cases, [31] proved lower bounds on the label complexity of active learning that are order wise same as passive learning. These findings have motivated more refined goals for active learning - abstain on hard to classify points and do well on the rest of the points. This idea is captured by the Chow's excess risk [7] and some of the recent works [60, 59, 47, 72] have proved exponential savings in label complexity for active learning when the goal is to minimize Chow's excess risk. The classifier learned by these methods is equipped with the abstain option and hence it can be readily applied for auto-labeling. However, the problem of misspecification of the hypothesis class still remains. Nevertheless, it would be interesting future work to explore the connections between auto-labeling and active learning with abstention. We also note that similar works on learning with abstention are done in the context of passive learning [10].

Another closely related line of work is the selective classification where the goal is to equip a given classifier with the option to abstain from the prediction in order to guarantee prediction quality. The foundations for selective classification are laid down in [17, 67, 18, 68] where they give results on the error rate in the prediction region and the coverage of a given classifier. However, they lack practical algorithms to find the prediction region. A recent work [26] proposes a new disagreement-based active learning strategy to learn a selective classifier.

Recent work studies a practical algorithm for threshold-based selective classification on deep neural networks [25]. The algorithm estimates the prediction threshold using training samples and they bound the error rate of the selective classifier using [23]. We note that their result is applicable to a specific setting of a given classifier. In contrast, in the TBAL algorithm analyzed in this paper, selective classification is done in each round and the classifiers are not given a priori but instead learned via ERM on training data which is adaptively sampled in each round.

Another related work [49] studies an algorithm similar to TBAL for auto-labeling. Their emphasis is on the cost of training incurred when these systems use large-scale model classes for auto-labeling. They propose an algorithm to predict the training set size that minimizes the overall cost and provides an empirical evaluation.

Well-calibrated uncertainty scores are essential to the success of threshold-based auto-labeling. However, in practice, such scores are often hard to get. Moreover, neural networks can produce overconfident ( unreliable) scores [29]. Fortunately, there are plenty of methods in the literature to deal with this problem [46, 69]. More recently, various approaches have been proposed for uncertainty calibration for neural networks [24, 42, 65, 35, 40, 56]. A detailed study of calibration methods and their impact on auto-labeling is beyond the scope of this work and left as future work.

There is another line of work emerging towards auto-labeling that does not rely on getting human labels but instead uses potentially noisy but cheaply available sources to infer labels [51, 52, 22]. The focus of this paper, however, is on analyzing the performance of TBAL algorithms [58, 2] that have emerged recently as auto-labeling solutions in systems.

B Definitions and Notation

B.1 Basic Definitions

Let X be the instance space and p ( x ) be a density function supported on X . For any x i ∈ X let y i be its true label. Let X = { x 1 , . . . , x N } be a set of N i.i.d samples drawn from X . Let set S ⊆ X denote a non-empty sub-region of X and S ⊆ X ∩ S be a set of n > 0 i.i.d. samples.

Definition B.1. (Hypothesis Class with Abstain) We can think of the function g along with set T as inducing an extended hypothesis class H ( T,g ) . Let H T,g = H× T . For any function ( h, t ) ∈ H ( T,g ) is defined as

Here ( h, t )( x ) = ⊥ means the hypothesis ( h, t ) abstains in classifying the point x . Otherwise, it is equal to h ( x ) .

The subset S ( h, t ) ⊆ S where ( h, t ) does not abstain and its complement ¯ S ( h, t ) where ( h, t ) abstains, are defined as follows,

̸

Probability Definitions. The probability P ( S ) of subset S ⊆ X and the conditional probability of any subset S ′ ⊆ S are given as follows,

The empirical probabilities of S and S ′ ⊆ S are defined as follows,

Loss Functions. The loss functions are defined as follows,

̸

Error Definitions. Define the conditional error in set S ⊆ X as follows,

Then, the conditional error in set S ( h, t ) i.e. the subset of S on which ( h, t ) does not abstain,

Similarly, define their empirical counterparts as follows,

Note that,

̸

Rademacher Complexity. The Rademacher complexities for the function classes induced by the H , T, g and the loss functions are defined as follows,

B.2 Glossary

The notation is summarized in Table 5 below. More detailed notation is in section B.1.

C Proofs

C.1 Proofs for the General Setup

We begin by restating the theorem here and then give the proof.

Theorem 3.2. (Overall Auto-Labeling Error and Coverage) Let k denote the number of rounds of the TBAL Algorithm 1. Let n ( i ) v , n ( i ) a denote the number of validation and auto-labeled points at epoch i and n ( i ) = | X ( i ) | . Let X pool ( A k ) be the set of auto-labeled points at the end of round k . N ( k ) a = ∑ k i =1 n ( i ) a denote the total number of auto-labeled points. Then, for any δ ∈ (0 , 1) , with probability at least 1 -δ ,

Proof. Recall the definition of auto-labeling error,

Here, m ( i ) a is the number of auto-labeling mistakes made by the Algorithm in the i th round and ̂ E a ( ˆ h i , ˆ t i | X ( i ) ) is the auto-labeling error in that round. Note that we cannot observe these quantities since the true labels for the auto-labeled points are not available. To estimate the auto-labeling error of each round we make use of validation data. Using the validation data we first get an upper bound on the true error rate of the auto-labeling region i.e. E a ( ˆ h i , ˆ t i |X ( i ) ) in terms of the auto-labeling error on the validation data ̂ E a ( ˆ h i , ˆ t i | X ( i ) v ) and then get an upper bound on empirical auto-labeling error rate ̂ E a ( ˆ h i , ˆ t i | X ( i ) ) using the true error rate of the auto-labeling region.

We get these bounds by application of Lemma C.1 with δ 3 = δ/ 4 k for each round and then apply union bound over all k epochs. Note that we have to apply the lemma twice, first to get the concentration bound w.r.t the validation data and second to get the concentration w.r.t to the auto-labeled points.

Substituting E a ( ˆ h i , ˆ t i |X ( i ) ) by its upper confidence bound on the validation data.

Having an upper bound on the empirical auto-labeling error for i th round gives us an upper bound on the number of auto-labeling mistakes m ( i ) a made in that round. It allows us to upper bound the total auto-labeling mistakes in all k rounds and thus the overall auto-labeling error as detailed below,

Since we have an upper bound on the empirical auto-labeling error in each round, we have an upper bound for each m ( i ) a , which are used as follows to get the bound on the auto-labeling error,

The last term is simplified as follows,

The last inequality follows from the application of the inequality || u || 1 ≤ √ k || u || 2 for any vector u ∈ R k . Here we let u = [ √ n (1) a , . . . , √ n ( k ) a ] , and since ∀ i √ n ( i ) a > 0 so, ∑ k i =1 √ n a ( i ) = || u || 1

To get the bound on coverage we follow the same steps except that we can use all the unlabeled pool of size n ( i ) to estimate the coverage in each round which gives us the bound in terms of n ( i ) and N as follows,

We bound the first term as follows,

Substituting it back we get,

For the last step we use the inequality √ a + √ b ≤ √ 2( a + b ) for any a, b ∈ R + .

Rademacher complexity and validation error trade-off. The bound contains validation errors at different thresholds. We revisit the definition of validation error appearing in the bound. Let X v = { x 1 , . . . x n v } be the validation samples and y i be the label corresponding to x i . Given any h ∈ H and the confidence function g , we have different subsets X v ( h, t ) of the validation points for

̸

which the model's confidence is higher than t . More precisely, X v ( h, t ) = { x i ∈ X v : g ( h, x i ) ≥ t } and the validation error ˆ E a ( h, t | X v ) is computed on each of these subsets as follows, ˆ E a ( h, t | X v ) = 1 | X v ( h,t ) | ∑ x i ∈ X v ( h,t ) 1 ( h ( x i ) = y i ) this error is different from the overall validation error which is computed over the entire set of validation points X v :

̸

The TBAL method computes ˆ E a ( h, t | X v ) at different thresholds ( t ) and selects the threshold at which it is at most ϵ . Thus even if the overall validation error ˆ E ( h | X v ) is bad, there could still be regions in the space where the conditional validation error ˆ E a ( h, t | X v ) is small. This can be easily seen in Figure 2 in the paper. Here we are doing auto-labeling using a linear function class that has low Rademacher complexity. All the models in this class have high overall validation error ˆ E ( h | X v ) but there are subsets where the conditional validation error ˆ E a ( h, t | X v ) is small and TBAL is able to find those subsets. Thus we see that working with low Rademacher complexity classes might lead to high overall validation error. However, it does not affect TBAL as long as there are regions of low conditional validation error ˆ E a ( h, t | X v ) . Furthermore, our upper bound on excess auto-labeling error depends on ˆ E a ( h, t | X v ) which due to the TBAL procedure is at most ϵ and hence it is not in conflict with the Rademacher complexity term.

Next, we state the result for uniform convergence between E a ( h, t ∣ ∣ S ) , ̂ E a ( h, t ∣ ∣ S ) and give its proof.

Lemma C.1. For any δ 3 , p 0 ∈ (0 , 1) , let S and S be defined as above. Let P ( h, t |S ) ≥ p 0 and ˆ P ( h, t | S ) ≥ p 0 ∀ ( h, t ) ∈ H T,g , the following holds w.p. at least 1 -δ 3 / 2

̸

Proof. We begin with proving one side of the inequality and the other side is shown by following the same steps. The proof is based on applying the uniform convergence results for ̂ E ( h, t | S ) and ̂ P ( h, t | S ) from Lemma C.2. The main difficulty here is that E S [ ̂ E a ( h, t ∣ ∣ S ) ] = E a ( h, t ∣ ∣ S ) , so we cannot directly get the above result from standard uniform convergence bounds.

We prove it, by using the results from the Lemma C.2 and restricting the region S such that it has probability mass at least p 0 .

By definitions of E a ( h, t ∣ ∣ S ) and ̂ E a ( h, t ∣ ∣ S ) we have,

Let ξ 1 = √ (1 /n ) log(2 /δ 1 ) , ξ 2 = √ (1 /n ) log(2 /δ 2 ) . From lemma C.2 we have,

Plugging in the above definitions of errors in equation (5) we get,

Substituting ̂ P ( h, t | S ) from equation 6 in the above equation, we get the following w.p. (1 -δ 1 / 2)(1 -δ 2 / 2) , ∀ ( h, t ) ∈ H T,g ,

Using upper bound ̂ E a ( h, t ∣ ∣ S ) ≤ 1 in the second and third terms,

0

Letting δ 1 = δ 2 = δ 3 and ξ 1 = ξ 2 = ξ · p 0 2 gives ξ = √ 4 p 2 0 n log ( 2 δ 3 ) and

This proves one side of the result, and the other side of the result follows similarly.

Lemma C.2. Let S ⊆ X be a sub-region of X and S = { x 1 , . . . , x n } be a set of n i.i.d samples in S drawn from distribution P x . Let { y 1 , . . . y n } be the corresponding true labels, let R n ( H T,g ) be the rademacher complexity of class H T,g then for any δ 1 , δ 2 ∈ (0 , 1) we have,

Proof. The proof is similar to the standard proofs for Rademacher complexity based generalization error bound. Since we work with the modified loss function and hypothesis class to include the abstain option, for completeness we give the proof here. The proofs for error and probability bounds are very much the same except for the change in the loss function. We give the proof for the error bound here.

The result follows by applying McDiarmid's inequality on the function ϕ ( S ) defined as below,

To apply McDiarmid's inequality we first show that ϕ ( S ) satisfies the bounded difference property (Lemma C.4). This gives us,

Using the bound on E S [ ϕ ( S )] from Lemma C.3 we get,

Similarly, the bound for the other side is obtained which holds w.p. 1 -δ 1 / 2 , and combining both we get eq. (9).

The bound of probabilities is obtained by following the same steps as above but with a different loss function, ℓ ⊥ , since P ( h, t |S ) is the probability mass of the region where ( h, t ) does not abstain.

Lemma C.3. Let S ⊆ X be a sub-region of X and S = { x 1 , . . . , x n } be a set of n i.i.d samples in S drawn from distribution P x . Let { y 1 , . . . y n } be the corresponding true labels and let R n ( H T,g ) be the Rademacher complexity of the function class H T,g defined over n i.i.d. samples. Then we have,

Proof. Let ˜ S = { ˜ x 1 , ˜ x 2 , . . . ˜ x n } be another set of independent draws from the same distribution as of S and let the corresponding labels be { ˜ y 1 , . . . ˜ y n } . These samples are usually termed as ghost samples and do not need to be counted in the sample complexity.

In the last step, we used the upper bound on the Rademacher complexity from Lemma C.5.

Lemma C.4. (Bounded Difference) Let S be a set of i.i.d samples from P x then for ϕ ( S ) := sup ( h,t ) ∈H T,g E ( h, t |S ) -̂ E ( h, t | S ) , with probability at least 1 -δ ,

Proof. It is proved by showing that ϕ ( S ) satisfies the conditions (in particular the bounded difference assumption) needed for the application of McDiarmid Inequality. To see this, Let S = { x 1 , x 2 , . . . x i , . . . , x n } and let S ′ = { x 1 , x 2 , . . . x ′ i , . . . , x n } , i.e. S and S ′ may differ only on the i th sample.

̸

The last step follows since ℓ ⊥ 0 -1 is a 0-1 loss function so letting ℓ ⊥ 0 -1 ( h, t, x i , y i ) = 1 and ℓ ⊥ 0 -1 ( h, t, x ′ i , y ′ i ) = 0 gives an upper bound on the difference. Thus we can apply McDiarmid Inequality here and get the bound.

The relationship between the Rademacher complexities is obtained using the following Lemma C.5 due to [16].

Lemma C.5. ([16]) Let ℓ 0 -1 , ℓ ⊥ , ℓ ⊥ 0 -1 be the loss functions defined as above and the Rademacher complexities on n i.i.d. samples S be R n ( H , ℓ 0 -1 ) , R n ( H T,g , ℓ ⊥ ) , R n ( H T,g , ℓ ⊥ 0 -1 ) respectively. Then,

Detailed proof of this lemma can be found in [16]. The result follows by expressing ℓ 0 -1 · ℓ ⊥ as ( ℓ 0 -1 + ℓ ⊥ -1) + and then applying Talagrand's contraction lemma [39].

C.2 Bounds for Finite VC-Dimension Classes

Here we specialize the auto-labeling error and coverage bounds to the setting of finite VC-dimension classes and then instantiate for a specific setting of homogeneous linear classifiers and uniform distribution.

Lemma C.6. [43] (Corollary 3.8 and 3.18). Let the VC-dimension of function class induced by F be any class of functions from X ↦→ Y ∪ {⊥} , and ℓ : Y ∪ {⊥} ↦→ { 0 , 1 } be a 0-1 function. Then,

Corollary C.7. (Auto-Labeling Error and Coverage for Finite VC-dimension Classes) Let k denote the number of rounds of TBAL algorithm 1. Let V ( H T,g ) = d Let X pool ( A k ) be the set of auto-labeled points at the end of round k . N ( k ) a = ∑ k i =1 n ( i ) a denote the total number of auto-labeled points. With probability at least 1 -δ ,

Proof. The proof follows by substituting the Rademacher complexity bounds for finite VC dimension function classes from Lemma C.6 in the general result from Theorem 3.2.

We first simplify the terms dependent on n ( i ) v as follows. Here we use the inequality √ a + √ b ≤ √ 2( a + b ) for any a, b ∈ R + .

Next, we simplify the terms dependent on n ( i ) a as follows. First, we substitute the Rademacher complexity using the bound in Lemma C.6 and then apply the same steps as in the proof of Theorem 3.2 to bound ∑ k i =1 √ n ( i ) a /N ( k ) a by √ k/N ( k ) a followed by the application of √ a + √ b ≤ √ 2( a + b ) to get the final term.

C.3 Homogeneous Linear Classifiers with Uniform Distribution

Here we instantiate Theorem 3.2 for the case of homogeneous linear separators under the uniform distribution in the realizable setting. Formally, let P x be a uniform distribution supported on the unit ball in R d , X = { x ∈ R d : || x || ≤ 1 } . Let W = { w ∈ R d : || w || 2 = 1 } = S d and H = { x ↦→ sign ( ⟨ w , x ⟩ ) ∀ w ∈ W} , the score function is given by g ( h, x ) = g ( w , x ) = |⟨ w , x ⟩| and set T = [0 , 1] . For simplicity, we will use W in place of H .

Corollary 3.4. (Overall Auto-Labeling Error and Coverage) Let ˆ w i , ˆ t i be the ERM solution and the auto-labeling margin threshold respectively at epoch i . Let n ( i ) v , n ( i ) a denote the number of validation and auto-labeled points at epoch i . Let the auto-labeling algorithm run for k -epochs. Then, for any

δ ∈ (0 , 1) , w.p. at least 1 -δ/ 2 ,

and w.p. at least 1 -δ/ 2

Proof. The bound on auto-labeling error follows directly from Theorem C.7 as the VC dimension for this setting is d . For the coverage bound, we utilize the fact that the distribution P x is the uniform distribution over the unit ball. This enables us to obtain explicit lower bounds on the coverage. The details are given in Lemma C.8 and Lemma C.9.

Lemma C.8. Let the auto-labeling algorithm run for k -epochs and let ˆ w i , ˆ t i be the ERM solution and the auto-labeling margin threshold respectively at epoch i . Let X ( i ) be the unlabeled region at the beginning of epoch i , then we have,

Proof. Let X ( ˆ w i , t i ) = { x ∈ X : |⟨ ˆ w i , x ⟩| ≥ ˆ t i } denote the region that can be auto-labeled by ˆ w i , ˆ t i . However, since in each round the remaining region is X ( i ) the actual auto-labeled region of epoch i is X ( i ) a = { x ∈ X ( i ) : |⟨ ˆ w i , x ⟩| ≥ ˆ t i } . Let ¯ X ( ˆ w i , t i ) denote the complement of set X ( ˆ w i , t i ) .

Now observe that X a = ∪ k i =1 X ( i ) a and X ( ˆ w k , ˆ t k ) ⊆ X a because any x ∈ X ( ˆ w k , ˆ t k ) is either auto-labeled in previous rounds i < k or if not then it will be auto-labeled in the k th round. More specifically, any x ∈ X ( ˆ w k , ˆ t k ) is either in ∪ k -1 i =1 X ( i ) a and if not then it must be in X ( k ) a . Thus the sum of probabilities,

The last step used Lemma 4 from [3]) with γ 1 = ˆ t i and γ 2 = 0 to upper bound P ( ¯ X ( ˆ w i , ˆ t i )) by ˆ t i √ 4 d/π . The lemma is stated as follows in Lemma C.9,

Lemma C.9. ([3] (Lemma 4)) Let d ≥ 2 and let x = [ x 1 , . . . x d ] be uniformly distributed in the d -dimensional unit ball. Given γ 1 ∈ [0 , 1] , γ 2 ∈ [0 , 1] , we have:

C.4 Lower Bound

Lemma 3.3. Let c 1 , c 2 and σ > 0 . Let x i ∈ X be a set of n i.i.d. points from X with corresponding true labels y i . Given ( h, t ) ∈ H T,g , let E [( ℓ ⊥ 0 -1 ( h, t, x i , y i ) -E ( h, t |X ) ) 2 ] = σ 2 i > σ 2 for every x i for σ i > 0 and let ∑ n i σ 2 i ≥ c 1 then for every ϵ ∈ [0 , ∑ n i =1 σ 2 i √ c 1 ] with n v < 12 σ 2 ϵ 2 log(4 c 2 ) the following holds w.p. at least 1 / 4 , E a ( h, t |X ) > ̂ E a ( h, t | X ) + ϵ.

Proof. It follows by application of Feller's result stated in lemma C.10.

Lemma C.10. (Feller, Lower Bound on Tail Probability of Sum of Independent Random Variables) There exists positive universal constants c 1 and c 2 such that for any set of independent random variables X 1 , . . . , X m satisfying E [ X i ] = 0 and | X i | ≤ M , for every i ∈ { 1 , . . . , m } , if ∑ m i =1 E [( X ) 2 i ] ≥ c 1 , then for every ϵ ∈ [0 , ∑ i m =1 E [( X i ) 2 ] } M √ c 1 ]

D Additional Experiments

In this section, we discuss additional experiments on the role of hypothesis class in auto-labeling datasets and experiments for studying the role of confidence function in auto-labeling. Finally, we visualize PaCMAP embeddings of the CIFAR-10 and MNIST data points to get a sense of auto-labeling regions in various rounds of the algorithm.

D.1 Additional Experiments on Role of the Hypothesis Class

First, we provide details of the datasets,

XOR is a synthetic dataset. Recall that it is created by uniformly drawing points from 4 circles, each centered at the corners of a square of with side length 4 centered at the origin. Points in the diagonally opposite balls belong to the same class. We generate a total of N = 10 , 000 samples, out of which we keep 8 , 000 in X pool and 2 , 000 in the validation pool X val .

MNIST [15] is a standard image dataset of hand-written digits. We randomly split the standard training set into X pool and the validation pool X val of sizes 48,000 and 12,000 respectively. While training a linear classifier on this dataset we flatten the 28 × 28 images to vectors of size 784.

TBAL Output

AL+SC Output

Queried +ve

Queried -ve

Auto-Labeled +ve

Auto-Labeled -ve

Unlabeled

20

60

100

140

0

10

20

30

Auto-Labeling Error (%)

20

60

100

140

0

20

40

60

80

100

Coverage (%)

TBAL

AL+SC

AL

PL+SC

PL

a

Figure 5: Comparison of Threshold-Based Auto-Labeling (TBAL) and Active-Learning followed by Selective Classification (AL+SC) on XOR-dataset. Left figure (a) shows samples that were auto-labeled, queried, and left unlabeled by these methods. Right figure (b) shows the auto-labeling error and coverage achieved. The lines show the mean and the shaded region shows 1-standard deviation estimated over 10 trials with different random seeds.

XORExperiment. We run the TBAL algorithm 1 with an error tolerance of ϵ a = 1% . we use 20% of N q as seed training data and keep query size n b as 5% of N q . We compare it with active learning and active learning followed by selective classification. The given function class and selective classifier are both linear for all the algorithms. The results are shown in Figure 5. Clearly, there is no linear classifier that can correctly classify this data. We note that there are multiple optimal classifiers in the function class of linear classifiers and they will all incur an error of 25% . So, active learning algorithms can only output models that make at least 25% error. If we naively use the output model for auto-labeling, we can obtain near full coverage but incur 25% auto-labeling error. If we use the model output by active learning with threshold-based selective classification, then it can attain lower error in labeling. However, it can only label ≈ 25% of the unlabeled data. In contrast, the TBAL algorithm can label almost all of the data accurately, i.e., attain close to 100% coverage, with an error close to 1% auto-labeling error.

MNIST Experiment. For training LeNet [37] we use SGD with a learning rate of 0.1, batch size of 32, and train for 20 epochs. We use auto-labeling error threshold ϵ a = 5% . We use 20% of N q as seed training data and keep query size n b as 5% of N q . The results are presented in Figure 6 we observe that TBAL using less powerful models can still yield highly accurate datasets with a significant fraction of points labeled by the models. This confirms the notion that bad models can still provide good datasets.

D.2 Role of Confidence Function

The confidence function g is used to obtain uncertainty scores is an important factor in auto-labeling. In particular, for threshold-based auto-labeling we expect the scores of correctly classified and incorrectly classified points to be reasonably well separated and if this is not the case then the

200

600

1000

1400

5

15

25

Auto-Labeling Error (%)

200

600

1000

1400

0

20

40

60

80

100

Coverage (%)

TBAL

AL+SC

AL

PL+SC

PL

a

200

600

1000

1400

5

15

25

Auto-Labeling Error (%)

200

600

1000

1400

0

20

40

60

80

100

Coverage (%)

TBAL

AL+SC

AL

PL+SC

PL

a

(a) Auto-labeling MNIST data using a linear classifier. The validation size used here is 12k.

(b) Auto-labeling MNIST data using LeNet classifier. The validation size used here is 12k.

10K

15K

20K

25K

30K

10

30

50

70

Auto-Labeling Error (%)

10K

15K

20K

25K

30K

0

20

40

60

80

Coverage (%)

TBAL

AL+SC

AL

PL+SC

PL

a

10K

15K

20K

25K

30K

10

30

50

70

Auto-Labeling Error (%)

10K

15K

20K

25K

30K

0

20

40

60

80

Coverage (%)

TBAL

AL+SC

AL

PL+SC

PL

a

(a) Auto-labeling CIFAR-10 data using a small network and softmax scores. Validation size = 10k.

(b) Auto-labeling CIFAR-10 data using a small network and energy scores. Validation size = 10k.

algorithm will struggle to find a good threshold even if the given classifier has good accuracy in certain regions.

Setup. We perform auto-labeling on the CIFAR-10 dataset using a small CNN network with 2 convolution layers followed by 3 fully connected layers [48]. We use two different scores for auto-labeling, a) Usual softmax output b) Energy score with temperature = 1 [38]. We vary the maximum number of training samples N q and keep 20% of N q as seed samples and query points in the batches of 10% of N q . The model is trained for 50 epochs, using SGD with a learning rate of 0.05, batch size = 256, weight decay = 5 e -4 , and momentum=0.9, and use ϵ a = 10% .

Results. The results with softmax scores and energy scores used as confidence functions can be seen in Figures 8(a) and 8(b) respectively. We see that for both of these cases, TBAL does not obtain a coverage of more than ≈ 6% . We observe that using the energy score as the confidence function performed marginally better than using the softmax scores. We note that this is the case even though the test accuracies of the trained models were around 50% for most of the rounds. Note that CIFAR-10 has 10 classes, so an accuracy of 50% is much better than random guessing and one would expect to be able to auto-label a significant chunk of the data with such a model. However, the softmax scores and energy scores are not well calibrated, and therefore, when used as confidence functions, they result in a poor separation between correct and incorrect predictions by the model. This can be seen in Figure 8 where neither of the softmax and energy scores provides a good separation between the correct and incorrect predictions. We can also see that the energy score is marginally better in terms of the separation, which allows it to achieve slightly better auto-labeling coverage in comparison to using softmax scores. This suggests that more investigation is needed to understand the properties of good confidence functions for auto-labeling which is left to future work.

0.2

0.4

0.6

0.8

1.0

Softmax Score

0

500

1000

1500

2000

Count

corrects

incorrects

0

20

40

60

80

100

120

Energy Score

0

200

400

600

800

Count

corrects

incorrects

(a) Histogram of scores in round 2.

0.2

0.4

0.6

0.8

1.0

Softmax Score

0

500

1000

1500

2000

2500

Count

corrects

incorrects

0

20

40

60

80

100

Energy Score

0

100

200

300

400

500

600

Count

corrects

incorrects

(b) Histogram of scores in round 6.

For a more detailed visualization of the rounds of TBAL for this experiment, see Figures 10 and 11 in the Appendix.

D.3 Detailed Results

In the main paper, we omitted AL, PL, and PL+SC for clarity and due to lack of space. We provide results including these baselines in the following tables for IMDB, Tiny-ImageNet, and Unit-Ball datasets.

D.4 Auto Labeling Visualization

In this section, we visualize the process of TBAL. We use the dimensionality reduction method, PaCMAP [66], to visualize the features of the samples. For neural network models, we visualize the PaCMAP embeddings of the penultimate layer's output and for linear models, we use PaCMAP on the raw features. In these figures, each row corresponds to one TBAL round. Each figure shows a few selected rounds of auto-labeling. Each figure has four columns (left to right), which show: a ) The samples that are labeled by TBAL in the round are shown in that row. b ) The embeddings for training samples in that round. c ) The embeddings for validation data points in that round. d ) The score distribution for the validation dataset in that round.

In Figure 9 we see visualizations for auto-labeling on the MNIST data using linear models. In this setting the data exhibits clustering structure in the PaCMAP embeddings learned on the raw features and the confidence (probability) scores produced are also reasonably well calibrated which leads to good auto-labeling performance.

The visualizations for the process of TBAL on CIFAR-10 using the small network (a small CNN network with 2 convolution layers followed by 3 fully connected layers [48]) with energy scores and soft-max scores for confidence functions are shown in Figures 10 and 11 respectively. We note that both the energy scores and soft-max scores do not seem to be calibrated to the correctness of the predicted labels which makes it difficult to identify subsets of unlabeled data where the current hypothesis in each round could have potentially auto-labeled. We also note that the test accuracies of the trained models were around 50% for most of the rounds of TBAL even though the small network model is not a powerful enough model class for this dataset. Note that CIFAR-10 has 10 classes, so the accuracy of 50% is much better than random guessing and one would expect to be able to auto-label a reasonably large chunk of the data with such a model if accompanied by a good confidence function. This highlights the important role that the confidence function plays in a TBAL system and more investigation is needed which is left to future work.

Note that, in our auto-labeling implementation we find class specific thresholds. In these figures, we show the histograms of scores for all classes for simplicity. We want to emphasize that the visualization figures in this section are 2D representations (approximation) of the high-dimensional features (either of the penultimate layer or the raw features).

Round1 AutoLabeled:18882

Training Samples: 160

ValidationSamples:12000

ValidationScoreDistribution

20

20

20

1000

corrects

15

15

incorrects

10

10

10

800

5

5

quno

600

0

400

5

-5

5

-10

-10

-10

200

15

-10

0

10

20

15

-10

0

10

20

15

-10

0

10

20

0

0.2

0.4

0.6

0.8

1.0

Score

Round 2

AutoLabeled:4502

Training Samples:320

ValidationSamples:7236

ValidationScoreDistribution

20

20

20

500

corrects

15

15

15

incorrects

10

10

10

400

5。

300

200

5

5

5

-10

-10

-10

100

15

-10

0

10

20

15

10

0

10

20

15

-10

0

10

20

0

0.2

0.4

0.6

0.8

1.0

Score

Round 3AutoLabeled:2920

Training Samples: 480

Validation Samples:6123

ValidationScoreDistribution

20

15

20

15

20

15

400

corrects

incorrects

10

10

10

300

5

50

。

200

5

5

-5

10

10

-10

100

15

-10

0

10

20

15

-10

0

10

20

15

-10

0

10

20

0

0.2

0.4

0.6

0.8

1.0

Score

Round 9Auto Labeled:408

Training Samples:1440

ValidationSamples:5436

ValidationScoreDistribution

15

20

20

15

20

15

400

corrects

incorrects

10

10

10

300

5

5

5

0

。

-10

5

-10

5

-10

5

100

15

-10

0

10

20

15

10

0

10

20

-15

-10

0

10

20

0

0.2

0.4

0.6

0.8

1.0

Score

Round 10AutoLabeled:432

Training Samples:1600

ValidationSamples:4645

ValidationScoreDistribution

20

20

20

400

corrects

15

15

15

incorrects

10

10

1

300

5

5

0

。

200

5

5

5

10

-10

-10

100

15

15

-10

0

10

20

-10

10

10

20

0

0.2

0.4

0.6

0.8

1.0

unlabeled

0

6

Score

Round 1 Auto Labeled: 519

Training Samples: 5000

Validation Samples: 10000

ValidationScore Distribution

corrects

6

400

incorrects

0OE

0

Z-

-2

200

4

6

6

100

8

8

8

-10

-5

0

10

-10

5

0

5

10

10

-5

0

5

10

0

0

10

15

20

25

Score

Round 2 Auto Labeled: 513

Training Samples: 7500

Validation Samples:9856

Validation ScoreDistribution

10.0

10.0

10.0

400

corrects

7.5

7.5

350

incorrects

5.0

5.0

5.0

300

2.5

2.5

2.5

250

0.0

0.0

0.0

2.5

2.5

2.5

150

5.0

5.0

5.0

100

7.5

7.5

7.5

50

10.0

10.0

10.0

10

5

0

5

10

5

0

5

10

-5

5

10

0

10

1520

25

30

58

Score

Round 3 Auto Labeled:395

Training Samples: 10000

Validation Samples: 9716

Validation Score Distribution

7.5

7.5

7.5

350

corrects

incorrects

5.0

5.0

5.0

300

2.5

2.5

2.5

250

0.0

0.0

0.0

2.5

2.5

2.5

150

5.0

5.0

5.0

100

7.5

7.5

7.5

50

-10

5

0

5

10

10

-5

5

10

-10

S-

0

5

10

0

0

10

20

30

Score

Round 8 Auto Labeled: 110

Training Samples: 22500

Validation Samples: 9614

Validation Score Distribution

7.5

7.5

7.5

600

700

corrects

incorrects

5.0

5.0

5.0

500

2.5

2.5

2.5

0.0

0.0

0.0

2.5

2.5

2.5

0OE

5.0

5.0

5.0

200

7.5

7.5

7.5

100

10

-5

0

5

10

-10

5

0

5

10

-10

5

0

5

10

0

0

10

1.5

Score

Round 9 Auto Labeled: 244

Training Samples: 25000

Validation Samples: 9468

Validation Score Distribution

7.5

7.5

7.5

700

corrects

incorrects

5.0

5.0

5.0

600

2.5

2.5

2.5

500

0.0

0.0

0.0

2.5

2.5

2.5

300

5.0

5.0

5.0

200

7.5

7.5

7.5

100

10.0

-10

5

0

unlabeled

10

10.0

airplane

-10

automobile

5

0

bird

5

10

cat

deer

-10

5

0

frog

5

horse

10

ship

0

Score

10

15

6op

6

Round 1 Auto Labeled: 0

Training Samples: 5000

Validation Samples: 10000

Validation Score Distribution

7.5

7.5

7.5

3000

incorrects

corrects

5.0

5.0

5.0

2500

2.5

2.5

2.5

2000

0.0

0.0

0.0

1500

2.5

2.5

2.5

5.0

5.0

5.0

1000

7.5

7.5

7.5

500

10.0

-10

5

0

5

10

10.0

-10

5

0

5

10

10.0

10

5

0

5

10

0

0.2

0.4

Score

0.6

0.8

1.0

Round 2 Auto Labeled: 138

Training Samples: 7500

Validation Samples: 10000

Validation Score Distribution

10.0

7.5

10.0

7.5

10.0

1.51

2500

corrects

incorrects

5.0

5.0

5.0

2000

2.5

2.5

2.5

0.0

0.0

0.0

2.5

2.5

2.5

1000

5.0

5.0

5.0

7.5

7.5

7.5

500

10.0

-10

S-

0

5

10.0

10.0 7.5 5.0 2.50.0

2.55.0

7.5

10.0

10

-5

0

5

0.2

0.4

0.6

0.8

1.0

Score

10.0

Round 7 Auto Labeled: 487

10.0

Training Samples: 20000

10.0

Validation Samples: 9968

ValidationScore Distribution

7.5

7.5

7.5

2500

incorrects

corrects

5.0

5.0

5.0

2000

2.5

2.5

2.5

1500

0.0

0.0

0.0

8

2.5

2.5

2.5

1000

5.0

5.0

5.0

500

7.5

7.5

7.5

10

5

0

5

10

10

5

0

5

10

10

5

0

5

10

0

0.2

0.4

Score

0.6

0.8

1.0

Round 8 Auto Labeled:101

Training Samples: 22500

Validation Samples: 9763

Validation Score Distribution

8

8

corrects

2500

incorrects

2000

0

0

2

2

1000

4

6

6

6

500

8-

8

-10

5

0

5

10

-10

5

0

5

10

-10

5

0

5

10

0.2

0.4

0.6

0.8

1.0

Score

Round 9 Auto Labeled: 0

Training Samples: 25000

Validation Samples: 9734

Validation Score Distribution

2500

corrects

incorrects

2000

1000

10

10

10

500

15

15

15

-10

S-

0

5

10

10

5

0

5

10

10

5

0

5

10

0.2

0.4

0.6

8'0

1.0

unlabeled

airplane

automobile

bird

cat

deer

6op

60.y

horse

ship

truck

Score

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies an important problem. It theoretically analyzes TBAL systems and develops tradeoffs between\nthe quantity of manually-labeled data and the quantity and quality of auto-labeled data. Empirical results on real/synthetic data also support their findings.\nTheir results provide two crucial insights. when having access to sufficient validation data and a good confidence function, even bad\nmodels can reliably label at least some data, meanwhile, the quantity requirement of the validation data can be high.\n\nSTRENGTHS:\n1. This paper is the first to study the theoretic performance of TBAL, which is important in practice.\n2. For the proposed TBAL, this paper gives theoretic guarantees on the auto-labeling error and the coverage using Rademacher Complexity and PAC theory.\n3. The paper designs a series of experiments including synthetic data, and real data. The experiments do reveal some interesting observations (though some observations are unexpected, need more disscusions, see W3.)\n4. The writing is good, easy to understand\n\nWEAKNESSES:\nW1. The goal of the paper seems a little confusing. On the one hand, it aims to study an existing popular TBAL system.\nOn the other hand, it seems that the studied TBAL algorithm is in fact proposed in this current paper.\n\nW2. Several questions need to be clarified (possibly in the related work section): \n(1) Is the proposed TBAL the same with any existing popular TBAL?\n(2) Whether the theories apply to an existing popular TBAL system, what if their workflows are different?\n(3) The paper claims that the TBAL is popular. But it didn't provide any evidence for its popularity, e.g., use cases, scenarios.\n\nW3. In experiments, it seems AL+SC can outperform TBAL (in terms of error rate, given that the difference in coverage is not significant).\nThis is very different from Figure 2, why this can happen? Please add more discussions for explaining the different performance among the 4 competing methods.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an analysis of Threshold-Based Auto-Labeling (TBAL) systems by deriving sample complexity bounds on the amount of human-labeled validation data required for guaranteeing the quality of the machine-labeled data. The paper provides two main insights: (1) large amounts of unlabeled data can be automatically/accurately labeled by weak models, and (2) a downside of TBAL systems is the need of potentially prohibitive amounts of necessary validation data.\n\nSTRENGTHS:\nTo the best of this reviewer's knowledge, this is the first paper that derives sample complexity bounds on the amount of validation data required for the confident use of TBAL systems. The contributions appear to be novel, and the paper is well-written & reasonably easy to follow. The authors provide both a theoretical analysis and an empirical validation for TBAL systems. Figure 1 is extremely intuitive and summarizes very well TBAL systems for readers unfamiliar with the problem.\n\nWEAKNESSES:\nThe paper could further benefit from additional discussion along the key findings. \n\n1) while Fig 2.a is crisp & intuitive, Fig 2.b is less so. The authors should (1) explain why the coverage of AL & PL is decreasing as the training sample increases, and (2) discuss why the coverage of AL+DC & PL+SC does not really change for various amounts of training data\n\n2) for Fig 3, the authors should (1) drop PL+SC with a note saying that AL+SC is always comparable/better, (2) explain why the coverage is so low for both IMDB & Tiny ImageNet, (3) discuss the usefulness of a TBAL approach with a coverage of about 25%, and (4) \"zoom in\" on an in-depth, 1:1 comparison between TBAL and AL+SC, which seem to be performing quite similarly (TBAL with a slightly better coverage, while AL+SC with a slightly better auto-labeling error); this comparison & potential trade-offs analysis should represent a major contribution of the paper.\n\nQUESTIONS:\n1) on line 268, the paper says that it is using five datasets, but it only lists four (Unit-Ball, Tiny-Imagenet, IMDB Reviews, and CIFAR-10). In the APPENDICES, the authors also bring up XOR & MNIST, for a total of six, rather than five, datasets \n\n2) In the main paper, why don't you show the results for all 4 datasets (see above)?\n\n3) would a scatter plot of AutoLabel Error vs Coverage help illuminate any trade-offs/insights for the domains in which the error/coverage are not totally/mostly flat?\n\n4) line 261: could you please explain why is Algo 2 is estimating a single threshold for all classes, while in Section 4 you estimate per-class thresholds?\n\nLIMITATIONS:\nCould you please discuss in depth the practical impact/usefulness of a TBAL-created dataset that has only 25% coverage?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThreshold-based auto-labelling (TBAL) is widely adopted in academic and industrial settings and this is the first work that studies TBAL from a theoretical standpoint, answering the questions of 1) Does TBAL works even if our models are not perfect and 2) what are the key requirements to having a successful TBAL pipeline? The paper suggests that even with a low-performant model, TBAL still helps accelerates the annotations. However, to facilitate a successful TBAL system, a potentially prohibitively large validation dataset might be needed. After anchoring the keys of TBAL, the paper experiments with several simple settings to validate the effectiveness of TBAL.\n\nSTRENGTHS:\nTBAL is widely adopted in both academic and industrial settings and the paper is the first that study it theoretically. The derivation is sound and can be easily drawn connections with the active learning literature. I expect more follow-up work in the future. \nThe strength of this work include\n- Novelty: first work to study TBAL theoretically \n- Clarity: The delivery is easy to follow\n\nWEAKNESSES:\nAlthough the paper is novel, section 3 and section 4 are disconnected to me. Section 3 gives us a bound on the error rate and coverage of TBAL, which provides insightful relations between the performance and different design factors. \nHowever, I cannot find the bound in Figure 3 and Figure 4. I expect to see how tight the bound is empirically.\n\nOn the other hand, Eq. 2 also provides insights into the empirical validation error (term a). I expect the authors to push the limits for using a more expressive feature extractor and discuss the relation between the empirical validation error and the TBAL error rate empirically.\n\nQUESTIONS:\n- L204 suggests to use a hypothesis class that induces lower Rademacher complexities to reduce the second term in Eq. 2. However, a hypothesis class (e.g., Lipchitz-constrained neural networks) with lower Rademacher complexities usually leads to higher validation errors, therefore, increasing the first term in Eq. 1. If so, do the authors still suggest practitioners to use a hypothesis class that indices lower Rademacher complexities. How to balance the trade-off here?\n    - This matters since a practitioner wants a rule of thumb for using TBAL. It’s important to discuss the trade-off here.\n- Minor: As mentioned in #weakness, the current experimental results show that annotating 5k TinyImageNet effectively requires a validation dataset with 5k images, which does not sound “promising”.  Have the authors tried to push the limits a little bit further by using a better feature extractor?\n\nLIMITATIONS:\nThe paper provides a novel view of TBAL, but the theoretical and empirical parts are not connected well. The authors should provide more numbers on how tight the proposed bound is under auto-labelling.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}