MCAL: Minimum Cost Human-Machine Active Labeling
Authors: Hang Qiu
Krishna Chintalapudi
Ramesh Govindan
Venue: ICLR 2023 poster (2023)
Keywords: Active Labeling
Groundtruth Annotation
Dataset Labeling

Abstract: Today, ground-truth generation uses data sets annotated by cloud-based annotation services. These services rely on human annotation, which can be prohibitively expensive. In this paper, we consider the problem of hybrid human-machine labeling, which trains a classifier to accurately auto-label part of the data set. However, training the classifier can be expensive too. We propose an iterative approach that minimizes total overall cost by, at each step, jointly determining which samples to label using humans and which to label using the trained classifier. We validate our approach on well known public data sets such as Fashion-MNIST, CIFAR-10, CIFAR-100, and ImageNet. In some cases, our approach has 6× lower overall cost relative to human labeling the entire data set, and is always cheaper than the cheapest competing strategy.

Published as a conference paper at ICLR 2023
MCAL: MINIMUM COST HUMAN-MACHING ACTIVE LABELING
Hang Qiu', Krishna Chintalapudi', Ramesh Govindan' University of Southern California, "Microsoft Research [hangqiu, ramesh ) @usc. edu, arch.ntaa@microooft..om
ABSTRACT
Today, ground-truth generation uses data sets annotated by cloud-based annotation services. These services rely on human annotation, which can be prohibitively expensive. In this paper, we consider the problem of hybrid human- labeling, which trains a classifier to accurately auto-label part of the data set. However, training the classifier can be expensive too. We propose an iterative approach that minimizes total overall cost by, at each step, jointly determining which samples to label using humans and which to label using the trained classifier. We validate our approach on well known public data sets such CIFAR-10, 10, CIFAR-100, and ImageNet. In some cases, our approach has 6x lower overall cost relative to human labeling the entire data set, and is always cheaper than the cheapest competing strategy.
Reshion-MNISS
INTRODUCTION
Ground-trute truud is crucial for training and testing ML models. Generating accurate ground-truth was cumbersome until the recent emergence of cloud-based human annotation services (SageMaker (2021); Google (2021); Figure-Fight (2021)). Users of these services submit data sets and receive, in return, annotations on each data item in the data set. Because these services typically employ humans to generate ground-truth, annotation costs can be prohibitively high especially for large data sets.
Hybrid Human-machine Annotations. In this paper, we explore using a hybrid numan-maching approach to reduce annotation costs (in S) where humans only annotate subset of the data items; a machine learning model trained on this annotated data annotates the rest. The accuracy of a model trained However, a user of an annotation service might choose to avail of this trade-off if (a) targeting a slightly lower annotation quality can significantly reduce costs, (b) the cost of training model to a higher accuracy is itself prohibitive. Consequently, this paper focuses on the design of a hybrid numan-machine annotation scheme that minimizes the overall cost of annotating the entire data set (including the cost of training the model) while ensuring that the overall annotation accuracy, relative to human annotations, higher than a pre-specified target (e.g.. 95%).
subset of the data set will typically be lower than that of human annotators.
Challenges. In this paper, we consider specific annotation task, multi-class labeling. We assume that the user of an annotation service provides a set X of data be labeled and a classifier D to use for machine labeling. Then, the goal is to find subset B C X human-labeled samples to train D, and use the classifier to label the rest, minimizing total cost while ensuring the target accuracy. straw man approach might seek to predict human-labeled subset B in single shot. This is hard to do because depends on several factors: (a) the classifier architecture and how much accuracy it can achieve, (b) how "hard" the dataset is, (c) the cost of training and labeling, and (d) the target accuracy. Complex models may provide a high accuracy, their training costs may be too high and potentially offset the gains obtained through machine- generated annotations. Some data data-points in dataset are more informative as compared to the rest from a model training perspective. Identifying the "right" machine-labeling can minimize the total labeling cost.
data subset for human-
Approach. this paper we propose a novel technique, MCAI! (Minimum Cost Active Labeling), that addresses these challenges and is able minimize annotation cost across diverse data sets. At its
MCAL is available https: httpa//:///tttttoo//// hww.ub..cooo//a/gggaa///ea

---

Published as a conference paper at ICLR 2023
Label
Label
+ Annotations
Data +
Data +
Sample
Sample
Model
ML
ML
Min-Cost Active Labeling
Active Learning
Differences between MCAL and Active Learning, Active learning outputs an ML model using few samples from the data set. MCAL completely annotates and outputs the dataset. MCAL must also use the ML model to annotate samples reliably (red arrow).
FIGURE 1:
core, MCAL learns on-the-fly an accuracy model that, given a number of samples to human-label, and number to machine label, can predict the overall accuracy of the resultant set of labeled samples. Intuitively, this model implicitly captures the complexity of the classifier and the data set. It also uses cost model for training and labeling.
MCAL proceeds iteratively. At each step, it uses the accuracy model and the cost model to search for the best combination of the number of samples to human-label and machine-label that would minimize total cost. It obtains the human-labels, trains the classifier D with these, dynamically updates the accuracy model, and machine-labels un-labeled samples if it determines that additional training cannot further reduce cost.
MCAL resembles active learning in determining which samples in the data set to select for human- labeling. However, it differs from active learning (Fig. 1) in its goals; active learning seeks to train a classifier with given target accuracy, while MCAL attempts to label complete data set within a given error bound. In addition, active learning does not consider training costs, as MCAL does.
This paper makes the following contributions:
It casts the minimum-cost labeling problem in an optimization framework ($2) that minimizes total cost by jointly selecting which samples to human label and which to machine label. This framing requires a cost model and an accuracy model, discussed above ($3). For the former, MCAL assumes that total training cost at each step is proportional to training set size (and derives the cost model parameters using profiling
real hardware). For the latter, MCAL leverages the growing body of literature suggesting that a truncated power-law governs the relationship between model error and training set size (Cho et al. (2015); Hestness al. (2017); Sala (2019)).
The MCAL algorithm ($4) refines the power-law parameters, then does fast search for the combination of human- and machine-labeles samples that minimizes the total cost. MCAL uses an active learning metric select samples to human-label. But because includes labelin step, not all metrics work well for MCAL. Specifically, core-set based sample selection is not the best choice for MCAL; the resulting classifier machine-labee labels fewer samples.
MCAL extends easily to the case where the user supplies multiple candidate architectures for the classifier. It trains each classifier up to the point where it is able to confidently predict which architecture can achieve the lowest overall cost.
Evaluations ($5) various popular benchmark data sets show that MCAL achieves lower than the lowest-cost labeling achieved by an oracle active learning strategy. automatically adapts its strategy to match the complexity of the data set. For example, it labels most of the Fashion data set using a trained classifier. At the other end. it chooses to label CIFAR-100 mostly using humans, since estimates training costs be prohibitive. Finally, it labels little over half of CIFAR-10 using classifier. MCAL is up to 6x cheaper compared to human labeling all images. It is able to achieve these savings, in part, by carefully determining the training size while accounting for training costs; cost savings due to active learning range from 20-32% for Fashion and CIFAR-10.
2 PROBLEM FORMULATION
In this section, formalize the intuitions presented in $1. The input to MCAL is an unlabeled data set X and a target error rate bound e. Suppose that MCAL trains a classifier (BB) using human generated labels for some B C X. Let the error rate of the classifier D(B) over the remaining unlabeled data
2

---

Published as a conference paper at ICLR 2023
using D(B) be c(X\B), If ((B) machine-generatedd labels for this remaining data, the overall ground-truth error rate for X would be, e(X) = (1 -1- (BII \ B). If eXX) 2 E, then, this would violate the maximum emror rate requirement.
However, D( is still able to generate accurate labels for a carefully chosen subset S(D, B) C. CNB (e.g., comprising only those that ((B) is very confident about). After generating labels for S(D, B) using D(B), labels for the remaining X B \ S(D, B) can be once again generated by humans. The overall error rate of the generated ground-truth then would be {[S((D, B) (()((())) B)). E(S(D, B)) is the error rate of generating labels over S(D, B) using D(B) and is, in general, higher for larger (S, B. Let S* (D. B) be the largest possible (D, B) that ensures that the overall error rate is less than E. Then, overall cost of generating labels is:
== (IXNS* B))) Cr. + C.(((
(1)
where, C2 is the cost of human labeling for single data item and (D()))) the total cost of generating D(B including the cost of optimizing B and training D(B). Given this, MCAL achieves the minimum cost by jointly selecting S* (D, B) and B subject to the accuracy constraint:
C'
argmin C, s.t. (1S" (2 <e S1,D,B),B
(2)
3 COST PREDICTION MODELS
MCAL must determine the optimal value of the training size B and its corresponding maximum machine-labeles set S* (D, B) that minimizes C. In order to make optimal choices, MCAL must be able to predict C as a function of the choices. C in turn depends on (S* B)] and the training cost (.((( (Eqn. 1). Thus, MCAL actually constructs two predictors, one each for |.S (SS B)] ($3.1) and C, (DB)) ($3.2). These predictors rely on two sample selection functions ($3.3): from among the remaining un-labeled samples, M() selects, which to human-labe label for training; L(-) selects those that the classifier (BB) can machine-label within the error constraint E.
3.1 ESTIMATING MACHINE-LABELLN PERFORMANCE
To maximize total cost reduction without violating the overall accuracy constraint (Eqn. 1), MCAL must determine a maximal fraction of samples 0* selected by T.(-) for machine labeling. To predict the model error <(50(D) (D())) we leverage recent work ($6) that observes that, for many tasks and many models, the generalization error vs. training set size is well-modeled by power-law (Hestness et al. (2017); Sala (2019)) ( <((S(D(())) = agg [ "0). However, it is well-known that most power-laws experience a fall-off (Burroughs (2001)) high values of the independent variable. To model this, we use an upper-truncated power-law (Burroughs (2001)):
IBI
(D())) az T0 %
where dgg Yoo kg are the power-law parameters.
Eqn. 3 can better predict the generalization error (Fig. 2) based on the training size than a power-law, especially at larger values of BJ. Fig. shows that more samples help calibrate the truncated power law to better predict the falloff. Other models and data sets show similar results (see Appendix $F).
((S will increase monotonically with since increasing 0 has the effect of adding data that D is progressively less confident about (ranked by L(.)). Lacking a parametric model for this dependence, to find 0*, MCAL generates plwer-law models E(S" (D(B)) for various discrete (0, 1) ($4). MCAL obtains 6* for a given B by searching across the predicted E(S" (D(B)) While also depends on the data acquisition batch size (ő, see $3.2), when the total accumulated training sample size is large enough, the error rate starts to converge, hence the dependence is insignificant. Fig. 4 shows this variation is < 1% especially for smaller values of A.
3.2 ESTIMATING ACTIVE LEARNING TRAINING COSTS
Active learning iteratively obtains human labels for batch size of 8 items ranked using the sample selection function M (.) and adds them to the training set B to retrain the classifier D. A smaller
Evaluation of labels assumes perfect human-lahels( B).
3

---

Published as a conference paper at ICLR 2023
Power Law
Truncaled Power Law
Eroor (8 (D(a1))
(DIB)
(D(B))
Broor (8) (D(aa))
1011
Error (8' (D(B))))
9.0
30*
10
points
points
(D/B)))
pointa
10
10
points
15 points
0.2
10
10
2000
16000
16000
4.5%
9.20%
0.50%
FIGURE 2: Fitting generalization error (e(S" (D(S"(D(B))) (D(B))) agasint train- ing size power law and trun- cated power law (CIFAR-10 10 using RESNETI8 for various 0).
FIGURE 3: Error prediction im- proves with increasing number of emor estimates CIFAR-10 using RESNETI8.
FIGURE 4: Dependence of (D(B))
is "small" especially towards the end of active learning. Here, (B) = 16, 000) for CIFAR-10 using RESNETI8.
typically makes the active learning more effective: allows for achieving a lower error for potentially smaller B through more frequent sampling, but also significantly increases the training cost due to frequent re-training. Choosing an appropriate training sample acquisition batch size is thus an important aspect of minimizing overall cost.
The training cost depends on training time, which turn is proportional to the data size ([B) and the number of epochs. A common strategy is to use fixed number of epochs per iteration, so the training cost in each iteration is proportional to [B]. Assuming new data samples added in each iteration, the total training cost follows
C(((B)) (B) -
(4)
MCAL can accommodate other training cost models?.
3.3 SAMPLE SELECTION
The sample selection functions (.) and L(.) are qualitatively different. The former selects samples to obtain accurate classifier, the latter determines which samples a given classifier is most confident about (so they can be machine-labeled) For L(.), we simply use the margin metric (Scheffer et al. (2001)), the score difference between the highest and second highest ranked labels (see Fig. 5).
(.) is similar to sample selection for active learning. Prior work has considered coreset (Sener & Savarese (2017)) and uncertainty (Schef- fer al. (2001)) based ap- proaches. The k-center selec- tion method (Wolf (2011)), an instance of the former, uses the feature- ceature-space distance between the last activation layer output to iteratively select data points farthest from existing centers. Uncertainty-bant based approaches include max-entropy (Dagan & Engelson (1995)), which samples data with highest entropy in activation output, least-confidence (Culotta & McCallum (2005)), and margin.
Machine Labeling Aecunnnn Cuaee
selection Metric (ML)) (M.)) Value
3.0
2.5
0.0
2.0
Min. K-centers Distance
Entropy
0.6
Confidence
Margin
1.0
Confidence
Min. Kccenters Distarice
0.4
Entropy
0.5
0.4
0.6
0.0
0.2
0.4
0.6
0.8
1.0
Top ranked % samples
Margin
FIGURE 5: Machine labeling accu- racy of samples ranked by L(.)
FIGURE 6: Sampling metric (M)) comparison
We evaluate MCAL using these different choices for (.), but find that ($5), for a subtle reason, uncertainty-basse approaches perform better. Unlike with active learning. where the goal is to select samples to match the generalization error the entire data set, MCAL'S goal is to select samples to train classifier that can machine label the largest number of samples possible. When MCAL uses margin or least confidence for L(),, the samples selected have high accuracy (close to 100%,
For example, if the number of epochs is proportional to [B] in which case C,D)(B)) can have cubic dependency B3]
4

---

Published as a conference paper at ICLR 2023
Fig. 5)* The samples selected by core-set based algorithm such as k-centers is poorly correlated with accuracy (Fig. 5) and margin (Fig. 6).
4 THE MCAL ALGORITHM
The MCAL algorithm (Alg. I, see appendix $A) takes as input an active learning metric (.). the data set X, the classifier 'D (e.g., RESNET18) and parametric models for training cost (e.g., Eqn. 4), and for error rate as a function of training size (e.g., the truncated power law in Eqn. 3). The algorithm operates in two phases. In the first phase, it uses estimates obtained during active learning to learn the parameters of one truncated power-law model for each discrete machine-label fraction 0, and uses the cost measurements to learn the parameters for the training-cost model. In the second phase, having the models, it can estimate and refine the optimal machine- labeling subset S (D, B) and the corresponding training set B that produce the optimal cost C It can also estimate the optimal batch size Sept for this cost. It terminates when adding more samples to B is counter productive. It trains a classifier to label S* (D, B) and use human labels for the remaining unlabeled samples.
To start with, MCAL randomly selects a test set T (e.g. [T] = 55 of (X) and obtains human labels to test and measure the performance of D. Next, MCAL initializes B Bo by randomly selecting ón samples from X (e.g., 1% of X in our implementation) and obtaining human labels for these. Then, MCAL trains D using By and uses T to estimate the generalization errors er (S (B)))) for various values of 0 E (0, 1) (we chose in increments of 0.05 (0.05, 0.1,.... 1}), using T and M.).
Next, the main loop of MCAL begins. Similar with active learning, MCAI, selects, in each step, ò samples, ranked by M(.), obtains their labels and adds them to B, then trains DD them. The primary difference with active learning is that MCAL, in every iteration, estimates the model parameters for machine-labeling error E(S (D(B)) for each 0 with various training size B, then uses these to estimate the best combination of B and @ that will achieve the lowest corresponding overall cost C At the end of this iteration, MCAL can answer the question: "How many human generated labels must be obtained into B to train D, in order to minimize C?" ($3).
The estimated model parameters for the training cost (D(() and the machine-label error (D(()) may not be stable in the first few iterations given limited data for the fit. To de- termine the model parameters are stable, MCAL compares the estimated C* obtained from the previous iteration to the current. If the difference is small (S 5%, in our implementation), the model is considered to be stable for use.
After the predictive models have stabilized, we can rely on the estimates of the optimal training size Bopt, to calculate the final number of labels to be obtained into B. At this point MCAL adjusts 5 to reduce the training cost when it is possible to do so. MCAL can do this because it targets relatively high accuracy for D(B). For these high targets, it is important to continue to improve model parameter estimates (e.g., the parameters for the truncated power law), and active learning can help achieve this. Fig. shows how the fit to the truncated power law improves as more points are added. Finally, unlike active learning, MCAL adapts ö, proceeding faster Bont to reduce training cost. Fig. 4 shows that, for most values of 0, the choice of 5 does not affect the final classifier accuracy much. However, it can significantly impact training cost ($3).
This loop terminates when total cost obtained in a step is higher than that obtained in the previous step. At this point, MCAL simply trains the classifier using the last predicted optimal training size Bopt, then human labels any remaining unlabeled samples.
Extending MCAL to selecting the cheapest DNN architecture. In what we have described so far, we have assumed that MCAL is given candidate classifier D. However, it is trivial to extend MCAL to the case when the data set curator supplies a small number (typically 2-4) of candidate classifiers (D1, D2,-.. J. In this case, MCAL can generate separate prediction models for each of the classifiers and pick the one that minimizes C once the model parameters have stabilized. This does not inflate the cost significantly since the training costs until this time are over small sizes of B.
Accommodtting a budget constraint. Instead of a constraint of labeling error, MCAL can be modified to accommodate other constraints, such as a limited total budget. Its algorithm can search
'The results are from Res18 trained over 8K samples (CIFAR10); actual numbers will vary by dataset, training size, and model used.
5

---

Published as a conference paper at ICLR 2023
for the lowest error satisfying given budget constraint. Specifically, with the same model for estimation of network error and total cost ($3), instead of searching the (D, B) for minimum total search for minimum
cost while error constraint is satisfied (line 18 of Alg. 1, Appendix $A), estimated error while the total cost is within budget. The difference is: in the former case, we can always resort human labeling when error constraint cannot be satisfied; in the latter case, we can only sacrifice the total accuracy by stopping the training process and taking the model's 's output when the money budget is too low.
EVALUATION
In this section, we evaluate the performance of MCAL over popular classification data sets: Fashion- MNIST, CIFAR-10, CIFAR-100, and ImageNet. We chose these data sets to demonstrate that MCAL can work effectively across different difficulty levels, We use three popular DNN architectures RESNET50, RESNET18 (He et al. (2016)), and CNN18 (RESNETI8 without the skip connections). These architectures span the range of architectural complexity with differing training costs and achievable accuracy. This allows us to demonstrate how MCAL can effectively select the most cost efficient architecture among available choices. We also use two different labeling services: Amazon SageMaker (2021) at $0.04/image and Satyam (Qiu et al. (2018)) at $0.003/image This allows us to demonstrate how MCAL adapts to changing human labeling costs. Finally, parametric model fitting costs are negligible compared to training, metric profiling, and human labeling costs.
MCAL evaluates different sampling methods discussed (Fig. 6) and uses margin to rank and select samples. At each active learning iteration, it trains the model over 200 epochs with 10x learning rate reduction at 80, 120, 160, 180 epochs, and a mini- mini-batch-size of 256 samples (Keras (2021)). Training is performed on virtual machines with 4 NVIDIA K80 GPUs at cost of 3.6 USD/hr. In all experiments, unless otherwise specified, the labeling accuracy requirement & was set at 5%.
5.1 REDUCTION IN LABELING COSTS USING MCAL
MCAL automatically makes three key decisions to mini- mize overall labeling cost. a) selects the subset of images
5000
Human Labeling MCAL
that the classifier should be trained on (IB) opt). b) adapts 5 across active learning iterations to keep training costs in check, and c) selects the best DNN architecture from among set of candidates. In this section, we demonstrate that MCAL provides significant overall cost benefits at the expense of (5%) degradation in label quality. Further, it even outperforms active learning assisted by an oracle with optimal ō value.
4000
Oracle AL CNN18
Oracle AL RESNET16
Oracle AL RESNETSO
Cost [8]
3000
2000
1000
Fashion CIFAR-10CIFAR-100
Fig. 7 depicts the total labeling costs incurred when using Amazon labeling services for three different schemes: i) when the entire data set is labeled by Amazon labeling services, ii) MCAL for 5 5%. and ii) active learning with an oracle to choose ô for each DNN model.
Total cost labeling for various data sets, for i) Human labeling, ii) MCAL, and iii) Oracle assisted AL with various DNN architectures.
FIGURE 7:
Tbl. lists the numerical values of the costs (in S) for human labeling and MCAL. To calculate the total labeling error, we compare the machine labeling results on S* (D, Bopt) and human labeling results on X 5 SD, Bopt) against the ground The human labeling costs are calculated based on the prices of Amazon labeling services SageMaker (2021) and Satyam (Qiu et al. (2018)).
Cost Saving Compared to Human Labeling. From Fig. and Tbl. 1, MCAL provides an overall cost saving of 86%, 67% and 30% for Fashion, CIFAR-10 10 and 100 respectively. As expected, the savings depend on the difficulty of the classification task. The "harder" the dataset the higher the savings. Tbl. 1 also shows the number of samples in B used to train D, well as the number of samples |SS] labeled using D. For Fashion, MCAL labels only 6. % of the data to train the classifier and uses it to label 85% of the data set. For CIFAR-10 10, it trains using 22% of the data set and labels about 65% of the data using the classifier. CIFAR- 100 requires more data to train the classifier to a high accuracy is able to label only 10% of the data using the classifier.

---

Published as a conference paper at ICLR 2023
Data Sct
Labeling
[X]
[2]
[X]
[S]
DNN
Error
Human
MCAL Cost (S)
MCAL
Service
Selected
Cost ($)
Savings
Fashion
Amazon
6.1%
85.0%
Res18
4.0%
2800
400
86%
Satyam
8,4%
85.0%
Res18
4.0%
210
29
86%
CIFAR 10
Amazon
22.2%
65.0%
Res18
2.4%
2400
792
67%
Satyam
27.0%
65.0%
Res18
2.4%
180
63
63%
CIFAR
Amazon
32.0%
10.0%
Res18
0.4%
2400
1698
29%
100
Satyam
57.6%
20.0%
Res18
1%
180
139
23%
TABLE 1: Summary of Results
CNNIS RESNET10 RESNETSO
CNNIS S-RESNETI RESNETSO
Total Cost Labal (5
2500
RESSETT0 RESNETSO
Total Cost to Labal [5]
Human Labeling
2000
Human Labeling
Total Cost to Label
2000
Human Labeling
3000
1.50
1900
MCAL
2000
1000
300
1000
MCAL
1000
MCAL
500
1415
AL Batch Size (0) [X]
AL Batch Size (6) (% of [X]
AL Batch Size (o) 1% of (X]
FIGURE 8: MCAL active Iearn- ing with different batch sizes (Fashion, Amazon).
FIGURE 9: MCAL active learn- ing with different batch sizes 8 learning with different batch sizes 6 (CIFAR-100, Amazon).
FIGURE 10:
MCAL active
(CIFAR-10, Amazon).
Cost Savings from Active Learning. In the absence of MCAL, how much cost savings would one obtain using naive Active Learning? As described in $3, the overall cost of AL depends on the batch size the DNN architecture used for classification, as well as how "hard" it is to classify the data set. In order to examine these dependencies, we performed active learning using values of ă between 1 1% to 20% of [X|, to label each data set using the different DNN architectures until the desired overall labeling error constraint was met. The contribution to overall labeling error is zero for human annotated images and dictated by the classifier performance for machine labeled images.
MCAL AL. Figures 8, 9 and 10 show the overall cost for each of the three data sets using different DNN architectures. The optimal value of ó is indicated by a circle in each of the figures. Further, the cost of labeling using humans only well as MCAL cost is indicated using dashed lines for reference. As shown in the figure, MCAL outperforms AL even with optimal choice of 8. The choice of can significantly affect the overall cost, by up 4-5x for hard data sets.
Training cost. Figures 19, 20, and 21 (see appendix ŞE) depict the AL training costs for each of the data sets using three different DNNs. The training cost can vary significantly with 5. While for Fashion, there is a 2x reduction in training costs, it is about 5x for CIFAR-10 10 and CIFAR-100.
Dependence on sample selection method M.). Fig. 11 shows the total cost of MCAL using different sample selection functions (.). As $3.3 describes, k-center is qualitatively different from uncertainty-bant methods in the way it selects samples. In Fig. 11, for CIFARIO on RESNETI8 (other results are similar, omitted for brevity), while all metrics reduce cost relative to human- labeling the entire data set, uncertainty-basee based metries have 25% lower cost compared to k-center because the latter machine-labels fewer samples. Thus, for active labeling, an uncertainty- based metric is better for selecting samples for classifier training.
Dependence on batch size 8. Fig. 12 depicts the fraction of images that were machine labeled for the different data sets and DNN architectures trained with different batch sizes. As seen in Fig. 12, lower d values allow AL to adapt at finer granularity, resulting in higher number of machine labeled images. Increasing from 1% to 15% results in a 10-15% fewer images being machine labeled.
Dependence on data set complexity. As seen from Figures 8-10, and 12, the training costs well as potential gains from machine labeling depend on the complexity of the data set. While, using a small 5 (2%) is beneficial for Fashion, the number is larger for CIFAR-10 10. For CIFAR-100, using active learning does not help at all as the high training costs overwhelm benefits from machine labeling.
A second dimension of complexity is the number of samples per class. CIFAR-100 has 600 per class, CIFAR- 10 has 6000. To demonstrate that MCAL adapts to varying numbers of samples per class, Fig. 13 shows experiment in which MCAL subsets of CIFAR-10, with 1000-5000
7

---

Published as a conference paper at ICLR 2023
2000
100
2000
Haren Lebeling
Mabling on
50
Classiter Predicted Images [%)
88 40
Fashion
FaaNon--RESee RESNETSO
CIFRR10 RESNETSO
Faatton- RESNETIS
2000
NICAL
Mediroiabbiing Fradtion
150
Predicted Imaggee/
a
CIFAR10 RESHET16
8
OIARR10 Cherr
1500
:00
1000
40
. 8 8 & 1
CIFAR100 RESNETE
Clocdl
CIFAR10
1000
800
0
Rondom
Gready & centar
Max estropy
Magin
CIFAR100
20
2000
4000
5000
Batch Size 1% Data Setj
Samples per Class
FIGURE
Total cost and
Fraction of machine labeled images (IS (D()))//|XI)
FIGURE 12:
FIGURE 13: Total cost and
suachine-labeling fraction of MCAL using different sampling methods (RESNET18, 10F
labeling fraction of MCAL using RESNETI8 on various subset sizes of CIFAR-10
using Naive AL for different 6.
DNN Architecture RESNETI8
Data Set
Labeling
CNN:1
RESNETS0
Service
Aupl
Cost (S)
Savings
Cost (S)
Savings
Agpl
Cost (S)
Savings
Fushion
Ammzon
1.7%
438.37
84.3%
6.7%
429.27
84.6%
3.3%
452.36
83.8%
Satyam
1.7%
49.87
76.3%
6.7%
20.77
75.3%
3.3%
63.86
69.6%
CIFAR
Amazon
10%
1377.24
34.3%
6.7%
10%
1128.65
53.0%
10
Saryam
16.7%
235.81
-31,0%
6.7%
129.60
28.0%
10%
149.63
16.9%
CIFAR
Amazon
13.3%
2520.79
-5%%
16.7%
2184.60
9.0%
13.3%
2915.80
-21 5%
Satyam
16.7%
407.04
-126.1%
16.7%
297.60
-65.3%
16.7%
805.76
334.66
TABLE 2: Oracle Assisted Active Learning
randomly selected samples per class. With 1000 samples per class, the majority of samples were used for training MCAL can machine-labe only 30%. With 5000 samples per class, this fraction goes up to 65%, resulting in increased cost savings relative to human-labeling the entire data set.
Dependence DNN architecture. While a larger DNN has the potential for higher accuracy, its training costs may be significantly higher and may potentially offset savings due to machine labeling. As seen from Figures 8-10, and 12, even though RESNET50 is able to achieve higher prediction quality and machine-labels more images, its high training cost offsets these gains. CNN18 18 on the other hand incurs much lower training costs, however, its poor performance leads to few images being machine-labeled. RESNETI8 provides for a better compromise resulting in overall lower cost.
MCAL Imagenet. We have explored applying MCAL Imagenet using EfficientNetBO (Tan & Le (2019)). Relative to CIFAR- 10 data set, Imagenet (with over 1.2 M images) is challenging for MCAL because: it has more classes (1000). fewer samples per class (avg. 1200), and the training cost is 60-200x higher than that for RESNET-18 For these reasons, MCAL trains the network to over 80% accuracy up 454K images, and decides, because cannot machine label any, to human-label the entire data set. As with CIFAR-100, for complicated data sets, MCAL still makes the correct decision.
Summary. Tbl. 2 provides the optimal choices for 8 and the optimal cost savings obtained for various DNNs. Comparing these values with Tbl. 1, we conclude that: MCAL outperforms Naive AL across various choices of DNN architectures and 5 by automatically picking the right architecture, adapting suitably, and selecting the right subset of images to be human labeled.
5.2 GAINS FROM ACTIVE LEARNING
Figures 14 and 15 ( see appendix SB) show the overall labeling cost with and without AL for the three data sets using Amazon and Satyam labeling services. The percentage cost gains are brackets. While Fashion and CIFAR- 10 show a gain of about 20% for both data sets, the gains are low in the case of CIFAR-106 because most of the images in that data set were labeled by humans and active learning did not have an opportunity to improve significantly. The gains are higher for Satyam, since training costs are relatively higher in that cost model: active learning accounted for 25-31 for Fashion and CIFAR-I0's costs, and even CIFAR-100 benefited from this.
'To make this decision, MCAL terminates when it has spent more than 2% (x = 10 in our evaluations) of the human labeling cost train the classifier hefore reaching the desired accuracy. So, for any highly complex data set for which it decides to human-label all samples, MCAL pays small "exploration tax" before termination.
8

---

Published as a conference paper at ICLR 2023
5.3 EFFECT OF RELAXING OTHER ASSUMPTIONS
Cheaper Labeling Cost, Intuitively, with cheaper labeling costs MCAL should use more human labeling train the classifier. This in turn should enable larger fraction of data to be labeled by the classifier. To validate this, we used the Satyam (Qiu et al. (2018)) labeling service (with 10x lower labeling cost). The effect of this reduction most evident for CIFAR- 100 in Tbl. as MCAL chooses to train the classifier using 57.6% of the data (instead of 32% using Amazon labeling service). This increases the classifier's accuracy allowing it to machine-laber 10% more of the data set. For other data sets, the differences are less dramatic (they use 2.5-5% more data to train the classifier, more details in appendix $C, Figures 16, 17, and 18), The numerical values the optimal as well as the corresponding cost savings provided in Table 2: MCAL achieves lower overall cost compared to all these possible choices in this case as well.
Relaxing Accuracy Requirement. In appendix $D, examine the quality-cost trade-off by relaxing the accuracy target from 95% to 90% to quantify its impact on additional cost savings. Fashion achieves 30% cost reductions; many more images are labeled by the classifier. CIFAR-10 and CIFAR- also show 10-15% gains.
6 RELATED WORK
Active learning (Settles (2010)) aims to reduce labeling cost in training a model, by iteratively selecting the most informative or representative samples for labeling. Early work focused on designing metrics for sample selection based on coreset selection (Sener & Savarese (2017)), margin sampling (Scheffer et al. (2001); Jiang & Gupta (2019)), region- sampling (Cortes et al. (2020)), max entropy (Dagan & Engelson (1995)) and least confidence (Culotta & McCallum (2005)). Recent work has focused on developing metrics tailored to specific tasks, such as classífication (Coleman al. (2020)), detection (Brust al. (2019)), and segmentation (Yang et al. (2017)), or for specialized settings such as when costs depend upon the label (Krishnamurthy et al. (2017)), or for a hierarchy of labels (Hu et al. (2019)). Other work in this area has explored variants of the problem of sample models (Coleman et al. (2020)), model assertions (Kang
selection: leveraging selection-via-pron al. (2020),, model structure (Wang al. (2017)), using model ensembles improve sampling efficacy (Beluch et al. (2018)), incorporating both uncertainty and diversity (Ash et al. (2020)), or using sepe-supe mining of samples for active learning to avoid data set skew (Wang et al. (2018)). MCAL uses active learning and can accommodate multiple sample selection metrics.
Training cost figures prominently in the literature on hyper-parannn chitecture search. Prior work has attempted to predict learning curves to prune hyper-parameter search (Klein et al. (2017)), develop effective search strategy within a given budget (Lu et al. (2019)), or build a model to characterize maximum achievable accuracy to enable fast triage during architec- ture search (Istrate et al. (2019),, all with the goal of reducing training cost. The literature on active learning has recognized the high cost of training for large datasets and has explored using cheaper models (Coleman et al. (2020)) to select samples. MCAL solves a different problem (dataset labeling) and explicitly incorporates training cost in reasoning about which samples to select.
tuning, especially for ar-
Also relevant is the empirical work that has observed a power-law relationship between generalization error and training set size (Beery et al. (2018); Johnson et al. (2018); Figueroa et al. (2012)) across wide variety of tasks and models. MCAL builds upon this observation, and learns the parameters of a truncated pawer-law model with few samples possible.
CONCLUSIONS
Motivated by the need of data engineering for increasingly advanced MI. applications as well as the the prohibitíve human labeling cost, this paper asks: "How to label data set at minimum cost"? To do this, it trains a classifier using a set B from the data set to label S samples, and uses humans to label the rest. The key challenge is to balance human- (B) vs. machine-labels (S) that minimizes total cost. MCAL jointly optimizes the selection by modeling the active learning accuracy vs. training size as truncated paw, and search for minimum cost strategy while respecting error constraints. The evaluation shows that it can achieve up to 6x lower cost than using humans, and is always cheaper than active learning with the lowest-cost batch size.

---

Published as a conference paper at ICLR 2023
REFERENCES
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds, In International Conference on Learning Representations, 2020.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In The European Conference on Computer Vision (ECCV), September 2018.
William H. Beluch, Tim Genewein, Andreas Nürnberger, and Jan M. Köhler. The power of ensembles for active learning in image classification. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Clemens-Alexander Brust, Christoph Käding, and Joachim Denzler. Active learning for deep object detection. Proceedings of the 14th International Joint Conference Computer Vision, Imaging and Computer Graphics Theory and Applications, 2019.
Stephen M. Burroughs. Upper-truncated power laws and self-similar criticality in geophysical processes. PhD thesis, University of South Florida, July 2001.
Junghwan Cho, Kyewook Lee, Ellie Shin, Garry Choy, and Synho Do. How much data is needed to train a medical image deep learning system achieve necessary high accuracy?, 2015.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations, 2020. URL https: / //openzeview. wwwp://eeeeetttittteeteeeeit..ee http://weeee....oooe---g00000000 BdgBb.5bV.DD..
Corinna Cortes, Giulia DeSalvo, Claudio Gentile, Mehryar Mohri, and Ningshan Zhang. Adaptive region-based active learning, 2020.
Aron Culotta and Andrew McCallum. Reducing labeling effort for structured prediction tasks. volume 2, pp. 746-751, 01 2005.
Ido Dagan and Sean P. Engelson. Committee-basee sampling for training probabilistic classifiess In In Proceedings of the Twelfth International Conference on Machine Learning, pp. 150-157. Morgan Kaufmann, 1995.
Rosa Figueroa, Qing Zeng-Tred Treitler, Sasikiran Kandula, and Long Ngo. Predicting sample size required for classification performance. BMC medical informatics and decision making, 12:8, 02 2012.
Figure-Fight. Figure eight data annotation platform. 2021.
com/,
Google. Google ai platform data labeling service. https://cloud.googg com/ data-Labeling/docee 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.
Peiyun Hu, Zack Lipton, Anima Anandkumar, and Deva Ramanan. Active learning with partial feedback. In International Conference on Learning Representations, 2019.
R. Istrate, F. Scheidegger, G. Mariani, D. Nikolopoulos, C. Bekas, and A. C. I. Malossi. Tapas: Train- less accuracy predictor for architecture search. Proceedings of the AAAI Conference Artificial Intelligence, 33:3927-3934. 3934, Jul 2019. ISSN 2159-5399, doi: 10. 1099/aa/aai....m wat.aaaaii.ii...3.... 33301.33113397 URL http://dt doi org 100 9 aa00/ v33i 01 33013927.
10

---

Published as a conference paper at ICLR 2023
Heinrich Jiang and Maya Gupta. Minimum marginn active learning, 2019.
Mark Johnson, Peter Anderson, Mark Dras, and Mark Steedman. Predicting accuracy large datasets from smaller pilot data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 450-455, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 12158/6/11/118-07222 URL https: //www. aclweb. org/ http//aathooooyl-eoeae
Daniel Kang. Deepti Raghavan, Peter Bailis, and Matei Zaharia. Model assertions for monitoring and improving ml models. In I. Dhillon, D. Papailiopoulos, and V. Sze (eds.), Proceedings of Machine Learning and Systems, volume 2, pp. 481-496. 2020.
Keras. Keras Documentation: Training Resnet on CIFAR-10.
1effar0_rreset/// 2021.
Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. In ICLR, 2017.
A. Krishnamurthy, A. Agarwal, T-K Huang. H. Daumé III, and J. Langford. Active learning for cost-sensitive classification. CoRR, abs/1703.00014, 2017. URL 1703.01 01014.
Zhiyun Lu, Liyu Chen, Chao-Kai Chiang, and Fei Sha. straint. Proceedings of the Twenty- International Joint Conference on Artificial Intelligence, Aug 2019. doi: 10.24963/jjci,2219999 2019//99.
tuning under budget con-
Hang Qiu, Krishna Chintalapudi, and Ramesh Govindan. Satyam: Democratizing groundtruth for machine vision. arXiv preprint arXiv: 1811:022 2018.
SageMaker. Amazon sagemaker data labeling. https:/ /occs aws amazon. com/ Sagemakerr/atest///ee dss///////////////////////////// sms ddata-labelle html, 2021.
Vittorio Sala. Power law scaling of test error versus number of training images for deep convolutional neural networks. In Proc. of SPIE, volume 1 11059 of Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, pp. 105914, Jun 2019.
doi: 10.1117/12.25258111
Tobias Scheffer, Christian Decomain, and Stefan Wrobel. Active hidden markov models for infor- mation extraction. CAIDA '01, pp. 309-318, Berlin, Heidelberg, 2001. Springer- Springer-Verlag. ISBN 3540425810.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach, 2017.
Burr Settles. Active learning literature survey. Technical report, 2010.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6005-6144, PMLR, 2019.
Keze Wang, Dongyu Zhang, Ya Li, Ruimao Zhang, and Liang Lin. Cost-effective active learning for deep image classification. IEEE Transactions Circuits and Systems for Video Technology, 27 (12):2591-2600, Dec 2017. ISSN 1558-2205.
Keze Wang, Xiaopeng Yan, Dongyu Zhang, Lei Zhang, and Liang Lin. Towards human-machine cooperation: Self-supervised sample mining for object detection. CoRR, abs/18033.00) 909609867, 2018.
Gent Wolf. Facility location:
concepts, models, algorithms and case studies. series: Contributions to management science: edited by zanjirani farahani, reza and hekmatfar, masoud, 2011.
Lin Yang. Yizhe Zhang. Jianxu Chen, Siyuan Zhang, and Danny Z. Chen. Suggestive annotation: A deep active learning framework for biomedical image segmentation. Lecture Notes Computer Science, pp. 399-407, 2017. ISSN 1611-3344
11

---

Published as a conference paper at ICLR 2023
A MCAL ALGORITHM
Algorithm MCAL
Inputs:
An active learning metric M (.). classifier D, set of unlaheled images (X), parametric model to prediet (2(D(B) and parametrie model to predict c(5 (P()))
1: human generated
randomly sampled
V..
2: Obtain human generated labels for randomly sampled data items Bg X. [Bo| 6a.
3: Train 2 D)) and test classitier over
4: Recurd training cost C2(D(Bp)
5: for E (Omins.... (muzz) do
6: Fsstimate or (5" (Doo)) using and M(-)
7: end for
8: Initialization: Cee O, Cold 0, 6o. Bupt Bo
9:
while
C(oppt + 8)
10:
human generated labels c X B;-1 comprising [b,| = samples ranked using M()
11: B. B.-1 Ub;
12: Train (B,)) and lest clamsifier over
13:
estimate (P(()) using Bxl, Cx(D(B))) for 0 E 10min... -._ Omazi do
Record C₁ (P).:))
14:
15:
Estimmate (s (D(())) using M()
16:
Fstimate and update
model parameters (0g.26, 26, from Eqn.
c(SS()(B))
using
(IBEl, oT (s" (D(B.))
17:
For
18:
Find C C", Bupt described in Section
19:
if ([Cr.e C A then
20:
argming Supt (Boppel
21:
6apt
22:
end
23:
C. C.ew
24:
1+1+1
25:
end while
26:
Use D(Bopt) and L(-) to (D, Bopt)
27:
Annotate residual (' Bopt)
B COST SAVINGS ON ACTIVE LEARNING
Figures 14 and 15 show the overall labeling cost with and without AL for the three data sets using Amazon and Satyam respectively The percentage cost gains are in brackets. While Fashion and CIFAR-10 10 show a gain of about 20% for both data sets, the gains are low the case of CIFAR-100 because most of the images in that data set were labeled by humans and active learning did not have an opportunity to improve significantly. The gains are higher for Satyam, since training costs relatively higher in that cost model: active learning accounted for 25-31% for Fashion and CIFAR-10' 10's costs, and even CIFAR-106 benefited from this.
MCAL With AL
(-1.5%)
Total Cost to Label [$]
IMCAL With (7.9%)
Total Cost to Label
2000
MCAL Without AL
IMCAL Without AL
150
1500
100
(31.7%)
1000
(19.1%)
(25.5%)
500
(21.4%)
Fashion CIFAR-10 CIFAR-100
Fashion CIFAR-10 CIFAR-100
FIGURE 14: Active learning gains using Amazon La- FIGURE 15: Active learning gains using Satyam La-
beling
beling
12

---

Published as a conference paper at ICLR 2023
C RESULTS ON CHEAPER LABELING SERVICE
Figures 16, 17, and 18 depict the effect of using Satyam as the labeling service. As seen these figures, the lower labeling cost alters the tradeoff curves. The figures also depict the corresponding MCAL cost as well as the human labeling cost for reference. MCAL achieves a lower overall cost compared all these possible choices in this case as well.
D EFFECT OF RELAXING ACCURACY REQUIREMENT
Tbl. depicts the fraction images that were machine-labeld by the classifier [S| and the number of samples used to train it [[B]|X]] for each of the data sets. Comparing with Tbl. 1, for Fashion MCAL predicts more images by using smaller number training images. For CIFAR-10 and CIFAR-100, it makes different decision and uses more training images to increase the classifier accuracy to enable more images to be machine-labeled. Further, RESNET18 continues to be the optimal architecture for all three data sets. As seen from Tbl. 3, MCAL ensures the accuracy target of 90% for all the data sets. Tbl. 3 also captures savings with respect to human labeling while using an accuracy guarantee of 90%. However, the savings do not dramatically increase indicating that most of the cost gain comes in reducing the accuracy requirement to 95% from 100%.
Data Set
S X
DNN
Labeling
Cost
Selected
Accuracy
Savings
Fashion
4.4%
90.0%
RES18
91.9%
88.9%
CIFAR-IO 10
25.9%
75.0%
RES18
94.7%
70.5%
CIFAR- 100
64.0%
25.0%
RES18
98.4%
39.1%
TABLE 3: Relaxing Error Constraints to 10%
E TRAINING COST
Fig. 19, Fig. 20, Fig. 21 shows the training cost portion from the total cost (Fig. 8,Fig. 9, Fig. 10). The training cost can vary significantly with 5. While for Fashion, there is a 22 reduction in training costs, it is about 5x for CIFAR-10 and CIFAR-100 data sets.
F POWER-LAW AND TRUNCATED POWER-LAW FIT
In this section, we show power law and truncated power law fitting results on all combinations of datasets and models. Fig. 22, Fig. 23, Fig. 24 show fitting results CIFAR-10. Fig. 25, Fig. 26, Fig. 27 on CIFAR-100. As an example, we show the fitting results on the error profile of f = 50% in all figures in this section. In all combinations of datasets and models, while using more points gives higher accuracy and better prediction, both power-lav and truncated power-law can get stable and precise prediction using very limited number of small sample points.
13

---

Published as a conference paper at ICLR 2023
250
1000
3000
CHNVE
CNNIE
CHNNNIE RESNETI0 RESNETS0
Total Cosl to (8)
200
Human Labeling
RESNETIO T10
RESNETS0
Total Total Cost to Label [5)
800
C RESNET1B
RESNETSO
Total Cost to Labal
B2500
Human Labeling
2000
1.50
MCAL
600
Human Labeling
1500
100
400
MCAL
50
200
100
10
15
20
10
20
Batch (6) [X]
Batch (4) [K]
AL Batch Size 1% [K]
FIGURE 16: Performance of MCAL compared to active learning with dif- ferent batch sizes on Fashion us- ing Satyam labeling
FIGURE 17: Performance of MCAL compared to active learning with dif- ferent batch sizes on CIFAR-10 using Satyam labeling
FIGURE 18: Performance of MCAL compared to active learning with dif- ferent batch sizes on CIFAR- 100 using Satyam labeling
100
1000
2000
Total AL Training Cost 85]
CNNNE RESWEESO
Total AL Training Cont (B3
RESNETSO
RESNETIO
RESNETIS
RESWETSO
600
Total Training
8000-
500
10
15
10
15
20
10
15
20
AL Batch Size 1% of Data Set]
AL Batch Size 1% Data Set]
AL Batch Size [% of Data Swi]
FIGURE 19: AL training cost (in
FIGURE 20: AL training cost (in
FIGURE 21: AL training cost (in
5) as a function of batch size (8)
$) as a function of batch size (6) for
$) as a function of batch size (6) for CIFAR- 100 Data Set using CNNI8, RESNETIS and RESNET50.
for Fashion Data Set using CNNI8,
CIFAR-10 10 Data Set using CNN18, RESNETI8 and RESNET50.
RESNETIS and RESNET50.
A30-
EITO moll
Bror rofic
paints
0/55
2.10
EFTS rofflc
DORMS
DECETT Oruncsted) CENTEA
points (unnctted) panta
poRnts Imuncsted)
D.C1
ponta
SOMES Bruncttedi
penns bruncated
panns (buncated)
prant
paints
panes
prast Drsoratedi
parts (mncsttd)
pants [manosted)
paints
pantt
pontaa Oruncaled)
pants (nuncalad)
pasts (iuncalad)
0.01
2003
4000
9000
39200
2000
3200
5020
19200
2020
9000
5020
39200
Tamig 20
Tasing Sue
T Tanin Ste
FIGURE 22: Pawer-law and Trun- cated Power-law fits on CIFAR-10 using CNN18
FIGURE 23: Power-law and Trun- cated Power-law fits on CIFAR-10 using RESNET18
FIGURE 24: Power-law and Trun- cated Power-law fits on CIFAR- 10 using RESNETS0
EITEr Picfee
BTO reffic
3 polroo
paiee
poire:
posts (inacette)
paints Stuncates!
partts Druncatess
poins
paintz
points 20luncates
2000
4200
8000
18200
2000
0002
8000
18200
2000
0002
8000
16200
Trininn 2:2
Bainin S25
Bainnn Sic
FIGURE 25: law and Trun- cated Power-law fits on CIFAR- 100 using CNN18
FIGURE 26: Pawer-law and Trun- cated Power-law on CIFAR-100 using RESNET18
FIGURE 27: Power-law and Trun- cated Power-law fits on CIFAR- 100 using RESNETSO
14


Reviews:
[{"review_id": "tkaah9MHmSB", "reviewer": "Reviewer_hXWK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a system for training of machine learning classification models. The main focus\nof the paper is to reduce the overal cost of training. The total cost considered in the paper is a\ncombination of the cost due to human labeling and the cost due to computation (e.g. renting a GPU on\nAmazon Cloud).\n\nIn order to reduce the training cost the paper proposes to rely on combination of active learning and\nautomatic labeling.\n\nThe training in the proposed MCAL approach proceeds iteratively. On each iteration the approach\nselects which examples should be labeled by human annotators, and which examples are assigned labels\nautomatically using a classifier trained on previously labeled data (using automatic + human labels).", "summary_of_the_review": "I refer to my text in the \"Strength And Weaknesses\" section. I would like to avoid repeating myself here.", "strength_and_weaknesses": "I generally have a negative impression about the paper. I can see that the approach is trying to\naddress a valid practical problem. The overall approach sounds very much like a common sense, i.e.\nlet's label some data points with already pre-trained model, and let human's label \"interesting\"\nexamples. The paper relies on standard metrics to assess what to give humans for labeling and how to\nchoose examples for automatic labels (i.e. examples where classifier is confident about the label).\n\nThe main technical contribution of the paper appears to be the use of the power law to predict how\nmany examples to label automatically (Sec. 3.1) and the overall system (Sec. 4).\n\nAfter going through the paper several times, what I am left with is the following: The paper\npresents an idea that sounds like a common sense, which is followed by hard to comprehend notation\nand cost model (Sec. 2,3), followed by description of standard mechanisms of sample selection (Sec.\n3.3), followed by hard to understand Sec. 4 that describes the algorithm.\n\nThe text does not communicate the ideas well unfortunately, and at least in my case it leaves the\nreader unexcited about the approach. Perhaps using clear real-world examples would help here.\n\nFor example, I don't understand why power law in Sec. 4 even applies to the approach presented in\nthe paper. I can see how such power law works for increasingly larger training sets labeled by\nhumans. However, why does this law apply to automatically labeled examples, where adding larger\nnumber of examples would result in adding more of the incorrectly labeled examples.\n\nI addition I am not taking it for granted that \"auto-labeling\" generally works. Why is automatically\nlabeling examples where classifier is already confident going to help with classifying examples\nwhere classifier is not confident? I understand that it might be the case, but I disagree that this\ncan be taken for granted.\n\nPrior to reading this paper I believed that the main questions for the type of approaches proposed\nin the paper is how to choose what to label manually, and how to choose what to label automatically.\nIf we knew how to do that well we would be in good shape to construct such systems. The paper does\nnot contribute to addressing these questions, and the paper didn't convince me that it makes other\nvaluable contributions.\n\nI found some statements in the paper which I would like to have clarified:\n1. \"it chooses to label CIFAR-100 mostly using humans\",\nI am very curious why this is the case, and why similar outcome does not happen for ImageNet.\n\n2. \"MCAL’s goal is to select samples to train a classifier that can machine label the largest number\nof samples possible.\"\nThat is certainly a good goal, but I do not understand how the approach achieves that. All I see in\nthe text is some application of the power law, whereas the problem above appears to be too complex to be\naddressed by a power law.\n\n3. \"When MCAL uses margin or least confidence for L(.), the samples selected have high accuracy (close to 100%, Fig. 5)\n. The samples selected by a core-set based algorithm such as k-centers is poorly correlated\nwith accuracy (Fig. 5) and margin (Fig. 6).\"\nI din't quite understand what is meant by these sentences. Why is accuracy relevant here if we are\nselecting examples to be labeled and to be used for training of the next-iteration classifiers?\nAlso, if we select examples that have high confidence already why would such examples improve the\nclassifier?\n\nFinally, I also doubt that the topic of the paper is within the scope of ICLR. For me ICLR stands\nfor interesting new representations (i.e. neural network models), or perhaps interesting techniques\nfor training neural networks. This paper sounds like a system paper that would perhaps be a good fit\nfor WACV (Winter Conference on Applications of Computer Vision) or some other more applied\nconference (e.g. International Conference on Computer Vision Systems).\n\nPositive:\nThe overall observation that in addition to human labeling cost there is also a computational cost is\nvaluable.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jO-FlwTQX6X", "reviewer": "Reviewer_Dg4t", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an ‘active labeling’ (inspired from active learning) based approach to automatically label parts of a dataset given some error bounds by jointly optimizing for human labeling and machine labeling costs (eg. training costs) together. It formulates the problem as a minimum-cost labeling problem in an optimization framework that minimizes total cost by jointly selecting which samples to human label and which to machine label. A major assumption of this work is that it considers the truncated power-law distribution to mimic how model error behaves with training set size. ", "summary_of_the_review": "This paper presents some practical ideas for cost reduction for data labeling which is growing need today. The work presents a solid extensible framework where one can set various thresholds for performing the labeling tasks. The methods have been testing on multiple datasets and understanding the role of various parameters have been explored and discussed well in the paper.", "strength_and_weaknesses": "Strength: \n- The paper presents approach that allows control over accuracy and cost with the cost minimization objective to launch data labeling task for developing a high accuracy model training.\n- The approach scales well with complexity of the data and also suggests (with a ‘exploration tax’) if it’s feasible to automatically label the data or not (given the size and complexity)\n- The approach can use any active learning metric for selecting samples for human-labeling. Also they found empirically that some popular metrics (like coreset) don’t work as well as others with this objective. \n- The approach is extensible to allow multiple model architectures \n\nWeakness:\n- It is not clear if the approach would work with a dataset containing imbalanced distributions and/or open classes to decide which ones to human-level vs machine-label. Also not clear if the power-law distribution would follow in the situation. \n- Fig. 4 needs more explanation - the role of batch size for model errors is low which is unintuitive. \n- The human-labeling is assumed to be oracle in this work - how realistic is this assumption - if the model can accommodate imperfect oracles in human labels, the method would become more realistic. \n- Fig. 6 needs more explanation \n- It is not clear how does ‘fast search’ work in the method. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HAIh53XkBS", "reviewer": "Reviewer_9gqM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors presents a novel active learning setup using a \"hybrid human-machine labeling\" framework to reduce the cost of labeling/annotating large datasets. They compared their approach on a number of public datasets and claims to provide a 6x overall reduction in cost to entire human labeling of the dataset.", "summary_of_the_review": "Overall, the paper discusses an intuitive and simple solution to gather ground truths for large datasets and reduce the overall labeling costs for the process. While there are avenues for improving the paper, the results are impressive and general enough to have a significant impact.\n\nApart from the previously highlighted strengths and weakness, the authors may also want to address the following:\n\n- Section 3.1, the intuition for the upper-truncated power-law could be motivated better. For example, a plot showing the effects of various values of the parameters from equation (3) could be useful to understand the desired shaping of the rewards\n- MCAL Algorithm -provide better details about the fitting procedure. Discuss about the cost for the optimization function\n- Provide intuitions behind the sample selection - e.g. how does the ranking of samples selected change with reduced labeling cost etc. Investigate possible issues with fairness / group fairness in selection of samples\n- In recent years, a number of papers have identified labeling/data issues in the original benchmarks. The authors may want to compare if their proposed solution is able to circumvent those issues", "strength_and_weaknesses": "Some of the key strengths of the paper are as follows:\n\n- The authors presented a general solution to a very important problem. With the impressive performance of AI/DL models for various tasks, the proposed method could be very important in operationalizing AI for real-world problems. Further, the MCAL Algorithm is not dependent on the underlying DL architecture / task and thus could be applied as a meta learner over multiple problems\n- The authors have presented evaluations across multiple datasets/publicly available benchmarks to support the claim about the effectiveness of the solution\n- Further, the detailed analysis about the reduction in labeling cost from multiple view points provides better insights for the method to be operationalized\n\nThe paper could improve upon the following aspects\n\n- The presentation of the paper can be improved upon. The two components of MCAL algorithm is hard to follow. Symbols are sometime introduced before they are properly defined (e.g. delta in section 3.1, 3rd line from end of the section; first introduced in 3.2) and intuitions behind certain selections not presented well\n- While the authors provide interesting insights about the drivers of the reduced cost, the discussion is lacking depth when considering the problem in a holistic manner. For example, in addition to labeling cost, other metrics of interest can include turnover time which can be affected as part of this multi-loop process\n- In a similar manner, more details/analysis on which samples are selected during the AL phase would be useful. The authors presented a comparative analysis of the sample selection candidates - however, the intuition and effect of the selected samples may need further analysis. This could also be useful for downstream tasks in identifying \"good enough\" solutions under budget constraints", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster