The Speed-up Factor: A Quantitative Multi-Iteration Active Learning Performance Metric
Authors: Hannes Kath
Thiago S. Gouvêa
Daniel Sonntag
Venue: Accepted by TMLR ()

Abstract: Machine learning models excel with abundant annotated data, but annotation is often costly and time-intensive.
Active learning (AL) aims to improve the performance-to-annotation ratio by using query methods (QMs) to iteratively select the most informative samples.
While AL research focuses mainly on QM development, the evaluation of this iterative process lacks appropriate performance metrics.
This work reviews eight years of AL evaluation literature and formally introduces the speed-up factor, a quantitative multi-iteration QM performance metric that indicates the fraction of samples needed to match random sampling performance.
Using four datasets from diverse domains and seven QMs of various types, we empirically evaluate the speed-up factor and compare it with state-of-the-art AL performance metrics. 
The results confirm the assumptions underlying the speed-up factor, demonstrate its accuracy in capturing the described fraction, and reveal its superior stability across iterations.

Published in Transactions on Machine Learning Research (annury//2200
The Speed-up Factor: A Quantitative Multi-lteration Active Learning Performance Metric
Hannes Kath
hannes.kath @dfki.
Deportment of Applied Artificial Intelligence University of Oldenbury
Oldenburg, Germany Deportment of Interactive Machine Learning German Research Center for Artificial Intelligence (DFKT) Oldenburg, Germany
Thiago S. Gouvéa
thiago.gouee gowvea @dfki.
Department of Interective Mochine Learning German Research Center for Artificial Intelligence (DFKT) Oldenburg, Germany
Daniel Sonntag
daniel. & de
Department of Applied Artificial Intelliencc Universily of Oldenbury
Oldenburg, Germany
Department of Interactive Machine Learning German Research Center for Artificial Intelligence (DFKT) Saarbrücken, Germang
Reviewed OpenReview: https: //
https://openrevieu openreview. net/ forum? id= g6hRb6fETo
Abstract
Machine learning models excel with abundant annotated data, but annotation is often costly and time-intensive. Active lcarning (AL) aims to improve the perfornancsto-ammotateee ratio by using query methods (QMs) to iteratively select the most. informative samples. While AL research focuses mainly on QM development, the evaluation of this iterative pro- coss lacks appropriate performance metrics. This work reviews eight years of AL evaluation literature and formally introduces the speed-up factor, quantitative multi-iteration QM performance metric that indicates the fraction of samples needed to match random sam- pling performance. Using four datasets from diverse domains and seven QMs of various types, we empirically evaluate the speed-up factor and compare it with state-of-the-ar AL performance metrics. The results confirm the assumptions underlying the speed-up factor, demonstrate its accuracy in capturing the described fraction, and reveal its superior stability across iterations.
1 Introduction
Machine learning focuses developing and training models that identify general patterns from given data. These models perform exceptionally well when trained on large amounts of annotated data, carning them the term data hungry (Van Der Ploeg et al. 2014). While data collection is straightforward in many applications, data annolation tends to be time-consuming and costly. Current research aims to maximize model performance while minimizing human annotation effort.
A leading approach to reducing annotation costs is active learning (AL) Instead of randomly selecting unlabelled data samples for annotation, AT. query methods (QMs) choose samples that optimize model

---

Published in Transactions on Machine Learning Research (annuayy/2200
1 1-1+6
D
DAL
Di=1
Di
(2)
(1)
Oj
Q
(3)
D
D'
Di
Figure 1: Active learning [AL] The dataset D is split into the evaluation set DE and the AL set DAL- (1) A query method uses the current unlabelled dataset Di= and labelled dataset Dj_11 to query samples (Qt) from Di (2) Qi is labelled (Q)) by human expert. (3) Update datasets: Di- Di- Oi and
D2 =
-D) D/.-. U2.
performance, improving the informativeness of the labelled dataset and enabling faster convergence with fewer annotations (Settles] 2009). Figure illustrates the AL process from the data perspective. The dataset D is initially divided into the representative evaluation set DE and the AL set DAL. At iteration t, Di and D denote the previous unlabelled and labelled dataset. (1) A query method QM(D' " selects n (Di) -i ,1) samples from D/=1 to form the unlabelled query 2 where n the query size, also called batch size. (2) O6 is labelled by human experts and becomes 2i. (3) Datasets are updated: Di - Oi and Dj, DD U 24, The model o' (Dir D2,) ) is trained and the performance p*("*, DE) on De is assessed. The next iteration begins unless predefined stopping criterion is met. Common stopping criteria include the labelling budget. [ the performance level Pt, or the allocated annotation time.
QMs can be categorized into three types of methods: Prediction- based methods utilize model performance to select the next query, e.g., uncertainty sampling prioritizes samples near the decision boundary. based methods choose samples based on the internal data structure, e.g., diversity sampling selects queries that reprosent the entire input space. Model-based methods focus on changes in the model, c.g., prioritizing samples with high model influence, irrespective of their labels (Tharwat & Schenck 2023). While research on AL primarily focuses on developing more efficient QMs) relatively little attention has been paid to their empirical evaluation. QQM evaluation is typically conducted by emulating [LL a labelled dataset is used, with all labels concealed from the model and revealed only when the respective samples are selected by the QM for annotation. In contrast to most machine learning lasks evaluated on final performance, AL requires evaluation across multiple iterations. Yet, the literature lacks a quantitative metric that evaluates QM performance across multiple itcrations.
This work roduces the speed-up factor, a quantitative multi-iteration QM performance metric that quan- tifies the fraction of samples a QM needs to reach random sampling performance. Section 2 provides a literature revicw of the past eight years on AAL ovaluation. Scction Scetion3formaa formally introduces the proposed meth- ods for AL evaluation. Section !| outlines the experimental setup for our empirical evaluation. Section 5 describes the experimental results. Section 0 analyzes the results. Section concludes the study.
2 Related Work
We provide a comprehensive overview of empirical qquery method (QM) evaluation through structured and unstructured literature reviews, focusing on evaluation criteria, comparison baselines, and result presentation.
The structured litcrature review covers proccedings from six major artificial intelligence conferences: the Association for the Advancement of Artificial Intelligence (AAAI), the Conference on Computer Vision and Pattern Recognition (CVPR), the International Conference Learning Representations (ICLR), the International Conference on Machine Learning (ICML), the International Joint Conference on Artificial Intelligence (IJCAI), and the Conference on Neural Information Processing Systems (NeurIPS). The search criteria include publications from 2017 to 2024, with the term factive learning' appearing in the title, abstract, or keywords.
Table summarizes the quantitative findings. The selection includes 449 papers. Excluded papers are those: focusing on related tasks (49); applying AL without. evaluation (37); evaluating AL aspects other than
2

---

Published in Transactions on Machine Learning Research (January 22026)
Table 1: Results of the literature review on active learning evaluation, showing the total number of papers identified through the selection criteria, those empirically evaluating query method (QM), introducing new QM, evaluating specific criteria, using particular baselines for comparison, and presenting evaluations in defined formats.
Evaluation Presentation
Evaluation Criteria
Baselines
On Stopping Criterion
On Each Iteration
Other
Venue
Total Eval New Perfor- Process Positive
Random Other Budget Performance
Performance Performance Location Ablation over Budget
QM
QM
mance
Time Samples Sampling QM((s
over Time Samples
Study
AAAI
104
68
62
68
13
2
47
65
35
58
2
5
20
CVPR
55
49
41
49
3
44
47
35
8
45
0
9
30
ICLR
48
35
33
35
0
32
33
21
31
0
9
14
ICML
76
44
38
44
4
0
36
42
25
3
40
0
2
2
DCAI
46
35
34
35
0
22
34
15
3
30
0
3
5
NeurIP
113
70
67
70
14
2
55
63
36
5
62
9
23
Other
7
0
5
0
0
0
Σ
449
306
280
306
57
7
241
289
169
26
271
3
37
Q (25), e.g., model, complexity, or social factors; evaluating QMs only theorctically (22); not about AL
(8) e.g., mentioning 'interactive learning'; or surveys of AL (3). A total of 306 papers empirically evaluate
MMs with 280 introducing a new QM
2.1 Active Learning Evaluation Trends
Evaluation Criteria. All 306 papers empirically evaluating QMs criterion, highlighting its central importance in [LL The performance of a single iteration is primarily mea- sured using accuracy (170), F1 score (44) and mean average precision (30). Additionally, 57 papers evaluate computational requirements using QM processing time, and 7 papers use the number positive samples selected by the QM
include performance as evaluation
Baselines. The performance of QM is primarily evaluated by emulating AL most frequently random sampling (241).
ALand and comparing it to baselines,
Evaluation Presentation. Presenting the evaluation results of QM performance not trivial. QMs aim maximize the performance-to-buugee ratio. The simplest approach using either budget or performance as the stopping criterion and comparing the numerical result for the other variable against baselines in the final iteration, when the criterion is met. In literature review, 169 papers compare the final performance using budget as the stopping criterion, while 26 papers compare the final budget using performance the stopping criterion. While it is straightforward to present evaluations for the final iteration, there are significant caveats to this approach: The stopping criterion varies by setup, making QM results highly dependent on when AL is stopped. Performance differences between QMs change over iterations and generally decrease in later iterations as all QMs converge to the same performance level. Thus, focusing solely on results from a single iteration fails adequately represent the overall process. A more comprehensive approach is to present the performance for each iteration. The most common method, used in 271 papers, is to present learning curve that plots performance loss against the size of the labelled dataset. Learning curve evaluation is performed qualitatively through visual inspection. While effective for few clcarly separated curves, visual inspection lacks quantitative, comprehensible results and poses challenges when comparing multiple QMs across different datasets (Pupo et al. 2018a). Another qualitative method, used in 37 papers, involves visualizing the locations of selected samples from a QMs in latent space, using dimensionality reduction techniques such t-SNE or UMAP. Another quantitative method, used in 94 papers, involves evaluating specific components of QM in isolated environments through ablation studies.
3

---

Published in Transactions on Machine Learning Research (January /2026)
2.2 Active Learning Evaluation Methods
Single-Iteration Evaluation. Research on enhancing empirical QM evaluation primarily focuses on single-iteration assessments, presented either quantitatively for single, typically the final, iteration qualitatively as learning curve. This research can be categorized into the development of new metrics, methods, or frameworks for improving empirical e Metric development aims to design metrics that more effectively capture [M performance than traditional measures such as accuracy or F1 score. [aa- oub et al. (2017) introduce the efficiency score, defined as the ratio of test accuracy to budget size, to identify the most efficient. budget size for annotation. (Shi el al. (2021) introduce the label-wise average precision improvement, defined as the difference between the average precision of the [Q and the baseline, divided by the average precision of the bascline. Rather than introducing new performance metric, five papers in our literature review (e.g., (Hartford et al. 2020 Kim et al.) 2021 present the percentage performance improvements of Q over random sampling, instead of presenting metric values as learning curves. Met development aims to enhance the validity of the evaluation process. Kong et al. (2022) argue that due to the abundance of large unlabelled datasets in [AL] scenarios, evaluation should be compared to semi-supervised learning, which incorporates both labelled and unlabelled data, rather than to supervised learning. Other papers use statistical significance tests, such as the t-test (Hu & Zhang, 2018 Mohamadi et al. 2022) or
the Mann-Whitmne U-test, (Bullard et al., 2019) to evaluate the signilicance of performance differences be- tween QMs Framework development aims to provide clear guidelines for the comprehensive, quantitative, and reproducible derivation of results. By reimplementing AL experiments, Munjal et al. (2022) find that reliable empirical QM evaluation requires repeating experiments under varying training settings (e.g., model architecture, budget size), incorporating regularization techniques, and tuning hyper-parameters at each iteration. Building on this, Lüth et al. (2023) propose an evaluation framework that suggests assessing QMs across various datascts, including imbalancod ones, with different initial budgets and query sizes. The framework suggests luning hyper-pec paeer-paameeerrs on the starting budget and keeping them fixed, since grid-search fine-tuning at each iteration, as proposed by Munjal et al. (2022) is impractical due to high computational requirements. The framework additionally evaluates QMs against self-supervised learning, semi-supervised learning, and their combinations with QMs
Multi-Iteration Evaluation. Riis et al. 2022 highlight the lack of a metric to assess QM performance
across iterations, without offering solution. Simple approaches for quantitative metrics that cover multiple
iterations include averaging the learning curve (Tang & Huang 2022 Gao et al., 2018 and caleulating
the area under the learning curve (Cawley 2011 Pupo et. al. 2018b Pupo & Ventura 2018). Pupo et al.
(2018a) observe that these metrics neglect crucial information about intermediate results and proposes more sophisticated evaluation method. Based on the assumption that the performance difference between two Q.s ideally increases over successive iterations, this difference is calculated for each iteration (Lermed cut-point), and the iterations are ranked according to the magnitude of this difference. A QM is considered superior if a statistically significant correlation exists between the obtained ranking and the ideal descending ranking. While these metrics quantify results across multiple iterations, Kath et al. (2024b) highlight their high sensitivity the number of evaluated iterations. Once the saturation point is reached, performance tends to stabilize. Subsequent iterations continuously equalize and increase both the mean value and the area under the learning curve for different QMs and also randomize the ranking of performance differences, resulting in low stability of the metrics when evaluated across different iteration numbers. Without presenting theoretical or empirical validation, Kath et al. (2024b) use a metric similar to the speed-up factor to evaluate AL performance on bioacoustic data. Using the approximation function p(x) = (1 - e ē-²), where á represents the performance, the ceiling performance (computed using all samples for training). and a the number of labelled training samples, the scaling parameter is fitted to the scatter points of the experimental results using least squares. After fitting the results of a QM and of the random sampling baseline, bym denotes the fraction of samples the QM requires to match the performance of random sampling.
brand
2.3 Motivation for new Active Learning Metric
Summarizing the trends in our literature review, the empirical evaluation of QMs primarily focuses on the performance criterion, presented either through quantitative results of single iteration, which fail to
4

---

Published in Transactions on Machine Learning Research (January
0.8
Performance (p)
0.6
Ceiling Perf. (a.)
1
random (scatter) random (connected) random (prana)
-
0.4
AL (scatter) (connected) AL (PAL)
0.2
200
400
600
800
1000 1200 1400
# added training samples (x)
Figure 2: Learning curve schematic for random sampling and active learning (AL) showing scatter points, approximation by direct connection, and approximation by using Pqm(() = aoo Bqm with 0xc = 0.9, ag -0.2, brand 600 and bAL- 150.
reliably represent the overall process, or through qualitative visual inspections of learning curves across iterations, which lack quantitative and comprehensible results. While metrics such as accuracy, F1 score, and mean average precision effectively capture model's performance for a single iteration, the field of active learning lacks a comprehensive metric for quantitatively comparing results across multiple iterations. Empirical evaluation of a QM typically involves comparing performance against baselines, predominantly using random sampling. The speed-up factor
quantitative performance metric for evaluating QMs over multiple iterations, indicating the fraction of samples required to match random sampling performance. It generalizes the metric used by Kath et al. (2024b) to various types of learning curves by employing a set of approximation functions and provides comprehensive theoretical and empirical evidence supporting its suitability as an AL performance metric.
3 Methods
The performance of a machine learning model trained on a labelled dataset of a given size can be effectively represented by various performance metrics, e.g. accuracy. During the AL process, the size of the labelled dataset increases iteratively, resulting performance value for each iteration. [erforrman is typically visualized by plotting the performance values against the labelling budget for each iteration, shown in Figure 2 Although the data points discrete, they commonly visualized
continuous graph by approximating them through direct point connection referred to here the learning curve (connected)
scen in, e.g., (Dayoub et al. 2017 Gao et al.) 2018 Hartford et al. 2020 Hu & Zhang 2018 Bullard et al.
2019 Cawley 2011). This paper introduces & novel method to approximate the learning curve, enabling the derivation of quantitative results. Our method grounded on four assumptions:
A1: t= 0
Pqm (x)()) c1, Vqm E QM, a E [0.
A2: Dj= = Pam (1)()) C2 Vqm E QM, 02 E [0
A3: Pqm Pam( (I(())
Pam (x(1 - 1)), Vqm E QM, E (1, ..., bmax)
A4: Zum (p)
C3.qm: Vqm QM, Vp
(p)nd(p)
Al states that the performance of the initial iteration is the same for all QMs A2 states that the performance is the same for all QMss if the unlabelled dataset is empty. Before starting the AL process (t 0) and when
DAL is completely labelled (D', = 0), the composition of D4, is independent of the QM This results in identical trained models and, consequently, the same performance.
A3 states that the lcarning curve strictly monotonically increasing. While several factors may cause deviations from this strict monotonicity, the error introduced by this assumption minimal, model performance generally improves with the addition of training data.
5

---

Published in Transactions on Machine Learning Research (January /202)
Dataset
Learning Curves
Speed-up factor (connected)
1.0
0.88
Dimension 2
Performance
0.8
0.87
Xyr Fprr/Xarrd
0.6
0.86
0.4
-5-
0.85
Random (connected)
0.2
AL (connected)
-10
Class
Class 1
0.84
0.0
-5
10
500
1000
1500
2000
0.855
0.865
0.875
Dimension
# added training samples
Performance
Figure 3: Synthetic illustration of A4. Left: Two-dimensional dataset with 4 000 samples and 90/10 class imbalance. Center: Connected learning curves for random sampling and the query method ratio max using (x) obtained by inverting the connected learning curves.
logistic regression. Right: Ratio Frand (P)
A4 states that the ratio of samples required by a QM to achieve a certain performance, compared to random sampling, is approximately constant across performance levels. Here, C3.qm represents this theoretical fraction, which the proposed speed-up factor estimates empirically. This assumption is based on the hypothesis that the information content of queries selected by QM remains roughly constant. While this trend is qualitatively observable in prior work (e-g., (Kath al., 2024c] Rakesh & Jain 2021 Cai et al., 2021)),
not aware of any quantitative examination of it. At most, this ratio's 's constancy can be inferred from tables reporting the number of samples required to reach a given performance (e.g., Table I in (Gal et al. 2017p). However, such data are rarely provided; as Table [ shows, only 26 of 449 papers use performance as a stopping criterion. To further illustrate A4, we generate a synthetic two-dimensional dataset with 4 0000 samples and two classes in a 90/10 10 balance (Figure 3 left). Using a representative subset of 2 000 samples as the evaluation set, we emulate AL on the remaining 000 samples with an initial budget and query size of 20 using a logistic regression model. We then compute the connected learning curves for
random sampling and for the QM ratio max (Figure 3 center), which is described in section 4.2 After
(rand(p)) (Figure right). As the labelled set is independently composed before AL begins (A1) and after all samples have been labelled (A22,, the performance at these endpoints is equal and the ratio therefore approaches 1. Figure 3 (right) confirms this behaviour at the initial point, whereas the curve does not reach 1 at the final point because variance in the learning curve (Figure 3 center) induces instability in the inversion near the endpoints. In the intermodiato region, however, the ratio remains approximately constant, supporting A4.
inverting these curves, we compute the ratio (um(p)
Instead of connecting the data points create the learning curve, we propose approximate it using a function (x) = k(.)) where x is the number of training samples added to Dj, bhm is QMescccifie parameter controlling the curve shape, is an approximation function and p.m (x) is the approximated performance value (see Figure 2 To satisfy Al A4, f must be independent of bym and the range [o, 1] for x (A1), have constant asymptote for lim in range [o, 1] (A2), and be strictly monotonically increasing (43). Since is always divided by bym, A4 is inherently satisfied. This follows directly from the
definition of the speed-up factor S, which represents the fraction of samples required by QM to achieve the same model performance as when using random sampling (pqm Prana).
S Iqm
P- (Pqu)) bqm Dqm
const.
(1)
Frand
8- ( brand brand
Considering those restrictions, be freely selected based the experimental setup. The following experiments set of the two most frequently used performance approximation functions VVering &
Loog 2023
8¹_ ax (1-ea a0 Equa
and
122 ²= 0.00
+
ap
Fagar
6

---

Published in Transactions on Machine Learning Research (January
Table 2: Multi-label datasets, including name, domain, number of classes (#C), class presence statistics, and the sizes of the active learning and the evaluation sct.
Class pres.
[%]]
Dataset
Domain
#C
min
max
mean
[D AE
|Dal
CARInA
Audio
37
5.3
73.1
26.6
78 73931
19732
MS COCO
Image
80
0.2
54.2
3.6
94 504
23
Reuters
Text
31
0.3
22.2
2.2
37 896
15675
Scene
Tabular
15.1
22.1
17.9
1927
480
Given experimental setup and the results of noM QMs aso is set to the average model performance across when all samples are used for training, and do.pt and dp,5² are derived from the results of the initial iteration (z= 0). Since the learning curve reflects QM performance and DL consists of samples selected randomly or from prior annotation sessions, equal across QMs the r-axis of the learning curve presents the number of added training samples with x = 'Dj 1 D- Using least squares, bym,ga and bqm.pa are estimated for all QMs The root mean square error (RMSE) between each QM result and the approximations f and Pum is computed and averaged across QMs
QMs
The approximation function that minimizes the RMSE is then
selected to approximate AL performance.
4 Experiments
The experiments have three objectives: (1) validate the four assumptions Al A4 underlying the speed-up factor (see section 3): (2) identify which properties must be fixed and which may vary for valid use of the speed-up factor, based whether A1-A4 agnostic to them. We extend the framework of Lüth et al.) (2023) (see section 2.2 varying dataset, AL setupp model, and training properties detailed in the remainder
of this section; and (3) evaluate the stability and robustness of the speed-up factor across diverse settings.
All results are averaged over multiple random seeds.
4.1 Datasets
Relevant dataset properties are label type, domain, number of classes, class balance, and size of DAL As label type choose multi-label datasets, since they are standard in real-world scenarios and generalize multi-class datasets, which in turn extend single-class datasets. Additional results on multi-class datasets are provided in appendix A Table lists selected datasets varying in domain, number of classes, class balance, and size of DAL
CARInA (Kath et al. 2022) is audio dataset derived from read German Wikipedia articles. We use the Complete subset, treat phonemes as classes, segment files into 1-second snippets, and ensure no speaker overlap between DAL and DE.
MS COCO (Lin al. 2014) is an image dataset with common objects.
1997 is text dataset of news titles and articles. We treat topics as classes, and concatenate each title with its article.
Reuters-2155 21578 [ewiis
Scene [Boutell al. 2004 is a tabular datasct of image attributes with landscapes as classes.
We use two complementary experimental settings. First, we emulate AL on the complcte datasets using a stop budget of 400 samples, which rellects a realistic annotation budgel in large-scale scenarios. Second, to analyse the behaviour of performance metrics when the unlabelled dataset is exhausted, we construct a ropresentative 000-sample subset (2k) for cach dataset and annotate all samples to observe the effect when D', = Evaluation is performed on the identical held-out evaluation set Dg for both settings, using the macro F1 score the performance metric per iteration to ensure equal class importance.

---

Published in Transactions on Machine Learning Research (January /2026
4.2 Active Learning Setup
Relevant AL properties are initial budget [P],, query size 1 stop budget, QM
type, QM performance,
and QM computational requirements, We set ( and 1 to equal values (2, 20, or 200), ensuring all classes are represented in (D]) when feasible, to avoid initial class selection by chance. We compare different stop budgets by treating intermediate iterations as final iteration. We choose basic, state-of-the-ar and multi-label-specifie QMs] selecting 5 5 of the samples randomly to ensure all samples remain eligible for
selection.
Random sampling as baseline selects samples arbitrarily.
Ratio max (Monarch, 2021 is basic uncertainty sampling QM that computes an uncertainty score per class. We select the maximum score per sample, following Kath et al. (2024c).
K-means Clustering (Monarch, 2021 is basic diversity sampling QM Following Kath et al. (2024a), we reduce embeddings to five dimensions using principal component analysis for improved performance, set the number of clusters equal to the query size, and select samples closest to each cluster centroid.
BADGE (Ash et al. 2020 is a statc-of-tho-art QM that combines uncertainty and diversity sam- pling. Using hypothetical labels, leverages the gradient embedding of the model the sampling space, selecting query of samples likely to produce high gradients across diverse neurons.
BALD Gal et al. 2017 is state-of-tho-art
QM that combines uncertainty sampling and mutual information, balancing prediction entropy and expected entropy across Monte Carlo samples to select the most informative samples.
CRW Esuli & Sobastiani 2009) is a basic multi-label-speefiic QM that weights class performances
to sample in round-robin manner, selecting more samples based the uncertainty score of classes with low performance.
BEAL (Wang et al., 2024) is a state-of-tho-art multi-label-speeitt QM that uses Bayesian deep learning to derive the model's posterior predictive distribution, and an expected confidence-based acquisition function to select uncertain samples.
4.3 Models
Relevant model properties are architecture, complexity, and weight initialization. Selected models include both basic and state-of-the-art variants with varying architectures and complexity:
wav2vec 2.0 Baevski et al. 2020 for audio,
ResNet-50 (He et al.
2016 with ImageNet weights for images,
BERT [Devlin al.
2019 for text, and
single fully connected layer for tabular data.
We compare three weight initialization strategies: random initialization, self-supervised learning using rota- tion prediction on DAL [Gidaris et al. 2018). and transfer learning.
4.4 Training
Relevant training properties training paradigms, layer status and dyper-paraneee Selected training paradigms include supervised Icarning (SL), SL with data augmentation (tripling 1), and semi-SL with pseudo-labelling (Lee) 2013). Initializing transfer learning weights, we compare fine-tuning the last n layers (1, 2, 5, or all). Syyper-paraneter include binary cross-entropy loss with logistic activation for multi-label classification, treating cach output node as an independent indicator of class presence. Training proceeds for up to 500 epochs with early stopping triggered from epoch 300 onward, using minimum delta of 0.1 and a patience of 10 epochs, with the best weights restored.

---

Published in Transactions on Machine Learning Research (January /202)
5 Results
Due to the large number of relevant properties, testing all possible combinations infeasible. Therefore, we adopt a two-step evaluation strategy. In the first step (Figure 4A-B), we fix the dataset to MS COCO2k and the QM to ratio max, and evaluate the effects of [D2), 1Qyl, training paradigm, weight initialization, and tune-tuning depth. Each property is varied independently, using the following default configuration: (P%, = 1 = 20, supervised learning, transfer learning weights for initialization, and fine-tuming of the last layer only, In the second step, we fix these properties to their default values and evaluate across datasets, models, QMs and stop budgets (Figure 4C-I).
5.1 Effect of Budget and Training Settings
Figure 4A shows the learning curves (connected), as commonly used in the literature, by varying (D)] and 11 training paradigm, weight initialization, and fine-tuning depth. At iterations where the training set is independent of the QM at t and when Di- 0-performance remains consistent across (D), and 10). but varies with changes to model or training parameters. Performance generally increases over iterations.
Figure A shows the fraction of samples required by QM to match random sampling performance, derived fam(p)
by inverting the learning curves (comnected) in Figure 4A and computing (pand(p): called the speed-up factor (connected). remains stable except when performance consistently low, such as when fine-tuning all layers. We note that, for this experimental setup, the speed-up factor exceeds 1, indicating that random sampling outperforms the QM The metric remains fully interpretable: for example, value of 1.1 means that the QM requires 10 % more labelled samples to achieve the same performance as random sampling. As these experiments focus on validating the speed-up factor rather than the superiority of specific QMs such cases do not compromisc the motric's validity or uscfulness.
5.2 Effect of Datasets, Models, Query Methods, and Stop Budgets
Figure CC shows the learning curves (comnected) of all QMs on all 2k datasets. While some QMs like BEAL on MS COCO2k, clearly outperform others, most QMs performance not visually distinguishable. Perfor- mance is consistent across QMs when the training set is independent of the QM increases monotonically.
Figure shows the speed-up factor (connected) derived from Figure 4 exhibiting low volatility across QMs and datascts. While the speed-up factor (connected) provides detailed QM performance insights, it
(qm(() Vp. As shown in Table I most experiments use budget as the stopping criterion. Figure D shows learning curves (connected) on the complete datasets with a stop budget. of 1400, where Dí * 0 and not all QMs reach maximum performance e.g., BALD on MS COCO, The speed-up factor (connected) requires annotating the entire dataset with both QM and random sampling, making it impractical; therefore, we aim to approximate its mean value.
requires both the QM and random sampling to reach all performance levels to compute #rand())
Figure E shows the model performance for one QM and random sampling using DI,Vt, Vt, termed the learning curve (ccatter), and the approximated learning curve á as described in section 3 The speed-up factor S, computed as in eq. [ is shown in the top left of Figure 4 and as a dashed line in Figure 4D, where its close approximation S (connected) is visible.
Figure shows the learning curves (scatter) and P for all complete datasets treated as single-class (with all but one class discarded), illustrating the impact of varying QM performance on metric stability, shown in Figure
Figure 4 and show the performance metric changes when varying the stop budget for the QM and dataset used in Figure 4 and H, comparing with the mean of the learning curve (LC mean), area under the learning curve (AULC) normalized by random sampling, the cut-point method (Pupo et al. 2018a), and the fixed version of the speed-up factor (Kath et al. 2024b). LC mean shows high volatility across experiments. AULC (normalized) is stable when QM performance resembles random sampling performance
9

---

Published in Transactions on Machine Learning Research (January /202)
COCO.2
COCO
COCO.
COCO.
Pertmmatc
.24
Wesgd Bandem
Rog AAAA Mols
Sudger sue
Leaming
27
1000 17k0 2005
1100 1763
210 sãe zio tede 1250 156 2000
240 zio 1903 1250 trss 2010
addedd tnsning camples
addeed toaring
xodds training
12
Ta Farsciom
We ged ind baator
1-2
-ns
5:35
0.20
0:23
5.50
0.23
0.42
0.0
0.23
0,30
5.25
0.40
0.04
2.05
5:00
0.20
0.22
0.23
0.20
5.25
2.3
0.35
0.40
Per Norforancc
Parfomance
Pefformance
Porformence
CARInAG:
MS COCOy
Reu5er--21570
Sceneg
200
150
1000 1250 1500 1750 2002
250 500 150 1000 1250
1300 3750 2000
250 500 150
1000
1250
1950 1750 2000
250 500 720 1203 1250 1506 1750 2010
addve baaiigg wumples
uddie buiring sampks
# Jodes trannng
astted vannng samples
14
1.25
CR
SRADGE
0.4
650
5.55
3,50 Pertomante
0.10
0:33 a.n 6dc ads ata a.5t Rertrmance
Perlomance
Pertomancc
5-0.00
5-1.3
5.0.0
Sandon
kencans mrdin
SAUUE Sandi
max
250
1000 1752 150 17i0 2005 abied burning sumple
150 10coo 1250 1500 3793 2000 addued buaring samplo
zi0
sàe 15
tode 1750 1550 1750 2000
zi2. rio 1500 1250 152 assedd zanngaamptt
Jodes trintig samp
16
100-
15
V6 (tica)
mAST
cillppirss
50-
800
2000
1220
1420
1850
1503 2002
500
1500
1293
1402
1800
1800 2000
820
1020
1200
2400
2003
2000
2000
802
2002
2010 1200
1400
1500
1000
Step Budgit
Stop B.0ge!
St0 Buoget
500 budget
CARInA
M5 COCO
Reuters-21578
Sceme
Pertmmance
15
200
197
1930
1602
e00
1002
200
500
1400
200
1960 1200 1820
added tranning samples
added traiiing
addes
added
5.0.32
5-0.2
2.57
godan
BADGE random
rardim
pi's max
200
607
2010
3250 3002
203
coo
2003
1202 3400
200
450
500
030
1000
1250 1400
200
400
630
002 1300 1200
A. Put Nerte Charogt (4)
comples
traiting
200
ratlo
102
LC maan
40-
manN
5
10
20
900
500 1050 1100
1210
1500
1002
730
803
302
1030
11:0
1202
1290
1400
500
950
1000
1100
1200
13D0
1400
500
900
1000
1100
1200
1300 1820
Stop Budjet
Stap Bidgel
Sios Burge:
Sto0 sudget
Logend:
BADGE
BALD
BEAL
kimaars
ratio
Figure 4: Empirical active learning (AL) results. 'Performance' refers to macro F1 score. Query method [QM] color and marker coding is shown in the legend below the figure; only panels with deviations include individual legends. Using multiple random seeds, μ denotes the mean, a the standard deviation and SEM the standart error of the mean. A B: MS COCO2,, all classes, QM ratio max, 5 random seeds. A) Learning curve (connected), μ + SEM. B) Speed-up factor (connected), μ. C-F: 2k datasets, all classes, 30 random D) Speed-up factor (connected)), all QMs μ. E)
seeds. C) Learning curve (connected), all QMs
/a +
Learning curve (scatter + 6 fit),, single QM # o.. F)
AL Performance metric stability, single QM μ. G-I:
Complete datasets, 5 random seeds. G) Learning curve (connected), all classes, all QMs + SEM. H)
Learning curve (scatter + i fitt), single class, single QM # SEM. 1) AL Performance metric stability, single class (same as in H), single QM (4.

---

Published in Transactions on Machine Learning Research (January /202)
Learning Curves
Stop Budget
Speed-up Factor
5=1.83
Port Metric stabllity (Qualitative)
Superior
Performance
0.6
0.4
random
Finnd k-means
0.2
0.0
Inferior
200
400
600
8C0
1000
1200 1400
200
400 600
800
1000
1200
1400
200
400
600
800
1000
1200
1400
training samples
training samples
training samples
Figure Metric stability comparing the stop budget method and the proposed speed-up factor. Left: Learning curves for random sampling and k-means on the single-label subset of Reuters-21575 (also shown in Figure H Center: Performance decisions over the stop budget using the stop budget method. Right: Performance decisions over the stop budget using the speed-up factor.
Assumption dependence on relevant properties. Check marks indicate assumptions (A1- hold despite property variation. Experiment indicate supporting figures.
Table 3:
Variation-Agnoste
Dimension Property
Experiment
A1
A2
A3
A4
Dataset
Label Type
Figs.
and
6
X
X
v
x
Domain
Fig.
x
X
V
X
# Classes
Fig.
X
V
X
Class Balance
Fig.
X
V
X
Sample Size
Fig.
V
X
Active
Initial Budget
Fig.
B
V
V
V
Learning
Query Size
Fig.
V
J
V
Sctup
Stop Budget
Fig.
F
J
V
QM
Type
Fig.
QM
Performance
Fig.
QM
Comp. Req.
Fig.
Model
Architocture
Fig.
X
Complexity
Fig.
4C-1
Weight Initialisation
Fig. AA-B
Training
Training Paradigm
Fig.
A
B
X
Layer Status
Fig.
4A-B
X
(e.g., Reuters-21578p.) but shows high volatility otherwise (e.g., MS COCO). Cut-points either accept reject the hypothesis that the QM significantly outperforms random sampling. This is stable before the learning curve saturates (c.g., CARInAak), but it bocomes volatile afterwards (c.g., Scono). Both S (fix) and S show high stability across datasets, with S consistently exhibiting improved stability.
Figure extends the performance stability analysis of the multi-iteration metrics from Figure 4 and the widely used stop budget method. Comparing the learning curves before and after saturation (Figure 5 left), the stop budget method shows instability after the learning curves saturate (Figure 5 center), whereas the speed-up factor remains stable throughout the experiment (Figure 5 right).
6 Discussion
We present findings on the three objectives outlined in section 4 testing assumptions A1-AA A1-A4, identifying fixed versus variable properties, and assessing speed-up factor stability.
Results confirm the intuitive claim that performance matches t 0 and when D 0 across Qs sechanged-Supportinng supporting Al and A2. Our results consistently
(0) the utilized training set remains unchanged
11

---

Published in Transactions on Machine Learning Research (annuary/2200
show increasing learning curves, supporting A3. Figure 4B and D show that Pam (p)
(rand(p) remains approximately
constant across all p. and Figure and E demonstrate that its mean is well approximated by the speed-up factor, supporting, A4. The only exception occurs in Figure col. 3, where performance remains low across monitored iterations; Figure A col. 4 indicates this low performance likely results from feature distortion and weer-fitting caused by fine-tuning multiple pre-trained layers on small, distant datasets, whereas updating only the last one or two layers preserves general representations (Kornblith et al., 2019 Kumar et al. 2022).
Our results show that A3 generally holds, whereas A1, A2, and A4 depend on fixing dataset, model, and training properties, as summarized in Table 3 This implies that QM comparisons require identical dataset, model, and training sctups, since these factors influence both the imitial (A1) and final (A2) performance, well as the number of samples each method needs to match random sampling performance (A4). This dependency is expected, performance inherently depends on the dataset, model, and training configuration, and because QMs do not perform equally well across models and tasks (Lowell et al. 2019).
Comparing metric stability under varying stop budgets (see Figure 4F and I) reveals that both the mean and area under the learning curve are sensitive to performance improvements. The cut-points metric is stable before learning curve saburation but becomes volatile afterward, as it assumes increasing performance gaps between QMss contradicting the theoretically and empirically supported assumption A2. The fixed and proposed speed-up factors exhibit the highest stability, with the proposed method demonstrating overall superior stability. As expected, using only the final iteration as performance measure is unstable after the learning curve saturates (see Figure 5).
We further evaluated the sensitivity of the speed-up factor to the chosen set of approximation functions (see appendix C for details). Overall, we observe low sensitivity across datasets and QMs
Sensitivity
increases for strongly non-monotonic learning curves or when performance remains consistently low until the final iteration. Based on qualitative analysis, we recommend computing the speed-up factor only when the learning curve is approximately monotonic and the normalized performance gain Penst Plaitaa exceeds
Posiling Piitia
approximately 20 %-30%. %.
7
Conclusion and Limitations
7.1
Conclusion
This work introduces quantitative multi-iteration AL performance metric, termed the speed-up factor. A comprehensive literature review shows that most existing AL evaluation methods are either quantitative but limited to single iteration, qualitative across iterations. The speed-up factor, derived from learning curve approximation based on four assumptions, quantifies the fraction of samples QM requires to match random sampling performance. Experiments validate these assumptions, demonstrate their agnosticism to all relevant AL setup properties, and show that the speed-up factor offers superior stability compared to existing metrics. Conclusively, the speed-up factor provides a robust, interpretable, and practically moaningful metric that enables quantitative multi-iteration evaluation QM performance.
7.2 Limitations
Limitations of this work include restricting the set of approximation functions to two commonly used func- tions that cover key families of learning curves. Specifically, E corresponds to a Weibull curve with shape parameter k = 1, and &² corresponds to a generalized logistic curve with shape paramcter 1. Together, these provide robust basis for computing the speed-up factor. While additional functions could be con- sidered, they must satisfy the assumptions A1-A4, which excludes non- monotonic approximations such unconstrained polynomials, splines, or neural network regressors.
Another limitation stems from the dependence on A1-44. While Al generally holds, A2 may be violated in online-learnngg scenarios without a fixed ceiling performance, and A3 or A4 can fail under unstable training, resulting non-monotonic or highly noisy learning curves. Because the speed-up factor relies on lcarning curves that can be smoothly approximated and are sufficiently monotonic for reliable inversion, strong fluctuations can cause the metric to become unstable or unreliable. Smoothing averaging across runs
12

---

Published in Transactions on Machine Learning Research (annary//0200
mitigate this issue, but if substantial non-monotonicity persists, recommend not using the speed-up factor.
Furthermore, the speed-up factor becomes unreliable when performance remains consistently low. This occurs in two situations: (1) when the number of labelled samples is very small and performance remains close to the initial value, making largely independent. of the QM as stated Al (see Figure 3 right); and (2) when the finaliterationn performance remains substantially below the ceiling performance, causing the learning-curve approximation to be dominated by stochastic variance.
Moreover, this elaboration does not consider computational requirements; an approach to address this is detailed in appendix B
8 Acknowledgments
This research is part of the Computational Sustainability & Technology project area and has been supported by the Ministry for Science and Culture of Lower Saxony (MWK), the Endowed Chair of Applied Artificial Intelligence, Oldenburg University, and DFKT.
References
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.nel, 2020. URL
httpss://openreview www.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina. Balcan, and Hsuan-Tien Lin (eds.), Advances in Newral Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL. html
www.iinggnneuuuisppeee httprrpp.ccpppppa//////////////e hash//
Matthew R. Boutell, Jiebo Luo, Xipeng Shen, and Christopher M. Brown, Learning multi-label scene classification. Pattern Recogmit., 3779)11757-1771 2004. doi: 10.116/1PAAT00.2204..
URL
wowp///....///.......///..///... 1tt.:/00ttpppcooog....oooooooooo 2004.03.009
Kalesha Bullard, Yannick Schroecker, and Sonia Chernova, Active learning within constrained environments through imitation of an expert questioner. In Sarit Krans (ed.), Proceedings of the Twenty-Eiin Eighth In- ternational Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 2045-2052. ijcai.org, 2019. doi: 10.24963 /T0/0////CAA.211//2222 URL http::ttp:///e /ttp:////o///.....///. wtgg//..49933/11.. 24963/1jcai. 2019/283
Lile Cai, Xun Xu, Jun Hao Liew, and Chuan Sheng Foo. Revisiting superpixels for ac- tive learning in semantic segmentation with realistic annotation costs. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 10988 10997. Computer Vision Foundation / IEEE, 2021. doi: 10019//VVRA4437.220 URL httpttts///peeacccessss thecvf Active_LLarriie in in__emmaticc
htt.aa/teeee////2/////////////// Revisiting www.eiitinggSuperiiielll.sssssss for Ltr_SeaaniicSSmmmnatttaaaae Segmentation With Wel__Reaasst_CC__ _as__2211____________ html
Gavin C. Cawley. Baseline methods for active learning. In Isabelle Guyon, Gavin C. Cawley, Gideon Dror, Vincent Lemaire, and Alexander R. Statnikov (eds.), Active Learning and Eaperimental Design workshop, In conjunction with AISTATS 2010, Sardinio, Italy, May 16, 2010, volume 16 of JMLR Proceedings, pp. mlr crress/v16/aaxleyy.1. cawleellllaalaaaaa..aaala pdf
47-57. JMLR.org, 2011. URL
Feras Dayoub, Niko Sünderhauf, and Peter I. Corke. Episode-based active learning with bayesian neural networks. In 2017 IEEE Conference on Compuler Vision and Pallern Recognition Workshops, CVPR
https://cst.... dik.ttffkkddd.
13

---

Published in Transactions on Machine Learning Research (annuayy/0200
Workshops 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 498-500 IEEE Computer Society, 2017. doi: 10.1109/CVPRW 2011.77.
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidi- rectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (ods.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- pratational Linguislies: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171-4186. Association for Computational Linguistics, 2019. doi: (018653/V1/N1990
URL battps:///oi..oe
Andrea Esuli and Fabrizio Sebastiani. Active learning strategies for multi-label text classification. In Mohand Boughanem, Catherine Berrut, Josiane Mothe, and Chantal Soulé-Dupuy Dupuy (eds.), Advances in Information Retrieval, 31th European Conference on IR Research, ECIR 2009, Toulouse, France, April 6-9, 2009. Proceedings, volume 5478 of Lecture Notes in Computer Science, pp. 113. Springer, 2009. doi: 10. 1007 1778-3-642-009988 _22. URL https::////o/ http::://0ol.rrg111100000//////1 3-2t:///0000007770-------2000000 3-000000--------08---777- 12.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August. 2017, volume 70 of Proceedings of Machine Learning Research, pp- 1183-1192, PMLR, 2017. URL
mml.rinng...rr....mm ppress/v0/gaa117// http:///naa........ree////////aa
Li Gao, Hong Yang, Chuan Zhou, Jia Wu, Shirui Pan, and Yue Hu. Active discriminative network represen- tation learning. In Jérôme Lang (cd.). Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 19-19, 2018, Stockholm, Sweden, pp. 2142-2148, ijcai.
2018. doi: 10.24963/ 110////08/A2 /266. URT. htttts://.ii.orrgg/gto http:////.....2.....aaa..../////
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https:
//openreview.o net weanurreieewnne//orunnniii net:/forumii--S1AAA2
Jason S. Hartford, Kevin Leyton-Brown. Hadas Raviv, Dan Padnos, Shahar Lev, and Barak Lenz. Exemplar guided active learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (cds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: N0.1109/CVPR.201666 URL attpp:///oi.... wag.ii.org//n org/10 10. 1109/CVPR. 2016.90
Zehong Hu and Jie Zhang. A novel strategy for active task assignment in crowd labeling. In Jérôme Lang (cd.); Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Slockholm, Sweden, pp. 1538 1545. ijeai.org, 2018. doi: 10. (0.21963/JJCAA.20188/ 24963/1j6c3ia1 2018/213
URL https://doi.org:
Hannes Kath, Simon Stone, Stefan Rapp, and Peter Birkholz. CARInA A Corpus of Aligned German Read Speech Including Annotations. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022 pp. 6157 6161. IEEE, 2022. doi: 10.1109/ICASSPP3B00 2022.974616... URL https://doii /ttts://doo.orr//000 10. 1119/ICASSP43992...oo 2022 222229741660
Hannes Kath, Thiago S. Gouvéa, and Daniel Sonntag. Active and transfer learning for efficient identification of species in multi-label bioacoustic datasets. In Proceedings of the 2024 International Conference on Information Technology for Social Good, GoodIT 2024. Bremen, Germany, September 4-6, 2024, pp. 22 URL https:://oii org/10. 1145/36775266 0t5.//36777555...86555 3678635
25. ACM, 2024a. doi: 0111145/3677525.366
Hannes Kath, Thiago S. Gouvéa, and Daniel Sonntag. Active learning in multi-label classification of bioa- coustic data. In Andreas Hotho and Sebastian Rudolph (eds.), KI 2024: Advances in Artificial Intelligence
14

---

Published in Transactions on Machine Learning Research (January/ /2026)
4 7th German Conference on AI, Würzzurg, Germany, September 25-27, 2024, Proceedings, volume 14992 of Lecture Noles in Compuler Science, pp. 114-127. Springer, 2024b. doi:
9.
Hannes Kath, Patricia P. Serafini, Ivan B. Campos, Thiago S. Gouvéa, and Daniel Sonntag. Leveraging transfer learning and active learning for data annotation in passive acoustic monitoring of wildlife. Fcol. Informatics, 82: 2024c. doi: 10.1016/J.ECOINN 202.EIII..22221121111 2024.102710. URL
j.ccinff
Kwanyoung Kim, Dongwon Park, Kwang In Kim, and Se Young Chun. Task-aware variational ad- versarial active learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 8166-8175, Computer Vision Foundation IEEE, 2021. doi: 10. 1109/CVPR44437. 20000000 URL htttps://openaccesst thecvf Task- Variaatonali Adversarial SettiveLLearinng CVPR_______________ Ct__C_C__R22_PPPeeee_ html
Seo Tack Kong, Soomin Jcon, Dongbin Na, Jaewon Lee, Hong-Seok Lee, and Kyu-Hwan Jung. A neural pre-conditioning active learning algorithm to reduce label complexity. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http:/ttp:/ppaper httml html
files/
Simon Kornblith, Jonathon Shiens, and Quoe V. Le. Do better imagenet models transfer better? In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 2661-2671. Computer Vision Foundation IEEE, 2019. doi: 10.1109/CVPR.
2019.00277. URL htttp:///peaacceec/c/c
thecvf www.ccoooemnnnc.cc cotpeettteeetertcecccce 2019/htmm/Kornbltta. Do Better Las_22119______ html
ImageNeet_Moder Models_ Transfer Breanser_Better___
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL htttps://ww..... www.wsstttoooooooooooo www.tooooooooo.oriiiii wwtp:eeeee/eeeeealiiiaaaa featunes--eeeuu--eeeeee 2009- 20.9-TR..p0. TR.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang, Fine-tuning can distort pretrained features and underperform out-of-distribattin.. In The Tenth International Conference on Learning Representations, TCLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL
Dong- Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, 2013.
David Lewis. Reuters-21578 text categorization cartrpaiii..ttttttttttteeooooono collection. 1997. URL http::://bii.orr/11024442333
September
Tsung- Yi Lin, Michael Maire, Serge J. Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (ods.), Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pp. 740-755. Springer, 2014. doi: 10.1007/998-3-319 30902-20001111111111 11 48. URL
https://doi.orgg/ wtt.///...r//....0.///...// 10001/007/87--3199 3-1------1--------
David Lowell, Zachary C. Lipton, and Byron C. Wallace. Practical obstacles to deploying active learning. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 21 30. Association for Computational Linguisties, 2019. doi: 10.1663/V11/D19-1003 URL https://doi. 1ttp//1.11863//1//11-111
Carsten T. Lüth, Till Bungert, Lukas Klein, and Paul F. Jaeger. Navigating the pitfalls of active learn- ing evaluation: A systematic framework for mcaningful performance assessment. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Newral
15

---

Published in Transactions on Machine Learning Research (January /2026)
Information Processing Systems 36: Annuul Conference Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:://papers.niiptt nips. cc/ html
paper
Mohamad Amin Mohamadi, Wonho Bae, and Danica J. Sutherland. Making look-ahead active learning strategies feasible with neural tangent kernels. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. httppttt::/ppaerr... nips ccapps.ccpppppee ccppaper_pp.. feerrr_fll__ppp hilesspapeer2022/ //sss//h/ 51224001baalr1222665444444343333 Abstract-
html
Robert Monarch. Mluman-in-the-Low Machine Learning: Active Learning and Annotation for Human- centered AI. Simon and Schuster, July 2021. ISBN 978-1-61729-674-1
Pratock Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, and Shadab Khan. Towards robust and reproducible active learning using neural networks. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 223 232. IEEE, 2022. doi: LLLL9/CVPR5268... 0w25:26688..22.0000000 00022. URL httpp://ea /ttp::///do...e /tg////ooor///////ooo/ooo 10. 101.009//VV222888888 2022. 00032
Oscar Gabriel Reyes Pupo and Sebastián Ventura. Evolutionary strategy perform batch-mode active learning on multi-label data. ACM Trans, Intell. Syst. Technol., 9(4):46:1 46:26, 2018. doi: 10.1145/
3161606. URL hatppa://boi.org/10... 1145/3161666
Oscar Gabriel Reyes Pupo, Abdulrahman H. Altalhi, and Scbastián Ventura. Statistical comparisons of active learning strategies over multiple datasets. Knowl. Based Syst., 14:2274-288, 2018a. doi: 10.1016/
J.KNOSYS.20188 20NSSY............ .033.
Oscar Gabriel Reyes Pupo, Carlos Morell, and Sebastián Ventura. Effective active learning strategy for multi-label learning. Newrocomputing. 273:494 508, 2018b. doi: 10.006//..EEUCOM...0....8 URL https://doi. wt..oo/..........1
10. www.wm...m....m 201.0c............ 18.08..00 000
Vineeth Rakesh and Swayambhoo Jain. Efficacy of bayesian neural networks in active learning. In IEEE Con- ference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021, pp. 2601-2609. Computer Vision Foundation IEEE, 2021. doi: 10.1109 CVPP//300..2222022222 20221. 00294. URI. httph:tpp::///pt h/oppacccs//pecccssssssss www.aaacccccssstccc..cceeeeeee the.vvf.com/conteetiee hwtp:///connttete/////////DD///0
Baatttm//aaessaaaee Barisshhffficca Efficacy_ccccc of
Bayesian Neural_ Neural_Networks, in Active_Learre Swwpeiinng__P_RReee__ 0PPPPW_2021_00 10001_pppee_pppee
Christoffer Riis, Francisco Antunes, Frederik Boe Hüttel, Carlos Lima Azevedo, and Francisco Pereira. Bayesian active learning with fully bayesian gaussian processes. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022 New Orleans, LA, USA, November 28 December 9, 2022, 2022.
Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. A dataset and taxonomy for urban sound research. In Kien A. Hua, Yong Rui, Ralf Steinmetz, Alan Hanjalic, Apostol Natsev, and Wenwu Zhu (eds.), Proceedings of the ACM International Conference on Multimedia, MM '14, Orlando, FL, USA, November 03 07, 2014, pp. 1041-1044. ACM, 2014. doi: 20.145/2647868.265500 URL https://doo. org/ 10.
Burr Settles. Active learning literature survey. Technical report, University of Wisconsin-Madiion,, 2009.
Weishi Shi, Dayou Yu, and Qi Yu. A gaussian process-bayesian bernoulli mixture model for multi-label active learning. Tn Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jen- nifer Wortman Vaughan (cds.), Advances in Neural Information Processing Systems 94: Annual Confer- ence on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 2021.
David Slate. Letter Recognition. https///di..rr/10.4433/.
UCI
Machine
Learning Repository, 1991.
DOL:
16

---

Published in Transactions on Machine Learning Research (January/200
Ting-Peng Tang and Sheng-Jun Huang. Active learning for multiple target models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
Alaa Tharwat and Wolfram Schenck. A survey active learning: Statc-of-tho-art, practical challenges and research directions. Mathemalics, 11(4), 2023. ISSN 2227-7390. doi: 10.3390/math1 1040820. URL
attps://wwer mdpi 20000/22222779/11//////3909
Tjeerd Van Der Ploeg, Peter C Austin, and Ewout W Steyerberg. Modern modelling techniques data hungry: simulation study for predicting dichotomous endpoints. BMC medical research methodology, 14: 1 13, 2014.
Tom J. Viering and Marco Loog. The shape of learning curves: review. IEEE Trons. Pattern Anal.
Mach. Intell,, 45(6))7799 7819, 2023. doi: 001109/PAAM12202.3227777 URL ptttps://doi.org/// 1109/TPAMI 2022 3220744
Qunbo Wang, Hangu Zhang, Wentao Zhang, Lin Dai, Yu Liang, and Haobin Shi. Deep active learning for multi label text classification. Scientific Reports, 14(1):28246, Nov 2024. ISSN 2045-2322. doi: 10.1038/
+41596-004-79249--7 URL [ttt::://6ob..o0///0000////11100
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classifica- tion. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.); Advances Neural Information Processing Systems 28: Anmual Conference on Newral Information Pro- cessing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 649 657, 2015. (RRpst/tp::/ html
proceedings http:///c..aa..a../o///////aaaaa
17

---

Published in Transactions on Machine Learning Research (January /2026
A Experiments on Multi-Class Datasets
To align with common practice in the literature, we additionally conducted experiments on multi-class datasets.
A.1
Datasets
Table
lists selected multi-class datasets varying in domain, number of classes, class balance and size of
D DAL.
UrbanSound8k (Salamon et al. 2014 an audio dataset urban sound categories. We segment
files into 1-second snippets.
CIFAR-10
Krizhevsky 2009 is an image dataset with common objects.
AG News (Zhang al. 2015 is text dataset of news titles and articles. We treat topics classes, and concatenate each title with its article.
Letter Recognition classes.
Slate 1991 is a tabular dataset of image attributes with letters (A Z) as
Multi-class datasets, including name, domain, number of classes (#C), class presence statistics, and the sizes of the active learning and the evaluation set.
Table 4:
Class pres. [%]
Dataset
Domain
#C
min
max
mean
D AL
[D g
UrbanSound8k
Audio
10
4.3
11.5
10.0
6971
1 761
CIFAR-10
Image
10
10.0
10.0
10.0
50.000
10000
AG News
Text
4
25.0
25.0
25.0
120000
7600
Letter Recognition
Tabular
26
3.7
4.1
3.9
157
143
A.2 Experiments
Compared to the multi-label experiments, the multi-class experiments use the same active lcarning sctup configuration. However, the model configuration differs by employing a sofumax activation in the final layer instead of sigmoid, and training uses categorical cross-entropy loss in place of binary cross-entropy loss.
A.3 Results
Figure G shows experimental results on multi-class datasets, applying the two-step evaluation strategy out- lined in section 5
Results and implications align with those for multi-label datasets discussed sections and with the difference that multi-class classification is generally easier, as each instance belongs to single class, simpli- lying decision boundaries. This leads faster convergence of the learning curves, shown in Figure A
C, E. G, and H. Even for curves with extended saturation periods, such CIFAR-10g in Figure 6 the required sample fraction for QQMto to match random sampling remains approximately constant (Figure ()) and the proposed speed-up factor remains stable (Figure FF).
18

---

Published in Transactions on Machine Learning Research (January /202)
CIFAR-10
1IF
CIFAR-10
CIFAR-10g
Porfemance
A)
Inbiing Mindin nn
Wesht intoalattion on fundon
75-
200
Spris,
250
s20
19ke 1962 1950 2000
250 soo 750 1200 1250 19s 1792 2000
zio 1000 1253 1520 1750 2003 addied toanig smmple
252 50a 1650 1750 2002
aoded barring
oddid bo samples
trining samplis
1.2
B)
Intin Budget (ivery
Ste
2an
Camio
Farsfer learting
(reellune Layers (aet N) 0.5 Performmnce
0:45
0.70
033
Parformance
Rertormance
Ferfermance
UrbanSoundBka,
CIFAR-10,
AG News.
Letter Recognitions
6-
250 sto 75c 1060 1250 1505 3750 2000
250
sio 250 1360 1250 1596 1753 2000
250 500 750 1605 1250 1556 1750 2003
253
500
1000 1254 1560 1750 2005 taddettaiiny soopes
750
added trainng samples
added traring samples
addiestaaing ammples
D)
5
12-
2.00
0.30
0.40 0.45 2.30
0.35
0:2
6.00
6.75
0.55
5.50
0.85
0.70
0.7
0.25
0.13 0.15
5 0.53
5 0.97
5 0.
5 55
Pertommnte
05-
BALD Aus Res
BASG:
Ree
250 $20 750 1960 1250 1992
1750 2000
Z50 scn 750 1200 1250 Tode 1792 2000
500 750
1000 1259 1550 1750 2003 *addedtaning amples
253 500 7in
1000 1252 1500 1750 2002 vaddectiiinng cemples
R. Per MADIC (honse
addec traiiig camples
addtd tos ring strples
100
SALD
100
75-
kmbrrs
100
75-
D4DGE
45 mean
Ola max
50-
Cu poirs
50-
20
25.
25.
500
1000
1203
1400
1002
1900
2000
500 1:00 1200
1400
1500
1830
2000
300
1000
1220
1400
1600
1800
2003
800
1000
1200
1400
1500
1500
2002
Stop Budget
500 Sudget
5002 Suzges
Slop Sudge:
Urban SoundBS
CIFAR-10
Letter Recognition
--
403
1200
1000
1403
2000 1200 3800
*adiestaaiy
0.6
1.0z
9 0.91
0.07
HD
SALD
gendam
gandum Find
SADGE
100
Pkus
200
503
600
503
10co
1200
1400
203
450
600
903
1363
1200
1400
200
420
600
895
1000
1200
1403
250
400
610
alc
1030
1200 1005
RL-UR AL- Morro Chunge [g]
added truiring samplica
# udded bariig ring samples
added tia sumplea
uddec trainig samplea
RAD
100
5
LC maan
LC mEan
outyorrts
20
10
300
202
3000
11.00
1100
1400
100
1000 1100
1203
1300
2400
800
2000
1200
1300
2402
700
200
1002
1132
1500
2002
Legere
Figure 6: Empirical aattve learning (AL) results. 'Performance' refers to macro Fl score. Using multiple random seeds, I denotes the mean, a the standard deviation and SEM the standart error of the mean. A-B: Results on CIFAR-1021: all classes, QM ratio max, 5 random seeds. A) Learning curve (connected),
a + SEM. B) Speed-up factor (connected), μ. C-F: Results on 2k datasets, all classes, 30 random seeds.
C) Learning curve (connected), all QMs
+ a. D) Speed-up factor (connected), all QMs p. E) Learning
curve (scatter + P fit), single QM == F) AL Performance metric stability, single QM μ.. G-I: Results
on complete datasets, all classes, 5 random seeds. G) Learning curve (connected), all QMs + SEM. H)
Learning curve (scatter + á fit),, single QM μ + SEM. 1) AL Performance metric stability, single QM μ..
19

---

Published in Transactions on Machine Learning Research (January /2026)
B Evaluation of Computational Requirements
Computational requirements of query method (QM) are critical in active learning, as they directly impact the feasibility and efficiency of iteralive sample selection in real-world applications. However, Table | shows that these requirements are often overlooked, with only 57 of 306 papers evaluating the processing time of QMs) Since computational requirements are orthogonal to performance, they are not reflected in the speed-up factor.
B.1 Query Methods
We propose a combined theoretical and empirical evaluation by reporting the complexity of each QM as a function of relevant factors (Table and presenting normalized processing times (Figure 77.
The processing time of a QM depends on factors such as ( [PE), model complexity, number of classes, data dimensionality, and query size. We propose expressing theoretical complexity based on these depen- dencies to enable informed, application-specific assessments, summarized in Table 5 Complementing the theoretical view, we propose reporting empirical processing times to capture practical efficiency. Figure 7 shows normalized processing times for the 2k subset and complete datasels used in this study.
As an example, Table identifies BADGE as the most computationally expensive method, which confirmed empirically Figure The strong dependency on [D and Relasses is evident, as processing time decreases with smaller [D]] (Figure A C; later iteralions) and fewer classes (e.g., Scenezx and Scene).
Table 5: Complexity of query methods (QMs) in dependence ( [DL model complexity (Φ). number of classes, data dimensions, and query size. Degree of dependence: (none), o(n) (sublinear), Onn) (linear), and w(n) (superlinear).
QM
[D2]
Nelasses
Rdim data
Rqueryy
random
((n)
o(n)
ratio max
o(n) O(n)
((n)
O
((n) o(n)
k-means
(mn)
(
O(n)
BADGE
w(n)
O(n) w(n)
w(n)
O(n) O(n)
BALD
(mn)
O(n) O(n)
O(n)
O(n) o(n)
CRW
o(n)
O(n) O(n)
O(n)
O(n) o(n)
BEAL ((m) O(n) O(n)
O(n)
O(n) o(n)
B.2 Speed- Up Factor
the speed-up factor does not influence subsequent query selection and therefore only needs be computed after the final iteration, where it is used assess the overall performance of QM While it must still be evaluated by fitting the parameter bym to the scattered learning curve using nonlinear least squares scipy.optimize optimize. curve fit 71 this computation does not introduce any per-iteration overhead within the active learning workflow.
Unlike QMs
(c.g., scipy.
The computational complexity of this fitting step depends on the mumber parameters 0 of the curve (here, beg and thus k = (8) = 1), the number of data points n, and the number of optimization iterations i. Each iteration computes residuals r, Y/ - fx, f) and processes the Jacobian matrix, resulting in total complexity of O(i n is + i- 12),, which reduces to O(i n) for k 1.
httprr//w.......ooo////////////e curve.fit.
20

---

Published in Transactions on Machine Learning Research (January 2202)
CARInAu
MS COCO
Reuters-21578.
Scenez:
A)
107
1300 1750
520
CARInA
MS COCO
Reuters-21578
Scene
101
10t
1202
100
100
3003
3005
CIFAR-10.,
News,,
Recognitions:
10t
C)
1250
Recognition
10
10'
D)
101
10
1200 added
800
1000
1200
400
1200
Figure 7: Normalized processing time of the active learning experiments, averaged over multiple random soods. A) Multi-label 2k datasets, random sceds. B) Multi-label complete datasets, 5 random scods. C) Multi-class 2k datasets, 30 random seeds. D) Multi-class complete datasels, 5 random seeds.
C Sensitivity Analysis of the Approximation Functions
To assess the robustness of the speed-up factor with respect to the choice of approximation function, we compare the applied functions,
8² ax (1 e
and
1 = 000
1 1 E
40
Tqua
across all datasets and query methods. We compute two sensitivity measures: the relative difference
A = [S( S(p²)
S(D') + S(P))
and the logarithnic difference
Alog log (og(S((p))) log log(
Table t reports the quantitative results. Across all experiments the 2k datasets, both Arel and Atog remain below 5 % (with the exception of BEAL. on Scene at 5.97 %), indicating that the speed-up factor is largely insensitive to the choice of approximation function when the full dataset is annotated. Experiments on the complete datascts generally show both Arel and Algg below %. However, comparing the sonsitivity results with the learning curves shown in Figure D reveals that QMs with pronounced deviations from monotonic learning curves (e.g., BALD on CARInA) or low fial-iteration performance (e.g., BALD on MS COCO) exhibit increased sensitivity the choice of approximation function.
21

---

Published in Transactions on Machine Learning Research (lanuar/2226)) 2006)
Table 6: Relative (Aret) and logarithmic (Atog) differences of the speed-up factor computed using the two approximation functions & and p².
Dataset
Dataset Type
Query Method
S
S
rel
[%]
log
[%
CARInA
2k
BADGE
0.97
0.98
0.69
0.69
BALD
1.13
1.10
2.84
2.84
BEAL
1.16
1.13
3.19
3.19
k-means
0.90
0.92
2.59
2.59
CRW
0.91
0.92
1.27
1.27
ratio max
0.95
0.96
1.28
1.28
complete
BADGE
0.95
0.96
1.02
1.02
BALD
2.42
2.13
12.44
12.45
BEAL
1.15
1.11
3.95
3.95
k-means
0.88
0.91
3.56
3.56
CRW
0.76
0.80
5.12
5.12
ratio max
0.84
0.87
4.35
4.36
MS COCO
2k
BALD
1.26
1.26
0.30
0.30
BEAL
0.84
0.87
3.36
3.36
k-means
0.99
0.99
0.16
0.16
CRW
1.07
1.06
0.22
0.22
ralio max
1.10
1.11
0.84
0.84
complete
BALD
2.62
1.78
38.14
38.61
BEAL
0.91
0.93
2.16
2.16
k-means
1.00
1.00
0.11
0.11
CRW
1.08
1.06
2.42
2.42
ratio max
1.17
1.12
4.84
4.84
Reuters-2155 21578 2k
BADGE
0.96
0.98
1.71
1.71
BALD
1.09
1.05
3.19
3.19
BEAL
0.93
0.93
0.04
0.04
k-means
0.95
0.95
0.03
0.03
CRW
0.95
0.97
2.77
2.77
ratio max
1.09
1.07
1.71
1.71
complete
BADGE
1.00
1.00
0.11
0.11
BALD
1.80
1.23
37.66
38.12
BEAL
0.82
0.91
10.80
10.81
k-means
0.88
0.95
8.04
8.04
CRW
1.48
1.18
22.95
23.05
ratio
1.58
1.21
26.52
26.68
Scone
2k
BADGE
1.02
1.02
0.04
0.04
BALD
0.79
0.80
0.87
0.87
BEAT.
0.76
0.81
5.97
5.97
k-means
0.95
0.97
1.57
1.57
CRW
1.09
1.09
0.68
0.68
ralio max
1.04
1.06
2.13
2.13
complete
BADGE
1.04
1.03
0.97
0.97
BALD
0.65
0.72
10.80
10.81
BEAL
0.63
0.74
16.62
16.66
k-means
0.91
0.95
4.79
4.79
CRW
1.13
1.09
3.74
3.74
ralio
0.98
1.02
3.40
3.40
22


Reviews:
[{"review_id": "QI9tHb5ZHF", "reviewer": "Reviewer_e43h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript proposes a new quantitative metric, the speed-up factor, for evaluating active learning query methods across multiple iterations. The authors formalize the metric through a set of assumptions and describe a methodology for approximating learning curves using parametric functions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The paper provides an extensive and well-organized review of evaluation practices in active learning. The quantitative summary table and the survey of methodologies used across major conferences offer strong motivation for the introduction of a new metric.\n- The assumptions underlying the speed-up factor are presented clearly and supported by logical argumentation. The rationale aligns with observations reported in prior active learning research.\n- The empirical evaluation is notably comprehensive.\n- The proposed metric is intuitively interpretable, capturing the proportion of samples required for a method to match the performance of random sampling.", "audience_yn": "Yes", "audience_explanation": "Machine learning models benefit from large quantities of labeled data, yet annotation is often costly and time-consuming. Active learning seeks to improve annotation efficiency by using query methods to iteratively select informative samples.", "requested_changes": "- Additional explanation, references to prior work, and empirical evidence are needed to substantiate Assumption 4, which posits that the ratio of samples required to reach a given performance level, relative to random sampling, remains approximately constant across performance thresholds. As this assumption is central to the metric, further theoretical discussion or a small synthetic example would considerably strengthen the manuscript.\n- Table 3 suggests that Assumptions 1, 2, and 4 fail to hold under changes in the dataset, model architecture, or training paradigm. This indicates that the speed-up factor is not universally applicable and instead depends on a fixed experimental configuration. Although this limitation is noted, a more detailed discussion would be valuable, particularly for practitioners.\n- The study evaluates only two functional forms for curve approximation (exponential and logistic). While these are common choices, the learning-curve literature encompasses broader families such as power-law or Weibull models. Including at least a sensitivity analysis, perhaps in the appendix, would improve robustness.\n- Although the speed-up factor does not explicitly account for computational overhead, repeatedly fitting parametric functions may be costly in large-scale experiments. A brief analysis of the computational complexity would be helpful.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1oASCIEy28", "reviewer": "Reviewer_SEBr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces the speed-up factor, a new quantitative metric for evaluating the performance of active learning query methods across multiple iterations. It measures how many samples a method needs to reach the performance achieved by random sampling, providing an interpretable notion of labeling efficiency. The authors support the metric with theoretical assumptions, validate it on diverse datasets and query methods, and show that it is more stable and reproducible than existing metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides extensive empirical evaluations to support the claims.", "audience_yn": "Yes", "audience_explanation": "Active learning is an important area in the community. This paper proposes a method for evaluation.", "requested_changes": "Questions:\n\n1.\tThe authors chose two common functions to approximate the learning curve. It would be beneficial to include a brief discussion on the metric's sensitivity to this choice. For instance, what happens if neither function provides a particularly good fit (e.g., in cases of non-standard learning curve shapes)?\n2.\tDid the authors consider other alternative approximation functions beyond the two cited? If so, could other curve-fitting models (e.g., polynomial, spline, or neural network regressors) improve the approximation robustness compared to the chosen forms?\n3.\tHow would the speed-up factor behave when AL is applied to non-monotonic or noisy learning curves (e.g., in unstable training scenarios)?\n4.\tThe paper mentions that when the performance remains consistently low, the speed-up factor cannot be reliably computed. Could you provide more specific, quantitative guidance for identifying when the speed-up factor is unreliable? For instance, is there a performance threshold below which the metric should be used with caution?\n\nRequested Changes:\n\n1.\tSeveral minor language and formatting issues appear in the manuscript. For example, in Section 5, “most QMs’s performance” should be corrected to “most QMs’ performance.” In addition, figure and table references are inconsistently formatted (e.g., “figure 3A” vs. “Figure 3A”, “Table” vs. “table”), and should be standardized throughout the paper.\n\n2.\tWhile the paper analyzes the computational complexity of query methods, it lacks an explicit discussion of the computational overhead introduced by calculating the new metric, the speed-up factor itself. Providing a complexity analysis is essential for readers to fully assess its practical implementation.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "k17nb53UBc", "reviewer": "Reviewer_r83Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In their paper \"The speed-up factor: a quantitative multi-iteration active learning performance metric\", the authors review the literature on active learning and suggest a novel metric for quantifying the performance of an active learning algorithm (\"speed-up factor\"). In a nutshell, they fit exponent learning curves for the active learning algorithm and for the random sampling approach, and define speed-up factor as the ratio of the two exponents. Then they do a benchmark of active learning algorithms and argue that this new metric (speed-up factor) does make sense.\n\nStrengths: The authors do a very thorough systematic review of existing active learning evaluations, manually annotating ~500 papers on active learning. I really liked this part. The suggested approach with the speed-up factor sounds very natural to me. The benchmark seems to be set up in a reasonable way.\n\nWeaknesses: The suggest metric seems very obvious (disclaimer: active learning is not my field). It's also been used before (in Kath et al). The authors argue that most of the prior work uses other, more simplistic, metrics (based on thresholding) and I believe them; but they do not really show why this is important, in practice. For example, they do *not* seem to argue that their metric yields more sensible  conclusions in their benchmark, compared to more standard metrics... so the paper does not really convince me that this new metric is really necessary.\n\nThis makes me a bit on the fence about this paper. I think TMLR would be a good venue to publish a systematic review of AC evaluations, even without any new metric. But if the paper does suggest a new metric, I want more clarity on what are the benefits, especially if the authors run a whole benchmark. So I am reserving my judgment on the paper until after the authors reply.", "claims_and_evidence_yn": "No", "claims_explanation": "The claims in the review part are supported by clear evidence, yes. The claim that the new metric is a _sensible_ metric is also supported. The claim that the new metric is _better_ than the old metric is IMHO not sufficiently supported.", "audience_yn": "Yes", "audience_explanation": "Yes, the paper is on-topic for TMLR and could potentially be a good fit.", "requested_changes": "MAJOR COMMENTS\n\n* The authors argue that the new metric (speed-up factor) is better than the stopping-criterion approaches. But they do not give any direct evidence for that. Can the benchmark results be somehow used to show that? Is the ranking of algorithms more sensible or more self-consistent using your metric compared to stopping-on-budget? Can you at least show that your metric is not fully redundant with stopping-on-budget?\n\n* Page 5 mentions \"a pool of the two\" functions. This is unclear. What does a \"pool\" mean? How are these two functions even different? How do you choose which one to use in any given experiment? Why only these two? Two is not much of a \"pool\". Would always using one of them be insufficient? Can you show that?\n\n* Section 4.1: I did not quite understand the experimental setup. Why do you do separate experiments with 1400 labeled samples and with 2000 labeled samples? Why not do everything on 2000 labeled samples? Also, why do you call the situation with 1400 labeled samples \"complete\"? Some datasets have |D_AL| of almost 100,000. I got very confused here, please explain this better.\n\n* The Results section is very difficult to parse, because it's one block of text without any subsections (and the 2nd paragraph is really long), and the corresponding figure is one GIGANTIC figure with 36 panels. This is hard to follow. I would suggest to split the figure into multiple figures and the section into multiple subsections, or \\paragraphs, or somehow give it more structure.\n\n* The way you define the speed-up factor, the lower the better, right? For example, the speed-up factor in Figure is around 0.2, correct? If so, I am extremely confused by Figure 3B that shows speed-up factors above 1. Does it mean that active learning fails here? And performs worse than random sampling? When I look at Figure 3A, I don't see any big difference between active learning and random sampling, implying S of around 1. Also in panel E and in panel G the random sampling seems to perform nearly the best. I am _very_ confused by that. If active learning fails in all these cases, then why are you even showing these results? I thought the idea of active learning is that it is supposed to be more efficient than random sampling!! Please explain.\n\n* The main conslusions of the experiments seem to be that assumptions A1-A4 are confirmed. But A1 and A2 must be true by construction, A3 must be true for any reasonable algorithm, and A4 looks like a sensible simple assumption. If all these experiments are done only to demonstrate A1-A4, then it feels underwhelming. Maybe I am missing something. See also my first bullet point.\n\n* As the authors write, the speed-up factor is basically taken from the Kath et al paper. I don't know who the authors of this paper are, but I hope it's the same authors as in Kath et al., otherwise I feel it is a bit too bold to take a metric from Kath et al., and say that you \"introduce\" this metric. If it's the same or overlapping authors, then it's fine, because of course this paper does go beyond Kath et al. Unfortunately there is no way for the authors to answer this point without reavealing their identities, which is not allowed. So please do not answer this bullet point. Still, I wanted to say it.\n\n\nMINOR COMMENTS\n\n* All citations should be changed to use \\citep and \\citet as appropriate. Currently it looks like \\citet (or maybe just \\cite) was used everywhere which is not according to TMLR style guide.\n\n* The first time \"AL\" appears in the Introduction it should be deabbreviated (\"active learning (AL)\").\n\n* page 5: c_{3, qm} (in assumption A4) and S (in Equation 1) seem to be the same thing. Why two different notations? That's confusing.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)