An Expanded Benchmark that Rediscovers and Affirms the Edge of Uncertainty Sampling for Active Learning in Tabular Datasets
Authors: Po-Yi Lu
Yi-Jie Cheng
Chun-Liang Li
Hsuan-Tien Lin
Venue: Accepted by TMLR ()

Abstract: Active Learning (AL) addresses the crucial challenge of enabling machines to efficiently gather labeled examples through strategic queries. Among the many AL strategies, Uncertainty Sampling (US) stands out as one of the most widely adopted. US queries the example(s) that the current model finds uncertain, proving to be both straightforward and effective. Despite claims in the literature suggesting superior alternatives to US, community-wide acceptance remains elusive. In fact, existing benchmarks for tabular datasets present conflicting conclusions on the continued competitiveness of US. In this study, we review the literature on AL strategies in the last decade and build the most comprehensive open-source AL benchmark to date to understand the relative merits of different AL strategies. The benchmark surpasses existing ones by encompassing a broader coverage of strategies, models, and data. Through our investigation of the conflicting conclusions in existing tabular AL benchmarks by evaluation under broad AL experimental settings, we uncover fresh insights into the often-overlooked issue of using machine learning models--**model compatibility** in the context of US. Specifically, we notice that adopting the different models for the querying unlabeled examples and learning tasks would degrade US's effectiveness. Notably, our findings affirm that US maintains a competitive edge over other strategies when paired with compatible models. These findings have practical implications and provide a concrete recipe for AL practitioners, empowering them to make informed decisions when working with tabular classifications with limited labeled data. The code for this project is available on https://github.com/ariapoy/active-learning-benchmark.

Published in Transactions on Machine Learning Research (06/2025)
An Expanded Benchmark that Rediscovers and Affirms the Edge of Uncertainty Sampling for Active Learning in Tabular Datasets
Po-Yi Lu National Taiwan University, Taipei, Taiwon
409944015@csic. ntu, edu. tw
Yi-Jic Cheng
eva.cheng1214 @grmail. com
National Taiwan University, Taipei, Taiwen
Chun-Liang Li
chamlial@cs..ushiingtto.. edu
University of Washington, WA, USA
Hsuan-Tion Tien Lin
htline@csic. nlu, edu, bw
National Taiwan University, Taipei, Taiwon
Reviewed on OpenReview: https: // openreview. net/ forum? id 855yo1Ub t2
Abstract
Active Learning (AL.) addresses the crucial challenge of enabling machines to efficiently gather labeled examples through strategic queries. Among the many AL strategies, Un- certainty Sampling (US) stands out one of the most widely adopted. US queries the example(s) that the current model finds uncertain, proving to be both straightforward and effective. Despite claims in the literature suggesting superior alternatives to US, community- wide acceptance remains elusive. In fact, existing benchmarks for tabular datasets present conflicting conclusions on the continued competitiveness of US. In this study, we review the literature on AL strategics in the last decade and build the most comprehensive open- source AL benchmark to date understand the relative merits of different AL strategics. The benehmark surpasses existing ones by encompassing broader coverage of strategies, models, and data. Through our investigation of the conflicting conclusions in existing tab- ular AL benchmarks by evaluation under broad AL experimental settings, we uncover fresh insights into the often-overlooked issue of using machine learning models model compati- bility in the context of US. Specifically, we notice that adopting the different models for the querying unlabeled examples and learning tasks would degrade US's effectiveness. Notably, our findings affirm that US maintains a competitive edge over other strategies when paired with compatible models. These findings have practical implications and provide concrete recipe for AL practitioners, empowering them to make informed decisions when working with tabular classifications with limited labeled data. The code for this project is available on /attps://githii thub. http//////rrragpp//areeeeeeeeeea lew.rainnn-nb-nchcrrr.. benchmark
1 Introduction
Supervised learning models can achieve competitive results with sufficient high-quality labeled data. How- ever, acquiring such data can be costly in specific domains. This situation calls for Active Learning (AL), lcarning paradigm that strategically selects the most valuable unlabeled examples for labeling. AL has the capability of achieving better performance with lower labeling costs, which has been widely studied and applied in various domains, such computer vision
Li & Guo 2013 Demir et. al. 2015 Beluch et al. 2018

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: Comparison between Yang Loog 2018); Zhan et al. (2021) and our benchmark. (D) means
aspects of datasets; (M) means aspects of base models; (Q) means aspects of query strategies; (A) means
aspects of analysis; (0) means aspects of an open source tool. Our benchmark fetches up lacking query
strategies in Yang & Loog 2018 and lacking analysis in Zhan et al. (2021 to provide comprehensive
comparison.
Yang & Loog 2018
Zhan et al. 2021
Ours
(D)
More than 100K examples
(D) More than 400 features
(M) LR
(M) RBFSVM
(M) RF
(Q) Model uncertainty
(Q)
Bayesian uncertainty
(Q)
Data diversity
(Q)
Hybrid criteria
(Q) Redesigned
learning
framework
(A) AUBC
(A)
Average ranking
(A)
Comparison with Uniform
(0)
Released datasets
(0)
Unified AL protocol
(O) Analysis tools
2021 Schröder et al. 2021 Kishaan et al. 2020). and biology and medical fields (Hao et al. 2020 Nath et al., 2020 Logan et al. 2022).
natural language processing Liu et al.
Among the many AL strategies, Uncertainty Sampling (US) stands out as straightforward and efficient query strategy by selecting the most uncertain examples for labeling based the model's prediction con- fidence. US has demonstrated success across multiple applications (Kishaan et al. 2020 Narayanan et al.
2020
Nath et al. 2020); while US is widely used, several AL studies have developed more sophisticated
query strategies to address specific limitations in particular scenarios (Donmez et al. 2007 Huang et al. 2010
Li et al. 2015).
Two large-scale benchmarks for pool-based AL have been developed evaluate existing strategies for clas- sification on tabular datasets (Yang & Loog 2018 Zhan et al. 2021). However, they present conflicting conclusions regarding the preferred query strategies. While Yang & Loog (2018) suggested that the straight- forward US strategy excels across the majority of datasets, Zhan et al.
2021 argued that Learning Active
Learning (LAL) KKnyushkova et al. 2017) outperforms US.
Given the lack consistent comparisons across diverse contexts and the contradictory conclusions drawn from the previous two extensive benchmarks, there a critical need for a benchmark that accurately represents the current state of AL techniques in this field. Therefore, this work aims to build the most comprchensive AL benchmark compared to previous benchmarks, focusing on datasets, base models, query strategies, and analysis aspects, as highlighted in Table Our benchmark is the most comprehensive open-source framework to date, crafted by integrating transparent and unified interface. This unified interface cooperates with existing GitHub repositories, such as libact (Yang et al. 2017), Google AL playground (Yilei "Dolee" Yang 2017), ALiPy (Tang et al. 2019), ModAL (Danka & Horvath) scikit-activeml (Kotkee et al. 2021 1, and sets a new standard for future research.
Subsequently, we assess the performance of query strategies specifically for classifications on tabular data, which is widely used in various real-world applications due to its structured nature and the availability of
2

---

Published in Transactions on Machine Learning Research (06/2025)
diverse datasets. Our benchmarking results show that US is SOTA on 18 of the 29 binary-class datasets and 5 of the 7 multi-class datasets.
Furthermore, through our investigation under different AL experimental settings, we uncover the reason for the substandard performance of US Zhan et al. (2021) is model compatibility. The incompatibility between a model used within US querying the unlabeled examples (query-oriented model) and model being evaluated for the tasks (task-oriented model) degrades the performance because the queired examples might not be the most uncertain to the current task-oriented model. Through careful study, we affirm that US maintains competitive edge over other strategies when used with compatible settings on Logistic Regression (LR), Radial Basis Function kernel Support Vector Machine (RBFSVM), Random Forest (RF), and Gradient Boosting Decision Tree (GBDT). In summary, we recommend adopting US with compatible settings as the first choice for practitioners, providing a clear baseline for AL in real-world usage from the community.
In this work, we make the following contributions:
To our knowledge, our benchmark is the most comprehensive, surpassing existing benchmarks in terms of datasets, models, query strategies, and analyses.
We re-benchmark existing strategies for tabular datasets, demonstrating the US's competitiveness on most datasets, and, importantly, uncover profound insights into the often-overlooked issue of model compatibility in the context of US.
We ofler a reproducible and open-source benchmarking framework, which includes preparing datasets, an active learning process, and analysis tools to facilitate future research in the community.
2 Preliminary
In this section, we extend the Settles addressing the gap created by the lack of an open-source benchmark and highlighting significant developments in query strategies over the last decade. We also introduce the experimental protocol of our benchmark, which facilitates a deeper understanding of the critical components involved pool-based AL, helping readers to comprehensively evaluate the efficacy of different query strategies in this domain.
(2022)s 's literature to the current state of pool-based AL research,
2.1 Literature survey of pool-based active learning
Settles 2012
formalized the pool-based active learning protocol as follows:
Initial setup The process begins with small labeled pool D1 ((11,11),.................)))))) v1). (xN,8N)), where 2n e Rd is d-dimension features, Yn E y is label, and Di = N is the number of labeled examples, and a large unlabeled pool Du 1ZN+1....,NN1...... TN+M), where Dal = M is the number of unlabeled examples; and an oracle O that provides ground truth labels.
Execution setup The active learning algorithm operates over T rounds within total query budget, where each round involves querying the label of one unlabeled example from Da until the budget exhansted.
Query steps in each round
Query: Employ the query stratogy Q to select an example x, from Du-
Label: Acquire the label yj for x, from an oracle O(r,) = yj-
Update pools: Move the new labelod example from D, to Dj, i.., D Di (i,,((Z),,))) D,, D,, ((x,)).
4. Update the model: Retrain the model using the updated labeled pool Dr-
3

---

Published in Transactions on Machine Learning Research (06/2025)
Prediction the test set Finally, train the model g on the latest labeled pool Dj and make predictions on new examples from the unseen testing set Dic-
The critical element in pool-based active learning is the query strategy Q. A naïve uniform sampling (Uniform) method randomly selects unlabeled examples for labeling. Uniform does not utilize active learning strategies and serves primarily as a baseline. The overarching goal of active learning to develop a query strategy that outperforms the Uniform baseline, and there are already numerous query strategies available today. Based on |Settles 2012), we classify existing query strategies into six categories: model uncertainty, expected model changing, representation exploiting. hybrid criteria, Bayesian methods, and redesigned Learning Framework. In the next section, we first introduce different query strategies with an illustrative example, and then we further introduce each type of query strategy and their variants in detail.
2.1.1 An illustrative example of types of different methods
In this section, we illustrate the characteristics of six distinct types of query strategies. We denote negative examples with red points, positive examples with blue points, and unlabeled examples with gray points. Figure demonstrates the properties these methods, highlighting the queried example with a black square box. Given the model's decision boundary, marked with a green dashed line,
Model uncertainty strategy selects the example closest to the decision boundary, reflecting high marginal uncertainty.
Expected model changing chooses an example that, if labeled negative displayed in Figure ! would significantly change the current model to the new model displayed in sketched green line,
Representative Exploiting selects the centre of the densest cluster, which does not rely the current model.
Hybrid criteria balance uncertainty with density by querying uncertain examples in denser regions compared pure Uncertainty Sampling.
Bayesian methods identify examples within uncertain regions with high posterior variance, as illustrated by the colored areas Figure !
Redesigned Learning Framework selects the most rewarding query strategy and then queries a new example by it.
P
Model uncertainty
Expocced model changing
Representation exploiting
Hybrid criteria
Bayesian method
Redeigged learning framevork
Figure 1: Illustration examples of model uncertainty, expected model changing, representation exploiting, hybrid criteria, Bayesian method, and redesigned learning framework.
Given the high-level idea of different types of query strategies, we begin with model uncertainty to guide readers through the relationships and historical development of these query strategies.
Model uncertainty Uncertainty Sampling (US) is a prevalent query strategy in pool-based active learning, where it selects examples for labeling based on the degree of uncortainty regarding the model's prodiction. US assumes that examples about which the model is most uncertain are likely to yield the highest information gain upon being labeled. Various measures be employed to quantify uncertainty, including the margin
4

---

Published in Transactions on Machine Learning Research (06/2025)
score and entropy of the predictions of an examples in the unlabeled pool returned by the current model. In binary classification scenarios, using margin and entropy scores are equivalent in terms of defining model uncertainty (see Appendix B.5 Previous works have found that US is a strong baseline for most pool-based
active learning problems (Cawley 2011
Yang & Loog 2018 Karamcheti et al. 2021 Schröder et al. 2021
Bahri et al. 2022
In contrast to US, which relies on a single model to quantify uncertainty, Query By Committee (QBC) (Seung 1992p quantifies uncertainty through multiple models to address the sampling bias in US (Settles 2012 QBC operates on the principle of disagreement among a committee of models, each representing different model derived from the training set. Specifically, QBC selects the unlabeled example where there is the maximal disagreement among the committee members. Disagreement is measured by voting entropy, defined as the entropy of the distribution of the committee's votes. A higher voting entropy of an example indicates more significant disagreement and, consequently, a higher value for querying.
et al.
Expected model changing Previous query strategies aim to query the most informative example for the current model. In this category, we strive to query the most informative example to reduce the model's error in the future. For instance, Expected Error Reduction (EER) queries the highest expected error of the future output over an unlabeled pool, where the error would be estimated by the Monte- Monte-Carlo Carlo approach (Roy & McCallum 2001 Similarly, Variance Reduction (VR) estimates the variance of the model's output based on its Fisher information, which estimates the inverse of the lower bound on the variance of the model's parameters Cover reduction and the cost of obtaining the label to query the most informative example over an unlabeled pool effectively (Kpoor et al. 2007).
1999: [Schein & Ungar 2007]. Another method proposed the difference between the error
Representation exploiting US and QBC might perform poorly due to outliers or sampling bias that results in querying a rep-repreeettative example during the query process (Dasgupta & Hsu 2008 Yang et al. 2015 Shui et al. 2020). Although EER and VR take the input distribution into account via estimating expocted future crror over all unlabeled examples, these methods are computationally expensive, making them unsuitable for large datasets [Settles) 20122.. In this calegory, we depart, from strategies that rely model predictions and instead focus the prtuucurrereeresenttttee
of data, an approach we refer to as model-free. Hicrarchical Sampling (Hier) is a model-free reprosentation sampling method that exploits hierarchical clustering to explore the data structure of the unlabeled pool (Dasgupta & Hsu 2008). Hier randomly selects an example from the subtree of the hierarchical clustering tree to obtain its label. Then, the tree structure is iteratively updated by making the labels in the cluster more pure and focusing on the remaining impure chusters.
The query strategies mentioned in Settles (2012) are long-standing. However, the survey should be updated with the latest approaches. Graph Density (Graph) is also a model-free representation sampling method that exploits cluster structure by applying graph-based clustering techniques to the unlabeled pool without depending on any model. Similar to Graph, Core-Set uses K-Means clustering on the embedding space extracted from the data transformation (such as deep convolutional neural networks) and then queries unlabeled examples closest to the centers of clusters. [Snner & Savarese (2018) show that Core-Set works well on image classification tasks. Besides Graph and Core-Set, we could categorize recent query strategies into hybrid criteria, Bayesian method, and redesigned learning framework.
three categories:
Hybrid criteria Several works study the combination of uncertainty and diversity information to improve previous query strategies. For example, Density-Weight Uncertainty Sampling (DWUS) assumes that infor- mative examples should have both high uncertainty and be representative of the data distribution (Nguyen) & Smeulders 2004), so DWUS designs a weighted uncertainty score by averaging example's similarity to the remaining examples in the training set. Hinted Support Vector Machine (HintSVM) focused on selecting an example of an updated decision boundary that passes through unqueried regions instead of reducing its margin only (Li et al. 2015). QUerying Informative and Representative Examples (QUIRE) formulated the informativeness and representativeness with kernel matrices (Huang et al.] 2010), which characterizes the similarity between labeled examples and unlabeled examples, select an example with large self-similarity and large similarity to most remaining examples in the unlabeled pool. Representative Marginal Cluster
5

---

Published in Transactions on Machine Learning Research (06/2025)
Mean Sampling (MCM) queries examples within the model's margin closest to the K-Means centers in the embedding space (Xu et al. 2003), which inherits the benefits from Core-Set and US. Recently, Batch Mode Discriminative and Representative (BMDR) and Self-Paced Active Learning (SPAL) have been designed to query a batch of examples with claborated empirical risk minimization (Wang & Ye 2015 Tang & Huang. 2019). BMDR queries the example that expects to minimize the empirical risk on the labeled and unlabeled pools using a self-learning approach and distribution difference between the labeled pool and training set. Following the objective function of BMDR, SPAL modifies the constraint of the objective function 1 to
improve BMDR's performance. Please refer to Appendix B.5 for the detailed information.
Bayesian method Although QBC aims to query the most disagrecable example, the voter entropy might ignore each model's confidence regarding its predictions, polentially reducing efficiency. To address this issue, Bayesian Active Learning by Disagreement (BALD) queries the most uncertain example across the ensemble models but confident in the single model (Houlsby et al.) 2011). This approach can be interpreted as the conditional mutual information between the model's prediction and its parameters. BALD aims to query the example with high conditional mutual information, where the model's prediction is uncertain, but the model's parameters are certain.
Redesigned learning framework As the number of query strategies increases, some are designed to automatically select the optimal strategy from multiple heuristic query strategies. For example, Active Learning By Learning (ALBL) treats the learning problem a multi-armed bandit problem (Hsu & Ling 2015p. It thus selects the optimal strategy from set of query strategies and queries the example based on this strategy that maximizes the estimated reward at cach round. Learning Active Learning (LAL) formulates the query process as a regression problem to learn the strategy from various types of toy data (Konyushkova et al., 2017). LAL queries the example from the learned regression function, which predicts the potential error reduction.
2.1.2 Deep active learning
Besides previous query strategies for conventional machine learning models, such as Logistic Regression (LR). and Radial Basis Function kernel Support Vcctor Machine (RBESVM), Beck et al. (2021 and Zhan
et al. (0222 compared additional query strategies designed for deep learning models used in computer vision classification tasks. Their results show that US outperforms data diversity-based sampling strategies (Core- Set, Variational Adversarial Active Learning) (Sinha et al. 2019). Moreover, hybrid criteria query strategies, such Batch Active learning by Diverse Gradient Embeddings (BADGE) (Ash et al. 2019), Learning Loss for Active Learning (I.PI.) (Yoo & Kweon 2019). and Wasserstein Adversarial Active Learning (WAAT) (Shui et al. 2020), achieve competitive results better than US. Given that many recent works have taken the burden to study deep active learning on computer vision tasks (Zhan et al., 2022), transformer models (Rauch et al., 2023), and cross-domain scenarios (Werner et al.) 2024h, we consider "the use of deep active learning in tabular data" still requires a lot of effort to study deeply. Therefore, we focus on providing the practical guide and benchmark of active learning methods for tabular datasets in this work.
2.2 Experimental protocol for the benchmark
Section 2.1 depicts an abstract process of pool-based active learning. To concretize the experimental protocol for the benchmark, we illustrate the framework in Figure D In this framework, we define the training set as the union of the labeled pool and unlabeled pool, denoted as Drc D, D.. First, we split the datasct into disjoint training and testing sets, i.e., Der D.. 0, to simulate a real-world learning scenario. After splitting the dataset, sample from the labeled pool Dj within Dur and leave the remaining examples the unlabeled pool Du to set up the initial environment. Furthermore, we isolate a query-oriented model H from the task-oriented model G in Section 2.I The query-oriented model is used for selecting the most informative example during the query step while the task-oriented model is used for prediction on the test set, as depicted in Figure? 2
To distinguish the relationship between the query-oriented model and the task-oriented model, we define model compatibility as the setting where the examples obtained by the query-oriented model might be
6

---

Published in Transactions on Machine Learning Research (06/2025)
Label
testing set Dro
O 1/j
3
active learning algorithm
evaluation
Update Pools Dr. Do
A(D Du;h) h)
Di
1. train( g)
1. Update Model (i)
2. eval Die. g)
2. Query ( h)
training set Drc
Dr = { [Tx, }()}
h
g
Dr {(2mm]
query-one oriented model H
task-oriented model g
Figure 2: The Framework of Active Learning Experiments. Rectangles represent datasets including labeled pool, unlabeled pool, and test set. Rounded rectangles represent processes including an active learning algorithm, labeling, and evaluation. Circles represent models. In this work, we differentiate the relationship between two models: task-oriented and query-oriented.
different from using the -rientr model. After introducing the model compatibility, we can further discuss the query strategies that depend the query-oriented models to query new examples. For example, the query- query-oriented model in HintSVM and QUIRE are restricted to the SVM model due to their theoretical design contrasting model-free strategies, which do not rely on the query-oriented model (See Section 2.1
We notice that the setting of the model compatibility is an often-overlooked issue in previous benchmarks when they compared query strategics under the same task-oriented model (Yang & Loog 2018 Zhan et al. 2021). Although some works discuss the influence of using different models for query informative examples deep active learning (Yoo & Kweon 2019 Sinha al. 2019), it remains unclear in the context of Uncertainty Sampling, In this work, we denote the compatible query-oriented and task-oriented models for Uncertainty Sampling as US-Con US-Compatible (US-C) and con-comaatible models US-Non- US-Non-Compatinn Compatible (US-NC). Section 5.1 studies the impact of model compatibility on US to clarify the conflicting conclusion in previous benchmarks.
The benchmark aims to provide standardized framework for evaluating and comparing different query strategies a fair manner. Following (Guyon et al.) 2010 2011 Desreumaux & Lemaire 2020 Zhan et al. 2021), we utilize the Area Under the Budget Curve (AUBC) as summary metric to quantify the results of learning curves. A learning curve tracks the performance of model at each round of the active learning process, lypically using evaluation metrics such accuracy. AUBC provides concise way to compare the overall performance of different learning curves of query strategies. Figure 3 demonstrates that US, BALD, and LAL achieve higher accuracy more quickly than Uniform, corresponding to the mean AUBC of US (85.78%) and BALD (85.72%), which are better than LAL (85.52%), Uniform (84.77%), and Core-Set (84.47%) in detail. Furthermore, we report the accuracy of the data sizes and data utilization rates of the query strategy for more detail.
model under different labeled
3 Experimental settings
We employ most of the settings outlined in the prior benchmark (Zhan et al. 2021 For each dataset D, we reserve 40% as the unseen test set Die for performance evaluation. Then, for the remaining 60%, our default protocol is uniformly sampling few examples as the initial labeled pool Dj and leave the others the unlabeled pool D..
D = Der Dre
Drel = 0.4
Drr = Dj U Du,
[Di = & x [V],

---

Published in Transactions on Machine Learning Research (06/2025)
Learning curves on Australian
0.84
Accuracy
0.82
0.80
Uniform
US
BALD
0.76
Core-Set
LAL
50
100
150
200
250
300
350
400
labeled examples
Figure 3: The learning curves (test accuracy vs. number of labeled examples) of query strategies on Australian dataset (Chang & Lin 2011).
where k is the positive number to control the size of the initial labeled pool. In the following, we clarify the differences and expansions in our benchmark compared to the previous benchmarks Yang & Loog 2018
Zhan et al. 2021)
Remove improper use of query strategies The previous benchmark simultaneously evaluated query strategies that either support or do not support multi-class or batch size greater than one, may have affected the validity of claims when comparing results across different aspects (Zhan et al.) 2021). Therefore, restrict our evaluation only on valid use of query strategies to ensure consistency and fairness.
Include comprehensive datasets with unified format. We select 26 binary datasets from Zhan et al. 2021) and (Yang & Loog 2018). Besides, we ensure consistency in the source datasets and the composition of the initial labeled and unlabeled pools. For instance, we scaled raw data features to [-1, 1] for all datasets
We added datasets from the other tabular data benchmark (Grinsztajn et al. 2022 expand the coverage of the binary classifications. These datasets were selected based on their large-scale, class-imbalanced,, and high-dimennsional properties to better reflect real-world scenarios. We also extend our benchmark by adding 8 multi-class classification across diverse fields from UCI datasets (Dua & Grall 2017). Moreover, we exlend 2 domain-speeific datasets, including utilization of Vision Transformer (ViT) (Dosovitskiy et al. 2021) as feature extractor for CIFAR10 /Kiizhessyy et al.) 2009),, and BERT (Devlin et al. 2019 for IMDB Maas et al. 2011), to connect our findings with modern machine learning research. Please refer to Table 17 for the properties of 40 datasets.
(021) extended the query strategies from Yang & Loog 2018 to 17 query strategies. However, the redundancy of query strategies, such US and Informative Cluster Diverse (InfoDiv) (See Appendix B.5 for more detail.), may lead to repetitive and limited insights into the benchmark. Therefore, we only keep the most representative 12 query strategies: US, QBC, Hier, Graph, Core-Set, HintSVM, QUIRE, DWUS, MCM, BMDR, ALBL, and LAL. We further expand the
Include broad types of query strategies. Zhan
We retained the original scaling for some of the LIRSVM datasets, such Heart, lonosphere, and Sonar, which already scaled the range of [-1,

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: Settings of query-oriented models H for specific query strategies @.
H
Reason of choice
HintSVM
RBFSVM
the implementation in libact the implementation in libact the inheritance of [Zhan et al. (2021)
QUIRE
RBFSVM
QBC
LR(C = 0.1), RBFSVM, RF,
Linear Discriminant Analysis
ALBL
Combination of multiple Q with
the default settings in libact
same H: US, HintSVM
LAL
RF
the implementation in ALiPy
benchmark to explore broader range of query strategies by including BALD, a popular query strategy in deep learning (Gal et al. 2017
Adopt a tree-based model. Previous benchmarks studied Logistic Regression and RBFSVM In this work, we further studied tree-based models such as XGBoost (Chen & Guestrin 2016 and Random For- est (Breiman 2001h (See Appendix as recommended by the earlier benchmark for tabular datasets (Grin- sztajn al.) 2022). To clarify the relationship between query-oricnted and task-oriented models, we report some query strategies that. do not use tree-based model the query-oriented model in Table 2
We disclose the construction of the initial labeled pool, data preprocessing steps, and the choice of models, which can significantly impact the experimental results (Ji et al. 2023). that saves participants time examin- ing the settings and critical considerations for designing active learning experiments. Notably, Ji et al. (2023) recommended using consistent "initial sets" across multiple runs to minimize the impact of randomness and ensure fair comparisons. our study, randomness arises from two main sources: the train-test split used to derive datasets Du. Die and the construction of initial sets for D1, Du (See Section 3). While randomness from the train-test split is unavoidable without predefined training and test sets, adhere Ji et al. (2023) 's suggestions by keeping the initial labeled sets fixed to mitigate randomness. A detailed investigation into
the effects of randomness due to the train-test split is presented in Appendix
C.3 In summary, our findings
indicate that Uncertainty Sampling exhibits consistent performance across most datasets, confirming its stability within our benchmark.
Handle errors and exceptions of experiments. We report the issues encountered and solutions when we conduct experiments. Because current modules cannot support cold-start problems we run the experiments repeatedly and skip any seed that lacks labels in the training test set at the initial setup. For exccution, set a maximum rumning time of 72 hours for executing query strategy on a dataset to ensure completion within reasonable time (Denote 'TLE' in Table G).
This section outlines the necessary information to conduct experiments for the benchmark. In our imple- mentation and report, we strive to ensure the reprodueibility of all results under these specific settings and processes corresponding Figure 2 Furthermore, we compare our settings and results with the existing benchmark (Zhan al.) 2021 in Appendix B and Appendix C covering any additional modifications or improvements needed.
4 Benchmarking results
This section presents the benchmarking results for XGBoost in Table 3 We repeated experiments 100 times for small datasets with a size less than 2000 (Ks = 100) and 10 times for large datasets (K1 = 10). We set a total query budget of 3000 to reduce rumning time for large datasets. Next, we verify the superiority of Uncertainty Sampling over other query strategies. Furthermore, we investigate whether existing query
We also reproduce the previous benchmarks with different base models in Appeudix
The cold- start problem is that some classes are under-represented in the initial labeled pool Yuan et. al. 2020).
2020 Brangbour
9

---

Published in Transactions on Machine Learning Research (06/2025)
Table 3: Benchharking results of XGBoost. The numbers are mean AUBC (f better). We report the bascline method (Uniform), the best query strategy with its mean AUBC (BEST QS, BEST), and the worst query strategy with its mean AUBC (WORST_QS) WORST).
Uniform
BEST QS
BEST
WORST QS
WORST
Appendicitis
81.51%
US
82.85%
DWUS
80.74%
Sonar
75.10%
US
76.06%
Core-Set
74.11%
Parkinsons
84.24%
US
86.63%
HintSVM
83.12%
Ex8b
84.21%
LAL
85.16%
DWUS
83.05%
Heart
78.37%
BALD
79.35%
HintSVM
77.83%
Haberman
67.69%
US
69.17%
HintSVM
66.82%
Ionosphere
87.96%
US
89.95%
DWUS
81.85%
Cleanl
76.54%
US
78.99%
Graph
76.30%
Breast
95.57%
LAL
96.31%
DWUS
91.85%
Wdbe
94.07%
LAL
95.24%
HintSVM
93.96%
Australian
84.77%
US
85.78%
HintSVM
83.89%
Diabetes
72.62%
US
73.62% HintSVM
71.49%
Mammographic
79.46%
BALD
80.78%
DWUS
78.80%
Ex8a
92.06%
Core-Set
94.07%
HintSVM
84.75%
Tic
90.11%
US
90.65%
DWUS
89.11%
German
72.68%
US
74.03%
DWUS
71.78%
Splice
91.89%
US
93.76%
DWUS
89.51%
Gcloudb
87.85%
LAL
88.68%
QUIRE
85.99%
Gcloudub
92.98%
US
94.30%
DWUS
86.12%
Checkerboard
98.72%
LAL
99.49%
DWUS
86.83%
Spambase
93.16%
US
94.51% HintSVM
91.09%
Banana
87.70%
LAL
88.45%
HintSVM
79.70%
Phoneme
85.78%
US
87.77%
DWUS
82.42%
Ringnorm
93.76%
US
95.46%
Core-Set
64.58%
Twonorm
95.43%
US
96.39%
HintSVM
83.38%
Phishing
94.20%
US
96.24%
DWUS
91.68%
Covertype
74.11%
US
76.64%
DWUS
61.34%
Bioresponse
72.92%
BALD
74.50%
Core-Set
72.04%
Pol
96.03%
BALD 97.62%
HintSVM
90.52%
strategies bring more benefits than Uniform for each dataset. After comparing the performance of query strategies in binary classification datasets, we further initiate the experiments for multi-class classification and domain-specifie problems to enrich the scope of this work. In addition, we reproduced the benchmarking results from (Zhan et al. 2021 with RBFSVM in Appendix C and constructed the new benchmark for RF in Appendix
4.1 Verify superiority
Referring Table we observe that US attains the highest mean AUBC among all query strategies on 18 datasets, indicating its superior performance compared to other query strategies average. The remaining dominant query strategics are LAL and BALD, which achieve the highest AUBC on 6 and 4 datascts, respectively.
Besides AUBC, we also observe learning curves from different perspectives. Specifically, we check the model's accuracy with varying ratios of labeled examples on each dataset. Table the model's accuracy with 20% labeled examples on each dataset, and US outperforms other query strategies on more than half (15) datasets. Please refer to Appendix A for more comparisons under other ratios. Beyond using fixed query
10

---

Published in Transactions on Machine Learning Research (06/2025)
Table 4: Accuracy (f is better) of the model with 20% labeled examples: We report the model's accuracy with 20% labeled examples on cach dataset. The scores with bold indicate the best performance, and with italics indicate the second-place performance on dataset. "TLE" means a query strategy exceeds the time limit.
Uniiorm
QBC
BALD
Hier
Graph
Cres-Set
HIISVV
QUIRE
DWUS
MCM
BMDR
ALBI
LAA.
Sonar
67.22%
67 29%
66.55%
86.73%
68.45%
67. 67.83%
06.25%
65.88%
07.993
68.15%
86, 86.82%
68.56%
19
61. 61.25%
67.32%
Parkinooss
79.26%
80.71%
79.36%
80.512
79.40%
78.90%
78.68%
78.41%
78.03%
29,09%
#0,1006
78,81%
50.27%
Escilo
Heare
80.55%
76-73%
80.69%
1956
79.98%
76.06%
77
70.37%
80.24%
75. 64.6
76.01%
79.135%
76.6096
61.11%
74.82%
80. 80.32%
76. 96%
$1.265
75
78. 76%
UR
75.912
81.96%
51.74%
76 76.30%
80 60%
Haberman
09.247
70. 61.55
69.257%
10.160%
68.54%%
$3.247
67.68%
81. 81.845
68.43%
67 61%
68.1175
71.04%
69.1555
70 70.00% 69.07%
Tonosphere
63.58%
86. 96%
63.84%
Cleanl
68.34%
69.86%
08.00
09.1000
68.6
68.667%
66.7496
67.59%
68. 12%
68
07.965
67.
66.1295
09.319%
Breset
96.1756
96.78%
46.17%
95.
05.545
95
94.344%
14. 94.77%
96 68%
Wilbe
92.9156
95.34%
92.600%
02.9475
03.0047
91.94%
92 02200
92- 5226
95.56%
Disbetes
Arsttaaiion
71.8476
83.6336
85.55%
78.96%
83.77%
72.0
83 6
72
53.000%
T1..7000 6
T1.
89 97M4
70. 1
81. 11.95%
71 71-5116
82
5227754
-
82
88
88
72. 72.80%
85. 1.%%
Mammographie
Excaa
86.22
79.6676
82.51% 5
68.55%
79.087%
83.9400
ST $7.9455
79. 49%
79.1736
91.
79.2274
78. 18.70%
80.41% 10.4
78.
79 12020
2000
820988
80
91.885
88 67%
Tie
Gorman
89.25%
71:2336
72.999%
90.43%
71 71.37%
71 71.045
89.725
70.4000
89
39.3300
71. 71.61%
88, 18.122%
-
80
71
850
85,
68
11.7%
49,588
89 3229
73. 12.11%
Sphie
89.0776
922.55%
8878
95.600
89 0002
86.7480
$1.11
88-9912
84 1000
91.388
96
Gelondb
87.82%%
899.55%
87. 17.96%
87. 17.9999
88.1988
1999
88
56.275
84 64.22%
85, $5
89,498
87 8
Gelondub
95.25%
94.11%
91.
01.92%
86-5823 22
86
63. 11%
85. 62.6
80
50.2155
91.0
Spanlose
Chookebbord
98, 7636
92:47%
94.84%
96 80%
08 80%
92.68 0
90 35%
92. 5.
92 1
92.000
DK 5099
68. 95%
01. 40%%
88. 72%
92.50%
70 9
92. 24
191
46%%
00
99
BsTLE
87 4674
87. .3.%
87.465 46%
57.66%
87. 53%
88 0023
71 71.37%
76. 11%
74, 1..
58,49
86
55.945
Phoneme
83.0736
87. 1956
83.73%
87.24%
844 66%
84.195
84.700%
80. 80.83%
T1.5%
78 60%
96.36%
66.52%
Ringtorm
92.79%
7535
93.33%
95.777%
92.355%
92.4015
51.8676
57.27%
TLE%
55.715%
92,4
92.5500
Phishing
Twenorm
Cownttye
93.415(
722.9156
S1..9%%
96.19%
76.26%
96.61%
95.04%%
72.05%
90.533%
75.86%
96.04%
95.07%%
9/2 83%
11..%
95.24%%
98.35%
64.5455
98, 1864
96.73%
11.636
91, 73%
62.00%
79.31%
11.E%
T14%
TLE%
88.11%
94.20%
50.017%
45,7VM
90.60%
96.
71.54%
95.529%
000
69.900
T3.92%
95.907%
50.9
Diccesporae
09.966%
73.14%%
71.05%
72.897%
40.4
71.11%
06..00%
70%
T1E%
60 0
TLES
10.339
72.07% .7.00%
Pol
95.29%
98.19%
95.52%
98.1
95. 40%
86.38%
93.68%
80.75%
TLE%
95.29%
97.58%
TLES
95.18%%
97. 97.73%
budget in Table 4 we also check the metric of the number of queried examples required to reach 99% of the
model's performance trained on the full budget. Table 5 demonstrates that Uncertainty Sampling stably
achieves first place or second place on 21 datasots, which shows consistent conclusion with Table 3 and Table 4 More results are revealed in Appendix A
Finally, we verify the ranking performance of query strategies across multiple datasets. Specifically, we assess the average and standard deviation of the rankings by sceds of the query strategy on cach datasct. Then, we apply the Friedman test with a 5% significance level to test for statistical significance. The p-values of the Friedman test are less than 5% for all datasets, indicating that the performance differences between query strategies are statistically significant. Table G demonstrates that US ranks first on 18 datasets, and LAL, BALD, and MCM often achieve second and third ranks.
These results show that the straightforward and efficient US outperforms others on most datasets. These outcomes also correspond to previous work claiming US is the strong baseline with LR (Yang & Loog] 2018) and RBFSVM, which we re- re-benchmarked in Appendix [ We recommend that practitioners initiate their pool-based active learning projects with US
4.2 Verify usefulness
We investigate the usefulness of query strategies in Section 4.2 The analysis of usefulness can uncover which query strategy brings more benefits than Uniform, offering practitioners a reality check on the effectiveness of query strategy. Specifically, we investigate the improvement of the optimal stopping point of query strategies over Uniform. The optimal stopping point is the point where the model achieves the target accuracy with the least number of labcled examples. We refer to the data utilization rate (Culver et al. 2006), which is the number of labeled examples to achieve the target accuracy divided by the number of labeled examples required by Uniform. In this benchmark, we set the target accuracy as the accuracy with the total query budget minus 0.01. Table shows the data utilization rate of the optimal stopping point of query strategics over Uniform. We observe that US, BALD, MCM, and LAL achieve higher data utilization rate than Uniform on most datasets.
11

---

Published in Transactions on Machine Learning Research (06/2025)
Table 5: The minimum number queried examples required to reach 99% accuracy (L is better) of the model: The bold indicates the first place and italics indicates the second place. 'TLE' means a query strategy exceeds the time limit.
Uniform
US
QBC
BALD
Hier
Graph
Core Set
HintSVM
QUIRE
DWUS
MCM
BMDR
ALBL
LAL
Appendicitis
28.44
25.58
29.32
25.08
25.30
27.50
26.03
25.63
26,522
31.49
24.71
28.68
25.43
24.74
Somer
73.45
65.70
68.1
63.81
72:34
75.91
77.40
73.35
76.83
T6.70
70.33
71.45
67.15
63.78
Parkinsons
66.44
44.56
66.81
47.26
58.10
75.95
65.68
81.31
70.40
67.65
49.78
60.99
59.36
45.41
ExSb
51.02
42.87
54.42
46.34
45.86
55.63
48.27
58.98
52.16
66.05
43.28
49.59
43.56
45.09
Heart
47.75
41.12
48.64
40.14
51.24
45.84
48.88
52.44
46.48
51.28
44.87
45.48
45.19
40.35
Haberman
25.53
24.59
28.73
25.35
27.02
27.17
29.75
30.51
29.40
28.74
26.36
26.81
28.67
25.13
lonosphere
97.04
53.38
08.79
55.
85.08
88,46
104.41
96.79
179.88
06.03
120.41
$7.49
Cleani
207.79
152.39
207.99
155. 50
201. 93
211.52
191.98
199.64
196.
207.79
168.59
199. 24
180.07
166.30
Treast
70.81
38.79
65.98
$8.02
56.25
90.22
61.85
64.05
288.68
40.14
81.19
39.19
30.39
Wibe
102.53
48.87
111.69
48,00
89.52
106.51
88.53
110.64
92.47
110.06
51.92
100.71
49.67
44.94
Australian
88.71
55.39
79.43
59.18
86.36
101.76
119.76
96.14
104.74
06.84
96.08
77.46
64.00
Diabetes
56.67
49.28
S7.35 50
54.89
53.78
58.07
39.12
104.25
62.42
68, 29
57.51
56.83
56.63
44.90
Macnrogrrahic
30.29
28.73
39.66
27.09
42.00
37.80
35.29
50.08
30.52
31.67
28.13
27.10
Exsa
263.54
19L.79
250.51
193.59
241-49
261.52
154.35
379.03
363.91
312.23
188.39
204.05
367.55
268.98
Tic
144.75
50.68
148.10
89.12
137.00
151,89
194,80
142.64
185,20
119.46
TLE
168.54
132.92
German
178.81
124.03
163.55 55
135. 11
184.64
177.15
177.50
164.53
169.58
256.71
144.83
TLE
146.80
129.
Splice
310.57
134.00
305-19
140.50
307.94
285.83
264.18
321.10
290.00
444.19
158.67
302.19
274.96
214.72
Geloadb
56.49
36. 53
63.12
38.09
48.29
71.45
42,10
116.48
24
73.58
37.54
45.44
45.08
34.30
Celonduh
239.57
60
228.82
140.76
194.34
252.24
309.11
510.70
355.72
512.86
150.74
270.05
206.95
119.19
Checlerboard
140.40
233.72
146.03
91.92
100.84
121.44
150.
528.25
529.48
619.52
72.11
107.50 50
151.65
37.22
Spombase
941.80
965.50
247.50
T80.30
1060.40
991.00
1670.10
TLE
965.40
322.60
TLE
734.70
124.20
BaTLEa
540.70
456.30
450.10
537. 10
509. 10
$79.50
319. 22
2081.00
1590.80 80
2618.00 8.00
489.60
TLE
729.50
221.90
Phoneme
1742.80
611.30
1665-60
622, 710
1248.10
1750.50
1332.40
1792.80
TLE
2600.90
750.70
TLE
1501.40
738.90
Ringnorm
1917. 10
386.90
1150.40 40
410.00
1400.60
1352, 20
2564.60
1959.
TLE 2066.00 .0
546.80
TLE
1189.30
922.80
Twonorm
$25.50
179.40
622.
189..00
$74.90
531.10
338.20
2638.30
TLE 921.80
222.00
TLE
382.50
375. 23
Phishing
1080.50
262.60
1386.00 60
273.50
1244.70
1396.00
1151.10
20.00
TLE 2500.00
374.50
TLE
578.00
322.90
Covertyp
1981
796.50
2115.80
842.50
TLE
20.00
TLE
TLE
TLE
20,00
TLE
TLE
TLE
1006.00
Dioresponse
1182.30
771.00
1262 50
793.50
1112.20
1160.90 80
1068. 80
TLE
TLE
1182.30
908.50
TLE
TLE
777.40
Pul
1001.00
288.90
958. 70
272.40
930.90
865.00
1501.00
300.00
TLE
1001,00
101.30
TLE
1236,60
312.20
To further investigate the usefulness of US, chock the improved accuracy (r) of US, BALD, Core-Set, and LAL over Uniform on effective dataset (Coverlype) and ineffective dataset. ( (Checkerboard) on average with different scales of the total budget. Figure 4 shows that the performance of US and BALD gains significant benefits on large scale dataset. However, US suffers from the sampling bias on Checkerboard with a small budget, while BALD is more stable. We notice that a query strategy with good performance brings more benefits at the early stage of the learning process.
Differance of accutacy on Covertype
Difference accuracy on Checkebbard
5.20
0.04
LAL
3.00
20% botal budget
0.02
Fertomantce Inperrmm11
5.00
5.06
-0.02
0.50
BALD
Care Sec.
0.53% total busget
505
1020
1530
2006
2503
200
400
607
500 1000
No. of libe oxanples
No. abdedd exama
Figure 4: Mean difference of accuracy (improvement) of a query strategy from Uniform on Covertype (left) and Checkerboard (right). Note that there results of Core-Set on Covertype due to the time limit (TLE).
12

---

Published in Transactions on Machine Learning Research (06/2025)
Table 6: Average Ranking of Query Strategies (4 better): We report query strategies with the best average ranking. The scores with or mean the 1st, 2nd and 3rd performance on datasct. 'TLE" means a query strategy exceeds the time limit.
US
QBC
BALD
Hier
Graph
Core-Set
HintVMM
QUIRE
DWUS
MCM
BMDR
ALBL
LAL
Appendiciris
4.83
7.50
5.51
7.63
8.30
7.34
7.82
9.12
5.20
7.85
6.92
5.83
Sonar
4.79
4.97
7.52
6.19
Parkinsous
3.39
8.41
3.79*
7.03
8.37
10.92
8.47
8.74
4.59
7.26
6.31
4.24'
Ex8b
5.31
7.95
6.34
7.60
5.83
9.09
7.42
10.07
5.08
7.13
5.50
5.40
Heart
4.89
7.72
4.76'
7.66
8.17
7.18
9.65
8.34
6.43
6.33
7.10
7.04
5.73'
Haberman
1.38
7.69
4.50
7.04
8.10
8.97
9.76
9.55
6.07
4.74
7.07
7.95
5.12
Tonosphere
2.67!
7.19
2.89a
7.12
8.22
8:28
9.38
10.10
12.74
4.51
10.08
4.30
3.52
Cleand
2.61
8.13
9.96*
8.33
8.76
7.87
8.47
7.39
8.59
5.30
TLE
5.53
4.06'
Breast
3.98
8.72
72
7.14
10.02
7.43
8.85
7.50
12.97
4.24
8.64
4.88
2.88
Wilbe
3.67
9:47
3.44
8.15
9.35
8.41
9.57
9.02
9.54
3.95
9.59
3.71
3.13
Australian
2.80
7.75
3.90
7.70
7.47
8.97
10.38
8.65
4.66
8.30
8.32
4.44"
Diabetes
3.64
7.15
4.354
6,88
7.09
10.44
10.01
4,98
7.20
8.14
3.51
Mammographic
3.46
S.45
7.51
Exca
4.37
4.42
5,90
6.16
1.70
11.68
10.75
10.24
4.43
8.91
3.31
Tic
2.65
5.66
9.99=
6.47
4.59
6.16
German
3.10
7.29
3.52
7,81
7.23
6.87
8.21
10.92
4.14
TLE
6.02
5.45
Splice
1.52
7.30
1.86
7.18
8.61
9.29
9.82
6.40
11.62
3.37
TLE
6.62
4.41
Geloudb
4.24
7.25
4.69
7.51
8.19
5.93
10.71
11.01
11.67
4.02
7.07
5.56
3.12
Geloudub
2.52
6.41
3.68*
4.91
7.13
8.75
12.44
10.70
12.45
4.51
7.60
7.05
2.85
Checkerboard
6.37
7.15
4.94
5.03
7.44
6.54
11.36
11.48
12.81
3. 3.72"
4.58
8.34
1.24
Spambase
1.50
7,80
10
6,80
8.10
8.30
11.00
TLE
7,80
3.30
TLE
6.20
3.50
Banana
5.20
5.70
5.70
33.00
8.20
3.00P
10.60
TLE
10.40
5.00
TLE
7.20
1.40
Phoneme
1.60
8.10
2.00
5.20
8.40
6.40
10.00
TLE
10.90
3.30
TLE
7.00
3.10
Ringnorm
1.40a
5.10
1.60a
6.30
8.00
10.50
9.00
TLE
10.50
3.00
TLE
6.30
4.30
Twonnrm
7.40
TLE
10.00
Phishing
1.40a
7.90
1.60*
7.20
8.70
6,20
10.10
TLE
10.90
3.60
TLE
3.40
Covertype
1.40
3.80
1.90a
TLE
5.00
TLE
TLE
TLE
6.00
TLE
TLE
TLE
2.90
Bloresponse
1.90
6.40
1.60'
6.40
6.20
8.60
TLE
TLE
6.60
3.70
TLE
TLE
3.60
Pol
1.80
5.80
1.50
6.20
9.80
9.20
11.00
TLE
6.70
4.00
TLE
7.30
2.70'
Tables A demonstrate that the Checkerboard and Banana datasets pose challenges for Uncertainty Sam- pling. Both are synthetic two-dimensional datasets introduced in previous works (Alcala-Fdez et al. 2010 Konyushkova et al. 2017). To study the cause of the Uncertainty Sampling's failure, we visualize scatter plots at selected rounds throughout the active learning process. Specifically, we examine the size of labeled pool at 20, 200, and 800 for Checkerboard, while 20, 400, and 800 for Banana. These rounds correspond to the initial labeled pool, the round at which Uncertainty Sampling performs worse than Uniform, and the round at which Uncertainty Sampling achieves comparable performance Uniform, as observed in the learning curves for each dataset presented in Figure
Figures G (Checkerboard) and 7 (Banana) illustrate existing unexplored regions either at the initial during intermediate rounds. Our analysis reveals that when datasets have multiple overlapping positive and negative regions, Uncertainty Sampling lends to query examples from these overlapping regions rather than exploring less-covered regions, particularly due to the uneven distribution the initial labeled pool. In this paragraph, we analyze the possible reasons for the failure US. Other related works investigating the causes of US failure is discussed in Section 5.4
4.3 Expanded benchmarks: multi-class classifications and domain-specific data
4.3.1
Multi-class classification datasets
We extend the evaluation to include F Filmulti-cla classification problems to demonstrate the broad scope of the benchmark and improve the validity of the US competitive edge. The multi-class datasets cover several fields of applications such as biology, physics, climate, business, healtheare, and social science, which reveals
We exclude RT T2022 for this experiment because the highly imbalanced ratio of the dataset results lacking classes the training when initializing the labeled pool by the default protocol (See Section
13

---

Published in Transactions on Machine Learning Research (06/2025)
Table 7: Data utilization rate (L is better): We report the data utilization rate of query strategies. The bold indicates the first place and italics indicates the second place. The scores with pink color indicate that the query strategy does not provide more benefits than Uniform. 'TLE' means a query strategy exceeds the time limit.
U8
QBC
BALD
Ilerr
Graph
Core Set
QUIRE
DWUS
MCM
BMDR
ALBL
LAL
Appendicits
77.90%5
03.07%
71.10%
58-47%
84.5735
77.22%
73.31%
81.7458
101.34%
73.7315
58.00%
72.73%
71.815
Sount
94.00%
99.31%
92.74%
100.9096
109.4195
B16.1596
307.00 62%
130.0676
105.5496
108.2496
200.
94.73%
Parkinsons
65.14%
100.21%
70.28%
86.74%
116.07%
99.84%
123.6450
100.3450
101.74%
74.09%%
02.52%
90.79%
07.39%
Ex5b
90.08%
85%
93. 96%
96. 50%
107 2655
11.14%
126. 007
104.1 10%
139-30%5
90.31.3
103.0
90.10%
03, 70A6
Heart
86.30/
107.95%
84.11%
177.1716
97. 50.5
109.
113.07%%
103.6256
109-4855
86.77%
91.87%
80.8056
Haberman
82.19%
25.22%
52. 67.%
111. 45%
1005 32%
120. 8009
161 25%
815
107.
912 1959
318.2756
1000
Toncephere
61.01%
111. 800
63, 5995
105,50 86%
108.5 9.95
06.76 76.7
110.
111.235
209 0215
76. 131
130 62%
78. 0..5
65, 46%
Clann
75.775%
96966
90.94%
2005
94.659%
48%
96
82.711%
99.600%
92.04456
Brose
00.355
100.64%
66, F498
59.34%
147 4796
05.755%
100, 65%
96, 699
404 555%
71 71.57%
153.419
66 73
49.26%
Willae
64.8055
138-42%
68 79.9
10s
188. 7605
41%
23H 111%
116.98%
117.16%5
6% 4224
119.89%
64.4%
69.28%
Austzollan
77.16%
115.07%
2799
107 19056
128. 4890
844.82%
350
127. 201%
190-52256
92.17%%
128, 128.82%
111 125
Dishrses
03.00%
102.01%
107. 05%
89.8 59.919%
114.9598
108. 809
181 5005
112 201
171.75%
04. 10.009
105,489
101 62
76.47%
Mammographic
02.0755
120.26%
18%
13411356
125. 125.8495
112 112.22%
237 911
103 2000
202 17%
88.0105
119.100
82 669
79.15%
ExNa
5038256
101.76%
20%
57. 18%
110.1296
62.62%%
1344-1406
75.06%
84.022%
69.71%
Thc
80, 79
128.2 22%
79. 62%
118 5255
137 8899
15 45%
147. 2015
11 52%
164.545
L05. 3499
TLE
137 67%
118. 439G
German
89.07%
110.68%
1003 79
125 8256
122.5195 96
139. 699%
110.0
124 66G
100.7396
113.27%
TLE
113 113:415
95.44%
Splice
50.12%
109.42%
06.96
130.8456
101 5676
965.6495
110.00%
57. 0222
163. 10%6
105.909
4056
Geloudh
73.44%
124.36%
F1. 91.99 56
100, 1250
145. 6995
78.12%%
222 5055
250 A350
2016
72. 12.86
85,25%
72 2005
64. 865%
Geloudub
07.80%
118.30%
79.222%
1002 1856
1311 47%6
168.099
1111111
15% 188
303.5756
MT. 16%
106.31%
705
06.35%
Checkoeboordd
251 5258
140.07%
65%
1.0.3
115.66 6415
15c.
520,0855 0875
543, 188
660.77%5
72.377
105.04%
150 2850
39. 50%
Spammhase
25.29%%
97.50%
0116
60%
9776
0099
136.13%
TLP.
101.91%
TLE
Bannna
111 5155
46.73%
227
1000 54166
117 4776
66. 76566
44%.34%
074.3/56
TLE
1.02 5056
43.66%
Phoneme
35.59%
97.05%
35, 5755
73.3-1%
100
50.849%
107.005
TLE
165.20/5
45. 8555
TLE
82.
44.67%
Ringnorm
$1.06%
91.
5996
208.70 7099
158
TLE
195-4815
44.9995
TLE
94.07%
73
Twonorm
84.69%
116.91%
36. 73996
112 166
1408.9756
70.30%
529.37%
TLE
178.405
43.0225
TLE
73.00%
74.5456
Phishing
27. %0
131.84%
28, 295
117.785
137 8055
101 8999
57.30%
TLE
261.87%6
37. 2575
TLE
38.399%
31.98%
Covertype
$1.2335
40.35%
TLE
TLE
5010
TLE
TLE
TLE
Bloresponse
63.05536
105.42%
64.56%
80.29%
7656
91. 91.78%%
TLE
TLE
10000075
73.9595
TLE
TLE
06.7256
Pul
29.9850
96.56%%
27.82%
96 41 5%
08.11 11%
162. 182.589
34.08%
TL.
100.00%5
11.0695
TLE
126. 70%
S1 8-455
Learning curves on Chec beecre
Learning curves on Banana
1.03
6.50
0.92
2.5
0.92
5.50
0.07
0.03
0.70
0.72
0.73
5.05
0.63
1n/iomm US
US
Unfim
630
cio
1035
s6o
1050
1930
2000
2500
3300
No. of bib and moumples
No. of abriee reamooie
Figure 5: The learning curves (test accuracy vs. number of labeled examples) of query strategies on Checker- board (left) and Banana (right).
the value of this benchmark for real-world applications (See Table 17 for details). As an initiating demo, we only adopt the query strategies that are valid for the multi-class classifications and representative of each category of query strategy described in Section Therefore, we choose US, BALD, MCM, and Core-Set. In particular, the different uncortainty measures US behave differently for multi-class classifications (Sctties 2012), so we compare the least confidence (US-LC), the smallest margin (US-SM), and the maximum entropy (US-MR) to verify their performance.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Dveceerbeard Skee 4 mued with Accenacy 0.80
Orocoetoaard Sked murd 330 Acceracy 000
Cwckerborr (Seed 4 di 783 TastAccaroo
Figure The scatter plots of Checkerboard at 20 (left), 200 (middle), and 800 (right) labeled examples. We denote red points negative examples, blue points as positive examples, and gray points as unlabeled examples. We mark example with a cross in labeled pool Dj and others with dots.
Uarura idead 2t round wwit Accurace G:l
Oarura iSend 23 mand 380 wit Kcatncc $:
Barana clead Lo round 200 last Acsurssyy 0.56
Figure The scatter plots of Banana at (left), 200 (middle), and 800 (right) labeled examples. The format the same the Figure
Table 8 demonstrates the superiority of uncertainty sampling with margins (US-SM) performing well in multi-class classifications. Although US-SM does not achieve first place in Iris and Wine, the difference between all query strategies, including Uniform, is insignificant in these datasets. In this experiment, we verify that US-SM with compatible XGBoost. models would significantly improve the model's performance for the multi-class classifications.
4.3.2 Domain-specift datasets
Our benchmark also includes domain-speciff datasets. One is CIFAR-10 Krizhevsky et al. 2009 which belongs to the computer vision (CV) domain; the other is IMDB Maas et al. 2011 belonging to the urtural language process (NLP) domain. We incorporate deep learning models, such as the ViT feature extractor for CIFAR-10 and BERT tokenizer for IMDB, to transform the images and texts to embedding spaces Then, we treat them as tabular datasets and follow the same active learning process described in Section
Table 9 shows that Uncertainty Sampling with different measures is still competitive to Uniform. In particu- lar, US-ME stands out from other uncertainly measures, and all query strategies achieve similar performance in this experiment. The results indicate that by utilizing the feature extractor to convert image or text data to tabular format, domain-specific tabular data essentially differs from the regular tabular structure.
In this section, we initialize the preliminary investigation on active learning for domain-specific scenarios. Specifically, we utilize feature extractors for CIFAR-10 10 and IMDB convert domain-specific data sets to tabular data sets and verify the feasibility of US for this protocol. However, there is still a lack of comparisons for other approaches. For example, the feature extractors for CIFAR-10 and IMDB chose are pre-trained on large-scale datasets and might be considered the 'external knowledge for the benchmark. Such "external knowledge' would be obtained by semi-supervised learning or self-supervised learning on both labeled and unlabeled pools (Zhang el al. 2023) or from foundation models (Guple el al. 2024). We leave the
We used the pre-trained VIT (bttps://hugginggaceeco wwt..////////etttteeeeeeeeee basttt--bbe--pcchh---- and BERT tokenizer https: //wuggingface.co/gge///// www.://ogeebee.tteeeeoooooeeeeee base- uncased] from Hugging Face repository
15

---

Published in Transactions on Machine Learning Research (06/2025)
Table 8: 'Mean + standard deviation' AUBC of XGBoost for multi-class classifications (1 is better). The scores with ' 2, denote the 1st, 2nd and 3rd performance on a dataset. 'TLE' denotes a query strategy that exceeds the time limit. The scores with underline denote its mean AUBC is greater than Uniform's 's 'mean standard deviation' AUBC.
Uniform
US-SM
US-LC
US-ME
BALD
Core-Set
MCM
Iris
92.05% 1.88%
94.05% 94.05% 10.85%
84.06% 1. 10.84%
94.01% 10.91%
94.07%++0885
94.45%.11.551%%
02.87% 10.5%%
Wine
92.40%+1..20 1.%
92.05%++0.500%
02. 02.888+00 +0. 4784
02.94%++0.52
90.14%% +-.52%
92.85%%
91.94% 9..%++0..%%%
Abalome
86.06% 27%
90.14%110.10%
90.04%
10%*
80.92% 1..1%
90.06% + 11%6*
86.51%61:0.699
89.57% 10,10..11 17.1%%
Academic Success
91.17%:::00 21%
92.30%.=0.0850
22, 22.28% (99)
02.21%++0.0995
92 22..9%%.0 08%
91. 11.20%%)222 25%
91.7%% 19%
Satellite
74.02%6:10.12 92%
74.80%
04.44% 26%
74.33%%110.3139
74.49%=00
25%
74.00% 26%
74.38%.10..01
Dry Besn
22.3994=:007777 17%
23.08%=0.000%
22, 95% 09%
22. 1..00%%
22.50%=001113
22. 02.3319622000000 015%
22.73%:22.006 08%
Diabotes 130
13.9278+00-43%%
35.26%%=0.44%%
54 64.39%% 8.78
53.77%%
TLE
54. 64.133%++011897
64.04%%++0.68%%
Table 9: 'Mean + standard deviation' AUBC of XGBoost for multi-class classifications (1 is better). The scores with 2, 2, or denote the 1st, 2nd and 3rd performance on a dataset. 'TLE' denoles a query strategy that exceeds the time limit. The scores with underline denote its mean AUBC is greater than Uniform's 's 'mcan standard deviation' AUBC.
Uniform
US-SM
US-LC 48.48% 48%
US-ME
BALD
Core-Set
MCM
CIFAR-10 97.59%
49% +0.02%
+0.00%
98.47% + 0.00%
98.51 t0..%% 93.98%
+0.68% 93.89%:10.06%
IMDB 93.75%:1..09%
93.88% 10.06%
10.0%.0006%
99.91 020 068%*
13.90%
investigation on the protocol choice for domain-specific datasets and its impact active learning methods in future work.
5 Analysis of uncertainty sampling
In this section, we first study the impact of model compatibility on Uncertainty Sampling, which clarifies the conflicting conclusions between our benchmark and the previous work of Zhan et al. (2021) (Section 5.1
Then, we extend the benchmark by evaluating the usefulness of Uncertainty Sampling on three real-world datasets, which are large-scale high-dimension used in the recent tabular benchmark (Grinsztajn et al., 2022) (Section 5.2) Lastly, we study the sensitivity of active learning protocols for Uncertainty Sampling, including imbalanced dalasets with the one-shol protocol, hyper-paramettess/ model complexity, and query batch sizes (Section 5.3 We also present the limitations of Uncertainty Sampling to remind AL practitioners of the ineffectiveness of some tabular classification scenarios (Section 5.4
5.1 Impact of non-compatible models for uncertainty sampling
In contrast to the broader performance comparisons in earlier sections, Section 5.1 focuses on the model compatibility with US. Our investigation demonstrates that the incompatibility between query-oriented and task-oriented models significantly influences the performance of US. An example of model incompatibility is that the previous benchmark adopted US with LR(C 1) as the query-oriented model and RBFSVM as the task-oriented model /Zhan et al. 2021 6 Through careful analysis, we found that when non non-compatible models are used (denoted as US-NC), the performance of US (denoted as US-C) notably drops, as shown in Table 19 This drop is primarily due to the misaligument of the decision boundaries between the query- oriented and task-oriented models, which can lead the query-oriented model to select samples that are not the most uncertain for the task-oriented model, illustrated in Figure In summary, our benchmarking highlights that by utilizing compatible models, US-C consistently performs better than US-NC average.
We compare different combinations of query-oriented and task-oriented models based on LR, RBFSVM, and RF. Figure g and Appendix C.4 emphasize that compatible model pairs perform better than non-compatible
6 6See
Zhan
2021
implementation for more details https://github. http://thhubbccom/inen/e cow.S/neZ/A/// www.ineeeggggleeeeeeeeoe google-binary-ee
ConparativeSurveyy...ac htt.a///oo.oooaaaaaaaaeeaaaaaaaa
16

---

Published in Transactions on Machine Learning Research (06/2025)
Figure Given RBFSVM as the task-oriented model, we study the non-compatible query-oriented model with LR(C- 0.1). The red and blue points represent labeled examples. The gray points represent unlabeled examples. The cyan and magenta lines indicate the decision boundarics of query models LR(C 0.1) and RBFSVM trained current labeled examples. If we adopt US, the non-compatible setting queries sample (orange circle) that is most uncertain to LR(C* 0.1) rather than the most uncertain sample to RBFSVM (red cirele).
model pairs for US, evident across 22 datasets, where the optimal AUBC score occurs with compatible models, i.e, the highest AUBC score is found along the diagonal. Although some results demonstrate that non-compatible models are slightly better than compatible models, such as Splice and Banana Figure these instances were exceptions rather than the norm our benchmark.
IR(C=1)
SVM(RBF)
RF
[R(C-1)
SVM(RBF)
RF
LR(C-1)
85.42%
84.50%
85.81%
UR(C-1)
92.79%
93.80%
95.87%
SVM(RBF)
85.42%
84.92%
85.87%
SVM(RBF)-
92.68%
94.68%
96.52%
RF
85.41%
84.72%
86.15%
RF-
92.57%
94.39%
96.73%
Figure 9: Mean AUBC of query-oriented model (rows) and task-oriented model (columns) Australian (Ieft) and Phishing (right)
In summary, we advocate for the default use of compatible model parings in US for practical applications. This setting simplifies the model selection process and can potentially yield better performance across various datasets.
17

---

Published in Transactions on Machine Learning Research (06/2025)
5.2 Extending the usefulness of uncertainty sampling
We extend the existing benchmark to real-world datasets used in another tabular data benchmark (Grinsztajn et al. 2022 to demonstrate the usefulness of US within our current benchmark and its potential applicability and benefits across a more comprehensive array of real-world datasets. Real-world datasets include a larger number of examples, such as Pol and Covertype, and higher dimensions, such as Bioresponse. By extending our evaluation to these datasets, we aim to illustrate that the consistent usefulness of US is not limited to the existing benchmark.
In Figure 10 similar to Section 4.2 US could bring more benefits than Uniform the early stage. These results affirm that US has potential applicable approach across large-scale and high-dimension scenarios, which encourages the exploration of US in broader applications.
ferennce of accuracy Bioresponse
Differencc of couracy Pol
0.05
0.04
Cere-set
BALD
Lotal budget
5.00
0.02
Pertemmace Pppoomeett
0.02
-6.35
-1.70
10% of
1290
1750
300
1020
2200
Figure 10: Mean difference of AUBC (improvement) of query strategy from Uniform on Bioresponse (left) and Pol (right).
5.3
Sensitive of active learning protocols for uncertainty sampling
5.3.1
Outcomes of the imbalanced datasets with the one-shot protocol
We collect imbalanced datasets our benchmark (See Table 17 for details). evaluate the results under imbalanced data, we report the weighted F1 scord of datasets. We especially check the datasets with an imbalance ratio r 4 with the one-shot protocol, where at least one label is used for each class. This scenario is practical, that the new project initialized with only few data for cach class with unknown label distribution of the test set.
Table 10 demonstrates that no query strategy is significantly superior on most datasets. In particular, all query strategics cannot have significant improvement over Uniform in Appendicitis, Myocardial, Abalone, and Diabetes 130. We argue that existing aclive learning algorithmns lack consideration of imbalanced data and expect practitioners to investigate Uncertainty Sampling on imbalanced tabular datasets in future work
5.3.2 Outcomes of different hyper-parameter for the query-oriented model
Previous sections illustrate that Uncertainty Sampling with compatible XGBoost task-oriented and query- oriented models is strong baseline for our benchmark. In this section, verify whether the change in the model complexity of the query-oriented model would influence the Uncertainty Sampling. The model
The weighted F1 usually used for imbalanced datascts handle uncqual class distribution issucs.
Recent work investigates active learning for imbalanced datasets the CV domain and suggests incorporating balancing step into the labeling process to mitigate imbalance within the labeled pool (Aggarwal al. 2020
18

---

Published in Transactions on Machine Learning Research (06/2025)
Table 10: 'Mean + standard deviation' weighted F1 score XGBoost for imbalanced classifications with the one-shot protocol at 20% total budget (f is better). The scores with bold denote the best performance on a dataset. The scores with underline denote its mean AUBC is greater than Uniform's 'mean standard deviation' AUBC.
Data
Uniform
US-SM
Core-Set
BALD
MCM
Appendicitis
80.07%44.4450 439%
80.15%418.004%
44.20%117.7344
80.87%
80.38%.88.599%
Tic
88.99%+1.300
91.8%%+1.147%
88. 88.18%+1.660 1.99%
01.02%.11.170%%
90.13%+1.480%
Myocardial
95.14%++0.423%
95.50%++..144%
95.21%+0..189
95.58%
9..66%+00096%
Abalone
10.98%%11.0011 %
20.97%:11.09%
19.57%:11.0179
20.99%.11.293%%
20.69%11..07%%
Dry Bean
90.83%+
92.27% 0.220%
90.70%+0.6622%
92.29%++0.20%%
91.33%++0.50%%
Diabetes 130
49.85%+0..8
50.52%%10.872%
49.41%:10.199%
TLE
49 49.31%+11.339%%
RT-1oT2022
98.52%
0..028%
83.80%1110.681%
99.72%+10053%%
98 98.93%
complexity of the XGBoost model controlled by the hyper-parameters, c.g., we could reduce the propor- tional number of leaves in the trees by adjusting www.cchllwweigge in XGBoost. Therefore, we further launched thought experiments that compare the default and best hyper-parameetrss of the query-oriented model on small datasets over the same structure with different syper-parameters. Concretely, follow the apper-parameters tuning process in previous work (Grinsztajn et al. 2022 to get the best of the XGBoost models for each dataset. Then, we create the XGBoost models with these hyper-parameters and repeat the same active learning process described in Section 3
Figure 11 demonstrates that the query-oriented model with the best prper-parameters obtains slightly better mean AUBC than the default typer-parameters on most datasets. We conjecture that XGBoost, which has high model complexity, performs stably
benchmark; hence, the tuning hyper-parameters do not affect these results too much. In summary, the compatible models with the default hyper-parameters of XG would achieve good performance for most tabular datasets.
1.00
Deleck
Bist
0.55
0.40
0.0-
2.75
0.2
Ersa
Figure 11: Mean and standard deviation AUBC of US with the default and the best hyper- parameessssssers for XGBoost the query-oriented model for each dataset. Note. We the best hyper-parameters for XGBoost as the task-oriented model.
5.3.3 Outcomes of different query batch sizes
Early studies design the active learning algorithms based on a serial query, i.e., query an example at each round the early stage. However, a scrial query would be incfficient when the size of the query-oricnted model increases or the training process bocomes slower. Increasing the query batch size B each round becomes the choice to overcome these issues. Nevertheless, directly selecting the most B-highest uncertain
19

---

Published in Transactions on Machine Learning Research (06/2025)
Table 11: Accuracy (f is better) of the model with 3000 labeled examples for the batch size B = 1, = 10, and 100 on large-scale datasets.
Data
1 (Original)
10
100
Covertype
79.34%
79.29%
79.08%
Pol 98.29%
98.40%
98.39%
Phishing
96.80%
96.75%
96.74%
Dry Bean
92.63%
92.60%
92.60%
Diabetes 130
56.48%
56.37%
56.35%
examples for Uncertainty Sampling might query redundant examples, resulting in inefficiency. In this section, we study the impact of increasing the batch size US without designing complicated techniques.
Table I1 shows the degrading of Uncertainty Sampling when increasing batch size from 1 to 10, and 100, indicating the existing ineffective examples queried by batch-mode Uncertainty Sampling. While previous works made an effort to design hybrid criteria such as BMDR and SPAL to improve batch-mode Uncer- tainty Sampling, most methods suffer from the computational cost (in Appendix E and only bring small benefits compared to Uniform (in Table 7). For future work, we suggest that the researcher consider the trade-off between the improvement over Uniform and the computational cost of their design.
5.4 Limitations of uncertainty sampling
Previous benchmarks show that query strategies may not. outperform Uniform in specific settings Karamcheti et al. 2021 Munjal et al. 2022). tasks (Yang& & Loog 2018 Desreumaux & Lemaire 2020
Our findings demonstrated in Table 7 also indicate that uncertainty sampling does not excel on datasets like Checkerboard and Banana. Several works study possible reasons for the failure of Uncertainty Sam-
pling (Mussmann & Liang 2018 Karamcheti et al. 2021 Tifrea et al. 2022 to realize the applicability of
active learning algorithms. underscores the need to explore robust baselines for pool-based active learning, particularly in real-world scenarios (Lu et al., 2023
In this work, we give the preliminary results of exploration Uncertainty Sampling for tabular datasets, cov- ering binary classification and multi-class classification. Although Uncertainly Sampling provides promising for some of these scenarios, the usefulness and effectiveness of Uncertainty Sampling still have a gap and are unclear in domain-specific classification and imbalance learning (Johnson & Khoshgoftaar 2019). We believe there are still potential research directions, and our protocol could help rescarchors explore more scenarios in future works.
6 Conclusion
This work presents the most comprehensive survey and open-source benchmark for active learning to date. Our benchmark, with its transparent and unified interface, incorporates existing GitHub repositories, pro- viding a thorough and up-to-date comparison of active learning query strategics. We equip Uncertainty Sampling with compatible models and affirm that remains superior to other active learning strategies well as Uniform Sampling most of the datasets. Furthermore, we discover that Uncertainty Sampling be affected by the incompatibility between query-oriented and task-oriented models, resulting in dis- crepancies between previous benchmarks. Our affirmation suggests Uncertainty Sampling with compatible query-oriented and lask-oriented models as a first-hand choice for practitioners. These insights not only enhance the community's comprehension of current active learning strategies but also establish a foundation for future rescarch with this practical guide. We anticipate extending our framework to encompass diverse domains like vision and languages and incorporating various models such as deep neural networks, as outlined in Appendix F for future exploration.
20

---

Published in Transactions on Machine Learning Research (06/2025)
Broader Impact Statement
Active learning is a long-lerm research topic in machine learning, yet achieving consensus on the best strategies within the community is challenging. This work starts from the tabular data to build the most comprchensive open-source active learning benchmark to date. We affirm that Uncertainty Sampling (US) remains superior to other active learning strategies and Uniform on most datasets. We also clarify confficting conclusions in previous benchmarks by carefully verifying previous settings. Our work will benefit the active learning community by providing a transparent and unified framework for evaluating active learning strategies compared to a strong bascline-US. US with compatible settings. We hope work will help practitioners check the reality of existing active learning strategies and settings for different domains. Moreover, re-examine the potential issues in existing benchmarks, such as neglected settings and unpublished analysis steps.
Studying active learning beyond pursuing high accuracy is also important. For example, some AL works studied ML fairness (Anahideh et al. 2022 and privacy issues (Feyisetan et al. 2019) for ethical consid- crations, which are essential for using new MLI/AL techniques for other fields. To study AL for different real-world scenarios, we trust our benchmark and experimental protocol is the solid foundation for future works.
Acknowledgments
We thank the anonymous reviewers, the responsible action editor and members of Computational Learning Lab for their constructive feedback and positive interactions. This work is supported by the National Science and Technology Council in Taiwan via NSTC 113-2628-E-0022008 NSTC 112-6288-F-002-033 We thank to National Center for High-performance Computing (NCHC) of National Applied Research Laboratories (NARLabs) in Taiwan for providing computational and storage resources.
and NSTC
References
Umang Aggarwal, Adrian Popescu, and Céline Hudelot. Active learning for imbalanced datasets. In Pro- ceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 1437-1437, 2020.
Jesus Aleala-Fdez, Alberto Fernández, Julián Luengo, J. Derrac, S Garc'ia, Luciano Sanchez, and Francisco Herrera. Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework. Journal of Mulliple- Valued Logic and Soft Compuling, 17:255 287, 01 2010.
Hadis Anahidch, Abolfazl Asudeh, and Saravanan Thirumurnganathan. Fair active learning. Expert Systems with Ayplicalions, 199:116981, 2022.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alckh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprinl arXiv:1906 03671, 2019.
Dara Bahri, Heinrich Jiang, Tal Schuster, and Afshin Rostamizadch. Is margin all you need? an extensive empirical study of active learning on tabular data. arXiv preprint arXiv:2210.0355 03.0822,, 2022.
Nathan Bock, Durga Sivasubramanian, Apurva Dani, Ganesh Ramakrishnan, and Rishabh Iyer. Effective evaluation of deep active learning on image classification tasks. arXiv preprinl arXic:2106. 15324,, 2021.
William Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE conference on computer vision and pollern recognition, pp. 9368 00779977 2018.
Etienne Brangbour, Pierrick Bruncau, Thomas Tamisier, and Stéphane Marchand-Mailer Active learning with crowdsourcing for the cold start of imbalanced classifiers. In CDVE, pp. 192-20 201, 2020. URL http://000000///0100000/0//-/--/
Leo Breiman. Random forests. Machine learning, 45:5 32, 2001.
Clemens-Alexander Brust, Christoph Käding, and Joachim Denzler. Active learning for deep object detection. arXiv preprint Xic: 09870,98755.. 2018.
21

---

Published in Transactions on Machine Learning Research (06/2025)
Wenbin Cai, Muhan Zhang, and Ya Zhang. Batch mode active learning for regression with expected model change. TEEE transactions on newral nebworks and learning systems, 28(7):1668-1688) 2016.
Gavin C Cawley. Baseline methods for active learning. In Active Learning and Paperimental Design workshop In conjunction with AISTATS 2010, pp. 47 57. JMLR Workshop and Conference Proceedings, 2011.
Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1 27:27, 2011. Software available al. http://www. csie. ntu.
edu. htt.bb///cjlii/llbee
Rita Chattopadhyay, Zheng Wang, Wei Fan, Tan Davidson, Sethuraman Panchanathan, and Jieping Ye. Batch mode active sampling based on marginal probability distribution matching. KDD proceedings. International Conference on Knowledge Discovery & Data Mining, 2012:741-749, 2012.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pp. 785- 794, New York, NY, USA, 2016. ACM. ISBN 978-1-4503-4232-2. doi: 02.144//233662.29397778 URL
http://doi... acm. www..acc........ wwa.moorg/1...0 99939999111199299999299999383331
2939785
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Matt Culver, Deng Kun, and Stephen Scott. Active learning to maximize area under the roc curve. In Siath International Conference on Data Mining (ICDM'06) pp. 149-158, 2006. doi: 10. 2006.....
Tivadar Danka and Peter Horvath. modAL: A modular active learning framework for Python. URL https:
//github.com/moet hwtho//doooootooooooooooaaa available arXiv at http::://arxti..rr/bbbs.ss /tt.//////b////000999/9999
Sanjoy Dasgupta and Daniel Hsu. Hicrarchical sampling for active learning. In Proceedings of the 25th international conference on Machine learning, pp. 208 215, 2008.
Emre Demir, Zehra Cataltepe, Umit Ekmekci, Mateusz Budník, and Laurent Besacier. Unsupervised Active Learning For Video Annotation. In ICML Active Learning Workshop 2015, Lille, France, July 2015. URL
https://hal.. saaaaaee//ceee///000550005
Louis Desreumaux and Vincent Lemaire. Learning active learning at the crossroads? evaluation and discus- sion. arXiv e-prints, art arXiv:2012.09631, December 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and 2019.
short popers), pp.
Pinar Donmez, Jaime G. Carbonell, and Paul N. Bennett. Dual strategy active learning. In Joost N. Kok, Jacck Koronacki, Raomon Lopez de Mantaras, Stan Matwin, Dunja Mladenič, and Andrzej Skowron (cds.), Machine Learning: ECML 2007, pp. 116 127, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg. ISBN 978-3-540-74958-5.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. URL httppp////oggrrrree..ee/e/rrrrre
Dheeru Dua and Casey Graff, UCI machine learning repository, 2017. URL http://archive. wcw.uci............. edu/
m1
Sandra Ebert, Mario Fritz, and Bernt Schiele. Ralf: reinforced active learning formulation for object class recognition. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 8626-3633 2012. doi: 10. (CC.C.202..2....
22

---

Published in Transactions on Machine Learning Research (06/2025)
Oluwascyi Feyisetan, Thomas Drake, Borja Balle, and Tom Diethe. Privacy-pressevving active learning
sensitive data for user intent classification. 2019. URL. https://www wttp:///wwwwwaaa..ooo.m amazon www.aoww..ceenneeeuulaaaaaioio.i
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference on machine learning, pp. 1183-1199 1192. PMLR, 2017.
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-basee models still outperform deep learning on typical tabular data? Advances in Neuval Information Processing Systems, 35:507-520, 2022.
Sanket Rajan Gupte, Josiah Aklilu, Jeffrey Nirschl, and Serena Yeung-Levy, Revisiting active learning in the era of vision foundation models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
URL https:://ppnnzeiier..eeet http:pppwwwaaaaaoooooooooooooaaa
Isabelle Guyon, Gavin Cawley Gideon Dror, and Vincent Lemaire. Design and analysis of the weei 2010 active learning challenge. In The 2010 International Joint Conference on Neural Networks (IJCNN), pp.
1 8, 2010. doi: 10.1109/1JCNP CTT.://NNN........... 2010.596566.
Isabelle Guyon, Gavin C. Cawley, Gideon Dror, and Vincent. Lemaire. Results of the active learning challenge. In Isabelle Guyon, Gavin Cawley Gideon Dror, Vincent Lemaire, and Alexander Statnikov (ods.), Active Learning and Experimental Design workshop In conjunction with AISTATS 2010, volume 16 of Proceedings of Machine Learning Research, pp. 19 45, Sardinia, Italy, 16 May 2011. PMLR. URL https::/ppoceedding laraaggsmmr..e. www..r.reeeevvvvv htress//11/gguoonaa..ttt guwgo/la..anmmmm
Zhongkai Hao, Chengqiang Lu, Zhenya Huang, Hao Wang, Zheyuan Hu, Qi Liu, Enhong Chen, and Cheekong Lee. Asgn: An active scmi-supervised graph neural network for molecular property prediction. In Pro- ceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,
pp. 731-752, 2020.
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Mátó Lengyel. Bayesian active learning for classifi- cation and preference learning. arXiv preprint 1rx:x:1112.571 2011.
Wei-Ning Hsu and Hsuan- -ien Lin. Active learning by learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 2659 2665, Jamuary 2015.
Sheng-jun Huang, Rong Jin, and Zhi-Hua Zhou. Active learning by querying informative and representative examples. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 23. Curran Associates, Tnc., 2010. URL https:///roceedd:/
Paperr pdff
Yilin Ji, Daniel Kaestner, Oliver Wirth, and Christian Wressnegger. Randomness is the root of all evil: More reliable evaluation of deep active learning. In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3932-3941, 2023. doi: 10.1 10.1109/ /t///WCVV66882220 2et.22223000030
Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of big data, 6(1):1 54, 2019.
Ashish Kapoor, Eric Horvitz, and Sumit Basu. Selective supervision: guiding supervised learning with decision-theorett active learning. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAF'7, pp. 877-882, San Francisco, CA, USA, 2007. Morgan Kaufmann Publishers Inc.
Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher Manning. Mind your outliers! in- vestigating the negative impact of outliers on active learning for visual question answering. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Inter- national Joint Conference on Natural Language Processing (Volume Long Papers), pp. 7265-7281, Online, August 2021. Association for Computational Linguistics. doi: 10. (0.1885//v1 18653/v1/2021111.1 http//22a.aalloong.... acl-long..164 URL http:p::://ccclaooooooooooo////p wat.002..cclllooo acl-long 564
23

---

Published in Transactions on Machine Learning Research (06/2025)
Jeeveswaran Kishaan, Mohandass Muthuraja, Deebul Nair, and Paul-Gerhard Plöger. Using active learning for assisted short answer grading. 2020.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:// Paper.
proceedings
Daniel Kottke, Adrian Calma, Denis Huseljic, GM Krempl, Bernhard Sick, et al. Challenges of reliable, realistic and comparable active learning evaluation. In Proceedings of the Workshop and Tutorial Interactive Adaptive Learning, pp. 2-14, 2017.
Daniel Kottke, Marek Herde, Tuan Pham Minh, Alexander Benz, Pascal Mergard, Atal Roghman, Christoph Sandrock, and Bernhard Sick. scikit-activeml: A Library and Toolbox for Active Learning Algorithms. Preprints, 2021. doi: 10. 222044/peepinn22220000001... URL httpp:://gttuub.cco/ssskkkk/// scikit-activen
Jan Kremer, Kim Steenstrup Pedersen, and Christian Igel. Active learning with support vector machines. Wiley Tnl. Rev. Data Min. and Knowl. Disc., 4(4):313 326, jul 2014. ISSN 1942-4787. doi: 10.1002/widhe /wddm. 1132. URL https:///ooiii org/10../1. htt.g//..100////111 1et.1000/wwiiii 1132
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Chun-Liang Li, Chun-Sung Ferng, and Hsuan-Tien Lin. Active learning using hint information. Newral computation, 27(8))1738-1700 2015.
Xin Li and Yuhong Guo. Adaptive active learning for image classification. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 859 866, 2013.
Qiang Liu, Yanqiao Zhu, Zhaocheng Liu, Yufeng Zhang, and Shu Wu. Deep active learning for text classifica- tion with diverse interpretations. In Proceedings of the 30th ACM International Conference on Information & Knowledge Monagement, pp. 3263 3267, 2021.
Yash-yee Logan, Mohit Prabhushankar, and Ghassan AlRegib. Decal: Deployable clinical active learning. arXiv preprint Xrx:::2206. 10120, 2022.
Po-Yi Lu, Chun-Liang Li, and Hsuan-Tier Lin. more robust baseline for active learning by injecting randomness to uncertainty sampling. In Proceedings of the Al and HCI Workshop @ ICML, July 2023.
Carsten T Lüth, Till J Bungert, Lukas Klein, and Paul F Jaeger. Toward realistic evaluation of deep active learning algorithms in image classification. arXiv preprint arXiv:2301.1062, 2023.
Andrew L.. Maas, Raymond F. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pp. 142 150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL attp:://wwt wwwwwb.cl.bbb. org/ htt///atoooog//111//0111
Prateck Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, and Shadab Khan. Towards robust and reproducible active learning using neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 223-232, 2022.
Stephen Mussmann and Percy Liang. On the relationship between data efficiency and error for uncertainty sampling. In International Conference on Machine Learning, pp. 3674-3682. PMLR, 2018.
S Deepak Narayanan, Apoorv Agnihotri, and Nipun Batra. Active learning for air quality station deployment. 2020.
V. Nath, Dong Yang, Bennett A. Landman, Daguang Xu, and Holger R. Roth. Diminishing uncertainty within the training pool: Active learning for medical image segmentation. IEEE Transactions on Medical Imaging, 10:2534-2544 2020.
24

---

Published in Transactions on Machine Learning Research (06/2025)
Hieu T Nguyen and Arnold Smeulders. Active learning using pre-chsterrngg In Proceedings of the twenty-first international conference on Machine learning, pp- 79, 2004.
Lukas Rauch, Malthias Aßenmacher, Denis Huseljie, Moritz Wirth, Bernd Bischl, and Bernhard Sick. Ac- tiveglae: A benchmark for deep active learning with transformers. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 55-74. Springer, 2023.
Nicholas Roy and Andrew McCallum. Toward optimal active learning through sampling estimation of error reduction. In Proceedings of the Lighteenth International Conference Machine Learning, ICML '01, pp. 441 448, San Francisco, CA, USA, 2001. Morgan Kaulmann Publishers Inc. ISBN 1558607781.
Andrew I. Schein and Lyle H. Ungar. Active learning for logistic regression: evaluation. Machine
Learning, 68(3):233- 2007. doi: 1010007ss10944-0--0000-- URL https:///od.orr/10..000 610954-007-5019-53
Christopher Schröder, Andreas Niekler, and Martin Potthast. Revisiting uncertainty-bbssed query strategies for active learning with transformers. In Findings, 2021. URL https://api..emmaaiiccccoho org/
Cor3psSDD223828823
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A corc-set approach. In International Conference on Learning Representations, 2018. URL http:://opearreriieeeef//rooorrr
H1aluk-RW
Burr Settles. Active Learning, volume 6. Springer, 2012. doi: 102200/00-220DD1000000000A000 URL
6e3g68kaf=R
H Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Proceedings of the fifth annual workshop Computational learning theory, pp. 287 7944 1992.
Changjian Shui, Fan Zhou, Christian Gagné, and Boyu Wang. Deep active learning: Unified and principled method for query and training. In International Conference Artificial Intelligence and Statistics, pp. 1308-1318. PMLR, 2020.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 5972 5981, 2019.
Ying-Peng Tang and Sheng-Jun Huang. Self-paced active learning: Query the right thing at the right time. Proceedings of the AAAI Conference on Artificial Intelligence, 33(00):5117-5514, Jul. 2019. doi: 10.1609/anai, (116009/aai.v330001. 00.160//aaaiv3331133311111111 v33101.330111111 URL. https:///sss..... aaai. www.saa...a.................. http:///wwa..aaaa////////aaaaaaa
Ying-Peng Tang, Guo-Xiang Li, and Sheng-Jun Huang. ALiPy: Active learning in python. Technical report. Nanjing University of Aeronautics and Astronautics, 1 2019. URL https://github. wOw/NUAA-ALNAAAAA AL/ALiPy
available arXiv preprint https://arxiv wwwpsiiiv.rrgaaeo 03802
Alexandru Tifrea, Jacob Clarysse, and Fanny Yang. Uniform versus uncertainty sampling: When being active less efficient than staying passive. arXiv preprint orXiv:2212.0077,, 2022.
Holger Trittenbach, Adrian Englhardt, and Klemens Böhm. An overview and a benchmark of active learning for outlier detection with one-class classifiers. Expert Systems with Applications, 168:114372. 2021. ISSN 0957-4174. doi: https: wwtp:////..................a.. scteaccc//rrttcaa/////////777772
URL https://ww.sccieeddrees. com/
Zheng Wang and Jieping Ye. Querying discriminative and representative samples for batch mode active learning. ACM Trans. Knowt. Discov. Data, 9(3), feb 2015. ISSN 1556-4681, doi: 10.1145/2700408.. URL
https://aoi.org//1 111111101111//0/000
Thorben Werner, Johannes Burchert, Maximilian Stubbemann, and Lars Thimee. cross-domain benchmark for active learning. In The Thirty-eight Conference on Neural Information Processing Systems Dalasets and Benchmarks Track, 2024. URL httppp///pppprreeeee.aaee/r////p
25

---

Published in Transactions on Machine Learning Research (06/2025)
Dongrui Wu, Chin-Teng Lin, and Jian Huang. Active learning for regression using greedy sampling. Infor- mation Sciences, 474:90- 100,, 2019.
Zhao Xu, Kai Yu, Volker Tresp, Xiaowei Xu, and Jizhi Wang. Representative sampling for text classification using support vector machines. In Fabrizio Sebastiani (ed.), Advances in Information Retrieval, pp. 393- 407, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg. ISBN 978-3-540-36618-8
Yao- Yuan Yang, Shao-Chuar Lee, Yu-An Chung, Tung-En Wu, Si-An Chen, and Hsuan-Tiee Lin. libact: Pool-based active learning in python. Technical report, National Taiwan University, 10 2017. URL https: available arXiv preprint
//ggthubb.com//tuulllaaa////////
Yazhou Yang and Marco Loog. A benchmark and comparison of active learning for logistic regression. Pallern Recognition, 83:401 13:44,1115,, 2018. ISSN 0031-3203. doi: https::/dobi.org/10..000 http:///d..oogpll....oooppo..o.p patcoo.2018.06.000 URL
Yi Yang, Zhigang Ma, Feiping Nie, Xiaojun Chang, and Alexander G Hauptmann. Multi-class active learning by uncertainty sampling with diversity maximization. International Journal of Computer Vision, 113:113 127, 2015.
rostamiz Yilei "Dolee" Yang. Active learning playground. http:////iiiiuu..oooooooo//////s com 2017. URL /ectps://githan com cww/ httm://glll/aaveeeeeniiiii
Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 93-102, 2019.
Michelle Yuan, Hsuan- Tisunn-Tien Lin, and Jordan
Cold-start active learning through self-supervised language modeling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), November 2020.
Tianning Yuan, Fang Wan, Mengying Fu, Jianzhuang Liu, Songcen Xu, Xiangyang Ji, and Qixiang Ye. Multiple instance active learning for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognátion, pp. 5330-5339, 2021.
Xueying Zhan, Huan Liu, Qing Li, and Antoni B. Chan. A comparative survey: Benchmarking for pool-based active learning. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 4679-4686. International Joint Confercnces on Artificial Intelligence URL https://doi.
Organization, 8 2021. doi: /tt/://///jjai22222 Survey Track.
24933/jjaaa/iaiaai http://aaaaa..1//./..// 2021/634
Xueying Zhan, Qingzhong Wang, Kuan-hao Huang, Haoyi Xiong, Dejing Dou, and Antoni B Chan. A comparative survey of deep active learning. orXiv preprint arXiv:2209. 19450, 2022.
Hongjing Zhang, SS Ravi, and Ian Davidson. A graph-based approach for active learning in regression. In Proceedings of the 2020 SIAM International Conference Data Mining, pp. 280 288. SIAM, 2020.
Jifan Zhang, Yifang Chen, Gregory Canal, Stephen Mussmann, Arnav M Das, Gantavya Bhatt, Yinglun Zhu, Jellrey Bilmes, Simon Shaolei Du, Kevin Jamieson, al. Labelbench: A comprehensive framework for benchmarking adaptive label-efficient learning. arXiv preprint or Xim:2306.09910 2023.
Zhisong Zhang, Emma Strubell, and Eduard Hovy. A survey of active learning for natural language process- ing. Xixi preprint arXiv:210. 10109, 2022.
26

---

Published in Transactions on Machine Learning Research (06/2025)
Table 12: Accuracy of the model with 10% labeled examples: We report the accuracy of the model with 10% labeled examples on each dataset. The bold indicates the first place. 'TLE' means query strategy exceeds the time limit.
Uniform
US
QBC
BALD
Hier
Graph:
Core-Set
HinSVMM
QUIRE
DWCS
MCM
BMDR
ALBL
LAL
Ceoul
Jounsphere
62-6055
74.99%
62.65%
75.43%
74.94%
02.52%
74.91%
62.39%
74.57%
63.36%
61.07%
74.86%
75.82%
62.52%
61.93%
74.91%
62. 6056
75.30%
74.31%
02.60%
61. 61.11%
74.53%%
10 G1.71%
75.00%
62.82%
75.25%
02.85%
75.96%
Wilbe
Breast
90.5456
98 33.33%
90.28%
94.95%
93. 1556
90.585%
01 3356
94. .956
00.099%
94.26%
90.32%
87.78%
91. 01.3355
98.6156
93, 7437
90.499%
90.965%
565
90.077
90-49%
43. 13.1680
40.29%%
94 94.799
00
92.322
95.88%
Asstralioa
81.00%
83.13%
81.177
62.9656
81.17%8
80.44%
76.49%
78.117
77.38%
79.7556
80,9555
00, 200
80, 10.
92.00%
Diahetes
70:2256
72.07%
70.52%%
71 .00%
70.865
69.58%
00. 00.2556
09.3456
68 88%
09.3076
70 70.00%
70. 10.299
70.-33%
71 11.08%
ExSaa
Mammographic
86.5456
79.5355
70.92%
88.
79. 62%
70.7
19.3876
66.63%
87. 03
78. 600%
79.
86,00%
78.32%
83.99%
53.
85.39%
79.1
76.65%
76.52%
79.077
46%
78-2226
71.75%
10%
78. 369
$1-0255
82. 22.90%
TLE
79 79.19%%
87.5%
80,3830
82.25%
91.50%
57.0%
German
69:2955
70.7%%
69.4656
70.3755
1%
69..60%
68% 14636
69. 1836
60 28%
68-4796
T1.
70.45%
Splice
83.1676
84.98%
82.0076
76.067
78.42%
63.033
$1 $1,093
90.61%
$1.79%
92.71
Gelourth
86.3656
81.11%
86.30%%
83.90%
87.1495
84.2156
64.6
84. 1776
47.00%
87.57%
K9. 15%
Gelondub
88.1976
88.91%
88.41%%
92.97%
68%
79.8886
97.70%
86.01%
90.77%
Carderboard
97.9255
95.81%%
97.089%
97.677%
97,9318
96.52%
96.78%
18%1
57.089 77.09%
75.27%
98, 0008
98.23%
94.78%
99. 80%
Spamisse
90.775G
98.885%
89, 99.99%
983 002%
88. 96.996%
85.98%
TLE
90.85%
93.11%
TLE
91. 91.00%
93. 11.01%%%
Hanona
85.7636
433.200%
96.17%
852 7756
886 5889
83.44%
87.511%
69. 7676
6-4%
70.87%
84.51%
T1.::
85.211%
43%
Phoneree
81 6866
85.15%
80.97%
811.77%
62, 30%%
80.62%
81.45%
76.9856
TLE
76.87%
83, 62%
TLE
81.47%
83. 53.61%
Kingnorn
89. 5536
003.89%
99.922%
94.0/2%
899%
57.35%
64.1776
05.700%
TLE
67.73%
9220395
T1.
886.11%
89.
Twonorm
03.9775
95..92%
03.556%
915.96%%
03.311%
03.733%
994 4555
80.799%
TLE
02.317%
95.7099
TLE
04.50%%
94.42%
Phishing
12.1755
94.62%
02.065
94.844%
91.70%
91.18%
22%6
90.97%
TLE
87.24%
8018
93,2015
94,28%
Covettype
70.9895
73.70%
70.22%
733 45%
TLE
66.
TLE
61.73%
TLE
58. 76%
67 5109
TLE
66. 66.92%
71.90%
Blorepoone
66-6456
68 396
66.
68.99%%
65.45%
63-522 53.522%
6D.6456
06.64%
67.77%
66630%
67 87.88%
Pol
93.2455
96.67%
93.27%
96.84%
92.96%
80.26%
75.70%
75.77%
TLE
93.21%
95.21%
TLE
93.07%
96.15%
A Detailed benchmarking results of XGBoost
We present more scttings of the benchmarking results for XGBoost for verifying the superiority of query strategies in Section 4.1 We check the accuracy of the model with different ratios, e.g., 10% and 30% of labeled examples on each dataset. Tables 12 and also confirm that US outperforms other query strategies on most datasets. It is worth mentioning that LAL achieves good performance on Geloudb, Geloudub, and Checkerboard when the ratio of labeled examples is 10%. However, these datasets are synthetic, and their features may be more similar to the pre-trained datasets used by LAL, resulting in LAL's exceptional performance on these datasets.
We also extend the Table 5 to the number of queried examples required to reach 90% and 95% of the model's performance trained on the full budget. Table I4 demonstrates that Uncertainty Sampling stably achicves first place or second place on 14 datascts. Similarly, Table 15 demonstrates that Uncortainty Sampling achieves first place or second place on 15 datasets. These results verify that Uncertainty Sampling is competitive in our benchmark.
B Revision of Zhan et al. (2021)
In this section, we reveal and revise descriptions in Zhan et al. 22021) to study the conflicting conclusions in previous benchmarks and provide clear information to the active learning community. We appreciate that Zhan et al. (2021) published their source code on GitHub @ Thus we could examine the difference from our scttings.
B.1 Experimental Settings
Inputs and base models. At the initial setup, Zhan et al. (2021) employed a random split of 60% of the dataset for the training set and the remaining 40% for the testing set. No pre-processing was applied to the dataset, and fixed random seeds were used to ensure consistency in the training and testing sets across repeated experiments. They used an RBFSVM as the lask-oriented model for evaluating the query strategies.
27

---

Published in Transactions on Machine Learning Research (06/2025)
Table 13: Accuracy of the model with 30% labeled examples: We report the accuracy of the model with 30% labeled examples on each dataset. The bold indicates the first place. 'TLE' means query strategy exceeds the time limit.
Unilorm
US
QBC
BALD
Hier
Graph
Core-Sel
HintSM
QUIRE
DWUS
MCM
BMDR
ALBL
LAL
Sonar
71.18%
71.71%
70.75%
71.90%
70.80%
71 12022
67.89%
69.06%
70.20%
71,
71.42%
70.10%
T2-37%
71.78%
71.99%
Parkfnsoms
81.44%
84.95%
81.88%6
84.35%
81.87%
81 .15956
81.58%
80.507%
81.74%
91.74%
81.7450
51.74%
83.74
53.74%
82.67%
83.21%
94.08%%
Heart
Exeha
Haberman
68.87%
82.70%
77.78%
71.21%
79.78%
84.21%
77.90%
08. 18.41%%
#2.955%
79. 79.50%
53.145
70.92%
2776
77.85%
69. 58%
60.11%
66 900%
82.54%
77.845
68. 53.5.%
78.07%
84.42%
67. 49%
76.76%
81.99%
17.999
67. 85500
78.999
79.25%
7L
Cleund
lonosphere
72.59%
85.97%
74.79%%
90.30%
86. 76.717
72.527%
74. 26.66
72. 12.364
87.21%
87.4156
772 72.40%
-
11.8776
-
844072
7998222
71%
72.54
722.442
72.009
78.50%%
74.50%%
Breast
95.05%
96. 699%
95.7775
96, 265
96, 96.5276
66.0656
86.7
96
95
Wille
Alsttaciion
84.00%
93.735%
85, 95%
25%
93.77%
16. 1
95. 959
84 56%
9- 11.28%
85 0006
98 8055
91.07%
84.
$4.50%
92 $2.964
82 8028
03
96
96
85.9455
95.74
95.
95. 99%
77%
Mammographiic
Dishetes
79.535%
72.60%
62. 0376
74..7%%
70.22
72.84%
73.84%%
70. 0220
733 72.72%
7% 28.877
73.10%
79. 10%%
70-3476
77 860
71.055
71
72.54%
72.64%
81 81.89%
722 72.75%
Tie
ExSa
90.122%
91.08%%
92. 63%
01%
00. 12.1
91.300
92, 92,7566
91 90. 11%111 0
11 11.200
91.1115
94.96%
59. $0.58%
89 37%
77. 67.6889
90. 1
80, 10.0
222 88
22
92
92.7550
$8.0699
93.89%
Spline
German
91.53%%
72.16%%
73..3%%
95%47%
91.53% 1
72.500%
15. 15.1
73. 23.5
72.330 0336
91 3677
72.71
90 90,499% 1
72-1156
80. 80.333
72 72-4076
T1.
91.70%
87.215%
70.830
T1.
91.807
91.
72
91.52%%
Gelouab
88.09%
59. 0675
66 66.25 235
86
56. 06.500
58
Caockerbard
Gelondub
98.299%
92.67%
07.22%
38%
10.10%
92.52%
190 7006
99.6119 9
015 33.3374
10.1
7055
90 90.00%
8-.03% -
87.96%
11
81, 1
16 91.81%
91.76%
1
Spambase
13.10%%
15.14%%
90.22%
96. 11.6
93..4%%
53.0075
92. 00.900
90.0037
TLE
95.16%
94.88%
T1E6
97.49%
95, 95.04%
Banana
88.02%%
69.08%
87.83%
88.988
87 77.866%
87.90 $7.9655
14..
79.78%%
76.635
TLE TLE
87.43%
88.985
Phoneme
85.14%
88.47%
85.14%
88. 2015
96. 46.02%
84.0035
85.505%
83.4556
TLE
80.04%
15.1
TLE
55.15%
88.000%
Ringnora
94.22%
96. 155%
94.59%%
96.32%
90. 60.38%
94.01%
51.04%
64.64%
TLE
15.099%
94.06%
94.94%
Twommr
95.07%
96.76%
95.67%
96.77%
95, 76%
95.7356
96.03%
77.35%
TLE
95.08%
80.75%
TIE
90.02%
96.57%
Phishing
04.009%
86.56%%
033667%
56.5596
USA 79%
983 19.69996
93 83%
91.9656
TLE
03.1
36.38%
T1.
94.81%
96.60%
Coveitype
73.544%
76.30%
73. 10%
20.2555
TLE
65. 8256
TLE
63.94%
TLE
61.21%
13-1
TLE
69.72%
74.64%
Diorespoase
72.11%%
74. 10%
72.51%
74..2%%
71.91%
73 2006
00. 89%
69. 11%
TLE
72.11
73.33%
TLE
71.70%
73.7
Pol
96.34%%
96.28%
96.22%
25%
96.70%
94.80%
91.99%
TLE
96.34%%
98.14%
95.98%
98.25%
Table 14: The minimum number of queried examples required to reach 90% accuracy (4 is better) of the model: The bold indicates the first place and italics indicates the second place. 'TLE' means a query strategy exceeds the time limit.
Unifora
US
QBC
BALD
Hier
Grapha
Core-Set
HiniS VM
QUIRE
DWUS
MCM
BMDR
ALBL
LAL
Appendicitis
20.88
20.62
21.16
20.71
21.32
20.85
20.98
21.09
21.82
20.60
21.04
20.87
20.82
Sunar
31.95
33.91
31.65
31.23
33.62
35.54
39.05
38. 70
33.17
35. 75
31.23
31.54
33.81
67
Parkinsens
25.29
26.81
25.0/2
27.32
32.16
27.29
26.
24.09
24.74
24.61
23.28
Exchh
25.01
23.57
24.60
23.24
24.67
23.34
23.02
27. 20
24.15
26.
23.36
21.03
23.13
22.07
Hourt
23.35
23.20
23.34
82. 76
23.25
23.93
22.76
24.40
22.98
24.64
23.74
23.85
22.69
23.13
Habernas
20.15
80.32
20.55
20.51
20,00 GD
20.62
22.00 09.09
86
21.21
20.54
20.26
20.53
20. 40
20.17
lonosplaere
35.05
30.38
32,74
30.02
32,77 79
38.32
32. 22.26
35, 12
39,80
93, 23
32:91
50.42
27.56
28. 05
Cleanl
97.02
85,98
85.01 04
102.05
101.63
966.0
97.00
96
84.58
Breast
21.01
20.88
20.95
22.25
20.89
21.60
20.95
21. 133
20.85
11.26
20.88
21.50
20.55 53
20.35
Wilbe
23.33
23.00
23.38
22.97
63
22.04
56
22.42
213 41
23.00
24.37
53
20.92
Austrralian
25.02
24.35
24.61
23.85
25.30
26.25
30.30
33.27
28.87
26. 19
27. .65
26.27
26.90
84. 32
Disbetes
22.42
23.02
222.50
22.30
22. D22
22.88
22.53
212
222.80
25.03
23.36
23.37
21.86
22.16
Matumographic
21.07
20,28
20.40
20.70
20.39 399
20.44
20.49
20, 29
20.33
20.99
20.83
20.39
20.53
20.38
ExcSa
74.56
76.86
76.89
75.54
79.67
71.65
55.77
248.91
214.78
154.66
52:22
63.65
99.55
58. 75
Tic
22.50
21.60
23.68
22.06
22.40
24.23
22.12
24.19
25.99
27.35
21.70
TLE
21.55
22.04
German
29.63
27.T2
30.15
88. 29
29.5- 5-
251.44
30.92
33. 47
352.72
35.41
28.32
TLF
30.26
29.29
Splice
71.31
57.28
197.18
107.19
70.71
80,91
69.37
Gelondb
22.37
22.99
22.41
22,18
22,18
21.08
20.92
21.53
22.42
21.02
22.84
21.15
21.19
21.28
Geloudub
38.13
38.29
36..9L
$5.03
36.18
41.46
35.00 .6
84.29
60.09
206 32
44.59
35.10
38.94
55.51
Checlorboard
39.15
95.46
11.08
11.61
37.96
11.37
33,60
136.14
93.89
420.74
45.56
30.83
53.35
28.22
Spanhuse
63.70
37.10
53.10
40.70
59.80
91.50
83.60
184.30
TLE
61.60
46.90
TLE
56.10
38. 10
FTLFaa
73.10
222.50
85,200
201.90
79.. 10
195,110
67.00
1040.10
617.60
777.70
173.90
TIF.
127.90
79.20
Phoneme
120.50
20
1933.7 70
101.70
99. 60
227. (0)
166.20
325.10
TLE
573.90
112.50
TLE
138.80
91.40
Ringnorm
165.90 60
101.20
162.77 70
101.70
209.60
178.60
2255.10 (I
1333.50
218130
172.20
TLE
138.50
Twonorm
$2.50
38.40
65.70
399.10
78.70
68.60
55.60
1359.60
TLE
$0.60
54L.1
TLE
50.70
39.00
Phishing
22.30
22.40
23.20
21.00
25.50
30.80
24.00
25.80
TLE
52.40
22.00
TLE
21.30
22,80
Covertype
164.00
120.80
130.10
146.00
TLE
1689.20
TLE
TLE
TLE
20,00
TLE
TLE
TLE
129.70
Biorepease
291.00
70
225. 10
1741.0
267.10
249. 10
150.10
TLE
TLE
291.00
222.55 50
TLE
TLE
213.20
Pol
86.00
62.80
79.60
69.20
105.80
649.20
286.70
531.60
TLE
86.00
96.40
TLE
76.30
58.60
Query strategics. To compare the performance of 17 query strategics, they implemented random sam- pling and all query strategies using different librarics. The libact library provided implementations for Uncertainty Sampling (US), Query by Committee (QBC), Hinted Support Vector Machine (HintSVM),
28

---

Published in Transactions on Machine Learning Research (06/2025)
Table 15: The minimum number of queried examples required to reach 99% accuracy (L is better) of the model: The bold indicates the first place and italics indicates the second place. 'TLE' means a query strategy exceeds the time limit.
Uniform
US
QBCC
BALD
Hier
Graph
Core-Set
HinsSVM
QUIRE
DWUS
MCM
BMDR
ALBL
LAL
Appendicitas
24.56
22.339
24.72
22,86
23.05
24.23
23.33
25.05
24.41
26.31
22.24
24.22
23.36
22.46
Sonar
51.31
48.78
51.26
47.09
51.27
53.33
55.92
55.22
51.48
53.32
50.29
49.14
49.01
46.11
Parleinsons
41.80
29.74
41.95
91.95
36.36
44.87
42.01
52.58
40.87
41.77
32. 14
36.16
34.49
31.69
Fxsh
34.15
30.30
34. 44.30
.70
32. 04
34.86
29. 73
39.04
33.11
412.1
30.27
312.50
30.51
28.12
Heart
-3
84
31.19
30.53
30,13
27.62
Haberman
22.17
21.35
21.32
21.27
21.70
21.59
22. 86
23, 73
22.91
21.95
21.06
21.49
21 53
21.36
Jonosphere
$7.59
27
51.75
38.94
54.52
56.78
53.82
58.17
61.56
145.
45.75
39.71
37.18
Cleanl
151.98
116.8T
149.53
117.15
152.53
150.88
1455.5
145,98
1/3.03
154.98
129.95
149.07
129 58.58
131.23
Breast
27,80
26.48
27.98
26.80
24.51
38.88
26.27
28,
24.37
110.20
26.50
29.27
23.51
21.66
Wilbe
33.43
33.02
35.83
29.50
35.15
38.93
30.10
36.22
37.32
33.26
37.70
26.67
25.62
Australion
36.07
89. 977
34.00 00
29.72
35.67
37.48
43.39
033 25
43.54
43.85
35.10
39.84
38. 38.07
30. 29
Diabetes
30.10
31.92
15
15.
33.58
29.
26.06
Mamnoggaapicc
22.50
21.44
22.24
21 .22
23. 13
23.50
21.87
222.0
21.11
25.04
23.82
21.67
21.16
21.01
ExSa
137.20
131/24
136,96
127. 19
133.07
131.67
87.53
303.87
292. 35
230.73 73
125 00
107. 80
213.37
98. 51
Tic
38.18
31.05
39.00
31.64
36,90 90
43.12
36.85
18.37
42.04
66.66
35.24
TLE
34.47
35.07
German
77.75
53.58
61.17
58.96 18.
73. 46
72.13
75.27
66.40
70.90
99.70
57.47
TLE
58.21
54.55
Splice
136.05
85.07
136.84
86.94
134.67
160 62
172.06
133.49
263.52
106.11
138.05
137.54 54
112.66
Geloudb
28.93
28.1 11
32.15
26-13
28. 58
39. 38
25.
32 32.25
39.50
37 :28
27.56
26.09
27 03
24.57
Gelondub
80.46
70.17
78.74
50-73
78
76.47
89:61
325.77
178. 15
364.91
93.60
88.56
99.21
59.37
Checkerbord
55.70
195.19
59.653
28
47. 50
68.96
54.86
276.80
239. 73
G84.71
56.07
$9.10
85.12
32.18
Spembase
223.00
81.20
256.90
78.70
150.20
205,
252. 20
620.00
TLE
220.00
114.10
TLE
167 80
96.90
BaTLEa
141.90
337.30
153.70
349.30
144.40
313.00
105,80
1474.50
S88. 20
1761.10
286.30
TLE
162.30
112.00
Phonene
509.70
226.40
527:30
233. 20
427.00
442.00
445.00
680.00
TLE
1455.90
303.60
TLF
556.80
256.00
Ringnorm
390.44
354.40
169.2
646.80
2337.70 .70
15003.00
TLE 2621 14
253
TLE
442.80
361.00
Twomorm
127.50
10
123.50
61.30
176.20
140.80
121.30
2320.40
TLE 218.30
90.20
TLE
89.70
73:30
Phishing
99.00
47.20
101.40
46.80
75.00
215.70
$3.30
138.90
TLE 693.40
70.70
TLE
80.50
53.00
Covertype
612.20
351, 90
758.80
218. 10
TLE
561.70
TLE
TLE
TLE
20.00
TLE
TLE
TLE
121.80
Biorespouse
602.40
.374. 70
540.10
354.60
560.40 10
529.00 00
713.60
TLE
TLE
602. 40
496.50
TLE
TLE
447.30
Pol
224.10
131.40
227.20
122.00
290.30
772.60
429.40
1009.80
TLE
224.10
175.20
TLE
205.30
124.59
QUerying Informative and REpresentative Examples (QUIRE), Active Learning by Learning (ALBL), Den- sity Weighted Uncertainty Sampling (DWUS), and Variation Reduction (VR). The Google library included Random Sampling (Uniform), (Centtrr-Greed, (KCenter or Corc-Set), Margin-basod Uncertainty Sampling (Margin), Graph Density (Graph), Hierarchical Sampling (Hier), Informative Cluster Diverse (InloDiv), and Representative Sampling (MCM). The ALiPy library contributed Estimation of Error Reduction (EER), BMDR, SPAL, and LAL. Besides, they proposed the Beam-Search Oracle (BSO) as a reference to approx- imate the optimal sequence of queried samples that maximizes performance on the testing set, aiming to assess the potential improvement space for query strategies on specific datasets. Through reviewing their re- leased source code, we identified differences between the task-oriented and query-oriented models for specific query strategies. Table 16 highlights the discrepancies between the two models for each query strategy To In particular, Margin and US (US-C and US-NC in our notation) are variant settings for Uncertainty Sampling. We further discuss such differences in Section 5.1 In re-benchmarking (Appendix C we adopt RBFSVM for a query strategy and evaluation by default.
Experimental design. The active learning algorithm was stopped when the total budget was equal to the size of the unlabeled pool, T= ". They collected the testing accuracy at each round to construct a learning curve, and the AUBC was calculated to summarize the performance of a query strategy on dataset. To ensure reliable results, they conducted Ks = 100 repeated experiments for small datasets (n < 2000) and KL = 10 repeated experiments for large datasets (n 2 2000), where n represents the size of the dataset. Finally, they compute the average AUBCs across repeated experiments for each query strategy on each dataset.
Analysis mothods. Zhan et al. (2021) benchmarked the pool-bascd active learning for classifications on 35 datasets, including 26 binary-class and multi-class datasets collected from LIBSVM and UCI (Chang & Lin 2011 Dua & Graff 2017). They provided the data properties such the number of samples n, dimension d, and imbalance ratio IR, where the imbalance ratio is the proportion of negative labels to the
10 settings are different from their source code for Google and ALP
29

---

Published in Transactions on Machine Learning Research (06/2025)
Table 16: Settings of query-oriented models H. for specific query strategies Q Zhan al. 2021
H
US
Zhan et al. 202
US-NC (Ours)
LR(C 0.1)
QBC
LR(C = 1). SVM(Linear,
probability = True),
SVM(RBF, probability
True), Linear Discriminant
Analysis
ALBL
Combination of QSs with same
H: US-C, US-NC, HintSVM
VR EER
LR(C 1) SVM(RBF,
probability =
True)
number of positive labels
[f(24, M) yt +1}
IR K(z,, M5) MJ = -11-
They employed these metrics to analyze the results from different aspects to explain the results of the query strategy's s performance dataset. We agree with their core idea of the analysis methods and believe their benchmark benefits the research community. However, we observe that the conclusion of their work differs from several previous works. For example, Zhan et al. (2021) claimed that LAL performs better on binary datasets than Uncertainty Sampling while not in the other benchmnrrk (Yang & Loog 2018). The evidence urges us to re-implement the active learning benchmark to clarify the conflicting claims.
B.2 Benchmarking datasets
3 records the datasets used in the previous benchmark (Zhan et al. 2021 However, we discover that the attributes of datasots are different. We report the revision Table 17 via Zhan al. (2021) Our new version'.
Section
B.3 Failure of the Reproducing Uniform
Table 20 (Table 21 shows the significant difference between ours and Zhan el al. (021) We noticed an
implementation error in the previous benchmark. In Google, Uniform assumees that the data has already been shuffled However, the implementation in Zhan et al. (021) does not shuffie the unlabcled pool first.
[Code=Python]
lec__daaa,labeeed____ dict_data labeled data, test data, unlabele__ dataa_ddta = data (dataset (faaaaet_aett dest_size n_labeled)
print (unlabeled data)
results indices of unlabeled pool
#[3, 4, 5, 10, 11, 13, 15, 16, 20, 23, 24, 26, 27, 29, 30, 31, 33, 36, 37, 41, 43, 44, 45
# 49, 50, 51, 53, 54, 55, 57, 63, 64, 65, 70, 73, 75, 77, 78, 79, 83, 84, 86, 87, 88, 89, # 91, 92, 95, 97, 102, 105, 110, 112, 114, 115, 121, 122, 127, 128, 131, 132, 136, 137, # 139, 140, 144, 148, 150, 151, 155, 157, 159, 160, 161, 162, 164, 165, 167, 168, 172, 175, \ # 176, 177, 178, 181, 182, 183, 184, 185, 186, 187, 188, 190, 191, 193, 194, 197, 198, 199,
baseline- baseline-google wwwgggoooglllrrre-ee binary.yyHL3311
30

---

Published in Transactions on Machine Learning Research (06/2025)
Table 17: Benchmarking datasets in this work and revision of Table 2
Zhan et al.
2021
Domain
k
d
Appendicitis
Health and Medicine Biology
2
4.05
7
106
Iris
3
1
4
150
Wine
Physics and Chemistry
3
1.48
13
178
Sonar
Physics and Chemistry
2
1
60
108->208
Parkinsons
Health and Medicine
2
3.06
195
Ex8b
Synthetic
2
206->210
Heart
Health and Medicine
2
13
270
Haberman
Health and Medicine
2
2
3
306
Tonosphere
Physics and Chemistry
2
1
34
351
Cleanl
Physics and Chemistry Health and Medicine Health and Medicine Business
2
1
168
166
475->477
Breast
2
1
10
478
Wdbe
2
1
30
569
Australiian
2
690
Diabetes
Health and Medicine
2
8
768
Mammographic
Health and Medicine
2
1
5
830
Ex8a
Synthetic
2
1
2
863->766
Tic
Games
2
6
9
958
German
Social Science
2
2
20-+24
1000
Splice
Biology
2
1
61-+60
1000
Gcloudb
Synthetic
2
1000
Gcloudub
Synthotic
2
2-+2.03
1000
Checkerboard
Synthetic
2
1-+182
1600
Myocardial
Health and Medicine
2
20.52
111
1700
Bioresponsc
Biology
2
1
419
3434
Abalone
Biology
21
114.83
8
4168
Academic Success
Social Science
3
2.78
36
4424
Spambase
Computer Science
2
1-1.54
4601
Banana
Synthetic
2
1
5300
Phoneme
Speech
2
2
5404
Satellite
Climate and Environment
6
2.45
36
6435
Ringnorm
Synthetic
2
1
21-+20
7400
Twonorm
Synthetic
2
1
50-+20
7400
Pol
Synthetic
2
1
26
10082
Phishing
Computer Science
2
1
2456++1108
Dry Bean
Biology
6.79
13611
IMDB
Text
2
1
768
50000
CIFAR-10
Image
10
1
768
60000
Diabetes 130-US Hospitals
Health and Medicine
3
4.83
2463
101766
RT-1oT2022
Engineering
12
3380.68
94
123117
Covertype
Biology
2
1
10
566602
# 202, 203, 204, 205, 207, 208]
We also modily their Uniform implementation by shuffle the unlabeled data.. Then, obtain similar results based on their source code, see Table I8
[Code=Python]
dict_data, labeled data, test test_data, umlabeled_date Applitdatt(ddtaasee (ataaaatttttttteepat.. filepath, test _ast_size, n labeled)
31

---

Published in Transactions on Machine Learning Research (06/2025)
Table 18:
Comparing different train/test/labele splits on Sonar: first column reprot and reproducing
results in Zhan et al. (2021), second column in our implementation, and the third column is reproducing
results after we revise Zhan et al. (2021) code.
Uniform
Report and code
Our code
Modified code
in Zhan et al.
in Zhan et al
2021
2021
Google
0.6274*
0.7513
0.7577
libact
0.7520
0.7543
ALiPy
0.7556
0.7579
random. seltalssufffllleeeee (unuabeeeelaeeeeeddeeee _
The unshuffled implementation Google significantly impacts binary classification datasots, such as Sonar, Clean1, and Spambuse. Also, it affects Exša and German, which enlarges the difference AUBCs between Uniform and other query strategies. Due to this experience, suggest practitioners ensure the correct- ness of the baseline method by comparing different implementations before conducting the benchmarking experiments.
B.4 Query Strategy and Implementation
We revise some descriptions of the query strategies in Zhan et al. (0021)
(1) 'Graph Density (Graph) is a typical parallel-form combined strategy that balances the uncertainty and representative based measure simultaneously via a time-varying parameter (Ebert et al. 2012
(2) 'Marginal Probability based Batch Mode AL (Margin) Chattopadhyay et al. (2012) selects a batch that makes the marginal probability the new labeled set similar to the one of the unlabeled set via optimization by Maximum Mean Discrepancy (MMD).
(3) Kremer et al. (0044) proposed an SVM-based AL strategy by minimizing the distances between data points and classification hyperplane (HintSVM):
Issue (1): Although Ebert et al. (2022) proposed the reinforcement learning method select uncertainty and diversity sample(s) during the procedure, Google YYilei "Dolee" Yang) 2017) does not implement the whole procedure but only the diversity sampling method method.
13 Thus, we should categorize it as diversity-based
Issuc (2): Google YYilei "Dolee" Yang 2017) ) docs not use MMD to measure the distance. The implementation
is uncertainty sampling with a margin score mentioned in the survey paper (Settles] 2012 Therefore, we should categorize it to uncortainty-based method.
Issue (3): libact (Yang al.) 2017) implemented HintSVM based on the work of Li al. 2015 rather than
Kremer et al. (2014)
B.5 Relationship between query strategies
We provide additional evidence explain the relationship between query strategies, which supports our experimental results.
(1) US-C and InfoDiv should be the same when the query batch size is one.
(2) Different uncertainty measurements should be the same in the binary classification, indicating that different uncertainty measurements do not canse differences between US-C and US-NC.
18 http://::e...aaaoooooooooo.ooaan mettatctt/erpppp.ddddstt....
32

---

Published in Transactions on Machine Learning Research (06/2025)
(3) SPAL changes the condition of variables used for discriminative and representative objective func- tions in BMDR.
Issue (1): InfoDiv clusters unlabeled samples into several clusters, then selects uncertain samples and keeps the same cluster distribution simultancously!! Therefore, it is the same when we set the B = 1 to query the most uncertain sample. Zhan et al. (2021) provided the different numbers of US-C and InfoDiv in Table 4, which might have resulted from using the different batch sizes of these query stralegies.
Issue (2): The least confidence, margin, and entropy are monotonic functions with a peak equal to P(y
+1 x) = 0.5 binary classification, such that all of these unccrtainty measurements would query the same
point (Setless 2012).
Tssue (3) The optimization problem in BMDR is
Di
min
Σ
Uvs - w wT o(xx))) + +||w=]/
=b, i=1
Dul
(1)
aj |||l o(x,))1 2/w TTx;))
+ 3(a Kia + ka),
where (xx) is the feature mapping, À is the hyper-parameter for the regularization term, 3 is the hyper- parameter for the diversity term, 1/p,l means ones vector with length of the unlabelled pool [Dul. K1 is defined as K1 = 1/2 KEU, where KUu means the kernel matrix with sub-index U of unlabelled pool Du. SPAL only changes a T p.1 b to e
B.6 Comparison between Zhan et al. (2021) and Yang & Loog (0018)
Yang & Loog 2018 propose the first benchmark for pool-based active learning for the conventional Logistic Regression model. The work compares query strategies that could be categorized into model uncertainty and hybrid criteria. In datasets, they adopt 44 binary datasets and follow data pre-processing in Chang & Lin 22011) To compare performance across different query strategies, they also
Arca Under the Learning Curve with accuracy to show the average performance of query strategy, named AUBC in Zhan et al. (2021). Furthermore, they compare the performance of each query strategy by average rank and improvement (win/tice tie//oss) from random sampling, which has the same purpose as our work (See Section
and Section 4.2
C Re-benchmarking results of Zhan et al. (2021)
After we accomplish experiments under the settings in Appendix B.1 we obtain the benchmarking results for RBESVM with the form (query strategy, dataset, seed, Di], accuracy) for each round. A (random) seed corresponds to the different training sets, test sets, and initial label pool splits for a dataset. We collect the accuracy at each round (IDi), accuracy) to plot. a learning curve for query strategy on datasel with seed and summarize it the mean AUBC in Table 19 Our re- re-benchmarking results show that Uncertainty Sampling with compatible models (US-C) outperforms the other query strategies most datasets.
C.1 Statistical comparison of re- re-benchmarking results
We show our re-benchmarking results for RBFSVM side-by-side with Zhan al. 2021 Table 3 in Table 20 To determine if there is statistical difference between the two benchmarking results, we construct the confidence interval with the t-distribution of mean AUBCs. If a result in Zhan et al. (2021) falls outside the interval, their mean significantly differs from ours. We notice significant differences in Uniform on several
datasets in Table 20 Therefore, we focus on comparing Uniform Table 21 demonstrating our mean and
14
diverse.
33

---

Published in Transactions on Machine Learning Research (06/2025)
Table 19: Mean AUBC of Query Strategies: We report query strategies with mean of repeated experiments.
Uniforn
US-NC
QU0:
Graph
Corn-Set
HinBVM
1SWU5
LAI.
Appenoiantie
65.95%
81.51%
86.095
51.54%
84.418
64.965
K3-96%
84.145
56.19%
63.68%
K3.305
63.005%
84.21%
81.57%
61.18%
84.15%
15.89%
61.35%
Sonuc
Parkinsons
75.003%
55.15%
77.883
85.313
85.115
70.54%
55-1176
77.88%
76.75%
84.05 09%
78.75%
58.5116
83 83.00%
75.005
83.575
75.645
50-915
76.585
74.549
83.35%
74.065
51.75%
K3.055
75.115
74.559
81.78%
77.51%
55 S30153
75-98%
81.83%
TLE
-4.0%%
76,255
816356
70.708
Rabhs
58.33 5.19
89.81 89.813
59.395
53.8186
50.3006
88 84-50%
85.813
58 5656
89 1575
N0.555
88-08 175
26 1.7
89 115
Hrst
40.5:7%
81.57%
$1.20%
SL-378
81:305
50.501
90.31%
80.753
50.545
81.00%
50,355
41.01%
90,5735
81 21.541
92,0735
+1.145%
51. 11.246
Uebernmnn
73.1225
72. 72.99%
724025
72:986
734055
T01456
71.005
73.013
73.045
72.7%%
72425
72. 15
71 70.10%
72
TLE
73.974
77A 77411%
Ioncoplace
91.8 008
53.007
91.977
00.005
92.785
12.195
01. 01.305
92.045
01.85%
51. 01.585
80.6455
90.155
87.065
02.961
92:525
02 685
Cleanl
61.709
84.32%
88.41%
54.155
85.25
10.155
TLE
81.865
41.00%%
73.0235
70.9755
61.8555
81. 81.7005
T1E
THE
59.365
Reoee
96.14 14%
56. 51%
96 555
16:345
16:315
18 5555
96.175
56.1555
16 985%
96 98.9455
99 W0.00
08:355
10 665
Wilbe
90.2575
166 5236
90 90.91%
06:325
m255%
56:125
10 00.30%
05.025
50.4355
10,189
95.8335
05.04%
58.1256
Australiazi
64.8355
85. 55.04%%
84 59.5994
55.0055
88 1144
54.756
81.77%
84.785
88 54.72%
85 85015
Diabetes
Mananngrapiie
81.255
74.24%
A1.6435
74.799
81.659
74.229
74.795
51,9455
81613
74725
74.175
51:355
T4.38%
48.23%
71.515
81
81:05
74:265
51:225
741615
96 58.455
$0.985
743665
74.706
81.624
02.55 22.225
53.50%
81
74.7155
#1.685
51.325
81.220
74.455
74
51 51.3953
ReBa
65, 03.5225
88. 01
82.835
58.0155
60105
55 55-555
88 53599
83 115
15.555
88 5559
81.580
88 2359
Ti:
67.18%
67.5005
87.185
87 871005
671155
67 100
$8.00
87 103
83 3
87.125
63.155
German
73, 0.55
74.17%
73.8754
T1 1000
1100%
11.
Gcoodds
Spiice
40 6255
80.000 20.5075
89,
101 0955
88.68%
075
85015
71 100
100%
87.62%
88 3855
TLE
40 4556
Gcionduhh
94. 4075
94.89%
94
63.17%
TLE
Cleckerboanl
97.
91,249
96 130
94.57%
86-22
TLE
06 1575
Spanbbee
172.05
M1 10
11
87. 97%
18 100
81 1.%
11
565
Plancue
62
53.55%
121
50.528
88185
THE
TIE
83.0%
53.956
81.81%
81,575
88 33.0%%
THE
TLE
62.47%
52 42%
Bingnoco
17. 807
97.07%
17 407
07.1
TLE
TLE
07.66%
11%
94 94.77%
TLE
00.40%
T1E
TLE
97 817
Texcuma
07.63%%
07.6335
97.638
97.00
TLE
TLE
97.628
07 64%
07. 775
003065
TLE
07.849
07.03%
TLE
TLE
62%
Plishing
03.825
94.005
98.01%
01605
94.11%
TLE
TLE
95.90%
06.27%
94.005
22.965
TLE
80.2855
41.
TLE
TLE
94.20%
04.295
Table 20: We report our AUBCs (%) with Table 3 in Zhan et al. (2021) side-by-side. A score denoted
with format: Zhan et al. (2021) ours. The symbol ** indicates a significant difference with the
significance level a = 5%.
Uniform
BSO
Avg
BEST
BEST_Q Q$
WORST
WORST_QG QS
Appondicitis
84%
83.95%
88%
88.37%
84%
84.23%
86%
84.37%
EER MCM
83%
83.90%
DWUS HinSVM
Sonar
62%
71.63%
83%
88, 10%%
76%
+ 75.60%
78%
77.62%
LAL - US-C
73% - 73.57%
HimISVM HHiuISVM
Parkinsous
84%
83.05%*
88, 88.28%
85%
83.97%
86%
31%
QBC-
US-C
83% 81.78%
HinISVM MHIuISVM
ExSb
87%
88. 53%
92%
93.76%
89%
+ 88, 88%
91%
89 89.81%
SPAL- - US-C
80% - 80.99%
HimISVM +HiuISM
Heart
81%
80.51%
89.30%
79%
50.99%
63%
8L.7%
InfoDiv
US.C
72%
80.39%*
DWUS
HimSVM
Haherman
73%
73.08%
75%
78.0695*
73%
72.95%
74%
73.19%
BMDR BMDR
72% 72.44%
QUIRE
QUIRE
Tanospbere
90%
91. 80%
03%
95.459%
91%
+ 91. 91.60%
98%
93.00%
LAL +
US-C
88% + 87.93%
HinSVM
DWUS
Cleaul
65%
81 83%
57%
92.19%*
81%
81.97%
84%
81 25%
LAL
US-C
73%
76.95%
HimISVM -HinISM
Breast
95%
96 36.16%
90%
97.6055*
96%
+ 96.19%
96%
96 36.
SPAL + US.C
95%
95.82
DWUS + VR
Wilce
95%
95.39%%
97%
98.415%*
96%
95.87%
97%
96.52%
LAL
US-C
94%
95.04%
EER DWUS
Australian
86%
84 84%
88%
90.446%*
80%
- 84.
86%
+ 86 65.04%
Cono-Set - US-C
82% - 84.44%
DWTS HInSVM
Diabeles
74%
74.21%
82.57%
74%
72.1
73%
71.91%
Cor- Sel +Cure-Set
69%
72.27%
EER - DWUS
Mammmggaphiic
82%
81.30%
8.%
$5.0395*
82%
+ 81.44%
83%
81.78%
MCM - MCM
80% - 79.99%
EER DWUS
ExeBa
84%
85.39%
87%
88.2899 24.28%**
845
84.62%
96%
87.88
US-C
79.11% 80%
QUIRE
DWUS
Tie
87%
88%
87%
90.7779
87%
87.17%
87%
87.20%
FER US-C
87% 86.00%
QUIRE
QUIRE
German
73% + 73. 40%
78%
74%
+ 73.65%
74%
+ 74.17%
QBC - US-C
72%
72.68%
DWUS
DWUS
Splice
81%
80.75%
87%
91.02%*
79%
80.08%
82%
82.31%
QBC
MCM
68%
75.18%
EER Core-Set
Golondls
89%
89.5 59.52%
90%
90.915%*
89%
-9 80 80.207%
90%
+ 89.81%
Graph US-C
87% - 87.48%
HimtSVM -MllnnSVM
Gelondah
94%
94.37%
96%
96.83%
90%
93.72%
95%
95.60%
QBC US-C
80%
89.299
EER Coro-Set
Checkerboard
98%
97. 81%
90%
99.72%
94%
- 92%
90%
98 78%74%
Core-Sel
-Curee-Set
90%
90.43%
VR - DWUS
Spanbase
69%
91.03%
TLE
88%
91.14%%
92%
92.05%
QBC
US-C
09%
89.85%
DWUS- HimSVM
Banana
90%
89. 20%
TLE
85% + 86.90%
89%
39.30%
Hier + Core-Set
78% + 80.50%
QUIRE -
US-NC
Phonerne
82%
82.54%
TLE
825 82.49%
83%
QBIC
MCM
80% 80.83%
HInSVM HHMMSVM
Ringrorm
98%
97. 77.76%
TLE
9556 V7. 00%
98%
97 86%
LAL
US-C
80%
93.46%
DWTS DWUS
Twosscmu
98%
97.539 97.58%
TLE
98%
+ 97. 775
98%
97 97%
Core Sei
US-C
97% + 97.31%
DWUS DWUS
Phishing
93%
93.82%
TLE
94%
93.65%
95%
91.60%
LAL
US-C
92%
$9.23%
Gmaph DWUS
standard deviation (SD) AUBC of Uniform and the mean AUBC of Uniform reported by Zhan et al. 2021 There are 13, nearly half of the datasets, significantly different from the existing benchmark with significance level a = 5%. Furthermore, we perform better on most datasets except for Parkinsons and Mammogruphic. 1% of mean AUBC is larger than previous work on 8 datasets, especially for Sonar, Cleant, and Spambase. Following the same procedure of statistical testing, Table 22 demonstrates BSO of ours and Zhan et al. (201). This phenomenon is more evident in BSO than in Uniform. We still get significantly different and better performances on most datasets except for Aypendicilis.
C.2 Verify usefulness
Zhan et al.
2021 verified the applicability of a query strategy by several aspects:
34

---

Published in Transactions on Machine Learning Research (06/2025)
Table 21: Reporducing Failure of Uniform
Mean
SD
Zhan et al.
2021
5%
1%
Appendicitis
83.95%
3.63%
83.6%
In
Sonar
74.63%
3.79%
61.7%
Out
Out
Parkinsons
83.05%
3.68%
84.0%
Out
Ex8b
88.53%
2.80%
86.6%
Out
Out
Heart
80.51%
2.79%
80.8%
Haberman
73.08%
2.70%
72.7%
In
In
lonosphere
91.80%
1.78%
90.1%
Out
Out
Cleanl
81.83%
1.94%
64.9%
Out
Out
Breast
96.16%
0.90%
95.4%
Out
Out
Wdbe
95.39%
1.30%
95.2%
In
In
Australian
84.83%
1.58%
84.6%
In
In
Diabetes
74.24%
1.52%
73.6%
Out
Out
Mammographic
81.30%
1.98%
81.9%
Out
Out
Ex8a
85.39%
2.17%
83.8%
Out
Out
Tic
87.18%
1.53%
87.0%
In
In
German
73.40%
1.73%
72.6%
Out
Out
Splice
80.75%
1.61%
80.6%
In
In
Gcloudb
89.52%
1.17%
89.3%
Geloudub
94.37%
0.96%
94.2%
In
In Out In In In
Checkerboard
97.81%
0.59%
97.8%
Spambase
91.03%
0.57%
68.5%
Out
Banana
89. 89.26%
0.38%
89.5%
In
Phoneme
82.54%
1.01%
82.2%
In
Ringnorm
97.76%
0.21%
97.6%
Out
Twonorm
97.53%
0.19%
97.6%
In
Phishing
93.82%
0.48%
92.6%
Out
Out
Low/high dimension view (LD for d < 50, HD for 2 50),
Data scale view (SS for < 1000, LS for n 2 1000),
Data halance/imbalanne /imbolance view (BAT for y < 1.5, IMB for y 2 1.5).
They compare these aspects with score
8..= max (AUBCBsO,s AUBCUS, AUBCLAL. AUBC,,s
Specifically, they grouped 64. by different aspects to generate the metric for the report
8,00 Esco 6q.5 1(8 E E
where e is one a dataset's dimension, scale, or class-balande balance views. We re-benchmark results and denote the rank of the query strategy with superscript in Table 23 Table shows that the US-C (InfoDiv) and MCM occupy the first and second ranks in different aspects, and the QBC keeps the third rank. The results are unlike those of Zhan et al. (2021) h except for the QBC performance well both of us. We explain the reason for the same performance of US-C and InfoDiv in Appendix B.5
Using score 5g.r to ascertain the applicability of several query strategics is straightforward. However, it could bring an issue: BSO oulperforms query stralegies significantly on most datasets in our benchmarking results. We cannot exclude those remaining large-scale datasets without BSO, i.e., n > 1000, having the same pattern, such that their results could impact different aspocts. Therefore, we replace 8 with the improvement of query strategy over Uniform, i.c., Tq.skk Section 4.2 bocause Uniform the baseline and most efficient across all experiments, which is essential to complete.
35

---

Published in Transactions on Machine Learning Research (06/2025)
Table 22: Reporducing Failure of BSQ
Mean
SD
Zhan al.
2021
5%
1%
Appendicitis
88.37%
2.95%
88.1%
In
Sonar
88.40%
2.84%
83.0%
Out
Out
Parkinsons
88.28%
3.19%
86.5%
Out
Out
Ex8b
93.76% 73.76%
1.82%
92.4%
Out
Out
Heart
89.30%
2.17%
84.8%
Out
Out
Haberman
78.96%
3.05%
75.1%
Out
Out
lonosphere
95.45%
1.42%
93.3%
Out
Out
Cleanl
92.19%
1.69%
87.1%
Out
Out
Breast
97.60%
0.67%
96.1%
Out
Out
Wdbe
98.41%
0.65%
97.3%
Out
Out
Australian
90.46%
1.48%
87.8%
Out
Out
Diabetes
82.57%
1.70%
78.4%
Out
Out
Mammographic
85.03%
1.97%
84.4%
Out
Out
Ex8a
88.28%
2.03%
87.3%
Out
Out
Tic
90.77%
2.27%
87.3%
Out
Out
German
82.08%
2.01%
78.3%
Out
Out
Splice
91.02%
1.18%
87.1%
Out
Out
Gcloudb
90.91%
1.09%
90.1%
Out
Out
Geloudub
96.83%
0.78%
96.3%
Out Out
Checkerboard
99.72%
0.36%
99.2%
Out Out
Table 23: Verifying Applicability with 6;
LD
HD
LS
BAL
IMB
US-NC
4.77
4.16
8.12
5.36
3.96
5.09
4.39
QBC
3.83
3.15
7.573
5.02
2.20
4.05
3.57
HintSVM
5.91
4.92
11.37
6.77
4.73
6.25
5.51
QUIRE
5.96
5.08
11.54
6.13
5.60
6.94
4.98
ALBL
4.20
3.49
8.06
5.37
2.59
1.15
3.90
DWUS
6.20
5.46
10.24
6.71
5.50
5.46
VR
5.04
4.26
12.02
5.43
4.13
5.36
4.72
Core-Set
4.92
3.78
11.20
5.79
3.72
5.35
4.42
US-C
3.50
2.891
6.86
4.62
1.97
3.72
3.24
Graph
4.62
3.72
9.58
5.77
3.05
4.98
4.20
Hier
4.22
3.41
8.69
5.53
2.43
4.49
3.90
InfoDiv
3.50
2.89
6.861
1.62
1.97
3.72
3.21
MCM
3.56
2.91
6.982
4.68
2.03
3.80
3.27
EER
5.21
4.18
11.09
5.33
4.86
6.13
4.30
BMDR
5.61
4.57
11.50
5.77
5.11
6.33
4.89
SPAL
5.90
4.69
12.32
5.67
6.77
6.56
5.17
LAL
4.14
3.41
8.14
5.27
2.59
4.37
3.86
The other issue is heuristically grouping the views into binary category and averaging the performance with the same views 5G without reporting SDs. These analysis methods may be biased when the propertics of datasets are not. balanced. To address this issue, we plot a matrix of scatter plots that directly demonstrates the improvement of US-C for each property on all datasets with different colors. Figure [2 shows a low correlation (Irl 0.4) and no apparent patterns between propertics and the improvement of US-C, indicating that Our analysis results do not support the claims of 'Method aspects' in the existing benchmark (Zhan) et al. 2021 either. In conclusion, want to emphasize that revealing the analysis methods is
36

---

Published in Transactions on Machine Learning Research (06/2025)
Table 24: Mean and standard deviation of AUBCs of query strategies under no-ftxed and fized initial sets.
Uniform Table
US Table 3
3
FixInit
FixImit
Heart Mammographic
78.37%
77. 77.88%+2.58%
79. 79.32%
78.82%14.44%%
80.75%+1.399 79.14%+1.59% 8..66%+00.55% 85.78%+0.566%
79.46%+1.500%
9..53%++1.88
Phoneme
8..87%+0.388%
8..67%+0.37%
important as the experimental settings because the analysis method employed will influence the conclusion.
AUBCIUS-CI AUBC(Uniform)
IR
0.05
-01167
r=
00.313
r=-0.122
0.00
-0.05
2.5
5.0
100
10000
Figure 12: A matrix of scatter plots of the improvement of US-C
C.3 The impact of initial sets
The current results are based on the train-test split described in Section 3 followed by random sampling select the initial labeled and unlabeled pools. In this section, we follow Ji et al. (2233 recommendation to check the influence of the consistent initial set (labeled pool and unlabeled pool) Uniform and Uncertainty Sampling. Specifically, we investigate two settings on Heart, Mammographic, and Phoneme datasets:
No fixed train-test split and no fixed initial sets (Table 3 current results),
Fixed initial sets and no fixed train-test split (Fixlnit).
The similar standard deviations between no-fixed (Table and fixed initial sets in Table 24 indicate that fixing the seed of the initial sets across different train-test splits does not lead to significant differences. This is mainly because the primary source of randomness comes from the train-lest split itself. Because the datasets in our benchmark do not have predefined train-test splits, we would not eliminate the randomness by fixing the train-test split in our experiments. Although the randomness of the train-test split exists, our benchmarking results show that Uncertainty Sampling performs consistently on most datasets.
C.4 More on analysis of non-compatible models for uncertainty sampling
Section 5.1 demonstrates results involving different combinations of query-oriented and task-oriented models on Checkerboard and Gcloudb datasets. We reveal more datasets from Figure [3] to Figure [7 These re- sults still hold for the compatible models for uncortainty sampling outperform non-compatible ones on most datasets, ie., the diagonal entries of the heatmap are larger than non-diagonal entries. Figure [8 demon- strates that non-compatible models achieve slightly better performance than compatible models. When
37

---

Published in Transactions on Machine Learning Research (06/2025)
query-oriented and task-oriented models heterogeneous, conjecture that it could improve uncertainty sampling by exploring more diverse examples like the hybrid criteria approach (Settles 2012 Sinha et al. 2019h.
D Benchmarking results of Random Forest
This section follows the analysis procodure in Section to benchmark Random Forest (RF) on the same datasets. The analysis results are listed as follows:
1. Verify the superiority by comparing the mean AUBC of query strategies in Table 25
Verify the superiority by comparing the average accuracy of the model with 20% of the total budget Table 20
3. Verify the superiority by comparing the average ranking of query strategies in Table 27
4. Verify the usefulness by comparing the data utilization rate of query strategies in Table
These results are consistent with the previous benchmarking results in Section and Appendix We conclude that the uncertainty sampling with compatible RF models gains superiority and usefulness for tabular datasets.
E
Computational resource
We test the time of an experiment for query strategy running on a datasct. Our resource is: DELL PowerEdge R730 with CPU Intel Xeon E5-2640 v3 @2. 22H6GHz * and memory 192 GB, We report the computational time for a query strategy for each dataset per round in Table 29 following the setting in Section 3
Note that this work does not optimize libact, Google, and ALiPy performance. If practitioners discover inefficiett implementation, please contact us by mail or leave issues on GitHub.
F Limitations, related benchmarks, and future works
While we intentionally constrain our benchmark's scope to maintain fairness and reproducibility, this focus might give the impression of limitations. We encourage practitioners to explore active learning techniques in broader tasks and domains. For example, ample room exists to investigate active learning's applicability
in areas like regression problems, object detection, and natural language processing (Cai et al. 2016 Wu
ct al. 2019 Zhang al.] 2020 Yuan al. 2021 Brust ct al. 2018 Zhang et al., 2022
Evaluating the performance of query strategy is challenge in benchmarking. Kottke et al. 2017 and Trittenbach et al. (2021) propose metrics such as Deficiency score, Data Utilization Rate, Start Quality, and Average End Quality to summarize the performance of a query strategy from learning curves. Our implementation saves querying results at each round, enabling thorough analysis without costly re-runs, which empowers researchers to develop novel metrics and methods.
The stability of experimental results another challenge to fair comparison. Ji al. 2023 : Lüth et al.) (0 Munjal et al. (2022) have revealed variations in performance metrics stemming from different query strategies, causing inconsistent results and claims in previous rescarch. They suggest standardizing experi- mental settings like data augmentation, neural network structures, and optimizers. These findings emphasize the sensitivity of active learning algorithms to experimental settings, critical consideration for future work.
38

---

Published in Transactions on Machine Learning Research (06/2025)
Table 25: Benchmarking results of Random Forest. The numbers are mean AUBC (1, %). We report the baseline method (Uniform), the best query strategy with its mean AUBC (BEST QS, BEST). and the worst query strategy with its mean AUBC (WORST QS, WORST) across datasets in Table 25
Uniform
BEST QS
BEST
WORST QS
WORST
Appendicitis
83.70%
US
84.48%
DWUS
83.12%
Sonar
75.66%
US
77.31%
HintSVM
74.75%
Parkinsons
84.31%
US
86.61%
HintSVM
82.79%
Ex8b
85.97%
US
87.07%
HintSVM
84.67%
Heart
80.19%
DWUS
80.93%
HintSVM
79.64%
Haberman
69.56%
70.61%
QUIRE
68.91%
Tonosphere
90.98%
BALD
92.08%
HintSVM
87.22%
Cleanl
79.15%
BALD
82.09%
HintSVM
75.18%
Breast
96.42%
US
96.82%
DWUS
95.44%
Wdbc
94.29%
LAL
95.32%
HintSVM
93.92%
Australian
85.77%
US
86.20%
DWUS
85.55%
Diabetes
74.60%
LAL
74.97%
DWUS
73.59%
Mammographic
79.36%
LAL
80.82%
DWUS
78.82%
Ex8a
93.09%
BALD
95.50%
HintSVM
87.65%
Tic
86.36%
Core-Set
86.43%
DWUS
85.48%
German
74.02%
US
74.74%
DWUS
72.86%
Splice
90.49%
MCM
91.52%
Core-Sel
84.17%
Geloudb
88.33%
LAL
88.96%
QUIRE
86.50%
Geloudub
93.83%
BALD
95.34%
HintSVM
87.38%
Checkerboard
99.24%
LAL
99.67%
DWUS
95.00%
Spambase
93.54%
BALD
94.74%
HintSVM
92.11%
Banana
88,25%
LAL
88.82%
DWUS
81.54%
Phoneme
86.63%
BALD
88.81%
HintSVM
84.75%
Ringnorm
94.15%
US
95.66%
Core-Set
70.55%
Twonorm
96.60%
BALD
96.88%
HintSVM
94.78%
Phishing
95.61%
US
96.68%
HintSVM
94.24%
Covertype
76.47%
US
79.20%
Uniform
76.47%
Bioresponse
73.57%
US
74.83%
Uniform
73.57%
Pol
96.58%
US
97.85%
Uniform
96.58%
39

---

Published in Transactions on Machine Learning Research (06/2025)
Table 26: Accuracy of the model with 20% labeled examples: We report the accuracy of the model with 20% labeled examples on each dataset. The scores with bold mean the best performance on a dataset. 'TLE' means a query strategy exceeds the time limit.
Uniform
US
QDC
BALD
Ilerr
Graph
Coze-Set
HisSVV
QUIRE
DWUS
MCM
DMDR
ALBL
LAL
Sonar
68. 98%
09: 99%9
69.965%
70.45%
69.11%%
70.77%
68.57%
69.407%
70.1156
70.15%
69.81%%
70.92%
70.33%
70.
Parkinrous
61.04%
82.21%%
81.56%
80.53%
81,08%
91.97%
81.41%
81.21%
61.94%
82.44%
83 41%
82.45%
93.62%
82.56%
82.04%
83.83%
81.85%
83.045
82.61%%
53.20%
83.43%
13.1887
83.
Heart
78.57%
79.42%
77.807
78.97%
78.89%
79.08%
78.81%
77.4078
78. 78.78%
79.6055
78.54%
78.83%
lonosphere
Haberman
88 6336
70.115
71.63%
91. 34%
70.24%
88.42%
71,58%
91.9196
11.888%
80, 19%
70.67%
88. 88.50%5
70.799
83. 83.14%%
69. 69.31%
69.57%
82.3095
68. 76%
84
71.17%
70.19%
70.594
88.7398
90. 90.87%
71.
Cleun
71 71.01%
72: 94%
70.82%
77..4276
71.25%
71.2775
66.915%
66.6377
71.31%
72.3430
72. 14.5
Breast
96.35%
97. 18%
96.511%
97. 07.14
96.66%
96.3035
65.705
96.36%%
96 96.93%%
96
96
96.3196
90.89%
97. 97.23%%
Wilbe
98.56%
95.12 12.%
92,699%
18%
Diabetes
Arattaaiion
73-84%
84.97%
86.175%
13%
74.07%
55.2
86 860009
73
85.
42%
85.544
734 73.958
73 3
84 84.69%
855
46%
73.
55.00%
85.47%
74 74.33%
85. 969%
Excaa
Mammographile
79.74%
89.915%
82.46%
95.05%
79.75%
89.58%
80
16
93 13.1
S1.19%
77.98% 9999
80.97%%
10000
96
87.17%
80.6236
6-2 22%
94L, 66%
Tic
German
86 929
72.869%
73. 62%
56. 922%
12.64%
80.84%
88
87
72.
022%
72. 755
96
933
72.53%
80 56%
86
72.5799
73.22%%
33344
73.72%
TLE
Splice
87 65%
86.77%
67.100
90%
Gelonbb
Gelonduh
88 277
02.02%%
89. 25%
31%
92
88 58%
24%
88
54.45%
85.
10.1000
84.000
84.
84.999
16
98%
888
80.0000
29.2536
03
89. 89.58%
Spanlose
Barama
Checkeeboard
09 2855
88.09%
87 97.977
99. 14%
80 25%
95. 32%
99.52%
000%
99,1395
87. 2222
90
29,000
92
90.5699
90. 2996
75.085
90.15%%
10
99.0090
90.206
66
2000
87:277 2333
99.
92.80%
10%%
99. 99.88%
95 95.08%
89 16%
Phoneme
84.44%
58 (11%
84.41%
85.
55.5115
81 81.399%
83.1456
12%
84
84.4666
87 87.5
Ringtorm
53.7355
96.91%
93.907%
92,9015
50.999
16.1
50.003 7
60.
92.883%
90. 307
Phishing
Twosorm
94.83%
98.47%
96.85%
56. 83%
90.52%
96
95.30%
90.0099
96.61%
96.8800 86..00%
93.07%%
92.91%
91.79% -
-
92, 799
TLE
96.07%
95.37%
96. 85%
83%
Cownttype
74.78%
76. 91%
TL.
TLE
TLE
TLR
T1.
TLP.
TIR
TLE
Biocesgunzee
70.63%
73.03%
TLE
72.61% -
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
Pol
05.94%
98.23%
TLE
08 222%
TLE
TLE
TLE
TLE
TLE
TLES
TLE
TLE
TLK
TLK
Table 27: Average Ranking of Query Strategies: We report query strategies with the best average ranking. The scores with 2,2 or mcan the 1st, 2nd and 3rd performance on datasct. 'TLE' mcans a query strategy exceods the time limit.
US
QBC
BALD
Hier
Graph
Core-Set
HintSVM
QUIRE
DWUS
MCM
BMDR
ALBL
LAL
Appendicitis
4.81
7.76
5.44
7.30
7.74
7.31
8.43
7.89
8.85
5.63
7.81
6.47
5.56I
Sonar
3.691
6.95
3.97'
7.86
7.77
8.15
8.87
7.71
7.83
3.96
TLF
6.52
4.72
Parkinsons
2.97
8.55
3.08:
6,81
10.06
9.53
11.61
9.34
7.17
3.72
7.77
6.6G
3.73
ExSb
4.50a
5.08
4.37
7.56
8.60
6.04
10.10
8:26
8.90
5.133
7.28
6.82
5.36
Heart
6,06
7.49
6.26
7,46
8.40
7.86
9.42
8.24
4.90
5.74P
6.34
6.63
6.20
Haberman
5.071
7.22
5.33
7.42
7.26
8.38
9.35
9.71
5.252
5.299
7.10
7.90
5.72
Ionosphere
2.43*
6.53
2. 40
6,29
7.27
9.96
11.50
11.55
11.25
2.90
8.03
7.17
3.72
Cleanl
2.69*
7.76
2.44
7.40
9.25
9.49
11.60
7.15
7.86
2.96'
TLE
5.58
3.82
Breast
3.44
8.35
3. 6.1
9.60
8.36
7.66
9.05
12.24
3. 3.600
8.54
5.94
3.83
Wibe
3.05
9.30
2. S0
8.41
9.33
9.58
10.67
9.42
9.31
3.32
8.19
4.49
2.93
Australian
4.77a
7.48
4.581
78
8.50
7.25
8.89
5.69
8.26
6.82
5.663
Diabetes
5.53
6.75
5.77"
7.22
6.83
6.30
8.27
7.78
10.86
6.66
7.75
5.05
Mammographic
3.92a
9.62
3.73
7.66
8.23
6.96
6.74
8.54
3.41
ExcSa
2.37
7.93
2.39
7.27
7,95
4.10
12.14
11.78
11.86
3.14²
6.43
9.83
3.81
Tic
7.49
4.53"
7.66
6.58
5.02
3,89
6.15
7.01
9.01
7.50
TLE
4.82
8.01
German
3.80
7.89
4.19P
8..4
7.55
7.07
8.71
7.55
12.52
4.34
8.21
5.67
5.06
Splice
2.72
6.46
2.817
6.15
9.43
11.59
9.86
6.20
10.77
2.72a
TLE
5.23
4.06
Geloudb
4.81
7.34
4.76'
7.65
8.25
6.73
11.28
10.51
10.25
4.90
6.76
4.53
3.23'
Geloudub
2.174
7.34
2.46'
5.49
6.63
8.48
12,90
11.21
11.56
2.87
7.48
7,80
4.31
Checkerbourd
3.00P
7.67
3.22
6..47
7.49
7.20
12.18
11.67
11.81
3.15P
5.83
8.38
2.90
Spambese
2.40P
7.50
1.
5.50
9.10
9.30
11.00
TLE
7.60
2.60
TLE
6.00
3.40
Ba TLEa
3.10a
6.90
2.90
7.00
9.60
5.60
11.60
12.00
12.4
3.30
6.10
8.30
2.20
Phoneme
2.40a
8.60
1.901
5.00
8.60
6.50
11.60
9.60
11.30
2.20
TLF
6.80
3.50
Ringnorm
1.40
6.00
1.80F
4.60
8.00
11.70
9.30
11.30
9.70
3
TLE
5,90
5.20
Twonorm
1.90
6.20
1.80
9.50
5.50
4.80
10.90
TLE
8.00
2.30
TLE
8.80
6.30
Phishing
1.50
7.00
2.30P
5.70
7,50
5.80
9.80
TLE
9.20
2,20
TLE
4.00
TLE
Cavertype
1.40
TLE
1.60P
TLF
TLF
TLE
TLF
TLE
TLE
TLE
TLF
TLE
TLF
Bioresponse
1.40
TLE
1.60"
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
Pol
1.30
TLE
1.70P
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
40

---

Published in Transactions on Machine Learning Research (06/2025)
Table 28: Data utilization rate of query strategies. The scores with 2 or mean the 1st, 2nd and 3rd performance on dataset. 'TLE' means a query strategy exceeds the time limit.
US
QBC
BALD
Hier
Graph
Core-Set
HMwSVM
QUIRE
DWUS
MCM
BMDR
ALBL
LAL
Appendicttss
72.68%
15.27%%
72.37%
83777%
84.57%
T8.46%6
94.03%
7U.50%
96.47%
73.37%
77.20%
76.18%
68.32%
Souar
53.21%
90.98%%
79.75%
103. 93%
165.195
109.28%
113. 81%
100. 109.00%
98. 38.23%
82.09%
93.92%
91.02%
81.00%
Paricinsons
06.78%
104-47%
65 65%
89.015
115.53%
113.00%
125. 06%
109 88%
19.199
71.19% 19.199
71.19%
Exöb
72.10%
100.5126
75.02%
108. 96%
105.5435
22.265
15g. 12%
107.66%
201.85%
78.497%
07.42%
92.0775
78-03%
Heart
83.52%
133966
80.04%
87 8394
98. 14.8
165. 58%
126. 9119
145.543%
96.76%
85.71%
88
26.30%
94.1
2039
84 565
Haberman
208.29%
166.2516
86.2276
127.40%
117.9955
155. 79235
194, 887
169. 7667
110. 120.28%
84.047%
108. 169%
191-4058
94-10%
Tonospbore
71.0056
409.4736
112.155 133
118. 9636
184.4256
B01 0919
199. 788
966.99%
75.05%
TLK
117.5994
78. 14%
Cleam
66.38%
101.5176
68.547%
96- 56-3355
119. 0395
165. 7995
125. 7999
98.0007
99.31%%
67.755%
104. 9599
75-8226
Becast
56.07%
91.30%%
58.339%
S3 407
7255
92.76%
94.25%%
90.92%
842.119
58.36%
124-45%
78.14% 44%
50. 50%
Wilbe
48.497%
138.8807
49.577%
100 02--022
110. 9235
119.6895
147. 17..99%
58%
219. 5.4%
52.565%
04 8006
62-335
Australian
71.45%
95.51%
73.109%
99 06.324
25%
114.8435
106. 655
208, 7978
125. 022
73.67%% 0
101, 769
92 92,5544
117- 100000
74. 20%2
Tie
Diabetes
Ex8a
Manmmographic
70.399
43. 13.73%
95.333
72.07%
96.117%
158.1276
14
80. 45%
71.869
22 2300
76, 1000
69 20000
96 9579
109,999
122, 0
124.919 81.
57. 61.61%
82. 1573
61.27%
169.488%
119. 51%
165. 84%
120. 9499
100. 54
87. 87.55%
175. 197
1959. 1966
116. 755%
128. 244%
178. 0.5%
165. 71.
209. 716
88 1996
64 166699
68.9
101
111. 7899
900%
98.200
96
61 141
149-324
50 36%
92 93%
TLE
51.80%
German
92.08%
64%
96.819%
122.34 9
194, 60%
144. 124.97%
1.90 447
927.0555
136. 066.899
196, 7
92.01% %
Gcloads
Splice
77. 77.9836
61.60%
147.19%
108. 1.230
63. 63.36%
79 25%
101. 14
166. 61%
147.55%
164.52% 091.5555566636
194, 99%
488. 1794
144. 6129
97. 14V4
149.
181
65%
06.2980
77
66
77.97%
98.029 1.
101 14%
96.84%
64.59%
84.31%
Gdoodab
46. 41%6
105, 36
48 277
84 1
65
119.
119.68% 68%
168
588%
47.57%
123. 129
1922 9.
03%
Chederhourd
80.4236
73966
70%
184. 184.0555
1798
10G. 199
141.992
Spsubaae
Ramana
65.459%
22. 54596666
1366
109. 3196
191.01% 47.70%
19.14%
94 565
122. 51%
132. 9156
83.74%
132. 64%
2822 8019
4 0114
966
96.355%
691 691.08%
21.76%
56.67%
TLE
194.
194.17%
104,
81%
52
25. 40%
Phonetae
30.787%
102.9917
33. 9699
68.401% 401%
100.165 1695
72.78%
116. 8299
92, 92.83%
107.11%
34.62%%
87.11%
83.245% -
30. 12%
Ringnorn
27.9556
134.897%
449%
(42. 9019
866. 1495
0899
914.91%
36.
TLE T
208. 100
194-483
41%
Thonorm:
Phishing
54.1176
18.389
90.133%
115.07%
42.611%
20.2495
285. 71.
102. 50% 20%
142. 0798
114, 8955
97.522% -
98 98%
9/22. 6299
TLE
TLE
151.29 251.26%
724%
20.96%
3R. 58.00%%
TLE
TLE
59.511
184.42%
22.77%
68.
Covertype
45.700%
TLE
46.787%
TLE
116.98%
TLE
TLE
TLE
217.86%
TLE
TLE
TIE
TLE
Biorespanae
70.869
TLE
72.539
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
Pol
17.78%
TLE
17.21%
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
TLE
Table 29: The computational time of a query strategy (column) on a dataset (row) with format 'min- utes:soconds! 'TLF' denotes a query strategy that exceeds the time limit.
Uniforn
QBC
Hire
Grapia
Core-See
QUIRE
DWUS
MCM
BMDR
ALDL.
LAG
Aspendittii
0u.20.500 50de
0mi.290.
Bmi2 G81s
Rud aots
224-1122
TLE
0m422.372
Dual2 175s
Sunar
Bna 22
0m 75%x
0m100.20 21
0es 0000
(nd 278
6m5.851
0mo 22.
Imi 855.
Bnd 16
fimt 158x
Parktoons
that.
tma1 1046
0m422.01 M15e
147s
0m447 707s
mads 2016
cm2 5565
12m
East:
0u2.000
0use1.20 26
50.0
tas 13
Imi7
Rnd 505s
062.6665
12u48 st0.
0744
BudT
Hoart
Unat. 191>
0m 071
00 790k
0et4 2086
ond 198
Gmb 9524
0ms Til.
Switt 6104
trat 185e
0m 151
Stmd 57%
0mt0 201-
12n Malo
Hoberman
0447
Ruzo
5mi4 780s
0aa4 usts
41224125
15m44
0m40
7aal7
lomoplice
0us3.012
Tmii 700.
1157.515
50a20
12m 246e
Clean!
21
0m31
3016
0m115.776
3m23 517
23054.5716
Beasst
0uad,575
0u152.967
1m15.005
0m17.907
40u7 140s
1m030.344
00m28.
Wille
0ra1.507x
0m155.070
1n 00
0un 125
0m155.07>
0m22.671
S1m 17tx
0m19 230
100m17.124
S
Austrolian
tta7.00 2006
1m 5525
1m33. S15e
2042 3016
0t114.00 86
2m225574
0m26-101
50m11 sote
teall 1825
0m15 00 is
189m 1275
1m 1854
52m11
Diabstes
1mf2. 502-
0u116.178
0m200
0m39.770
6öm25 .02s
RaG 0333
0u200.00 Do
112m52.504
1m40..90
Manmographie
Brdd
1m0.024
1mt4 56%
twl 753
(1117.1776
0m20) 8005
0m301
simbe 13tx
(ra a/2x
(m21.
1i:
61m52 790k
1m52 301-
5om17 473e
Easo
Baad 2015
1m20 227
2u55 8966
0tu14 Titl
Gm221 1825
0m243874s
63m33.
7345 7346
0m177
32%
TLE
1m35.725
53m10.54
Tir
0ta12.1844 182x
1m14.257
2nb 724+
4u:00 65
0u22 801x
0uff7 GuS1.253.
106ml 296w
1m56. 145x
TLE
2m:66.30
8im22 dels.
German
Unil11 1226s
4m36.813
1mi
1m.5
130m31 Jsst
2m 5356
0mb1. 0ill
dm
88m12 Sits
Splice
0ast0.100:
Su21,511
fautt.733 11a:001150s
0u41. 197t
2m2.672
6m32.2675
11an2 (12:
tuatt 22th
1m21.
8m29..50
85m50,
Gelenil
1m11.800
187x
(1121.119e
6m31 187.
0m31 ass.
16n13 sise
(n122.9611
63
Swr21 185-
Alm1l. (w7s
Gelondub
Ims 108s
Im47.
0m31 2316
15m7 ssis
1936
0m2558411
85mal
3m018.005
Sran22 M4
Chuckerbourd
0ud22.076a
1m228.007
Limi. 470b
1u7.754a
Im16,älis
92m10, 770
lafi 420a
1m209.77
22fm188.300
4u122.000
19lud0 450a
41

---

Published in Transactions on Machine Learning Research (06/2025)
LR(C-1)
SVM(RBF)
LR(C-1)
SVM(RBF)
LR(C-1)
84.61%
84.25%
84.16%
LR(C-1)
90.67%
89.20%
86.75%
SVM(RBF)-
84.21%
84.31%
84.15%
SVM(RBF)
90.56%
89.83%
86.92%
RF
84.49%
84.31%
84.34%
90.32%
89.65%
87.09%
LR(C-1)
SVM(RBF)
RF
LR(C-1)
SVM(RBF)
LR(C=1)
73.99%
73.50%
70.19%
LR(C=1)-
97.03%
95.81%
95.24%
SVM(RBF)
73.46%
73.13%
70.28%
SVM(RBF)
96.98%
96.54%
95.18%
RF
73.48%
73.21%
70.69%
RF 96.92%
96.22%
95.40%
Figure 13: Mean AUBC of query-oriented model and task-oriented model on group 1. (Compatible LRs achieve best results.): Appendicitis (top-left), ExSb (top-right), Habermon (bottom-left), and Wdbe (bottom- right).
42

---

Published in Transactions on Machine Learning Research (06/2025)
LR(C-1)
SVM(RBF)
LR(C-1)
SVM(RBF)
LR(C-1)
76.06%
74.18%
74.23%
LR(C-1)
82.87%
81.17%
79.84%
SVM(RBF)
75.92%
74.80%
74.42%
SVM(RBF)
82.23%
81.68%
80.18%
RF
75.98%
74.50%
74.74%
82.22%
81.45%
80.72%
LR(C-1)
SVM(RBF)
RF
LR(C-1)
SVM(RBF)
LR(C=1)
95.69%
94.13%
93.55%
LR(C=1)-
97.70%
97.55%
96.13%
SVM(RBF)
95.38%
95.61%
94.68%
SVM(RBF)
97.70%
97.66%
96.44%
RF
95.42%
95.08%
95.29%
RF 97.70%
97.66%
96.88%
Figure 14: Mean AUBC of query-oriented model and task-oriented model on group 1. (Compatible LRs achieve best results.): Diabetes (top-left), Mammographic (top-right), Geloudub (bottom-left), and Twonorm (bottom-right).
43

---

Published in Transactions on Machine Learning Research (06/2025)
LR(C=1)
SVM(RBF)
RF
LR(C=1)
SVM(RBF)
RF
LR(C=1)
74.54%
15.82%
76.23%
UR(C=1)
85.94%
91.60%
89.97%
SVM(RBF)
73.74%
77.68%
76.79%
SVM(RBF)-
86.10%
92.95%
91.34%
73.71%
76.91%
76.99%
86.30%
92.68%
92.04%
LR(C=1)
SVM(RBF)
LR(C=1)
SVM(RBF)
LR(C=1)
79.49%
82.87%
80.72%
LR(C=1)-
86. 86.92%
87.04%
86.04%
SVM(RBF)
78.57%
84.28%
81.65%
SVM(RBF)-
83.29%
87.05%
85.38%
RF
78.75%
83.87%
82.10%
RF
84.77%
87.04%
85.80%
IR(C=1)
SVM(RRF)
RF
[R(C=1)
SVM(RRF)
RF
LR(C=1)
87.91%
88.55%
78.72%
UR(C=1)
75.80%
97.66%
94.42%
SVM/RBF)-
87.84%
89.86%
88.52%
SVM(RBF)-
74.99%
97.83%
94.54%
RT 87.58%
89.43%
88.71%
RF 68.33%
97.80%
95.62%
Figure 15: Mean AUBC of query-oriented model and task-oriented model on group 2. (Compatible SVMs achieve best results.): Sonar (top-left), Jonosphere (top-right), Clean1 (middlc-left), Tic (middle-left). Goloudb (bottom-left), and Ringnorm (bottom-right).
44

---

Published in Transactions on Machine Learning Research (06/2025)
LR(C-1)
SVM(RBF)
LR(C-1)
SVM(RBF)
LR(C-1)
83.66%
84.67%
84.46%
LR(C-1)
96.40%
96.28%
96.75%
SVM(RBF)-
84.40%
85.55%
86.03%
SVM(RBF)-
96.43%
96.30%
96.75%
RF
82.59%
84.91%
86.89%
96.41%
96.2 96.23%
96.84%
LR(C-1)
SVM(RBF)
RF
LR(C-1)
SVM(RBF)
LR(C=1)
85.42%
84.50%
85.81%
LR(C=1)-
68.89%
82.51%
89.06%
SVM(RBF)
85.42%
84.92%
85.87%
SVM(RBF)
67.89%
87.94%
92.80%
RF
85.41%
84.72%
86.15%
RF 67.33%
86.23%
95.46%
Figure 16: Mean AUBC of query-oriented model and task-oriented model on group 3. (Compatible RFs achieve best results.): Parkinsons (top-left), Breast (top-right), Australian (bottom-left), and ExSa (bottom- right).
45

---

Published in Transactions on Machine Learning Research (06/2025)
LR(C-1)
SVM(RBF)
LR(C-1)
SVM(RBF)
LR(C-1)
74.49%
73.94%
74.40%
LR(C-1)
92.24%
90.37%
93.58%
SVM(RBF)
74.62%
74.17%
74.59%
SVM(RBF)-
91.70%
92.15%
93.98%
RF
74.49%
74.09%
74.76%
91.20%
91.34%
94.73%
LR(C-1)
SVM(RBF)
RF
LR(C-1)
SVM(RBF)
LR(C=1)-
75.66%
82.33%
86.99%
LR(C=1)-
92.79%
93.80%
95.87%
SVM(RBF)
71.61%
83.37%
88.02%
SVM(RBF)
92.68%
94.68%
96.52%
RF 73.25%
82.93%
88.71%
RF 92.57%
94.39%
96.73%
Figure 17: Mean AUBC of query-oriented model and task-oriented model on group 3. (Compatible RFs achieve best results.): German (top-left), Spambase (top-right), Phoneme (bottom-left), and Phishing (bottom-right).
46

---

Published in Transactions on Machine Learning Research (06/2025)
LR(C-1)
SVM(RBF)
LR(C-1)
SVM(RBF)
LR(C-1)
81.65%
80.98%
80.30%
LR(C-1)
76.94%
81.25%
91.36%
SVM(RBF)-
81.74%
81.63%
80.67%
SVM(RBF)-
76.76%
82.30%
91.81%
RF
81.49%
81.21%
80.66%
76.21%
81.59%
91.50%
LR(C-1)
SVM(RBF)
RF
LR(C-1)
SVM(RBF)
LR(C=1)
76.09%
90.61%
95.61%
LR(C=1)-
56.85%
78.98%
79.69%
SVM(RBF)
67.80%
98.52%
99.50%
SVM(RBF)
55.97%
88.25%
87.10%
RF 60.94%
97.15%
99.45%
RF 53.09%
89.34%
88.74%
Figure 18: Mean AUBC of query-oriented model and task-oriented model on group 5. -Non-Compatible models achieve best results.): Heart (top-left), Splice (top-right), Checkerboard (bottom-left), and Banana (bottom-right).
47


Reviews:
[{"review_id": "D24cGXu5As", "reviewer": "Reviewer_6RaL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper's main strength and contribution is the comprehensiveness of the studied datasets (Table 3) and the various algorithmic & data characteristics (Table 1). It also performs a fairly comprehensive review of the novel active learning approaches proposed over the last decade.\n\nThe paper has two main weaknesses: the experimental setup and the presentation. \n\nWith respect  to the experimental setup, in order to correctly assess an active learner's impact on a given dataset, one most quantify how much cheaper (i.e., fewer labeled examples) it is to reach (near) SOTA performance compared to the uniform sampling scenario. For example, rather than using a fixed querying budget (e.g., 20% or 2K or 3K of the examples) that leads to a (possibly) mediocre performance, use as metric \"the number of examples required to reach 90%, 95%, and 99% of the performance of training on all the available data.\" The values in Tables 3 & 4 show that US often outperforms other methods on the chosen budgets, but it does compare this performance against the  SOTA results when training on all labeled data. \n\nWith respect to the actual presentation, it could easily be improved to make the paper more readable:\n- the abstract uses the query/task-oriented  terminology that is only introduced in section 2.2\n- make it clear both in the abstract & the intro that your focus is on binary clarification; right now, only in section 3 (2nd pgf) you make it clear that this is the case\n- in Section 2.1, the Model uncertainty part is presented fairly well, while the other approaches (Expected model changes, Representation exploiting, and Hybrid) are a lot vaguer and more poorly organized. Please add references to the \"Expected model changes\" paragraph, and explain what do you mean by model-free. Ideally, you should add a new sub-section with an illustrative example that INTUITIVELY compares & contrasts the 4 main types of approaches and their main variants.\n- the second paragraph in 2.2 is poorly written (\"might or might not be\"), and too complicated; a simpler, intuitive 1-sentence version should be part of your abstract\n- please add comprehensive, intuitive discussion on the impact of the initial labeled pool. It seems to be a key issue, for which you are simplify referring the reader to a 2022 paper. Depending on whether you solved or just alleviated this issue, this part may be a major contribution or, alternatively, red flag\n- in Table 5, US performs poorly on Chekerboard and Banana. Please analyze and discuss in detail the root causes\n- in all tables, make it easy to find the top-2 values (eg, bold & italic, respectively) \n\n\n\n \nOTHER COMMENTS\n- explain in detail and add references to the vague statement \"However, these strategies often lack a fair and unified comparison across different\ncontexts.\"\n- avoid weak statements such as the \"could be\" - as in \"Our benchmark could be the most comprehensive ...\" Either it is, or it is not.\n- replace the weak statement \"US remains competitive on most datasets\" by \"US is SOTA on X of the 26 studied datasets\n- you are repeatedly using incorrectly the expression \"first-hand\"\n- Settles' literature review of 2009 did not \"initiate\"  the study of pool-based AL, which started a decade earlier, in the 1990s\n- Fig 2 - add a reference to the Australian dataset\n- define the \"cold start\" issue from section 3\n- Section 4,1: \"statistically exist\" --> - define \"statistically significant\"\n- the caption of Table 3 refers itself", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a comprehensive study of various state of the art (SOTA) active learning methods for binary classification on tabular data. It also high-lights the importance of model compatibility to uncertainty-sampling outperforming the other SOTA active learning algorithms.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see above", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5fnp483ALu", "reviewer": "Reviewer_9t1Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### **Strengths**  \n\n**Comprehensive and Rigorous Benchmark**  \n- The paper presents one of the most extensive Active Learning (AL) benchmarks to date, surpassing prior works in:\n  - Dataset variety: 29 datasets covering different domains and scales\n  - Model diversity: Including Logistic Regression (LR), RBF Support Vector Machines (RBFSVM), Random Forests (RF), and Gradient Boosting Decision Trees (GBDT)\n- Query strategy coverage: Evaluating 12 different AL strategies systematically  \n- It systematically evaluates 12 AL strategies across 29 datasets, ensuring broad applicability.  \n- The benchmark includes standardized protocols and open-source tools, which enhance reproducibility and facilitate further research.  \n\n**Resolution of Conflicting Findings in AL Research**  \n\n- The authors identify and address a key flaw in previous benchmarks: model incompatibility between query selection and task-oriented models. Specifically, they distinguish between:\n  - Query-oriented models (H): Used to select which examples to label\n  - Task-oriented models (G): Used to make final predictions on test data\n\nBy controlling for this compatibility factor and showing that mismatches between H and G can significantly degrade performance, they provide a more accurate assessment of Uncertainty Sampling (US) and clarify why prior studies reached contradictory conclusions.\n\n\n**Empirical and Statistical Rigor**  \n- The study employs multiple evaluation metrics and thorough statistical analysis:\n  - Comprehensive performance comparisons across datasets (Tables 3-4)\n  - Detailed ranking analysis showing strategy performance (Table 5) \n  - Data utilization rate analysis (Table 6)\n- Statistical significance testing using Friedman tests (p < 0.05) to validate that performance differences between strategies are not due to chance\n- The study employs multiple evaluation metrics, including Area Under the Budget Curve (AUBC) and average ranking statistics.  \n- It includes **statistical significance testing (e.g., Friedman tests)** to validate claims.  \n- The benchmark results are thoroughly documented, with clear performance trends across datasets.  \n\n**Practical Implications**  \n- The findings provide actionable insights for practitioners, advocating for Uncertainty Sampling (US) as the default choice when paired with a compatible model.  \n- The benchmark offers a transparent, standardized framework that can serve as a foundation for future AL research and applications.  \n\n---\n\n### **Weaknesses**  \n\n**Exclusion of Deep Learning-Based Active Learning**  \n- The benchmark is **limited to traditional machine learning models** (e.g., Logistic Regression, SVMs, Random Forests).  \n- Modern AL techniques for deep learning models (e.g., CNNs, Transformers) are not included, making the conclusions less applicable.  \n\n**Lack of Computational Efficiency Analysis**  \n- The paper does not analyze the runtime or computational cost of different AL strategies.  \n- This omission is critical for practitioners deciding between accuracy gains and computational feasibility.  \n\n**Limited Exploration of Real-World Challenges**  \n- The benchmark focuses primarily on tabular datasets and does not address challenges like imbalanced data distributions. \n- Expanding to these areas would make the benchmark more representative of real-world AL deployment scenarios.  \n\n*Over-Reliance on AUBC as the Primary Metric**  \n- AUBC provides a high-level summary but does not capture finer details of AL performance, such as:  \n  - Early-stage performance (some strategies may be better in the first few queries but degrade later).  \n  - Budget-limited scenarios (e.g., when the labeling budget is very small).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an expanded benchmark for Active Learning (AL) that aims to resolve conflicting conclusions in previous AL benchmarks regarding the effectiveness of Uncertainty Sampling (US). The primary contributions of the paper are as follows:\n\n**Comprehensive AL Benchmark**\n- The authors develop the most extensive AL benchmark to date, covering 29 datasets, 12 query strategies, and multiple base models (Logistic Regression, SVMs, Random Forest, and Gradient Boosting Decision Trees).\n- The benchmark surpasses Yang & Loog (2018) and Zhan et al. (2021) by offering a broader evaluation framework, including new datasets, models, and query strategies.\n- The authors provide a transparent, open-source implementation to ensure reproducibility and enable further research.\n\n**Findings on Uncertainty Sampling (US) Performance**\n- The study provides clear guidelines for AL practitioners, recommending US as the default choice when paired with a compatible model.\n\n**Methodological Contributions**\n- The paper introduces a new experimental protocol for benchmarking AL strategies fairly, ensuring that query selection and model evaluation are properly aligned.\n- The authors analyze AL performance using AUBC (Area Under the Budget Curve), average ranking statistics, and accuracy comparisons at different query budgets.\n\nThis work improves the empirical evaluation of AL strategies by addressing benchmarking inconsistencies, introducing a more rigorous experimental protocol, and providing open-source tools. The insights on model compatibility have major implications for how AL strategies should be compared and deployed in practice.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical Revisions (Necessary for Acceptance)**\n\n- Clarify the Scope of the Benchmark in the Title and Abstract: The current title and abstract suggest that the benchmark is broadly applicable to active learning, but the study is limited to binary classification, tabular data and classical ML models. This can lead to misinterpretation of its generalizability.\n\n**Recommended Improvements (Not Essential for Acceptance, but Strengthen the Paper)**\n- Expand the Literature Review to Include Deep Learning-Based AL Methods\n- Include Computational Efficiency Comparisons\n- Consider Evaluating AL on More Challenging Scenarios (Imbalanced datasets)", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IzFJrLimEI", "reviewer": "Reviewer_V5HM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. **Comprehensive Benchmark Design**:\n   - The paper offers a **well-structured, comprehensive benchmark**, covering a wide range of datasets, query strategies, and base models. This extensive coverage sets a **new standard for evaluating Active Learning (AL)** techniques.\n   - The **transparent and reproducible** open-source framework is a significant contribution, addressing a major gap in previous AL research by enabling standardized comparisons across different settings.\n\n2. **Clarification of Model Compatibility**:\n   - The authors provide an insightful analysis on the importance of **model compatibility** between query-oriented and task-oriented models, introducing the concepts of **US-Compatible (US-C)** and **US-Non-Compatible (US-NC)**.\n   - This analysis resolves conflicting conclusions in prior benchmarks and offers a **clear and practical recommendation** for practitioners, enhancing the understanding of the conditions under which **Uncertainty Sampling (US)** excels.\n\n3. **Robust Experimental Protocol**:\n   - The submission features a **well-defined experimental protocol**, including detailed preprocessing steps, dataset configurations, and evaluation metrics such as **Area Under the Budget Curve (AUBC)**.\n   - The use of multiple **base models** (e.g., Logistic Regression, RBFSVM, Random Forest, XGBoost) demonstrates the generalizability of the benchmark results across different machine learning paradigms.\n\n4. **Practical Relevance and Extensibility**:\n   - The findings are **highly relevant for real-world applications**, providing actionable insights for AL practitioners on when and how to apply **Uncertainty Sampling (US)** effectively.\n   - The open-source nature of the benchmark and its **modular design** make it easy to extend, encouraging further exploration in areas like **deep learning models** and **domain-specific AL tasks** (e.g., vision, language).\n\n5. **High-Quality Analysis and Visualization**:\n   - The paper includes a thorough analysis of the benchmarking results, with **clear visualizations** (e.g., learning curves, mean difference plots) that effectively convey the performance trends of different AL strategies.\n   - The use of **statistical tests**, such as the Friedman test, adds rigor to the performance comparisons, providing evidence for the significance of the results.\n\n### Weaker Elements and Areas for Improvement\n\n1. **Limited Scope of Datasets**:\n   - Although the benchmark includes a variety of tabular datasets, it **lacks diversity** in terms of **domain-specific datasets** (e.g., text, image data). Incorporating **deep learning scenarios** and **more complex data types** could strengthen the generalizability of the findings.\n   - The focus on **binary classification** may limit the applicability of the benchmark to more **complex, multi-class** or **imbalanced** dataset scenarios.\n\n2. **Dependency on Traditional Machine Learning Models**:\n   - The analysis heavily relies on **traditional machine learning models** (e.g., Logistic Regression, Random Forest). While these models are effective for tabular data, the relevance of the results to **deep learning models** used in modern applications (e.g., neural networks in NLP or CV) remains unexplored.\n   - The paper could benefit from evaluating AL strategies on **deep neural networks**, especially given the increasing adoption of these models in industry and academia.\n\n3. **Overemphasis on Uncertainty Sampling**:\n   - While the focus on **Uncertainty Sampling (US)** is justified by its widespread use, the submission might have **overemphasized its advantages**. Additional analysis comparing **state-of-the-art AL strategies** (e.g., BADGE, LPL, WAAL) across different model types could provide a more balanced view.\n   - The paper could include a discussion on the **limitations** of US, such as its potential biases and sensitivity to **outliers**, and suggest scenarios where alternative strategies might be more effective.\n\n4. **Limited Exploration of Hyperparameter Sensitivity**:\n   - The benchmark does not thoroughly address the **impact of hyperparameter choices** on the performance of different query strategies and base models. Given that hyperparameters can significantly influence AL outcomes, this could be an important area for further investigation.\n   - Including an analysis of **sensitivity to key hyperparameters** (e.g., query batch size, model complexity) would add depth to the findings and help practitioners better understand the robustness of each strategy.\n\n5. **Potential Issues with Reproducibility**:\n   - While the framework aims for reproducibility, the **complexity of the experimental setup** (e.g., model compatibility settings, preprocessing choices) might present challenges for replication by other researchers.\n   - Providing **detailed documentation** and **examples** for setting up the experiments, as well as sharing **pre-trained models** and configurations, could help address this concern.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a comprehensive review and benchmarking of **Active Learning (AL)** strategies, focusing on the widely used **Uncertainty Sampling (US)** method. The key contributions and new knowledge introduced by the paper include:\n\n1. **Expanded Open-Source Benchmark**:\n   - The authors present the **most comprehensive open-source AL benchmark** to date, surpassing previous benchmarks in terms of the number of datasets, models, query strategies, and analysis tools.\n   - The benchmark unifies previous comparisons by integrating a **transparent and consistent experimental protocol**, allowing for **reproducible evaluation** across various AL strategies.\n\n2. **Uncovering the Role of Model Compatibility**:\n   - The study investigates the often-overlooked issue of **model compatibility** between the query-oriented model (used for selecting samples) and the task-oriented model (used for predictions).\n   - The authors demonstrate that **Uncertainty Sampling (US)** remains competitive when paired with compatible models, clarifying conflicting conclusions found in previous benchmarks.\n   - The paper introduces the terms **US-Compatible (US-C)** and **US-Non-Compatible (US-NC)** to differentiate between these scenarios, showing that **US-C** consistently outperforms **US-NC**.\n\n3. **Comprehensive Analysis of AL Strategies**:\n   - The benchmark includes a broader range of AL query strategies (e.g., **US**, **QBC**, **BALD**, **Core-Set**), covering **traditional machine learning models** (Logistic Regression, RBFSVM, Random Forest) and **tree-based models** (XGBoost).\n   - The analysis highlights the **robust performance of US**, especially in scenarios involving **binary classification on tabular datasets**, with US achieving the highest **Area Under the Budget Curve (AUBC)** on most datasets.\n\n4. **New Insights and Practical Recommendations**:\n   - The study provides fresh insights into the **effectiveness and limitations** of AL strategies, demonstrating that US is not only a strong baseline but also a preferred choice when used with compatible settings.\n   - The findings suggest practical guidelines for AL practitioners, emphasizing the importance of model compatibility and offering a **clear recipe for selecting AL strategies** in real-world applications.\n\n5. **Open-Source Framework for Future Research**:\n   - The benchmark framework is designed to be extensible, allowing future research to include **diverse domains** (e.g., vision, language) and **advanced models** (e.g., deep neural networks).\n   - The open-source nature of the benchmark facilitates collaboration and standardizes the evaluation of AL strategies, addressing the **need for transparent and fair comparisons**.\n\nIn summary, the paper reaffirms the competitiveness of Uncertainty Sampling, resolves inconsistencies in previous benchmarks, and establishes a strong foundation for future studies in Active Learning. The new benchmark provides a **practical, reproducible, and comprehensive tool** for evaluating and selecting AL strategies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical Adjustments**\n\n1. **Expand the Scope of Datasets**:\n   - **Issue**: The current benchmark predominantly focuses on tabular datasets and binary classification tasks.\n   - **Proposed Adjustment**: Extend the evaluation to include more diverse datasets, such as **multi-class**, **imbalanced datasets**, and **domain-specific data** (e.g., text, image data). This will demonstrate the **generalizability** of the benchmark across a broader range of applications and enhance the validity of the findings.\n   - **Justification**: Expanding the dataset scope is crucial for showcasing the real-world applicability of the proposed benchmark and avoiding limitations in generalizability.\n\n2. **Include Analysis with Deep Learning Models**:\n   - **Issue**: The benchmark relies heavily on traditional machine learning models like Logistic Regression and Random Forest, which may not reflect modern usage trends in machine learning.\n   - **Proposed Adjustment**: Incorporate **deep learning models**, such as neural networks used in NLP (e.g., BERT, LSTM) or computer vision (e.g., ResNet, CNNs), to evaluate the performance of Active Learning (AL) strategies in **deep learning scenarios**.\n   - **Justification**: This addition is critical for aligning the benchmark with current industry and research practices, where deep learning models are widely adopted. It will also address a gap in the analysis and increase the impact of the work.\n\n3. **Provide Detailed Hyperparameter Analysis**:\n   - **Issue**: The current submission does not address the impact of **hyperparameters** on the performance of AL strategies.\n   - **Proposed Adjustment**: Include an analysis of **hyperparameter sensitivity**, particularly focusing on key parameters like **query batch size**, **model complexity**, and **sampling thresholds**. This could involve experiments where these hyperparameters are systematically varied to observe their effects.\n   - **Justification**: Hyperparameter choices can significantly affect the outcomes of AL experiments. Addressing this sensitivity is critical for ensuring the robustness of the benchmark results and guiding practitioners in setting up their experiments.\n\n4. **Clarify the Limitations of Uncertainty Sampling**:\n   - **Issue**: The paper heavily emphasizes the advantages of Uncertainty Sampling (US) without adequately discussing its **limitations**.\n   - **Proposed Adjustment**: Provide a section that explicitly outlines the **potential drawbacks** of US, such as sensitivity to **outliers**, sampling bias, and limitations in cases of **data drift** or **distribution shifts**.\n   - **Justification**: A balanced discussion of the strengths and weaknesses of US is essential for a fair evaluation of AL strategies and for helping readers understand scenarios where alternative methods might be more suitable.\n\n**Enhancements That Would Strengthen the Work**\n\n5. **Improve Documentation and Reproducibility**:\n   - **Issue**: The complexity of the experimental setup, including preprocessing and model compatibility settings, may pose challenges for other researchers attempting to reproduce the results.\n   - **Proposed Adjustment**: Provide **detailed documentation** and **tutorials**, including clear instructions for setting up experiments, code examples, and a description of key dependencies. Consider releasing **pre-trained models** and **configuration files** for ease of replication.\n   - **Justification**: While not critical for acceptance, this improvement would significantly enhance the usability and impact of the benchmark, making it a more valuable resource for the community.\n\n6. **Incorporate State-of-the-Art AL Strategies**:\n   - **Issue**: The benchmark includes many well-known AL strategies but lacks a comparison with some **recent advancements**, such as BADGE, LPL, and WAAL, which have shown promising results in specific scenarios.\n   - **Proposed Adjustment**: Extend the benchmark to include a comparison with these **state-of-the-art AL methods**, particularly focusing on their performance in scenarios where traditional methods like US might fail (e.g., high-dimensional feature spaces, non-i.i.d data).\n   - **Justification**: This adjustment would provide a more comprehensive evaluation and ensure the benchmark remains relevant as the field of AL evolves.\n\n7. **Expand Discussion on Broader Impacts and Future Work**:\n   - **Issue**: The current broader impact statement focuses primarily on the benefits to the AL community but does not address potential risks or ethical considerations.\n   - **Proposed Adjustment**: Include a more thorough discussion on the **ethical implications** of using AL strategies, particularly in sensitive applications (e.g., medical data annotation, legal text analysis). Additionally, provide more specific directions for **future work**, such as evaluating AL strategies in **low-resource settings** or integrating with **semi-supervised learning** approaches.\n   - **Justification**: This enhancement would strengthen the broader impact statement and demonstrate a proactive approach to ethical considerations, which is increasingly valued in machine learning research.\n\n8. **Visualize Model Compatibility Impact More Clearly**:\n   - **Issue**: While the paper discusses the importance of model compatibility, the visualizations (e.g., decision boundary illustrations) could be made clearer.\n   - **Proposed Adjustment**: Improve the **quality and clarity of visual aids**, using well-labeled plots that highlight the differences in performance between US-Compatible (US-C) and US-Non-Compatible (US-NC) settings across various datasets and models.\n   - **Justification**: Enhanced visual representations would make the key insights more accessible and help reinforce the importance of model compatibility in a more digestible format.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I recommend accepting this paper based on the thorough reviews, the authors' comprehensive responses, and the revisions to the paper.

All three reviewers have provided positive recommendations, and the authors have addressed the concerns raised during the review process.

I want to thank the authors and reviewers for their patience as this paper went through the review process. I'm very glad to note the positive interactions that have improved the paper towards acceptance.
Decision: Accept (poster)