Is margin all you need? An extensive empirical study of active learning on tabular data
Authors: Dara Bahri
Heinrich Jiang
Tal Schuster
Afshin Rostamizadeh
Venue: Submitted to ICLR 2023 (2023)

Abstract: Given a labeled training set and a collection of unlabeled data, the goal of active learning (AL) is to identify the best unlabeled points to label. In this comprehensive study, we analyze the performance of a variety of AL algorithms on deep neural networks trained on 69 real-world tabular classification datasets from the OpenML-CC18 benchmark. We consider different data regimes and the effect of self-supervised model pre-training. Surprisingly, we find that the classical margin sampling technique matches or outperforms all others, including current state-of-art, in a wide range of experimental settings. To researchers, we hope to encourage rigorous benchmarking against margin, and to practitioners facing tabular data labeling constraints that hyper-parameter-free margin may often be all they need.

Under review as a conference paper at ICLR 2023
IS MARGIN ALL YOU NEED? AN EXTENSIVE EMPIRI- CAL STUDY OF DEEP ACTIVE LEARNING ON TABULAR DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Given labeled training set and a collection of unlabeled data, the goal of active learning (AL) to identify the best unlabeled points to label. In this compre- hensive study, we analyze the performance of a variety of AL algorithms on deep neural networks trained on 69 real-world tabular classification datasets from the OpenML-CC18 benchmark. We consider different data regimes and the effect of self-supervised model pre-training. Surprisingly. we find that the classical margin sampling technique matches outperforms all others, including current state-of- art, in a wide range of experimental settings. To researchers, we hope to encourage rigorous benchmarking against margin, and to practitioners facing tabular data la- beling constraints that ryper-parameter-Free margin may often be all they need.
INTRODUCTION
Active learning (AL.), the problem of identifying examples to label, is an important problem in machine learning since obtaining labels for data is oftentimes a costly manual process. Being able to efficiently select which points to label can reduce the cost of model learning tremendously. High- quality data is key component in any machine learning system and has a very large influence on the results of that system (Cortes et al., 1994; Gudivada et al., 2017; Willemink et al., 2020); thus, improving data curation can potentially be fruitful for the entire ML pipeline.
Margin sampling, also referred to as uncertainty sampling (Lewis et al., 1996; MacKay, 1992), is classical active learning technique that chooses the classifier's most uncertain examples label. In the context of modern deep neural networks, the margin method scores each example by the differ- ence between the top two confidence (e.g. softmax) scores the model's prediction. In practical and industrial settings, margin is used extensively in wide range of areas including computational drug discovery (Reker Schneider, 2015; Warmuth et al., 2001), magnetic resonance imaging (Liebgott et al., 2016), named entity recognition (Shen et al., 2017), well predictive models for weather (Chen et al., 2012), autonomous driving (Hussein et al., 2016), network traffic (Shahraki et al., 2021), and financial fraud prediction (Karlos et al., 2017).
Since the margin sampling method is very simple, it seems particularly appealing to try to modify and improve on it, or even develop more complex AL methods to replace Indeed, many papers in the literature have proposed such methods that, at least in the particular settings considered, consistently outperform margin. In this paper, we put this intuition the test by doing head- to-head comparison of margin with number of recently proposed state-of-the-an active learning methods across variety of tabular datasets. We show that in the end. margin matches or outperforms all other methods consistently in almost all situations. Thus, our results suggest that practitioners of active learning working with tabular datasets, similar to the ones we consider here, should keep things simple and stick to the good old margin method.
In many previous AL studies, the improvements over margin are oftentimes only in settings that are not representative of all practical use cases. One such scenario is the large-batch case, where the number of examples to be labeled at once large. It is often argued that margin is not the optimal strategy in this situation because it exhausts the labeling budget very narrow set of points close to decision boundary of the model and introducing more diversity would have helped (Huo & Tang, 2014; Sener& & Savarese, 2017; Cai et al., 2021). However, some studies find that the number of

---

Under review as a conference paper at ICLR 2023
examples to be labeled at once has to be very high before there is advantage over margin (Jiang & Gupta, 2021) and in practice a large batch of examples usually does not need to be labeled at once, and it is to the learners' advantage to smaller batch sizes so that datapoints get labeled, such information can be incorporated to re-train the model and thus choose the next examples in a more informed way. It is important to point out, however, that in some cases, re-training the model is very costly (Citovsky et al., 2021). In that case, gathering a larger batch could be beneficial. In this study, we focus on the practically more common setting of AL that allows frequent retraining of the model.
Many papers also only restrict their study to only a couple of benchmark datasets and while such proposals may outperform margin, these results don't necessarily carry over to a broader set of datasets and thus such studies may have the unintended consequence of overfitting to the dataset.
In the real world live active learning setting, examples are sent to human labelers and thus we don't have the luxury of comparing multiple active learning methods or even tuning the hyper- parameters of single method, without incurring significantly higher labeling cost. Instead, we have to commit to a single active learning method oftentimes without much information. Our results on the OpenML-CCI18 benchmark suggest that in almost all cases when training with tabular data, is safe for practitioners to commit to margin sampling (which comes with the welcome property of and have the peace of mind that other alternatives wouldn't t
not having additional have performed better in statistically significant way.
2 RELATED WORK
There have been a number of works in the literature providing empirical analysis of active learn- ing procedures in the context of tabular data. Schein & Ungar (2007) studies active learning pro- cedures for logistic regression and show that margin sampling performs most favorably. Ramirez- Loaiza et al. (2017) show that with simple datasets and models, margin sampling performs better compared to random and mnery-by-Commitee if accuracy is the metric, while they found random performs best under the AUC metric. Pereira-Santos et al. (2019) provides a investigation of the performance of active learning strategies with various models including SVMs, random forests, and nearest neighbors. They find that using margin with random forests was the strongest combination. Our study focuses on the accuracy metric and also shows that margin is the strongest baseline, but is much more relevant to the modern deep learning setting and with comparison to a much more expanded set of baselines and datasets. Our focus on neural networks is timely as recent work (Bahri et al., 2021) showed that neural networks often outperform traditional approaches for modeling tab- ular data, like Gradient Boosted Decision Trees (Chen & Guestrin, 2016), particularly when they are pre-trained in the way we explore here. To our knowledge we provide the most comprehensive and practically relevant empirical study of active learning baselines on neural networks thus far.
There have also been empirical evaluations of active learning procedures in the non-tabular case. Hu et al. (2021) showed that margin attained the best average performance of the baselines tested on two image and three text classification tasks across variety of neural network architectures and labeling budgets. Munjal et al. (2022) showed that on the image classification benchmarks CIFAR- 10, CIF0R-100, and ImageNet, under strong regularization, none of the numerous active learning baselines they tested had a meaningful advantage over random sampling. We hypothesize that this may be due the initial network having too little information (i.e. no pre-training and small initial seed set) for active learning to be effective and conclusions may be different otherwise. It is also worth noting that many active learning studies in computer vision only present results on few benchmark datasets (Munjal et al.. 2022; Sener & Savarese, 2017; Beluch et al., 2018; Emam et al., 2021: Mottaghi & Yeung, 2019; Hu et al., 2018), and while they may have promising results on such datasets, it's unclear how they translate to a wider set of computer vision datasets. We show that many of these ideas do not perform well when put to the test on our extensive tabular dataset setting. Dor et al. (2020) evaluated various active learning baselines for BERT and showed in most cases, margin provided the most statistically significant advantage over passive learning. One useful direction for future work is establishing an extensive empirical study for computer vision and NLP.
While our study is empirical, it is worth mentioning that despite being such a simple and classical baseline, margin is difficult to analyze theoretically and there remains little theoretical understanding of the method. Balcan et al. (2007) provides learning bounds for a modification of margin where ex- amples are labeled in batches where the batch sizes depend on predetermined thresholds and assume
2

---

Under review as a conference paper at ICLR 2023
that the data is drawn on the unit ball and the set of classifiers the linear separators the ball that pass through the center of the ball (i.e. no bias term). Wang & Singh (2016) provide a noise- adaptive extension under this style of analysis. Recently, Raj & Bach (2022) proposed a general family of margin-based active learning procedures for SGDD-bsee learners that comes with theoret- ical guarantees; however these algorithm require predetermined sampling function and hence, like the previous works, does not provide any guarantees for the classical margin procedure with fixed batch size. Without such theoretical understanding of popular active learning procedures, having comprehensive and robust empirical studies become even more important for our understanding of active learning.
3 PROBLEM STATEMENT
We begin with brief overview of pool-based active learning. One has initial sample of S labeledd training examples Tn = f( y:)} - where I; E RD and y; E N. We also assume a pool of unlabeled examples
== {z I PD that can be selected for labeling, for a grand total of Nioual 1, 2, .... T, select B examples (active learning batch size) in
points. The goal is for rounds each round from Z to be labeled and added to the train set I (and removed from Z) to produce the new labeled set I,
3.1 SCARF
Recently, Bahri al. (2021) proposed technique for pre-training deep networks on tabular datasets they term SCARE, Leveraging self-supervised contrastive learning in spirit similar to vision- oriented SimCLR (Chen et al., 2020), the method teaches the model embed an example z; and its corrupted view ã; closer together in space than I; and f. the corrupted view of different point zy- They show that pre-training boosts test accuracy on the same benchmark datasets we consider here even when the labels are noisy, and when labeled data limited and semi-supervised methods are used. We investigate the effect of SCARF pre-training on AL methods.
3.2 ACTIVE LEARNING BASELINES
Margin, Entropy, Least Confident. These three popular methods score candidates using the un- certainty of a single model, p. It is assumed that at round f, p is trained on the labeled set thus far to select the next batch of examples with highest uncertainty scores in 2. We seek points with the smallest margin, largest entropy, or largest least confident (LC) scores, defined as follows:
Margin(x) (x) = p(u = (x) p(y = (=()))a:), where
y1 (z) arg max c/z), 32(x)) == arg max p(y c/r)
c
Entropy (z)o : Σ ply = -| logp(y = clæ).
LC(x) := maxp(y=caa)
=
Random-Margin. A 50-50 mix of random and margin; common way to enhance diversity. Half of the batch chosen based on margin, and the other half of the examples are randomly selected.
Min-Margin (Jiang & Gupta, 2021). An extension of margin that uses bootstrapped models to increase the diversity of the chosen batch. K 25 models trained on bootstrap samples drawn from the active set (where the bootstrap is done on a per-class basis with the sample size the same the original training dataset size), and the minimum margin across the K models is used as the score.
Typical Clustering (TypiClust) (Hacohen et al., 2022). A method that uses self-supervised em- beddings to balance selection of "typical" or representative points with diverse ones as follows. At round f, all N pre-trained embeddings are clustered into II-1l + B clusters using K-means and then the most typical examples from the B largest uncovered clusters (i.e. clusters containing
3

---

Under review as a conference paper at ICLR 2023
no points from I;-1) are selected. Given that cosine similarity is the distance used when learning SCARF embeddings, we use spherical K K-means and define the typicality score as:
Typicality(s)
Σ
1 1 [-CooSim(r; J,)
K
(,E))((2)
I is chosen as min Noua, max(20) [x()))}, where [C(x)] is the size of the cluster containing æ.
Maximum Entropy (MaxEnt) and BALD. These Bayesian-based approaches use M models drawn from posterior. Oftentimes, Monte-Care dropout (MC-dropout) used, wherein a single model trained with dropout and then M different dropout masks are sampled and applied during inference (Gal et al., 2017). This can be seen as model inference using different models with weights {wm 1.M
For maximum entropy, we score using the model's entropy H:
H|gla] == -
M EEts ply ply= = cla, wm) log
ply cla, wm))
M
BALD (Houlsby et al., 2011) estimates the mutual information (MI) between the datapoints and the model weights, the idea being that points with large MI between the predicted label and weights have a larger impact on the trained model's performance. The measure, denoted I, is approximated as:
[[ya=] == [yla]]
M > Σ p(y = cla,, wm) log c/2, Wm)
We use M = 25 and dropout rate of 0.5 as done in prior work (Beluch et al., 2018). For consis- tency, dropout not applied during pre-training, only fine-tuning.
BADGE (Ash et al., 2019). Batch Active learning by Diverse Gradient Embeddings (BADGE) uses the loss gradient of the neural network's final dense layer for each unlabeled sample, where the loss is computed using the model's most likely label for the sample. The gradient embeddings are clustered using the K K-means++ seeding algorithm (Arthur & Vassilvitskii, 2006) and the centroids the samples added to the labeled set. The gradient embedding for a sample captures how uncer- tain the model is about the sample's label while the clustering provides diversity in sample selection. We use the sklearn. c uster. kmeans _p usspppp. us function with the default settings.
CoreSet (Sener & Savarese, 2017). Selects points to optimally cover the samples in embedding space. Specifically, each acquisition round, it grows the active set one sample time for B iterations. In each iteration, the candidate point x, that maximizes the distance between itself and its closest neighbor 2y in the current active set added. We use Fuclidean distance on the activations of the penultimate layer (the layer immediately before the classification head), as done in Citovsky al. (2021).
Margin-Density (Nguyen & Smeulders, 2004). Scores candidates by the product of their margin and their density estimates, so as to increase diversity. The density is computed by first clustering the penultimate layer activations of the current model on all |z| candidate points via K Kemaass. Then, the density score of candidate v; is computed as: (C(xa))/122 where C(rx) is the cluster containing Jzz We use min (20, [2]] clusters.
Cluster-Margin (Citovsky et al., 2021). Designed as a way to increase diversity in the extremely large batch size (100K-IM) IM) setting where continuously model retraining can be expensive, Cluster- Margin prescribes a two step procedure. First, after the model is trained on the seed set, penultimate layer activations for all points are extracted and clustered using agglomerative clustering. This clustering is done only once. During each acquisition round, candidates with the m x B lowest margin scores (denoted M) along with their clusters C' are retrieved. CM is sorted ascendingly by cluster size and cycled through in order, selecting single example at random from each cluster. After sampling from the final (i.e. largest) cluster, points are repeatedly sampled from the smallest unsaturated cluster until total of B points have been acquired. We explore the same settings as they do: m 1.25 as well as m = 10. We use Scikit-Leam's 's (Pedregosa et al., 2011) agglomerative clustering API with Euclidean distance, average linkage, and number of clusters set to [Nwata/ /mm]-
4

---

Under review as a conference paper at ICLR 2023
Query-by-Comnitte (QBC) Beluch al. (2018). Uses the disagreement among models in an ensemble, or committee, of K models trained on the active set at each iteration. Like Munjal et al. (2022), we use the variance ratio which was shown to give superior results. It is computed as v = 1 fm/K K, where fm is the number of predictions in the modal class. We set K = 25. As noted in prior work, differences among the committee members largely stem from differences in random initialization of the model than from random mini-batch ordering. Thus, when evaluating with pre-training, we use randomly initialized non-pre-trained members.
PowerrMargin and PowerBALD. Like many other AL methods, both margin and BALD were designed for the case when points are acquired one at a time (i.e. batch size 1). Recently, Kirsch et al. (2021) proposed a simple and efficient technique for extending any single sample acquisition method to the batch setting. Letting { represent the scores for the candidate points Z, instead of selecting topk { they propose selecting topk {s, + e (softmax variant) or topk (log(s;) + } (power variant), where & ~ Gumbel(0, B )) The driving insight, derived from the popular Softmax- Gumbel trick (Maddison et al., 2014; Gumbel, 1954; Kool et al., 2019), is that topk {s; + < is the same as sampling stochastically without replacement from 2 where the sampling distribution exp([fs.) Σ, exp((a,)) E (1...., (1....,121) ). As they recommend, we use the power variant with for both BALD and margin with Margin(-) used for the latter.
Categorical
121)):
EXPERIMENTS
4.1 SETUP
Active Learning Setting. We consider batch bashd AL in this work. Starting with a seed set To of labeled points drawn from the training split, we select the best B points- according to the examined AL method-from the remainder of training to be labeled and added to our active set at each acquisition round. We do this iteratively for T rounds or until the training dataset is exhausted, whichever happens first.
In order to get clear picture into the performance of AL. algorithms across active learning settings of practical interest, construct the following scenarios, fixing T' 20. Small: [Zo| 30, B 10. Medium: [To| = 100, B 50. Large: [Zo| = 300, B = 200.
Datasets. We consider the same 69 tabular datasets used in Bahri et al. (2021) and perform the same pre-processing steps. Concretely, these are all the datasets from the public OpenML-CC18 classification benchmark' under the CC-BY licence, less MNIST, Fashion-MNIST, and CIFARIO as they are vision-centric. We pre-process as follows: if a feature column is always missing, we drop it. Otherwise, if the feature is categorical, we fill in missing entries with the mode, most frequent, category computed over the full dataset. For numerical features, we impute it with the mean. We represent categorical features by one-hot encoding. We z-score normalize (i.e. subtract the mean and divide by the standard deviation) all numerical features of every dataset except three (OpenML dataset ids 4134, 28, and 1468). which are left unscaled. For each OpenML dataset, we form 80% /20% train/test splits where a different split is generated for each of the 20 trials and all methods use the same splits. Unsupervised SCARF pre-training uses the features (and not the labels) of the entire train split- 70% for training and the remaining 10% as a validation set for early stopping.
Model Architectures and Training. Our model consists of backbone followed by classification head (a single affine projection down to number of classes). The backbone is a 5-layer ReLU deep net with 256 units per layer. When the model is SCARF pre-trained, pre-training head, 2-layer ReLU net with 256 units per layer, is attached to the output of the backbone. After pre- pre-training the backbone with the pre-training head, the head is discarded; both the backbone and the classification head are updated during supervised fine-tuning. We use the recommended settings for SCARF 60% of the feature indices are corrupted and no temperature scaling (i.e. T 1). We pre-train for a maximum of 1000 epochs, early stopping with patience 3 on static validation set built from 20 epochs of the validation data. We train all models with the Adam optimizer using default learning
hhttps://dccs openml wtt../pwww...mmmmmoomm.oo.mm www.gbbencccaartt..
5

---

Under review as a conference paper at ICLR 2023
AUBC, w/o pre-training
chaster Margin
hand
powerMargg gin
mar Bm
HandMane gin
Ang
cereset
typictut
dlsterttrpi Margin (10)
marginn Dainsiy naly
minMange ginn
mat sant
powerdalL
RADGE
rand
O4E 043
ESA
0:23
410
820
837
200 04G
% rel. gain in AUBC wo pre traininin
mergin
R1/11
FASTRCTIEE
30
2004 9014 S0Y
Margin
H141
17,33
ent ont-
MangMrg
5424
328
003
2013
06-3922100
0:23
230
2204 224430422000000
coreset
typiclsst
49
405
0/33
6193
0496
429
clusterMargin (1.25)
clusterMargin (10)
5400
30T
SCT
0/77
533
DG2
13473010
0:23
1/55
003
120
A4D
QBC
marginDeesity ty MAr
8X2
6/
LA75
12160
004
L
margin Mavernarrgin ent
coress
1i 223
OBC 100
dnvvg/aao
Ea0VE
minMargin
26.292200
2207 200
maxent 4556
309
0.21
224
LOT
21:2
Buew mannnnn
623
929
0:4
0:33
244
824
422
3:24
921
0/3
PowerBALD
073
RADOF
H116
5:1
AB
20 9a46 X/ Fal 2O0ERCG-R
AUBC, w/ pre-training
Cluster Margin Drrgini 25)
PowerrMrrgh gin margin gin
Fandi gin
coreeet
dusterttargh righ (10)
maargint Denaity
minmerrgh maaxent
DowerBaLO BALO
BADGE
land
rand
AMT Wie
2035
4188
3551
820
% rel., gain In AUBC w/ pre-training
margin
HU43
120220-362717
3424
2001 2633363
4
powerMargin
5328
12033300825000
3222
2404362333003
ent 0305
2013927100
07200
331
32/22
2
rendMergin
VIX
wn1
W/1
V10
211
coreset
sypiclust
44
303
OCT
023
DGL
0:25
4156
201
800
002
(223
1504
301
clusteraargin (1.25)
clusterNergin (10)
026
072
S:3
1/2
121192212
0:22
2251
127
0322
L373 100
2011
margineensiy 1474 hw74
60
112136066600
D427
1asge ent
messet
weweennoraae ta 231
(100
QBC
bensity
maxent
BALD
poweraae
BADGE
minMargin 0727
06-372200
1222
maxent 6527 3ST
220
Lh293412
11.0
525
mi
1944
0:55
4119
175
4214
POWerBALD
BADGE
H010 4/4 20 2/1 12000000100360066
2/L
32:23
A/
1/7
20
Figure 1: Medium scale win and box plots. The box plots shown are unfiltered (those filtered by p-value are shown in the Appendix). We see with and without pre-training, margin matches or out- performs alternatives on nearly all datasets for which there was statistically significant difference. For example, without pre-training, it outperforms random all 41 of 41 times, CoreSet 41 of 41 times, and BALD 37 of 37 times. The relative gain over random is about 1-3%. See $4.2 for details on the statistical computation.
rate 0.001 and batch size of 128. For supervised training, we minimize the cross-entropy loss for 30 epochs.
Implementation and Infrastructure. Methods were implemented using the Keras API of Tensor- flow 2.0. Experiments were run on a cloud cluster of CPUs, and we used on the order of one million CPU core hours in total for the experiments.

---

Under review as a conference paper at ICLR 2023
dataset id 300 (top) w/o pre-training
dataset id 300 (top) w/ pre-training
0.97
0.00
1est accuracy
test accuracy
0.60
0.02
0.75
fand
OSC
disturMarge
1200
0.75
rand
LC
dustarMarge 120)
murginDerally
margh
murainBeriity
Margin
mintargin
0.70
Margin
miMargin
0.65
0.63
0.60
typiiiunt
BADGE
0.60
typtciunt
BADGI
0.55
Custaraarin (2.25)
CastiaMargin 11.25)
number of batches
number of batches
dataset id 1462 (bottom) w/o pre-training
dataset id 1462 (bottom) w/ pre-training
3.005
1.005
1.003
1.000
accuracy
0.995
0.990
0.905
marghh
menghCennty
menghOenety
Marge
misMa gi
6.900
Marge
minMa gn
ert
maceet
miseent
grrMaarga
BALD
0.975
GrdMargg
BALD
POWBRBALD
POWHEDALD
tysktu
0.900
Trsktu
19
11
batches
Figure 2: Medium scale AL curves. Margin has strong, stable performance across rounds for its best (top) and worst (bottom) datasets alike, with and without pre-training. Average dataset is in the Appendix.
4.2 EVALUATION METHOOSS
We largely follow the evaluation methodologies of Bahri et al. (2021), which we briefly review.
Win matrix. Given M methods, we compute "win" matrix W of size M M, where the (i.j) entry is defined as:
Dee L [method i beats j on dataset d]
Way Eee 1 [method beats j on dataset d] + 1 [method i loses to j on dataset d]
"Beats" and "loses" only defined when the means with unequal variance and p-value of 0.01). A win ratio of 0/1 means that out of the 69 (pairwise) comparisons, only
statistical tie (using Welch's t-test
significant and it was loss. Since 0/1 and 0/69 have the same value but the latter is more confident indication that is worse than j. we present the values in fractional form and use a heat map.
Box plots. The win matrix effectively conveys how often one method beats another but does not capture the degree by which. To that end, for each method, we compute the relative percent im- provement over the random sampling baseline on each dataset. We then build box-plots depicting the distribution the relative improvement across datasets. We show relative gains on all datasets as well as gains only statistically significant datasets where the means of the method and the ref- erence are different with p-value 0.1 (we use a larger p-luuee here than with the win ratio to capture more points).
We show win ratio and box plots for the Area Under the Budget Curve (AUBC) metric, where the x-axis is "number of batches acquired" and the y-axis is test accuracy. The trapezoidal rule is used to calculate the area.
Probability of Improvement. Following the methodology of Agarwal et al. (2021), estimate the probability that a method will outperform Margin when the dataset is unknown and we use stratified bootstrapping estimate confidence intervals of this statistic. See the Appendix for details.

---

Under review as a conference paper at ICLR 2023
AUBC, w/o pre-training
chaster Margin
hand
powerMargg gin
mar Bi
BaaaAa gin
cereset
typictutt
dlsterttrgi Margin (10)
marginn Daisiiy
QBC
minMargi Mirgin
masua sant
powerdalo
RADGE
rand
140 008
643
1:40
014
EST
BRGS 2013
14 04G
% rel. gain in AUBC w/o pre-training
mergin MU4e
111
Margin
187411/3
11,33
11
ent
206
022
00-022007
E122
330
MandMag
2323
1/3
023
26.091115
1/29
1233
271
2878 281332222
072
coreset
000
-5
typiclust
46
600
L0e
0/12
6159
SA12
AG4
-10
clusterMargin (1.25)
clusterMargin (10)
(10)
5424
303
D29
0.21
5:25
C122
16473217
0:23
2:13
022
03
-15
QBC-
marginDenssty by Mate
KFL
2710
411
319
by s
ent
Maaelaae [2 25)
seseeaaeiin gign (10)
QBC
marrginb Deniillllyy
minkkargen
rgin
maxent
BALD
powerBALD
BADGE
minMargin MAVA
V11
maxent 5328
LO3
0/21 8:23
11:30 922
3/22
2122
LO0
2022
022
BALD 201
544 0/4 3:42
D:3
220
3/35
120
042
3:2
200
0444
PowerBALD
1ST
1251
RADOF
W116 AM
40
2424
145
AUBC, w/ pre-training
cluster Margin Corginn 25)
powwerMarggn gin margin gin
Tannia Mandia gin
conteet
dustertaargh righ (10)
marggin Denaity
minMerggh maaxent
DOwerBaLD BALO
BADGE
land
rand
201 W/1
3118 REY
450
15
415
% rel. gain in AUBC w/ pre-training
margin
0527
0L22
DIV322800
34
2306
powerMargin
9026
313728009
1870
22
240430222602
3/1
entr p020
501
120
rendMergin M404 1n10
1.99
22
V10
50
PAOK
11/114044
592
coreset
-5
typiclust
44
429
125
022
9432
$143
0:2
2133
32T
022
220000044
002
-10
clusterNargin (1.25)
clusterNergin (10)
LX74
511
1/2
1115
W7
0127
STTY
RTD
WIR
EXDR
M25
-15
QBC 2093
-20
marginDensit 9070
5400
20-007007
poooe margin
Fanooe entt
coreser
clustenMargin (1. 25)
gin (10)
QBC
ma Denillyyy
minMargin
maxent
BALD
powerBALLD
BADGE
minMargin 0020
3:4
00-302600
2.6
1ATI
230425251992
maxent L500
225
10-18
3120
BALD Sit
1/4
WHB
1527
1141
500
424
31VE
1151
POWeBBALD
BADGE
0505
77
35
2/1
2326
77
25-153301
20
2222
5:
20
270837372613
Figure 3: Small scale win and box plots. The box plots shown unfiltered (those filtered by p- value are shown in the Appendix). We see that even here margin matches or outperforms alternatives, with and without pre-training, on nearly all datasets for which there was difference. Without pre- training, it outperforms random all 39 of 40 times, CoreSet 36 of 36 times, and BALD 44 of 44 times. The relative gain over random is about 1-4%.
4.3 MEDIUM SETTING RESULTS
Figure shows results for our medium scale active learning setup. Firstly, all baselines except for CoreSet, Max-Entropy, BALD, and PowerBALD methods are able to outperform the random sam- pling baseline on roughly half of all the datasets. Thus, actively selecting which points to label does in fact help model performance in this regime. We find that with and without pre-training, margin slightly outperforms Least Confident and Entropy (the other uncertainty baselines) along with QBC, cesttering-eentrii baselines TypiClust
Margin-Density and Min-Margin. It significantly outperforms and Cluster- Margin,,,,,,,,, if the latter uses an average cluster size of 10. When Cluster-Marre uses an average cluster size of 1 1.25, performance similar margin. This makes sense, since in this case,
8

---

Under review as a conference paper at ICLR 2023
dataset id 29 (middle) w/ pre-training
dataset id 1468 (middle) w/ pre-training
0.92
test accuracy
test accuracy
0.03
0.82
find
disturNarge
1200
rand
distarMarie 120)
0.00
margh
margi-Derally
LC
margh
merginBeralty
Done Margin
mintargin
Margin
miMargin
0.78
0.73
typiiiunt
BADGE
typtciunt
BADGI
CusteMargin (2.25)
number of batches
number of batches
Figure 4: Small scale AL curves. Like the medium setting, margin has strong, stable performance across rounds for its best, average, and worst datasets alike, with and without pre-training. See Appendix for best and worst datasets.
the algorithm first selects the roughly B lowest margin points, and each point will more or less have its own cluster (except perhaps if two datapoints are near duplicates), sampling from these clus- ters will just return the same lowest margin points. From the box plots we see that margin beats random by relative 1-3% average. Methods that tied slightly underperformed margin the win plots have a comparable relative gain over random, whereas the gains for others are near zero negative (in the case of BALD). For each pre-training setting, Figure 2 shows AL curves for the datasets margin performs best, average, and worst on, compared to random (filtering using p-value of 0.1). We clearly see margin near or the top of the pack in all cases and across all acquisition rounds.
4.4 SMALL SETUP RESULTS
Somewhat our surprise, the trends in the small setup are similar those for the medium scale setup. A priori it seemed that with a seed set size of merely 30, embeddings uncertainties derived from the current model would be untrustworthy and that random sampling would often be optimal. However, we find that even in this regime, margin and other uncertainty-basee baselines provide a boost. Observing performance after the 7th acquisition round, we see that as before, CoreSet, TypiClust, PowerBALD, and Max-Entropy perform similar to random while BALD underperforms substantially.
4.5 LARGE SETUP RESULTS
In this setting we use larger seed set and larger batch, to test whether starting with a more accurate underlying model will benefit alternatives more than margin and whether margin's 's naive top-k batch acquisition approach would be overshadowed by the other baselines' diversity- promoting ones. With the caveat that what sizes are considered small or large subjective and application specific, we observe no differences in high level trends in this setting. Results are presented in Figures 6, 7, and 8, shown in the Appendix.
5 CONCLUSION
In this work, we question whether many active learning strategies, new and old alike, can really outperform simple margin sampling when deep neural networks are trained small to medium- sized tabular datasets. We analyzed a diverse set of methods on 69 real-world datasets with and without pre-training under different seed set and batch sizes, and we found that method was able to outperform margin sampling in any statistically remarkable way. Margin has no hyper- parameters and is consistently strong across all settings explored. We cannot recommend a single better method for practitioners faced with data labeling constraints, especially in the case tabular data, baseline be henchmarked against by researchers in the field.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural informa- fion processing systems, 34:29304-299355 2021.
David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. Technical report, Stanford, 2006.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient arXiv:/906.0300 2019.
gradient lower bounds. arXiv preprint
Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: sulf-super contrastive learning using random feature corruption. arXiv preprint arXiv:200.11117, 2021.
Maria-Florina Balcan, Andrei Broder, and Tong Zhang. Margin based active learning. Interna- tional Conference on Computational Learning T'heory, pp. 35-50. Springer, 2007.
William H Beluch, Tim Genewein, Andreas Nürneerger, and Jan M Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9368-9377. 2018.
Lile Cai, Xun Xu, Lining Zhang, and Chuan-Sheng Foo. Exploring spatial diversity for region-based active learning. IEFE Transactions on Image Processing, 30:8702-8712, 2021.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, pp. 785-794, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Zichong Chen, Feng Yang, Albrecht Lindner, Guillermo Barrenetxea, and Martin Vetterli. Howis the weather: Automatic inference from images. In 2012 19th IEEE International conference on image processing, pp. 1853-1856. IEFE, 2012.
Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Ros- tamizadeh, and Sanjiv Kumar. Batch active learning at scale. Advances in Neural Information Processing Systems, 34:1 1933-1199944, 2021.
Corinna Cortes, Lawrence D Jackel, and Wan-Ping Chiang. Limits on learning machine accuracy imposed by data quality. Advances in Neural Information Processing Systems, 7, 1994.
Liat Ein Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. Active learning for bert: an empir- ical study. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 7949-7962, 2020.
Zeyad Ali Sami Emam, Hong-Min Chu, Yeng-Yeh Chiang, Wojeiech Czaja, Richard Leapman, Micah Goldblum, and Tom Goldstein. Active learning at the imagenct seale. arXiv preprint arXiv:2111.12880 2021.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1 1183-1192 192. PMLR, 2017.
Venkat Gudivada, Amy Apon, and Junhua Ding. Data quality considerations for big data and ma- chine learning: Going beyond data cleaning and transformations. International Journal on Ad- vances Software, 10(1):1-20, 2017.
Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: a series of lectures, volume 33. US Government Printing Office, 1954.
10

---

Under review as a conference paper at ICLR 2023
Guy Hacohen, Avihu Dekel, and Daphna Weinshall. Active learning budget: Opposite strategies suit high and low budgets. arXiv preprint arXiv:2202.02794, 2022.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprini arXiv: 2011.
Peiyun Hu, Zachary C Lipton, Anima Anandkumar, and Deva Ramanan. Active learning with partial feedback. arXiv preprint 1rXiv:182..0744 2018.
Qiang Hu, Yuejun Guo, Maxime Cordy, Xiaofei Xie, Wei Ma, Mike Papadakis, and Yves Le Traon. Towards exploring the limitations of active learning: An empirical study. In 202/ 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 917-929. IEEE, 2021.
Lian-Zhi Huo and Ping Tang. A batch-mode active learning algorithm using region- region-partttoonnng diversity for svm classifier. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 7(4):003-10046, 2014.
Ahmed Hussein, Mohamed Medhat Gaber, and Eyad Elyan. Deep active learning for autonomous navigation. In International Conference on Engineering Applications of Neural Networks, pp- 3-17. Springer, 2016.
Heinrich Jiang and Maya R Gupta. Bootstrapping for batch active sampling. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 3086-3096, 2021.
Stamatis Karlos, Georgios Kostopoulos, Sotiris Kotsiantis, and Vassilis Tampakas. Using active
learning methods for predicting fraudulent financial statements. In International Conference Engineering Applications of Neural Networks, pp. 351-362. Springer, 2017.
Andreas Kirsch, Sebastian Farquhar, and Yarin Gal. A simple baseline for batch active learning with stochastic acquisition functions. arXiv preprint arXiv:2 120599 2021.
Wouter Kool, Herke Van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. International Conference on Machine Learning, pp. 3499-3508. PMLR, 2019.
David D Lewis, Robert E Schapire, James P Callan, and Ron Papka. Training algorithms for lin- ear text classifiers. In Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 298-306, 1996.
Annika Liebgott, Thomas Küstner, Sergios Gatidis, Fritz Schick, and Bin Yang. Active learning for magnetic resonance image quality assessment. In 2016 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 922-926. IEEE, 2016.
David JC MacKay. The evidence framework applied to classification networks. Neural computation, 4(5):720-736, 1992.
Chris J Maddison, Daniel Tarlow, and Tom Minka. A sampling. Advances in neural information processing systems, 27, 2014.
Mottaghi and Serena Yeung. Adversarial representation active learning. arXiv preprint 10X1v:191..09720. 2019.
Prateek Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, and Shadab Khan. Towards ro- bust and reproducible active learning using neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 223-232, 2022.
Hieu Nguyen and Arnold Smeulders. Active learning using pre-clustering. In Proceedings of the fwenty-first international conference on Machine learning, pp. 79, 2004.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Coumapeau, M. Brucher, M. Perrot. and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2833 2011.
11

---

Under review as a conference paper at ICLR 2023
Davi Pereira-Santoss Ricardo Bastos Cavalcante Prudêncio, and André CPLF de Carvalho. Empiri- cal investigation of active learning strategies. Neurocomputting. 326:15-21. 2019.
Anant Raj and Francis Bach. Convergence of uncertainty sampling for active learning. In Interna- tional Conference on Machine Learning, pp. 18310-18331. PMLR, 2022.
Maria E Ramirez-Loaiza, Manali Sharma, Geet Kumar, and Mustafa Bilgic. Active learning: an empirical study of common baselines. Data mining and knowledge discovery, 31(2):287-313, 2017.
Daniel Reker and Gisbert Schneider. Active-lcarning strategies in computer-assisted drug discovery. Drug discovery today, (0(4):458-466) 2015.
Andrew 1 Schein and Lyle H Ungar. Active learning for logistic regression: evaluation, Machine Learning. 68(3):235-263 2007.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 700800049, 2017.
Amin Shahraki, Mahmoud Abbasi, Amir Taherkordi, and Anca Delia Jureut. Active learning for network traffic classification: technical study. IEEE Transactions Cognitive Communications and Networking, 8(1):422-433) 2021.
Yanyao Shen, Hyokun Yun, Zachary C Lipton, Yakov Kronrod, and Animashree Anandkumar. Deep active learning for named entity recognition. arXiv preprint arXiv: arXiv:1707.05923 2017.
Yining Wang and Aarti Singh. Noise-adaptive margin-based active learning and lower bounds under tsybakov noise condition. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Manfred KK Warmuth, Gunnar Rätsch, Michael Mathicson, Jun Liao, and Christian Lemmen. Ac- tive learning in the drug discovery process. Advances in Neural information processing systems, 14, 2001.
Martin J Willemink, Wojciech A Koszek, Cailin Hardell, Jie Wu, Dominik Fleischmann, Hugh Harvey, Les R Folio, Ronald M Summers, Daniel L Rubin, and Matthew P Lungren. Preparing medical imaging data for machine learning. Radiology, 295(1):4-15, 2020.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
We present results that were omitted from the main text.
A.1 PROBABILITY IMPROVEMENT: AN ALTERNATIVE TO WIN RATIO PLOTS
Agarwal al. (2021) suggests various statistically sound strategies for comparing the performance of methods across tasks (or datasets) in the presence of stochastic factors. One such strategy is prob- ability of improvement, which we briefly review for completeness (see the paper for more details).
Let X and Y be the scalar performance metric (higher is better) of algorithms X and Y, and Xm. (Y,)) be the performance X (Y) on dataset m. Suppose we observe N samples of Xm and K samples of Ym. We use the Mann-Wil Whitney U-statistic:
1, if <x,
P(Xm mm) NK
(( Um.))
where S(x,y)
if y I,
2
0,
if > x.
M Σ P(Xm > Ym), 1=1
P(X M
where Tm.i represents the performance of X on trial i on dataset m. We perform stratified bootstrap sampling (re-sampling 200 times from Xm,1:N and Ym,1.K for each dataset m) and then show violin plots of the bootstrap sampling distribution of the U-statistic (i.e. probability improvement). the upper CT is higher than threshold of 0.75, then the results said be statistically meaningful as per the Neyman-Pearee Pearson statistical testing criterion.
In Figure 5 we plot the probability that method beats Margin the AUBC metrie. We find that this probability is less than around 0.50 and the quantiles are well below 0.75 so we can say that when the dataset task is unknown, method outperforms Margin in any reasonably statistically meaningful way.
Margin), using AUBC, w/o pre-traihing
0.5
Marqin), using AUBC, pre pre-training
0.5
0.5
0.4
0.4
0.1
LC
200
LC
Roc L. waweeee
ReSomnAa dreu
Margini, using AUBC, w/o pre traiiing
Marqin), using AUBC, pre training
0.5
0.6
0.4
0.3
0.3
0.2
0.2
L.C.
100 U.
2000
6.1
200
Apys Copsunctuttnne
2-0000
D
Figure 5: Probability of Improvement charts for small (top) and medium (bottom) settings. We see that the alternatives Margin do not beat Margin in statistically meaningful way.
A.2 LARGE SETTING RESULTS
13

---

Under review as a conference paper at ICLR 2023
AUBC, w/o pre-training
rand
powerrMarge Margia marg gin
tandblang gin
end
Cluster ClusterMargio Margin (3
coresiet
traiciast
BustarMargin
margin Dernity
(10)
qec
minMargi Mangnn
maa xort
DOWNHBALO BALD
BADGE
rend
1PT RPS
1110 6m
111
6024
1012
% rel. gain in AUBC wo pre-training
margin 4027
2424 12
30
D1L
powerMargin
ent RAC0
89
35
BILL
25
E324
03
631
2215282423122
399
rendiergin
corsset
364 0GE 1/11
3/13
404
521
2011
0224
typiclust
02 356
o28
3:20 5033
3022
022
8/16
200
025
clusterNargin (1.25)
4027
247429242005
clusterNergin (10)
WV
1228 trte
QBC
marginDenstt ty 2322
413
60
50
501
7V
262422022
20
29
69
margin
powerMarget
ent
(OT)
336
Aasuure
dnvggaaoo
a9.wwa
minMargin
maxent 5323
008
0:3
416
162233110
0:2
E273
271
1:14
EN73
powerBALD
as
303
813
1010
L12
BADGE 3002
112334342800
D626
281825222262
AUBC, w/ pre-training
cluster Marerraaarii (10)
Clustertarg arrgin
margin margin Dariity eatt
rand
m gin
PrwerMarrg
randllarn gin
ert
Coreser
traccust
OBC
minn
Margaariin
moxer xara
powerBALL
RADGE
204
apS
122
601
30
627
893
A01
602
015
401
% rel. gain in AUBC w/ pre-training
margin
2503
21
30
17
26082800
2522
4/4
20
547419012222
Dt
3
powerMargin
2
ent PUP2
217
1212 107
611
aanMargin
1253
coraset
506
303
024
typiclust
02
528
022
1:22
D/2
3/13
920
022
413
022
clusterMargin (1.25)
-2
chasterMargin (10)
40%
6117
1011
Q714
111
0BC-
marginDenstt ty 2072
4/2
62
30
57
W1
10132000
3M
C4T4
86G
ent
1OT
GALD
minMargin
maxent L522 213
524
002
0C2
3112
112033112
022
224
300
422
powerBALD
A6
301
026
877
013
10300
Q214
022
A22
BADGE 2007
26-028009
2727
66
Figure Win and unfiltered box plots for the large setting.
14

---

Under review as a conference paper at ICLR 2023
% rel. gain in AUBC wo pre-training
% rel. gain AUBC pre-training
4
3
1
0
-1
-2
-2
-3
-3
margin
powerMargin
ent
randMargin
typiclust
clusterMargin 25)
clusterMary Mariin (10)
QBC
marginDensity
minMargin
maxent
BALD
powerBALD
BADGE
LC
margin
powerMargin
ent
randMargin
typiclust
clusterMargin 11.25)
clusterMargi (10)
QBC
marginDensity
Margin
maxent
BALD
powerBALD
BADGE
Figure 7: Box plots for the large setting filtered using a p-value of 0.1.
dataset id 6 (top) w/o pre pre-training
dataset id 6 (top) w/ pre-training
0.95
0.90
test accuracy
test accuracy
0.80
Rind
OsC
disterMagg 1200
0.82
Tard)
dustamMagg 2000
QaC
nargin
margi-
0.75
margin
DarginDuralty
0.73
Dowe Hargin
Margin
mvMtargi
0.70
0.70
0.88
typiciutt
BADGE
typtciat
BADGI
0.63
CusteMaargn
0.60
CusterMargn 11.25)
number of batches
number of batches
dataset id 1461 (middle) w/o pre-training
dataset id 1489 (middle) w/ pre-training
0.010
0.505
0.900
test accuracy
0.86
lend
(lutee Marg 1100
dusterNarie (20)
OBC
105t
Morg
meight
mangrDDeetl
ent
randParrgm
SALD
0.82
tomeet
power RAID
Tysidus
BADIEE
typiciat
BADGE
diaterMergin (.25)
CustorMargnn (1.25)
19
11 13
20
number of batches
number of batches
dataset 38 (bottom) w/o pre-training
dataset 38 (bottom) w/ pre-training
0.583
0.900
0.990
1OsT accuracy
8.975
accurac
0.975
Marg
duueeraarg (10)
0.970
080
nargis
mongabensty
margh
mangaCensty
Marge
menMa gn
menMarg
ert
moveent
0.963
anddarggi
0.953
0.955
dlustikarrin
0.550
number of batches
number batches
Figure 8: AL curves for the large setting.
15

---

Under review as a conference paper at ICLR 2023
A.3
REMAINING SMALL AND MEDIUM SCALE RESULTS
gain in AUBC pre-training
rel. gain in AUBC pre-training
6
-2
-2
-4
-4
-6
-6
LC
margin
powerMargin
ent
randMargin
typic/ /usc
clusterMargin (1.25)
clusterMargin (10)
QBC
marginDensity
minMargin
maxent
BALD
powerBALI BADGE
LC margin
powerMargin
ent
randMargin
coreset
typiclust
clusterMargin (1.25)
clusterMargin (10)
QBC
marginDensity
minMargin
maxent
BALD
powerBALD
BADGE
% rel. gain in AUBC w/o pre-training
% rel. gain in AUBC w/ pre-training
10
10
-5
-5
-10
-10
-15
-15
-20
-20
margin
powerMargin
ent
randMargin typiclust
clusterMargin (1 25)
clusterMargin (10)
QBC
Densit
minMargin
maxent
BALD
powerBALD BADGE
LC margin powerMargin ent
randMargin
coreset
typiclust.
clusterMargin (1.25)
clusterMargin (10)
QBC
marginDensity
minMargin
maxent
BALD
powerBALD
BADGE
Figure 9: Top: Box plots for the medium setting filtered using p-value 0.1. Bottom: Same plots for
small setting.
16

---

Under review as a conference paper at ICLR 2023
dataset id 1494 (middle) w/o pre-training
dataset id 1590 (middle) w/ pre-training
0.83
0.84
1651 accurac)
0.87
test accuracy
0.83
0.86
dst
disterMargi
0.62
dustenlarge 1300
8.85
0.84
TandWa
BALD
SALD
corssut
corisut
powe SALD BADGE
Typidlutt
BADGE
typtciutt
0.02
SkatarMargin (3.25)
ChotanMargg 13.25)
number of batches
number of batches
dataset id 40966 (top) w/o pre-training
dataset 40966 (top) w/ pre-training
test accuracy
test accuracy
0.5
curdeeorrh (1.)
Cuseer ginh (10)
DBC
maaggnn
margneDensity
marggn poworMarrh
mangnDeessyy
powertoorin
0.6
RADCE
RADGE
shude Margin (1.25)
Hargee (1.25)
number batches
number batches
dataset id 40978 (bottom) w/o pre-training
dataset id 6 (bottom) w/ pre-training
0.01
0.90
test accuracy
test accuraEy
0.65
0.42
0.85
land
diatenMarge (200
fand
disterNargi 130)
Ost
080
Targon
marginDerahh
Taggln
BeraiiBernily
0.80
Margin
miMargin
Margh
miMargg
0.5
0.75
typiciunt
BADGE
typiciutt
BADSI
CiokerMargi (1.25)
0.2
ChstenMargn 11.25)
number of batches
number of batches
Figure 10: AL curves: middle curves for medium setting (top), top and bottom curves for the small setting (bottom two).
17


Reviews:
[{"review_id": "ltLuHytxKJ", "reviewer": "Reviewer_h2sD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper compares a variety of active learning algorithms on 69 tabular datasets using a multilayer perceptron model (with and without unlabeled pretraining). The results show that margin-based (i.e. best versus second best) uncertainty sampling performs the best.", "summary_of_the_review": "This type of active learning evaluation paper (especially for tabular datasets) is a very useful addition to the active learning literature. However, the quality of the evaluation could be improved in some ways though.", "strength_and_weaknesses": "Strengths:\n - This paper empirically analyzes active learning algorithms (which is much needed) rather than defining yet another algorithm.\n - This paper studies tabular datasets which are under-explored in active learning experiments. \n - This paper compares a large variety of active learning algorithms and large number of datasets.\n\n\nWeaknesses:\n - This paper only evaluates with one model, a multilayer perceptron, which may be suboptimal for tabular datasets compared to tree-based methods.\n - With the very large number of active learning algorithms, it is admittedly hard to pick a list. However, I was surprised to see BADGE and BAIT were not on the list. While BADGE is computationally intensive, for image classification tasks, it works very well.\n - The evaluation criteria (performance after k rounds in 3 different batch settings) seems a bit arbitrary without much discussion.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UrhUktLD_Q", "reviewer": "Reviewer_58Ep", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper performs a comparison of different active learning strategies on tabular data sets when trained with deep learning models that are pre-trained with self-supervised learning. The key result is that margin sampling, which is also one of the easiest active learning strategies, consistently outperforms other strategies.\n", "summary_of_the_review": "Overall, the paper seems limited in scope and performs statistical analysis that is not clear. It could be my understanding of the stat analysis is wrong and if I am incorrect, I am happy to increase my score, but I am not yet sure about how to address the scope limitations.\n\n\n==============\nAFTER REBUTTAL\n\nI appreciate the authors for carefully considering my comments and exploring a more robust statistical setup to compare their methods. I have raised my score accordingly.", "strength_and_weaknesses": "Strengths\n- The experiments cover a large number of data sets and seeds\n- The paper concludes with clear guidelines for practitioners\n\nWeaknesses\n- The focus of the study seems narrow. The paper focuses on (1) tabular data with (2) deep learning models. This is limited as the vast majority of tabular ML in practice is done not with DL but with GBT, RF, etc.\n- The statistical testing seems inappropriate in that the number of tests that are being made is massive. My understanding may be incorrect so please correct me if I am wrong, but there are 49 x 20 x 3 tests performed and then the results are collated. At the very least it should be possible to reduce the number of tests to 16 x 16 by considering multiple measurements (i.e., seeds) and just doing a single test per method x method pair. An even simpler analysis would be for each data set, to perform a single ANOVA (if assumptions are met) with margin and other methods to demonstrate that margin actually outperforms all methods on each data set. This reduces the number of tests down to 49. There are statistical setups that could also be considered.\n- Why do you use a larger p-value in one setting and not in the other? The p-value is supposed to be a priori fixed and not modified to increase the number of wins. If the result is not significant for the pre-determined p-value then that in itself is a result\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FKQxXSQt98", "reviewer": "Reviewer_TZgZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is an empirical study of deep active learning (especially for margin based approaches). This paper consider different tasks (openml) and how model pre-training affect the performance.", "summary_of_the_review": "This is a well-written emprical study with sufficient comparative experiments about margin-based DAL approaches, however, the experimental analysis is somewhat weak and need to be improved.", "strength_and_weaknesses": "Strength: this empirical study tests many DAL approaches (mostly are margin-based approaches) on openml datasets. A meaningful finding is that margin-based approachs perform consistently better than random sampling across multiple various tasks.\n\nWeakness: some directions need to dig deeper.\n\n1. For w/ and w/o pre-training techniques, this paper only provide an overall performance, however, how w/ and w/o pre-training techniques affect the AL performance is complex, it is related to the similarity btw target and source domain and perform different on different tasks like [r1] (section 3.4, considering dataset with distribution shift). The author should dig more informations either from existing experimental results or conduct more experiments.\n\n2. For evaluation methods, both win matrix and box plots only present the performance on one round. As for Figure 4, there are too many curves with mixing colors. It's really hard to distinguish which is better across varying batches. Can use Area Under the Budget Curve (AUBC) [r1] or AUCLOG, which is the AUC of the learning curve when the X-axis is in log-scale [r2] or the average performance across multiple rounds (not recommended) to present an overall performance. Aslo, Figure 4 (and the similar figures) need to be re-designed by selecting more distinguishable color schemes or zoom in on some areas like [r3].\n\n3. A deeper and more confidential analysis of the experimental results is needed. For instance, at the end of Page 7, \"Methods that tied\nor slightly underperformed margin on the win plots have a comparable relative gain over random, whereas the gains for others are near zero or negative (in the case of BALD). We observe that Power-BALD outperforms BALD – this is, most likely, not because BALD is ill-suited for the batch AL setting but because PowerBALD’s additive noising process mimics (biased) random sampling.\" Firstly, \"most likely\" seems not confidential. Comprehensive study needs reasonable and confident analysis. Then, Why BALD is ill-suited for the batch AL setting? Does the author try different batch settings or there are some publications support the opinion? \n\n[r1] Zhan X, Wang Q, Huang K, et al. A comparative survey of deep active learning[J]. arXiv preprint arXiv:2203.13450, 2022.\n\n[r2] Mozafari B, Sarkar P, Franklin M, et al. Scaling up crowd-sourcing to very large datasets: a case for active learning[J]. Proceedings of the VLDB Endowment, 2014, 8(2): 125-136.\n\n[r3] Kothawade S, Beck N, Killamsetty K, et al. Similar: Submodular information measures based active learning in realistic scenarios[J]. Advances in Neural Information Processing Systems, 2021, 34: 18685-18697.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject