Accelerating Batch Active Learning Using Continual Learning Techniques
Authors: Arnav Mohanty Das
Gantavya Bhatt
Megh Manoj Bhalerao
Vianne R. Gao
Rui Yang
Jeff Bilmes
Venue: Accepted by TMLR ()

Abstract: A major problem with Active Learning (AL) is high training costs since models are typically retrained from scratch after every query round. We start by demonstrating that standard AL on neural networks with warm starting fails, both to accelerate training and to avoid catastrophic forgetting when using fine-tuning over AL query rounds.  We then develop a new class of techniques, circumventing this problem, by biasing further training towards previously labeled sets. We accomplish this by employing existing, and developing novel, replay-based Continual Learning (CL) algorithms that are effective at quickly learning the new without forgetting the old, especially when data comes from an evolving distribution. We call this paradigm \textit{"Continual Active Learning" (CAL)}.  We show CAL achieves significant speedups using a plethora of replay schemes that use model distillation and that select diverse/uncertain points from the history. We conduct experiments across many data domains, including natural language, vision, medical imaging, and computational biology, each with different neural architectures and dataset sizes. CAL consistently provides a $\sim$3x reduction in training time, while retaining performance and out-of-distribution robustness, showing its wide applicability.

Published in Transactions on Machine Learning Research (11/2023)
Accelerating Batch Active Learning Using Continual Learning Techniques
Arnav Das" Gantavya Bhatt 'University of Washington, Seattle {arnavmd2, gbhatt2, bilmea)Quw.edi
Megh Bhalerao Vianne Gao Rui Yang Jeff Bilmes'
Memorial Sloan Kettering Cancer Center
Reviewed on OpenRevicw:
Abstract
A major problem with Active Learning (AL) is high training costs since models are typically retrained from scratch after every query round. We start by demonstrating that standard AL on neural nelworks with warm starting fails, both to accelerate training and to avoid catastrophic forgetting when using fine-tuning over AL query rounds. We then develop new class of techniques, circunventing this problem, by biasing further training towards previously labeled sets. We accomplish this by employing existing, and developing novel, replay- based Continual Learning (CT.) algorithms that are effective at quickly learning the new without forgetting the old, especially when data comes from an evolving distribution. We call this paradigm 'Continual Active Learning" (CAL). We show CAL achieves significant speedups using a plethora of replay schemes that use model distillation and that select diverse/ uncertain points from the history. We conduct experiments across many data domains, including natural language, vision, medical imaging, and computational biology, each with different neural architectures and dataset sizes. CAL consistently provides ~3x reduction in training time, while retaining performance and out-of-distribution robustness, showing its wide applicability.
1 Introduction
While neural networks have been successful in a variety of different supervised settings, most such approaches are labeled-data hungry and require significant, computation. From a large pool of unlabeled data, active learning (AL) selects subsets of points label by imparting the learner with the ability to query human annotator. Such methods incrementally add points the labeled pool by repeatedly: (1) training model from scratch on the current labeled pool and (2) using some measure of model uncertainty and/or diversity
to select a set of points to query the annotator (Settles) 2009 2011 Wei et al., 2015 Ash et al. 2020 Killamsetty et al. 2021a). AL has been shown to reduce the amount of training data required but can be computationally expensive since it requires retraining a model, typically from scratch, after each query round.
A simple solution to warm start the model parameters between query rounds. However, the observed speedups tend still be limited since the model must make several passes through an ever-increasing pool of data. Moreover, warm starting alone in some cases can hurt generalization, (2020) and [eck et al. (2021). Another extension to this is to solely train the newly labeled batch of examples to avoid re-initialization. However, we show in Section g naive fine-tuning fails to retain accuracy on previously seen examples since the distribution of the query pool may drastically change with each round.
discussed in Ash & Adams
This problem of catastrophic forgetting while incrementally learning from a series of new tasks with shifting distributions a rentral question in another paradigın called Continual Learning (CL) (French) 1999. McCloskey & Cohen 1989 McClelland et al. 1995 Kirkpatrick et al. 2017ch. CL has recently gained popularity, and many algorithms have been introduced to allow models to quickly adapt to new tasks without.
EEqul contribution

---

Published in Transactions on Machine Learning Research (11/2023)
forgetting (Riemer et al., 2018 Lopez- Paz & Ranzato 2017 Chaudhry et al. Chaudhry et al. 2020 Kirkpatrick et al. (017b).
2019 Aljundi et al.
2019b
In this work, we propose Continual Active Learning (CAL which applies Continual Learning strategies to accelerate batch Active Learning. In CAL, we apply CL to enable the model to learn the newly labeled points without forgetting previously labeled points while using past samples efficiently using ryplay-based mods As such, we observe that CAL attains significant training time speedups over standard AL. This is beneficial for the following reasons: (1): As neural networks swell (Shoeybi et al. 2019), do the environmental and financial model training costs (Bender et al. 2021 Dhar 2020 Schwartz et al. 2020). Reducing the number of gradient updates required for AL will help mitigate such costs, especially with large-scale models. (2): Reducing AL computational requirements makes AL more accessible for edge computing, IoT, and low- resource device deployment (Senzaki & Hamelain 2021) such as with federated learning (Li et al. 2020). (3): Developing new AL sessorrttmss/cquiiiiiiii functions, or searching for architectures as done with NAS/Aul AAttoMM that are well-suited specifically for AL, can require hundreds or even thousands of runs. Since CAL's speodups are agnostic to the AL algorithm and the neural architecture, such experiments can be significantly sped up. Overall, the importance of speeding machine learning training processes is well recognized, as evidenced by the
plethora of efforts in the computing systems community (Jia ct al. 2022
Zhang ct al. 2017
Zheng et al. 2022
In addition, CAL demonstrates another practical application for CL methods. Many settings used to benchmark CL methods in recent work are somewhat contrived (Farquhar & Gal, 2018 Wallingford et al., 2023 Most CL work considers the class/ /omaain incremental setting, where only the samples that belong to a subset of the set of classes /domains of the original dataset are available to the model at any given time. This sctting need not be the only benchmark upon which CL methods are evaluated. We suggest that the evaluation of future new CL algorithms should be determined not only on tradi- tional CL evaluation schemes and benchmarks but also on their performance in the CAL setting.
1.005
CAL-ER
Avg. relative accuracy
CAL-MIR
1.000
x
CAL-DER
0.995
CAL-SD
CAL-SDS2
0.990
X
AL w/ WS
AL
0.985
1.0
1.5
2.5 Average Speedup
2.0
3.0
Figure 1: Summary of our results: Benchmarking average relative accuracy vs. speedup for the proposed methods (solid markers) against: the baseline. Each point is averaged across datasets and labeling budgets; top-right region is desiderata. This shows that proposed CAL methods (partic- ularly WAL-DER/SD/SDDS relative to the baseline (if not better) on multiple datasets and across different budgels, all while providing significant speedups.
In the present work, we are not using CL to improve or change AL querying strategies. We view this as both a strength of the present work and an oppor- tunity for future work. Firstly, it is a strength of our present work since any AL query strategy, both old and new, can in principle be applied in the CAL setting, both for speeding up the AL strategy and offering, as argued above, a test bed for CL. Indeed, a major AL research challenge includes combining AL with other techniques, as we believe we have done herein. Secondly, it is an opportunity since there is nothing inherent in the definition of CL that precludes the CL tasks from being dependent on the model as we show below. There may be a way for CL to open doors to intrinsically new AL querying policics of attaining new batches of unlabelod data. This we leave to the future. Our present core goal is to accelerate butch AL training via CL lechniques while preserving accuracy.
have equivalent performance
To the best of our knowledge, this application of CL algorithms to accelerate batch AL has never been explored. Our contributions can be summarized as follows: (1) We first demonstrate that batch active lcarning techniques can benefit. from continual learning techniques, and their merger creates a new class of techniques that we propose to call the "CAL framework." (2) We benchmark several existing CL methods (CAL-ER, CAL-DER, CAL-MIR) as well as novel methods (CAL-SD, CAL-SDS2) and evaluate them on diverse datasets based on the accuracy/spp /spuruyy/ppeduupp they can attain over standard AL. (3) We study speedup/performant /performance trade- offs on datasets that vary in modality (natural language, vision, medical imaging, and computational biology), ), neural architecture with varying degrees of computation (Transformers/ Caatiierrrr////////
data scale (including
Thhre have been papers published with titles containing the "Contimul Active Learning" phrase, but these do not merge Continual Learning with Active Learning as we do, hence our name.
2

---

Published in Transactions on Machine Learning Research (11/2023)
some larger datasets, one having 2M samples), and class-balance, And (4), lastly, we demonstrate that models trained with CAL and standard AT, models behave similarly, in that both classes of models attain similar uncertainty scores on held-out datasets and achieve similar robustness performance on -uttrf-distributionn data. Figure | summarizes our results, detailed later in the paper and greatly detailed in the appendices.
2 Related Work
Active learning (Atlas et al. 1989: Cohn et al.
1994 Wei et al.
2015
Killamsetty et al.
2021a Ash et al.
2020 has demonstrated label efficiency over passive learning. In addition, there has been extensive work on theoretical aspects of AL (Guillory et al. 2009 Hanneke 2009 2007
Balcan al.) 2010) where Hanneke
(2012) shows sample complexity advantages over passive learning in noise- free classifier learning for VC classes. More recently, Active Learning has also been studied as a procedure to incrementally learn the underlying data
distribution with the help of discrepancy framework (Mathelin et al. 2022 Cui & Sato 2020
Shui et al. 2019
Recently there has been an interest in speeding up active learning since most deep learning is computationally demanding. Kirsch et al. (019 Pinsler et al. (2019); Sennr & Savarese (2018) aim to reduce the number of query iterations by having large query batch sizes. However, they do not exploit the learned models from previous rounds for the subsequent ones and are therefore complementary to CAL Work such as Coleman let al. (2020a); Ertekin al. (2007); Mayer & Timofte (2020); Zhhu & Bento (2017): Zhang et al. (2023) speeds up the selection of the new query set by appropriately restricting the search space or by using generative methods. This work can be easily integrated into our framework because CAL works on the training side of active learning, not on the query selection. On the other hand, Lewis & Catlett (1994); ; Coleman et al.
(2020b); Yoo & Kweon (2019) use a smaller proxy model to reduce computation overhead, however, they still follow the standard active learning protocol, and therefore can be accelerated when integrated with CAL.
There exists work that explores continual/transte learning and active learning in the same context. Perkonigg et al. (2021) propose an approach that allows active learning to be applied to data streams of medical images by introducing a module that detects domain shifts. This is quite distinct from our work:
work uses CL algorithms to prevent catastrophic forgetting and to accelcrate learning. Zhou et al. (2021) study when standard active learning is used to fine-tune a pre-trained model, and employs transfer learning this does not consider continual learning and active learning logether, however, and is therefore not related to our work. Finally, Ayub & Fendley (2022) studies where a robot observes unlabeled data sampled from a shifting distribution, but does not explore active learning acceleration.
For preventing catastrophic forgetting, we mostly focus on replay-based algorithms that are state-of-the- art methods in CL. However, demonstrated in Section 4 on how active learning rounds can be viewed in a continual learning context, one can apply other methods such as EWC (Kirkpatrick et al. 2017a),
Bayesian divergence priors Li & Bilmes (2007), structural regularization (Li et al. 2021 or functional regularization (Titsias et al. 2020) as well.
The effect of warm-started model training on generalization and convergence speed has been explored by Ash & Adams (2020) which empirically demonstrates that a model that has been pretrained on a source dataset converges faster but exhibits worse generalization initialized model. However, that work only considers the setting where the source and target datasets are unbiased estimates of the same distribution. This is distinct from our work since the distributions we consider are all dependent on the model at each AL round. Furthermore, our work employs CL methods in addition to warm-starting, also not considered in Ash & Adams 2020
target dataset when compared to a randomly
3 Background
3.1 Batch Active Learning
Define [n] = {1, n), and let X' and y denote the input and output domains respectively. AL typically starts with an unlabeled datasct U = (ej), where each x; E X. The AL setting allows the model f. with parameters 8, to query a user for labels for any z E U. but the total number of labels limited to a budget b, where b S n. Throughout the work, we consider classification tasks so the output of fxx; is a probability
3

---

Published in Transactions on Machine Learning Research (11/2023)
Sond
Round
Round
Round
Cla
Cano
Class
Class
Selected Foints
1eat3
Teat
Teat
Feat
Figure 2: Visualization of standard AL when training a linear model on simple 2D syuthetic dataset. Each plot shows which samples were chosen by uncertainty sampling to be added to the labeled pool at given a query round. Notably, the distributions of the new labeled points vary significantly between rounds motivating CAL
distribution over classes. The goal of AL is to ensure that f can attain low error when trained only the set of b labeled points.
A Algorithm ! details the general AL procedure. Lines 3-6 construct the seed set Dj by randomly sampling a subset of points from V and labeling them. Lines 7-14 iteralively expand the labeled set for T rounds by training the model from random initialization on D, until convergence and selecting b, points (where Eieny by b) from UI based on some selection criteria that is dependent 0g. The selection criteria generally select samples based on model uncertainty and/or diversity (Tewis Gale 1994 Dagan & Ennelloo 1995; Settles Killamsetty et al., 2021a; Wei et al.) 2015 Ash al. 2020 Senne & Savarese 2017). In this work, we primarily consider uncertainty sampling Lewis & Gale (1994]
Dagan & Engelson 1995 Scttles though we
also test other selection criteria in Section
in the Appendix.
Uncertainty Sampling is widely- used practical AL method that selects Ur = (I1,...,To,} S U by choos- ing the samples that maximize a no- tion of model uncertainty. We consider
Algorithm 1 Batch Active Learning
1: procedure ACTIVELEARNING(//. u, bur, T)
1, 1, -+0
Initialise
3:
U4 4--l
Draw by camples from
4:
x (<2,)))) E44
Provide labela
entropy (Dagan & Engelsson 1995) as the uncertainty metric, so if (o( A
5:
u 1+ 1
P Remove unlabeled
+ SUD,
Add to labeled set
7:
while 1 1 T do
-Eijy (a; 0) log I (a;; 0);, then Uz+1 E argmax EmeA hg (x).
Randomly initialize Oinst
9:
0, + Train(f, 8mne 2)
ACu:|A|=b,
10:
Me+1 + Select (f. 8z, be) De+1 <- {(21,
D Select dg pointe based on Ps
11:
E U4
Distribution Shift As shown in Algo- rithm the set of samples that are labeled
12:
21 < 2 U4t1 + EU 1
13: 0r Train(s, 0init,
at given query round are dependent on the model parameters. Since the model is repeatedly updated, the distribution newly labeled samples may vary across different. AL rounds. This is illustrated in Figure where we perform standard AL in simplified setting. In the example, we use linear model to perform classification a 2D synthetic dataset and perform AL with entropy-based uncertainty sampling as the acquisition function. Upon visualizing the set of samples that are queried by the model at. different AL rounds, it is evident that distribution shift does occur. We demonstrate empirically in Figure 3 that training on newly labeled samples alone will cause catastrophic forgetting due to distribution shifts.
14:
return E, OT
3.2 Continual Learning
Dr> We define Di.H. Usenn D,. In CL, the dataset consists of T tasks )rr that are presented to the model sequentially, where D, = {x,, y.he,, N, are the task-t sample indices, and n, = [Ne]- At time t [T], the data/label pairs are sampled from the current task (x, y) ~ D., and the model has only limited access the history D1.-I. The CL objective to efficiently adapt the model to D, while ensuring performance the history does not appreciably degrade. We focus replay-based CL techniques that attempt to
4

---

Published in Transactions on Machine Learning Research (11/2023)
History Access
Continual Learning Papers
None
Partial
Full
Kirkpatrick et al. [0017b). Li & Hoiem (017))
x x
Aljundi et al. (2019b), Mai et al. (2020)
X
V
x
Buzzega et al. 2020), , Aljundi et al. (2019a)
Lopez-Pane Paz & Ranzato (2017) 7). Chaudhry et al. 2019
V
Ratcliff (9900)
Table 1: Different previous CL work organized by the memory access assumptions made in their settings. In the CAL setting, we do not need to assume that we have limited access to the history so we only consider CL methods that are applicable to the setting where the history is fully accessible. This category primarily contains replay-based methods.
approximately solve CL optimization by using samples from D14-1 to regularize the model while adapting to D,. Please refer to appendix B for more details on CL.
A ooutlines general replay-based
A Algorithm 2 Continual Learning
CL, where the objective is to adapt f pa- rameterized by 0g to while using sam-
1:
procedure CONTINUALTRAIN(U, Ou, D. H. m, m ml)) D. local
2:
++0
ples from the history H. Beurrent consists of m points randomly sampled from the current time's D, and Breplay consists of m(()) points chosen based on a criterion 6:
3:
while not converged do
4:
7
5:
Beurrent + ((z1, y:) ~
Sample from new tack.
Biuplay +- Select -1, H. mh) 0, + Updatees, -, Beurrent, Broplay)
Sumpie from history.
that selects from H. In line 6, 0, is com- puted based on an update rule that uti- lizes both Breplay and Beureent
7:
8: return 0,
Different CL methods assume varying degrees of access the history. In many practical settings, Pı:r too large to store in memory T is unknown CL algorithms assume limited or access Lo samples from the history. Some of these approaches remove unimportant samples from the history Aljundi et al. (2019b) Mai ct al. 2020), while others solely rely on regularizing the model paramcters without replaying any samples b Li & Hoiem (017). In the CAL setting, Di. is already stored in
from the history Kirkpatrick et al. (2017b); memory as is done in the standard AL setting so employing a CL algorithm that imposes a memory constraint would needlessly cause CAL to underperform. In Table [ a handful of prior CL works are sorted by what assumptions about memory /hiotoryy access they make; methods that do not make any memory constraint assumption are the most well-suited for CAL. A more comprehensive and complete overview of CL methods can be found in De Lange et al. (2022).
4 Blending Continual and Active Learning
A clear AL inefficiency that the model f is retrained from scratch on the entire labeled pool after every query round. One potential solution idca is to simply continue training the model only on the newly AL-queried samples and, via the process of warm starting, hope that history will not fade. Unfortunately for this approach, Figure (top) shows that when the model is warm-start trained only on the task t (samples labeled at AL round using entropy sampling), historical sample performance deteriorates precipitously while performance on the validation set flatlines. That is, at AL round t (x-axis), we continue to train the model until convergence on task & and track accuracy (y-axis) on each previous task and also on the validation set. The performance of task 1, after the initial drop, tracks that of the validation set, since task is a model agnostic initial unbiased random subset query of the training data. The performance task i, for i > 1, however, each of which is the result of model-condiiioned AL query, shows perilous historical forgetting. In the end, the model performs considerably worse on all of the historical tasks (aside from task 1) than on the validation set, even though it has been trained on those tasks and not the validation set. This experiment suggests that: (1) the distribution of each AT-queried task > 1 is different than the data distribution; (2) fine-tuning to task can result in catastrophic forgetting; and (3) techniques to combat catastrophic forgetting are necessary to effectively incorporate new information between successive AL rounds.
5

---

Published in Transactions on Machine Learning Research (11/2023)
Forgetting Finetuning
106
15
Accuracy (%)
Fargelting CAL-SDS2
166
V4
Taak
Query Round
Figure 3: A ResNet-18's performance CIFAR-10 the validation set (blue X) and task i for E (1,2....,9 9) (other colors) after training the warm-started model and training just on the new task data (top) compared to efficiently adapting newly labeled data with CAL method (bottom). In the top plot, at each round new 5% of the full dataset is added to the labeled pool and the previonsly trained model is further trained just on the new points (the isolated upper-right cyan point shows task 9's performance after round 9). After each round, the accuracy on the corresponding task does well, but all previous tasks' accuracies drop precipitously, demonstrating a form of catastrophic forgetting. In the bottom plot, the same setting is considered but the model adapted to the new data with CAL-SDS2, one of the CAL methods. Also, the validation set performance has flarlined despite continued training on new data when finetuning, but continues to increase when CAL-SDS2 is employed. This dernonstrates that naively line-tuning model to newly labeled points can work poorly.
The CAL approach, shown in Algorithm D uses CL techniques ameliorate catas- trophic forgetting. The key difference between CAL and Algorithm D is line 9. Instead of standard training, replay- based CL is used to adapt f to D, while retaining performance on D1a-1- The speedup comes from two sources: (1) we are computing gradient updates only
Algorithm 3 The general CAL approach.
I: procedure CAL((, U, b T, m, m
2:
1,1,
Inteialtse
3:
21 ~
Draw by camples from M Provide labols
4:
D. 111,,,)) E44
5:
+ 4:
E L-DD
7:
Randomly initialize A
while T do
9:
0. + Continua/Train(f. 0--1, De, Dist-1, m, m Ue Select 0, b.)
useful subset of the history Dia-1
10:
Seloot by points from Et
rather than all of it for reasonable choices
11:
De+1 + ((21,8)e E E4+11
of m()); and (2) the model converges
12:
1--44 U441 + EU DUDI 66
faster since it starts warm. The ability 13:
Pr + ContinualTrain// Or-1, Dr, Di.r m, m (h)
of CAL to combat catastrophic forget- 14:
return E, OT
ting is shown in Figure Figure 3 (bot-
tom). In the rest of the soction, define
(0) A ((a.))-Beurrnn [ (æ;; 0))] We next define and compare several CAL methods and assess their
performance based on their performance the test set and the speedup they attain compared to standard
AL.
Experience Replay (CAL-ER) is the simplest and oldest replay- method (Ratcliff 1990 Robins 1995). In this approach, Beurrent and Broplay are interleaved to create minibatch B of size + (4) and Breplay is chosen uniformly at random from D The parameters 0 of model f are updated based on the gradient computed on B.
Maximally Interfered Retricval (CAL-MIR) chooses (izo-m()) subset of points from D Pia-1 most likely to be forgotten (Aljundi et al. 2019a). Given a batch of m labeled samples Beurrent sampled from D, and model parameters 0, 0,, is computed by taking "virtual" gradient step i.e., O, L. where
6

---

Published in Transactions on Machine Learning Research (11/2023)
7 is the learning rate. Then for every example x in the history, SMIR(*) = ((f(x:)))) ([f(2; 8.). (Le.. the change in loss after taking single gradient step) is computed. The m(() samples with the highest. SMIR score are selected for Breplay, and the remainder is similar to experience replay. Beurrent and Breplay are concatenated to form the minibatch (as in CAI-ER), upon which the gradient update is computed. In practice, selection is done on random subset of Dia-1 for speed, since computing SMIR for every historical sample is prohibitively expensive.
Dark Experience Replay (CAI-DER) uses a distillation approach to regularize updates (Buzzega et al. 2020). Let g( 0) denote the pre-softmax logits of classifier f(z; ie., f(æ; 0) = softmax(g(x; 0)). In DER, every x' E Di:-1 has associated z' which corresponds to the model's logits at the end of the task when x was first observed - if x' E Dr, then x' _ g(x'; 02)) where l' E [t 1] and Oz. are the parameters obtained after round t'. DER minimizes LDEN (0) defined as:
CDEr (0) A Co(0)+ +
E (w/.8/ :/))=
la ()) =11 Bel ((<:0)))
(1)
where Breplay is a batch uniformly at randomly (w/o replacement) sampled from DI.-1, and a and 3 are tuneable hyperparameters. The first term ensures that samples from the current task are classified correctly. The second term consists of a classification loss and a mean squared error (MSE) based distillation loss applied historical samples.
Scaled Distillation (CAL-SD) (S) a new objective proposed this work defined via:
a DRL (sotttmaa)x()))) |1 f(x':0) (( +
Creplas (0) 2
E (w'.0/ Br
(2)
and then (SD A X₂ (0) + (1 A1) Creplay (0) where A & [DAI/(Da)+|Diaa-I)) Similar to CAL-DER, Creplny is a sum two terms: a distillation loss and a classification loss. The distillation loss expresses the KI- divergence between the posterior probabilities produced by f and softmax(2'), where 2' is defined in the DER section. We use KL-divergence instead of MSE loss on the logits so that the distillation and the classification losses have the same scale and dynamic range, and also since it allows the logits to drift by constant term that does not affect the softmax output but does effect the MSE loss. a E [0, 1] is tuneable hyperparameter.
The weight of each term is determined adaptively by a ciaabilitypplattciiii plasticity dilemmaa is commonly found in both biological and artificial neural networks (Abraham & Robins 2005; Mermillod et al. 2013). network stable if it can effectively retain past information but cannot. adapt to new tasks efficiently, whercas a network that is plastic can quickly learn new tasks but is prone to forgelting. The trade-off between stability and plasticity is a well-known constraint in CL (Mermillod let al.) 2013). For CAL, want the model to be plastic early on, and stable later We apply this intuition with de: higher values indicate higher plasticity, since minimizing the classification error of samples from the current task is prioritized. Since Dia-1 increases with 4, d decreases and the model becomes more stable in later training rounds.
trade-off term A A stability-
Scaled Distillation w/ Submodular Sampling (CAL-SDS2) CAL-SDS2 is another new CL approach we introduce in this work. CAL-SDS2 uses CAL-SD to regularize the model and uses a submodular sampling procedure to select a diverse representative set of history points to replay. Submodular functions are well- known to be suited to capture notions of diversity and representativeness (Lin & Bilmes 2011 Wei et al. 2015 Bilmes 2022) and the simple greedy algorithm can approximately maximize, under cardinality constraint.
monotone submodular function up to - e constant factor multiplicative guarantee (Fisher et al.
1978
Minoux 1978
Mirzasolciman et al. 2015). We define our submodular function G as:
G(S) A
Σ max Wij + A Allog 1+ Σ EMaa)) FEES
(3)
28 ZEA
The first term is a facility location function, where Wi is a similarity score between samples x; and zy. We use Wiy exp (-11z z 2 where z; is the penultimate layer of model for x4 and is hyperparameter.
7

---

Published in Transactions on Machine Learning Research (11/2023)
The second term is concave over modular function (Liu et al. 2013) and h(z;)) is standard AL mcasure of model uncertainty, such as entropy of the model's output distribution. Both terms are well known to be monotone non-decreasing submodular, as is their non-negatyvely weighted sum (Bilmes] 2022). The core reason for applying a concave function over a modde-uccettinty-scor--- keeping it as a pure modular fuuction, is to better align the modular uncertainly values with the facility location function. Otherwise, if we do not apply the concave function, the facility location function dominates during the early steps of the greedy algorithm and the modular function dominates in the later steps of greedy. In order to speed up SDS2 (to avoid the need to perform forward passes over the entire history before single step), we randomly subsample the history indices to produce A and re-compute forward passes to produce fresh 2; values for E A before re-constructing G(S), and we then perform submodular maximization; thus S C A C D The objective of CAL-SDS2 is to ensure that the set of samples that are replayed are both difficult and diverse, similar to the motivation of the heuristic employed in Wei et al. 2015
modular function, instead of
Baselines All proposed CAL methods are compared against two AL baselines. The first baseline is standard active learning, denoted as AL, which is no different from the procedure shown in Algorithm [ We also consider active learning with warm starting (AL w/ WS), which uses the converged model from the previous round to initialize the model for the current round. Both models retrain on the entire dataset at cach query round.
5 Experiments and Results
Aug ralaive socurey
COLA
CFAR1O
1,001
FMNIST
1.000
MNINNNST
CaIope
Amazzo Poletty
3.010
1.002
0.876
1.015
CAL MR
9.005
1.034
1.000
0.500
1,210
1.000
CAL-COR
CAL-SD
1,000
0.599
0.825
1.005
W/W
0.985
1.000
X
0.800
1.000
Average Speedup
Figure 4: Relative accuracy vs. speedup averaged over difffernnt labeling budgets. For every dataset, we have at least one CAL method, that is faster with same accuracy than baseline (lop right region is desiderata).
We evaluate the validation performance of the model when we train different fractions (b/n) of the full dataset. We compute the speedup attained by a CAL method by dividing the wall-clock training time of the bascline AL method over the wall-clock training time of the CAL method. We test the CAL methods on a variety of different datasets spanning multiple modalities. Two baselines do not utilize CAL which are standard AL (active learning) as well as AL w/ WS (active learning with warm starting but still training using all the presently available labeled data).
Our objective to demonstrate: (1) at least one CAL-based method exists that can match or outperform standard active learning technique while achieving a significant speedup for every budget and dataset and (2) models that have been trained using a CAL--baeed method behave no differently than standard models. We emphasize that the purpose of this work is not to champion a single method, but rather to showcase an assortment of approaches in the CAL paradigm that achieve different, performance/spesdup trade-offs. Lastly, we would like to point out that some of the CAL methods are ablations of each other. For example, CAL-ER is ablation for CAL-DER (or CAL-SD) when we replace the distillation component. Similarly, CAL-SD is ablation of CAL-SDS2, where we remove the submodular selection part.
5.1 Datasets and Experimental Setup
We use the following datasets, which span a spectrum of data modalities, scale (both in terms of dataset size, and model's computaatonaa//mmorry footprint), and class balance.
FMNIST: The FMNIST datasct consists of 70,000 28 x8 grayscale images of fashion items belonging to 10 classes Xiao et al. 2017). ResNet-18 architecture (He et al., (0066) with SGD used. We apply data augmentation, in
Beck et al. 2021) consisting of random horizontal flips and random croppings.
8

---

Published in Transactions on Machine Learning Research (11/2023)
CIFAR-10 consists of 60,000 32x32 color images with 10 different categories (Krizhevsky 2009 We use a ResNet-18 and use the SGD optimizer for all CIFAR-10 experiments. We apply data Sugmentation consisting of random horizontal flips and random croppings.
CIFAR-10:
MedMNIST: We use the DermaMNIST dataset within the McdMNIST collection Yang et al. 2021a by for performance evaluation of CAL on medical imaging modalities. Tt consists of 3-color channel dermatoscopy
images of 7 different skin diseases, originally obtained from [Codella et al. (2019); Tschandl al. (2018). A ResNet-18 architecture is used for all DermaMNIST experiments.
Amazon Polarity Review: (Zhang et al. 2015 is an NLP dataset consisting of reviews from Amazon
and their corresponding star-ratings (5 classes) which was used for active learning in Coleman et al. (2020b). Similar to the previous work we consider a. lotal unlabeled pool of size 2 million sentences and use a VDCNN-9 (Schwenk et al. 2017) architecture trained using Adam.
COLA: COLA WWrsttadt et al. 2018) aims to check the linguistic acceptability of sentence via binary classification. We consider an unlabeled size-7000 pool similar to Ein-Dor el. al. (2020) and use BERT (Devlin
et al. 2019 backbone trained using Adam.
Single-Cell Cell Type Identity Classification: Recent single-cell RNA sequencing (scRNA-seq) technolo- gies have enabled large-scale characterization of hundreds of thousands to millions of cells in complex tissues, and accurate cell type annotation is a crucial step in the study of such datasets. To this end, several deep learning models have been proposed to automatically label new scRNA-seq datasets (Xie et al., 2021 The HCL dataset is highly class-imbolanced and consists of scRNA-seq data for 562,977 cells across 63 cell types represented in 56 human tissues (Han et al. 2020). We the ACTINN model (Ma & Pellegrini 2019). four-layer multi-layer perceptron that prediets the cell-type for each cell given its expression of 28832 genes, and uses an SGD optimizer.
fyperparameters: Details about the specific hardware and the choices of hyperparameters used to train models for each technique can be found in Appendix A.5
Active Learning setup: As done in previous work (Coleman et al. 2020b Killamsetty et al. 2021a] for CIFAR10 and Amazon polarity review, budgets go from 10% to 50% in increments of 10% (results for other query sizes are presented in Appendix C.3 For FMNIST, MedMNIST, and Cell-type datasets, it goes from 10% to 30% in increments of 5%. Lastly, for COLA, we follow budget using absolute sizes from 200 to 1000 in increments of 200 (similar to Ein-Dor et al. (020)) We adopt the AL framework proposed in Beck et al. (2021] for all experiments. In the main paper, we here present results for an uncertainty sampling-band based acquisition function. However, we provide results using other acquissiiion functions Appendir
5.2
Performance vs Speedup
Budget=
Budget
Budget 30%
Aug. reliive accuncy
BACER CALAIR
Figure 5: Relative accuracy vs. speedup averaged over different datasets. For every given budget, we have at least one CAL method that is faster with the same (if not better) accuracy than the baseline.
1.00
1.00
X
1.00
CAL-OOR CAU-DD
0.99
1.00
0.99
CAL-S002 AL of We
0.98
0.99
AL
0.99
0.98
1
2 3
Average Speedup
Our goal is to show that for every dataset and budget, there exists at least one CAL method that performs equivalent (if not better) than bascline AL. However, with raw accuracics, it is difficult to compare different CAL methods and baseline AL over different. datasets at different. budgets. Therefore, we begin by observing the relative gain in accuracy over the AL baseline. Relative gains further make it feasible to take an average across the budgets, for given dataset, and take averages across the datasets, for a given budget. For every budget, we normalize the accuracies of each method by that of baseline AL. This makes the baseline accuracy always 1, irrespective of budget and dataset. Relative performances greater than indicate better than baseline accuracy (and the opposite for the less than 1 case). Having said that: (1) keeping the budgets fixed to 10%, 20%, and 30% and averaging over the datasots (except COLA, since it has a different budget)
We follow the query set sizes from Don
2020
9

---

Published in Transactions on Machine Learning Research (11/2023)
1.04
1.00
Figure 6: Comparison of CAL methods with the baseline on CIFAR-10C; absolute accuracies on individual bench- mark and its average difference with the baseline AL are reported in Appendix [A.2 table S and [0 respectively. All CAL methods perform within standard deviation of the slandard AL baseline, and CAL-SDS2 achieves the highest robust accuracy on average.
Aut rebivv Sc0/00)
1.01
1.00
0.99
0.97
0.98
CAL-DER proosdures
give us Figure 5 (2) koeping the dataset fixed, averaging the relative accuracy vs. speedups across different budgets give us Figure 4 and (3) further averaging the above across different datasets give us Figure I Methods the top right corner are preferable. For space reasons, we display only relative accuracies in the main paper, but all absolute accuracies and standard deviations for each method for every dataset and every budget are available in Appendix A.2
From the overall and per-dataset. depiction of CAL's performance (Figures and 4 respectively), it is evident that there erists a CAL method that attains a significant speedup over a standard AL technique for every dataset and budget while preserving test set accuracy. From Figure + we can further see that for some datasets (such as FMNIST and CIFAR-1O), CAL-ER, a non-distillation, and uniform sampling sampling-based method, only incur a minor drop in performance but attain the highest speodup. This suggests that naively biasing learning towards recent. tasks can be sufficient adapt the model to new set of points between AL rounds. However, we show in Figure it is not universally true for all the datasets (at different budgets). Hence, the methods which include some type of distillation term (CAL-DER, CAL-SD, CAL-SDS2) generally perform the best out of all CAL methods. We believe that the submodular sampling- based method (CAL-SDS2) can be accelerated using stochastic methods and results improved by considering other submodular functions, which we leave as future work. It should mentioned, however, that the concave function on h(x;)) was essential for CAL-SDS2's performance.
5.3 Comparison Between Standard and CAL Models
In this section, we next assess whether CAL training has any adverse effect on the final model's behavior. We first demonstrate that CAL docs not result in any deterioration of model robustness (Section 5.3.1 We then demonstrate that CAL models and baseline trained models are uncertain about similar set of unseen examples (Section 5.3.2). Lastly, in appendix A.3 we provide a sensitivity analysis of our proposed methods, where we demonstrate that CAL methods are robust to the changes to the hyperparameters.
5.3.1 Robustness
Test time distributions can vary from training distributions, so it is important to ensure that models generalize across different domains. Since models trained using CAL methods require significantly fewer gradient steps, the modified training procedure may produce fickler models that are less robust to domain shifts. To ensure against this, we evaluate CAL-method-trainee model robustness in this section. We consider CIFAR-10C Hendryeks Dietterich 2019), a dataset comprising 19 different corruptions each done at 5 levels of severity. For each model trained up to a 50% budget, we report the average classification accuracy over each corruption and compare it against the baseline in Figure each result is an average of over three random seeds. We note that most of the CAL methods perform statistically similarly standard active learning, all while providing significant acceleration. Moreover, average across all the tests, models trained with CAL-SDS2 are better than the models trained with CAL-DER, where we see difference of as much as 5% in corruptions such as glass blur; please refer to appendix A.3 table g table 9 and 10 for the absolute accuracies and statistical tests. Submodular sampling replays a diverse representative subset of history which is likely the reason behind CAL-SDS2's better robustness. The relationship of diversity with
robustness has also been explored previous works including Killamsetty et al. (2021b): Fang et al. (2022); Rozen et al. (019 Gong al. (2018).
10

---

Published in Transactions on Machine Learning Research (11/2023)
5.3.2 Correlation of Uncertainty Scores
For models trained using CAL techniques to be used as valid substitutes for standard AL models, these two classes of models need to query similar samples at each AL round. This is particularly important if AL is being used solely as a data subset selection procedure (where the user is concerned about the quality of the resulting labeled dataset as opposed to the final model). When using uncertainty sampling as the AL acquisition function, computing the Pearson correlation between the entropy scores of bascline and CAL models on the validation set after every query round is one way of determining this. Since most AL policies incorporate some notion of uncertainty, we hypothesize that the results of this experiment should extend to other acquisition functions as well. A similar analysis is done in Coleman et al. (2020b).
As seen in Figure 7 the Pearson correlation between all pairs of models is positive at every AL query round. Thus, the nature of samples that the models are uncertain about, and thus are likely to be chosen at each round, is similar between the CAL-trained models and bascline AL-trained models. A break- down of these correlations at every round is provided
In the grid of cells below-left, we show the mean, max, std. (clockwise starting left each cell) of cor- relations of the entroples of the models trained using each the seven techniques, where the statistics in each are computed
CAL-DIER
the different active query rounds. In general, high mean, and low std. correlation implies that the unlabeled samples each model is uncertain about are simi- lar each other. We positive mostly strong correlations and low stds, implying that CAL-based methods are not changing the inherent behavior of the active leaming processes, provid- ing further evi- dence for the utility
CAL-S
0,39
1.97
CAL-S052
100
in Figure [4 in the Appendix.
64
6.99
MAR
CALER
6 Conclusion and Future Work
6.42
031
60
We proposed the CAL framework, the first method to circumvent the problem of having to retrain mod- els between batch AL rounds. Across vision, natural language, medical imaging, and biological datasets, we show there is always CAL-based method that either matches or outperforms standard AL. while achieving considerable speedups. Since CAL is inde- pendent of the model archilecture and AL strategy, this framework applies to a broad range of settings.
CAL-MIR
690
355
197
6.44
6.
68
6M
617
the CAL
AL
033
844
80
80
045
10
methods.
AL ww/WS w5 ta
C4r
94
647
324
647
D43
127
220
CAL-DER
CAL-SO
CAL-SD52
CAL-ER
CALMIR
AL
AL W5
Figure 7: Mothod Entropy Correlation Statistics.
Empirical Extensions Future empirical directions may include the following: (1): CAL reduces the training time of the model, but not the AL query time although query time reductions could be an offshoot of this work; (2): CAL operates using existing AL query acquisition functions, but it is possible to tailor acquisition functions for CAL methods yielding further generalization and computational improvements; (3): while SDS2's use of submodularity helped robustness, additional submodular strategies can be used to further improve results and also for diverse AL query selection; and (4): CAL provides novel application for CL; future CL work can be partially assessed based on its CAL performance.
Theoretical Extensions The discrepancy--aseed Active Learning framework is promising to establish generalization error bounds for the replay-based CAL framework. Specifically, leveraging the generalization results presented in Theorem 2 of Cui & Sato (2020), the upper bound on the risk function based on true data distribution (referred to as IP in the theorem) is contingent upon the empirical distribution of the presently labeled data (referred to as Q in the theorem), that includes the newly acquired query batch.
In contrast to utilizing the entirety of the available labeled data, CAL strategically opts for subset of this data through the employment of replay methods, for acceleration. Due to this, the empirical distribution of examples chosen by CAL (call it Q') could deviate from Q. Consequently, one can modify the upper bound of the risk function based on true data distribution in Theorem 2 by adding the error term (which would depend on some notion of distance between Q and Q') which vanishes when we consider all the currently labeled examples, thus, recovering standard AL.
Acknowledgments
This work was supported in part by the CONIX Research Center, one of six centers in JUMP, a Semiconductor Research Corporation (SRC) program sponsored by DARPA, by the National Science Foundation under
11

---

Published in Transactions on Machine Learning Research (11/2023)
Grant Nos. IIS-2106937 and IIS-2148367, and by NIH/NHGRI U01 award HG009395. We thank Lilly Kumari, Tianyi Zhou, Shengjie Wang and all other MELODI lab members for their helpful discussions and feedback. We also thank the TMLR Action Editor and reviewers for their constructive comments.
References
Wickliffe C Abraham and Authony Robins. Memory retention- the synaptic stability versus plasticity dilemma. Trends Neurosci, 28(2):73 78, February 2005.
Rahaf Aljundi, Lucas Caccia, Fugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. CoRR,, abs//190.004744 2019a.
URL attp://arxiv. org/ wwwpa///...a/a/... 1908. .47422
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. arXiv preprint a 080.08 2019b.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 3884 3894. Curran Associates, Inc., 2020. URL attps://proocecingg..ut co/ppp//p.c/pppp////0///////
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. ArXiv, Abs/1996.03671,, 2020.
Les Atlas, David Cohn, and Richard Ladner. Training connectionist networks with queries and selective sampling. Advances in neural information processing systems, 2, 1989.
Ali Ayub and Carter Fendley. Few-shot continual active learning by a robot, 2022. URL https://arxii: //arxiv org/ 24100
Maria-Florina Balcan, Steve Hanneke, and Jemnifer Wortman Vaughan. The true sample complexity of active learning. Machine Learning, 80(2-3):111-133, April 2010. doi: 10.10777/109944001--777.. URL
Nathan Beck, Durga Sivasubramanian, Apurva Dani, Ganesh Ramakrishnan, and Rishabh K. Iyer. Effective evaluation of deep active learning image classification tasks. CoRR, hts/2106.15322. 2021. URL
https://arxiv org/ www./bg///////o..ee....oeeeaeeee hwtpg//bb///0...... 2106 2132.15324411111
Emily M. Bender, Timmit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, pp. 210-623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi: (0.1145/344218884445522 URL
https://boi.orr/11..1455.3 0ttp//10.114//344211888 3445222
Jelf A. Bilmes. Submodularity in machine learning and artificial intelligence. CoRR, abs 2tlp/2222.0013,, 2022.
URL pttpee//axxx..orr///////20000000
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and SIMONE CALDERARA. Dark
experience for general continual learning: a strong, simple basclinc. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.
15920 15930. Curran Associates, Inc., 2020.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In ICLR, 2019.
Arslan Chaudhry, Albert Gordo, David Lopez-Pazz Puneet K. Dokania, and Philip Torr. Using hindsight to an- chor past knowledge in continual learning, 2020. URL http///////////pppoe///////////a 1244KS
net/forum?idd-lee
12

---

Published in Transactions on Machine Learning Research (11/2023)
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M. Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic); 2019. URL htttps://arxa http://wxxvvvog..aaaa// www.aavv.ggga///abeeeeeee hwtpg//bb//992.. 03368
David A. Cohn, Les E. Atlas, and Richard E. Ladner. Improving generalization with active learning. Mach. Learn., 15())2201 221, 1994. URL
Cody Coleman, Edward Chou, Sean Culatana, Peter Bailis, Alexander C. Berg, Roshan Sumbaly, Matei Zaharia, and I. Zeki Yalniz. Similarity search for elficient active learning and search of rare concepts. CoRR, hes//000700007,, 2020a. URL https://arxiv org. wwtggabb/2200..000
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Effficient data selection for deep learning. In International Conference on Learning Representations, 2020b. URL http:s///ppaarrrreeeeer/////r/rp
Zhenghang Cui and Issei Sato. Active learning using discrepancy, [ttpp:://rallooromm..g.ttbbb//bb _e/7_cui__u _al___aaerr__ pdf 2020. [Accessed 21-11-2023).
Ido Dagan and Sean P. Engelson. Committec-based sampling for training probabilistic classifiers. In Proceedings of the Twelfth International Conference International Conference on Machine Learning, ICML pp. 150-157, San Francisco, CA, USA, 1995. Morgan Kaufmann Publishers Inc. ISBN 1558603778.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, (177)3336-33885 2022. doi: 10.1109/TPAMI 2021.3057446.
Jacob Devlin, Ming-Wi Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019.
Payal Dhar. The carbon impact of artificial intelligence. Nature Machine Intelligence, 2(8) 425, Aug 2020. ISSN 2522-5839. doi: 10.1038/s42256- 000-0000000229- URL httt::///0b..or///00000////22222
Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. Active Learning for BERT: An Empirical Study. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp.
Online, November 2020. Association for Computational Linguistics. doi:
URL https:://cllatthol.n 2rtp:///ooooyyor//220000000 emm00-mainmmmm..mmm 668
Seyda Ertekin, Jian Huang. Leon Bottou, and Lee Giles. Learning on the border: Active learning in imbalanced data classification. In Proceedings of the Sirteenth ACM Conference on Conference on Information and Knowledge Management, CIKM '07, pp. 127-136, New York, NY, USA, 2007. Association for Computing Machincry. ISBN 9781595038039. doi: 10. 011114/113144001322222 0021440.131444
URL
1145/
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip), 2022.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual lcarning. arXiv preprint N.N.xv805.0097 2018.
M.L. Fisher, G.L. Nemhauser, and L.A. Wolsey, An analysis of approximations for maximizing submodular set functions II. In Polyhedral combinalorics, 1978.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognilive sciences, 3(4): 128-135, 133-135,11 1999.
Zhiqiang Gong, Ping Zhong, and Weidong Hu. Diversity in machine learning. CoRR, abs/ 1////1870.07777 2018. URL http:// arxiv.v orggabs//8070.....
13

---

Published in Transactions on Machine Learning Research (11/2023)
Andrew Guillory, Erick Chastain, and Jeff Bilmes. Active learning as non-convex optimization. In Twelfth International Conference on Artificial Intelligence and Stalistics (ATSTAT), Clearwater Beach, Florida, April 2009.
Xiaoping Han, Ziming Zhou, Lijiang Fei, Huiyu Sun, Renying Wang, Yao Chen, Haide Chen, Jingjing Wang, Huanna Tang, Wenhao Ge, Yincong Zhou, Fang Ye, Mengmeng Jiang, Junqing Wu, Yanyu Xiao, Xiaoning Jia, Tingyue Zhang, Xiaojie Ma, Qi Zhang, Xueli Bai, Shujing Lai, Chengxuan Yu, Lijun Zhu, Rui Lin, Yuchi Gao, Min Wang, Yiqing Wu, Jianming Zhang, Renya Zhan, Saiyong Zhu, Hailan Hu, Changchun Wang, Ming Chen, He Huang, Tingbo Liang, Jianghua Chen, Weilin Wang, Dan Zhang, and Guoji Guo. Construction of a human cell landscape at single-cell level. Nature, 581 (7808)))3330 309 March 2020.
Steve Hanneke. A bound on the label complexity of agnostic active learning. In Proceedings of the 24th International Conference on Machine Learning, 1CML '07, pp. 353-300 New York, NY, USA, 2007.
Association for Computing Machinery. ISBN 9781595937933. doi: 10. 10.1445/1774466.775511 URL
https://doi.org// 1115//221444 1273541
Steve Hanneke. Theoretical foundations of active learning. Carnegie Mellon University, 2009.
Steve Hanneke. Activized learning: Transforming passive to active with improved label complexity. Journal of Machine Learning Research, 13(49):1469 1587, 2012. URL http://jmlr. httpa//////pppppe///////eeeeeee html
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10.1109/CVPR.2016..00
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Zhihao Jia, Colin Unger, Wei Wu, Sina Lin, Mandeep Baines, Vinay Ramakrishnaiah Carlos Efrain, Nirmal Prajapati, Pat McCormick, Jamaludin Mohd- Yusof, Xi Luo, Dheevatsa Mudigere, Jongsoo Park, Misha Smelyanskiy, and Alex Aiken. Unity: Accelerating DNN training through joint optimization of algebraic transformations and parallelization. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), Carlsbad, CA, July 2022. USENIX Association. URL https: //www. usenix org/ confereecceosdi2:: http:///sssseeepppeetttattoooo/ presentation//ia
KrishnaTepy Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh K. lyer. GLISTER: generalization based data subset selection for efficient and robust learning. In Thirty- Fifth AAAI Conference on Artificial Intelligence, AAAT 2021, Thirty-Thaie Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 8110-8118. AAAI Press, 2021a. URL https://ojs. aaai org/ hts//indxx.rom/ php/ http:///aaaatteeeieeeeeeeeeaaeee
Krishna Teja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh K. Iyer. RETRIEVE: coreset selection for efficiont and robust semi-supervised learning. CoRR, abs/2106.07766 2021b. URL https://arxiv. org/
0ts/2106.07766)
James Kirkpatrick, Razvan Pascamu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barrinskar Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521 3526, 201 7a. doi: 10. po073//aas.11118333 URL https:/ /tte:://wwwwww www. www.wwww..............o pnas. org/ doi /wwaa/aa//100 10..b//......../...eee.... 1611835114
James Kirkpatrick, Razvan Pascamu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521 -3326,,,,,,,-- 2017b. doi: 10. 10.1073/pmm) /nas.. 161 1835114.
Kieran Milan, John Quan, Tiago Ramalho, Agnieszka
14

---

Published in Transactions on Machine Learning Research (11/2023)
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Ctrabska-Barwinskee et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 3526, 2017c.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acqui- sition for deep bayesian active learning. In H. Wallach, H. Larochelle, A. Beygekzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL httpss///proceeding: www.rseeee.rp...e.. http://awwna....er..e.op//appppp pdf.
Alex Krizhevsky. Learning multiple layers of features from tiny images. pp. 32 33, 2009. URL https: //www. CB. www.ttooontteeommm parprtt.eet//zrrr//eeemgggggreee
David D. Lewis and Jason Catlett. Hoterogencous uncortainty sampling for supervised learning. In William W.
Cohen and Haym Hirsh (eds.), Machine Learning Proceediings 1994, pp. 148 156. Morgan Kaulmann, San Francisco (CA), 1994. ISBN 978-1-55860-333 335-6. doi: https://doo..rrg100.00110 1-778-1-588-00- 6.50026-X.
URL
David D. Lewis and William A. Gale. soquential algorithm for training text classifiers, 1994. URL
http://arin htt.aaaiii...aaaaaaaaaeaeaaeeaae owg.aos/ http:/cmm1g19990000/00 active learning roots.
Haoran Li, Aditya Krishnan, Jingfeng Wu, Soheil Kolouri, Praveen K. Pilly, and Vladimir Braverman.
Lifelong learning with sketched structural regularization. CoRR, abs/2104.086044, 2021. URL https
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEFE signal processing mogazine, 37(3):50-60) 60, 2020.
Xiao Li and Jeff Bilmes. A Bayesian divergence prior for classifier adaptation. In Artificial Intelligence and Statislics, pp. 275 282. PMLR, 2007.
Zhizhong Li and Derek Hoiem. Learning without forgetting, 2017.
Hui Lin and Jeff Bilmes. A class of submodular functions for document summarization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 510 520, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://cccannhoogg..rrrg////////
Yuzong Liu, Kai Wei, Katrin Kirchhoff, Yisong Song, and Jeff Bilmes. Submodular feature selection for high-dimensional acoustic score spaces. In 2013 IEFE International Conference on Acoustics, Speech and Signal Processing, pp. 7184-7188, 2013. doi: 10.1109/ICASSPP.2 201.5.59902111.633555
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Aduances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Feiyang Ma and Malteo Pellegrini. ACTINN: automated identification of cell types in single cell RNA sequencing. Bioinformatics, 36(2):533-553 07 2019. ISSN 1367-4803. doi: 10.1093/ 10.1093/bioinformal bioinformaties/btzaee
URL http:///...rrr......///ooooootoa
Zheda Mai, Dongsub Shim, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Adversarial shapley value experience replay for task-free contimual learning. CoRR, abs/2009 abs/2009.00093, 2020. URL
https://arxiv.. http://////abb//b//eeee 00093
Antoine De Mathelin, François Dehooger, Mathilde MOUGEOT, and Nicolas Vayatis. Diseropancy-based active learning for domain adaptation. In International Conference on Learning Representations, 2022.
URL
15

---

Published in Transactions on Machine Learning Research (11/2023)
C. Mayer and R. Timofte. Adversarial sampling for active learning. In 2020 IEEE Winter Conference Applications of Computer Vision (WACV), pp. Computer Society, doi: 10.1109 /ttp//AACV455.22220.000000 org/ hkg/10.1109//// 100.9//////AC5557222. 2020. 9093556
Los Alamitos, CA, USA, mar 2020. IEFE URL https://doi. htt.p/po...ee.cooopptorococciei
James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, )):41): 1995.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Martial Mermillod, Aurélia Bugaiska, and Patrick BONIN. The stability-plasticit dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in Psychology, 4, article/
2013. ISSN 1664-1078. doi: 10.3389/fpsyyg (0008//fpsg..213.00000 URL 10. (w.33389ffpppp 2013.00500
M. Minoux. Accelerated greedy algorithms for maximizing submodular set functions. In Optimization Techmiques, 1978.
Baharan Mirzasoleiman, Ashwinkumar Badanidiyuru, Amin Karbasi, Jan Vondrák, and Andreas Krause. Lazier than lazy greedy. In Proceedings of the AAAT Conference on Artäficial Intelligence, volume 29, 2015.
Matthias Perkonigg, Johannes Hofmanninger, and Georg Langs. Continual active learning for efficient. adaptation of machine learning models to changing image acquisition. In Lecture Notes in Computer Science, Lecture notes in computer science, pp. 649-660. Springer International Publishing, Cham, 2021.
Robert Pinsler, Jonathan Gordon, Eric Nalisnick, and José Miguel Hernández-Tobato. Bayesian batch active learning sparse subset approximation. In H. Wallach, Н. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https:// wwwpoiiiinnnn.uuiss..... / 04224500000177776444444444448880
pdf
R Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychol Rev, 97(2):285 308, April 1990.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learn- ing to learn without forgetting by maximizing transfer and minimizing interference. CoRR, abs/1810.11910, 2018.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2):123 146, 1995. doi: 10. 100800/085409995300333333 URL (tt.:ttree//aollo.r///0/000////0 http://00...00094009999555533311
Ohad Rozen, Vered Shwartz, Roce Aharoni, and Ido Dagan. Diversify your datascts: Analyzing generalization via controlled variance in adversarial datasets. In Proceedings of the 23rd Conference Computational
Natural Language Learning (CoNLL), pp. 196-205 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/vv v 111--199111
URL https://aclaatthhoolggo 1ttpa//tttoooooooooo//-/11111110
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai. Commun. ACM, 63(12):54 63, nov 2020. ISSN 0001-0782. doi: 10.1145/338187) URL https:///ol: pttps:://od..org/10.111193 1145/338.11
Holger Schwenk, Loic Barrault, Alexis Conneau, and Yann LeCun. Very deep convolutional networks for text. classificaaton. In EACL, 2017.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach, 2017. URL htttpps//aaxxiv..r/////////////
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018. URL /ttp::///oeeerrreieee. wwt.:rrriiiieeeouoonouoid..
Hialuk-RM RW
16

---

Published in Transactions on Machine Learning Research (11/2023)
Yuya Senzaki and Christian Hamelain. Active learning for deep neural networks on edge devices, 2021. URL
2106..10 10836
Burr Settles. Active learning. Morgan & Claypool Publishers, 2012.
Burr Settles. Active learning literature survey, Computer Sciences Technical Report 1648, University of Wisconsin Madison, 2009. URL http://axon: cs. btt.e//////aattnnn///ssss/////pp /settls
activelearning. pdf
Burr Settles. From theories to queries: Active learning in practice. In Isabelle Guyon, Gavin Cawley, Gideon Dror, Vincent Lemaire, and Alexander Statnikov (eds.), Active Learning and Experimental Design workshop In conjunction with AISTATS 2010, volume 16 of Proceedings of Machine Learning Research, pp. 1 18, Sardinia, Italy, 16 May 2011. PMLR. URL https: httppsss//pooceiings... mlr. wtt.....r...//oo/sttteellll..e.. html
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. CoRR, htts/199.000053, 2019. URL
08053
Changjian Shui, Fan Zhou, Christian Gagné, and Boyu Wang. Deep active learning: Unified and principled method for query and training. CoRR, abs/ abs/1911.09112, 2019. URL httpp://arxvv.org/ http:///r./b//.......o 19111199110001
Michalis K. Titsias, Jonathan Schwarz, Alexander G. de G. Matthews, Razvan Pascamu, and Yoe Whye Tch. Functional regularisation for continual learning with gaussian processes. In International Conference Learning Representations, 2020. URL
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5, 2018.
Matthew Wallingford, Aditya Kusupati, Keivan Alizadeh-Vahid, Aaron Walsman, Aniruddha Kembhavi, and Ali Farhadi. FLUID: A unified evaluation framework for flexible sequential data. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL httpps://pperreviet/ wtt..eee/oormmee.......eee..
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint Xiv:::1805. 12471, 2018
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submoduarrity in data subset selection and active learning, In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, TCML'15, pp. 1954-1963. JMLR.org, 2015.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrutions, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL acw.eb..wbbb...... brgganthologg//00000 emnlp- demos. 6
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, abbs/1708.07744 2017. URL
Bingbing Xie, Qin Jiang, Antonio Mora, and Xuri Li. Automatic cell type identification methods for single-cell RNA sequencing, Compulational and Sbructurul Biotechnology Journal, 19:5874 5887, 2021. ISSN 2001- 0370. doi: atticle/ppi/5200117770000-
Jiancheng Yang, Rui Shi, and Bingbing Ni. Modmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In IEEE 18th International Symposium on Biomedical Imaging (ISBT), pp. 191-195, 2021a.
17

---

Published in Transactions on Machine Learning Research (11/2023)
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2: A large-scale weight benchmark for 2d and 3d biomedical image classification. arXiv preprint Are::21100 14795, 2021b.
Donggeun Yoo and In So Kweon. Learning loss for active learning. CoRR, abs/1905.03677,, 2019. URL http://arxiv. http://www..rr/////...........
Haoyu Zhang, Logan Stafman, Andrew Or, and Michael J. Freedman. Slaq: Quality-driven scheduling for distributed machine learning. SoCC '17, pp. 390-404, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450350280. doi: 10.1145/3127477331774440 URL https://doi.org///
1145/312747..31274
Jifan Zhang, Yifang Chen, Gregory Canal, Stephen Mussmann, Yinglun Zhu, Simon Shaolei Du, Kevin Jamicson, and Robert Nowak. Labelbench: comprchensive framework for benchmarking labcl-officient learning, 2023.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. convoleeii networks for text classification. CoRR, abs/1509.00 016266 2015. URL
01626
Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Joseph E. Gonzalez, and lon Stoica. Alpa: Automating inter- and URL https:
intra-operator parallelism for distributed deep learning. CoRR, abs/2201.12023, 2022.
Zongwei Zhou, Jae Y. Shin, Suryakanth R. Gurudu, Michael B. Gotway, and Jianming Liang, Active, continual fine tuning of convolutional neural networks for reducing annotation efforts. Medical Image Analysis, 71:101997, 2021. ISSN 1361-8415. doi:
101.0.009...00.......0..119.1..1 URL https://wwas wwwpseccceeeee...cooooooooooococ http:///...a.otooooo.oooooocc/aa
Jia-Jie Zhu and José Bento. Generative adversarial active learning. CoRR, abs/ 170..0792, 2017. URL
htggabbs/1702.07555
18

---

Published in Transactions on Machine Learning Research (11/2023)
A Additional Experimental Details on Main Results
A.1 Procedure for normalized accuracy plots
Here we again explain the procedure to generate the normalized accuracy versus speedup plots, as reported in figures I and | These plots help us to understand and compare the performance of different CAL methods against the bascline AL. For every dataset, we first get the respective accuracies and speedups of CAL methods and baseline AL, for every budget. This also reported tabular form besides the scatter A.2
plot visualizations in section
For every budget, normalize the accuracics of each method by that of baseline AL. This makes the baseline always at 1, irrespective of budget and dataset. Relative performances greater than 1 indicate better than baseline accuracy (and similarly for the less than case). Note that, for CIFAR1O and Amazon polarity review, budgets go from 10% to 50% increments of 10%. However, for FMNIST, MedMNIST, and Celltype datasets, it goes from 10% to 30% in increments 5%. Lastly for COLA, the budgets are different from the above, therefore, don't 't include that when average across the dalaset at fixed budget. Having said that,
Keeping the budgets fixed to 10%, 20%, and 30% and averaging over the datasets (except COLA, since it has a different budget) will give us figure 5
Keeping the dataset fixed, averaging the relative accuracy v.s. speedups across different budgets will give us figure
Further averaging the above across different datasets will give us the main result figure
From the results above, can infer that CAL-DER and specialized methods such CAL-SD/SDS2 always provide a speedup, all while preserving the accuracy, if not betler.
A.2 Results in Tabular Form
In this section, we expand our results mentioned in section 5.2 In particular, we report the absolute accuracies for each dataset, at every budget, and plot against the observed speedup. All methods highlighted blue are methods that use CAL. Note that all the results this section are for uncertainty-base query pool acquisiiion functions. The choice of budget scale is taken from the previous works (Beek et al.) 2021 Ein-Dor al.] 2020). Each dataset has different complexity (and train on them using different neural architectures) ) therefore they differ in the labeling budget.
FMNIST Please refer to table 2 and figure
Budget 10%
Budget 15%
Budget 20%
Budget 25%
Budget = 30%
CALER
CAL-MIR
93
CAL-DER
Accuracy
CAL-SD
CAL-SD82
94
AL
X
X
94
X
2
2
3
2
3
3
1
3
Speedup
Figure 8: FMNIST Results
19

---

Published in Transactions on Machine Learning Research (11/2023)
Test Accuracy (%)
Factor Speedup
Method
10%
15%
20%
25%
30%
10%
15%
20%
25%
30%
CAL-ER
92.6 aa
93.9
94.5
94.9
94.9 a
1.5x
1.4x
2.0x
2.4x
2.8 x
CAL-MIR
92.6
93.9
94.5
94.9
94.9
0.9
1.2x
1.3x
1.5x
1.7x
CAL-DER
92.7
93.9
94.5 1
94.8 1.
94.9 1
1.4
2.0x
2.4x
2.7x
3.1x
CAL-SD
92.6 нл
94.0
94.5 0.1
94.8 +.
94.9 #.
1.4 x
2.0x
2.4x
2.7x
3.1x
CAL-SDS2
92.6
94.0
94.6
94.9
94.9
1.1x
1.5x
1.7x
1.9x
2.1x
AI. w/ WS
92.7 A oa
93.8 0.a
94.4 a 0.1
94.6 0.03
94.4
0.0
1.1x
1.4x
1.5x
1.3x
1.5x
A1.
92.6 9.
93.8
94.4 0.1
94.9
94.9 o
1.0x
1.0x
1.0x
1.0x
1.0x
Table 2: FMNIST Results
CIFAR10 Please refer to table
and figure
Budgel m 10%
Budget 20%
Budget 30%
Budget 40%
Budget 50%
83.5
90.0
93.8
CALEER
Accuracy (%)
92.8
94.2
CAL-MIR
89.8
93.6
CAL-DER
x
92.6
*
94.0
CAL-SD
93.4
89.4
92.4
93.8
AL WS
AL
82.0
89.2
92.2
2
3
4
5
2
Speedup
Figure CIFAR-10 Results
Test Accuracy (%)
Factor Speedup
Method
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
CAL-ER
a 0.5
89.9 0.
92.4 0.1
93.5 a o.1
93.7 x 0.
1.6x
2.8x
4.0x
5.3x
CAL-MIR
82.4
89.5
92.6 6.
93.6 0.1
93.8 0.
0.7 x
1.0x
1.4x
1.8x
2.2x
CAL-DER
83.5
90.0
92.3
93.1
93.4
1.4x
2.3x
3.2x
4.2x
5.2x
CAL-SD
83.0
90.0
92.7 6.2
93.3
93.9 1.
1.4x
2.2x
3.2x
4.1x
5.1x
CAL-SDS2
82.5 0.1
90.1 u.2
92.9
94.0
94.4 u.
1.1x
1.6x
2.1x
2.7x
3.4x
AL w/ WS
81.9 1
89.6 1
92.4
93.5 a
94.1 1o1
1.2x
1.1x
1.1x
L.1x
1.1x
AL
82.0 a o.a
89.1 1 0.
92.1 0.4
93.5 a 0.a
93.8 0.0
1.xx
1.0x
1.0x
1.0x
1.0x
Table 3: CIFAR-10 Results
MedMNIST Please refer to table and figure 10
Budget= 10%
Budget 15%
Budget 20%
Budget 25%
Budget 30%
70.0
70.0
CAL-ER
70.0
Accuracy (%)
70.0
68.0
68.0
67.5
CAL-DER
CAL-SD
66.0
CAL-SDS2
80.0
66.0
65.0
X
AL
64.0
57.5
84.0
64.0
62.5
2
3
3 Speedup
2
Figure 10: MedMNIST Results
20

---

Published in Transactions on Machine Learning Research (11/2023)
Test Accuracy (%)
Factor Speedup
Method
10%
15%
20%
25%
30%
10%
15%
20%
25%
30%
CAL-ER
64.7
63.1
68.4
66.1 17
67.5
1.4x
2.0x
2.4x
2.9x
3.4x
CAL-MIR
64.1
66.2
69.2
70.7
71.8
1.0x
1.2x
1.4x
1.6x
1.9x
CAL-DER
67.1 1 1.00
67.9 1.0
69.8 1.a
71.5 1.04
72.2 a 0.6
1.2x
1.6x
2.1x
2.6x
3.2x
CAL-SD
65.1
67.8
70.6
71.1 1.1.1
72.1 1a
1,2x
1.7x
2.0x
2.5x
2.9x
CAL-SDS2
66.1
68.1
69.8
71.6
72.5
1.1x
1,5x
1.7x
2.0x
2.2x
w/ WS
67.0 a 1-4
67.5 a 0.7
69.5 0.e
70.3 1.1
71.3 1.0
1.1x
1.3x
1.5x
1.7x
2.0x
AL
67.2 0.
68.3 ana
68.8
69.0 2:00
70.9
1.0x
1.0x
1.0x
1.0x
1.0x
Table 4: MedMNIST Results
Amazon Polarity Review
Please refer to table and figure 11
Budgel 10%
Budget 20%
Budget 30%
Budget 40%
Budget 50%
93.0
94.0
94.0
93.8
94.2
CALERR
92.5
94.0
CAL-MIR
93.8
Accuracy
92.0
93.5
93.5
93.6
93.8
CAL-SD
CAL-DER
91.5
93.0
93.5
93.2
AL WS
91.0
93.2
AL
92.5
93.0
X
93.2
x
2
5
2
5
3
1
Speedup
Figure 11: Amazon Polarity Results
Test Accuracy (%)
Factor Speedup
Method
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
CAL-ER
90.7 a.
92.4 = 1.2
93.5 0.
93.7 0.2
93.8 a 0.2
1.5x
3.5x
3.3x
4.1x
4.9x
CAL-MIR
92.0
92.9 0.1
93.3 0.
93.7
93.7
0.9x
1.3x
1.8x
2.3x
2.7x
CAL-DER
92.9
94.1
94.0
93.6
94.2
1.5x
2.4x
3.2x
4.0x
4.7x
CAL-SD
92.1
92.6
93.7 o.
93.8
94.1 a
1.5x
2.4x
3.2x
4.0x
4.7x
CAL-SDS2
92.6
93.2
93.6 o
93.8
94.1
1.2x
1.9x
2.5x
3.1x
3.7x
AL w/ WS
92.6
93.0 e.
93.0 o.
93.2
93.1 a
1.0x
1.0x
1.0x
1.0x
1.0x
Al.
92.8 0.8
93.1 0.
93.3 a
93.8 a 0.5
94.1 a 0.2
1.0x
1.0x
1.0x
1.0x
1.0x
Table 5: Amazon Polarity Results
COLA
Please refer to table 1 and figure 12
Budget 2.9%
Budget 5.7%
Budget 8.6%
Budget 11.4%
Budget 14.3%
75.0
77.5
78.0
79.0
CAL-ER
Accuracy (%))
74.0
77.0
78.5
CAL-DER
75.5
78.5
77.0
78.0
CAL-SD
CAL-SDS2
X
X
AL
77.5
72.0
75.0
76.0
76.5
X
77.0
2
5
3
5
3
Speedup
Figure 12: COLA Results
21

---

Published in Transactions on Machine Learning Research (11/2023)
Test Accuracy (%)
Factor Speedup
Method
2.9%
5.7%
8.6%
11.4%
14.3%
2.9%
5.7%
8.6%
11.4%
14.3%
CAL-ER
73.2
75.4 0.
76.4
77.0
77.3 1.4
1.7x
2.8x
3.9x
5.0x
6.1x
CAL-MIR
75.1
75.5
76.6
76.5
77.0
0.8x
1.2%
1.6x
2.0x
2.4x
CAL-DER
71.5 2.7
74.9 a.a
75.7 1.5
76.9
78.3 1.8.
1.7%
2.7x
3.7x
4.8x
5.8x
CAL-SD
73.6
74.9 1.1.1
77.7
76.4
78.0 0.M
1.7x
2.7x
3.7x
4.8x
5.8x
CAL-SDS2
74.7
75.5
77.2
78.1
79.2
1.4x
2.1x
2.9x
3.7x
4.5x
w/ WS
74.6 0.7
76.1 0.
76.3 x.
76.3 a 1.5
77.2 a 0.0
1.2x
1.7%
1.6x
1.8x
1.8x
AL
73.9
75.5 2.0.
76.6
76.3
77.3
1.0x
1.0x
1.0x
1.0x
1.0x
Table 6: COLA Results.
Single-Cell Cell Cell-Type Identity
Please refer to table 7 and figure
Budget 10%
Budgel 15%
Budget 20%
Budgel 25%
Budget 30%
CAL-ER
Accuracy (%)
CAL-MIR
90
CAL-DER
88
90
CAL-SD
CAL-SD82
89
AL wi WS
84
88
AL
88
2
3
2
3
3
1
3
1
2
3
Speedup
Figure 13: Single-Cel. Cell-Type Identity Classification Results
Test Accuracy (%)
Factor Speedup
Method
10%
15%
20%
25%
30%
10%
15%
20%
25%
30%
CAL-ER
86.3
88.3
89.7 o.
90.6
91.0
1.5x
2.0x
2.4x
2.9x
3.4x
CAL-MIR
86.3
88.3
89.7 oo
90.5
90.9
1.2x
1.6x
1.9x
2.2x
2.6x
CAL-DER
o.s
88.8 0.s
# u.
90.7 + 0.2
91.2 o.
CAL-SD
86.3
88.3
89.5 o.
90.3 u.2
90.8
1.4x
1.9x
2.3x
2.8x
3.3x
CAL-SDS2
86.3
88.2+
89.2 0.
90.1
90.6+
1.4x
1.9x
2.3x
2.8x
3,3x
AL. w/ WS
86.3 0.1
88.3 o.1
88.4 a.
88.2 0.8
88.1 i 0.
1.0x
1.0x
1.2x
1.6x
1.9x
AL
83.6
87.0
88.6 0.1
80.5
89.9
1.0x
1.0x
1.0x
1.0x
1.0x
Table Single-Cell Cell-Type Identity Classification Results
A.3 Out-of-distribution (OOD) generalization
Here we show results from the figure t in tabular form in table S We compare the performance of CAL methods with baseline AL for OOD generalization. Reported mean values and standard deviations are computed over three different random seeds. As shown in the figure 0 in the main paper, and the table below, CAL methods are as robust to perturbations as baseline AL, despite the speedup. Finally, in table 10 we report that CAL-SDS2 is better compared to other CAL methods, average.
22

---

Published in Transactions on Machine Learning Research (11/2023)
Corruption
CAL-ERR
CAI-MIR
CAL-DER
CAL-SD
CAL-SDS22
Al
AL WS
saturate
90.4
90.7 02
90.4
90.5
90.8
90.7
90.8
impulse noise
52.4
53.4
53.3
53.6
55.8
54.5
48.7
defocus blur
79.6
81.6
79.3
80.6
79.6
80.9
80.4
contrast
74.6 u.
76.1 0.
73.8 0.09
75.1 =
74.2 0.8
77.1 1.4
76.4 +
frost
76.0
76.1
73.8
75.4
76.9
75.4
77.0
speckle noise
61.7
61.7
60.9
61.6
64.3
61.7
59.4
pixelate
74.4 1.3
2.2 76.7
75.2 6.9
74.8 # 1.0
76.5 # 0.7
75.9 = 1.5
77.0 #
200m blur
74.4
74.4 2.
75.4
74.1
75.9
75.3
elastic transform
82.5
83.5
82.0
82.1
82.6
82.0
82.9
spatter
81.8
82.9
82.4
82.6 1
82.8
83.0
82.5
snow
80.3 a 0.4
80.2 07
79.5 a 6.8
79.7 A
80.8 1 0.1
80.1 0.5
80.6 1:
fog
86.8
86.9 1.00
85.9
86.3
86.5
86.6
87.9
Gaussian noise
46.6
46.6
44.9
46.7
50.7
45.9
43.2
brightuess
92.1 u.s
0.
92.1 6.3
92.2 # 0.1
92.5 # 0.1
92.7 + v.1
92.6 0.2
Gaussian blur
69.7
72.3 1:
69.3 2.2
71.3 1.
69.3 #
71.6
70.4 + 1.3
motion blur
75.1
77.1 +12
74.4
75.1
74.5
74.7
76,8
shot noise
58.6
58.5 2.9
57.3 A 4.
58.6 1
61.6 1 0.5
58.3 1
56.1 1:
jpeg compression
79.0
1.5
78.4 0.05
78.5
0.00
78.6
0.5
79.1
0.
77.7 & 0.1
77.7 1 0.1.1
glass blur
51.8
54.6 2.00
48.6
50.7
9.1
53.9
49.4
52.6 2.0.1
Table Accuracy (in %) comparison of CAL methods with the baseline on the CIFAR-10C dataset. Results
were reported as an average over three random seeds. Models trained with CAL procedure perform statistically similar to the one trained with baseline AL.
Corruption
CAL-ER/AL
CAL-MIR/AL
CAL-DER/AL
CAL-SD/AL
CAL-SDS2/AL
saturate
0.3957
0.9104
0.2118
0.4516
0.7961
impulse noise
0.32009
0.5692
0.7029
0.7134
0.5374
defocus blur
0).2777
0.2794
0.1561
0.7854
0.3178
contrast
0.0335 (+)
0.3023
0.0256 (-)
0.1020
0.0250 (-)
frost
0.5793
0.6639
0,2147
0.9548
0.2778
speckle noise
0.9873
0,9022
0.7409
0.9739
0.0017 (+)
pixelate
0.2243
0,2521
0.4661
0.3913
0.5636
blur
0.2824
0.5567
0.4279
0. 0.7238
0.3910
clastic transform
0,5482
0.1295
0.9975
0.9154
0.5176
spatter
0.1504
0.9401
0.4310
0.5843
0.7800
snow
0.7823
0.8905
0.3747
0.5455
0.1976
fog
0.4715
0.2534
0.2781
0.4512
0.7142
gaussian noise
0.3514
0.7485
0.7836
0.8056
0.0013 (+)
brightness
0.0309 (-)
0,2042
0.0341 (-)
0.0091 (-)
0.0474 (-)
gaussian blur
0.2969
0.5867
0.2605
0.8658
0.2997
motion blur
0.6474
0.0368 (+)
0.6263
0.6801
0.7135
shot noise
0.72522
0.8920
0.7311
0.9091
0.0004 (+)
jpeg compression
0.0110 (+)
0.1080
0.0016 (+)
0.0286 (+)
0.0000 (+)
glass blur
0.2820
0.1047
0.7633
0.5807
0.1182
Table 9: Pairwise p-values were computed to compare each Confidence- Aware Learning (CAL) method with the standard Active Learning (AT.) approach across various corruption tests. p-values less than 0.05 are highlighted in bold, and instances, where CAL methods outperform (+) or underperform (-) compared to standard AL, are indicated. Overall, our findings suggest. that the disparities between CAL and AL methods are typically not statistically significant, implying that CAL generally does not compromise robustness. When statistically significant difference does arise, CAL (particularly CAL-SDS2) tends to outperform AL more frequently than it falls short.
A.4 Correlation between CAL and baseline AL
In figure 14 we provide the Pearson correlation between the uncertainty scores of models on the held-out test. set,, before every query round. A positive correlation between CAL models and baseline AL models
23

---

Published in Transactions on Machine Learning Research (11/2023)
Method
Average Accuracy difference (in %))
CAL-ER
-0.34
CAL-MIR
0.49
CAL-DER
-0.96
CAL-SD
-0.17
CAL-SDS2
0.63
AL w/ WS
-0.32
Table 10: The average difference in the accuracy (in %) of models trained with baseline AL from models trained with CAL, on different benchmarks across the CIFAR10-C dataset. Higher numbers are better. We can see that CAL-SDS2 is better compared to other CAL methods, on average. We hypothesize that this can be attributed to submodular sampling from history.
before every query round suggests that the nature of examples chosen by the CAL models similar that of baseline AL models. Note that each entry the correlation matrix is averaged over three random seeds corresponding to the random initialization of each model.
24

---

Published in Transactions on Machine Learning Research (11/2023)
CALOER
0.53
2.19
0:41
0:41
0:42
0/42
2.39
0.39
0/4
0:31
5:4
0.31
0:35
0.31
C4L.30
0.51
0.
0:43
0.40
9:51
0.46
0.45
0.6
0.40
0.51
0.5
0:43
08
27
0.7
0.48
0.5
0:45
0.46
0:38
0.09
0.5
0:45 0.47
2.40
0.51
0.39
0:46
0.44
0.44
0.45
0.42
-0.
0.29
0:44
0:44
0:44
0:43
0.4
-15
AI M5
5.89
0.54
0.53
3.54
0.54
0:45
0:39
0:45
0:44
0:44
0:46
0:42
0:34
0:43
0:41
0:43
0:42
0.4
1
CALOER
0.43
0:63
0:41
0:37
0:45
0:41
0:44
0:43
0.4
0.39
0.54
0.09
0.54
0.53
0:48
0:45
0.
0.9
C4L.50
0:45
0:
0.36
0:40
0.47
0.54
0.54
0.58
0.59
022
0:45
0:53
0.54
0.6
0.53
0:54
0.58
2.30
0.4
0.49
0.45
0:40
0.41
0:40
0:52
0:47
0:53
0.5
0.44
A/
0.57
0.43
8:43
0.44
0:43
0.41
0.4
0:39
0:47
0.44
0:46
0:46
0:41
0:45
0:48
0:44
6:47
0:48
0.44
1
-55
CAL
0.57
0.5
0.55
0.54
0:48
0.47
0.61
0:55
0.59
0.59
0:51
0.48
0:63
0:55
0.61
0.6
0:54
0:49
04.500
2:57
0:57
0.81
3.51
0:63
0.59
0.84
0.63
0:54
0:51
0.61
0.59
0.65
0.64
0:57
0.52
2.9
GAL5052
0.56
0.50
0.0
0:55
0.59
0:58
0.59
0:51
0:48
0.7
0.7
0.64
0.59
0.64
0:56
0.6
00.6
288
3.40
0:51
0.54
0.53
0.44
0.54
0.57
0.51
0.56
0.50
0.45
AL
2.47
-55
0.48
0:51
0.47
0.44
0.40
0.52
0.48
0.51
0.51
0:45
CAUSh
AL
ALAINS
CALOOR
0k 04/4
CA
Pearson correlation between uncertainty scores of models held out set after every query round. We are showing nine query rounds in row-major order (so the top left after the first query round, the top middle is after the second, and so on). Positive correlation of uncertainty scores suggests that the nature of examples the models are uncertain about, and thus likely to be chasen at every query round, is similar between the CAL-trained models and baseline AL-trained models.
Figure 14:
A.5 Hyperparameters
For every dataset and every CAL/AL strategy, learning rate (Ir) and batch size (m) are chosen based on whichever setting achieves the highest performance on standard AL. This means CAL methods can still improve if we tune either of learning rate batch size. Replay size critical for the performance of continual learning algorithms. On the one hand, we do not want the replay batch size, m()), be too small since then will forget some history. But on the other hand, we also do not want m²) to be too large since there will be a computational cost associated with that. Considering the mentioned constraints, for all CAL methods, we, therefore, set replay size as mhh) E {m, 2m} (used in all CAL methods). Via experinentation on subset of the datasets, we found (h) less than or greater than this range sullered either from forgetting (when too small) or extra computation without an accuracy benefit (when too large). We set a E {0.1, 0.25, 0.5, 0.75] (used in CAL-DER, CAL-SD, and CAL-SDS2), / E (0.75, 1) (used in CAL-DER). The scale and size of the search space for and / is inspired from Buzzega et al. (2020) Lastly, o (0.1,1) (used in CAL-SDS2), and À E (0.1, 1, 10) (used in CAL-SDS2).
25

---

Published in Transactions on Machine Learning Research (11/2023)
We select the configuration for each CAL model that achieves the highest accuracy. "c" is the hyperparameter used in CAT-MIR and CAL-SDS2 to subsample the history before finding the m(h) samples to replay, but this parameter is not tuned for any of the presented results. We list the specific set of hyperparameters we use for all the main experimental results in this section.
A.5.1 FMNIST
All experiments for FMNIST used ResNet-18 with SGD optimizer, with learning rate of 0.01 and batch size of 64. For all the CAL methods, we fix mh)) 128. A NVIDIA GeForce RTX 1080 GPU was used to run all the reported experiments.
CAL-MIR 256
CAL-DER a 0.1, ==1
CAL-SD a 0.25
CAL-SDS2 256, 0.25, 0.1, À
A.5.2 CIFAR-10
All experiments for CIFAR-10 used a ResNet-18 18 with an SGD optimizer, with learning rate of 0.02 and a batch size of 20, For all the CAL methods, we fix (h) 40. Training done on an NVIDIA GeForce RTX 2080.
CAL-MIR 100
CAL-DER 0.1, 3 ==1
CAL-SD 0.25
CAL-SDS2 c= 100, a = 0.25, - 0.1, = 0.1
A.5.3 MedMNIST
All experiments for MedMNIST used ResNet-18 with Adam optimizer, with learning rate of 0.001 and batch size of 128. For all CAL methods, fix m 128. All reported models were trained on NVIDIA GeForce RTX 2080.
CAL-MIR 270
CAL-DER à = 0.1, 3 =1
CAL-SD a 0.5
CAL-SDS2 c 270, a 0.5, 0.1, X 10
A.5.4 Amazon Polarity Review
Throughout our experiments, we sample 2M sentences and use them as the total training set instead. We use Adam optimizer with default parameters with a learning rate of 0.001 and a batch size of 128 for epochs. For all the CAL methods, we fix (h) 128. All reported models were trained on an NVIDIA GeForce 1080 Ti.
CAL-MIR 256,
26

---

Published in Transactions on Machine Learning Research (11/2023)
CAL-DER 0.25, 3 0.75
CAL-SD a 0.5
CAL-SDS2 c 256, a 0.75, o 1, ==1
A.5.5 COLA
For all of our experiments use Huggingface's transformer library Wolf (2020) and use a maximum sentence length of 100. We use Adam optimizer and learning rate of 5 10 batch size of 25 and m 25. Models were trained on single NVIDIA GeForce 1080 Ti.
CAL-MIR 50
CAL-DER a 0.25, 3 0.75
CAL-SD a = 0.75
CAL-SDS2 c= 50, a 0.5, o 1 1.
A.5.6 Single-Cel Cell Cell-Type Identity Classification
All experiments use SGD optimizer with standard parameters with learning rate of 0.001 and a batch size 128. For all the CAL methods, we fix (h) 128. Training done on an NVIDIA A100-PCTE-40GB.
CAL-MIR c 200,
CAL-DER a 0.1, 3 1
CAL-SD ==1
CAL-SDS2 100, a 0.25, a- 0.1, A =
A.6
Hyperparameter Sensitivity Analysis for Proposed Methods
In this section, we provide hyperparameler sensitivity analysis methods for CAL-SD and CAL-SDS2 CIFAR-10. For CAL-SD, only a is tuned and the best value is used for CAL-SDS2. Therefore, we tuned hyperparameter for CAL-SD while a and À are tuned for CAL-SDS2. Note that the differences in final test accuracy at 50% budget on CIFAR-10 across different configurations are negligible, as shown the tables below.
0.1
0.25
0.75
93.79
93.90
93.58
93.38
Table 11:
CAL-SD Sensitivity
0.1
0.5
5
10
0.1
94.20
94.27
94.32
94.34
94.16
94.33
94.22
94.28
94.15
94.44
10
94.32
94.29
94.27
94.30
94.18
Table 12: CAL-SDS2 Sensitivity
27

---

Published in Transactions on Machine Learning Research (11/2023)
B A primer on Continual Learning
We define Din Uieini D.. In CL, the dataset consists of T tasks (DI,... Dr) that are presented to the model sequentially, where Dr = {(z,, He),, Ny are the task-t sample indices, and n, = [Neel- time E [T), the data/label pairs are sampled from the current task (x, y) ~ Dr: and the model has only limited access to the history D1.--1. If the model is trained only on Dr using standard optimization algorithms, the model will exhibit catastrophic forgetting. The CL objective is to efficiently adapt the model to D, while retaining the performance the history. Given loss function e: X ) + IR initial parameters 0,-1, and model f. 8, can be obtained as the solution to the CL optimization problem (Aljundi et al. 2019b Chaudhry
et al. 2019 Lopez-Paz & Ranzato 2017
min E (H.D)~,,
{V,,S<2;0)
(d' / e v'''
(v.I(c(:0))
8.b. (/)a-Di
E
E
C Results for Additional Active Learning Strategies
In this section, demonstrate that CAL methods can accelerate AL strategies other than entropy sampling without incurring any significant performance drops. We test multiple AL strategies on FMNIST Xiao et al. (2017) and CIFAR-10 Krizhevsky (2009). Note that the speedups are approximately the same as the ones reported in Section A since the training time is generally independent of the selected AL strategy.
C.1 Overview Strategies
Margin Score Sampling This strategy is another form uncertainty sampling Settles) (0099) as described in the main paper. Instead the entropy f(x; 0), the margin score used as the entropy score i.e., h(z) & 1- (f((xx;;); f(x; where and j are the indices corresponding the highest and second highest values of (z; 0) respectively.
FASS FASS Wei et al. (2015) is a two-staged selection method that uses both uncertainty sampling and submodular maximization. Initially, a set of samples A of cardinality c* by chosen from / using uncertainty sampling, where <>> is a tuneable hyperparameter. Nexi, U, constructed by greedily selecting samples that maximize submodular set function G 2- + R,, defined on ground set A. Entropy is once again used as the uncertainty metric for the initial stage. For the second stage, G is defined to be the facility location function Wei al. 2015) expressed below:
G(S) = Σ
max Wijs
(4)
ZEES
F.EA
where S C A and way a similarity score between samples I; and Ty- our experiments, Wij exp (-1| -z = )2o2) where z; is the penultimate layer representation of model f for I, and o hy- perparameter.
GLISTER GLISTER Killamsetty el al. 2021a) solves bi-level optimization problem in order to select
samples to label. Specifically, GLISTER solves
LLy (argmax (B, Sc
argmax
(5)
where Lyy is the log-likelihood on the validation set V, and Lor is the logelikelihood the subset. S.
28

---

Published in Transactions on Machine Learning Research (11/2023)
C.2 Results
Test Accuracy (%)
Method
10%
15%
20%
25%
30%
CAL-ER
92.8 o.1
94.1
94.8
95.1
95.2
CAL-MIR
92.6
94.1
94.9
95.0
95.2
CAL-DER
91.8
93.1
94.3
94.6 0.1
94,8 0.2
CAL-SD
92.5
93.8
94.8
95.
95.2 0.0
CAL-SDS2
87.8 1.1
93.4 0.1
94.6 # 0.1
95.0 0.2
95.2 0.1
AL w/ WS
92.8
94.0
94.6
94.8
95.0
AL
92.7
94.1 0.3
94.9
95.0
95.2
Table 13: FMNIST with Margin Score Sampling
Test Accuracy (%)
Method
10%
20%
30%
40%
50%
CAL-ER
81.5
89.3
92.2+ 0.2
93.4-
93.8 +
CAT-MIR
81.9
89.6
92.2
93.6
94.0
CAL-DER
83.0 0.2.2
89.5
92.2
93.2
93.6 0.0
CAL-SD
82.6 0.4
89.9
92.4
93.5
93.8 0.2
CAL-SDS2
82.5 +.2
90.2 0.2
92.5 0.2
93.8 0.2
94.1 0.1
AL w/ WS
83.1
90.3
93.0
93.5
93.6
AL
75.1
87.1
90.2+ o.s
92.0=
92.8 0.5
Table 14: CIFAR-10 with Margin Score Sampling
Test Accuracy (%)
Method
10%
15%
20%
25%
30%
CAL-ER
92.6 1 0.1
93.9 1 0.2
94.6
95.0
94.9 i 0.0
CAL-MIR
92.5 4.1
93.8
94.6
94.8
94.9
CAL-DER
92.7 0.1
93.8
94.5
94.7
95.0 +.2
CAL-SD
92.8 0.
93.9 1 0.1
94.7
94.8 0.3
94.9 1 0.1
CAL-SDS2
92.8 + 0.0
93.8
94.5 0.1
94.8 0.2
94.9 + 0.1
AL w/ WS
92.5
93.8
94.0
94.3
94.3 + 0.0
AL
92.7
93.9
94.5
94.7
94.8
Table 15: FMNIST with FASS
29

---

Published in Transactions on Machine Learning Research (11/2023)
Test Accuracy (%)
Method
10%
20%
30%
40%
50%
CAL-ER
82.2
89.8
92.5+
93.4+
93.7
CAT-MIR
82.2 0.3
89.4
92.3 0.1
93.4 0.0
93.5 4 0.1
CAL-DER
83.1
1 0.3
89.7
91.9
93.1 0.2
93.5 + 0.1
CAL-SD
83.0
90.0 0.a
92.5 à o.1
93.5 = 0.
94.0 0.1
CAL-SDS2
83.0 0.1.1
90.1 0.1
92.7 0.2
93.5 0.2
94.0 0.0
AL w/ WS
82.8
90.3 0.1
92.8 0.2
93.6
93.7 0.8
AL
72.5 2.0
86.6
90.1# 0.4
91.7* 0.2
92.9 + 0.2
Table 16: CIFAR-10 with FASS
Test Accuracy (%)
Method
10%
15%
20%
25%
30%
CAL-ER
92.6 + 0.0
93.9 + 0.2
94.3 0.1
94.7 + 0.1
94.7 + 0.2
CAL-MIR
92.5
93.9 : 0.4
94.3
94.4 = 0.2
94.6 : 0.
CAL-DER
92.7
93.9
94.3
94.7
94.9
CAL-SD
92.6
93.8
94.4
94.6
94.7 0.1
CAL-SDS2
92.6 0.1
93.9 0.2
94.4 0.2
94.6 0.3
94.7 0.2
AL w/ AL
WS
92.5 4 0.1
93.6
93.9 0.1
94.1 0.1
94.3 +: 0.1
92.5
93.8
94.2
94.6
94.7 + 0.2
Table 17: FMNTST with GLISTER
Test Accuracy (%)
Method
10%
20%
30%
40%
50%
CAL-ER
81.7 0.3
89.2
91.9
93.0 = 0.1
93.3 # 0.1
CAL-MIR
81.6 1 0.3
89.3 0.4
91.7 à 0.2
92.9 = 0.1
93.5 4 0.2
CAT-DER
82.8 0.4
89.5 0-4
91.7 4 0.4
92.8 0.6
93.1 1 0.2
CAL-SD
82.5 + 0.3.
89.6 + 0.2
92.1 # 0.2
93.1 = 0.2
93.8 # 0.1
CAL-SDS2
81.4
89.1
92.1 0.2
93.2
93.9
AL w/ WS
81.7 0.4
89.3 0.4
92.1 0.3
93.0 0.1
93.3 + 0.4
AL
81.0 1 0.6
88.5 0.5
91.5 4 0.5
93.0 = 0.2
93.4 4 0.3
Table 18: CIFAR-10 with GLISTER
C.3 Effect of Query Size
In this section, we demonstrate that speedups with CAL methods can be realized with different query sizes. We demonstrate this by comparing AL with CAL-SDS2 on CIFAR-10, with three different choices of query sizes. We use entropy based uncertainty sampling as the acquisition function. We observe that CAL-SDS2 achieves comparable performance to standard AL with different query sizes, and the speedup can increase when the query size is reduced.
30

---

Published in Transactions on Machine Learning Research (11/2023)
Test Accuracy (%)
Factor Speedup
Query Size
Method
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
79.1
88.6
91.7
93.5
1.0x
1.0x
1.0x
1.0x
1.0x
5000
CAL-SDS2
79.1
89.4
92.4
94.0
94.3
1.0x
1.1x
1.4x
1.7x
2.0x
AL CAL-SDS2
81.9
89.6
92.4
93.5
94.1
1.0x
1.0x
1.0x
1.0x
1.0x
2500
82.5
90.1
92.9
94.0
94.4
1.1x
1.6x
2.1x
2.7x
3.4x
AT CAL-SDS2
84.4
90.4
93.2
93.8
94.2
1.0x
1.0x
1.0x
1.0x
1.0x
1000
84.3
89.8
92.5
93.2
93.5
1.6x
3.1x
4.2x
5.3x
6.4x
Table 19: Effeet of Query Size on CTFAR-10
D Additional Details on Single-Cell Cell-Type Identity Classification Dataset
The human cell landscape (HCL) dataset consists of scRNA-seq data for 562,977 cells across 63 cell types represented in 56 human tissues. Fach cell lype may be present in multiple tissues. The cell type classes are highly imbalanced, with the rarest cell type, human embryonic stem cell, accounting for 0.06 % the total dataset and the most common, fibroblast, accounting for 6%. The raw data first normalizod for library size and scaled to 10000 reads in total, followed by log transformation. We visualize the da dataset using UMAP 15
Single Coll Call Type
AT2 cell
Fertal aciner end
Marmphage
Adrenal gland infiemmatory cell
Fetal chondrocyte
Mast cell
Antigen presenting cell (IPS high)
Fetal endocrine cell
Mesothelial cell
Astracyte
hetal enterscyse
Monocyte
eptteekal
Myntost
(Passmocyte)
brobbastt
Neutrophii
Neutrophal
Primorsial
Endotiiciaal
Profieeratig
Endotheiaal cell (APC)
cheet
Procrna tubule progentor
Indothelall cell jendothelaal mesenchymaal transitio)
Gestre endocrine cell
Sinusoidal endorhelaal cell Smooth muscle cell
Enterocyte
Gob et cell
Enterocyte progen for
Eepultsyyt/FFdddddmmd cell Immature sertol cell Sertii cell)
Stratilied cpithclial call
Eptteeial call
Stramal cell
Epttelial cel [ntermedited]
intercalated cel
Tcell
Erythroid cell
incemediated cell
Thyrsid folliculor cell
Frythroid progeniter cell (IP high)
Kitncy intercalated cell
Ursteric but
Fesciculate
Hente
Wentncle cardomyacyte
Neuron
M2 Macrophage
NESC
Figure 15: UMAP embedding of single cells in HCL annotated by their cell type.
31


Reviews:
[{"review_id": "m2psAdC0Lr", "reviewer": "Reviewer_RHXd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nImportance of the Research Problem: The paper addresses a significant problem in the field of active learning, which is the time cost. This is a relevant and challenging issue that has practical implications in various domains.\nNeat Ideas: The use of continual learning techniques to accelerate batch active learning is an interesting concept and shows potential for advancements in the field. It brings together two distinct research areas, which can contribute to the development of more efficient and effective active learning methods.\nExperimental Validation on Multiple Datasets: One commendable aspect of the paper is the thorough evaluation of the proposed approach on multiple datasets. Conducting experiments on diverse datasets strengthens the validity and applicability of the findings.\n\nWeaknesses:\nLack of Innovation: While the paper addresses an important problem and employs continual learning techniques, it falls short in terms of introducing substantial innovative contributions or novel algorithms. It would benefit from proposing new concepts or extensions to existing methods to differentiate itself from prior works. The contribution of Scaled Distillation is not very impressive, in my opinion, it doesn't have any fundamental difference from DER.\nComparatively Weak Methodology: The proposed approach (SD and SD2) is deemed less effective than existing methods in the field. It is crucial to clearly highlight the limitations and shortcomings of the proposed method and provide a comparative analysis that demonstrates its inferior performance compared to state-of-the-art alternatives.\nLimited Performance Analysis: The paper could provide more in-depth analysis and discussion of the experimental results. By thoroughly examining the performance metrics and presenting insightful interpretations, it would offer a better understanding of the current limitations of existing methods and areas for potential improvement.\nIn conclusion, the paper addresses an important problem and presents an interesting idea by exploring the use of continual learning techniques for accelerating batch active learning. However, it lacks sufficient innovation, falls short in terms of method effectiveness compared to existing approaches, and requires stronger analysis and discussion of experimental performance. To enhance the paper's contribution and clarify its limitations, further advancements in the proposed method and more extensive evaluation would strengthen its overall scientific value.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Overall, the paper explores an important problem and presents an intriguing idea in the area of batch active learning, specifically focusing on leveraging continual learning techniques for acceleration.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My main concerns are as follows:\nLack of Innovation: While the paper addresses an important problem and employs continual learning techniques, it falls short in terms of introducing substantial innovative contributions or novel algorithms. It would benefit from proposing new concepts or extensions to existing methods to differentiate itself from prior works. The contribution of Scaled Distillation is not very impressive, in my opinion, it doesn't have any fundamental difference from DER.\nComparatively Weak Methodology: The proposed approach (SD and SD2) is deemed less effective than existing methods in the field. It is crucial to clearly highlight the limitations and shortcomings of the proposed method and provide a comparative analysis that demonstrates its inferior performance compared to state-of-the-art alternatives.\nLimited Performance Analysis: The paper could provide more in-depth analysis and discussion of the experimental results. By thoroughly examining the performance metrics and presenting insightful interpretations, it would offer a better understanding of the current limitations of existing methods and areas for potential improvement.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0yq8OvgXIk", "reviewer": "Reviewer_PnH7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths #\n- the paper introduces a compatible marriage of two pre-existing fields\n- the paper conducts quite comprehensive experiments to validate their approach\n- paper is clearly written, with the experimental methodology capturing the important questions that the framework poses\n\n# Weaknesses #\nThere are a few things that are unclear from the paper that I would like resolved.\n1. Algorithms 1 and 3 return a set of labeled examples $\\mathcal{L}$ -- are the results reported in the paper the final results obtained from $\\theta_{T}$ -- the last iterate in the respective procedures, or are they results obtained from  :\n\n             (i) creating a new random initialization $\\theta$\n\n             (ii) training the randomly initialized model once on the final $\\mathcal{L}$ and then reporting it's final performance ?\n\n2. On page 6 -- section Baselines, the paper states *Both models retrain on the entire dataset at each query round*  -- with respect to the AL methods.  Since a big part of the paper is the speedup, it seems to me that this detail would mean that the other methods always trivially result in speeds up due to using only $m^{(h)}$ samples from the history (granted these $m^{(h)}$ samples are chosen smartly) .  I would therefore contend that the CL-ER method, should rather be considered a baseline AL method (AL-WS-Random Subsample) since it's AL with randomly chosen batch from the history in order tot reduce compute cost.  \n\n3.  Some results come without significance testing even though the error bars quite overlap. Could you report the p-values for say Figure 5. It's unclear if the effects here should be taken on face value", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a way of  speeding up batch active learning approaches by leveraging insights from continual learning. \nSpecifically, they propose to apply continual learning methods to adapting to data from new query rounds whilst also leveraging data from past query rounds instead of restarting model training from scratch. \nThey show that the continual learning based framework works better than warm restarts both in terms of generalization and speedups.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see weaknesses above as they also contain requests for clarification", "broader_impact_concerns": "No ethical concerns immediately obvious", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dmeGf4Eoie", "reviewer": "Reviewer_QLBV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "First of all, I am not familiar with the literature of AL and CL. My comments below are based on my understanding of the submission. I'd like to correct them if there is something inaccurate or wrong.\n\n**Strengths:** The proposed method is really simple, and I can see the improvement in learning speed in AL. The paper is written well, and easy to understand.\n\n**Weaknesses:**\n\n1. There is little theoretical analysis on the proposed method, and all the claims are supported by the empirical results.\n\n2. The authors may overclaimed the contributions. Personally, I hardly believe that such batch-based sampling methods, i.e., reducing the number of training samples in a reasonable way for better computational efficiency such as stochastic learning (e.g., SGD), have never been used in AL. Maybe it is not called CL in some methods. If truly nobody has tried this before for better computational efficiency in AL, then I think this submission has merit in the AL field. I am hoping that the other reviewers and AC can better justify their contributions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a simple strategy for active learning (AL) by integrating replay-based sampling approaches in continual learning (CL) with the classic AL framework, namely continual active learning (CAL), to accelerate the learning in AL. Some experimental results are shown to demonstrate the performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No specific changes are suggested.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considered batch active learning, with a particular focus on using continual learning techniques to speed up the training process in modern deep active learning. 

I think this paper considered an unexplored but important problem in active learning -- using continual learning without training from scratch. In fact, this AE has also been considering this issue for a while and is quite happy to see the integration of continual learning to address these training problems.

At the same time, some reviewers raised concerns about the manuscripts. Here are the main points and my suggestions. 

- [Justifications] Some reviewers think that this paper could be much stronger if a systematic analysis was done to understand the choice of continual learning methods. I noticed that the authors updated one paragraph to reflect this. For the final version, I would suggest a thorough discussion. For example, a comparative table to illustrate the pros and cons of different continual learning approaches in active learning.

- [Distribution shift/forgetting/theoretical support] Reviewer QLBV thought of a lack of rigorous analysis, and I wondered about the source of forgetting. Upon reflection, I would think that the theoretical support might be explained by a distribution shift viewpoint. I would recommend two related papers [1-2]. In particular, paper [1] considered a dynamic update (analogous to continual learning) in active learning, based on the distribution shift in paper [2]. I would like to suggest a brief discussion of the theoretical possibilities.

Based on these, I would recommend an accept with a minor revision. 

References:

[1] Active Learning Using Discrepancy. https://realworldml.github.io/files/cr/7_cui_paper.pdf

[2] Deep Active Learning: Unified and Principled Method for Query and Training. AISTATS, 2020.
Decision: Accept (poster)