Continual Active Learning
Authors: Arnav Mohanty Das
Gantavya Bhatt
Megh Manoj Bhalerao
Vianne R. Gao
Rui Yang
Jeff Bilmes
Venue: Submitted to ICLR 2023 (2023)
Keywords: Active Learning
Deep Learning
Efficient Machine Learning
Continual Learning

Abstract: While active learning (AL) improves the labeling efficiency of machine learning (by allowing models to query the labels of data samples), a major problem is that compute efficiency is decreased since models are typically retrained from scratch at each query round.  In this work, we develop a new framework that circumvents this problem by biasing further training towards the recently labeled sets, thereby complementing existing work on AL acceleration. We employ existing and novel replay-based Continual Learning (CL) algorithms that are effective at quickly learning new samples without forgetting previously learned information, especially when data comes from a shifting or evolving distribution. We call this compute-efficient active learning paradigm $\textit{``Continual Active Learning" (CAL)}$. We demonstrate that standard AL with warm starting fails, both to accelerate training, and that naive fine-tuning suffers from catastrophic forgetting due to distribution shifts over query rounds.  We then show CAL achieves significant speedups using a plethora of replay schemes that use model distillation, and that select diverse/uncertain points from the history, all while maintaining performance on par with standard AL.  We conduct experiments across many data domains, including natural language, vision, medical imaging, and computational biology, each with very different neural architectures (Transformers/CNNs/MLPs). CAL consistently provides a 2-6x reduction in training time, thus showing its applicability across differing modalities.

Under review as a conference paper at ICLR 2023
CONTINUAL ACTIVE LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
While active learning (AL) improves the labeling efficiency of machine learning (by allowing models to query the labels of data samples), major problem is that compute efficiency is decreased since models typically retrained from scratch at each query round. In this work, we develop a new framework that circum- vents this problem by biasing further training towards the recently labeled sets, thereby complementing existing work on AL acceleration. We employ existing and novel replay-based Continual Learning (CL) algorithms that are effective quickly learning new samples without forgetting previously learned information, especially when data comes from a shifting evolving distribution. We call this compute- -efputtefffiientt active learning paradigm "Continual Active Learning" (CAL). We demonstrate that standard AL with warm starting fails, both to accelerate training, and that naive fine-tuning suffers from catastrophic forgetting due to distri- bution shifts over query rounds. We then show CAL achieves significant speedups using a plethora of replay schemes that model distillation, and that select diverse/uncertain points from the history, all while maintaining performance on par with standard AL. We conduct experiments across many data domains, including natural language, vision, medical imaging, and computational biology, each with very different neural architectures Transformers/CNN/MMLL provides a 2-6x reduction in training time, thus showing its applicability across differing modalities.
CAL consistently
INTRODUCTION
While neural networks have heen immensely successful in a variety of different supervised settings, most deep learning approaches are data-hungry and require significant amounts of computational resources. From large pool of unlabeled data, active learning (AL) approaches select subsets of points to label by imparting the learner with the ability to query a human annotator. Such methods incrementally add points to the pool of labelled samples by 1) training a model from scratch on the current labelled pool and 2) using some measure of model uncertainty and/or diversity select set of points to query the annotator (Settles) 2009 2011 Wei et al.) 2015 Ash et al. 20200] Killamsetty al. 2021). AL has been shown reduce the amount of data required for training, but still be computationally expensive to employ since it requires retraining the model, typically from scratch, when new points are labelled at each round.
A simple way to tackle this problem is to warm start the model parameters between rounds to reduce the convergence time. However, the observed speedups tend to still be limited since the model must make several passes through an ever- ver-inereasing pool of data. Moreover, warm starting alone in some cases can hurt generalization, as discussed in Ash Adams (2020) and Beck al. (202T). Another extension to this is to solely train on the newly labeled batch of examples to avoid re-initialization. However, as we show in Section 3.3] naive fine-tuning fails to retain accuracy on previously seen examples since the distribution of the query pool may drastically change with each round.
This problem of forgeern while incrementally learning from a series of new tasks with shifting distribution is central question in another paradigm called Continual Learning (CL) (French 1999; McCloskey & Cohen 1989 McClelland et al., 1995 Kirkpatrick et al. (017c). CL has recently gained popularity, and many algorithms have been introduced to allow models to quickly adapt to new tasks without forgetting (Riemer et al. 2018 Lopez-Pat Paz & Ranzato 2017 Chaudhry
et al. 2019 Aljundi et al. 2019h, Chaudhry al.
2020 Kirkpatrick et al. 2017b)

---

Under review as a conference paper at ICLR 2023
In this work, propose Continual Active Learning (CAL), which applies continual learning strategies to accelerate batch active learning. In CAL, we propose applying CL to enable the model to learn the newly labeled points without forgetting previously labeled points while using past samples efficiently using replay-based methods. As such, we observe that CAL methods attain significant speedups over standard AL in terms of training time. Such speedups are beneficial for the following reasons:
As neural networks grow in size (Shoeybi et al.) 2019), the environmental and financial costs to train these models increase as well (Bender et al. 2021 Dhar 2020 Schwartz et al., 2020). Reducing the number of gradient updates required for AL will help mitigate such costs, especially with large-scale models.
Reducing the compute required for AL makes AL-based tools more accessible for de- ployment on edge computing platforms, IoTT, and other low-resource devices (Senzaki & Hamelain 2021).
Developing new AL algorithms/acqiisiiion functions, searching for architectures as done with NAS/AutoMI, that are well-suited specifically for AL can require hundreds or even thousands of runs. Since CAL's speedups are agnostic the AL algorithm and the neural architecture, such experiments can be significantly sped up.
The importance of speeding up the training process in machine learning is well recognized and is evidenced by the plethora of optimized machine learning training literature seen in the computing systems community (Zhihao Jia & Aiken Zhang et al.) 2017 Zheng et al.) 2022).
In addition, CAL demonstrates a practical application for CL methods. Many of the settings used to benchmark CL methods in recent works somewhat contrived and unrealistic. Most CL works consider the class/domain incremental setting, where only the samples that belong to a subset of the set of classes/domains of the original dataset are available to the model any given time. This setting rarely occurs in practice, representing the worst-case scenario and therefore should not be the only benchmark upon which CL methods are evaluated. We posit that the validity of future CL algorithms may be determined based on their performance in the CAL setting in addition to their performance in existing benchmarks.
To the best of our knowledge, this application of CL algorithms for batch AL has never been explored. Our contributions can be summarized as follows: (1) We first demonstrate that active learning can be viewed as continual learning problem and propose the CAL framework; (2) we benchmark several existing CL methods (CAL-ER, CAL-DER, CAL-MIR) as well as novel methods (CAL-SD, CAL-SDS2) and evaluate them on several datasets based on the accuracy/speedup they can attain over standard AL..
2 RELATED WORK
Active learning has demonstrated label efficiency (Wei et al. 2015 Killamsetty et al. 2021 Ash et al. 2020) over passive learning. In addition to these empirical advances there has been extensive work on theoretical aspects as well over the past decade (Hanneke) 2009 2007 Balcan et al. 2010) where
Hanneke (2012) shows sample complexity advantages over passive learning in noise- free classifier learning for VC classes. However, recently there has been an interest in speeding up active learning because most deep learning involves networks with a huge numbers of parameters.
Kirsch al. (2019): Pinsler al.) (2019) Senner & Savarese (2018) aim to reduce the number of query iterations by having large query batch sizes. However, they do not exploit the learned models from previous rounds for the subsequent ones and are therefore complementary to CAL Works such as Cooeman et al. (2020a); Ertekin et al. (2007): |Maerr & Timoftel (2020): Zhu & Bento (2017) speed up the selection of the new query set by appropriately restricting the search space or by using generative methods. These works can be easily integrated into our framework because CAL works on the training side of active learning, not on the query selection. On the other hand, Lewis & Catlett (994)): Coleman et al. (2020b): Yoo & Kweon (2019) use smaller proxy model to reduce computation overhead, however, they still follow the standard active learning protocol, and therefore can be accelerated when integrated with CAL
2

---

Under review as a conference paper at ICLR 2023
Lastly, there exist few prior works that explore continual/transfer learning and active learning in the same context. Perkonigg et al. (2021) propose an approach that allows active learning algorithms to be applied to data streams in the context of medical imaging, by introducing module that detects domain shifts. This differs from our work which uses algorithms that prevent catastrophie forgetting, accelerate active learning. [Zhou et al. (202T) consider setting in which standard active learning used to finetune a pre-trained model, and uses transfer learning to do so. Thus, this work does not consider continual learning and active learning in the same setting and is therefore related to our work.
On preventing catastrophic forgetting, in this work, we mostly focus on the replay-based algorithms that are currently saateoof-the-nt methods in continual learning. However, as demonstrated in Section 3.3 on how active learning rounds can be seen as continual learning, one can apply other methods such as EWC (Kirkpatrick et al.) (2177)), structural regularization (Li et al. 2021) or functional regularization based methods as well. (Titsias et al. 2020).
METHODS
3.1 BATCH ACTIVE LEARNING
Define [n] = {I, [I,..., ...,),, n). and let X' and y denote the input and output domains respectively, AL typically starts with unlabelled dataset the model f. with parameters 6, to query a user for labels for any x E U, but the total number of labels limited
[an)) where each 2; X', The AL setting allows
budget b, where < n. Throughout the work, we consider classification tasks so the output of (a; 0) is probability distribution over classes. The goal of AL is to ensure that f can attain low error when trained only on the set of labelled points.
Algorithm details the general AL procedure. Lines 3-6 construct the seed set D1, by randomly sampling subset of points from Z and labelling them. Lines 7-14 iteratively expand the labelled set for 7 rounds by training the model from random initialization on D; until convergence and selecting be points (where Eie(r) b, = b) from 2/ based on some selection criteria that dependent on 02. The selection criteria generally selects samples based model uncertainty and/or diversity (Lewis & Gale
1994 DDaaan & Engelson 1995 /Stttles) Killamsetty al. 2021 Wei al., (015) Ash al. 2020
Sener & Savarese| 2017). In this work, we primarily consider uncertainty sampling Lewis & Gale
(1994); & Engelson (1955): Settles though also test other selection criteria in Section A in
the Appendix.
Algorithm 1
procedure ACTIVELEARNINGL, Ut bI.T.
2:
t 1. +
> Initialize Draw by samples from U Provide labels
3:
Ui ~
4:
D. + (iI,, E4}
5:
u+ 1--1/4 lee
6:
LUD,
7:
while < do
8:
Randomly initialize Synit
9:
A, t (f.in(f.ee Oimis, L)
Ur Select(S, 0z, 0r,21
Select by points from li based O,
11:
Dr + ((r,, 31 x4 E Ue)
U u Mii LUD,tttH
13: return
Uncertainty Sampling is widely-used practical AL method that selects 41; = {I1,..., Th, to label from 24 by choosing the samples that maximize a notion of model uncertainty. We consider entropy 1995) as the uncertainty metric, so if h(xx) A -Evejy f(z; 0); log f(;; 0),
Dagan & Engelson,
then Cle E arg max A:A Eve h()).
3

---

Lopez-Paz & Under review as a conference paper at ICLR 2023
3.2 CONTINUAL LEARNING
We define Dim - UVE(M) Dj. In CL, the dataset consists of T tasks (Di,.., Dr} that are presented to the model sequentially, where Dr { [ejm,] and ny is the cardinality of Dr. At time E [T], the data/label pairs are sampled from the current task (ær, y) ~ Dr, and the model generally has only limited access to the history D..--1 The CL objective is to efficiently adapt the model to Dr while ensuring that performance on previously learnt tasks Dia-1 does not degrade appreciably. Ideally, given a loss function & X' x y + I. initial parameters - and a model f. 0g can be obtained by solving the CL optimization problem (Aljundi et al. 2019b Chaudhry et al. 2019
Ranzato 2017):
arg min (r.y)~D,
f(;(0)
s.t
E (r' (r'a')--rrr
E d'Dic-
In this work, foeus replay based CL techniques which attempt approximately solve the CL optimization problem by using samples from Dia-11 to regularize the model while adapting to Dr.
Algorithm 2 outlines the general replay-based CL algorithm, in which the objective is adapt parametrized by Op to 'D while using samples from the history M. Inside the training loop, Beurent consists of points randomly sampled from D. Breplay consists of m' points that chosen based on some customizable selection criteria from M. In line 6, 0, is computed based on some update rule that utilizes both Beepaa and Buurentt Note that many CL works also consider the problem of selecting which samples should be retained in M, which is relevant in the scenario where Dirr is too large to store in memory or when unknown (Aljundi al., (019b)). However, this constraint does not apply to the CAL setting, so in the subsequent sections we consider M Diat-1.
Algorithm 2
procedure CONTINUALTRAINGF, G, D. M, m, m²)
2:
1+11
3:
while not converged do Beurrent <- {Z 31) Jm D)
4:
Sample points from current task > Sample replay m' points from history
5:
Breplay <- Select (f 1, m²) A: < Update(f. (f, 8,-1, Beurren Brepla))
6:
7:
t +
8:
return
3.3 ACTIVE LEARNING AS CONTINUAL LEARNING
clear inefficiency of standard AL stems from the fact that the model f must be retrained from scratch on the labelled pool every round. In this work, we employ CL--inspired techniques adapt to the newly labelled points, while significantly reducing the number of updates needed on samples labelled in previous rounds.
We demonstrate that catastrophic forgetting indeed occurs in AL, when a model is fine-tuned only on the newly labelled points every round. In Figure 1, task indicates the set of points from the training dataset that were selected at the round t of querying based on entropy sampling, On the y-axis, we report the accuracy of each set immediately after the model has been fine-tuned on the points that were just labelled at a particular round.
is evident that the model forgets old information from the precipitous drops in performance for task t t- 1 as soon as the model is adapted to new task t when points are added to the labelled set. Note that task 1, after the initial drop, tends to increase performance the subsequent AL rounds since the points belonging to the initial round are chosen uniformly random (shown in Algorithm 1) and thus an unbiased estimate of the full dataset. This trend is generally not present in any of the later tasks, which sampled from distributions that conditioned on the model parameters 0,. It is also interesting to note that the model performs considerably worse on all of the tasks (aside from task 1) than it does on the test set, despite the fact that the model has been trained the labelled
4

---

Under review as a conference paper at ICLR 2023
pool. This experiment suggests that 1) the distribution of each task t >> 1 is distinct from the true data distribution and 2) techniques designed to combat catastrophic forgetting are necessary in order to effectively incorporate new information between successive AL rounds.
Forgeting Active
357
Ta41
30
Round
Figure 1: This figure shows the performance of ResNet-18 on CIFAR-10, in the active learning setting where the model is only trained on newly labelled points. At each round, 5% of the full dataset is added to the labelled pool.
Algorithm 3
1:
procedure CAL(F, U, m, m')
2:
1+
> Initialize
3:
ur~U
Draw b samples from 24 Provide labels
4:
D. + (i),, {
5:
u+ 1-UV/
6:
C LUD,
7:
while s T do
8:
02 &- ContinualTrain(f. f.-1, Di, D1--11, m, m.m.) m²) U4+1 Select(f, ff 0r. b,)
9:
D Select points from based on 0
10:
Dr+1 + {Ii, v:) r; {+++1
11:
U Me L LUD+
12:
return L
To ameliorate the problem of catastrophic forgetting, we use CL techniques. The continual active learning (CAL) approach shown in Algorithm 3. The key difference CAL from standard AL (Algorithm 1) can be found in line 8. Instead of standard training, replay-based CL is used to adapt Dr while retaining performance on D1:t-1- The speedup comes from two points: 1) the number of gradient updates computed for samples from Di:--1 is less than that of samples in DD; for reasonable choices of m' and 2) the model tends to converge faster since its parameters are warm-starter We compare several CAL methods and assess their performance based on their performance the test set and the speedup they attain compared to standard AL. In the rest of the section
Le
A
E (x.g)r-BBrrm
[e(u, f (x; 0))
Experience Replay (CAL-ER) is the simplest and oldest replay-basee based method (Ratcliff 1990. Robins 1995). this approach, Beurem and Beepla are interleaved to create a minibatch B of size m m and Breplay is chosen uniformly random from DI The parameters of model updated based on the gradient computed on B.
Maximally Interferred Retrieval (CAL-MIR) addresses the problem selecting samples from DIa-1, choosing the m' points that are most likely to be forgotten (Aljundi al. 2019a). Given batch of m labelled samples Beureett sampled from D, and model parameters 8, 8. is computed by 7/ L. where 17 is the learning rate. Then for every example x in the history, [z)) = {(f(x; [ f), y) - {x; 0.,,, y)) or the change in loss after taking single gradient step is computed. The m' samples with the highest 8MIR score selected to form
taking "virtual" gradient step i.c. 0,
5

---

Under review as a conference paper at ICLR 2023
Breplay- Beurent and Bieplay are concatenated together to form the minibatch (as in CAL-ER), upon which the gradient update is computed. In practice, selection is done on a random subset of D1:1-1 for speed.
Dark Experience Replay (CAL-DER) uses a distillation based approach to regularize updates (Buzzega et al. 2020). Suppose g( 0) denotes the presoftmax logits of classifier f(z; f) i.e [æ; softmax 0)). In DER, every x' E Di has an associated =' which corresponds to the logits produced by the model the end of the task when was first observed. In other words, if x' E Dr, then z' - gar'; ;0;)) where 1' [t 1] and 0% are the parameters obtained after round t'.
DER minimizes LDER as expressed below:
LDE 2 Le
E (xl (r
(a
((d;0)))
where Beurentt is a batch sampled from Dr. Beelay is a batch sampled from D1s-1 and a and are tuneable hyperparameters. The first term ensures that samples from the current task are classfied correctly. The second term consists of a classification loss and a mean squared error (MSE) based distillation loss that are applied on samples from the history.
Scaled Distillation (CAL-SD) is a new CL approach we propose in this work specifically tailored towards the CAL setting. SD addresses the stability-plasticil dilemma that is commonly found in both biological and artificial neural networks (Abraham & Robins 2005 Mermillod et al. 2013). A network stable if can effectively retain past information but cannot adapt to new tasks efficiently. whereas a network that is plastic can quickly learn new tasks but is prone to forgetting. The trade-off between stability and plasticity a well-known constraint in CL (Mermillod et al. (213). In the context of CAL, we would like the model to be plastic during the early rounds and stable during the later rounds. We apply this intuition to develop SD, which minimizes LSD at round t as shown below:
Creplay
E (f D-BBBas
[ax DKI. (softmax(z)) f(x'; 0)) + +1 (1-a))(00))) ((:(0()))))))
(3)
LSD de Le + (1 Nx) Freplay
(4)
where,
de
(5)
Pra-1
+
[2.
Similar to CAL-DER, Leppla sum of two losses: a distillation loss and classification loss. The distillation loss in Lreplay minimizes the KI. divergence between the posterior probabilities produced by I and softmax (='). where defined in the DER section. We use KL divergence term instead of MSE loss on the logits, that the distillation loss and the classification losses are the same scale. a E [o, 1] is a tuneable hyperparameter.
LsD is convex combination of the classification loss on the current task and Creplay. The weight of each term is determined adaptively by the stability/plasstictyy trade-off term A₃ Higher values of X indicate higher model plasticity, since minimizing the classification error of samples from the current task is prioritized. D increases with t, X₁ decreases and the model becomes more stable in the later rounds of training.
Scaled Distillation w/ Submodular Sampling (CAL-SDS2) CAL-SDS2 is another a new CL approach we introduce in this work. CAL-SDS2 uses CAL-SD to regularize the model and utilizes submodular sampling to select a diverse set of points from the history to replay. Submodular functions are well suited to capture notions of diversity and representativeness (Lin & Bilmes 2011 Wei et al. 2015) Bilmes] 2022). and the greedy algorithm can approximately maximize monotone submodular function up I -1 factor guarantee (Fisher et al. 1978 Minoux 1978 Mirzasoleiman et al. 2015). We define a submodular function G below:
G(S) 2
Σ E.EAA
max Way + A log 1+ h(xx))
(6)
ZyES

---

Under review as a conference paper at ICLR 2023
The first term of G' is the facility location function, where Wa is similarity score between samples e and 2z. In our experiments, Wij = exp (-1| 2j 112 (2 where z; is the penultimate layer representation of model f for Iy and a a hyperparameter. The second term a concave over modular function (Liu et al. 2013) and h((z;) is some measure of model uncertainty. In order to speed up SDS2, we randomly subsample from the history before performing submodular maximization so S A C D1:t-1 The objective of CAL-SDS2 is to ensure that the set of samples that replayed are both difficult and diverse, similar to the motivation of the heuristic employed in Wei et al. (2015).
4 RESULTS
In this section, we evaluate the validation performance of the model when we train on different fractions (b/n) of the full dataset. We compute the factor speedup attained by CAL method by dividing the runtime of AL over the runtime of the CAL method. We test the CAL methods on variety of different datasets spanning multiple modalities. The two methods that do not utilize CAL are AL w/ WS (Active Learning with Warm Starting) and AL. We plot speedup vs mean test accuracy (computed over three random seeds) different labelling budgets (b/nn) for each of the five datasets we consider in this work. Qualitatively, methods that are plotted towards the top right corners are preferable. The results are also available in tabular form in Appendix A. We adapt the AL framework proposed in Beck et al. (202T) for all experiments presented in this section. In the main paper, we show results for uncertainty sampling based acquisition function, but provide results on other acquisition functions as well in Appendix B. Our objective is to demonstrate 1) at least one CAL method exists that can match or outperform a standard active learning technique while achieving a significant speedup for every budget and dataset and 2) models that have been trained using a CAL method behave no differently than standard models.
4.1 EXPERIMENTAL. SETUP
FMNIST The FMNIST dataset is a dataset consisting of 70,000 28 grayscale images of fashion items belonging to 10 classes (Xiao et al. 2017). A R8sNet-18 architecture (He et al. 2016) and SGD is used. We apply data augmentations, as in Beck et al. (202T), consisting of random horizontal flips and random croppings. On this dataset, we find that a CAL method matches or outperforms the performance of standard AL in every setting we test 2
CIFAR-10 CIFAR-10 10 consists of 60,000 32x32 color images with 10 different categories (Krizhevsky 2009). We use ResNet- 18 and use the SGD optimizer for all CIFAR-10 experi- ments. We apply data augmentations consisting of random horizontal flips and random croppings. From the results shown in Figure there is at least one CAL method that outperforms standard AL for every budget that we examine.
MedMNIST We use the DermaMNIST dataset within the MedMNIST database ( YYag et al. 2021a b) for performance evaluation of CAL on medical imaging modalities. DermaMNIST consists of 3-color channel dermatoscope images of 7 different skin diseases, originally obtained from Codella et al.] (2019): Tschandl et al. (2018) A ResNet-18 architecture is used for all DermaMNIST experiemnts. All results are shown in Figure 4
Budgei - 10%
Budlge - 15%
Burgee - 20%
Budgert 25%
Budget - 50%
CAL-ER
34.0
94.5
CAL -MRR
24.6
54
CAL-DEE
23.9
94.5
CAL-SD
54.7
AL WB
38.8
54.4
51.4
3
3
2
Spewdup
Figure 2: FMNIST Results
7

---

Under review as a conference paper at ICLR 2023
Budgel + 10%
Budgel - 20%
Budger 30%
Burfgel 40%
Budget 50%
85.5
$3.0
59.8
29.5
30.0
52.0
92.6
30.8
20.8
%
54
94.
CALIER
CAL
CAL-MRR
62.5
92.4
CAL-SDS2
WS
19.
92.2
50
Speeduup
Figure 3: CIFAR-10 Results
Budgel= 10%
Budgal 20%
Budgal 25%
Budget= 90%
70.0
70.0
70.0
x
CAL-MRR
CALERR
62.6
26.0
08.0
680
675
CAL-SD
66.0
CAL-SD02
60.0
86.0
660
AL wf WS
64.0
625
Spesdup
Figure 4: MedMNIST Results
Amazon Polarity Similar to Coleman et al. (020b)), we use Amazon Polarity Review (Zhang et al. 2015) dataset, which is NIP dataset consisting of reviews from Amazon and their corresponding star-ratings (5 classes). We consider total unlabelled pool of size 2M sentences and use VDCNN-9 Schwenk al. (0117) architecture, trained with Adam optimizer. As observed from Figure 5 CAL methods achieve speedups while having competitive performance with standard AL procedure.
COLA (Warstadt et al. 2018) is an another commonly used NLP dataset, which was recently considered in Active Learning setting (Fin-Dor et al. (020). It aims to check linguistic acceptibility of sentence, that is, binary classification. We use BERT (Devlin et al. (019) backhone trained with Adam optimizer. We consider unlabled pool of size 7000 and remaining as test; similar to Ein- -or (et al. (2020) we use entropy sampling for the acquisition function and report accuracy. Figure 6 reports the performance and speedup of CAL methods with increasing budget, which shows their competitive performance with standard AL procedure.
Single-Cell Cell Type Identity Classification Recent single-cell RNA sequencing (scRNA-seq)
technologies has enabled large-scale characterization of hundreds of thousands to millions of cells in complex tissues, and accurate cell type annotation is a crucial step in the study of such datasets. To this end, several deep learning models have been proposed to automatically label new scRNA-seq datasets (Xie et al., 202T). The HCL dataset is a highly iaas-iimbllnceed dataset that consists of scRNA-seq data for 562,977 cells across 63 cell types represented in 56 human tissues. (Han et al., 2020). The data divided into training, validation and test sets via an 80/ 80/10/10 split whilst ensuring similar class proportions across splits. We use the ACTINN model (Ma & Pellegrini| 2019). a four-layer multi-layer perceptron that predicts the cell-type for each cell given its expression of 28832 genes, and use the SGD optimizer for all experiments. From the results shown in Figure 7 the majority of the CAL methods outperforms standard AL for every subset size that we examine.
Budget= 10%
Budget
Budget 30%
Budget 40%
Budget= 52%
34.0
94.0
93.8
CALEER
92.5
91.0
CAL AMR
32.0
38.5
338
50
CAL-SD
21.5
25.0
25.2
K35
AL WB
CAL-SDR2
92.6
$0.0
932
4
Spcodup
Figure 5: Amazon Polarity Results
8

---

Under review as a conference paper at ICLR 2023
Budgel = 2.9%
Budger = 5.7%
Budget 8.6%
Budgett 11.4%
16.3%
78.0
77.5
78.0
CALER
CAL-MR
Acarary
28.0
77.0
75.5
78,5
775
195
75.0
78.0
Speediup
Figure 6: COLA Results
Budget 20%
Budget 25%
$7.0
CALER
NoGur3yy (50
86.0
$0.5
88.5
30.0
CAL-MRR
$5.0
19.0
CAL-SDRR
57.5
89
AL WB
$4.0
$7.0
58.5
Spcodup
Figure 7: Single-Cell Cell-Type Identity Classification Results
4.2 SCORE CORRELATION BETWEEN STANDARD AND CAL MODELS
We test whether or not CAL models behave the same way as models that have been trained using standard AL. Specifically, we assess the degree to which the uncertainty scores of CAL models are correlated with standard models. In Figure we show the pairwise correlation between all the entropy scores of the models we used in the FMNIST and CIFAR-10 experiments at the end of training (after training on 50% of the data). From the results, it is evident that the all the entropy scores are positively correlated, providing an explanation as to why CAL models are able to perform on par with standard models.
ANNIST
CIAR
0-42
0.30
0.3
0.49
0.0
AL 95
0:73
0:43
04
0.41
3.34
AL
0:51
0:46
0.47
0.69
6.5
0.4
C8
CALER
0.27
0.4
C:42
C.45
0.46
2.44
06L
0.22
0.45
0:3
C:4
0.54
0.48
CAL
0::7
0:44
0.45
0:64
032
0.67
0.54
0,55
0.48
CALOER
0:46
0.44
0.46
CAL-0ER
0.25
0.49
0.55
0.49
006
-C.5
0:58
0:46
0:58
0:55
-85
CALOR
5002
200
Figure& 8: The correlation of entropy the test set between models trained using AL/CAL the end of FMNIST and CIFAR-10 experiments is shown.
5 CONCLUSION
In this work, we proposed the framework of CAL and demonstrated its efficacy in speeding up AL across multiple datasets by applying techniques adapted from CL. Across vision, natural language, medical imaging, and biological datasets, we observe that there is always CAL method that either matches or outperforms standard AL while achieving considerable speedups. Since CAL is independent of model architecture and AL strategy, this framework is applicable of settings. Furthermore, CAL provides a novel application for CL so future CL algorithms can be assessed based on their performance CAL well other existing CL benchmarks.
broad range

---

Under review as a conference paper at ICLR 2023
REFERENCES
Wickliffe C Abraham and Anthony Robins. Memory retention-the synaptic stability versus plasticity dilemma. Trends Neurosci, 28(2):73-78, February 2005.
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. CoRR, 04
has//990.047744 2019a. URL URLttp///arxx http://v.rggb/b/////aaaa/
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning, arXiv preprint arXiv:1903.08677 2019b.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. In
H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3884 3894. Curran Asso- ciates, Inc., 2020. URL 7Bcd255753300eee03333333333
nwwrips httpr//ss.ccppppp////////////// pdf
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. ArXiv, abs/1906.03671, 2020.
Maria-Florina Balcan, Steve Hanneke, and Jennifer Wortman Vaughan. The true sample com- plexity of active learning. Machine Learning, 80(23):111-139, April 2010. doi: 1007/
-10994-010-5174-y. UtL/tttps:///// /ttp:////.........0000//0/000000 010- 5174--
Nathan Beck, Durga Sivasubramanian, Apurva Dani, Ganesh Ramakrishnan, and Rishabh K. Iyer. Effective evaluation of deep active learning image classification tasks. CoRR, 2021. httpp:://aax.v...////bbbb////
Emily M. Bender, Timnit Gebru, Angelina McMillan- Majorr, and Shmargaret Shmitchell. On the
dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAceT '21, pp. 610-623, New York, NY, USA, 2021, Association for Computing Machinery. ISBN 9781450383097. doi:
011443344218834455959 URL / 1145:0011011111114222212112 3445992
Jeff A. Bilmes. Submodularity in machine learning and artificial intelligence. CoRR, abs/2202.00132. 2022. URL httph:ttp//aaate
www.//aaaaa...oooo 2202. 00132
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and SIMONE CALDERARA. Dark
experience for general continual learning: a strong, simple baseline. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 11920-159300 Curran Associates, Inc., 2020.
Arslan Chaudhry, Mare Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. In ICLR, 2019.
Arslan Chaudhry, Albert Gordo, David Lopez-Paz, Puneet K. Dokania, and Philip Tor. Using hindsight to anchor past knowledge in continual learning, 2020. [RLL:tttp:///peeeeee
net/ wtt.ormmmmi..eee1144444000
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M. Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic), 2019. URL http:://///xxx..og////
1902. 03368
Cody Coleman, Edward Chou, Sean Culatana, Peter Bailis, Alexander C. Berg, Roshan Sumbaly, Matei Zaharia, and Zeki Yalniz. Similarity search for efficient active learning and search of rare concepts. CoRR, abs/2007. .00077, 2020a.
00077
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations, 2020b. URLhttps: 7/openrevies net. owt.eeeeeeeeooooomoiioooooooaaa
10

---

Under review as a conference paper at ICLR 2023
Ido Dagan and Sean P. Engelson. Committee- based sampling for training probabilistic classifiers. In Proceedings of the Tweifth International Conference on International Conference on Machine Learning, ICML'95, pp. 150-157, San Francisco, CA. USA, 1995. Morgan Kaufmann Publishers Inc. ISBN 1538603778.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019.
Payal Dhar. The carbon impact of artificial intelligence. Nature Machine Intelligence, 2(8):423-4255
Aug 2020. ISSN 2522-5839. doi: 10. 10.10888-44256600000000922 URL
020
Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. Active Learning for BERT: An Em- pirical Study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 7949-7962, Online, November 2020. Association for Computational Linguistics. doi: 10.18853/vvl 1tt0:///1//00.mmmlpp.. mai.lp.mma......... URL https: htt.ttttts//aaatttttttooo 2020. emnlp ma i n. 638
org/
Seyda Ertekin, Jian Huang, Leon Bottou, and Lee Giles. Learning on the border: Active learning in imbalanced data classification. In Proceedings of the Sixteenth ACM Conference on Conference on Information and Knowledge Management. CIKM '07, pp. 127-136, New York. NY, USA. 2007. Association for Computing Machinery, ISBN 978 59959999,,,, doi: 29.1445/132144001321446 1321
URL /tt.:pps//ooo..r/11..111111
M.I. Fisher, G.I. Nemhauser, and L.A. Wolsey. An analysis of approximations for maximizing submodular set functions-II II. In Polyhedral combinatorics, 1978.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3 (4):128-135, 1999.
Xiaoping Han, Ziming Zhou, Lijiang Fei, Huiyu Sun, Renying Wang, Yao Chen, Haide Chen, Jingjing Wang, Huanna Tang, Wenhao Ge, Yincong Zhou, Fang Ye, Mengmeng Jiang. Junqing Wu, Yanyu Xiao, Xiaoning Jia, Tingyue Zhang, Xiaojie Ma. Qi Zhang, Xueli Bai, Shujing Lai, Chengxuan Yu, Lijun Zhu, Rui Lin, Yuchi Gao, Min Wang, Yiqing Wu, Jianming Zhang, Renya Zhan, Saiyong Zhu, Hailan Hu, Changchun Wang, Ming Chen, He Huang, Tingbo Liang, Jianghua Chen, Weilin Wang, Dan Zhang, and Guoji Guo. Construction of human cell landscape at single-cell level. Nature, 581 581(7808) 309, March 2020.
Steve Hanneke. A bound on the label complexity of agnostic active learning. In Proceedings of the 24th International Conference on Machine Learning, ICML, '07, pp. 353-360, New York, NY, USA. 2007. Association for Computing Machinery, ISBN 9781595937933., doi: 10.1145/1273396.
1273541. htthshts::// /tte::///oooii /oii www.lo.g./o........ 1tt.//111.111112114913939 1273541
Steve Hanneke. Theoretical foundations of active learning. Carnegie Mellon University, 2009.
Steve Hanneke. Activized learning:
Transforming passive to active with improved label complexity. Jourmal of Machine Learning Research, 13(49): 1469-1577, 2012. URL http: orgg/
/vaiss/v333 papers/ vv1vvv33//// miaeerss/v3/haaneee www/banneeeela....mmmmoaaa. html
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
doi: 10. 1099/CVR.2206..90
KrishnaTeja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh K. Iyer. GLIS- TER: generalization based data subset selection for efficient and robust learning. In Thirty-Fifth AAAI Conference Artificial Intelligence, AAAI 2021, Thirty-Thine Conference on Innova- tive Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence. EAAI 2021. Virtual Event, February 2-9, 2021, pp. 8110- 8818. AAAI Press, 2021. URL https: / /o aaai. org/ /wt//idxx....mm php/ php/AAAI AAAI CeAAI/artii /ari////ee
vi ew 6988
11

---

Under review as a conference paper at ICLR 2023
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiaoo Ramalho, Agnieszka Grabska Barkka--rriikkk,, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, (14(13):352) 7a. doi: 10.1073/pnas. 1835114. 14. URL https: /www pnas wwtp:ppaaa..rrg/o//////////.oo.. 1611835114
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka
Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-352 2017b. doi: 10.10733pnas.1611 16118351
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017c
Andreas Kirsch, Joost Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Fox. and R. Garnett (eds.), Advances in Neural Information Processing Systems. volume 32. Curran Associates, Inc., 2019. URL UR/.ttps///rrooeeeissss.os
cc/
Alex Krizhevsky. Learning multiple layers of features from tiny images. pp. 32-33, 2009. URL
wwtp://wwwccc www.sss.ooooooooooooooooooo. http://ww....ooooooooeaa/aaa//aa
featurrs 2009-0000200000 pd)
features
David D. Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In William W. Cohen and Haym Hirsh (eds.), Machine Learning Proceedings 1994, pp. 148-
156. Morgan Kaufmann, San Francisco (CA), 1994. ISBN 978- 978-1-55860-335-42 doi: https:
//bb..og/10.110669988-11 1-58000087788---------------303- 335-6000-3336600 6.5026665 URL https: http://wwwwiennncreeeeeeeee. com/ httm://siieeeeattcccee///////a/a 017777855588333333 /50.1//0017271111113333335555515
David D. Lewis and William A. Gale. A sequential algorithm for training text classifiers, 1994, URL http://arxiyg:
abs/ cmp / 9407020 active learning roots.
Haoran Li, Aditya Krishman, Jingfeng Wu, Soheil Kolouri, Praveen K. Pilly, and Vladimir Braverman.
Lifelong learning with sketched structural regularization. CoRR, abs/2104.08604, 2021. URL
httpss://arxiii org/ als / 024 08604
Hui Lin and Jeff Bilmes. A class of submodular functions for document summarization. In Pro- ceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 510-520, Portland, Oregon, USA, June 2011. Association for Compu- tational Linguistics. URL https: dtttss//aclanthol..o wwg. 1052
Yuzong Liu, Kai Wei, Katrin Kirchhoff, Yisong Song, and Jeff Bilmes. Submodular feature selection for high-dimensional acoustic score spaces. In 2013 IEEE International Conference Acoustics, Speech and Signal Processing, pp. 7184-7188, 2013.
doi: 10. 11111//AASP.0013666999999
David Lopez-Paz Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.). Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Feiyang Ma and Matteo Pellegrini. ACTINN: automated identification of cell types in single cell
RNA sequencing. Bioinformatics, 36(2):533-538, 2019. ISSN 1367-4803. doi: 10. 1093/ bioinformatics/itz292.. URL /tt.ttps:///oii..rr/////// 1093 http:///............ooo.oooooo..
C. Mayer and R. Timofte. Adversarial sampling for active learning. In 2020 IEEE Winter Conference Applications of Computer Vision (WACV), pp. 5060-3063 Los Alamitos, CA, USA, mar 2020. IEEE Computer Society. doi: 10.1100//ACV4455222200000
URtps//s: 9093555
12

---

Under review as a conference paper at ICLR 2023
James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Martial Mermillod, Aurélia Bugaiska, and Patrick BONIN. The stability-plasticity dilemma: in- vestigating the continuum from catastrophic forgetting to age-limited learning effects. Fron- fiers in Psychology, 4, 2013. ISSN 1664-1078. doi: 10.3388/fpsy
URL
https: www. www.onensssiii... org/ wtt.://ooo.a....e.......... 33890..333.111 2013. 00504
M. Minoux. Accelerated greedy algorithms for maximizing submodular set functions. In Optimization Techniques, 1978.
Baharan Mirzasoleiman, Ashwinkumar Badanidiyuru, Amin Karbasi, Jan Vondrák, and Andreas Krause. Lazier than lazy greedy. In Proceedings of the AAAI Conference Artificial Intelligence. volume 29, 2015.
Matthias Perkonigg, Johannes Hofmanninger, and Georg Langs. Continual active learning for efficient adaptation of machine learning models to changing image acquisition. In Lecture Notes in Computer Science, Lecture notes computer science, pp. 649-660. Springer International Publishing, Cham, 2021.
Robert Pinsler, Jonathan Gordon, Eric Nalisnick, and José Miguel Hernández-Lobato Bayesian batch active learning as sparse subset approximation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Fox. and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http//ttsss//ppcceciiigii http:///wwcoeeeiingg...e...pppp. cc/ pdf
Rateliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychol Rev, 97(2):285-308, April 1990.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. CoRR, abs/1810.111911 2018
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7 (2):123 146, 1995. doi: 10. 10.1000095440095553333,,8 URL httphhtts:/// TRLLattpp:://oll.o///0..11
09540099550039931
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai. ACM, 54-63, nov 2020. ISSN 0001-0782. doi: 10.334533381811. URL htttps://6oi.org//11 1145/3381837
Holger Schwenk, Loic Barrault, Alexis Conneau, and Yann LeCun. Very deep convolutional networks for text classification. In EACL, 2017.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach, 2017. hhtttps://axxiv.rrggsgoos
00489
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: core-set
approach. In International Conference on Learning Representations, 2018. URL https:/7
appenrevewwnne//fe wtwp://nnnn//oommm/ttttt1 BacehllllkkkPPe Tuk-RMi
Yuya Senzaki and Christian Hamelain. Active learning for deep neural networks on edge devices, www.aa/wwww....a/////booooo
2021. URL
Burr Settles. Active learning. Morgan & Claypool Publishers, 2012.
Burr Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Maddion,, 2009. URL /ttp://axoo... cs wwt.:yy....aaaaaeeeeeeeeeeeeeeee 778/Papers act ve learning.
13

---

Under review as a conference paper at ICLR 2023
Burr Settles. From theories to queries: Active learning in practice. In Isabelle Guyon, Gavin Cawley, Gideon Dror, Vincent Lemaire, and Alexander Statnikov (eds.), Active Learning and Experimental Design workshop In conjunction with AISTATS 2010, volume 16 of Proceedings of Machine Learning Research, pp. 1-18, Sardinia, Italy, 16 May 2011. PMLR. URI https //proceedings.. ml prwss v1 w/settsteese.
html
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron- Training multi- parameter language models using model parallelism. CoRR, abs/1909.08055 00053,, 2019.
1909/9999.000 08053
Michalis K. Titsias, Jonathan Schwarz, Alexander G. de G. Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning with gaussian processes. In International Conference Learning Representations, 2020. URL https: http://://ppeeeeee..eeeeeeee//ee
[d=HkxCZeHFDB
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5, 2018.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 1805. 12471, 2018.
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML'15, pp. 1954-1963. JMLR.org, 2015.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac. Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguisties. URL wttph:tt/////
ac wtt.://wwwwww.wowogggggoooo/o/// 20tp::/.22220mm...mmmmmmmmmm emnlp demos 6.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017. URL httt:://axii.org////
1708.0774)
Bingbing Xie, Qin Jiang, Antonio Mora, and Xuri Li. Automatic cell type identification methods for single-cell ma sequencing. Computational and Structural Biotechnology Journal, 19:5874- 5887, 2021. ISSN -03000 doi: http://doo.oog//1..006......... 2021 10.022. URL https
www. www.ssseeecceereececccc.... rect com/ wttpa//ttteeeeeerr...e/////////.
Jiancheng Yang. Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 191-195, 2021a.
Jiancheng Yang. Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2: A large-scale lightweight benchmark for 2d and 3d biomedical image classification. arXiv preprint arXiv:2 /0.::///0./// 14795, 2021b.
Donggeun Yoo and In So Kweon. Learning loss for active learning. CoRR, URL http:tttp://rxxvoorgggg http:axxww..xrggaa/aa
2019.
03677
Haoyu Zhang, Logan Stafman, Andrew Or, and Michael J. Freedman. Slaq: Quality-driven seabiy-- for distributed machine learning. SoCC '17, pp. 390-404, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450350220. doi: 10.114553127499.1174499 URL
https::/doi.orgg 115...111//17.111322222
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. flanatterlovvellooooouiiioo networks for text classification. CoRR, abs/1509.0166 2015. URL htw.:ttpp.//aart...orr//////////
14

---

Under review as a conference paper at ICLR 2023
Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang. Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Joseph E. Gonzalez, and Ion Stoica. Alpa: Automating inter- and intra-operator parallelism for distributed deep learning. CoRR, abs/2201.12023,, 2022. 2022.
URL hhttps::/aaxiv..oto abwpggg/ab//aaaa .2000...20... 12023
Wei Wu Sina Lin Mandeep Baines Vinay Ramakrishnaiah Carlos Efrain Nirmal Prajapati Pat Mc- Cormick Jamaludin Mohd- Yusof Xi Luo Dheevatsa Mudigere Jongsoo Park Misha Smelyanskiy Zhihao Jia, Colin Unger and Alex Aiken.
Zongwei Zhou, Jae Y. Shin, Suryakanth R. Gurudu, Michael B. Gotway, and Jianming Liang. Active continual fine tuning of convolutional neural networks for reducing annotation efforts. Medical Image Analysis, 71:101997, 2021. ISSN 1361-8415. doi: https://doi.org/10.11111
media.2021. 101999 URL https:/ www. www.::.ceiinediireee.ee.. com www. pii 011/9361141522100533
Jia-Jie Zhu and José Bento. Generative adversarial active learning. CoRR, httpp///00..095556 2017.
URL http:://a///xii.... org/ abs//102. 07956
A ADDITIONAL EXPERIMENTAL DETAILS ON MAIN RESULTS
A.1 RESULTS IN TABULAR FORM
In this section, we report all results presented in Section 3.1 and Section 3.2 tabular form. All methods highlighted in blue are methods that use CAL
Test Accuracy (%)
Factor Speedup
Method
10%
15
20 %
25%
30%
10%
15%
20
25
30 %
CAL-ER
92.6 a.
93.9 02
94.5 t.
94.9 + 02
94.9 0:02
1.5x
1.4x
2.0x
2.4x
2.8
CAL-MIR
92.6 =03
93.9 #02
94.5 6:00
94.9 t:
94.9 0:00
0.9 x
1.2x
1.3x
1.5x
1.7x
CAL-DER
92.7 201
93.9 1:1
94.5 t:0
94.8 6:
94.9 1:01
1.4 x
2.0x
2.4x
2.7x
3.1x
CAL-SD
92.6 =n1
94.0 1.2
94.5 1:01
94.8 02
94.9 1:01
1.4 x
20x
2.4x
2.7x
3.1x
CAL-SDS2
92.6 201
94.0 100
94.6 1:00
94.9 1:
94.9 1101
1.1x
1.5x
1.7x
1.9x
2.1x
AL w/ WS
92.7
93.8
94.4
94.6 + o.1
94.4
1.1x
1.4x
1.5x
1.5x
1.5x
AL
92.6 03
93.8
94.4
94.9
94.9 a.
1.0x
1.0x
1.0x
1.0%
1.0x
Table 1: FMNIST Results
Test Accuracy (%)
Factor Speedup
Method
10%
20%
30%
40%
50%
10%
30%
40 %
50%
CAL-ER
82,1 4.s
89.9 a.
92.4 a
93.5 =11
93.7 2:3
1,6x
2,8x
4.0x
5.3x
6.5x
CAL-MIR
82.4 104
89.5 1.03
92.6 a
93.6 =1
93.8 202
0.7 x
1.0x
1.4x
1.8x
2.2x
CAL-DER
83.5 1.1
90.0 14
92.3 i:1
93.1 =2
93.4 201
1.4x
2.3x
3.2x
4.2x
5.2x
CAL-SD
83.0 +.00
90.0 6.44
92.7 #2
93.3 as
93.9 +03
1.4x
2,2x
3.2x
4.1x
5.1x
CAL-SDS2
82,5 t.
90.1 6.
92.9 t4
94.0 02
94.4 #:1
1.1x
1.6x
2.1x
2.7x
3.4x
AL w/ WS
81.9
89.6 os
92.4
93.5
94.1
1.2x
1.1X
1.1X
1.1x
1.1x
AL
82.0
89.1
92.1
93.5
93.8
1.0x
1.0x
1.0x
1.0x
1.0x
Table 2: CIFAR-10 Results
15

---

Under review as a conference paper at ICLR 2023
Test Accuracy (%)
Factor Speedup
Method
10%
15%
20
25%
30%
10%
15 %
20% %
25%
30
CAL-ER
56.8
62.6
64.3
63.8
62.6
1.5x
2.3x
2.8x
3.3x
3.9x
CAL-MIR
60.8
66.0
64.3
68.3
69.5
1.1x
1.3x
1.3x
1.5x
1.7x
CAL-DER
65.5
69.1
69.9
70.1
71.9
1.3x
1.9x
2.2x
2.6x
3.0x
CAL-SD
62.9
68.5
69.3
70.8 06
70.7
1.3x
1,9x
2.2x
2.6x
3.0x
CAL-SDS2
61.3
29.1
69.4
70.2
70.7+
1.1X
1.5x
1.7x
2.0x
2.2x
w/ WS
66.0 1:
65.6 2.
69.4 a
69.7 2:
70.7 2:04
1.1x
1.4x
1.6x
2,1x
2.2x
AL
66.2 1.4
70.2 d 0.
68.7 a
69.4 2:
71.4 1.
1.0x
1.0%
1.0x
1.0x
1.0x
Table 3: MedMNIST Results
Test Accuracy (%)
Factor Speedup
Method
10%
20%
30%
40%
50%
10%
20%
30%
40 %
50% %
CAL-ER
90.7
92.4
93.5
93.7
93.8
1.5x
3.5x
3.3x
4.1x
4.9x
CAL-MIR
92.0
92.9
93.3
93.7
93.7
0.9x
1.3x
1.8x
2.3x
2.7x
CAL-DER
92.9
94.1
94.0
93.6
94.2
1.5x
2.4x
3.2x
4.0x
4.7x
CAL-SD
92.1
92.6 1:4
93.7
93.8 2:
94.1
1.5x
2,4x
3.2x
4.0x
4.7x
CAL-SDS2
92.6 1:33
93.2 1.
93.6
93.8
94.1
1.2x
1.9x
2.5x
3.1x
3.7x
AL w/ WS
92.6 os
93.0 6.0
93.0 a.
93.2 03
93.1 20
1.0x
1.0x
1.0x
1.0x
1.0x
AL
92.8 1.02
93.1 1:00
93.3 à 1.1
93.8 2:08
94.1 202
1.0x
1.0x
1.0x
1.0x
1.0x
Table 4: Amazon Polarity Results
Test Accuracy (%)
Factor Speedup
Method
2.9%
5.7%
8.6%
11.4%
14.3%
2.9%
5.7%
8.6%
11.4% %
14.3%
CAL-ER
73.2 1.
75.4 +.0
76.4 1.
77.0 20
77.3 1:
1.7x
2.8x
3.9x
5.0x
6.1x
CAL-MIR
75.1 402
75.5 1:
76.6 1
76.5 uu
77.0 23
0.8x
1.2x
1.6x
2.0x
2.4x
CAL-DER
71.5 22
74.9 1:
75.7 1:
76.9 1
78.3 =:08
1.7x
2.7x
3.7x
4.8x
5.8x
CAL-SD
73.6 19
74.9 +
77.7 1:
76.4+0.
78.0 =0
1.7x
2.7x
3.7x
4.8x
5.8x
CAL-SDS2
74.7 +28
75.5 1.
77.2 +:0
78.1 as
79.2 05
1.4x
2.1x
2.9x
3.7x
4.5x
AL w/ WS
74,6
76.1
76.3
76.3
77.2
1.2x
1.7x
1.6x
1.8x
1.8x
AL
73.9
75.5 405
76.6
76.3
77.3
1.0x
1.0x
1.0x
1.0x
1.0x
Table 5: COLA Results.
Test Accuracy (%)
Factor Speedup
Method
10%
15%
20%
25%
10%
15%
20%
%
%
CAL-ER
86.3 +..1
88.3 +.0.1
89.7 a3
90.6 u2
91.0 =0.1
1.5x
2.0x
2.4x
2.9x
3,4x
CAL-MIR
86.3 +..1
88.3 +.0.
89.7 +22
90.5
90.9 =02
1.2x
1.6x
1.9x
2.2x
2.6x
CAL-DER
86.9 +.03
88.8 +.
89,9 a3
90.7
91.2 =.11
1.4x
1.9x
2.3x
2.8x
3.3x
CAL-SD
86.3 6.
88.3 t.
89.5 02
90.3
90.8 o:22
1.4x
1.9x
2.3x
2.8x
3.3x
CAL-SDS2
86.3 6.1
88.2
89.2 n3
90.1
90.62
1.4x
1.9x
2.3x
2.8x
3.3x
WS
86.3 +
88.3 +.0.
88.4
us
88.1
1.0x
1.0x
1.2x
1.6x
1.9x
AL
83.6 + 1.0
87.0 +
89.9 +3
1.0x
1.0x
1.0x
1.0x
1.0x
Table 6: Single-Cell Cell-Type Identity Classification Results
A.2 HYPERPARAMETERS
For every dataset and every CAL/AL strategy, learning rate (Ir) and batch size (m) chosen based on whichever setting achieves highest performance on standard AL. For all CAL methods, replay size m' (m, 2m) (used in all CAL methods), {0.1, 0.25, 0.5, 0.75) (used in CAL-DER, CAL-SD,
16

---

Under review as a conference paper at ICLR 2023
and CAL-SDS2), 3 {0.75, 1} (used in CAL-DER), E {0.1,1) (used in CAL-SDS2), and E 10.1,1, (used in CAL-SDS2).
the hyperparameter used in CAL-MIR and CAL-SDS2
to subsample the history before finding the m' samples to replay, but this parameter is not tuned for any of the presented results. We list the specific set of hyperparameters we use for all the main experimental results in this section.
A.2.1 FMNIST
All experiments for FMNIST used a ResNet-18 18 with an SGD optimizer, with learning rate of 0.01 and batch size of 64. For all the CAL methods, we fix m' = 128. A NVIDIA GeForce RTX 1080 GPU was used to run all the reported experiments.
CAL-MIR = 256
CAL-DER a -1,
CAL-SD .25
CAL-SDS2 C = 256. 25, 0.1, ==
A.2.2 CIFAR-10
All experiments for CIFAR- used a ResNet-18 with an SGD optimizer, with learning rate of 0.02 and batch size of 20. For all the CAL methods, we fix m 40. Training is done NVIDIA GeForce RTX 2080.
CAL-MIR 100
CAL-DER a .1,
CAL-SD .25
CAL-SDS2 C= 100, 25, 0.1, 0.1
A.2.3 MEDMNIST
All experiments for MedMNIST used a ResNet-18 with an Adam optimizer, with learning rate of 0.001 and batch size of 128. For all CAL methods, fix m' 128. All reported models were trained on an NVIDIA GeForce RTX 2080.
CAL-MIR 270, m' 128
CAL-DER m' -128, a = .1, 3
CAL-SD m' 128, a .5
CAL-SDS2 270, m' 128, a 5, 0.1, x= 10
A.2.4 AMAZON POLARITY REVIEW
Throughout our experiments, we sample 2M sentences, and use them as the total training set instead.
We use Adam optimizer with standard parameters with learening rate of 0.001 and batch size 128. For all the CAL methods, we fix m' = 128. All reported models were trained on an NVIDIA GeForce 1080 Ti.
CAL-MIR C = 256,
17

---

Under review as a conference paper at ICLR 2023
CAL-DER 22, 3 0.75
CAL-SD
Q= .5
CAL-SDS2 C' = 256, 75, 1,1=1
A.2.5 COLA
For all our experiments we use Huggingface's transformer library Wolf al. (2020) and use
maximum sentence length of 100. We use Adam optimizer and learning rate of 5 10 use a batch size of 25 and m' 25. Models were trained on single NVIDIA GeForce 1080 Ti.
CAL-MIR 50
CAL-DER a 0.25, 5 0.75
CAL-SDS 0.75, 8 0.25
CAL-SDS2 C 50, 0.5, B 0.5, 1,
A.2.6 SINGLE-CEI CELLE CELL-TYPE IDENTITY CLASSIFICATION
All experiments use SGD optimizer with standard parameters with learning rate of 0.001 and batch size 128. For all the CAL methods, fix m 128. Training done an NVIDIA A100-PCIE-40GB.
CAL-MIR 200,
CAL-DER a == 31 =1
CAL-SD a a=1
CAL-SDS2 100, a = 0
B RESULTS FOR ADDITIONAL ACTIVE LEARNING STRATEGIES
In this section, demonstrate that CAL methods are able accelerate AL strategies other than entropy sampling without incurring any significant performance drops. test multiple AL strategies on and FMNIST Xiao al.) (2017) and CIFAR-10 Krizhevsky (2009). Note that the speedups are approximately the same as the ones reported Section since the training time is generally independent of the selected AI strategy.
B. OVERVIEW OF STRATEGIES
Margin Score Sampling This strategy is another form uncertainty sampling Settles (2009) as described in the main paper. Instead of the entropy of [x; 0), the margin score is used as the entropy score i.e. h(x) e 1 - (f(2; 0); f(x; 0)₂) where i and are the indices corresponding to the highest and second highest values of f(x;; 0) respectively.
FASS FASS Wei et al. (0155) is a two-staged selection method that uses both uncertainty sampling and submodular maximization. Initially, a set of samples A of cardinality * b, is chosen from 2 using uncertainty sampling, where e > 1 is a tuneable hyperparameter. Next, U, is constructed by greedily selecting samples that maximize submodular set function G 2 - R4 defined on ground set A. Entropy is once again used as the uncertainty metric for the initial stage. For the second stage, G is defined to be the facility location function Wei al. (015) expressed below:
18

---

Under review as a conference paper at ICLR 2023
G(S)
Σ
max Wifs
3GEA ByES
where S C A and wij similarity score between samples z; and zj. In our experiments, wi exp (-1 2j 1|2/ where 24 is the penultimate layer representation of model f for x; and a is hyperparameter.
GLISTER GLISTER Killamsetty et al. (2021) solves a bi-level optimization problem in order to select samples to label. Specifically, GLISTER solves
arg max LLy (arg max (0r V) 1Sus,
(8)
where LLv is the log- ligliiillhood on the validation set V, and LLT the log- log-likeldoood on the subset
S.
B.2 RESULTS
Test Accuracy (%)
Method
10 %
15 %
20%
25%
30 %
CAL-ER
92.8 2.11
94.1 1.1
94.8 1.
95.1
0:03
95.2 : 0.2
CAL-MIR
92.6 = 0.
94.1 0.
94.9 1:02
95.0 0:
95.2 0.2
CAL-DER
91.8 = 0.5
93.1 0.1
94.3 03
94.6 + 0.1
94.8 0.2
CAL-SD
92.5 #.1
93.8 0.1
94.8
95.1
95.2
CAL-SDS2
87.8 1.1
93.4 0.1
94.6
95.0 + 0.2
95.2 0.1
AL w/ WS
92.8 0.0
94.0 0.3
94.6 + o.1
94.8 + 0.1
95.0 0.2
AI.
92.7 8.1
94.1
94.9 o.1
95.0
95.2
Table 7: FMNIST with Margin Score Sampling
Test Accuracy (%)
Method
10 %
20 %
30%
40%
50%
CAL-ER
81.5
89.3
92.2+
93.4
93.8
CAL-MIR
81.9 0.1
89.6 + 02
92.2 + 0.4
93.6 + a.
94.0 0.2
CAL-DER
83.0
89.5 02
92.2 002
93.2 + 0.a
93.6
CAL-SD
82.6 0.4
89.9 0.
92.4 42
93.5 a.11
93.8 0.2
CAL-SDS2
82.5 0.2
90.2 02
92.5 1.02
93.8
94.1 0.1
AL WS
83.1 = 0.1
90.3 0.0.
93.0 1:02
93.5 0.03
93.6 :: 0.2
AL
75.1
87.1
90.2 o.s
92.0= 0.0
92.8 0.5
Table 8: CIFAR-10 with Margin Score Sampling
19

---

Under review as a conference paper at ICLR 2023
Test Accuracy (%)
Method
10 %
15 %
20%
25%
30 %
CAL-ER
92.6
93.9 =
94.6 002
95.0
94.9
CAL-MIR
92.5 0.1
93,8 03
94.6 1011
94.8 0.1
94.9 0.2
CAL-DER
92.7 2.11
93.8 1.11
94.5 10.1
94.7 1..1
95.0 0.2
CAL-SD
92.8
93.9 1.1
94.7 +.1
94.8 03
94.9
CAL-SDS2
92.8 0.0
93.8 0.02
94.5 0.1
94.8 4:02
94.9 0.1
AL w/ WS
92.5 0.1
93.8
+ 0.3
94.0 + oz
94.3 + 0.2
94.3 # 0.0
AL
92.7
93.9
0.1
94.5 0.1
94.7
94.8
Table FMNIST with FASS
Test Accuracy (%)
Method
10% %
20
30%
40%
50%
CAL-ER
82.2 02
89.8 02
92.5 n.2
93.44 0.4
93.7 0.2
CAL-MIR
82.2 0.3
89.4 0.2
92.3 0.
93.4 +: 0.0
93.5 0.1
CAL-DER
83.1 0.3
89.7 0.2
91.9 1.1.1
93.1 0:
93.5 0.1
CAL-SD
83.0 = 0.3
90.0 = 0.3
92.5 0.
93.5 + 0.1
94.0
= 0.1
CAL-SDS2
83.0
90.1 0.1
92.7 0
93.5 0.2
94.0
AL w/ WS
82.8
90.3
92.8
93.6
93.7
AL
72.5 2.0
86.6 + 0.4
90.1
91.7 0.2
92.9 0.2
Table 10: CIFAR-10 with FASS
Test Accuracy (%)
Method
10 %
15 %
20%
25%
30%
CAL-ER
92.6 = 0.0
93.9 02
94.3 1.1
94.7 +:
94.7 = 0.2
CAL-MIR
92.5 0.0
93.9 0.
94.3 102
94.4 0
94.6 0.1
CAL-DER
92.7 #.11
93.9 02
94.3 4:03
94.7 4:02
94.9 0.3
CAL-SD
92.6 #.1
93.8 0.0
94.4 0:
94.6 0.1
94.7 0.1
CAL-SDS2
92.6 0.1
93.9 22
94.4 102
94.6 + 0.3
94.7 = 0.2
AL w/ WS
92.5 0.1
93.6 =
93.9 +
94.1 + 0.1
94.3 0.1
AL
92.5
93.8 0.1
94.2 + 0.
94.6
94.7
Table 11: FMNIST with GLISTER
Test Accuracy (%)
Method
10
20%
30%
40%
50%
CAL-ER
81.7 0.3
89.2 0.2
91.9 1:02
93.0 0.0.1
93.3 0.1
CAL-MIR
81.6 203
89.3 04
91.7 1002
92.9 1.0.1
93.5
0.2
CAL-DER
82.8 04
89.5 2.4
91.7 +04
92.8 0.6
93.1
CAL-SD
82.5 0.3
89.6 02
92.1 02
93.1 +:0
93.8 + 0.1
CAL-SDS2
81.4 0.4
89.1
92.1
93.2 + 0.3
93,9
AL w/ WS
81.7
89.3
92.1
93.0
93.3
AI
81.0 0.6
88.5 2
91.5 0:
93.0 0.2
93.4 0.3
Table 12: CIFAR-10 with GLISTER
20

---

Under review as a conference paper at ICLR 2023
C ADDITIONAL DETAILS ON SINGLE-CELL CELL-TYPE IDENTITY CLASSIFICATION DATASET
The human cell landscape (HCL.) dataset consists of scRNA-seq data for 562,977 cells across 63 cell types represented in 56 human tissues. Fach cell type may be present in multiple tissues. The cell type classes are highly imbalanced, with the rarest cell type, human embryonic stem cell, accounting for 0,00006 % of the total dataset and the most common, fibroblast, accounting for 0.06%. The raw data first normalized for library-size and scaled 10000 reads in total, followed by log-trant traasffrrrmmiinn. We visualize the dataset using UMAP
Single Tye
A12 cell
hoaal achar coll
Macreshage
pland intermedtry soll
chanshocyts
call
Aarocyts
Fotal emorscyle
Manocyts
cell
Fotal cpstiuu progertor
Myrtad col
CB CO34
Fotal
Chandroyte
Fotal soricca muxce cal
Parceesn esccrie call
Promrrial
Castric
Proe
Drdotheial ls (eneeme ut mesanchruul transtor)
Custric endoorie
nusiball ardenwitatin cat
Drterzcyts
Coblet
Smooth mucle
Epthetaical
Stomal
Epeteitaii cal (nternedute))
intrraaineed aodd call
Teal
Tryrust
proerao
lamte NG Merrahaage
Ventrcte cartoomperyte
Firsa Neure
NESC
Figure 9: UMAP embedding of single cells in HCL annotated by their cell type.
21


Reviews:
[{"review_id": "stwc38vUeNL", "reviewer": "Reviewer_wxxe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to investigate the computational efficiency of Active Learning (AL). The authors argue that typical AL methods are computationally intensive, as models are retrained from scratch at every round. They propose to reframe the AL setting as a Continual Learning one, which allows them to leverage the vast literature on replay-based CL. The authors conduct a series of experiments testing a variety of existing and novel rehearsal based methods in their new Continual Active Learning setup. Interestingly, the authors show that using CL methods in Active Learning not only provides significant (2-6x) computational gains, but does so without incurring a loss in performance. ", "summary_of_the_review": "In summary, the authors address an important limitation in the field of Active Learning. I believe that with a more detailed analysis of the results, and a clear explanation as to why replay based methods can actually be more efficient and performant than their iid counterpart, would make this a strong paper. \nFor now, given that several key things are missing from the paper, I think this paper still needs work. Should the authors provide more insight and a clear response to the issue raised above, I believe the paper will be worthy of acceptance. \n\n\nedit : after the rebuttal I have increased my score to a weak accept.", "strength_and_weaknesses": "### Strengths \n1. The paper tackles an important, yet understudied, problem of active learning : the computation cost. Given the resources required for training every larger models, I agree with the authors, and believe that this cost should be addressed. This way, practitioners can compare the cost of labeling vs the cost of training models. \n2. The authors make an interesting (and novel to the best of my knowledge) connection with Continual Learning. By doing so, they show that standard CL techniques can be applied in the AL setting with success. \n3. The authors provide some experiments over different modalities and labelling budgets.\n\n### Weaknesses\n1. The paper is in dire need of a better discussion of the results. I say this for the following reason : The authors **show that using replay (CAL-ER) consistently outperforms regular Active Learning **. My understanding is that this finding directly contradicts the base experiment (figure 1) in (Ash & Adams) [1], where they found that doing full-replay from a warm-started model (CAL-ER), still underperforms a model trained from scratch on all the data (AL). In other words, I think the authors should provide insights (or at least discuss) how a CL model manages to outperform a model trained from scratch on all the data seen so far, which has long been seen as an upper bound in performance. \n2. On Catastrophic forgetting in AL (fig 1). How are the models cross-validated at every round in CAL ? I don't quite agree with using the word \"catastrophic forgetting\" to describe the behavior in figure 1, since the test accuracy (in general) gets better. It seems more like the the model is overfitting on the training samples of each round. \n3. The paper would greatly benefit from larger scale experiments. I think the question of compute efficiency is much more relevant in such settings. I understand the increased cost that's required to run these. \n\n[1] On Warm-Starting Neural Network Training, https://arxiv.org/pdf/1910.08475.pdf, Neurips 2020", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "eEQcAX69VVw", "reviewer": "Reviewer_wt23", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors state that the efficiency of traditional active learning (AL) should be significantly increased using approaches from continual learning (CL), when we consider past datapoints already used for training in AL as previous tasks. Authors test several standard CL approaches and newly proposed CL methods specifically designed for AL in the AL setting. On datasets FMNIST, CIFAR-10, MedMNIST, and Amazon Polarity, author demonstrate the ability to accelerate AL up to 2-6 times in some settings.", "summary_of_the_review": "This paper is not ready for publication.", "strength_and_weaknesses": "+ The paper is well written, the motivation is very clear.\n- The main problem is the insufficient review of related work and previous approaches. I need not only a citation of previous work, but also a thorough review of the novelty of the proposed algorithm over the existing literature.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "BVbLKK3KLr5", "reviewer": "Reviewer_nCXE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a continual learning technique for the active learning. In particular, a set of the continual learning techniques are adopted and integrated into each model training procedure at each cycle of active learning. The proposed CAL regards the newly-queried sample at each round a new task as in the continual learning setting, then apply continual learning techniques combined with warm-starting to active learning. As a result, CAL accelerates the active learning process by improving computational efficiency at the model training stage in active learning. The contribution is demonstrated with extensive experiments in benchmarks across different domains (NLP, vision, medical imaging, and computational biology).", "summary_of_the_review": "Based on the above comments, I recommend a reject on this paper.\n\nThis paper should significantly strengthen the integration of the active learning and continual learning, rather than simply superficially coupling both.", "strength_and_weaknesses": "The idea of this paper is well-received.\n\nMy main concern is listed as follows.\n1. the novelty. It is yet to be answered if the performance gain is truly brought by the active learning framework, or the promotion from the classifiers.\n2. this paper may seem a bit incremental to me where a lot of existing techniques were combined into a whole.\n3. following 2, the integration of the existing continual learning techniques are generally loose. In other words, how can we attain better querying policy via the integration of continual learning remains untapped.\n4. the establishment of the new setting needs further explanation.\n\nThe other more detailed weaknesses are:\n\n1. With all due respect, I may disagree with the author on the simple statement, \"active learning can be viewed as a continual learning problem\". While indeed that the active learning and continual learning seem to have something in common (i.e. new sample being added into the pool at every cycle), crucially a more vital stage of the active learning is the querying policy of attaining new batch of unlabeled data that are gauged to bring maximal performance gain. The paper didn't explain this aspect adequately.\n\n2. Section 3 is occupied by the existing settings or techniques. It seems to me that the section 3.1 and 3.2 fit the background interpretation while section 3.3 describes the existing continual learning techniques. These, however, seems to very much orthogonal to the active learning.\n\n3. Following 2, the major methodological innovation lies in the proposition of CAL-SD and CAL-SDS2. It generally lacks essential details that can correlate active learning and continual learning. For example, this paper emphasize a \"stability-plasticity\" problem in CAL-DS, but I don't see a clear connection with the active learning. The sub-modular sampling in CAL-SDS2 also seems orthogonal to active learning. To summarize, this paper seems to simply utilize the continual learning to an active learning setup incrementally.\n\n4. The empirical analysis in section 4.2 is a bit hard for me to understand, with a surprisingly short piece of analysis. I suggest the authors to focus on CAL-SD and CAL-SDS2 and offer an ablation analysis accordingly, such as the impact of $\\mathcal{L}_c$ and $\\mathcal{L}_{\\rm replay}$.\n\n5. A major problem for active learning is the costly querying process, which is not shown in this paper because the authors only apply continual learning to uncertainty sampling strategies, while there are many existing. I suggest that the authors should consider other types of baselines. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6J6R_Xq6rl", "reviewer": "Reviewer_HMvu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper suggests a set of selection criteria to training points in each AL step to avoid the so-called catastrophic forgetting which degrades model performance. A variety of heuristics is proposed some motivated by prior work and 2 new ones proposed by the authors (CAL-SD and CAL_SDS2).", "summary_of_the_review": "To this end I feel the paper still needs to be refined, and the empirical test show a trend that reduces the novelty and significance of the proposed sampling criteria for CAL. Nevertheless, this is an important problem and I would like to authors to continue its pursuance.\n\n", "strength_and_weaknesses": "Strengths\n1.\tThe problem tackled in this work is important improving the running time and performance of Active  Learning techniques.\n2.\tThe Authors show in their experiments that indeed it is important to do continual learning and that acceleration can be obtained in this manner.\n3.\tThe paper is well organized\nWeaknesses\n1.\tMy main concern is that the empirical results show the simple CAL-ER is the actual winner in acceleration in most cases, and often even the trade-off with accuracy makes the favorable one. Well,…that means that there is no point in any of the ‘intelligent’ criteria, but just to have a diverse set (which can be obtained by the simple uniform sampling of CAL-ER). I would like to get the authors opinion on this observation, please.\n2.\tThere are some parts that are unclear. For example, \na.\tthe setting of experiment in 3.3 isnt clear and confusing: its not clear if the model is fully trained first and with what data? What is the accuracy measured for? what does the statement ‘the model performs considerably worse on all of the tasks that is does on the test set? what are you measuring the performance on, if not the test set.\nb.\twhat are “reasonable choices of m’ “?\n3.\tI don’t understand how the CAL-MIR selection of m’ is done. Is it done exhaustively over all the loss minimizer subsets? That prohibitive at some point, right?\n4.\tHow is the selection of alpha and beta are done for (2) in the experiments?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject