IALE: Imitating Active Learner Ensembles
Authors: Christoffer Löffler
Christopher Mutschler
Venue:  (2021)
Keywords: active learning
imitating learning
ensembles

Abstract: Active learning (AL) prioritizes the labeling of the most informative data samples. However, the performance of AL heuristics depends on the structure of the underlying classifier model and the data. We propose an imitation learning scheme that imitates the selection of the best expert heuristic at each stage of the AL cycle in a batch-mode pool-based setting. We use DAGGER to train the policy on a dataset and later apply it to datasets from similar domains. With multiple AL heuristics as experts, the policy is able to reflect the choices of the best AL heuristics given the current state of the AL process. Our experiment on well-known datasets show that we both outperform state of the art imitation learners and heuristics.

Under review as a conference paper at ICLR 2021
IALE: IMITATING ACTIVE LEARNER ENSEMBLES
Anonymous authors
Paper under double-blind review
ABSTRACT
Active learning (AL) prioritizes the labeling of the most informative data samples. However, the performance of AL heuristics depends on the structure of the under- lying classifier model and the data. We propose imitation learning scheme that imitates the selection of the best expert heuristic at each stage of the AL. cycle in a batch-mode pool-based setting. We use DAGGER train the policy on dataset and later apply it to datasets from similar domains. With multiple AL heuristics as experts, the policy is able to reflect the choices of the best AL heuristies given the current state of the AL process. Our experiment on well-known datasets show that we both outperform state of the art imitation learners and heuristics.
INTRODUCTION
The high performance of deep learning on various tasks from computer vision (Voulodimos et al,, 2018) to natural language processing (NLP) (Barrault et al. 2019) also comes with disadvantages. One of their main drawbacks is the large amount of labeled training data they require. Obtaining such data is expensive and time-consuming and often requires domain expertise.
Active Learning (AL) is an iterative process where during every iteration an oracle (e.g. a human) is asked to label the most informative unlabeled data sample(s). In pool-based AL all data samples are available (while most of them are unlabeled). In batch-mode pool-based AL, we select unlabeled data samples from the pool in acquisition batches greater than Batch-mode AL decreases the number of AL iterations required and makes it easier for an oracle to label the data samples (Settles) 2009). As selection criteria we usually need to quantify how informative a label for particular sample is. Well-known criteria include heuristics such as model uncertainty (Gal et al.) 2017 Roth Small [0066] Wang & Shang, 2014] Ash et al., 2020),, data diversity (Sener & Savarese 2018), query-by berryy-y-committe (Beluch et al.) 2018). and expected model change (Settles et al.) 2008). As ideally we label the most informative data samples at each iteration, the performance of a machine learning model trained on labeled subset of the available data selected by AL strategy is better than that of a model that is trained on a randomly sampled subset the data.
Besides the above mentioned, in the recent past several other data-driven AL approaches emerged. Some are modelling the data distributions (Mahapatra et al. 2018:) Sinha et al.) 2019 Tonnaer] [2077] Hossain et al. 2018)
pre-processing step, or similarly metric-based meta-learning (Ravi & Larochelle 2018 Contardo et al.] (017)) as a clustering algorithm. Others focus on the heuristics and predict the best suitable one using a multi-armed bandits approach (Hsu & Lin 2015). Recent approaches that use reinforcement learning (RL) directly learn strategies from data (Woodward & Finn 2018 Bachman et al., 2017 Fang al. (017). Instead of pe-processing data or dealing with the selection of a suitable heuristic they aim to learn an optimal selection sequence on given task.
However, these pure RL approaches not only require a huge amount of samples they also do not resort to existing knowledge, such potentially available AL heuristics. Moreover, training the RI. agents usually very time-intensive as they are trained from scratch. Hence, imitation learning (II.) helps in settings where very few labeled training data and potent algorithmic expert available. IL aims to train, i.c., clone, a policy to transfer the expert to the related few data problem. While IL mitigates some of the previously mentioned issues of RL, current approaches are still limited with respect to their algorithmic expert and their acquisition size (including that of Liu et al. (018)), i.e., some only pick one sample per iteration, and were far only evaluated on NLP tasks.
We propose an batch-mode AL approach that enables larger acquisition sizes and that allows to make use of more diverse of experts from different heuristic families, i.e., uncertainty, diversity,

---

Under review as a conference paper at ICLR 2021
expected model-change, and byeemite- Our policy extends previous work (see Section by learning at which stage of the AL cycle which of the available strategies performs best. We use Dataset Aggregation (DAGGER) to train robust policy and apply it to other problems from similar domains (see Section 3p. We show that we can (1) train policy on image datasets such as IN, ST, Kuzushi ji -MNIST, and CTFAR- 0,, (2) transfer the policy between them, and (3) transfer the policy between different classifier architectures (see Section[4)
RELATED WORK
Next to the AL approaches for traditional ML models (Settles) 2009) also ones that are applicable to deep learning have been proposed (Gal et al. 2017; Sener & Savarese 2018 (Beluch et al.. 2018 Settles et al. 2008 Ash et al. 2020). Below we discuss AL strategies that are trained on data.
Generative Models. Explicitly modeled data distributions capture the informativeness that can be used to select samples based on diversity. (Sinha et al. (2019) propose a pool-based semi-supervised AL where a discriminator discriminates between labeled and unlabeled samples using the latent representations of a variational autoencoder. The representations are used to pick data points that are most diverse and representative (Tonnaer) 2017). Mirza & Osindero (2014) use a conditional generative adversarial network to generate samples with different characteristics from which the most informative are selected using the uncertainty measured by Bayesian neural network (Kendall) & Gal 2017: Mahapatra et al. 2018). Such approaches are similar to ours (as they capture dataset properties) but instead we model the dataset implicitly and infer a selection heuristic via imitation.
Metric Learning. Metric learners such as Ravi & Larochelle (2018) use set of statistics calculated from the clusters of un-/labeled samples in a Prototypical Network's 's (Snell et al., 2017) embedding space, or learn to rank (Li et al. 2020) large batches. Such statisties distances (e.g. Euclidean distance) or are otherwise converted into class probabilities. Two MLPs predict either a quality or diversity query selection using backpropagation and the REINFORCE gradient (Mnih & Rezende, 2016). However, while they rely on statisties over the classifier's s embedding and explicitly learn two strategies (quality and diversity)
a richer state and are not constrained to specific strategies.
Reinforcement Learning (RL), The AL cycle can be modeled as a sequential decision making problem. Woodward problem.WWodward & Finn (018) propose stream-based AL agent based on remory-auggenttee neural networks where an LSTM-based agent learns to decide whether to predict a class label or to query the oracle. Matching Networks (Bachman et al. 2017) extensions allow for pool-based AL. Fang et al. (2017) use Deep Q-Learning in stream- stream-based AL scenario for sentence segmentation. In contrast to them we consider batch-mode AL with acquisition sizes 2 1, and work on a pool- instead of stream stream-settings. While Baihmann et al. (2017) propose strategy to extend the RL-based approaches to pool setting, they do still not work on batches. Instead, we allow batches of arbitrary acquisition sizes. Fan et al.) 2018)) propose a meta-learning approach that trains a student-teacher pair via RL. The teacher also optimizes data teaching by selecting labeled samples from a mini- batch that lets the student learn faster. In contrast, our method learns to selects samples from an unlabeled pool, i.c., in a missing target scenario. The analogy of teacher-student is related, however, the objective, method and available (meta-) dlat learn good teacher (policy) are different.
Multi-armed Bandit (MAB). Baram et al. 2004 treat the online selection of AL heuristics from an ensemble as the choice in a multi- armed bandit problem. COMB uses the known EXP4 algorithm to solve it, and ranks AL heuristics according to a semi-supervised maximum entropy criterion (Clas- sification Entropy Maximization) over the samples in the pool. Building on this Hsu & Lin (2055) learn to select an AL strategy for an SVM-classifier, and use ampportacce-eiihteee accuracy exten- sion to EXP4 that better estimates each AL heuristics' performance improvement, as an unbiased estimator for the test accuracy. Furthermore, they reformulate the MAB setting so that the hcuristics are the bandits and the algorithm selects the one with the largest performance improvement, con- trast to COMB'S s formulation where unlabeled samples are the bandits. Chu & Lin (201)) extend Hsu & Lin 2015) to a setting where the selection of AL heuristics is done through a linear weighting, aggregating experience over multiple datasets. They adapt the semi-supervised reward scheme from Hsu & Lin 2015)) to work with their deterministic queries. In our own work, we instead learn a unified AL policy instead of selecting from set of available heuristics. This allows our policy to lear interpolation between batches of samples proposed by single heuristics and furthermore, to exploit the classifier's internal state, so that it is especially suited for deep learning models.
2

---

Under review as a conference paper at ICLR 2021
Imitation Learning (IL). Liu et al. (2018) propose neural network that learns AL strategy based on the classifier's loss on a validation set using Dataset Aggregation (DAGGER) (Ross et al, 2011). One of their key limitations is that only a single sample is labeled during every acquisition. As the DI. model is trained from scratch after every acquisition this results in a very slow active learning process and expensive expert-time is requested less efficiently (Kirsch et al. 2019 Sener & Savarese| 2018). Hence, we extend this work for batch-mode AL using a top-k-like loss function, and select more samples to increase the suitability deep learning and its efficiency (as do not retrain after each sample). We also incorporate recent ideas (Ash et al. (0200) to extend the state and imitate multiple AL heuristics. This computationally more efficient and leads to better results.
3 IALE: IMITATING AN ENSEMBLE OF ACTIVE LEARNERS
IALE learns an AL sampling strategy for similar tasks from multiple experts in a pool- setting. We train a policy with data consisting of states (i.e., that includes an encoding of the labeled data samples) and best expert actions (i.e., samples selected for labeling) collected over the AL. cycles. The policy is then used on a similar (but different) task. To see states that are unlikely to be pro- duced by the experts, DAGGER (Ross et al. 2011) collects a large set of states and actions over AL iterations. The policy network is trained all the previous states and actions after each iteration.
3.1 BACKGROUND
In pool-based AL we train a model M on dataset D by iteratively labeling data samples, Initially, M is trained on a small amount of labeled data Diab randomly sampled from the dataset. The rest of the data is considered as the unlabeled data pool D poot, i.e., D = Diab U DD Pooot From that point onwards during the AL iterations a subset of Dee is selected from Pooot by using an acquísition function a [,M (M,,poo) Poont) ). The data is labeled and then removed from D and added to Drab. The size of Dest is based on the acquisition size aeg (>1 I for batch-mode AL). The AL cycle continues until a labeling budget of B is reached. M is retrained after each acquisition to evaluate the performance boost with respect to the increased labeled dataset only (and not the additional training time).
The acquisition function is heuristic that uses the trained model M to decide which of the data samples in Dpool are most informative. For deep AL popular heuristics include uncertainty- based MC-Dropout (Gal et al. 2017), , serry-y-coomiitee--ee data diversity-based CoreSet (Sener & Savarese 2018), gradient- BADGE (Ash et al. 2020 and soft-max-based Con fonfidennee ideecee or Ent Entropy-sampling (Wang & Shang] 2014).
Ensembles (Beluch et al. 2018
MC-Dropout: uses a Monte-Carlo inference scheme based on a dropout layer to approximate the model's predictive uncertainty (Gal & Ghahramani]
(016). The heuristic (Gal et al. 2017) then uses these values to select the most uncertain samples. Ensembles (Beluch et al. 2018) model predictive uncertainty using a committee of N classifiers initialized with different random seeds. However, while at inference time we need to run only N forward-passer passes per sample (compared to MC-Dropout performing two dozen or more Monte-Carle passes), the training of N 1 additional deep models can become prohibitively expensive in many use-cases. CoreSet (Sener & Savarese 2018) aims to select diverse samples by solving the k-centers problem on the classifier's embed- dings. This involves minimizing the distance between each of the unlabeled data samples to its nearest labeled samples. BADGE uses (pscudo labels of) the magnitudes of the gradients in a batch to select samples by uncertainty, and the gradient directions together with k-means++ clustering to select samples by diversity. Soft-max-basee
heuristics fience and Entropy-samplin use predictive uncertainty and are computationally lightweight lower AL performance (Gal & Ghahramani] 2016 Ash et al., (2020) (Conf i dence selects the samples with the lowest class prob- ability and Entropy the ones with with largest entropy of their probability distribution).
3.2 LEARNING MULTIPLE EXPERTS
Instead of using specific heuristics we propose to learn the acquisition function using a policy net- work. Once the policy is trained on a source dataset it can be applied to different target datasets. [ sketches the idea. The policy network is a Multi-Layer Perceptron (MLP) trained to pre- dict the usefulness of labeling samples from the unlabeled data pool D poot for training the model M, similar an AL. acquisition function. As input the policy network takes the current state, consisting
3

---

Under review as a conference paper at ICLR 2021
3
Choose best expert
Random
Sample
Me (Pab)
(z)
0.8
0.2
Poss
Daub
MLP
BCE Loss
(Dua) M (ar
0.7
Disb
Selection
Target
Figure I: Training π to imitate experts E: (1) we pass samples from Dub and Dias through the
current classifier M; (2) the embeddings and the predictions are input n, whose output vector is compared with the target vector predicted by the best expert; (3) we calculate a loss and back- propagate the error through æ; (4) we extend the labeled pool data Diab by Deet and retrain M.
of the model M's embeddings of pool data next to other elements, extending similar work (Contardo) (ct al.) 2017] Konyushkova et al. 2017 Liu et al. 2018). see below. π then outputs the action to be taken at that step. Action here refers to AL acquisition, i.c., which of the unlabeled data samples should be labeled and added to the training data. æ learns the best actions from a set of experts which predict the best actions for a given AL state. A subset of the pool dataset Dub with size is used instead the whole pool dataset at each active learning iteration for training the policy.
States. As uses the state information to make decisions, state should be maximally compact but still unique, i.e., different situations should have different state encodings. Our state encoding uses types of information: (1) model-dependent parameters (that describe the state from the perspective of the model M and the already labeled samples Diab), and (2) AL-cyele-dependent parameters (that describe the elements of the samples Daub that we can choose to label). Together, these parameters form minimal description of a current state.
We use the following parameters describe the model-dependent aspects:
The mean of the already labeled data samples μa( MM (Diat)): the embedding Me of sam- ple by M is the output of the final layer (i.e., the layer before the soft-max layer in case of classification model), see Figure The size of this representation is independent of the (growing) size of Diab and thus will not become computational bottle-neck.
The ground-truth empirical distribution of class labels
EveD 1 -- (Drabl
Eve DIN.. 1B =
Diob
(Diabl
which is a normalized vector of length i, i.e., the number of classes, with percentage of occurrence per class using the labels of the already acquired data samples.
M's predicted empirical distribution of class labels for the labeled data EM((P...) (i.c., normalized vector as above but with predicted class labels instead of ground-truth).
The rationale for including both the ground truth and the predicted empirical distribution is to enable the policy to base its decisions on the model M's prediction errors. In other words, when the model makes mistakes already labeled samples for class (that were part of the model's training) likely needs more samples from those classes to correct the erroneous predictions.
parameters describe the data samples in D rub that we evaluate the cur- rent iteration. For each data sample x; E Daub we calculate M 's embedding (z,z,) ) in the same em- bedding space as the already labeled samples of our -deeendet parameters. We also predict each sample's 's label (x:). Similarly, we capture the expected model change via per-sample gra- dient information g (x:) in the embedding space for Dsub. using the method proposed in (Ash) et al. 2020). Hence, we consider the gradients of the loss at the embedding layer, given unlabeled samples with proxy labels, both as predictive of model uncertainty and expected model change. The proxy label is the most likely prediction M (2;) (determined by an argmax operation in the softmax layer). The magnitude of the loss gradient at the embedding layer then describes both the model's uncertainty and its expected change.
The AL-cycle-dependee
4

---

Under review as a conference paper at ICLR 2021
This information enables the policy to learn to select samples (1) where the model is uncertain (i.e., where it predicts the wrong labels), (2) where the model might gain most information (i.e. the loss is high), and (3) to learn to select more diverse samples from less well represented classes using the label statistics. Hence, we describe a state & as follows:
Melaod) M (M(())) (((M)(())))
(M(coo)
g (
(P(Ma(Diab)), DDwo: (MDiaa)):
Actions. The action of the MLP is a desirability score Pj for each unlabeled sample from Deub, i.e., Pi == Tss)). We choose the samples to be labeled based on this score. i.e., we choose the top-k ranked values, where k = acq, resulting in a binary selection vector ह = top-k(po, - , pn) with Σi= 0/ = acq. The ground truth actions provided by the experts are binary vectors of length k, where a 1 at index means that Fa should be selected for labeling (and indices of those sample that should not be labeled are 0). From here we can use binary cross entropy loss to update x''s o's weights:
n Σ log (p.) (1 Eog (1 -pa), 1=U
C(p.t)
where is the target vector provided by the best expert (similar to greedy multi-armed bandit approach (Hsu & Lin 2015)). This brings r's output closer to the suggestion of the best expert.
Our IL-based approach uses the experts to turn AL into supervised learning problem, i.e. the action of the best expert becomes the label for the current state s. Our choice of AL heuristics for the set of experts & includes particular types but is arbitrarily extendable. Using MC- Drrouuo Ensemble, CoreSet, BADGE, Confidence or Ent ropy allows us to only minimally modify the classifier model M. aims to learn certain derived properties from the state, such as model uncertainty or similarity. These measures are based on M's predictions and embeddings.
Our hypothesis is that learns to imitate the best suitable heuristic for each phase of the AL cycle, i.., starting with relying on one type of heuristics for selections of samples in the beginning and later switch to a fine-tuning using different one (see also Section 4.3). This is in line with pre- vious research that combines uncertainty- and density-based heuristics and that learns an adaptive combination framework that weights them over the training course (Li & Guo 2013).
3.3 POLICY TRAINING
Algorithm 1:
Imitating Active Learner Ensembles
Input: data D, labeled validation data Draat classifier M, budget B, experts E, acquisition size acg, subset size n. probability states S, actions
Our policy training builds on the intuition behind DAGGER, which is a well-known algorithm for IL that aims to train a policy by it- eratively growing dataset for su- pervised learning. The key idea is that the dataset includes the states that are likely to be visited over the course of solving problem (in other words, those state and action encodings that would have been vis- ited if we would follow a hard- coded AL strategy). To this end, it is common when using DAGGER to determine policy's 's next state by either following the current policy or an available expert (Ross et al.) 2011]. We thus grow list of state and action pairs, and randomly ei- ther choose expert or policy selections as the action.
A, random policy π (acq 2 1, n = 100).
for e 1... www.ocesssee...m maz do
Diah, D t split(D)
while |Dtasl < B do
M < initAndTrain (M, Drab)
D Doub + sample D poots n)
e" <- bestEEppet(( M, D sub, D vat)
Deel + e- Seleecttuurry(( M, D sub, acq)
S.A <- toState (Drut. Drab). toAction Deet)
10
if Rnd(0, 1) 2 p then
// We may choose I's selection
Deel + T T.SelectQuery D Doubi acq)
Dias + Diab U
15
Dpool + D Ppoot Dee!
Update policy using (S,A)
16
Each episode of the IL cycle lasts until the AL labeling budget is reached for episodes iterations. We aggregate the states and actions over all episodes, and continually train the policy on the pairs.
5

---

Under review as a conference paper at ICLR 2021
Class:
Class: 1
bag
bag
sneaker
coot
yu
na
ya
S
4
Figure 2: Examples for the three datasets MNIST, Fashion Matiiion-MMSSS MNIST, and Kuzushi ji ji-MNIST.
We use DAGGER to further randomize the exploration of D. Instead of always following the best expert's advice, we randomly follow the policy's prediction, and thus enrich the possible states.
Our II. approach for training is given in Algorithm| | At each AL cycle, we randomly sample subset sub of n 100 samples from the unlabeled pool 'Pooot (line 3). We find the best expert e* from set of experts & (line 8) by extending the training dataset by the expert selections (from Daub) and train a classifier each. This means that each expert constructs one batch according to its heuristic, e.g., batch composition could maximize model-change, and queries the oracle for labels. We choose the best expert by comparing the resulting classifiers' accuracies on the labeled validation dataset. We next set its acquisition as this iteration's 's chosen target and store state and action for the policy training (line 10). According to DAGGER we then flip a coin and depending on the probability p (line 11) either use the policy the best expert increase Drab for the next iteration (line 14). After each episode we retrain π on the state and action pairs (line 16).
4 EXPERIMENTS
We first describe our experimental setup (Section Next, we describe how we trained our pol- icy (Section 4.2) and evaluate approach by transferring it to test datasets, i.e., to FMNIS' ST and KMNIST (Section 4.3). Finally. we end with a discussion of our ablation studies and the limita- tions of our approach (Section 4.4). The source code is publicly at https:///ittie cr brispchris/ias /alsssialee.e. and can be used to reproduce our experimental results.
com/
4.1 EXPERIMENTAL SETUP
Datasets. We use the image classification datasets MNIST (LeCun et al. 1998), Fash i MNIST
(FMNIST) (Xiao et al. 2017). and Kuzushi ji ji-MNIST (KMNIST) (Clanuwat et al. 2018 for our evaluation. They all consist of 70, 000 grey-scale images (28 x 28px) in total for 10 classes. MNIST contains the handwritten digits 0 - 9, FMNIST contains images of clothing (i.e., bags, shoes, etc.), and KMNI ST consists of Hiragana characters, see Figure
To evaluate IALE we train a policy и and run it (3 repetitions) on unseen datasets along with the baselines. The similarity between FMNIST ST and MNIST (that has previously been shown (Nalis- nick et al., 2019)) and the difficulty of FMNIST (it has been shown to be a demanding dataset for AL methods (Hahn al., 20T9b)) make these datasets perfect combination evaluate IALE. Please find more results for transfering T's AL strategy learned from MNIST to CIFAR-10 10 in Ap- pendix A.3.3
Architectures of classifier M. We use the same model used to evaluate AL on MNIST data that has been used in previous research (Gal & Ghahramani] 2016). Our model has two convolutional layers, followed by max pooling and dense layer. We add dropout layers after the convolution and dense layers and use ReLU activations. A softmax layer allows for classification. We also provide additional results for all the methods with Resnet-18 (He et al. (0116) and with a two-layer MLP with a dense layer (256 neurons) followed by a soft-max layer (Ash et al. 2020) in Appendix A.3.2
Architecture of policy Our policy model neurons each. The first two dense layers are followed by a ReLU activation layer, whereas the final layer has only neuron and the output of this layer is passed onto sigmoid function constrain the outputs to the range [0, and to further process into an aggregating top-k operation.
MLP with three dense layers with 128
Baselines. For the evaluation of the performance of our AL method, implemented different well-known AL approaches from literature: Random Sampling MC-Dropout, Ensemble CoreSet, BADGE, Con dencce-anplinng Ent ropy-saplig and AL.LL (which adapted

---

Under review as a conference paper at ICLR 2021
075
14.5
DACGE
023
D4LE
4.8
DADGE
645
UAEE
Liberin
300
043
20
640
2003
Li
(a) MNIST
(b) F F.saasionnnne OMNNISTT
(c)
J1-MNIST
Figure 3: Active learning performance of the trained policy (trained on MNIST), compared to the baseline approaches and ALIL (Liu et al. 2018), and applied to MNIST, FMNIST and KMNIST datasets.
to work on image classification tasks). For a more detailed description of the baselines please see Appendix A.I.1
4.2 POLICY TRAINING
We use the MNIST dataset our source dataset on which we train our policy for 100 episodes, with each episode containing data from an AL cycle. The initial amount of labeled training data is 20 samples b.lass-balanced). At each step of the active learning process, 10 samples are labeled and added to the training data until a labeling budget B of 1, 000 is reached. We use the AL heuristics MC-Dropout, Ensemble, CoreSet, BADGE, Confi dennee and Entropy as experts, and use Denl with 100 labeled samples to score the acquisitions of the experts. The pool dataset is sampled with = 100 at each AL iteration. We choose 0.5 for DAGGER for means of comparison with the baselines (based preliminary experiments (see Appendix A.2.1
cetpprrtttio--xlptttttiioo We train the policy's ' MLP on the current and growing list of state and action pairs using the binary cross entropy loss from Equation and use the Adam optimizer (Kingma & Ba 2015) for 30 epochs with a learning rate of le 3, 31 0.9, 32 = 0.999, = 1e 8. and without any weight decay.
Figure 3a shows the results of our method in comparison to all the baseline approaches on the MNIST dataset, on which the policy was trained on. Our method consistently outperforms or is at least en par (towards the end, when enough representatives samples are labeled) with all the other methods. IALE performs better acquisitions than Ensemble and MC-Dropout for the impor- tant first half of the labeling budget, where matters the most. Moreover, IALE faster (see Appendix A.3.3). While MC -oo-ut requires 20 forward passes to decide which samples it ac- quires, and En Crisebbles N = 5 forward passes, one for each model, our approach requires only 2 inferences for only similar as the two more complex methods, even though it uses only the simple soft-max probabili- ties. While ropy beats random it is still not competitive. BADGE performs similar random sampling, which is due to the small acquisition size of 10 (the better performance of BADGE was reported with much larger acquisition sizes of 100 10,000 Ash al. (2020) as its mix of uncer- lainty and diversity heuristic benefits from these). The same applies to CoreSet, however, here it performs worst on average over all experiments. This finding is in line with previous research (Sinha et al. 2019 Hahn et al. 2019a) and can be attributed to a weakness of the utilized p-norm distance metric regarding digeeninnsi data, called the distance concentration phenomenon. The accu- racy of A on MNT ST is similar to CoreSet, however, ALII is designed to add only one sample to the training data at time (no batch-mode).
100 samples plus the labeled pool. Con Confidences setenccessmmling performs
Construction of acquisition. We compare IALE's chosen samples with the ones chosen by the baselines, see Figure We show this overlap in relation to the baselines in percent, and plot the val- ues over the 100 AL cycles using the MNIST dataset (more results can be found in Appendix A.2.2 We plot second-orded polynomials, fit to the percentages (given as dots) over 100 acquisitions of size 10. mostly imitates uncertainty-base heuristics, i..., soft-max heuristics and MC-Dropout, and BADGE (close behind). Interestingly, Ensembl is overlapping
the -iiienttttiiieriit--ii mostly at the beginning. CoreSet has the lowest overlap. Note that TALE'S acquisitions are build
7

---

Under review as a conference paper at ICLR 2021
Acje1 1015
Acprts 10
Asjet un183
Randan
629
Cas
002
021
035
KLE 000 co
301
20
36.5
Libering othent
A:0
300
043
659
20
Litl
2013
(a) Varying acquistion sizes.
(b) Varying [Daub].
(c) Experts
Figure 5: Ablation studies: IALE performance for different (a) acg and (b) n=| subl (on MNIST);
(c) IALE performance for different expert sets (on KMNI KM).
from combinations of the heuristics (instead of single votes). The percentages do not sum up to 1 as the experts are independent and may also overlap with each other.
4.3 POLICY TRANSFER
We investigate how works on a different dataset than the one that it has been trained on. Hence, we train π on the source dataset MNIST as in Section 4 and use for the AL problem on FMNIST and KMNIST. For the AL we again training dataset of 20 samples and add 10 samples per AL acquisition cycle until we reach a labeling budget of 1,000 samples. All the baselines are evaluated along with our method for comparison.
1.0
MC Dropout Badge
Essemble
Enerapy
0.8
Oxe nte
0.6
initial class-balanced labeled
0.2
0.0
20
40
60
80
100
Figures 3b and 3 show the performance of IALE along with the baselines on FMNIST and KMNIST, IALE con- sistently outperforms the baselines on both datasets. We can see that it learns combined and improved policy that outperforms the individual experts consistently and (sometimes) even with large margins. On FMNIST IALE the only method that actually beats Random Samp 1 ing (similar findings have previously been reported by Hahn et al. (019b)). IALE is consistently 1 3% better than Random Sampling on FMNIST, on the harder KMNIST dataset IALE is even 7 - 9% ahead. The baselines give mixed picture. ALIL does not achieve competitive performance never beats a random sampling strategy. We also see unstable performance for MC-Dropout and Ensembble, e, that generally perform similarly well. The simple soft-max heuristics Entropy and Confidence fail on FMNIST. CoreSet lags far behind, especially on KMNIST. BADGE always performs like random sampling, due to the aforementioned problematic acquisition size. Please find additional experiments in Appendix A.3 includincllgg wider range of classifier models (MLP, CNN, and Resnet- 18) and datasets (CIFAR-10).
Acqusition numbor
Figure 4: Overlap ratio.
any task and actually
4.4 ABLATION STUDIES
Hyperparameters. Two important parameters are the acquisition size aeq and the size of Drub- Figures 55 results for acq of 1 to 40 and size of D.ub between n=10 and n=10,, 000. As expected, TALE performs best at acq=l and worst at acq=40, n unchanged, because n limits the available choices, e.g., bad samples have to be chosen. Increasing to 1, 000 alleviates this issue. However, there is an upper limit to the size of Drub after which performance deteriorates again, see Figure 5b) This could be because random sub-sampling actually simplifies the selection of diverse, uncertain samples. The lower limit becomes apparent again when n is smaller than 10 times acg, with n acg essentially being random sampling. From our observations, should be 10- 100 times acg. From the small differences within this value range, it is suggested that our method is suitable for larger acquisition sizes for batch-mode AI, as its performance is not affected much.
8

---

Under review as a conference paper at ICLR 2021
Varying experts. To investigate the influence of experts, we leave out some types of experts: We categorize them into 4 groups. i.c. uncertainty (ModropEns), soft-max uncertainty (Ent diversity (Coreset) and hybrid (Badge), and leave subset out. We fully train each method on MNTST with B = 1. 1.000 and an acquisition size 10, and present the results of the evaluation on KMNIST in Figure 5c (more results including the ablation of state elements can be found in Ap- pendixes A.444]aam and and A.4.3).. We see that most combinations perform well compared to the baselines. However, leaving out uncertainty or soft-max uncertainty experts can decrease performance. Even though training time is longer with MC-Dropout, the gain in performance can be worth it. In con- trast, the soft-max uncertainty based heuristics are computationally cheap and yield good policies.
5 CONCLUSION
We proposed a novel imitation learning approach for active learning- Our method learns imitate the behavior of different active learning heuristics, such as uncertainty- diversity-, model change- and query-by-commiitee-
heuristics, on one initial dataset and model, and transfers the ob- tained knowledge to work on other (types of) datasets and models (that share an embedding space). Our policy network is simple MLP that learns to imitate the experts based on embeddings of the dataset samples. Our experiments the art consistently (despite being a batch-mode AL approach). An ablation study and analysis of the influence of certain hyper-parameters also shows the limitations of our approach. Future work investigates the relationship between acquisition sizes and sub-pools, and an analysis on how the state (embedding) enables to transfer its active learning strategy, potentially leading to the use of artificial data for finding optimal AL strategies suitable for specific active learning scenarios.
well-known datasets show that we outperform the state of
REFERENCES
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations (ICI.R). Virtual Conference, Formerly Addis Ababa Ethiopia, 2020.
Philip Bachman, Alessandro Sordoni, and Adam Trischler. Learning algorithms for active learning- In 34th International Conference on Machine Learning (ICML), pp. 301-310, Sydney, Australia, 2017.
Yoram Baram, Ran El- Yaniv, and Kobi Luz. Journal of Machine Learning Research, (5):255-291, 2004.
Loic Barrault, Ondłej Bojar, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Yvette Gra- ham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 conference ma- chine translation. In 4th Conference on Machine Translation, pp. 1-61, Florence, Italy, 2019.
William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9368 9977,99777758 Salt Lake City, UT, 2018.
Hong-Min Chu and Hsuan- Tien Lin. Can active learning experience be transferred? In 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 841-846. IEEE, 2016.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature, 2018
Gabriella Contardo, Ludovic Denoyer, and Thierry Artières. A Meta-Leaming Approach to One- Step Active-Learnin In International Workshop on Automatic Selection, Configuration and Composition of Machine Learning Algorithms, pp. 28-40, Skopje, Macedonia, 2017.
Yang Fan, Fei Tian, Tao Qin, Xiang- Yang Li, and Tie- Yan Liu. Learning to teach. arXiv preprint arXiv: 1805. 0366433 2018.

---

Under review as a conference paper at ICLR 2021
Meng Fang, Yuan Li, and Trevor Cohn. Learning how to active learn: A deep reinforcement learning approach. In Conference on Empirical Methods Natural Language Processing, pp. 595-605,
Copenhagen, Denmark, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout a Bayesian approximation: Representing model uncertainty in deep learning. In 33rd International Conference on Machine Learning (ICML),
New York, NY, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In 34th International Conference on Machine Learning (ICML), Sydney, Australia, 2017.
Lukas Hahn, Lutz Roese-Koentd Koerner, Peet Cremer, Urs Zimmermann, Ori Maoz, and Anton Kum- mert. On the robustness of active learning. In 5th Global Conference on Artificial Intelligence, volume 65 of EPiC Series in Computing, pp. 152-162. Bolzano, Italy, 2019a.
Lukas Hahn, Lutz mert. On the robustness of active learning. In 5th Global Conference on Artificial Intelligence, volume 65 EPiC Series in Computing, pp. 152-162, 2019b.
Peet Cremer, Urs Zimmermann, Ori Maoz, and Anton Kum-
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
HM Sajaad Hossain, MD Abdullah Al Haiz Khan, and Nirmalya Roy. Deactive: Scaling activity recognition with active deep learning. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2:66:1-66:22 2018.
Wei-Ning Hsu and Hsuan-Tien Lin. Active learning by learning. In 29th AAAI Conference Artificial Intelligence (AAAI), pp. 2659-2665, Austin, Texas, 2015.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems (NeurIPS), pp. 5574-5584, Long Beach, CA, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations (ICLR), 2015. URL http::t/tttiiiiie org/ abs/1412.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acqui- sition for deep bayesian active learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 1026-7037. Vancouver, Canada, 2019.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4225-4235, Long Beach, CA, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11))2278-235 22224,, 1998.
Minghan Li, Xialei Liu, Joost van de Weijer, and Bogdan Raducanu. Learning to rank for active learning: A listwise approach, 2020.
Xin Li and Yuhong Guo. Adaptive active learning for image classification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Portland, OR, 2013.
Ming Liu, Wray Buntine, and Gholamreza Haffari. Learning how to actively learn: A deep imitation learning approach. In 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1874-1883, 2018.
Dwarikanath Mahapatra, Behzad Bozorgtabar, dean-Philippe Thiran, and Mauricio Reyes. Efficient active learning for image classification and segmentation using sample selection and conditional generative adversarial network. International Conference on Medical Image Computing and Computer-Asiisted Intervention, pp. 580-588, Granada, Spain, 2018.
10

---

Under review as a conference paper at ICLR 2021
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1884, 2014.
Andriy Mnih and Danilo J. Rezende. Variational inference for monte carlo objectives. In 33rd International Conference Machine Learning (ICML), pp. 2188-2196, New York, NY, 2016.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan, Do deep generative models know what they don't it know? In International Conference on Learning Representations (ICLR), New Orleans, LA, 2019.
Sachin Ravi and Hugo Larochelle. Meta-learning for batch mode active learning. In 6th Intl. Conf. Learning Representations, Workshop Track Proc., Vancouver, Canaday, 2018.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. In 14th International Conference on Artificial Intel- ligence and Statistics (AISTATS), pp. 627-635, Ft. Lauderdale, FL, 2011.
Dan Roth and Kevin Small. Margin-based active learning for structured output spaces. Johannes Fürnkranz, Tobias Scheffer, and Myra Spiliopoulou (eds.), Machine Learning: ECML 2006, pp- 413-424, Berlin, Heidelberg. 2006. Springer Berlin Heidelberg.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference Learning Representations (ICLR), Vancouver, Canada, 2018.
Burr Settles. Active learning literature survey. Technical report, University of Wisconsin-Madison Department of Computer Sciences, 2009.
Burr Settles, Mark Craven, and Soumya Ray. Multiple-instance active learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 1289-1296, Vancouver, Canada, 2008.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In International Conference on Computer Vision (ICCV), pt.CCV)....992-35.5...2 5972-5981, Seoul, South Korea, 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances Neural Information Processing Systems (NeurIPS), pp. 4077-4087, Long Beach, 2017.
LMA Tonnaer. Active learning in vae latent space. Eindhwven University of Technology, 2017.
Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis. Deep learning for computer vision: A brief review. Comp. Intelligence and Neuroscience, 2018.
D. Wang and Y. Shang. A new active labeling method for deep learning. In 2014 International Joint Conference on Neural Networks (ICCN), pp. 112-119, 2014.
Mark Woodward and Chelsea Finn. Active one-shot learning. In Advances in Neural Information Processing Systems Workshops, Barcelona, Spain, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv:1708.077477 2017.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
In this section we provide an extension of the experiments section (Section I and feature additional results that support complete evaluation of IALE. We adhere the same section structure.
A.1 EXPERIMENTAL SETUP
A.1.1 BASELINES
In the following is short explanation of the baselines and experts that we used in our experiments:
1. Random Sampling randomly samples data points from the unlabeled pool.
2. MC- MC-Dropout (Gal et al. 2017) approximates the sample uncertainty of the model by repeatedly computing inferences of the sample, i.e., 20 times, with dropout enabled in the classification model
3. Ensemble (Beluch al. 2018) trains an ensemble of classifiers with different weight initializations. The uncertainty of the samples is quantified by the disagreement between the model predictions.
4. CoreSet (Sener & Savarese 2018) solves the k-center problem using the pool- embeddings of the last dense layer (128 neurons) before the softmax output to pick samples for labeling.
5. BADGE et.. 2020) uses the gradient of the loss (given pseudo labels), both its magni- tude and direction, for k-means++ clustering, to select uncertain and diverse samples from batch.
6. Sonfidencc--sapling (Wang & Shang 2014) selects samples with the lowest class probability of the soft-max predictions.
7. Ent rattroassamping (Wang Shang] 2014) calculates the soft-max class probabilities' entropy and then selects samples with the largest entropy, i.e., where the model is least certain.
8. ALIL (Liu et al, 2018): modify ALIL's implementation (that initially intended for NLP tasks) to work on image classification task. Due to the high runtime costs of running ALIL (as the acquisition size is 1), perform the training of ALIL for 20 episodes. We trained the ALIL policy network with labeling budget of 1, 000 and an up-scaled policy network comparable to that of our method along with similar M as
evaluate the other AL approaches. We left the coin-toss parameter at 0.5, and the k parameter for sequential selections from random subset of Droot at 10.
We use the variation ratio metric (Gal et al. 2017) to quantify and select the data samples for labeling from the uncertainty obtained from MC Dropoout and Ensembl heuristics. The variation ratio metric is given by its Bayesian definition (Gal et al. 2017) for data sample x E Ppool in Equation 3 and for an ensemble expert (Beluch et al. 2018 in Equation 4
variation-ratio(s)
maz,,lyla, D)
(3)
m N
where m is number of occurrences of the mode and N the number forward passes or number of models in the ensemble.
A.2 POLICY TRAINING
A.2.1 EXPLORATION
CONERATTIO-EXPLOTAATTTOO EXPLOITATION IN DAGGER
DAGGER uses hyper-parameter that determines how likely N predicts the next action, and thereby setting the next state, instead of using the best expert from E. this preliminary study we compare the influence it has either fix p 0.5 or to use exponential decay parameterized by the number 0.9epi We train the policy on MNIST for 100 episodes with a labeling budget of 1, 000 and acquisition size of 10 (as before). Our result is that the fixed policy
of the current episode epi:
12

---

A balanced (i.e., fixed) ratio does not emphasize over the other, whereas exponentially decay quickly relies on the policy for selecting new states of the dataset, and thus it trains on too few Under review as a conference paper at ICLR 2021
outperforms the exponential one by small margin for the transfer of the policy to another dataset than the trained one, which is in line with previous findings (Liu et al. 2018).
optimal states over the AL cycle.
A.2.2 OVERLAP RATIOS
We additionally show FMNIST and KMNIST overlap ratios in Fig. The policy chooses about half of the samples differently from any single baseline. The overlap is lower on FMNIST, where our method is the only one that beats random sampling. CoreSet overlaps least often and performs worst.
Conficance
Oripat
Ensenitaa
Conficence
Orgpat
Erainidi
Drapoat
Graendle
20
100
100
100
(a) MNIST
(b) Mashion-MIIN
(c) Suzushiij-MNELE
Figure 6: The overlap plots for all datasets MNIST. ST, FMNIST and KMNIST datasets.
A.3 POLICY TRANSFER
In this section we provide further studies on how our method performs in regard to applying it to unseen scenarios. These include that we (1) train the policy on other source datasets than MNIST, that we (2) apply the policy onto other classifiers than the one it was trained on, and that we (3) use different datasets and classifier models in training and application of the policy.
We aim to show that ı learns a (relatively) task-agnostic AL strategy, that works inference and outperforms the baselines, as long the state can be expressed similarly to the one that has been used during training.
A.3.1 VARYING SOURCE DATASETS
Our policy does not coincidentally perform well due to a specific source dataset that is most suit- able. In Figure unchanged
show two additional policies that were trained FMNTST KMNTST (with settings). We see comparable performance that indicates that TALE ac- tually learns to actively learn and not just remembers the source datasets makes not difference which datasets are chosen as the source and the target dataset.
13/6 Foiiy Nancons
04.5
EMA TN-ETTR Taics
0//
Nove
0..00
0.00
1.7
0377
1mm
0.50
11x
0722
C12
28
403
300
100
0290
200
Lowinnsterr
FX
903
80
300
100
(a) MNIST
(b) F Fashion--MME
(c) KuzushijjiMNN
Figure 7: We show the performance of three policies trained different datasets in comparison with Random (on MNIST, FMNIST and KMNIST).
13

---

Under review as a conference paper at ICLR 2021
A.3.2 CLASSIFIER ARCHITECTURES.
Our method is not bound to a specific classifier architecture. To show this, we evaluate our approach with MLP and Re classifiers. We use the same hyper-parameteess as with the CNN and only switch out the underlying classifier. All results for the experiments are given in Figure g We see the robustness of π over fundamentally different classifier architectures to 18 layers). The deviations for ResNet-18 18 are very large due to the very deep architecture and the modest amount of training
data. We use median filtering in Figures 8g
8h
and 8i
These experiments show that π can learn AL strategies for both very small and very deep architec- tures and still outperform baselines. Even though the strongest baselines, i.e., CoreSet and MC Dropout, come close to our method in accuracy. they are less versatile and require more computa- tional resources, that is especially noticeable on deeper architectures.
1.00
0.1
0.99
0.52
0.1
025
the
6.79
0.84
0.99
0.75g
1000
02%
1000
609
(a) MLP on MNIST
(b) MLP on FMNIST
(c) MLP on KMNIST
INLE
MLLE 5400
Confoonca Condill
Condall
0.71
Ranbon
Droogy
Mc Orgaa
0.7
9/gg
0.2
6.
(d) ResNet-18 18 on MNIST
(c) ResNet-18 18 on FMNIST
(f) ResNet- on KMNIST
Confiderze
10.5
Eepumbbe Chrpuea
Raerro
Drinary
0%
0.1
0.2
Labanng
Ubbirg vrtnn
(g) R8 on MNT ST
(h) 18 on FMNIST
(i) ResNet-18 18 on KMNIST
Figure& 8: MLP and ResNet-18 18 classifiers, data averaged or median filtered. Active learning perfor- mance of the trained policy in comparison with the baseline approaches on MNIST, FMNIST and KMNIST datasets.
A.3.3 GENERALIZATION
To show that x learns active learning independent from task and classifier, we conduct experiments where we mix both the source datasets and the classifiers, as only the state retains the same formula- tion. Since these states are the same for CNN and Resnet-18 classifiers (independant of the datasets) can transfer policy trained using Resnet-18 classifier to a CNN classifier and vice versa.
We report the results for applying π (trained on Resnet- 18 and MNIST) to a CNN and all the datasets in Figure TALE is always performing the top. These results convincingly show that our method
14

---

Under review as a conference paper at ICLR 2021
learns model- and task task-agnostic active learning strategy that transfers knowledge between datasets and even between classifier architectures.
1.6
042
0.90
6.5
1995
0.9
025
R1Z
BADGE
Oueda
56202
Gredal
B6002
AGG /62/6
Tardrn
fander
Tanders
100 DA2
1000
540g
(a) MN S
(b) F'ashi on ON- ST
(c) Kuzushi 11- I ST
Figure 9: Applying policy trained using Resnet-18 classifier (trained on MNTST) to a CNN-based classifier (on MNTST, FMNIST and KMNIST).
We conclude the transfer studies by applying pre-trained π to a Resnet-18 classifier on the CIFAR-10 (Krizhevsky) (009) (color) image classification dataset. Each image in CIFAR-10 is 32 x 32 pixels and has 3 color channels. The transfer of π is possible as the classifier's state is both invariant to the models and independent from the number of color channels of the image dataset and the input size. For our experiments we use two different क: л1 was trained using Resnet- 18 and MNIST (IALE Resnet)) and π2 was trained using CNN and MNIS ST (IALE CNN). We train the classifiers with an acquisition size of 10 until the labeling budget of 10.000 is reached.
The results are presented in Figure TO As the acquisition size of 10 results noisy curves we report the raw learning curves (Figure Tnd median filtered learning curves (Figure TOb). We report the interesting segment of the learning curve in more detail (filtered) Figures T0c] The results generally show the feasibility of transferring to both different classifiers and datasets. IALE is en par or better than Random, and the other baselines are either en par or worse than Random (some of them considerably).
Moreover, besides IALE's better accuracy compared to all other methods, its run time is highly competitive. Per experiment iteration (with 10.000 samples, acquired over 998 steps of size 10, each step including a complete re-training of Resnet-18 for 100 epochs), run times for IALE are 10:17:31 (h:m:s) a high performance GPU (NVidia V100) (9:45:12 for Random sampling), compared to 49:15:23 for Ensembles 14:23:18 for MC Dropout and 11:58:47 for BADGE. Only Conf (10: (1:01):01) and Ent ropy (10:05:21) are quicker (however, they both also perform worse than Random).
1.70
2.275
344
Randen
DLE Naan
Gaage
Gordeer
Coeder
God
Oripout
VC Orooe
400C
100 03
Lobetie
10000
Lonern
6200
2001
(a) full (raw)
(b) full (filtered)
(c) detail (filtered)
Figure 10: Full and enlarged segments of learning curve: Applying и trained on CNN (IALE CNN) or R8 (IALE Resnet), trained on MNIST, to a Resnet-18 classifier on CIFAR10.
While more experiments are certainly required to further emphasize these initial claims of general- izability to more diverse types of datasets, these findings are already very promising.
15

---

Under review as a conference paper at ICLR 2021
A.4 ABLATION STUDIES
A.4.1 HYPERPARAMETERS.
We report fine-granular steps of acquisition sizes in Figure Iaa with values between 1 and 10, plus 20 and 40, for [Daubl of 100. Overall, a clear difference not visible below 10 samples. For enhanced readability, we show a magnified section of the varied acquisition sizes and Daubl in Fig. 11b that clearly shows the benefits of tuning suub to suitable value for the acquisition size.
11.80
0.360
0.666
100 903880 1006 140re
Asg ett
0.840
Act sor
Acprt e.100
Appu5 --12)
0.800
Asge10, m0
Anpu21 1-322
Labelig offer
Loko ing offert
Varying acq sizes FMNIST.
Varying acy sizes MNIST
Figure 11: Evaluating the acquisition sizes from 1 to 10 on FMNIST, and varying different pool sizes on MNIST.
Acquisition sizes for baselines: We additionally compare the baseline active learning methods with our approach, as these exhibit different performance different acquisition sizes, see Fig. 12 We have included comparisons with acquisition sizes of either or 100 (1 or 3 repetitions). For method, for an acquisition size of we chose Deubl = 100 and for acquisition size of 100 we chose
[Duuhl = 2, 000. While the results show that IALE outperforms the baselines they also highlight the large effect that the acquisition size has on some of the baseline methods. For instance, CoreSet constructs better covers with larger batches, and RADGE increases its accuracy by constructing a representative sampling as well. At the same time the uncertainty-base methods, apart from Ent ropy, remain unaffected.
WNBT
C3
54002
DAOCE
CouSel
UCOrssa
Levding afrtt
Labeling afoor
(a) Acquisition size 1.
(b) Acquisitioon size 100.
Figure 12: Some experts are more suitable to other acquisition sizes (evaluated on MNIST).
A.4.2 VARYING EXPERTS
We present more results for variations of sets of experts in Figure 13 experts, and train the policies with the unchanged hyper-parametery and the CNN classifier, The results for all three datasets show that the generally high performance of IALE holds for the leave- one-out sets of experts, with the full set experts being consistently among the best performing policies.
We leave out some types of
16

---

Under review as a conference paper at ICLR 2021
0.30
Erpccar ALE Buttee
0470
5.850
0400
0726
AL6 to incriooe cooe
ladge
035
020
Librie
300
0730
200
659
200
1.00
(a) full state, MNIST.
(b) full state, FMNIST.
(c) full state, KMNIST.
Figure 13: The active learning performance for each (leave-one-out) set of experts.
A.4.3 VARYING STATE ELEMENTS
Next, we study the state more closely. For unlabeled samples, the state contains two types of rep- resentations for predictive uncertainty: the statistics predicted labels M (xx) and the gradient representations (M,)) In this study, we focus on leaving out one or the other. To get the full picture, again train sets of experts for reduced states.
In Figure 14 we see that dropping gradients generally decreases performance (bottom row), while dropping predicted labels (ar.) affects performance very little (top row). However, the influence of different sets of experts is more important. We cannot see that a particular set of states and experts generally outperforms others consistently (while the negative effect of leaving out ( is consistently visible). Overall, we find that using as many experts as available, combined with full state both performs well and works reliably. Even though training policy this way does guarantee the best performance, it always performs among with the group of best policies.
199
KLL e Covnel
2335
032
MAE
Coeer
ALE ALEraDreca
ULS Gemee
Male diie
Tandon
1700
Labeling oftor
(a) No predictions, MNIST
(b) No predictions, FMNIST
(c) No predictions, KMNIST
0.30
0.75
0.75
I Gonen
6.9
Raroo's
0.5
DLB Boreee
300
200
200
(d) No gradients, MNIST
(c) No gradients, FMNIST
(f) No gradients, MMNIST
Figure 14: For partial state (without predictions, without gradients), we plot active learning perfor- mance for each (leave-onc-out) set of experts.
17

---

Under review as a conference paper at ICLR 2021
A.4.4 ADDITIONAL DATASETS
We additionally run experiment harder datasets, i.e., SVHN and CIFAR 100 in Fig.
SVHN:
We train a 8 on SVHN with an acquisition size of 1,000 and a labeling budget of 16,000. We initially label 1, 1,000 samples and evaluate over repetitions. We show both the average results with variance in Fig. [5a and the smoothed averages for improved visibility in Fig, [5b] While the results exhibit some variance, we can clearly see that IALE performs best (and is the only AL methods that is consistently able to beat random sampling.
Next, we evaluate datasets with larger number of classes by generalizing the com- position of the state (removing prediction and empirical class distribution). We train two policies, one with a CNN classifier on MNIST and the second with a Resnet18 classifier on CIFAR- 100 Then, we apply the resulting policies in the following experiment, We train Resnet18 on the very hard CIFAR-100 100 dataset, with an acquisition size of 1,000 and a labeling budget 20000, and initially label 1,000 samples. We evaluate over repetitions. Fewer samples or less powerful net- work architectures tend to fail to converge in our experiments. The results Fig. show that both policies, i.e., transferred from CNN/MNIST and from Restnet 8/CIFARI 10 with arbitrary number of classes, perform best among all heuristics, and beat random sampling.
CIFAR-100:
8HHN
SVIN
2.21
CFAR 100
1.0
3.5
5.55
0.00
3.50
1.7
3.20
0.0
Ag
Condec
9.70
Cordee
(422 Meemit FATUE)
Onioy
C8
2500
4521
COCC
1000
12000
SEX
1.0
200
4000
600
1000
1200
1400
1000
230
2000
7500
1000
1930
1750
2000
(a) SVEN.
(b) SVHN.
CIFAR-100.
Figure 15: Increasingly difficult datasets: SVHN (complex structure, requires samples, average and smoothed plots) and CIFAR 1000 (100 classes).
18


Reviews:
[{"review_id": "XpBqiK38_2U", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Paper summary:\n\nIn this work, an imitation learning (AL) approach is proposed to imitate multiple active learning algorithms, in order to take their advantages to learn a better active learning algorithm. The main idea is to treat the active learning algorithms as experts and utilize the DAGGER algorithm for imitation learning. The proposed approach is evaluated on MNIST, fashion-MNIST, and Kuzushiji-MNIST, showing that the learned active learner outperforms baseline active learners, meanwhile is transferrable to other datasets.\n\n#### Advantages:\n\n- The idea of using imitation learning is interesting.\n\n- The survey of existing active learning approaches is quite thorough.\n\n#### Disadvantages:\n\n- The idea of using DAGGER to combine multiple active learners was explored in Liu et. al. https://www.aclweb.org/anthology/P18-1174. The technical novelty is thus limited. \n\n- The experiments are not sufficient for the following perspectives:\n\n  - The performance gain is somehow not very significant, the datasets are restricted to images with the same number of classes. \n\n  - The policy utilizes the features of training instances as part of the states. While I am in doubt of why such features can be transferrable to new datasets? \n\n  - There are no comparisons to the work of Liu et. al., which also utilizes DAGGER to imitate active learners.\n\nOverall evaluation:\n\nI think the paper proposes a reasonable solution, while the contributions and novelty are somehow below the standard of ICLR.\n\nAdditional:\n\n- The methodology for the proposed approach is similar to the \"learning to teach\" approach https://openreview.net/pdf?id=HJewuJWCZ, which learns a teaching policy for selecting the smallest set of labeled instances to accelerate learning. Adding some discussions to their work can be useful.\n\n- The \"test set\" in algorithm 1 should be \"validation set\".\n\n#### Update after rebuttal:\nThanks for the detailed feedback. While I still find the technical novelty is limited comparing to [Liu et. al.]. For this reason, my score remains unchanged.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OZEsYXHqrRO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach to learning active learning policies that rely on imitating the behaviour of known heuristics. The resulting learnt policies can successfully transfer between the related datasets. \n\nStrong points:\n- This paper focuses on the important problem and attempts to provide a simple solution that leverages the prior knowledge about the problem (existing heuristic solutions) while being data-driven.\n- The experimental studies are quite informative and detailed and provide analysis of the behaviour of the proposed solution.\n- The proposed method outperformed the presented baselines in most cases.\n- The experiments with policy transfer between different datasets seem very interesting and focus on the realistic and important setting.\n- The experiments with ablations by excluding various experts are informative and insightful.\n\nWeak points:\n- The novelty of the proposed method is not very strong: The main idea is to select the best behaviour among several heuristics, that was explored before (Hsu&Lin, 2015, 1, 2, 3), the learning mechanism relies on imitation learning, that was also explored in previous works for AL (Liu et al, 2018), and the implementation in terms of state parametrization seem to be similar to previous works too (Contardo et al, 2017, Konyushkova et al., 2017, Liu et al, 2018).\n- This paper puts an emphasis on the batch nature of the proposed method (for example, when contrasting against the related work). However, I did not understand how the policy training is particularly targeted to the batch setting.\n- The nature of the proposed method is the most similar to the algorithms that select a particular heuristic to use at each step of the training. While the authors mention some of such algorithms (Hsu&Lin, 2015), I believe a few others works deserve some attention [1, 2, 3]. Importantly, there is no comparison to the methods from this group.\n- The authors claim that the imitation learning is done by the DAGGER method. However, as far as I understand, DAGGER makes an assumption that an expert can provide an optimal action from any state as supervision. If I understand the method correctly, such an assumption cannot be satisfied in the setting of the paper. To my mind, it sounds that the method is closer to supervised learning with epsilon-greedy data collection.\n- Most experiments are performed on rather low dimensional data in MNIST, FMINST and KMNIST. The transfer to more complex datasets such as CIFAR is shown in the appendix, but the gains by the proposed method seem to be marginal.\n\n\nI am leaning towards the rejection of this paper in the light of limited novelty of the presented method and limited gains especially on more complex realistic datasets.\n\nQuestions:\n\n- Section 4.4 contains a curious and counter-intuitive observation: \"there is an upper limit to the size of D_{such} after which performance degrades again\". However, the authors did not provide any explanation of this phenomena. Why is this happening?\n- Could the authors elaborate a bit more on the connection between the proposed learning mechanism and DAGGER?\n- I think the most significant part of the method is the ways how the training data is collected with supervision of the experts. While this procedure is covered in section 3.3, I think it deserves a bit more attention. For example, I am still not sure if and how *batches* of data are selected by heuristics strategies.\n- Why do the authors say that the method is closer to the baselines \"towards the end obviously\"? Is the most of the data labelled at that point?\n- The learning curves in many experiments seem to be quite noisy. How many experiments are averaged?\n- How do the baselines compare to the proposed method with varying batch size (as some methods are particularly targeted to batch size=1 or large batch size>100)?\n\nAdditional comments:\n\n- The language in the paper often uses unusual word ordering and constructions that makes it difficult to read. For example, \"as ideally we label\", \"besides the above mentioned\", \"do still not work on batches\".\n- A lot of experimental results are presented by training and testing the policy on the same dataset. As this is a rather unrealistic setting and serves mostly as sanity check, less space in the paper could be dedicated to it.\n\n\n[1] Online choice of active learning algorithms. Y. Baram, R. El-Yaniv, and K. Luz. JMLR, 2004.\n[2] Can active learning experience be transferred? H.-M. Chu and H.-T. Lin. ICDM, 2016.\n[3] Active learning by learning. W.-N. Hsu and H.-T. Lin. In American Association for Artificial Intelligence Conference, 2015\n\n===Post rebuttal===\n\nI have read the authors response and would like to thank the authors for the clarifications. I am still inclined to keep my original score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NWi9-MRiKbp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a batch-mode active learning approach via imitation learning to derive a sample selection strategy for active learning. The proposed method makes use of a pool of existing heuristics, learning a sample selection strategy that rely on different heuristics in the pool depending on the different stage of the active learning process. The use of imitation learning makes sense, as it helps to alleviate the high sample complexity problem of deploying a reinforcement learning strategy. Furthermore, a batch-mode active learning strategy is desirable compared to the previous work (Liu et al. 2018) that uses a single sample. Experimental results on MNIST,  FMNIST, and KMNIST demonstrates the utility of the proposed method. I have the following concerns:\n\n1. While I think the idea of framing the active learning problem as an imitation learning problem has merits, it is unclear to me how the proposed framework is different from that of (Liu et al. 2018). The authors point out that (Liu et al. 2018) achieves single-sample mode active learning but the proposed framework achieves batch-mode active learning. However, the way that batch-mode is achieved in this paper seems trivial by computing and using the top-k instances according to certain metrics.\n\n2. It is also unclear how the pool of experts used in the paper generalizes beyond the setting in (Liu et al. 2018), and why this might be desirable. Given this concern and the previous one, it would be great if the authors can clarify the relationships between the two works so as to better highlight the specific contributions made in this work.\n\n3. In terms of empirical validation, is it possible to also compare the proposed method with (Liu et al. 2018)? Furthermore, the authors demonstrate the utility of the proposed method in three related datasets (MNIST,  FMNIST, and KMNIST). I am not sure if such empirical validation is sufficient. How about other datasets that contain digits such as SVHN? Is it possible to imitate on CIFAR 10 and transfer on CIFAR 100? How about NLP tasks that are reported in (Liu et al. 2018)? On the other hand, I think the ablation studies provided by the authors do help to demonstrate how different components of the framework will play and influence the eventual outcome.\n\nMiscellaneous:\n1. There is a missing comma in the last sentence of the paragraph right before Section 3.1?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In the discussion, all reviewers acknowledge the novelty of this paper, such as learning from a wide range of AL heuristics, and the ability to transfer the to tasks with arbitrary number of classes. They also think that the additional experiments provided by the authors improve the paper's empirical validity. 

However, a major issue raised by the reviewers is that the novelty (especially when compared with Liu et al) may not be enough for ICLR this time. One research direction (implicitly suggested by Reviewer 2) was to learn active learning strategies that go beyond selecting top-k scoring examples to explicitly account for batch diversity. We encourage the authors to address this in the next version.
Decision: Reject