Learning Active Learning in the Batch-Mode Setup with Ensembles of Active Learning Agents
Authors: Malte Ebner
Bernhard Kratzwald
Stefan Feuerriegel
Venue:  (2021)
Keywords: active learning
ensembles

Abstract: Supervised learning models perform best when trained on a lot of data, but annotating training data is very costly in some domains. Active learning aims to chose only the most informative subset of unlabelled samples for annotation, thus saving annotation cost. Several heuristics for choosing this subset have been developed, which use fix policies for this choice. They are easily understandable and applied. However, there is no heuristic performing optimal in all settings. This lead to the development of agents learning the best selection policy from data. They formulate active learning as a Markov decision process and applying reinforcement learning (RL) methods to it. Their advantage is that they are able to use many features and to adapt to the specific task.

Our paper proposes a new approach combining these advantages of learning active learning and heuristics: We propose to learn active learning using a parametrised ensemble of agents, where the parameters are learned using Monte Carlo policy search. As this approach can incorporate any active learning agent into its ensemble, it allows to increase the performance of every active learning agent by learning how to combine it with others.

Under review as a conference paper at ICLR 2021
LEARNING ACTIVE LEARNING IN THE BATCH-MODE SETUP WITH ENSEMBLES OF ACTIVE LEARNING AGENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Supervised learning models perform best when trained on of data, but anno- tating training data is very costly in some domains. Active learning aims to choose only the most informative subset of unlabelled samples for annotation, thus saving annotation cost. Several heuristics for choosing this subset have been developed, which use fixed policies for their choice. They are easily understandable and ap- plied. However, there is no heuristic performing optimal in all settings. This led to the development of agents learning the best selection policy from data. They formulate active learning as a Markov decision process and apply reinforcement learning (RL) methods to it. Their advantage is that they are able to use many features and adapt the specific task.
approach combining these advantages of learning active learning and heuristics: We propose to learn active learning using a parameterized ensemble of agents, where the parameters are learned using Monte Carlo policy search. As this approach can incorporate any active learning agent into its en- semble, it allows to increase the performance of every active learning agent by learning how to combine it with others.
Our paper proposes
INTRODUCTION
Supervised machine learning systems perform best when trained large amount of training data. Obtaining this data by labelling can cause huge time and cost efforts some domains. Active learning in the selective scenario overcomes this bottleneck by selecting subset of all unlabelled samples to be labelled such that the model trained on them learns as much as possible and achieves high accuracy (Cohn al. 1994).
Heuristic active learning agents choose the samples to be labelled using a fixed policy. They have known and predictable behaviour and their policy is easy understand (Settles al. 2008). However, they have two main disadvantages: First, they rarely combine different features. Second, it found that the best heuristic highly depends on the dataset and supervised learning model used (Lowell et al. 2018).
More recently, these shortcomings have been addressed by learning active learning directly from data (Konyushkova et al. 2017 2018 Bachman et al. 2017] Fang et al. 2017 Liu et al. 2018b a The authors formulate active learning as a Markov decision process and apply reinforcement learn- ing (RI.) methods like Q-Learning and imitation learning to it. While this approach promises to overcome the advantages of heuristics it introduces new problems: There the credit assignment problem (Minsky) 1961). the training is computationally very costly (Amodei et al.), and many results are not significant and hard to reproduce (Henderson et al. 2017).
Learning active learning with RL in the batch-mode setting has only received little attention in literature. One of the reasons is that choosing a batch of samples instead of a single one makes the action space exponentially bigger and thus finding the action maximizing value function not be done by iterating over all actions anymore. Furthermore, it makes it harder to attribute the reward specific parameter of the policy.

---

Under review as a conference paper at ICLR 2021
Our paper addresses these shortcomings by proposing active learning agent being weighted ensemble of other agents. The weights are learned using Monte Carlo policy search and black- box optimization. As several different agents can be used part of the ensemble. many different features can be included. By learning the weights of each agent, the ensemble can adapt to the dataset, model and optimization metric. Nonetheless, the learning is very robust and global, black- box optimization over a small number of parameters is much casier than reinforcement learning. Furthermore, the policy easily understandable and interpretable. The ensemble should not be seen as an alternative approach to current approaches, but rather an extension of them allowing to learn how to combine several approaches to further increase their performance.
We evaluate our approach using active learning tasks from different domains and with random forests, CNNs and LSTTS as classifiers. The experiments show that the ensemble consistently performs at least as good as the best agent it includes and sometimes even outperforms it by significant margin.
Our main contributions are:
We propose an approach to learning active learning using an ensemble of heuristics. combines the advantages of heuristics and approaches learning active learning. The experi- ments show that the theoretical advantages also translate into a high empirical performance.
We show with our experiments that it is very important to train the ensemble on a similar task it is evaluated on. This is contrary assumptions in earlier literature that a learning agent trained on synthetic dataset works well in completely different domains.
RELATED WORK
2.1 ACTIVE LEARNING HEURISTICS
Heuristic frameworks are active learning frameworks relying on engineered, fixed policies. Their performance depends highly on the active learning task, with no heuristic being able to outperform the others in all cases. There are three core ideas which kind of samples should be chosen to be labelled. They can be clearly distinguished, they rely on disjoint sets of features:
Informativeness sampling: One group of heuristics prefers to choose informative samples, which can be expected to change the supervised learning model a lot when added to the labelled set. Their features for sample can be calculated only given the current supervised learning model and the sample itself. The most popular heuristic in this group is uncertainty sampling (Lewis Gale 1994] Scheffer et al. 2001 Shannon 1948), while many others exist. Diversity sampling: This approach chooses samples which are dissimilar to already labelled samples. Its features are similarity metrics from sample to the current labelled set. It is nearly always combined with uncertainty based sampling. Representative sampling: This approach, also called density-based sampling, chooses samples which are representative for the whole dataset. In particular that means that outliers should not be chosen. Like diversity sampling, it is usually combined with uncertainty sampling.
The combination of features or heuristics from these three disjoint groups is used by many re- searchers to increase the performance of active learning agents (Wang et al. 2017 Sener & Savarese) 2017 Zhu al. 2009). There are many ways combine heuristics, e.g. possible to first choose a subset of candidate points using one heuristic and then choose the final points to be labelled us- ing the other heuristic, or one could directly define a heuristic combining two existing ones. While our approach with a learned ensemble is the first known approach to learn the best combination of heuristies, there exist many ideas in current literature, which could improve the performance of the proposed ensemble even further:
Wang et al. (0017) have added a diversity constraint to uncertainty sampling, and use a Quadratic Programming approach to enforce this constraint. Sennrr & Savarese) (2017) have described active learning as a sere-set-selectiee to each other and the labelled set and representative of the unlabelled set. These share that they propose advanced non- -reededd methods for choosing a batch of samples. A greedy choice is only a !-agoorithm, as already pointed out by Kirsch et al. (2019). Adapting our
problem. Thus, they try to choose samples which are dissimilar both
approaches
2

---

Under review as a conference paper at ICLR 2021
approach such that the samples are chosen non-greedily using the method by Wang et al. or Sener and Savarese is thus a promising future research direction.
Zhu et al. (2009) have proposed sampling by uncertainty and density approach choosing the sam- ples whose product of their uncertainty and density metric is highest. This idea can be used to extend the simple linear combination of heuristics proposed: The product of the utility values of two heuristics or any other combination of them can easily be included as the utility of an additional heuristic of the ensemble.
Ash et al. 2019) have proposed a batch-mode AL agent choosing samples such that their gradients diverse and used the k-means++ algorithm to choose the batch to be labelled. The diversity of gradients could be added as an additional feature for the ensemble we propose.
Active learning agents may perform better if they use different strategies in the beginning than in the end of the active learning episode. [Tang & Huang (2019) have shown that it helps if the active learning agent chooses easier samples in the beginning and harder samples only in the end. This idea could also be used to extend our approach, by letting it learn schedule of policies instead of having a constant policy.
2.2 LEARNING ACTIVE LEARNING
As the MDP of active learning is unknown for new tasks and datasets, methods to find the optimal policy given this MDP must rely on data. Thus, dynamic programming is not feasible, instead re- inforcement learning algorithms are the suitable method for this problem. In the sequential active learning setting. the action space is categorical mne-dimensional space allowing to efficiently cal- culate the action maximizing a function given the action. This makes the common reinforcement learning framework of Q-Learning suitable. It fits a regression function Q(observation, action) to the training data, which tries to predict the reward given the observation for each action. Its policy (obsercation) is simply choosing the action maximizing the Q-Value given this observation. This approach is used by Konyushkova et al. (2017 2018); [Bachman et al. (2017); Fang et al. (2017). Another approach is used by Liu et al. (2018b 8b al: They use imitation learning and train the agent to choose the action maximizing the reward directly without estimating the reward. Their policy to imitate is a so-called algorithmic expert: It a policy already knowing the true labels of the samples to choose and is thus able to calculate the improvement of the model if a sample is added to the labelled set. This equals evaluating the reward function without actually performing the step. They have applied this approach not only in the sequential, but also in the batch-mode active learning set- ting: Their agent sequentially chooses the samples to add to the batch. Our approach is also based on this idea of sequential batch-fillie -illiniging active learning.
The error decay can also be assumed to only depend on the number of samples in a cluster (Chang et al. 2020). They predicted the error decay using parameterized policy depending only on this single value. Because this policy is mostly parameterized heuristic, it only needs a small number of samples be trained dataset.
Learning batch-mode active learning has also partly been covered by Ravi & Larochelle (2018): They used a Q-learning algorithm to train a regression model to predict the improvement of the accuracy if a sample is added to the labelled set. Then they multiply this quality metric with a diversity metric gain final expert metric. The diversity metric takes the similarity of sample to other unlabelled samples into account. The policy is to choose the samples having the highest expert metric. While the quality metric is learned metric, of the quality metric and the diversity metric. Thus, this approach can be seen as a mixture of an engineered heuristic and a learning agent. Their idea use not only a combination of heuristics, but in particular a combination of learned active learning agent and a heuristic can also be used to improve the ensemble we propose: Any learned active learning agent can be added to the set of agents making up the ensemble.
a design choice to maximize the product
2.3 ENSEMBLES
In supervised learning, combinations of several different models can be combined to form semble. They combine the output of these models to calculate the final output. The combination itself might be learned by another supervised learning algorithm using the predictions of the other
3

---

Under review as a conference paper at ICLR 2021
models as input features. This approach is called stacking and usually performs better than the best method being part of the ensemble [Wolpert 1992). Learning the weights of an ensemble of agents can also be seen form of stacking, which further validates our claim that an ensemble of active learning agents performs better than the single best agent being part of it.
While ensembles of supervised learning models are widely used, there has been very little research on ensembles of agents. Nonetheless, it was found that ensembles of reinforcement learning algo- rithms using majority voting and Boltzmann multiplication perform significantly better than single algorithms [Wiering & Van Hasselt 2008).
Gao et al. (0188) have proposed to learn AL by learning to choose of three heuristics each step of the active learning process. Different to our approach, the heuristics they used and their experiments are restricted to learning network representations. Furthermore, their approach is only applied in the sequential AL setting.
FRAMEWORK FOR LEARNING BATCH-MODE ACTIVE LEARNING
Our framework for learning batch-mode active learning is split into three parts: First, batch- active learning is formalized as Markov decision process. Second, an ensemble of several different active learning is defined. Third, the parameters of this ensemble are learned using Monte Carlo policy and its objective function and optimizer are defined.
3.1 MARKOV DECISION PROCESS FOR SEQUENTIAL AATCH-FILLING ACTIVE LEARNING
The goal of every active learning agent is to choose the next batch of samples to be labelled based on features of the current active learning process such that the supervised learning model learns most. The MDP for pool-based active learning is described by the 4 tuple (S,A, P. R):
The state space S consists of the labelled and unlabelled set of samples and the model trained on the labelled set. As this space is extremely huge and can include millions of variables, it is very hard for an agent to learn from this state. Thus, we define an observation space, which includes features of the state which might be relevant to the active learning agent. Common metrics in this space are the predictions of sample and metries computed out it, the uncertainty of the prediction and similarities or distances of unlabelled samples to each other and labelled samples. Most active learning heuristics define a single metric to rely on and do not combine them.
The action space A is the categorical choice of one sample set. As the unlabelled set becomes smaller each step, the action space is variable.
batch of samples in the unlabelled
Following the frameworks for learning sequential active learning (Konyushkova et al. 2017 2018 Bachman et al., 2017| Fang et al., 2017) we are using reward shaping (Ng & Jordan 2003) and set the reward to be the improvement of the model accuracy after a full batch is labelled and the supervised learning model is re-trained on the new labelled set. This makes the MDP a minimal extension of existing MDPs to batch-mode AL and facilitates comparison with them.
Instead of defining the probability P(s'|s, a) of a new state given the current state and action, we define a transition function directly mapping the current state and action the new state. In the case of active learning this step includes the annotation of chosen sample or batch of samples, the re-training of the model with it, the change of both the labelled and unlabelled dataset, and the increase of the currently spent annotation budget.
Choosing a batch of samples to be annotated next is a very hard task: one wants to choose samples out of a pool of и unlabelled samples, there are (+) options. Thus, we set the definition of a step of one MDP for batch-mode active learning to be the choice of one sample which is added the batch. The corresponding pseudocode for this sequential batch-filling MDP is shown in Algorithm I In each step, the agent chooses the next sample to be added to the batch based on the current observation. As the observation includes similarity measures of unlabelled samples to samples in the batch, the agent can choose samples which are not too close to samples already in the batch. While the unlabelled set becomes smaller every step as sample leaves it and enters the batch, the labelled set is only updated if the batch is full (i.c. the io-cciiiti in the pseudocode is fulfilled): Then the labelled set is extended by all samples in the batch, thus all samples in the batch
4

---

Under review as a conference paper at ICLR 2021
annotated. As the labelled set updated, the supervised learning model can be retrained on it, changing the observation significantly.
Algorithm 1 Pseudocode of the step function of a batch-mode pool-hased active learning MDP function CneepP"nccciionsmmplle oosaommpleeTTelelllll Labell
D Duntobelled <- D untabelled batch batch U Sampleeooaabll
SatrnpeeTTabbbel
if |batch| == batchSize then
Diabelled t- Dlabelled batch.
batch +
new Accuraay f- -uperriseddeaamnnngglll...d... (Dibbetta)))))) reward +- new newAcccracy - old Accuracy
oldAccuracy + Accurayyyy
else
reward +-
end if
observation + get (ariiionnDuutaatttii Duntabelled; Drabelted; batch)
epochFiinisten + Labelted + |batch] >= annotation return observation, reward, FrochFinished end function
This definition of sequential batch batch-filling active learning combines the advantages of sequential and batch-mode active learning. The action per step is still one categorical choice like for sequential active learning, making it much easier for learning agent to learn which sample to choose next. As the labelling is only performed for a full batch, the practical advantages of batch-mode learning still preserved. However, there a theoretical disadvantage: Compared to true sequential learn- ing, the supervised learning model is not trained on the samples just chosen, and thus the features depending on the supervised learning model are not updated. Thus, it can be expected that a se- quential batch-filling agent performs slightly worse than true sequential agent, with the difference monotonically increasing with the batch size.
One shortcoming of this definition of the MDP is that the reward is only non-zero if a full batch was labelled. This makes reinforcement learning approaches like Q-learning difficult to apply, as is hard to attribute the reward to the choice of specific sample, i.e. single action. This could be solved by re-training the supervised learning model after each sample added to the batch to calculate fictive reward during the training of the RL approach. After this, the model has to be set back the previous state to keep the state and observation independent of this step. Such an approach would make it possible attribute each reward exactly sample, however, it has the disadvantage of needing batch- times more re-trainings the supervised learning model, thus increasing the computational complexity by the batch size. Hence, Monte Carlo policy search more effective than approaches like Q-Learning.
A general challenge of learning active learning is that each computation of reward needs the re- training of a supervised learning model, which is computationally very expensive for large models and complex problems. This property makes sample-efficiency an important criterion for choosing a suitable framework to learn active learning. Consequently, Q-Leaming and Monte Carlo policy gradient are unsuited methods. Model-based RI. also unsuited, because predicting the next state is difficult for two reasons: First, the state space has multiple features for each of many samples and thus very large. Second, predicting the next state essentially equals predicting the outcome of a neural network which is not easier than training the neural network itself.
3.2 WEIGHTED ENSEMBLE OF ACTIVE LEARNING AGENTS
We propose an active learning policy that is represented by an ensemble different active learning heuristics. The basic idea is have several different active learning agents which all assign measure for the utility of choosing a certain data point to be labelled next and each choose the data point with the highest utility. We evaluate this approach using ensemble of five agents:
The random sampling agent samples the utility of each sample independently from the same Gaus- sian distribution. The uncertainty sampling agent defines the utility of a sample as the entropy of
5

---

Under review as a conference paper at ICLR 2021
its prediction vector. The diversity sampling agent chooses samples which are dissimilar to samples already chosen for labelling. We defined its utility measure to be the minimum euclidean distance of sample to sample already in the labelled or batch set. The representative sampling agent chooses samples which are representative of the whole data distribution. In particular it aims not to choose outliers. We defined its utility as -1 the 5%-percentlle of the euclidean distance of sample to all other samples. The uncertainty diversity sampling agent defines its utility as the product of the utilities of the uncertainty and diversity sampling agent (Zhu al. 2009). The ensemble
agent chooses the sample having the highest overall utility defined as the weighted sum of the utility measure of each heuristic. The corresponding function is given Equation
ubility(ensemble) +
Σ 3i * utility(agent))
The beta-parameters could be chosen such that the agent performs exactly like one single heuristic by setting one 3 to high value and the others zero. Thus, assuming this agent learned the best 3-parameters for a task, its worst-case performance is the one of the best heuristic on this task, but it can also perform significantly better.
We do not claim that an ensemble made of these four simple heuristics outperforms ootee----he-att batch-mode active learning agents. We do claim, however, that the ensemble performs least as good as all agents contains, but can perform significantly better. Thus, any state-of-the-art active learning agent be added to the of agents making up the ensemble, which allows the ensemble to further improve the performance of this agent by combining it with others.
3.3 CHOICE OF PARAMETERS OF ENSEMBLE
The linear combination of active learning agents is policy parameterized by the 3-parameters. We decided to learn which parameters are good using Monte Carlo policy search. The objective is to maximize the performance of the supervised learning model after complete episode. The objective function is defined in Eqn. 2 and Algorithm 2 and calculates the performance of one choice of 3. Taking the mean of several active learning runs reduces the noise of the objective function. It is very hard to attribute the performance at the end of an episode to the choice of specific action in a specific state or to the choice of 33. Thus we do not solve the optimization problem using policy gradients, but treat it as a black-box function hyperparameter optimization problem instead. Methods like Bayesian optimization, random search, grid search, tree parzen estimators, or population-based training could be used to find the optimal solution. Among them, chose tree parzen estimators (Bergstra et al. 2013) as they are very sample-efficiee and search for the global optimum.
objectiveFuunctin//]3 R² R with n = number of heuristics making up the ensemble
(2)
Algorithm Pseudocode for the objective function of the ensemble agent Ensemblee.Agent
Require:
trainingTT ask,, aative_LarminngEneirmmee function objective Function(())
agent + Ensemblle
For i +- to numbee Episodes do
results +- active
agent)
per formanceslii t results.get Pin fir
end for
mean wareeffrrmann formance +- mean(per mean(performonees formances)
return mean Per formance
end function

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS AND RESULTS
4.1 SETTINGS
We compare our ensemble of five heuristics with each of these five heuristics and an ensemble not using the learned weights, but setting all weighs to the same value. We set the annotation budget such that the performance differences between the agents are clearly visible and set the batch size such that between and iterations could be performed. The starting size to the number of classes.
be proportional
For training the B-parameters, fixed the 3- 3-parameter for random sampling and chose log- normal distribution with a mean of and standard deviation of 2 as prior for the other parameters.
The later parts of this section show the performance of the active learning agent on the tasks as plots of the classification accuracy over the number of labelled samples. The plots for the different evaluation tasks all share the same structure: The dark shaded area around the curves of each agent report the 95% confidence intervals, the light shaded areas are the standard deviations. A suffix to the agent names in the form of" batchSize' denotes the batch size the agent used. There no such suffix for the random, diversity and representative sampling agent, as their features are independent of the batch.
4.2 EXPERIMENTS ON UCI DATASETS
We used datasets from the UCI machine learning repository (Dua Graff 2017). The datasets 2- breast cancer, 3-diabetes, 4-flare solar, 5-heart, 6-german, 7-mushrooms, 8-wavelorm, 9-wdbe were used for training and O-adult, 1 l-australian, and 10-spam for evaluation, similar to Konyushkova et al.) (218)). The objective function was set to be the harmonic mean of the final accuracies the ensemble reached on each of the 7 training tasks. This allows training on multiple datasets easily and thus improves generalization. The results for the l-australian evaluation task is shown in Figure T the results for the other evaluation tasks are quite similar.
4.3 EXPERIMENTS ON CHECKERBOARD DATASETS
The datasets checkerboard 2x2 and rotated checkerboard 2x2 were used as training tasks to train the learning agents, the checkerboard 4x4 task was used as evaluation task. All checkerboard tasks are XOR-like binary classification tasks with two-dimensional input data. We used random forest as classifier and the results are shown in Fig. [
4.4 EXPERIMENTS ON (FASHION-) MNIST TASKS
The image classification tasks use the MNIST [eCun et al. 2010) and fashion-MNIST (Xiao et al, 2017) dataset and three-layer convolutional neural network implemented in keras (Chollet al., 2015a) as classifier. Because Munjal et al. (2000) have shown that performance differences be- tween different active learning agents may vanish when we tried to maximize the performance of the random baseline using following techniques: random data augmentation, dropout, I.1- and 1.2 regularization and hyperparameter optimization using tree parzen estimators (Bergstra et al.) 2013). We did not use more advanced networks like the ResNet or VGG16 because they are computationally much more expensive and tend overfit their huge number of parameters on the very small training set size. Instead of calculating the distance metrics using the raw images, we used tSNE embeddings (Maaten Hinton 2008). We trained the ensem- ble agent on the MNIST task and evaluated it on the fashion-MNIST task, the results are shown in Fig.
regularization techniques used,
4.5 EXPERIMENTS ON QUESTION ANSWERING TASKS
The bAbI dataset Weston et al. 2015 is collection of question answering datasets of which two different challenges are used: one with single supporting fact and one with two supporting facts. The classifier used is using a long short term memory and is taken from a keras example, which follows Sukhbaatar al. 2015). Like for the image classification tasks, the hyperparameters were

---

Under review as a conference paper at ICLR 2021
Task: UCI Ivaustratian
Task: checherioard Acé
0.85
0.30
0.8
0.7
Diversity
Diversity
0.70
Ensembiee_
Ensemble_jg
Ensembiee Equal _Weght
0.00
Random
Representative
Recresentative
Random
Uncertaincy
Uncartainty 32
0.50
Uncertaincy
0.5
Uncartainty
10
25
23
100
150
200
number
Task: fachion NNST
Tosk: bA21 two supporting fects
0.80
Diversity
0.24
Ensemble Equal. Weight, 54
Random
0.22
Uncertainty_pe
0.60
Uncertainty Divesiyy
0.60
Dverstty
Ensemble
0.20
0.35
Randem
Resresentative
Uncertainty 54
0.16
Uncertainty. Diversty 64
100
150
200
250
300
350
50
130
150
200
250
300
350
number Samples.
number of Sampless
Figure 1: Results on evaluation tasks. The ensemble performs at least as good as the best heuristic, but sometimes significantly better.
optimized to minimize the loss on small random subset. We used the bAbl task with single supporting fact as training task and the one with two supporting facts as an evaluation task. For calculating the distances, we used word2vec embeddings (Mikolov et al. 2015). The results are shown in Fig.
4.6 ANALYSIS
The performance of the ensemble on the checkerboard and bAbl task was found to equal the one of the best heuristic, whereas it showed significant performance increase compared to the best heuristic on the UCI and fashion-MNIST task. The ensemble with all weights set to 1 performed equal or worse than the one with learned weights, which shows the importance of learning the optimal weights on a training task. Our approach has also shown to have the same behaviour with respect varying batch sizes uncertainty sampling, see Fig. 3 It performs better with smaller batch sizes and slightly worse with larger ones. The effect of larger batch sizes depends on the weight uncertainty and uncertainty diversity sampling, the other heuristics do not depend on the batch size at all.
4.7 INTERPRETATION LEARNED PARAMETERS
The ensemble learns the weights of the agents being part of it. These parameters can be easily inter- preted and thus allow us to understand the ensemble's policy better. Furthermore they might be used to find out which kind of samples could be additionally generated, if possible. The weights learned for the four different tasks or tasks groups are given Table 4.7 The weights were normalized by the sum of all weights, allowing better comparison of the weights leamed the different tasks.
For the UCI datasets, diversity sampling is very unimportant, which is indicated by the weight of diversity sampling being nearly zero. Thus, it can be assumed that the samples in the UCI datasets are already quite dissimilar to each other. The ensemble trained on the checkerboard datasets has weight of representative sampling being zero, which can be well explained by the fact that it does
8

---

Under review as a conference paper at ICLR 2021
random
uncertainty
diversity
representive
uncertainty diversity
UCI
0.21
0.24
0.01
0.18
0.15
checkerboard
0.15
0.14
0.30
0.00
0.25
MNIST
0.16
0.08
0.06
0.28
0.27
bAbl
0.00
0.00
0.00
1.00
0.00
Table I: Comparison of beta-parameters learned the different training task combinations. The greener, the higher the weight.
not contain any outliers. Both MNIST and bAbl have much more samples in the unlabelled set, explaining why representative sampling is quite important for them. The weights differ lot between the different training tasks, indicating the importance of training the parameters of the ensemble on similar dataset it is later applied on.
CONCLUSION
In this paper, we proposed novel approach for active learning using an ensemble of different active learning heuristics or trained agents. By learning the weights of linear combination of different agents in the ensemble, this approach can be easily adapted to different datasets and supervised learning models. To train the ensemble decision process. Experiments in different domains and using different supervised learning models have shown the effectiveness of our approach:
formalized batch-mode active learning as Markov
As expected, the worst-case performance of the ensemble is the one of the best heuristic, while outperformed the best one significantly some datasets. Furthermore, it was shown that it is very important to adapt the active learning agent to the dataset it applied on.
The most natural extension of ensemble is to make it up with more and learning agents. Additionally, we assume following future research directions to be very promising: First, different agent weights could be learned for different time points of the episode, allowing to change the policy with increasing accuracy. Second, the weights could be learned on a small subset of the dataset it is later applied on. This allows its application on tasks where no similar training task is available or where the ensemble generalizes badly between the training and application task.
advanced active
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané, Con- crete problems in ai safety. arxiv 2016. arXiv preprint arXiv: arXiv:/606.0m
Jordan Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.0366 036771,, 2019.
Philip Bachman, Alessandro Sordoni, and Adam Trischler. Learning algorithms for active learning- In Proceedings of the 34th International Conference on Machine Learning-Voll Volume 70, pp. 301- 310. JMLR. org, 2017.
James Bergstra, Daniel Yamins, and David Daniel Cox. Making a science of model search: Hyper- parameter optimization in hundreds of dimensions for vision architectures. 2013.
Haw-Shiuan Chang, Shankar Vembu, Sunil Mohan, Rheeya Uppaal, and Andrew McCallum. Us- ing eиror decay prediction overcome practical issues deep active learning for named entity recognition. Machine Learning, pp. 1-30, 2020.
François Chollet et al. Keras. https////s/////noooooooo////aoar
/ttp:/:///aaa.... 2015a.
François Chollet et al. babi dataset, 2015b.
python implementation of memory network trained the URL fttta:://gittuub.co//keeeer
com

---

Under review as a conference paper at ICLR 2021
David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine learning, 15(2):201-221, 1994.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL BtRp:::://archive
ics.uci..ee
Meng Fang, Yuan Li, and Trevor Cohn. Learning how to active learn: deep reinforcement learning approach. arXiv preprint arXiv: 170:::2388.3 2017.
Li Gao, Hong Yang, Chuan Zhou, Jia Wu, Shirui Pan, and Yue Hu. Active discriminative network representation learning. In ICCI International Joins Conference on Artificial Intelligence, 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06633 2017.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acqui- sition for deep bayesian active learning. In Advances in Neural Information Processing Systems, pp. 7024-7035, 2019.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. In Advances in Neural Information Processing Systems, pp. 4225-4235, 2017.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Discovering general-purpose purpose active learn- ing strategies. arXiv preprint arXiv: 1810.041 2018.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: hutp://yann. lecun. com/exdb/mnist, 2, 2010.
David D Lewis and William A Gale. sequential algorithm for training text classifiers. In SIGIR 94, pp. 3-12. Springer, 1994.
Ming Liu, Wray Buntine, and Gholamreza Haffari. Learning actively learn neural machine trans- lation. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pp. 334-344, Brussels, Belgium, October 201 8a. Association for Computational Linguisties. doi: 10. 10//3//11/K8-// 1/K18-1033, URL https:
ac web. org/
1013
Ming Liu, Wray Buntine, and Gholamreza Haffari. Learning how actively learn: deep im- itation learning approach. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1874-1883, Melboure,, Australia, July 2018b. Association for Computational Linguistics. doi: 10. 10.18633/1//118-1174 URL https: www. www.wb.wwbbb.. org/ /aathhology/P1--117
David Lowell, Zachary C. Lipton, and Byron C. Wallace. How transferable are the datasets collected by active learners? CoRR, 1aw/10877.000... 2018. URL http:://axxv..rr/////a////
04801
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2559 2008
Tomas Mikolov, Kai Chen, Gregory Corrado, and Jeffrey A Dean. Computing numeric represen- tations of words in a high figh--immnsiooal space, May 19 2015. US Patent 9,037.464.
Marvin Minsky. Steps toward artificial intelligence. Proceedings of the IRE, 49(1):8-30, 1961.
Prateek Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, and Shadab Khan. Towards robust and reproducible active learning using neural networks. arXiv, pp. arXiv-2002, 2020.
Andrew Y Ng and Michael I Jordan. Shaping and policy search in reinforcement learning. PhD thesis, University of California, Berkeley Berkeley, 2003.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Coumapeau,, M. Brucher, M. Perrot. and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2834 2011.
10

---

Under review as a conference paper at ICLR 2021
Sachin Ravi and Hugo Larochelle. Meta-learning for batch mode active learning. 2018.
Tobias Scheffer, Christian Decomain, and Stefan Wrobel. Active hidden markov models for in- formation extraction. In International Symposium on Intelligent Data Analysis, pp. 309-318.
Springer, 2001.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint wtXi::170.00445 2017.
Burr Settles, Mark Craven, and Lewis Friedland. Active learning with real annotation costs. In Proceedings of the NIPS workshop on cost-sensitive learning, pp. 1-10. Vancouver, CA, 2008.
Claude Elwood Shannon. A mathematical theory of communication. Bell system technical journal, 27())379-423,, 1948.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, al. End-to-end memory networks. In Advances in neural information processing systems, pp. 2440-2448, 2015.
Ling-Peng Tang and Sheng-Jun Huang. Self-paced active learning: Query the right thing at the right time. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp- 5117-5124, 2019.
Gaoang Wang, Jenq-Neng Hwang, Craig Rose, and Farron Wallace. Uncertainty sampling based active learning with diversity constraint by sparse selection. In 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP), pp. 1-6. IEEE, 2017.
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy 2015.
tasks. arXiv preprint arXiv:1502.05555
Marco A Wiering and Hado Van Hasselt. Ensemble algorithms in reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 38(4)993--936, 2008
David H Wolpert. Stacked generalization. Neural networks, 5(2):241 1992.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 2017 2017.
Jingbo Zhu, Huizhen Wang, Benjamin K Tsou, and Matthew Ma. Active learning with sampling by uncertainty and density for data annotations. IEEE Transactions on audio, speech, and language processing. (363):333-1333 2009.
A APPENDIX
A.1 DETAILS OF CLASSIFIERS
The classifier used for both the UCI and the checkerboard datasets the used by Konyushkoval et al. (2017): A random forest implemented with scikit-learn (Pedregosa et al. 2011) and 50 esti- mators. The other hyperparameters were set the default value. We chose this classifier for easier comparison with the work by Konyushkova et al. (2017).
The classifier used for the Mishhoon-MMNIST task is CNN with two convolutional layers, one inter- mediate dense layer and a softmax output layer. Intermediate dropout and max- maxx-ooliig layers were added. The exact structure and the setting hyperparameters can be found in the additional material in / ssupervil sed ea rninnn priinggtassssssssss takks//akk.. vis on takk liisss_iiiiee Visio in oin_CNN py: : f We used random data augmentation. We had also tried the ResNet16 architecture, but found it to have worse performance, probably because its large number of parameters was overfitting the few hundred images it was trained on. Furthermore, was computationally much more expensive. We chose CNN with data augmentation classifier as it is the state-of-the-art architecture for image classification.
The classifier used for the bAbl task is memory network (Sukhbaatar et al. 2015). The code is largely based on keras example (Chollet et al. 2015b). We chose this architecture as provides Mate-offthe-aa performance.

---

Under review as a conference paper at ICLR 2021
A.2 PERFORMANCE OF AL AGENTS WHEN TRAINED ON THE FULL DATASET
As shown Fig 2 the worst agents need about times more samples to reach the same accuracy the best agents. They also need much longer reach almost the same accuracy trained on the full dataset.
1.0
0.85
0.80
0.75
Diversity
Ensembre_
Ensemble_13
Fosembtie_
EepemtiieFFuul_ Figun Weigh
Frsemble F Weight
Random
0.5
Random
Representative
Recresentative
Linceralinty
Uncortainty
0.60
Uncertainty Diversity_
0.5
Uncortainty, Ceciraintt__Dverstte Divesitty
50
200
300
800
1000
Samples.
Samples
Figure 2: Results UCI and checkerboard task with the annotation budget set to the full dataset.
A.3 PERFORMANCE ENSEMBLE WITH OTHER BATCH SIZES
The plots in Fig. include the ensemble with both a lower and a higher batch size than the one it was trained on. Across all task, the performance the ensemble monotonically decreasing with the batch size. On the UCI dataset, the ensemble performs always slightly better than uncertainty sampling with the same batch size. On the checkerboard task, the ensembles with a batch size of 2 and 32 respectively have almost the same peformance, the one with batch size of 160 only performs slightly worse. On the fashion-MNISY task, the ensemble with batch size of 16 performs much better than the ones with bigger batch sizes. On the bAbl task, the ensemble had learned to rely almost completely on the representative sampling agent, whose features are independent of the batch size, thus the ensemble's performance is also independent of the batch size.
12

---

Under review as a conference paper at ICLR 2021
Task: UCI Ivaustration
Task: chacherizorrd Acé
0.80
0.8
Diversity
0.75
0.7
Ensemblc, 2
Intemble_
Ensemble. 32
Ensemble 24
Ensemble_
Ensemble
Laraalee___aaeeee Weight
0.55
Uncertairty 2
Random
Recresentative
Random
Uncertainty 24
Uncartainty 32
Uncertainty
0.5
Uncartainty
10
25
23
100
125
150
175
200
number
Task: fashion NNST
677
Diverstty
Freembie_11 16
Ersemble 160
Ensemble
Randem
Resresentative
Uncertainty. 54
Uncertainty. Diversiy
100
150
200
250
300
350
number Samples.
Figure 3: Results tasks with ensemble with other batch sizes.
A.4 DETAILS TRAINING TASKS
The ensemble trained on the UCI datasets 2- 2-breast-cancer through 9-wdbe was trained with starting size of 8, an annotation budget of 32, batch size of and the objective was set to the geometric mean of 64 episodes (8 per dataset).
The ensemble trained on the checkerboard datasets was trained with a starting size of 8, an annota- tion budget of 104, a batch size of 32 and the objective was set to the geometric mean of 64 episodes (32 per dataset).
The ensemble trained on the MNIST dataset was trained with starting size of 40, an annotation budget of 296, a batch size of 64 and the objective was set to the arithmetic mean of 2 episodes.
The ensemble trained the bAbl single supporting fact dataset was trained with a starting size of 40, an annotation budget 296, a batch size of 64 and the objective was set to the arithmetic mean of 2 episodes.
The starting size was set to 4 times the number classes for each task. The annotation budget and batch size were set such that the accuracy does not converge yet and 3 to 4 batches chosen per episode. The number of episodes run for each evaluation of the objective function was set such that the variance of the objective is approximately the chosen if multiple datasets were combined for training, so that the difficulty of each dataset does not influence its weight relative to the other ones. Otherwise the arithmetic mean was chosen.
across all tasks. A geometric mean was
A.5 DIFFICULTY OF FINDING THE OPTIMAL BETA-PARAMETERS
The question how likely it is, that the optimal 3 p-rrameters are found, can be broken down into to subquestions:
What does the objective function look like? The dataset, classifier and set heuristics to choose from determine the form of the objective function mapping the 8-parameters to the final accuracy of the classifier after a complete episode. Important characteristics of this function how noisy it is and whether it contains local minima saddle points making it
13

---

Under review as a conference paper at ICLR 2021
harder to find its optimum. We found for all four training tasks, that the objective function is very noisy, but does not contain any local minima.
How likely it, that a black-box optimizer finds the optimum of the objective function? Given the fact that only 4 continuous parameters can be optimized and the simple structure the objective function, can expected that most black-box optimizers can find an optimum quite easily. However, they should be able to handle the noisy nature of the objective function.
A.5.1 OBJECTIVE FUNCTION OF UCI TASK
The plots Fig. show the datapoints of many random evaluations of the objective function of the UCI task. The 6-parameters are again normalized by their sum. The four smaller plots show the accuracy over each of the 4 3-pparaeterss and include a fitted polynomial of second degree. It shows that high weights for uncertainty sampling should be preferred, which fits well to the fact, that the optimizer found that uncertainty sampling should be assigned the highest weight. The larger graph shows the accuracy over three of the four parameters. For all tasks, additional plots showing the accuracy over two parameters can be found in the supplementary material.
0.775
0.770
0.765
0.8
0.760
0.6
0.755
0.0
0.0
0.0
0.2
0.2
0,740
Diversity
Uncertainty
0,4
0.735
0.8
Figure 4: Objective function of the UCI task
A.5.2 OBJECTIVE FUNCTION OF CHECKERBOARD TASKS
The plots Fig. show the datapoints of many random evaluations of the objective function of the checkerboard tasks. It is easily visible, that assigning higher weights to Diversity and Uncertainty- Diversity sampling increases the performance. This further validates that the parameters found by the optimizer indeed good.
14

---

Under review as a conference paper at ICLR 2021
0.96
1.0
0.8
0.92
0.4
0.0
0.90
0.0
0.0
0.2
0.2
Diversity
0.4
0.4 Uncertainty
0.6
0.8
1.0
Figure 5: Objective function of the checkerboard tasks
A.5.3 OBJECTIVE FUNCTION OF MNIST TASK
The plots Fig. show the datapoints of many random evaluations of the objective function of the MNIST task. It is clearly visible that uncertainty sampling should be assigned low weight.
15

---

Under review as a conference paper at ICLR 2021
0.92
0,90
0.2
0.86
0.0
0.0
0.0
0.2
0.2
Diversity
Uncertainty
0.4
0.84
0,6
Figure 6: Objective function of the MNIST task
A.5.4 OBJECTIVE FUNCTION OF BABI TASK
The plots Fig. show the datapoints of many random evaluations of the objective function of the bAbl task. It is clearly visible that representative sampling performs best.
16

---

Under review as a conference paper at ICLR 2021
AM
0.4
0.8
0.6
0.4
0.0
0.0
0.0
0,
0.2
0,3
Diversity
0.4
0.5
Uncertainty
0,4
0.6
0.6
0.8
0,7
0.8
Figure 7: Objective function of the bAbl task
17


Reviews:
[{"review_id": "gTigTRwAlIb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper \"Learning Active Learning in the Batch-Mode Setup with Ensembles of Active Learning Agents\" proposes to deal with the problem of active learning via a weighted ensemble of agents. Each agent sequentially selects data to include in the batch to be labelled according to specific heuristics. Finally, the various agents are weighted according to parameters found by a gradient-less approach. \n\nWhile meta-learning of active learning is a very interesting and useful problem, I find the contribution of this paper too weak for a conference as ICLR. The approach is rather straightforward (only a linear combination of different heuristic agents), the experimental results are not fully convincing (there is no real gap between the ensemble and the best individual approach, so using the best agent at training time is maybe a strong alternative) and the paper lacks clarity and details. From my point of view the related work section should shortened to focus on mainly important aspects related to the presented work, a better detailed view (more formalized, less algorithmic) of the approach should be given, and some theoretical insights should be given before it could be considered for publication in a top machine learning conference. \n\n   ", "extra_scores": {}, "extra_text": {}}, {"review_id": "YRsNMf-hbPV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:\nThe paper proposes an algorithm for batch-mode active learning using an ensemble of 4 active learning heuristics. The basic idea is to use an ensemble of heuristics/agents as the utility function to select a batch of samples. The paper proposes to use black-box optimization for optimizing the ratio of combining the agents. The authors perform experiments on various datasets. The results show that the proposed method outperforms the baseline heuristics in most settings, and sometimes performs significantly better.\n\nReview:\n\nI would vote for rejection of this paper. \n\n1. The authors use the whole section 3.1 to describe the MDP formulation of the batch-mode active learning problem. However, the final method is not any RL algorithm but using BO. This is disappointing and also misleading. Why is policy-gradient/Q-learning not used? What will the performance be if we use RL algorithms? \n\n2. The experiments are only comparing to the baseline heuristics, and are missing comparison with other previously proposed learning AL methods as in Section 2.2. This is not the first paper on learning to active learn in batch-mode, see e.g.:\n\t Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford and Alekh Agarwal. Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds. ICLR 2020.\n\n3. The idea of ensembling different heuristics is not new either. For example, this paper uses a multi-armed bandit view to combine heuristics:\n\tGao L, Yang H, Zhou C, Wu J, Pan S, Hu Y. Active discriminative network representation learning. InIJCAI International Joint Conference on Artificial Intelligence 2018 Jan 1.\n\nMinor Comments:\n\n1. Last paragraph on page 4 - it should be $\\binom{n}{b}$ options to choose the batch (the orders do not matter for a batch).\n\n2. Algorithm 2: It is better to replace \"noEpisodes\" with \"numberEpisodes\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "SrwFtfy6S0G", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes learning a batch mode active learning (AL) policy as a weighted ensemble of existing AL techniques (or agents). In the proposed method, the ensemble weight vector (\\beta) is learnt from data. AL is simulated on a set of training tasks where performance for various choices of \\beta are estimated using a Monte Carlo approach. Subsequently, the optimal choice of \\beta is found using tree Parzen estimators, a black box hyperparameter optimization technique. Experimental results are shown on Checkerboards, Fashion MNIST, bAbI and 10 UCI datasets. \n\n+ves:\n+ The Related Work section is extensive and clearly explains the existing methods in literature, building up to the reasons behind developing the proposed approach\n+ The batch mode active learning framework is clearly defined and well-explained, thus setting the stage for the proposed ensemble based method. \n\nConcerns:\n\n- Sec 2.3, para 2, “In domains where it is not possible to learn the best parameters because there is no similar training task, an ensemble of active learning agents using such an engineered combination of them might be suitable.”\nThe sentence suggests that an ensemble of AL agents is suitable for domains where similar tasks are not available. However, the second contribution in Section 1 states the following. “We show with our experiments that it is very important to train the ensemble on a similar task it is evaluated on.” \nThis contribution seems to be in direct contradiction with the first statement. \n\n- In Sec 3.1, the reward is chosen as the improvement in current model’s accuracy after a step i.e., choosing a sample or a batch of samples. However, Sec 3.3 and Alg 2 suggest that the objective of the \\beta optimization problem is calculated as the final AL performance i.e., the performance of the supervised model at the end of the AL process. If this is the case, then reward shaping technique mentioned in Sec 3.1 might not be necessary, because the intermediate rewards are not used to learn the ensemble weights. This was not clear, kindly clarify if I missed something here.\n \n- Sec 4.6, para 2, “..ensemble trained on the checkerboard dataset has a weight of uncertainty sampling being zero..”. \nThe sentence says that the weight for uncertainty sampling is zero for checkerboard, however Table 1 shows that uncertainty has weight 0.93 for the checkerboard dataset.\n\n- The results would’ve been much stronger and more conclusive if the proposed method of learning ensemble weights had been compared against an ensemble of agents with equal weights (or random weights). Then, the superior performance of the proposed method would’ve strongly validated the need to learn ensemble weights, as opposed to giving them equal weights.\n\n- While it is suggested in the introduction that learning a few weight parameters is much easier than reinforcement learning, the results would’ve been more complete if details were included in the Experiments section, on the computational complexity of the proposed method, as compared to RL methods in terms of time taken. \n\n- The paper could have benefited with a more thorough discussion section. Since ensembling is a fairly well known idea, an in-depth discussion would’ve helped to understand how well ensembling works in case of active learning. For instance: \n(1) Why is there a major difference of three orders of magnitude between the weights for representative sampling for MNIST and bAbI? (Table 1)\n(2) In Table 1, for bAbI dataset, uncertainty and diversity have zero weight. Then, why does uncertainty-diversity sampling has a positive and high weight of 21.5?\n(3) It would’ve been interesting to see an ablation study on how strongly the prior of \\beta parameters would affect the final learned \\beta vector.\n(4) It is mentioned that if the \\beta parameters are learnt properly, then the proposed method will perform at least as good as the best heuristic, in the worst case. In this vein, a discussion on how likely the tree parzen method will attain those optimal parameters, could’ve been insightful. Also, it would help the community to see the limitations of the proposed method. For instance, how do the following factors affect the optimality of \\beta parameters? (a) complexity and size of the dataset (b) choice of the hyper parameter optimization technique. \n\n- The details of what classifiers were used for the experiments on UCI datasets is not provided. A short section could have been introduced as Supplementary material to provide these details. These may be important, especially considering there seem to be different choices of classifiers for different datasets. Why was this justified, and why would this observation generalize? Some ablation studies and analysis on one dataset as to how the performance would change if another classifier was used would help showcase the effectiveness of the proposed method.\n\nIn summary, the paper aims to propose ensembling as a simple alternative to computationally costly RL techniques for active learning. While the method is well-motivated, it is missing some key experiments (especially on the significance of the contribution) and analysis, and has a weak discussion/analysis section.\n\nMinor comments:\n(which did not affect the decision):\n\nQuite a few editorial mistakes were across the paper. Here are a few examples and their corrections. (Note that this is not an exhaustive list, and only examples of similar mistakes across the paper).\n[Abstract, line 2] chose -> choose\n[Abstract, line 5] fix -> fixed\n[Abstract, line 8] applying -> apply\n[Abstract, para 1 last sentence] “to adapt to” -> “adapt to”\n[Introduction, para 3 line 1] shortcoming -> shortcomings\n[Sec 2.1, para 2, line 9] which a -> which are\n[Sec 2.2, para 1, line 1] dataset -> datasets\n[Sec 2.3, sentence 1] “combined to an” -> combined to form an\n[Sec 3, line 3] ensemble of different active learning -> ensemble of different active learning methods\n\nPOST-REBUTTAL:\n\nI thank the authors for their response. While some of my concerns have been addressed, a few key questions haven't been answered.\n\n* The contribution is limited in novelty.\n\n* The general author response mentions that reward shaping is not used in the proposed method. In that case, Sec 3.1 seems a bit pedagogical and misleading, since MDP for AL is described in detail but is not even empirically compared with the proposed method.\n\n* Regarding the ablation studies on a different classifier, while I agree that SOTA networks need to perform well, the ablation study on a different classifier was suggested to rule out the effect of the choice of SOTA classifier in the effectiveness of the proposed method. Also, the authors respond to R1 that the gap between the ensembles' performance and each single classifier's performance is small since they chose SOTA models for the individual classifiers. This is perhaps even more reason to show how ensembling works when the base models are not SOTA.\n\n* Beyond just a statement in the response, it would have been good to see some empirical comparisons between the proposed method and, say, Q-learning-based RL methods - especially in terms of actual running time complexity.\n\n* I also agree with R3 that similar ideas have been explored before (papers cited in R3's review), and it is important to compare with those methods as baselines in the experiments.\n\n* Also, the choice of the datasets used is not justified appropriately.\n\nI stay with my original decision.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DIhD0wdeKAr", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce a novel, batch-mode ensemble approach to Learning to Active Learn that combines the best of both worlds: heuristic- and learning- based active learning. The main idea is to create an ensemble of parametrized active learning agents that will outperform any of the individual agents.\n\nThis is a well-written, easy to follow paper on an important topic. The work appears to be original, and the findings are significant. \n\nThe basic idea of the paper is to combine the-best-of-both-worlds of heuristic- and learning- based active learning. The authors introduce a simple & efficient way to do this. The empirical validation shows that the proposed approach is robust across a variety ML problems. \n\nThe paper would benefit by further strengthening up the empirical validation by answering the following questions:\n- is the proposed approach robust wrt sample size? In other words, what happens when, for all four evaluation tasks, we consider batch sizes of 1/2/48/16/32/64. Most of these graphs could be part of an appendix, with the main paper summarizing the findings\n- how many more labeled examples would take the \"loser\" approaches to reach the accuracy of the \"winning\" one(s)\n- how many more iterations (and at what cost) would the proposed approach reach state-of-the-art accuracy on each evaluation task?     \n\nOTHERS:\n- please run a spell-checker to avoid errors such as \"acitve learning\" (page 2) or \"lineracombination\" (page 8)", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose to linearly combine the utility functions of (batch) active learning algorithms. The linear combination coefficients are "learned" with Monte Carlo estimators to adapt the coefficients to different kinds of tasks automatically.

The reviewers find the presentation within the papers generally clear. The simplicity of the approach, which is highlighted in the authors' rebuttal, should be appreciated. The authors also addressed the issue of robustness with respect to the batch size. But the paper left quite a few unanswered issues even after the authors rebuttal. The novelty with respect to several earlier papers require clarification and concrete comparisons, such as the ones in reinforcement learning and bandit learning as pointed out by reviewers. The lack of comparisons to those existing works, both illustratively and empirically, is a key weakness of the current paper. A more careful study of RL setting (such as reward shaping) is also important to understand the value of the work. Finally, the gap between the ensemble approach and the single approach also deserves more investigation to justify the significance of the contribution.

Decision: Reject