Deep reinforced active learning for multi-class image classification
Authors: Emma Slade
Kim Branson
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement learning
active learning
image classification

Abstract: High accuracy medical image classification can be limited by the costs of acquiring more data as well as the time and expertise needed to label existing images. In this paper, we apply active learning to medical image classification, a method which aims to maximise model performance on a minimal subset from a larger pool of data. We present a new active learning framework, based on deep reinforcement learning, to learn an active learning query strategy to label images based on predictions from a convolutional neural network. Our framework modifies the deep-Q network formulation, allowing us to pick data based additionally on geometric arguments in the latent space of the classifier, allowing for high accuracy multi-class classification in a batch-based active learning setting, enabling the agent to label datapoints that are both diverse and about which it is most uncertain. We apply our framework to two medical imaging datasets and compare with standard query strategies as well as the most recent reinforcement learning based active learning approach for image classification.

Under review as a conference paper at ICLR 2023
DEEP REINFORCED ACTIVE LEARNING FOR MULTI- CLASS IMAGE CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
High accuracy medical image classification be limited by the costs of acquiring more data as well as the time and expertise needed to label existing images. In this paper, we apply active learning to medical image classification, method which aims to maximise model performance on a minimal subset from a larger pool of data. We present new active learning framework, based on deep reinforcement learning, to learn an active learning query strategy to label images based on predictions from a convolutional neural network. Our framework modifies the deep-Q network formulation, allowing to pick data based additionally on geometric arguments in the latent space of the classifier, allowing for high accuracy multi-class classification in a batch-based active learning setting, enabling the agent to label datapoints that are both diverse and about which it is most uncertain. We apply our framework to two medical imaging datasets and compare with standard query strategies as well as the most recent reinforcement learning based active learning approach for image classification.
INTRODUCTION
Modern methods in machine learning (ML), including deep learning (DL) frameworks, require large amounts of labelled data to train sufficiently well to obtain high performance. Depending on the training task, these data can be very expensive to obtain or annotate, to the extent that traditional approaches become prohibitively costly. Active learning (AL) aims to alleviate this problem by adaptively selecting training samples with the highest value to construct minimal training dataset with the most information for the ML model.
In order to select training samples with the most information, different strategies are used in different AL cycles which can be either constructed based on knowledge of the specific problem one is aiming to learn, or using theoretical criteria to approximate mathematical bounds information contained in the data. Standard query strategies in AL include the uncertainty-baat based approach (Lewis and
Gale 1994| Lewis and Catlett 1994; Shannon, 1948; [Schfferr al.) [2001] [Esuli and Sebastiani 2009 Seung et al. 1992; Dagan and Engelson] 1995), which aim to quantify the model uncertainty about the samples to be selected using different hand-crafted heuristies. Other approaches aim to
estimate the expected model change (Roy and Mccallum| 2000]; |Freytag et al. (014)) or employ
diversity-based approaches promote diversity in sampling (Bilgic and Getoor 2009 Gal al.)
2017
Nguyen and Smeulders) 2004).
Some approaches combine different techniques in hybrid-based query strategies, take into account the uncertainty and diversity of query samples (Ash et al., 2019 Zhdanov 2019 Shui et al., 2019 trade-off and reformulate Beluch al. 2018). Other methods leverage the 2015 Chu and Lin 2016) or reinforcement the AL framework
bandit problem (Hsu and Lin
learning problem (Ebert et al. 2012, Long and Hua 2015 Konyushkova et al. (217a)) which are however still limited by their reliance on hand-crafted strategies, opposed to learning new one.
The move towards combining deep learning methods with active learning, to combine the learning capability of the former in the context of high-dimensional data, with the data efficiency of the latter, have led to further methods development. However, combining the two is non-trin non-trivial; traditional active learning query strategies label samples one-by-one, and so batch-model deep active learning aims to use batch-based sample querying (Gal and Ghahramani] [015]] Gal et al. 2017]] Kirsch et al. 2019 Cardoso al. 2017) ensure efficiency in sampling the data.

---

Under review as a conference paper at ICLR 2023
Modern diversity- -wased approaches in deep active learning include the coreset approach (Sener and Savarese 2018 Killamsetty et al. 2020 Wei et al., 2015; Shen et al. 2017 Mirzasoleiman et al. 2019), which aim minimise the Euclidean distance between the sampled and unsampled data points in the latent space of the trained model. Whilst the coreset approach has been shown to work well for image classification tasks (Sener and Savarese 2018), the performance deteriorates the number of classes grows. Furthermore, as the dimensionality of the data grows, the distance measure between data points becomes indistinct due to the curse of dimensionality, Semi-supervised approaches (Sinha et al. 2019; Kim et al., 2020] Zhang et al. 2020) aim to alleviate this issue by using an adversarial network as a sampling strategy pick data with the largest amount of information in the latent space,
Manually designing the DL models in addition to AL query strategies requires both expert knowledge about the task at hand, as well as a lot of compute resources to train the DL. model. Furthermore, as the labelling heuristic is generally specific to the dataset of interest, there is little likelihood that the learnt acquisition function transferrable to other datasets, whereas one based on a meta-learning approach may be more easily applied to other data domains.
In this paper we combine active learning, deep learning and reinforcement learning into an end-to- end framework which can automate the design of the acquisition function for active learning with high-dimensional, multi-class data, in a pool-based active learning setting. We additionally introduce batched-labelling approach, enabling to label multiple datapoints at each step, allowing for much more efficient training.
By employing coreset- irreset-inspired methods, we encourage the reinforcement learning agent to label samples which maximise uncertainty and are also diverse. We apply our model to medical image classification datasets, covering binary and multi-label classification problems as well as differing imaging modalities. We add a range of noise to the images, in order to simulate a real-world annotation setting, and show that our framework is robust to high levels of noise. We compare the classification accuracies to a wide range of sampling methods, including (Konyushkova et al. 2018). the most similar framework to ours, and we show we outperform all other strategies.
2 RELATED WORK
Recent papers on combining active learning with reinforcement learning aim to instead learn a policy for labelling data from the unlabelled pool in order to maximise model performance. In (Bachman et al. 2017 Liu et al., 2018), they use information gathered from an expert oracle to learn the policy, whilst (Pang et al.) 2018 Padmakumar et al. 2018) use policy gradient methods to learn the acquisition function. Current papers which aim to combine deep reinforcement learning with active learning are not able to label more than one datapoint per step (Konyushkova et al. 2018 Woodward
and Finn, 2017 Pang et al. 2018 Padmakumar et al. 2018 Bachman et al. 2017 2017|1 Liu al.) 2018
Woodward and Finn 2017), 7). with the exception of (Casanova et al. 2020 which labels batches of
pixels for semantic segmentation tasks.
Many existing works on reinforced active learning focus on the simpler task of a stream-based active learning approach (Fang et al. 2017 Woodward and Finn 2017) are limited to binary
tasks (Konyushkova et al., 2018 Pang et al. 2018| et al. (001)). which limit their application to more general, and difficult classification tasks. In (Haussmann al. 2019). they learn an acquisition function using Bayesian neural network which is layered onto a bootstrapped existing heuristic.
In this work we focus on medical image classification tasks, which has attracted attention in the field of deep active learning, due to the cost of acquiring medical image data as well as the relatively small size of the datasets. Image segmentation tasks in active learning have traditionally used
uncertainty-basee based acquisition functions (Wen et al.] 2018; Smailagic et al.) [2018] Konyushkova al., 2016) Gal and Ghahramani] [015]] Gal et al.] [017] 7 Yang et 2017] Ozdemir al. 2018).. Generative adversarial network (GAN) based methods, used widely for image synthesis, have been used in order to add informative labelled data to limited training sets, which is directly applicable for active learning scenarios (Zhao et al., 2019 Mahapatra et al. 2018 Last et al.) 2020). There recently has been some research applying meta- leeta-la to medical image tasks in an active learning setting: in MedSelect (Smit et al. 2021) they use reinforcement learning to label medical images, and in (Konyushkova et al., 2017b) selection.
they use a regression model to learn a query strategy using greedy
2

---

Under review as a conference paper at ICLR 2023
Our work is closest (Konyushkova et al. 2018), in that use double deep Q-network (van Hasselt et al., 2015) (DDQN), a generalisation of the deep Q-network (Mnih et al. 2015) to learn the acquisition function and to avoid bias from overestimation of the action values and stabilise training, in a pool-based setting. Our work extends that of (Konyushkova et al. 2018) in several ways. Firstly, we show that multi-class, high dimensional setting, by integrating a deep learning approach into the framework. Secondly, we use batch-based active learning to label multiple datapoints improving on the efficiency of the framework. We also aim to perform the data labelling in much more realistic environment, and we add realistic levels of noise our data, specific to the differing image modalities, to simulate a real-world setting where the images are distorted.
not limited to binary classification tasks and can accurately predict in
time, thus greatly
3 METHODOLOGY
As stated above, our approach aims to reformulate the active learning framework as a reinforcement learning problem. by casting pool-based active learning as Markov decision process (MDP), inspired by (Konyushkova et al., 2018) and (Casanova et al., 2020) and then use reinforcement learning agent to find an optimal labelling strategy. By using not only a meta-learning approach, but full reinforcement learning approach, the selection strategies are learnt in data-driven way, thus removing the bias incurred from using hand-crafted acquisition functions.
3.1 ACTIVE LEARNING AS A REINFORCEMENT LEARNING TASK
In order to reinforcement learning learn query strategy, need cast active learning as a MDP. An MDP is defined by the tuple {(t, az, r++1, }. For each state 8t E S, the agent can perform actions a, to choose which sample(s) to annotate from an unlabelled dataset Ur E 'D. The data are then added to the labelled dataset L, E D. Let fr be classifier (in our case, a convolutional neural network) trained on the labelled data C.. The action a is then function of ft. L, and U4. Depending on the sample(s) chosen to be labelled by the agent, it then receives reward 1+
We can then formulate active learning an episodic MDP. We initialise the active learning loop with a small labelled set Lo 'D alongside the unlabelled dataset Uo = DD where (Mol >> /Lol. At each iteration we then
Train the classifier fa on the initial labelled data fg-
2. Compute the state s, as function the prediction of the CNN ft (x;) + Di. well as Li and Ur
[a E A by following policy x 8, +> a 3. The active learning agent selects N actions {af I==] that defines data T, Ud, to be labelled. Each action at corresponds to one image to be labelled.
4. An oracle labels the data with y and updates the sets f-1= C. (IIn vn) b Uet1 = Me' \ J.Y J
5. Train ft one iteration on the new labelled dataset.
6. The agent receives a reward r1+1 based on the performance of fe+1 on a test set.
The episodes terminate when they reach terminal state, which is when the labelling budget is met. Once the episode is terminated, we reinitialise the weights of the classifier as well as the labelled and unlabelled datasets. The aim of the agent is to learn a policy л maximise the expected discounted reward until some terminal state is reached
Q" (s = =E
E
Σ Tr
(I)
where we set the discount factor 7 = 0.99. The optimal value is then QT* We compute the reward at each time-step as the difference between the accuracy at the current and previous time-steps of the prediction of the classifier on hold-out set DR.
( Q) = max, Q a).
rt+1 ACC(@+1 (5z), 01) ACC( (6:), 0e) Dn.
3

---

Under review as a conference paper at ICLR 2023
3.1.1 STATE REPRESENTATION
2018) we represent the state space S using set-aside dataset D..- We use the sorted list of predictions of ft to represent As we use a CNN for our classifier, unlike (Konyushkova et al. 2018) where they use logistic regression or support vector classifier, we can represent the state for multi-class data and are not limited to binary tasks.
Following (Konyushkova
3.1.2 ACTION REPRESENTATION
In the active learning setting, taking action determines which datapoint(s) to label from the unlabelled pool. We generalise the standard DDQN architecture, by representing each action of as concatenation of 3 features: the score of the classifier fe on Fn and measures of distance between Vn and the labelled and unlabelled datasets in the latent space of the classifier. This enables us to label datapoints explicitly requiring diversity samples, and not just based on how uncertain the classifier is about the potential new datapoint. We compute distance measures in the feature space of the classifier inspired by the coreset approach of (Sener and Savarese, 2018), which, while coresets have been shown to be ineffective for high- ligg--imensional and multi-class data, we find that by including latent space distance measures, as opposed to distance measures directly in the space of the data, that training is substantially faster for little reduction in accuracy.
3.2 LEARNING A POLICY FOR DATA ANNOTATION
We employ DDQN (van Hasselt al. 2015 parameterised by multi- layer-- perceptron, é, to find T*. We train the DDQN with labeled training set 'Dr and compute rewards using a hold-out set DR.
Similarly to Casanovaa et al. 2020), we aim to annotate the unlabelled data at each step in a data- efficient manner, by labelling batches of data a time. As focused image classification, as opposed to semantic segmentation, we do not consider regions of images to be labelled, rather we wish label entire images at each step. We can therefore label N images by calculating the top-N Q-values at each step
N (at) /==1 = arg max
Σ (ss, af:
of E. = N
To stabilise training and alleviate the overestimation bias of the DQN framework, we use a target network with weights ø' using adapted DDQN architecture, to enable us work with actions represented by vectors, and the network is trained using the temporal difference (TD) error (Sutton) 1988)
Σ at E.AAl10 Q"EaT Q Qss. (::::0
Vt r +7Q 81+1.
arg max
4 EXPERIMENTS
In this section we wish to evaluate our method against different medical image datasets, which span differing imaging modalities. We will introduce the datasets we use, as well as the experimental set-up and baselines against which compare.
4.1 BASELINES
We will compare our method with several baselines. Firstly, work can be considered generalisation and extension of LAL-RL (Konyushkova et al., 2018), we compare our results against theirs. We then compare method with 'standard' active learning query strategies, implemented using the modAL package (Danka and Horvath). Namely, the baselines we compare against are
LAL-RL (Konyushkova et al. 2018), using logistic regression as the base classifier, as done in the original work.
Random sampling, where the datapoint to be labelled is picked random.
4

---

Under review as a conference paper at ICLR 2023
Margin sampling (Scheffer al. 2001). which selects the instances where the difference between the first most likely and second most likely classes are the smallest.
Entropy sampling (Shannon 1948). which selects the instances where the class probabili- ties have the largest entropy.
Uncertainty sampling (Lewis and Gale 1994) which selects the least sure instances for labelling.
Average confidence (Esuli and Sebastiani 2009 query strategy designed for multi-label classification.
For the baseline query strategies except for LAL-RL, use the same CNN classifier as the active learner as in our implementation, which a pre-trained resnet50 (He et al. 2015) model with a linear layer added ensure the correct number of labels classified for given dataset.
4.2 IMPLEMENTATION DETAILS
As we use pre-trained resnet50 CNN the classifier for our approach and those all benchmark methods except LAL-RL, we apply transforms to the images in the differing datasets consistent with the image size and normalisations required (see paper for full details, (He et al. 2015)). In order to ensure consistency across the benchmarks we apply these transformations to the data also when comparing against L.L.-RI.
To compute Q" (s, a) we use a multi-layered perceptron with 3 hidden layers and 128 neurons in each layer with ReLU activation function. We train the DDQN with a fixed learning rate of 0.0001, requiring the DDQN to train until the average reward obtained has stabilised using early stopping. As stated above, the classifier, f, the standard resnet50 CNN classifier with an additional linear layer to enable multi-label classification, which train for 200 epochs with fixed learning rate of 0.0001.
Due to the similarities between approach and those of (Konyushkova et al. 2018), all the overlap- ping reinforcement learning hyperparameters are kept consistent when running the experiments. In practice, this means that the classifier and action representations are the main parameters which changed between the two methods. Due to the very large compute resources required by LAL-RL, as will discuss in detail below, we only initialise the DDQN for both methods with 16 warm start random episodes. For each method, we show results on the accuracy of the classifier on the test set for each dataset, after having been trained on set number of datapoints. We show results for 5 different random initialisations, showing the mean and 68% confidence level intervals for all methods.
4.3 DATASETS
We evaluate our method two open source medical imaging datasets under CC BY 4.0 licence] with an aim to cover very different imaging modalities and types of data present in medical image classification tasks, and to look at the binary and multi-label classification scenarios
Pneumonia The first dataset consists of chest X-rays of paediatric patients (Kermany et al. 2018a b) with the classification task being to determine whether the patient presents with pneumonia or not.
Colorectal cancer with colorectal udenocarcinoma (Kather et al. 2016abb). The data consists of 8 differing types of tissue.
This 8-class classification problem consists of histopathology tiles from patients
4.4 RESULTS
We show in Fig. the accuracy of the differing query strategies as function of labelled datapoints for the pneumonia (left) and colorectal cancer histopathology (right) datasets. Error bands are the
https: crett vecommons. hwtpmmooonsss/////// org/1 censes/by/a 0/1ega code
The human biological samples were sourced ethically and their research use was in accord with the terms of the informed consents under IRB/EC approved protocol.
5

---

Under review as a conference paper at ICLR 2023
30.
marsh
marhh HATERS
canding
corfeence
31-
corfeeee
3.5
25-
Rester slavesiees
37-
36-
2.4
32-
18
20
30 50
26
00 30 306
0
30
76 90
330
Figure 1: Top: Accuracy on the test set as a function of labelled datapoints for the pneumonia (left) and colorectal cancer histopathology (right) dataset for the differing labelling strategies. Error bands are the 68% confidence level intervals over random initialisations of 5 seeds. Bottom: Same as before but with just the results of the two reinforcement learning approaches.
68% confidence level intervals over random initialisations of 5 seeds. As we wish to emphasise generalisation of our method over existing reinforcement learning based active learning query strategies, we plot the same results in the bottom panel for method and LAL-RL.
We observe that, regardless of query strategy, our method on average outperforms all others, impor- tantly reaching near-optical accuracy the test set for the colorectal cancer histopathology dataset with very few labelled datapoints. This suggests that, because our method is substantially more general than the other strategies, it is able better deal with the multi-class labelling task for the histopathology slides. Importantly, our method outperforms LAL-RL on the binary classification pneumonia dataset, for which LAL-RL is designed work with, whilst ours is agnostic the number of classes present in the data. The standard deviation of results both in our approach and LAL-RL are also much smaller than those of the traditional querying methods, suggesting that reinforcement learning based approach is in fact more stable with respect to random initialisations of the dataset and hyperparameters.
A fundamental difference between our approach and that of LAL-RL is how we compute the generalised action representation; i.e., how one determines the distances between the labelled and unlabelled datasets. Our approach is inspired by coreset methods (Sener and Savarese 2018). where we minimise the Euclidean distance between the sampled and unsampled data points in the latent space of the trained model. LAL--L, however, compute the pairwise distances in the space of the data, which, in the case of very high dimensional data as we have here, can become computationally intractable. Indeed, experiments show that training the colorectal cancer histopathology dataset for 5 episodes, allowing 20 datapoints to be annotated with the two reinforcement learning methods with identical overlapping hyperparameters is 267 times slower using LAL-RL than method.
4.4.1 COMMENT ON BENCHMARK RESULTS
As we mentioned above, this extremely slow performance requires us to use very lightweight models for the two reinforcement learning based approaches in order to be able benchmark between them, preventing the true predictive power either of them from being accurately represented. We therefore emphasise that the results we show in this section are only indicative of the general improvement of

---

Under review as a conference paper at ICLR 2023
30.
344
Targhianple
Haghhaateg laTg-g sereiing
sereding ertrney sormeii
3.5
Rester 21012
Figure 2: Same as in Fig. model for our method.
1 but additionally showing results for the two datasets with slightly larger
26.
35-
naghiaategg seroeigg
FEEEST seroeing
entreg sorrel 13
32-
35"
Sudder
Runfer
Figure 3: Test accuracy on the colorectal cancer histopathology histopathology dataset where training noise has been added to 10% (left) of the dataset and 100% (right).
our approach over other query strategies and by no means the best predictive power possible for our method on the two datasets we show in this work.
Indeed, by simply increasing the number of warm start episodes from 16 to 128 for our method,
so that the replay buffer is filled with a better selection of episodes from which to sample well as adding one additional datapoint per step using Eq. equation and keeping all other parameters the same, we find up to 11.5% improvement in test accuracy for the pneumonia dataset and 23.3% for the colorectal cancer histopathology dataset, as we show in Fig. 2 where we additionally plot the results for this slightly larger model. We see that in the multi-class case, the small increase in size of our DDQN drastically improves performance on the dataset, as well as greatly reducing the spread of results and is thus much more stable. We therefore conclude that method can easily outperform other active learning query strategies in terms of: predictive power, stability, and efficiency, (near-) optimal performance is found by labelling very few data points.
4.4.2 ROBUSTNESS TO NOISE
In this section we modify the training data account for noise in the data. We randomly augment the training images by randomly zooming the images, rotating them by up to = / 112 radians, and adding random Gaussian multiplicative noise using the MONAI package (MONA 2022). a medical imaging deep learning package which provides image transforms of relevance to differing image modalities. We ran experiments for the aforementioned noise applied to 10% of the training set as well to 50% and 100%, to cover range of eventualities.
In Fig. 3 we again show the test accuracy on the colorectal cancer histopathology dataset for the differing query strategies where training noise has been added to 10% (left) of the dataset and 100% (right). It is clear that all query strategies are, to an extent, robust to noise the data, which suggests that there is overfitting in any of the cases. We quantify this in Table I where we show the accuracy on the test set for differing query strategies after labelling 10 datapoints with differing levels noise added to the training data and in all cases the mean accuracy is consistent within the error bands.
7

---

Under review as a conference paper at ICLR 2023
Strmlogy
No noise
10%
100%
Ous
0.0209 0.0051
0.6348 =: 0.09:55
0.6534 =: 0. 0.0226
LAA-RR
0.3151 0.0238
0.3143 = 0.0633
0.2978 = 0. 0.0463
Uncastinty sampling
0.4952
0.4813
0.4800
0.00670
Random sampling
0.4698
0.0340
0.4516 2: 0385
0.4552
0697
Mirgin sampling
0.5063 1: 0.0204
0.5111 = 04.5
0.4849
= .628
Entropy sampling.
0.4560 0.0168
0.5020 0.0673
0.5067
0.0689
Average confidence
1.00731
Table Accuracy on the test set for the colorectal cancer histopathology dataset for differing query strategies after labelling 10 datapoints with differing levels of noise added to the training data.
6688
10
Figure 4: Accuracy on the test set as a function of the number of datapoints labelled at each step, corresponding to the top-N Q-values determined by the reinforcement learning algorithm.
4.4.3 PICKING N-DATAPOINTS AT A TIME
As we discuss above, one of the main advantages of our method is the ability to label multiple datapoints at each stage the active learning cycle. We do this by labelling the top-N Q-values at each step, following Eq. equation 3 It is natural to consider several consequences of labelling data in this way. Firstly, by labelling lower-ranked datapoints each step, one may ask whether the quality of prediction of the model deteriorates as increase N. Secondly, does the model require any additional time to perform these additional labellings, as the size the training set now allowed to increase.
To answer these questions, we ran our DDQN model on the colorectal cancer histopathology dataset, training for fixed number of episodes and allowing 50 total label annotations each episode. At cach step, we allow the agent to label from (1..10} datapoints, and as before, compute the accuracy on the test set. We chose the colorectal cancer histopathology dataset here the contrast between the other reinforcement learning informed query strategy, LAL-RL, most pronounced. We show in Fig. 4 the mean and 68% confidence level intervals, averaged over 5 seeds, of this procedure. is clear that labelling more than datapoint per step can greatly improve the prediction accuracy of the model, suggesting our prior results in Sec. 4.4 are conservative with respect the potential prediction power of our model. Beyond allowing N > I, however, within the eтror bands as reported, it is not obvious that labelling more data at each step this dataset helps (or indeed, hinders), prediction accuracy.
The training time for the differing values of N was also computed, and we show the test accuracies for the differing values N shown in Fig. 4 alongside the average time for the training in seconds. We can see that the training time does not begin to deteriorate until N >> 8, which we can does not even come at the gain of improving test accuracy for this dataset. is therefore reasonable to consider 1 < N < 9 to be logical choice of datapoints to label in this experiment.
5 DISCUSSION
In this paper we have introduced novel method for active learning query strategies, inspired by previous work which reformulates the active learning cycle as a Markov decision process. Similarly to previous work (Konyushkova et al. 2018), use deep reinforcement learning approach to cast the task of labelling data as actions for an agent to pick in an environment. Unlike previous reinforcement learning based active learning query strategies, however, aim present much
8

---

Under review as a conference paper at ICLR 2023
Test acc
Time (sccondss
0.6468 = 0.0197
1511
2 0.7147 0.0220
1633
0.7151 0.0307
4
0,6762 0. 0272
3313
5
0.7295 0.0168
2123
6
0.7091 0.0185
1222
0.0521
8
0.6016 0200
1290
9
0.7084 0.0148
3374
0,6939 0.0206
6504
Table 2: Test accuracy and training time for our model labelling N datapoints at each step.
more general approach to active learning, by enabling multi-label prediction as opposed to just binary classification. In addition, by enabling the agent to pick an arbitrary number of datapoints each step in the MDP, our approach can easily be applied to segmentation tasks where we would like to be able to label multiple pixels at each step, such in (Casanova al. 2020).
We have applied our method to the task of medical image classification, in the binary and multi- label class case, and across differing image modalities. We have shown that, even with extremely conservative estimates, our approach outperforms both standard query strategies, as well as the current state of the reinforcement learning based approach, LAL-RL. Importantly, our method has been shown to be much more computationally efficient, as optimal accuracy on the test set can be reached with very few labelled data points, and by sampling the data in the latent space of our classifier, requires much less compute resources than LAL-RL. This is a very relevant result in the space of medical imaging, as, depending on the imaging data, one may be limited by the number of samples one can obtain but each datapoint is very high-ddmensional. In the healthcare domain, high-d, low-n datasets are abundant, whilst traditional active learning- haaning-based approaches generally assume one have access large pool of unlabelled data.
Both LAL-RL and our method aim to answer the same question; namely, can we create an active learning framework which contains minimal bias in how data is labelled at each stage? We take this one step further and consider; can we create an active learning framework which can do this task but apply it to the deep learning paradigm of active learning? In this paradigm, ourselves to binary classification tasks, we wish to be able to add large batches of data at each step as opposed just one datapoint, and we also need to be able deal with high-dimensional data. We solve all these problems in our approach: in the first instance, we can clearly outperform LAL-RL on an eight-laber classification task. In the second instance, by ranking the top-N Q-values at each step, we show we obtain good performance (at improved accuracy and minimal loss of speed), and finally, by using a coreset-inspired approach for determining the action representation, we are much more able to deal with very high dimensional data.
do not wish to limit
5.1 OUTLOOK
There are several avenues for exploration to extend this work. We mention above that image segmentation would be an interesting application of our approach: indeed, although here we just consider image classification, our framework is task-agnostic and can in theory be applied to arbitrary active learning cycles. As we discuss above, although we have empirically seen that our coreset- inspired approach to ranking Q-values works well, it known that this method breaks down in the multi-label regime, and so determining heuristic which is more efficient for multi-label classification would be importance to improve performance of our method.
REFERENCES
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. CoRR, 2019.
Philip Bachman, Alessandro Sordoni, and Adam Trischler. Learning algorithms for active learning. CoRR, 2017.

---

Under review as a conference paper at ICLR 2023
William H. Beluch, Tim Genewein, Andreas Nurnberger, and Jan M. Kohler. The power of ensembles for active learning in image classification. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9368-9366 9377, 2018.
Mustafa Bilgic and Lise Getoor. Link-based active learning. 01 2009.
Thiago N.C. Cardoso, Rodrigo M. Silva, Sérgio Canuto, Mirella M. Moro, and Marcos A. Gonçalves. Ranked batch-mode active learning. Information Sciences, 379:313-337, 2017. ISSN 0020-0255.
Arantxa Casanova, Pedro O. Pinheiro, Negar Rostamzadeh, and Christopher Pal. Reinforced active learning for image segmentation. CoRR, 2020.
Hong-Min Chu and Hsuan- Tien Lin. Can active learning experience be transferred? In 2016 IEEE 16th International Conference on Duta Mining (ICDM), pages 841-846, 2016.
Ido Dagan and Sean P. Engelson. Committec-based sampling for training probabilistic classifiers, In Armand Prieditis and Stuart Russell, editors, Machine Learning Proceedings 1995, pages 150-157. Morgan Kaufmann, San Francisco (CA), 1995. ISBN 978-1-55860-377-6
Tivadar Danka and Peter Horvath. modAL: modular active learning framework for Python. URL https://githin www.o///oooooooooooooooeoeeeeeee c-ott/////ccotttteeo//o.oo modAL
Sandra Ebert, Mario Fritz, and Bert Schiele. Ralf: A reinforced active learning formulation for object class recognition. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3626-36666 3633,,633,20 2012.
Andrea Esuli and Fabrizio Sebastiani. Active learning strategies for multi-label text classification. pages 102-113, 04 2009. ISBN 978-3-642-00957-44
Meng Fang, Yuan Li, and Trevor Cohn. Learning how active learn: A deep reinforcement learning approach. CoRR, 2017.
Alexander Freytag, Erik Rodner, and Joachim Denzler. Selecting influential examples: Active learning with expected model output changes. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014, pages 562-577, Cham, 2014. Springer International Publishing.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning, 2015.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. CoRR, 2017.
Manuel Haussmann, Fred A. Hamprecht, and Melih Kandemir. Deep active learning with adaptive acquisition, 2019.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, 2015.
Wei-Ning Hsu and Hsuan disuan-Tien Lin. Active learning by learning. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1), Feb. 2015.
Jakob Nikolas Kather, Cleo-Aron Weis, Francesco Bianconi, Susanne M. Melchers, Lothar R. Schad, Timo Gaiser, Alexander Marx, and Frank Gerrit Zöllner. Multi-class texture analysis in colorectal cancer histology. Scientific reports, 187988-279988 Jun 2016a. ISSN 2045-2322. doi: 10.1038/ssep27988..
Jakob Nikolas Kather, Frank Gerrit Zöllner. Francesco Bianconi, Susamne M Melchers, Lothar R Schad, Timo Gaiser, Alexander Marx, and Cleo-Aron Weis. Collection of textures colorectal cancer histology, May 2016b.
10

---

Under review as a conference paper at ICLR 2023
Daniel S. Kermany, Michael Goldbaum, Wenjia Cai, Carolina C.S. Valentim, Huiying Liang, Sally L. Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, Justin Dong, Made K. Prasadha, Jacqueline Pei. Magdalene Y.L. Ting, Jie Zhu, Christina Li, Sierra Hewett, Jason Dong, Ian Ziyar, Alexander Shi, Runze Zhang, Lianghong Zheng, Rui Hou, William Shi, Xin Fu, Yaou Duan, Viet A.N. Huu, Cindy Wen, Edward D. Zhang, Charlotte I.. Zhang, Oulan Li, Xiaobo Wang. Michael A. Singer, Xiaodong Sun, Jie Xu, Ali Tafreshi, M. Anthony Lewis, Huimin Xia, and Kang Zhang. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell. 172(5):1122-1115 1131.e9, 2018: 8a. ISSN 0092-8674.
Daniel S. Kermany, Kang Zhang, and Michael H. Goldbaum. Labeled optical coherence tomography (oct) and chest x-ray images for classification. 2018b.
KrishnaTeja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh K. Iyer. GLISTER: generalization based data subset selection for efficient and robust learning. CoRR, 2020.
Kwan- Young Kim, Dongwon Park, Kwang In Kim, and Se Young Chun. Task-aware variational adversarial active learning. CoRR, 2020.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. CoRR, 2019.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Geometry in active learning for binary and multi-class image segmentation. CoRR, 2016.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from real and synthetic data. CoRR, 2017a
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017b.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Discovering general general-purpose purpose active learning strategies. CoRR, 2018.
F. Last, Tassilo Klein, Mahdyar Ravanbakhsh, Moin Nabi, K. Batmanghelich, and Volker Tresp. nluman-maching collaboration for medical image segmentation. ICASSP 2020 2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1040-1044, 2020.
David D. Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. pages 148 156, 1994.
David D. Lewis and William A. Gale. A sequential algorithm for training text classifiers. CoRR, 1994.
Ming Liu, Wray Buntine, and Gholamreza Haffari. Learning how to actively learn: A deep imitation learning approach. In Proceedings of the 56th Annual Meeting of the Association for Computa- fional Linguistics (Volume 1: Long Pupers), pages 1874-1883, Melbourne, Australia, July 2018. Association for Computational Linguistics.
Zimo Liu, Jingya Wang, Shaogang Gong, Dacheng Tao, and Huchuan Lu. Deep reinforcement active learning for Conference Computer Vision (ICCV), pages 6121-6130, 2019.
person re-identification. In 2019 IEEE/CVF International
Chengjiang Long and Gang Hua. Multi-class multi-annotater active learning with robust gaussian process for visual recognition. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2839-28477 2015.
11

---

Under review as a conference paper at ICLR 2023
Dwarikanath Mahapatra, Behzad Bozorgtabar, Jean-Philippe Thiran, and Mauricio Reyes. Efficient active learning for image classification and segmentation using a sample selection and conditional generative adversarial network. In Alejandro F. Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-I and Gabor Fichtinger, editors, Medical Image Computing and Computer Assisted Intervention MICCAI 2018, pages 580-588, Cham, 2018. Springer International Pub- lishing. ISBN 978-3-033-00093422
Baharan Mirzasoleiman, Jeff A. Bilmes, and Jure Leskovec. Data sketching for faster training of machine learning models. CoRR, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-5333 -333, February 2015. ISSN 00280836.
MONA. Monai: Medical open network for ai, February 2022. URL https://doi.org:
5281/zenodo. 6114127,
Hieu T. Nguyen and Amold Smeulders. Active learning using pre-clustering. In Proceedings of the Twenty-Fite International Conference on Machine Learning. ICML '04, page 79, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581 138385.
Firat Ozdemir, Zixuan Peng, Christine Tanner, Philipp Fuernstahl, and Oreun Goksel. Active learning for segmentation by optimizing content information for maximal entropy. In Deep Learning in Medicul Image Analysis and Multimodal Learning for Clinical Decision Support, pages 183-191, Cham, 2018. Springer International Publishing. ISBN 978-3-030-00889-5.
Aishwarya Padmakumar, Peter Stone, and Raymond Mooney. Learning policy for opportunistic active learning. CoRR, 2018.
Kunkun Pang, Mingzhi Dong, Yang Wu, and Timothy M. Hospedales. Meta-learning transferable active learning policies by deep reinforcement learning. CoRR, 2018.
Nicholas Roy and Andrew Mecallum. Toward optimal active learning through monte carlo estimation of error reduction. 01 2001.
Tobias Scheffer, Christian Decomain, and Stefan Wrobel. Active hidden markov models for infor- mation extraction. IDA '01, page 309-318, Berlin, Heidelberg, 2001. Springer-Vertee ISBN 3540425810.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018
H. S. Seung, M. Opper, and H. Sompolinsky. Query by committee, 1992.
C. E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27(3): 379-423, 1948.
Yanyao Shen, Hyokun Yun, Zachary C. Lipton, Yakov Kronrod, and Animashree Anandkumar. Deep active learning for named entity recognition. CoRR, 2017.
Changjian Shui, Fan Zhou, Christian Gagné, and Boyu Wang. Deep active learning: Unified and principled method for query and training. CoRR, 2019.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. CoRR, 2019.
Asim Smailagic, Pedro Costa, Hae Young Noh, Devesh Walawalkar, Kartik Khandelwal, Adrian Galdran, Mostafa Mirshekari, Jonathon Fagert, Susu Xu, Pei Zhang, and Aurélio J. C. Campilho. Medal: Accurate and robust deep active learning for medical image analysis. 2018 17th IEEE International Conference Machine Learning and Applications (ICMIA), pages 481 1888 2018.
12

---

Under review as a conference paper at ICLR 2023
Akshay Smit, Damir Vrabac, Yujie He, Andrew Y. Ng, Andrew L. Beam, and Pranav Rajpurkar. Medselect: Selective labeling for medical image classification combining meta-learning with deep reinforcement learning. CoRR, 2021.
Richard S. Sutton. Learning to predict by the methods of temporal differences. Mach. Learn., 3(1):
9-44, aug 1988. ISSN 0885-6125.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double learning. CoRR, 2015.
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1954-1963, Lille, France, 07-09 Jul 2015. PMLR.
Si Wen, Tahsin Kure, Le Hou, Joel Saltz, Rajarsi Gupta, Rebecca Batiste, Tianhao Zhao, Vu Nguyen, Dimitris Samaras, and Wei Zhu. Comparison of different classifiers with active learning to support quality control in nucleus segmentation in pathology images. AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science, 2017:227-2366 05 2018.
Colin White, Mikhail Khodak, Renbo Tu, Shital Shah, Sébastien Bubeck, and Debadeepta Dey. A deeper look at zero-cost proxies for lightweight nas, ICLR Blog Track, 2022. URL https:// https://iclr-
Mark Woodward and Chelsea Finn. Active one-shot learning. CoRR, 2017.
Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z. Chen. Suggestive annotation: A deep active learning framework for biomedical image segmentation. CoRR, 2017.
Beichen Zhang, Liang Li, Shijie Yang, Shuhui Wang, Zheng-Jun Zha, and Qingming Huang. State- relabeling adversarial active learning. CoRR, 2020.
Amy Zhao, Guha Balakrishnan, Frédo Durand, John V. Guttag, and Adrian V. Dalca. Data augmenta- tion using learned transforms for one-shot medical image segmentation. CoRR, 2019.
Fedor Zhdanov. Diverse mini-bated batch active learning, 2019.
13


Reviews:
[{"review_id": "sx192SEgPlW", "reviewer": "Reviewer_CZBN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose deep-Q learning to learn an active learning strategy for medical image classification using deep learning models. The authors propose to add a geometric distance in the embedding space to the actions and then select the top-n Q values to compose the batch in bool-based batch-mode active learning. They claim that this (action/problem) formulation leads to uncertain and diverse batches.", "summary_of_the_review": "This review considers the claims wrt. the conducted experiments and uncited related work. The experiments are not supporting the claims, and the there is no theoretical justification for them either. Furthermore, the novelty is doubtful given prior publications, and there are many other issues with the submission.", "strength_and_weaknesses": "**Strengths:**\n- The paper is (at least partially) well-written and easy to \n- The problem is interesting and relevant to the community\n\n**Weaknesses**:\n- Novelty is weak, the top-n operator was previously proposed for deep active learning [1] and also being trained within an RL-style framework\n- The mathematical notation is inconsistent\n- Several central claims are not supported by the experiments: (i) policy learning diversity sampling (I doubt this, as I cannot see this in the introduced methodology), (ii) that the learnt acquisition function can be transferred between datasets, (iii) that only little computational resources are required in relation to current approaches.\n- The experimental study has several weaknesses (see more details below).\n\n\n****Methodology****\n\n- It is not clear how you minimize the Euclidean distance, as you say in Section 4.4 (last paragraph). Isnt it just a part of the action \n- The paper introduces the MDP using a four-tuple $\\{(s_t, a_t, r_{t+1}, s_{t+1})\\}$ – usually that is not correct as an MDP is defined over a set and not over concrete time-step samples, i.e., { $\\mathcal{S},\\mathcal{A},\\mathcal{R},[\\mathcal{P}], \\gamma$ }, where $\\mathcal{S}$ is a set of states, $\\mathcal{A}$ a set of possible actions, $\\mathcal{R}$ a reward function, $\\mathcal{P}$ a probability matrix that describes the system dynamics, and $\\gamma$ the discount factor. Elements such as $s_t$ or $a_t$ are concrete elements from those sets at particular time-steps but they do not define the MDP. \n- “5. Train f_t one iteration on the new labelled dataset” – what do you mean with one iteration? One epoch? If not: then I doubt that we see a full RL setting here. Does it make a difference if we label datapoint A first and then B versus labeling B first and then A?\n- Equ. 1 is incorrect: $\\gamma$ should be to the power of $i$. How is $i$ bounded? (I guess over the length of the episode, but there is a lot of notation being inconsistent here…)\n- $Q^{\\pi\\star}$ $\\rightarrow$ $Q^{\\pi^\\star}$\n- “The optimal value is the $Q^{\\pi^\\star} (s,a)$ = $max_\\pi Q^\\pi (s,a)$ – please refer to standard RL literature to clear abuse of notation\n- State representation: I have two questions on that: (1) Don’t you include any classifier-model-specific parameters into the state? If not, I doubt that the model explicitly learns diversity-based strategies…, (2) The state representation seems to be a bit misaligned and too minimal. I guess you formulate a POMDP here.\n- Sec. 3.1.2 was unclear to me for quite some time and only became clear after iterating together with Section 3.2. The authors should rewrite this part to make it more accessible and more intuitive.\n- Equation 4 is unclear. The argmax-operator as defined (although I am not sure if this is common-sense to do it like this) returns a set of actions but it is used as a scalar for computation. This is undefined. Or do you use the set over A as a proxy for actions? But then it is inconsistent with the description in 3.1.2\n\n**Experimental results**\n\n- Claims of superior performance should be better supported with suitable stronger baseline methods in experiments section (Coreset [Sener 2018], Ensemble [Beluch 2018], MC-Dropout [Gal 2017], IALE [Loeffler 2022], BADGE [Ash 2020 (please see the final reference paper instead of your cited preprint], BatchBALD [Kirsch2019, please see the final reference paper instead of your cited preprint], ...).\n- No ablation study of the differences between Konyushkova et al. (2018) and the proposed method. That would be interesting to see.\n- Why do you show only the 68% confidence interval for your results?\n- Please define what N is in the main experiments (I guess it is 4)\n- Figure 1ff: the accuracy at point 0 (annotions = 0) seem to be wrong. Is it the performance of the classifier after the warm-up? (then, I guess it has already seen 64 samples (16 episodes à 4 samples). Shouldn’t the graphs start at 64? At x=10 the classifier has seen 74 samples, right? (64 random + 10 selected by heuristic)\n- To make your method more comparable, how does your method perform compared to the baselines on commonly used image classification datasets such as CIFAR10/100, SVHN, and MNIST variants, with commonly used classifier architectures (ResNet18 etc)?\n- The presentation of results in Figs. 1-3 is unintuitive and there is little to no benefit to active learning visible. Why is that? I would also expect to run AL at least for 1K annotations to see how things evolve…\n- How do the warm-start episodes exactly work? This seems important, can you perform a study on the effect and cost of it?\n- The experiments on batch-sizes between 1 and 10 have several issues:\n   - Is each datapoint in Figure 4 a separate run of the policy from 0 to 50 samples? If yes, why is the graph in Fig. 4 connecting the separate runs as if they are continuous? I’d like to see each run from 0 to 50 over each other in one figure.\n   - What is the warmup?\n   - Training time measurements seem noisy between 1222 and 3374, the N=10 datapoint seems like an outlier from this, how does it perform for N=20, 50, 100, 1000?\n   -Why is 1 worse than the others?\n- To me runtime or computation time seems to be a real issue. In fact the authors also mention that throughout the paper but there is no results or analysis on that.\n- Section 4.2.2: to me this section is a bit misleading. I do not see why this aspect seems to be important here.\n- What about model transfer, i.e, learning the policy on one dataset and the applying it to another?\n\n**Some minor points to the experiments**\n- How do you differentiate from MedSelect?\n- The results in Fig. 1,2,3 do not present the results well, due to the point at 0 labeled samples that can be removed. Also, no sub-figs used there.\n- Table 1: it would help to highlight the best-performing method\n\n**Missing references and prior work & updated citations**\n- [Loeffler 2022] Christoffer Loeffler and Christopher Mutschler: “IALE: Imitating Active Learner Ensembles”. In: Journal of Machine Learning Research 23 (2022) 1-29\n- [Ash 2020] Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations (ICLR), 2020. \n- [Kirsch 2019] Andreas Kirsch, Joost van Amersfoort, Yarin Gal. BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning. NeurIPS, 2019. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "FMM0n5kYg_", "reviewer": "Reviewer_6vUj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a reinforcement learning framework for active learning for deep learning where the examples to be annotated are selected by a policy. The policy is learned by a DQN to maximize (discounted) reward. The paper seems to be built upon works of Konyushkova et al. 2017b, 2018.", "summary_of_the_review": "Although the proposed method his promising, I do not think the current manuscript is ready for publication at a major venue yet. The paper needs to clarify its novelty and conduct extensive experiments to show its merit on real data (preferably natural images).", "strength_and_weaknesses": "A principled framework, that is also learnable, for active learning would be very useful for active learning in my opinion.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "17ADsi5QfZ", "reviewer": "Reviewer_tpMj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a deep learning based reinforcement active learning method for image classification. The approach uses the deep-Q network for learning to pick informative data points. The contributions are a method that allows for the selection of multiple data points in each round, to be more amenable to batch based optimization. Points that maximize uncertainty and diversity is prioritized. The approach is evaluated on two datasets, one containing chest X-rays with the goal of classifying pneumonia and the other being histopathology images of patients with colorectal cancer.\n", "summary_of_the_review": "I am really not sure what to make of this work. It could be good with more work, but as it is, there are just too many uncertain aspects.", "strength_and_weaknesses": "Strengths\n---------\n- The paper addresses the important and relevant topic of how to minimize the amount of labelled data needed for training deep learning based image classification methods.\n\n- The ideas pursued in the work make sense and the particular combination of methods and problem has not been seen before as far as I know.\n\n- The method is compared to a good number of reference methods and seems to improve on them.\n\nWeaknesses\n----------\n- The two datasets does not seem particularly well chosen for this experiment. The Pneumonia being worse than the Colorectal cancer dataset, but both seem to converge already after 10 added \"annotations\". As this appears to be the first point on the plot with information, it does not illustrate the convergence properties of the different methods well. As the iterations after the first 10 \"annotations\" do not seem to improve results, I am wondering what the role of the reinforcement learning aspect is and how much the improvement in performance of the proposed method is simply due to the implemented combination of uncertainty and diversity based sampling.\n\n- I find the novelty to be a bit weak. Casanova et al., 2020 (referenced in paper) already explored very similar techniques for segmentation. An experiment with classification and slightly different approach is a valid contribution if the experimentation was done right, but I am not sure what to conclude from the results I see and the authors comments are not helpful (see comment above).\n\n- Why does the test accuracy seem to have a lower bound at 0.6 for the pneumonia dataset?\n\n- The paragraphs at the bottom of page 3, and equations 1 and 2 are a bit unclear to me. Could you elaborate and provide some intution for the content of these equations?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject