Reinforced active learning for image segmentation
Authors: Arantxa Casanova
Pedro O. Pinheiro
Negar Rostamzadeh
Christopher J. Pal
Venue:  (2020)
Keywords: semantic segmentation
active learning
reinforcement learning

Abstract: Learning-based approaches for semantic segmentation have two inherent challenges. First, acquiring pixel-wise labels is expensive and time-consuming. Second, realistic segmentation datasets are highly unbalanced: some categories are much more abundant than others, biasing the performance to the most represented ones. In this paper, we are interested in focusing human labelling effort on a small subset of a larger pool of data, minimizing this effort while maximizing performance of a segmentation model on a hold-out set. We present a new active learning strategy for semantic segmentation based on deep reinforcement learning (RL). An agent learns a policy to select a subset of small informative image regions -- opposed to entire images -- to be labeled, from a pool of unlabeled data. The region selection decision is made based on predictions and uncertainties of the segmentation model being trained. Our method proposes a new modification of the deep Q-network (DQN) formulation for active learning, adapting it to the large-scale nature of semantic segmentation problems. We test the proof of concept in CamVid and provide results in the large-scale dataset Cityscapes. On Cityscapes, our deep RL region-based DQN approach requires roughly 30% less additional labeled data than our most competitive baseline to reach the same performance. Moreover, we find that our method asks for more labels of under-represented categories compared to the baselines, improving their performance and helping to mitigate class imbalance.

Published as a conference paper at ICLR 2020
REINFORCED ACTIVE LEARNING FOR IMAGE SEGMENTATION
Arantxa Casanova"
Pedro O. Pinheiro
Negar Rostamzadeh ElementAI
Ecole Polytechnique de Montréal Mila, Quebec Artificial Intelligence Institute ElementAI
ElementAI
Christopher J. Pal
École Polytechnique de Montréal Mila, Quebec Artificial Intelligence Institute ElementAI
ABSTRACT
Learning-based approaches for semantic segmentation have two inherent chal- lenges. First, acquiring pixel-wise labels is expensive and time-consuming. Second, realistic segmentation datasets highly unbalanced: some categories much more abundant than others, biasing the performance to the most represented ones. In this paper, we interested in focusing human labelling effort small subset of a larger pool of data, minimizing this effort while maximizing performance of a segmentation model a hold-out set. We present new active learning strategy for semantic segmentation based on deep reinforcement learning (RL). An agent learns policy to select a subset of small informative image regions opposed entire images to be labeled, from pool of unlabeled data. The region selection decision made based on predictions and uncertainties of the segmentation model being trained. Our method proposes new modification of the deep Q-network (DQN) formulation for active learning, adapting it to the large-scale nature semantic segmentation problems. We test the proof of concept in Cam Vid and provide results in the large-scale dataset Cityscapes. On Cityscapes, our deep RL region-based DQN approach requires roughly 30% less additional labeled data than our most competitive baseline to reach the same performance. Moreover, we find that our method asks for more labels of under-represented categories compared to the baselines, improving their performance and helping to mitigate class imbalance.
INTRODUCTION
Semantic segmentation, the task of labelling an image pixel-by-pixel with the category it belongs to. is critical for a variety of applications such as autonomous driving (Müller et al., 2018 Wang & Pan 2018), robot manipulation (Schwarz et al. 2018), embodied question answering
et 2019) and
biomedical image analysis (Ronneberger et al. 2015). Convolutional neural networks (Lecun et al. bbse)d methods have achieved excellent results on large-scale supervised semantic segmentation, in which we assume pixel-level annotations 2014 Long et al. 2015). For such models to work, however, they need large amount of level-level annotations that may require costly human labor (Cordts et al. 2016 Bearman al., 2016).
available (Farabet et al.) 2013 Pinheiro & Collobert
Current semantic segmentation datasets have pixel-wise annotations for each image. This standard approach has two important issues: (i) pixel-level labelling extremely time consuming. For example, annotation and quality control required more than 1.5h per image (on average) on Cityscapes (Cordis) al. 2016), popular dataset used for benchmarking semantic segmentation methods. (ii) Class imbalance in the data is typically extreme. Certain categories (such as 'building' or 'sky') can appear
"Wor done while interning ElementAT

---

Published as a conference paper at ICLR 2020
Figure I: (Left) Input image from Cityscapes dataset (Cordts al. 2016), with selected regions by our method to be labeled. (Right) Retrieved ground truth annotation for the selected regions. Our method focuses on small objects and under-representedd classes, such as bicycles. pedestrians and poles. Best viewed in color.
with two orders of magnitude more frequently than others (e.g. 'pedestrian' or 'bicycle'). This can lead to undesired biases and performance properties for learned models.
This is specially relevant when we want collect annotated data with human in the loop create a new dataset or to add more labeled data to an existing one. We can tackle the aforementioned problems by selecting, in an efficient and effective way, which regions of the images should be labeled next. Active learning (AL) is a well-established field that studies precisely this: selecting the most informative samples to label so that a learning algorithm will perform better with less data than a non-selective approach, such as labelling the entire collection of data. Active learning methods can be roughly divided in two groups: (i) methods that combine different namually-desighe
AL strategies (Roy & McCallum] 2001 Osugi et al., 2005; Gal et al. 2017: Baram et al.) [2004]
Chu & Lin 2016, Hsu & Lin| 2015 Ebert et al. 2012 Long Hua 2015) and (ii) data-driven
AL approaches (Bachman et al. [017]] Fang al. 2017 Konyushkova et al. 2017 Woodward &
Finn 2016 Ravi & Larochelle| 2018]] Konyushkova et al. 2018). that learn which samples are most informative to train model using information of the model itself. Although label acquisition for
semantic segmentation is more costly and time consuming than image classification, there has been considerably less work in active learning for semantic segmentation (Dutt Jain & Grauman 2016
Mackowiak et al., 2018 Vezhnevets et al. 2012 Konyushkova et al. 2015 Gorriz et al. 2017
Yang
al.
2017). and they focus on hand-crafted strategies.
Current AL techniques that use reinforcement learning (Konyushkova et al. 2018 Fang et al. 2017
Woodward & Finn 2016, Pang et al./ [2188] Padmakumar et al., 20188] (0018//Bachae Bachman et al.) 2017 focus on labelling one sample per step until a budget of labels is met. In semantic segmentation, this would translate into labelling a single region per step. This is highly inefficient, since each step involves updating the segmentation network and computing the rewards. In this work, we propose an end-to-end method learn an active learning strategy for semantic segmentation with reinforcement learning by directly maximizing the performance metric we care about, Intersection over Union (IoU). We aim at learning a policy from the data that finds the most informative regions on a set of unlabeled images and asks for its labels, such that segmentation network can achieve high-quality performance with a minimum number of labeled pixels. Selecting regions, instead of entire images, allows the algorithm to focus on the most relevant parts of the images, as shown in Figure TT Although class imbalance in segmentation datasets has been previously addressed in (Badrinarayanaa et al.) Chan et al. 2019 Sudre et al.) 2017), among others, they try to solve a problem that arises from the data collection process. We show that our proposed method can help mitigate the problem at its source, i.e. in the data annotation itself. Because our method maximizes the mean IoU per class, indirectly learns to ask for more labels of regions with under-represed inder-represented classes, compared to the baselines. Moreover, we propose and explore a vatch-mode active learning approach that uses an adapted DQN to efficiently chose batches of regions for labelling at each step.
2hi7
To the best of our knowledge, all current approaches for active learning in semantic segmentation rely on hand-crafted active learning heuristics. However, learning a labelling policy from the data could allow the query agent to ask for labeled data as a function of the data characteristics and class imbalances, that may vary between datasets. Our main contributions can be summarized as follows: (i) we learn a RL-based acquisition function for region-basee active learning for segmentation, (ii) we formulate active learning framework with batch-mode DQN, which labels multiple regions in parallel at each active learning iteration (a more efficient strategy for large-scale datasets that compatible with standard mini-batch gradient descent), and (iii) test the proof of concept
2

---

Published as a conference paper at ICLR 2020
in CimVid (Brostow al. 2008) dataset and provide results in Cityscapes (Cordts et al. 2016)
dataset, beating& a recent state-of of-the-art technique known as BALD (Gal et al. 2017). 7. a widely used untropy-based selection criterion and uniform sampling baselines.
RELATED WORK
Active learning. Traditional active learning techniques focus estimating the sample informative-
ness using hand-crafted heuristics derived from sample uncertainty: employing entropy (Shannon
1948). query-by-committe
(Dagan & Engelson 1995 Shannon 1948 Freund et al.) 1993), max-
imizing the error reduction Roy & McCallum 2001 I. disagreement between experts (Dagan &
Engelson| 1995| Freund al.
1993) or Bayesian methods that need estimate the posterior distri-
Gal et al., 2017). Some approaches combine different techniques to trade-offs (Osugi et al.) 2005), on a bandit formulation (Baram et al. 2004, Chu & Lin 2016 Hsu & Lin| 2015)) and on
bution (Houlsby et al. [201 lap
improve AL performance. For instance, relying on applorationeexploiante
reinforcement learning (Hbert et al., 2012 Long & Hual 2015). However, these methods are still limited in the sense that they combine hand-crafted strategies instead of learning new ones. More recent active learning methods rely on an acquisition function that estimates the sample informa- tiveness with a learned metric. Konyushkova et al. (2017) estimate the error reduction of labelling particular sample, choosing the ones that maximize the error reduction. Wang et al. (2017) introduce cost-effective approach that also uses confident predictions as pseudo ground truth labels.
AL with reinforcement learning, Recently, reinforcement learning has gained attention as method to learn labelling policy that directly maximizes the learning algorithm performance. For in- stance, Liu et al. (2018): [Bachman et al. (2017) leverage expert knowledge from oracle policies to learn labelling policy, and Pang al. (2118); Padmakumar al. (2018) rely on policy gradient methods to learn the acquisition function. In different approach, some methods gather all labeled data in one big step. In Contardo et al. (017)) all samples are chosen in one step with a bi-directional RNN for the task of one-shot learning. In Sener & Savarese (2018), they propose to select batch of representative samples that maximize the coverage of the entire unlabeled set. However, the bounded core-set loss used tends to perform worse when the number of classes grows.
More similar to our approach, some prior works propose to learn the acquisition function with a Deep Q-Network (DQN) (Mnih et al. (013) formulation. These works have examined both freeam-based active learning (Fang et al. 2017;; Woodward Finn, 2016), where unlabeled samples are provided one by one, and the decision is to label it or not, and pool-based active learning (Konyushkova et al.) 2018), where all the unlabeled data is provided beforehand, and the decision is later taken on which samples to choose. The work of Konyushkova et al. (2018) is the closest to ours. Similar to them, method also leverages the benefits of Q-learning ( (Watkins & Dayan 1992) to tackle pool- based AL. Contrary them, deal with much more complex problem: semantic segmentation versus simple classification on UCI repository (Dua & Graff] (017). The large-scale nature of the problem requires us use very different definition of actions, states and rewards. Moreover, we need adapt the DQN formulation to allow the problem to be computationally feasible.
AL for semantic segmentation. Active learning for semantic segmentation has been relatively less explored than other tasks, potentially due to its large-scale nature. For instance, Dutt Jain & Grauman (2016) combine metries (defined on hand-crafted heuristics) that encourage the diversity and representativeness of labeled samples. Some rely on unsupervised superpixel-based over- segmentation (Vezhnevets et al., 2012 Konyushkova al., 2015) - and highly depend on the quality of the super-pixel segmentation. Others focus on foreground Yoreground-backgroundd segmentation of biomedical images (Gorriz et al. 2017; Yang et al. 20017), also using hand-crafted heuristics. /Settles et al. (2008); Vijyanarasimhan & Grauman (2009); Mackowiak et al. (2018) focus on cost-efrective approaches, proposing -amuullly-deiigan acquisition functions based on the cost of labeling images or regions of images. However, this information is not always given, restricting their applicability.
Mackowiak et al. (2018) focus on cost-effective approaches, where the cost of labeling an image is not considered equal for all images. Similar our work, they use region-based approach to cope with the large number of samples on a segmentation dataset. Contrary to us, their labelling strategy is based on manually defined heuristies, limiting the representability of the acquisition function. To the best of our knowledge, our work is the first to apply data-driven RL-based approach to the problem of active learning for semantic segmentation.
3

---

Published as a conference paper at ICLR 2020
3 METHOD
We are interested in selecting a small number of regions cropped from images in the original dataset) from a large unlabeled set maximize the performance of segmentation network f. parameterized by 8. This process is done iteratively until given budget B of labeled samples is achieved. At each iteration f, a query network x. parameterized by ø. selects K regions to be labeled by an oracle from large unlabeled set U4 These samples are added to the labeled set L,, that is used to train the segmentation network /. The performance is measured with a standard semantic segmentation metric, Intersection-over-iiee
(IoU).
We cast the AL problem within Markov decision process (MDP) formulation, inspired by other work such as (Padmakumar et al. 2018 Fang et al., 2017 Bachman et al. 2017: Pang al. [0188] Konyushkova et al.) 2018). We model the query network π as a reinforcement learning agent, specifically deep Q-network (Mnih et al.] 2013). This data-driven approach allows the model to learn selection strategies based solely on prior AL experience. Our formulation differs from other approaches by the task we address, the definitions of states, actions and rewards, and the reinforcement learning algorithm we use to find the optimal policy.
3.1 ACTIVE LEARNING WITH REINFORCEMENT LEARNING FOR SEGMENTATION
In our setting, we use four different data splits. To train π, we define a subset of labeled data Dr to play the active learning game for several episodes and learn good acquisition function that maximizes performance with a budget of B regions. The query network is evaluated on a different split Dv. We use separate subset DR obtain the reward signal by evaluating the segmentation network on it. The set Ds DDsl < |Drr) is used to construct the state representation.
The MDP is defined with the sequence of transitions {(sE, ae, rt+1; 8t41)) For every state 8t S (function of the segmentation network at timestep 1), the agent can perform actions a, E A to choose which samples from Ue annotate. The action at {af K J==1 composed of K sub-actions, is function of the segmentation network, the labeled and the unlabeled set. Each sub-action asks for a specific region to be labeled. Then, receives reward r++1 based on the improvement in mean loU per class after training the segmentation network with the selected samples. Note that states and actions do not depend on the specific architecture of the segmentation network. We are interested in finding a policy to select samples that maximize the segmentation performance. We use deep Q-network (Mnih et al. 2013) and samples from an experience buffer to train the query network m.
Each episode elapses total of T steps. We start by setting the segmentation network I to set of initial weights Og and with no annotated data, i.e., Lo = @ and Uy = Dr- At each iteration 4, the following steps arc done:
1. The state 8t is computed as function of St and Ds-
2. A restricted action space is built with K pools P with N regions, sampled uniformly from the unlabeled set Uz. For each region in each pool, we compute its sub-action representation a.".
3. The query agent selects K sub-actions (at 1==1 with e-greedy policy. Each sub-action af is defined as selecting one region x& (out of N) to annotate from pool Pk.
4. An oracle labels the regions and the sets are updated: Li+1 = L, U {(z HK) and
Ut+1 = Ue {
5. The segmentation network fe+1 is trained one iteration on the recently added regions {
6. The agent receives the reward Tr+1 as the difference of performance between f and fo on Dr.
Figure 2 depicts this training algorithm. We consider the termination of each episode when the budget B of labeled regions is met, i.e., [Ce| = B. Once the episode is terminated, we restart the weights of the segmentation network f to the initial weights lg. set Lo = @ and Uo Dr, and restart the episode. We train the query policy æ by simulating several episodes and updating its weights at each
We chose non-overlapping squares regions (similar to (Mackowikk et al. 2018)) Other choices of regions could also he valid, but we consider region design choice selection he the scope of this work.
4

---

Published as a conference paper at ICLR 2020
Labeled set
L
Add regions to labeled set
5. Train
Choose action
a
Segmentation network
Unlabeled set
2. Build action pools
U1
Pl
Build state
Query Network π
S
p
State set 5
Reward set DRi
6. Compute reward
t+1
Figure The query network π trained during several episodes e with MDP transitions {(8,, az T++1; (++1)] The state computed function of segmentation network and state set Ds. K unlabeled pools Pf are sampled uniformly from the unlabeled set Uu The representation of their possible sub-actions are computed using f. labeled set L. and unlabeled set 2e. 3) Query network selects action as. composed of K sub-actions at. Each of them is chosen from its corresponding pool. 4) Selected regions are labeled and added to La (and removed from Mt). 5) Segmentation network f trained with those new labeled samples. 6) Reward re+1 is obtained from DR. This loop continues until budget B of labeled regions is achieved.
timestep by sampling transitions {(E,, ae, Tt+1, 8t+1)} from the experience replay buffer More details in Section
State representation. We would like to the state of the segmentation network f as the MDP state. Unfortunately, it is not straightforward to embed f into state representation. Follow- ing Konyushkova al. (2177) we represent the state space S with the help of set-aside set Ds. We use a small subset of data from the train set, making sure it contains a significant representation of all classes. We consider it to be representative set of the dataset, and that any improvement in the segmentation performance on subset Ds will translate into an improvement over the full datase? We use the predictions of the segmentation network f. on D5 to create global representation of state (step in Figure 2).
We need compact representation avoid intensive memory usage due the pixel-wise predictions. The samples in Ds are split in patches, and compact feature vectors are computed for all of them. Then, each region is encoded by the concatenation of two of features: one based on class predictions of f, and the other on its prediction uncertainty, represented as the Shannon entropy (Shannon) 1948). The first set of features (i) (normalized) count of the number of pixels that are predicted each category. This feature encodes the segmentation prediction spatial information, less important for small patches. Moreover, we measure the uncertainty of the predictor with the entropy over the probability of predicted classes. For each region, compute the entropy of each pixel location to obtain a spatial entropy map. To compress this representation, apply min, average and max-po max-poolingg to the entropy map obtain downsampled feature maps. The second set of features (ii) is thus obtained by flattening these entropy features and concatenating them.
given patch while dismissing the
Finally, the state & is represented by an ensemble of the feature representation of each region in Ds. Figure A. illustrates how s, is computed from each region.
Action representation. In setting, taking action means asking for the pixel-wise annota- tion of an unlabeled region. Due to the large-scale nature of semantic segmentation, would be prohibitively expensive compute features for each region in the unlabeled set each step. For this reason, instead, at each step f, we approximate the whole unlabeled set by sampling K pools of unlabeled regions Pk. each containing (uniformly) sampled regions. For each region, we compute its sub-actt sub-action representation a. (step 2 in Figure
In practice, we found that the state set needs have similar class distribution that of the train

---

Published as a conference paper at ICLR 2020
Each sub-action d; k. is a concatenation of four different features: the entropy and class distribution features (as in the state representation), a measure of similarity between the region 2i and the labeled set and another between the region and the unlabeled set. The intuition is that the query network could learn to build a more class-balanced labeled set while still taking representative samples from the unlabeled set. This could help mitigate the hard imbalance of the segmentation datasets and improve overall performance.
For each candidate region, x in pool P/ we compute the KL divergence between the class distributions of the prediction map of region æ (estimated as normalized counts of predicted pixels in each category) and the class distributions of each labeled and unlabeled regions (using the ground- truth annotations and network predictions, respectively). For the labeled set, we compute a KL divergence score between each of the labeled regions' class distribution and the one of region x. Summarizing all these KI. divergences could be done by taking the maximum or summing them. However, obtain more informative features, we compute normalized histogram of KI. divergence scores, resulting in a distribution of similarities. As an example, if we were to sum all the scores, having half of the labeled regions with a KL divergence would be equivalent to have all labeled regions with KL divergence of c/2. The latter could be more interesting, since it means there are no labeled regions with the same class distribution as x. For the unlabeled set we follow the same procedure, resulting in another distribution of KI. divergences. how we
zero and the other half with a value c,
Both of them are concatenated and added to the action representation. Figure represent each possible action in a pool.
Based on early experimentation, learning the state and action representations directly with a CNN does not provide strong enough features for the reinforcement learning framework to converge to good solution. An ablation study on the state and action components can be found in Appendix E.
3.2 BATCH MODE DQN
The desired query agent should follow an optimal policy. This policy maps each state an action that maximizes the expected sum of future rewards. We rely on a DQN (Mnih et al. (013). parameterized by Φ, to find an optimal policy.
We train our DQN with labeled set Dr and compute the rewards in held-out split DR. As mentioned above, the query agent in our method selects regions before transitioning to the next state. We assume that each region is selected independently. as in the case where K annotators label one region in parallel. this case, the action a, is composed of K independent sub-actions (a ) k==1 each with restricted action space, avoiding the combinatorial explosion of the action space. To ease computation and avoid selecting repeated regions in the same time-step, we restrict each sub-action at to select region Fę in Pf defined as:
arggax Q(s a ".)): KEP
for each {l,..., K} action take timestep t.
The network is trained by optimizing a loss based on temporal difference (TD) error (Sutton 1988).
The loss is defined as the expectation over decomposed transitions To {(+,, at r 1141)),
obtained from the standard transitions {($t, at, rt+1, 8(+1)), by approximating r4a so T2t1:
ET (ut - Q(se, ai; o))²
(2)
where E is the experience replay buffer and yt the TD target for each sub-action.
To stabilize the training, we used a target network with weights ø' and the double DQN (Van Hasselt et al.] 2016) formulation. The action selection and evaluation is decoupled; the action is selected with the target network and is evaluated with the query network. The TD target for each sub-action represented as:
v r441 + (((44 argmax (4n
knn
Ai EPI,,
where y is discount factor.
We also tried to optimize the network with average of Q-values over all sub-actions as in y Te+1 + Σ. Q(eet1, at,1) and a() = 1 E. Q(+,, at but performed worse.

---

Published as a conference paper at ICLR 2020
This formulation is valid under the approximation that the sub-actions independent of each other, conditioned on the state. We observed that increasing the number of sub-actions K per step eases computation and does not hinder segmentation performance. We provide ablation study on the effect of in Appendix E.3
4 EXPERIMENTS
We start this section by describing the datasets that we use evaluate our method, the experimental setup, and the basclines. We evaluate the algorithm in Camvid as proof of concept and we show large-scale results on Cityscapes.
4.1 EXPERIMENTAL SETUP
Although we can apply active learning setting with unlabeled data with a human in the loop that labels selected regions, we test our approach in fully labeled datasets, where it is casier to mask out the labels of a part of the data and reveal them when the active learning algorithm selects them.
Cam (Brostow et al. 2008). This dataset consists of street scene view images, with the reso- lution of 360x480 and II categories. It has 370, 104 and 234 images for train, validation and test set, respectively. We split the train with uniform sampling in 110 labeled images (from where we get 10 images to represent the state set Ds and the rest for Dr), and 260 images to build Dy where we evaluate and compare our acquisition function the baselines. The state is chosen to be representative of DD. by restricting the sampling of Dy to have similar class distribution to the one of Dr. Each image split into 24 regions of dimension 90 We use the dataset's validation set for DR- We report the final segmentation results on the test set. In our experiments, we chose K 24 regions per step. Our model is quite robust to the number of regions selected at each time step (see Appendix E.3)
Cityscapes (Cordts et al. 2016). It is also composed of real street scene views, with image resolution of 2048 1024 and 19 semantic categories. The train with fine-grained segmentation labels has 2975 images and the validation dataset of 500 images. We uniformly sampled 360 labeled images from the train set. Out of these. 10 images represent Ds. 150 build Dr and 200, DR. where we get our rewards. The remaining 2615 images of train set are used for Dv, as if they were unlabeled. We report the results in the validation set (test set not available). Each image is split in 128 regions of dimension 128x x 128. We chose K = 256 regions per step.
Implementation details. The split DR is used to get the rewards for the DQN and also for hyper- parameter selection, that are chosen according to the best setup for both baselines and our method. We report the average and standard deviation of the 5 different runs (5 random seeds). As data augmentation, we use random horizontal flips and random crops of 224 x 224. For more details, please refer to Appendix B on supplementary material.
Evaluation. The query network trained on Dr with small, fixed budget (0.5k regions for Camvid and 4k regions for Cityscapes) to encourage picking regions that will boost the performance in an heavily scarce data regime. The learned acquisition function, as well as the baselines, is evaluated on Dy. where we ask for labels until the budget is met, for different budgets. Note that the baselines do not have any learnable component.
Once the budget is reached, we train the segmentation network f with Lr until convergence (with early stopping in D))) For fair comparison, all methods" segmentation network has been pre-trained (initial weights 0o) on GTA dataset (Richter et al., (2166), a synthetic dataset where high amounts of labeled data can be obtained without human effort, and Dr (where we had labels to train the DQN). We evaluate the final segmentation performance (measured in mean IoU) on the test set of Cam Vid and the validation set of Cityscapes.
4.2 RESULTS
7

---

Published as a conference paper at ICLR 2020
0.64
0.650
0.625
0.600
0.62
Validation teu
0.575
0.550
0.525
90% Supervised all data Entropy
0.60
0.500
Uniform
0.475
Ouns
OK
0.5K (8%)
0.75K (12%)
15 (10%)
1.23K (20%)
1.5k (20%)
0.450
0K 24 36-
8K.
12K (3%)
(6%)
32K (9%)
Budger
Budget
(a) Active learning in Cam
(b) Active learning in Cityscapes
Figure 4: Performance of several methods with increasing active learning budget, expressed as the number of 128x 128 pixel regions labeled and the % of additional labeled data, All methods have been pretrained with GTAV and a small subset of their target datasets. Budget indicates additional number of regions labeled (and the percentage of unlabeled data used). The dashed line represents the 96% of the total performance achieved by the segmentation network with fully-supervised training (having access to all labels). We report the mean and standard deviation of 5 runs
Results Cam Vid. We compare our results against three distinct baselines: (i) U is the uniform random sampling of the regions to label at each step out of all possible regions in the pool, (ii) H is an uncertainty sampling method that selects the regions with maximum cumulative pixel-wise Shannon entropy. (iii) B picks regions with
maximum cumulative pixel-wise BALD (Houlsby et al.) 2011b Gal et al. (017)) metric. We 20 iterations of MC-Dropout (Gal & Ghahramani 2016) (instead of
100, as in (Gal et al. 2017)) for computational reasons. In preliminary experiments, we did not observe any improvement using over 20 iterations. In Camvid, we use
Extrzw Figure 3: En-
pool size of 10 for our method, H, B and 50 for U. In Cityscapes, we have access to more data so we use pool sizes of 500, 200, 200 and 100 respectively for U, H, B and our method. Pool sizes were selected according to the best validation mean IoU.
tropy of class distributions ob- tained from pix- els of selected re- gions.
Figure 4a shows results on CamVid for different budget sizes. Our method out- performs the baselines for every fixed budget, except for 1.5k regions, where we achieve similar performance selecting 1.5k regions already reaches past 98% of maximum performance, where differences be- tween our method and H are negligible. Surprisingly, B is worse than U, specially for small budgets, where training with the newly acquired labels does not provide any additional information. It overfits quickly to the training, getting a worst result that with the initial weights. In general, all results have a high variance due to the low regime of data we are working in. In Appendix E.2 advantages of labeling small regions instead of full images.
H. We argue that the dataset has small number of images and
we show the
Results in Cityscapes. Figure 4bgure4bjstr shows results on Cityscapes for different budgets. Here, also observe that our method outperforms the baselines for all budgets points. Labelling 20k regions, corresponding only 6% of the total pixels (additional performance of 64.5% mean loU. This is 96% of the performance of the segmentation network it had access all labeled pixels. To reach the same performance. requires an additional 6k labeled regions (around 30% more pixels, equivalent to an extra 45 images). In this larger dataset, performs better than random, showing that for the task of segmentation, B might start to show its benefits only for considerably large budgets. Table T shows the per-class IoU for the evaluated methods (with a fixed budget). Our method works specially well for under-represented classes, such as Person, Motorcycle to under-represented classes than baselines. Note that this is a side effect of directly optimizing for the mean IoU and defining class- representations for states and actions. Figure shows the entropy of the distribution of selected pixels of the final labeled set (for a budget of 12k regions) for Cityscapes. The higher the entropy means closer to uniform distribution over classes, and our method
the labeled data in Dr), we obtain
Bicycle, among others. Indeed, our method selects more pixels belonging
8

---

Published as a conference paper at ICLR 2020
Side-
Build-
Traffic
Traffic
Vege- tation
Method
Road
Walk
ing
Wall
Fence
Pole
Light
Sign
Terrain
U
96.67
76.63
88.48
33.89
36.00
52.80
$4.27
60.84
90.27
52.34
H
95.60
72.08
88.06
35.30
44.59
52.43
53.70
61.38
90.08
51.87
B
95.25
69.37
88,75
32.28
44.36
53.81
58.84
64.79
90.27
50.51
Ours
96.19
74.24
88,46
33.56
42.28
53.28
57.18
63.61
90.20
51.84
Motor-
Sky
Person
Rider
Car
Truck
Bus
Train
Bicycle
mloU
cycle
92.57
69.66
31.82
90.13
27.04
43.41
23.30
32,98
63.64
58.78
88.27
72.69
40.85
90.46
42.40
58.88
33.63
43.17
68.08
62.29
93.33
71.16
39.08
88.38
34.23
30.35
37.37
66.67
60.64
Ours
91.32
73.30
45.22
90.91
42.14
58.84
35.97
45.14
69.35
63.32
Table I: Per category IoU and mean IoU [%] on Cityscapes validation set, for a budget 12k regions. For clarity, only the mean 5 runs is reported. Results with standard deviations C.I
has the highest entropy. Appendix C shows the distribution from which the entropy computed and Appendix presents some qualitative results, showing what each method decides to label for some images.
CONCLUSION
We propose data-driven, region-based method for active learning for semantic segmentation, based on reinforcement learning. The goal is to alleviate the costly process of obtaining pixel-wise labels with human in the loop. We propose new modification of DQN formulation to learn the acquisition function, adapted to the large-scale nature of semantic segmentation. This provides computationally efficient solution that uses less labeled data than competitive baselines, while achieving the same performance. Moreover, by directly optimizing for the per-class mean IoU and defining class-aware representations for states and actions, our method asks for more labels of under-representte classes compared baselines. This improves the performance and helps mitigate class imbalance. As future work, we highlight the possibility of designing a better region definition, that could help improve the overall results, and adding domain adaptation for the learnt policy, to transfer it between datasets.
ACKNOWLEDGEMENTS
We thank NSERC and PROMPT. We would also like to thank the team at ElementAl for supporting this research and providing useful feedback.
REFERENCES
Philip Bachman, Alessandro Sordoni, and Adam Trischler. Learning algorithms for active learning. In ICML, 2017.
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. PAMI, 2017.
Yoram Baram, Ran El Yaniv, and Kobi Luz. Online choice of active learning algorithms. JMLR, 2004.
Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. What's the point: Semantic segmentation with point supervision. In ECCV, 2016.
Gabriel J. Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recogni- tion using structure from motion point clouds. In ECCV, 2008
Robin Chan, Matthias Rotmmann, Fabian Hüger, Peter Schlicht, and Hanno Gottschalk. Applica- tion of decision rules for handling class imbalance in semantic segmentation. arXiv preprint arXiv:1901.08394, 2019.
Hong-Min Chu and Hsuan-Tien Lin. Can active learning experience be transferred? In ICMD, 2016.

---

Published as a conference paper at ICLR 2020
Gabriella Contardo, Ludovic Denoyer, and Thierry Artières. A meta-learning approach one-step active-learning. International Workshop on Automatic Selection, Configuration and Composition of Machine Learning Algorithms. CEUR, 2017.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Ido Dagan and Sean P. Engelson. Committee- sampling for training probabilisticc classifiers. In ICML, 1995.
J. Deng, W. Dong, R. Socher, L.-. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hicrarchical Image Database. In CVPR, 2009.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:://acchhhit ics. www.ciicee edu/ml
Suyog Dutt Jain and Kristen Grauman. Active image segmentation propagation. In CVPR, 2016.
Sandra Ebert, Mario Fritz, and Bernt Schiele. Ralf: reinforced active learning formulation for object class recognition. In CVPR, 2012.
Meng Fang, Yuan Li, and Trevor Cohn. Learning how active learn: A deep reinforcement learning approach. In EMNLP, 2017.
Clement Farabet, Camille Couprie, Laurent Najman, and Yann LeCun. Learning hierarchical features for scene labeling. PAMI, 2013.
Yoav Freund, Sebastian Seung, Shamir, and Naftali Tishby. Information, Prediction, and Query by Committee. In NIPS, 1993.
Y Gal, R Islam, and Z Ghahramani. Deep bayesian active learning with image data. ICML, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
Mare Gorriz, Axel Carlier, Emmanuel Faure, and Xavier Giró Nieto. Cost-effective active learning for melanoma segmentation. MLAH: Machine Learning for Health Workshop, NIPS, 2017.
He, Mari Ostendorf, Xiaodong He, Jianshu Chen, Jianfeng Gao, Lihong Li, Li Deng. Deep reinforcement learning with combinatorial action space for predicting popular reddit threads. In EMNLP, 2016.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:/112.57455 201 la.
Neil Houlsby, Ferenc Huszar, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv, 2011b. lb.
Wei-Ning Hsu and Hsuan-Tien Lin. Active learning by learning. In AAAI, 2015.
Alexander Kirillov, Ross B. Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. CVPR, 2019.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Introducing geometry in active learning for image segmentation. In ICCV, 2015.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. In NIPS, 2017.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Discovering general-purposs active learning strategies. arXiv preprint arXiv:1810.04111 2018.
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-basce learning applied to document recognition. In Proceedings of the IEEE, 1998.
10

---

Published as a conference paper at ICLR 2020
Tsung- Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Ming Liu, Wray Buntine, and Gholamreza Haffari. Learning how to actively learn: A deep imitation learning approach. In ACL, 2018.
Chengjiang Long and Gang Hua. Multi-class multi-annotator active learning with robust gaussian process for visual recognition. In ICCV, 2015.
Jonathan Long. Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Radek Mackowiak, Philip Lenz, Omair Ghori, Ferran Diego, Oliver Lange, and Carsten Rother. Cereals-cost-effectiiee region-based active learning for semantic segmentation. BMVC, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. NIPS Deep Learning Workshop. 2013.
Matthias Müller, Alexey Dosovitskiy, Bernard Ghanem, and Vladen Koltun. Driving policy transfer via modularity and abstraction. CoRL, 2018.
Thomas Osugi, Deng Kim, and Stephen Scott. Balancing exploration and exploitation: A new algorithm for active machine learning. ICDM, 2005.
Aishwarya Padmakumar, Peter Stone, and Raymond Mooney. Learning a policy for opportunistic active learning. In EMNLP, 2018
Kunkun Pang, Mingzhi Dong, Yang Wu, and Timothy Hospedales. Meta-learning transferable active learning policies by deep reinforcement learning. arXiv preprint 1rXiv: 04798, 2018.
Pedro O. Pinhciro and Ronan Collobert. Recurrent convolutional neural networks for scene labeling. In ICML, 2014.
Sachin Ravi and Hugo Larochelle. Meta-learning for batch mode active learning. In ICLR Workshop, 2018.
Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In ECCV, 2016.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
Nicholas Roy and Andrew McCallum. Toward optimal active learning through monte carlo estimation of error reduction. ICML, 2001.
Max Schwarz, Anton Milan, Arul Selvam Periyasamy, and Sven Behnke. Rgb-d object detection and semantic segmentation for autonomous manipulation in clutter. IJRR, 2018.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. ICLR, 2018.
Burr Settles, Mark Craven, and Lewis Friedland. Active learning with real annotation costs. In NIPS, 2008.
Claude Elwood Shannon. mathematical theory of communication. Bell system technical journal, 1948.
Carole H Sudre, Wengi Li, Tom Vercauteren, Sebastien Ourselin, and M Jorge Cardoso. Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, 2017.
Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning. 1988.
11

---

Published as a conference paper at ICLR 2020
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. AAAI, 2016.
Alexander Vezhnevets, Joachim M Buhmann, and Vittorio Ferrari. Active learning for semantic segmentation with expected change. In CVPR, 2012.
Sudheendra Vijayanaraimman and Kristen Grauman. What's going to cost you?: Predicting effort vs. informativeness for multi-label image annotations. In CVPR, 2009.
Keze Wang, Dongyu Zhang, Ya Li, Ruimao Zhang, and Liang Lin. Cost-effective active learning for deep image classification. IEEE Transactions on Circuits and Systems for Video Technology, 2017.
Wenfu Wang and Zhijie Pan. Dsnet for real-time driving scene semantic segmentation. arXiv preprint arXiv:1812.07044 2018.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 1992.
Mark Woodward and Chelsea Finn. Active one-shot learning. NIPS Deep RL Workshop, 2016.
Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z Chen. Suggestive annotation: A deep active learning framework for biomedical image segmentation. In MICCAI, 2017.
Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L. Berg, and Dhruv Batra. Multi-target embodied question answering. In CVPR, 2019.
12

---

Published as a conference paper at ICLR 2020
A STATE AND ACTION REPRESENTATION DETAILS
In this section, we provide illustrations that show more details on how the state and action are built. Figure A. shows how build the state representation and Figure A. A. how to compute the action representation of particular region.
HQ
Entropy features
Class distr. features
Ply
argmax))
St
State representation
Enrropy
Cass
ply (x)
Labeled
set
divergence
PL
pools
P
(b) Action representation
Figure A.1: (a) Each region v, Ds is represented as a concatenation two features, one based on entropy and the other on class predictions. The final state &t is the concatenation of the features for all regions. (b) Each region Ją in pool Ps is represented as concatenation of four features: entropy-based features, class predictions and two KI. divergence distributions, comparing each region with the labeled and unlabeled set.
B EXTENDED EXPERIMENTAL SETUP
The segmentation network f is adaptation of feature pyramid network (Lin al. 2017 for semantic segmentation (similar to the segmentation branch of (Kirillov et al. 2019)). with a ResNet- 50 backbone (He et al. 2016), pretrained ImageNet (Deng al. 2009). The network is pretrained on the full train set of large-scale synthetic dataset, GTAV (Richter et al. (2166), therefore not requiring much human labelling effort. Moreover, this dataset has the advantage possessing the same categories as real datasets we experiment with.
The query network ,, depicted in Figure B.
is composed of paths, one compute state features and another to compute action features. fusing them at the end. Each of the layers are composed of Batch Normalization, ReLU activation and a fully-connecter layer. The state path and action path are composed of 4 and 3 layers, respectively, with final layer that fuses them together to get the global features; these are gated with a sigmoid, controlled by the KI. distance distributions in action representation. The weights updated at each step of the active learning loop. by sampling batches experience tuples from an experience replay buffer, sized 600 and 3200 for Camvid and Cityscapes, respectively.
We train both networks with stochastic gradient descent (SGD) with momentum. We use the same learning rate for both the segmentation and query networks; 10-4 and 10 for Cityscapes and Camvid respectively. Weight decay is set to 10 for the segmentation network and 10 for the query network. We used training batch size of 32 for Camvid and 16 for Cityscapes.
13

---

Published as a conference paper at ICLR 2020
S
k
04
x x|_
TX Ni
1 N
x 256
x 512
N, x 128
256
N₂ x
Flatten
128
Flatten
N
128
128
256
(sty at
knn
Figure B.1: The DQN takes a state representation &, and action representation for a possihle action (labeling region zı in an unlabeled pool Pf ). Ny are the number of state and action features (class distributions and entropy- based features), and Nsrar the number of features for the KL divergence distributions. Features are computed for both representations separately with layers composed of Batch Normalization, ReLU activation and fully connected layers. Both feature vectors are flattened and concatenated, apply final linear layer that obtains a score as a single scalar. The Q-values are computed as the gated score, where the gate controlled by a feature representation from the KL distance distributions of the action representation.
C CLASS FREQUENCIES AND PERFORMANCE PER CLASS
We show in Figure C.1 a more detailed plot for the class frequencies of regions that each of the methods chooses for labeling. As the entropy of the class distributions in Figure show, our method picks more regions containing under-represented classes. Specially, it asks labels for more Person, Rider, Train, Motorcycle and Bicycle pixels.
We observe that our B baseline picks more than 50% of pixels for only 3 classes that are over- represented or have a medium representation: Building, Vegetation and Sky. This could explain why the performance is worse than the H baseline.
Moreover, table C.I
we extend Table by adding the standard deviation for each result.
D QUALITATIVE RESULTS
We compare our method qualitatively with the baselines in Figure D.I Baseline U asks for random patches. Our method tends pick more regions with the under-representee classes and small objects. For instance, in the first image in the left, our method asks for several regions of a Train, that almost has no samples in the training data. In the second image, focuses on Person, Bicycle and Poles. In the third image, it asks for labels of the traffic lights and pedestrian on bicycle. Baselines B and H select some of those relevant regions, but miss lot of them.
E ABLATION STUDIES
In this section, we provide an ablation study on the state and action representation, the effect of labeling small regions versus full images, and the comparison of taking different regions per step.
14

---

Published as a conference paper at ICLR 2020
Data split
0.30
0.25
Frequency
Qurs
0.15
0.30
0.05
0.00
ro0d
adeessik
buiiding
wall
fence
prie
traffic light trotffi sign
legitation
lerrain
gk1
person
ider
car
bruck
txop
bntn
motarcy
Bcyde
yil
Figure C.I: Class frequencies [%] Cityscapes for selected regions to label after the active learning acquisition for different methods. "Data split" frequencies refer to the proportion of classes in the unlabeled data split, where we reveal the masks for the purpose of showing the underlying class frequencies. In this split is where all methods perform active learning, in the setting where mask out the labels (D.). Budget used: 12k regions. ease of visualization, only plot mean runs. Void label represents all pixels for which we do not assign any labels.
Method
Road
Sidewalk
Building
Wall
Fence
96.67 + 0.09
76.63 = 0.51
88.48 + 0.12
33.89 + 1.11
36.00 + 2.12
H Shannon
1948
95.60 + 0.33
72.08 = 1.28
88.06 + 0.42
35.30 + 1.73
44.59 + 1.62
B (Gal et al
2017
95.25 + 0.28
69.37 = 0.94
88.75 = 0.18
32.28 + 0.88
44.36 + 1.12
Ours
96.19 + 0.23
74.24 1.50
88.46 + 0.23
33.56 2.30
42.28 + 1.40
Pole
Traffic Light
Traffic Sign
Vegetation
Terrain
U
52.80 + 0.41
54.27 + 1.34
60.84 + 0.99
90.27 + 0.14
52.34 + 1.38
II Shannon
1948
52.43 0.31
53.70
1.48
61.38 1 0.81
90.08 1 0.16
51.87 1 0.79
(Gal al.
2017
53.81 1 0.30
58.84
0.50
64.79
0.34
90.27 1 0.15
50.51 1
Ours
53.28 1 0.51
57.18 + 1.92
63.61 + 1.47
90,20 1 0.26
51.84 1 1.62
Sky
Person
Rider
Car
Truck
92.57 + 0.30
69.66 0.62
31.82 2.66
90.13 = 0.01
27.04 + 2.16
Shannon
1948
88.27 + 3.26
72.69 + 0.53
40.85 + 1.85
90.46 +0.38
42.40 + 1.96
B Gal et al.
2017
93.33 + 0.24
71.16 + 0.47
39.08 + 1.26
88.38 + 0.29
34.23
+ 1.24
Ours
91.32 + 1.06
73.30 + 0.43
45.22 + 2.75
90.91 + 0.23
42.14 + 1.41
Bus
Train
Motorcycle
Bicycle
mIoU
U
43.41 + 2.80
23.30 + 2.52
32.98 + 3.81
63.64 + 0.33
58.78 + 0,29
H (Shannon
1948
58.88 +: 2.97
33.63 + 4.76
43.17 1.33
68.08 + 0.38
62.29 1 0.55
B (Gat et al. 2017
43.41 4:.34
30.35 = 3.18
37.37 + 0.79
66.67 + 0.67
60.64 + 0.49
Ours
58.84 + 4.15
35.97 3.50
45.14 2.34
69.35 + 0.90
63.32 + 0.93
Table C.I: Per category and mean ToU [%], on Cityscapes validation set, for budget of 12k regions. Both the mean and standard deviation of runs is reported.
E. STATE AND ACTION REPRESENTATION
Here, we analyze the incremental effect of our design choices for the state and action representation on Cityscapes. We use pooling operations min, average, max - to compress the entropy map of the region and use it in the state and action representation. Also, KL divergences are added to the latter. As seen in Table E. using only the max-pooled entropy map (Ours- 1H), the performance is slightly worse than H. When we combine the information of the 3 pooled entropy maps (Ours 3H), we outperform H baseline. Moreover, when adding the two distribution of KL distances to our action representation (Ours 3H KL): between possible regions to label and the labeled set and between the region and the unlabeled set, we further increase the performance, getting our best state and action representations.
15

---

Published as a conference paper at ICLR 2020
ENSATO
U
B
H
Ours
Qualitative results Cityscapes after running the active learning algorithm with a budget 2k regions. The first row consists on input images, the second shows the what picks, the third, the fourth H, and the last row shows what our method picks. Best viewed in color.
Figure D.I:
Pool size
State
20
100
200
54.62 + 0.60
54.92 + 0.59
55.15 + 0.64
(Shannon
1948
57.41 + 0.17
57.55 + 0.60
57.48 0.96
Gal el al.
2017
56.73 = 0.20
0.32 56.99
56.44 -: 0.77
Ours 1H
56.89 + 1.22
57.29 0.67
57.62 0.96
Ours 3H
57.65 - 0.74
58.10 + 1.16
57.65 1.30
Ours 3H+KL
57.67 1 0.92
58.95
0.59
59.18 0.62
Table F.1: Contribution to the validation mean ToU performance [%] of Cityscapes dataset, for budget of 4K and for each of the components of our state representation, compared the baselines. Mean and standard deviation of runs is reported.
E.2 REGION VS. IMAGE ANNOTATION
In this subsection, we analyze the effect of asking for labels regions instead of full images and the effect of the number of regions per step. We compare the validation IoU when asking for pixel-wise labels for entire images versus pixel-wise labels for small regions. In the first case, we ask for one image each step and, for the latter, we ask for 24 regions per step (pixel-wise, equivalent one image). As it shown in Table E.2 asking for entire image labels has similar performance for all methods, that resemble Uniform performance when asking for region labels. This indicates that, in order to select more informative samples, it useful to split the images into patches (crops) and be able to disregard regions that only contain over-represented classes of the dataset.
16

---

Published as a conference paper at ICLR 2020
E.3 INFLUENCE OF STEP REGIONS
Empirically, our selector network is quite robust to the number of regions per step, as seen in Table E.3 Therefore, select 24 regions for Cam Vid, the that yielded best results. This more efficient to train than taking one region per step.
U
H
Ours
Full im.
69.64 1 0.33
69.46 1 0.15
69.66 + 0.21 69.44 0.22
24 R
70.35 + 0.71 70.40 + 0.65 70.63 = 0.77 71.85 = 0.68
Table E.2: Comparison between labeling full image and 24 non-overlapping square regions (pixel-wise, equivalent full image), for different methods. Performance measured terms of validation mean IoU performance [%] Cam Vid dataset, for budget 0.5k. In the first row, results for "full im.", one entire image is labeled at each step (region size equal to the size of the image). In the second row, "24 R" results for labeling 24 regions each step. Pool size selected the one that performed better, out of 10, 20, 50 and 100. Results reported with the and standard deviation of 5 runs.
Regions per step
Val IoU [%]
71.10 + 0.75
70.93 = 0.70
24
71.85 = 0.68
36
71.24 = 0.49
48
71.25 + 1.17
71.20 + 0.53
Table E.3: Results of varying the number of regions to be labeled at each step by our method. Performance is measured in terms of validation mean IoU performance [%] in Cam dataset, for a budget of 0.5k Results are reported with the and standard deviation of runs.
17


Reviews:
[{"review_id": "r1guYvhkcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary #\nThe paper works on active learning for semantic segmentation, aiming to annotate as few \"blocks/patches\" as possible while training a strong model. The authors proposed to learn a query policy via Q learning, and design states and actions specifically for segmentation. The experimental results show that the learned policy can attend to informative patches and rare classes to learn a model faster and efficiently.\n\n# Strength #\nS1. The paper is well-motivated; the references are quite sufficient.\nS2. The paper clearly states the challenges when applying RL algorithms like Q-learning to image segmentation, which should serve as good guidance for other future work.\n\n#Weakness/comments#\nW1. The writing of the technical part can be strengthened.  The authors deferred the state and action design entirely to the supplementary, while they are the main contributions to the paper.\n\nW2. The proposed algorithms seem to be highly time-consuming. The actions require pairwise comparison, and at every step, the models need to evaluate all the validation images to get the reward.\n\nW3. If I understand correctly, the authors use part of the training data D_T, D_S (of an existing dataset) together with the validation data D_R to learn the policy, and then use the learned policy to select patches from the remaining training data D_V to train the segmentation model. I have two questions.\n1) The labeled data involved in policy training is indeed quite large (validation plus part of the training, D_S + D_T + D_R). Does it mean that to learn a good active learning policy we indeed need a large number of labeled data?\n2) Since (D_S, D_T, and D_R) are used to learn the policy, they should be treated as available training data for segmentation that all the compared algorithms can use without spending the budget. In other words, all the compared algorithms (U, H, B) should use those data to fine-tune a pre-train segmentation network before they start to acquire data from D_V. It would be great if the authors can clarify this.\n\nW4. In applying the policy for selecting patches from D_V, do the authors update the model once on the selected patches, or do the authors train with them for multiple iterations together with other previous selected patches? Since deep neural nets are known to forget what has been learned (i.e., catastrophic forgetting), it's better if the author could clarify this.\n\nW5. The authors include an upper bound in Fig. 4; however, I didn't find the explanation. Why the proposed methods can outperform the upper bound with all the training data, even with only 24% of data?\n\n#Rebuttal#\n\nPlease discuss W1-W5.\n\n- Annotating an entire image is definitely easier to annotators than annotating patches. Could the authors discuss how to design an active learning algorithm by selecting informative images to annotate, and maybe compare to such a method?\n\n- Can the authors discuss Figure 3 more? As H is based on maximum entropy, why is it outperformed by the proposed method? \n\n- There is no explicit mechanism to prevent that the k actions select similar patches. Can the authors provide more discussion?\n\n# Post rebuttal\nThe authors responded to most of my concerns. I'd like the authors to incorporate all their responses into the manuscript or the appendix so that future readers can better understand the concepts and details.  I would like to raise the score to borderline (4 or 5). I modified my scores to weak accept (6) since there is no option in between.\n\nOne concern I still have is W3. 2). Given only 360 images are available, it might be inappropriate to use D_T + D_S (roughly 160 images?) to pre-train baseline models and then use D_R with \"200\" images for validation (early stopping). It will be more appropriate to use, for example, 70% of 360 images for training. This is supported by that many modern datasets use a much larger training set than the validation set. Therefore, I would highly suggest the authors redoing the baseline methods; otherwise, future work that re-splits the data (this is totally valid!) from the 360 images might easily achieve higher accuracy.\n\nThe authors must also reorganize the paper, taking W1 into account. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1eSMTzp9H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a method to use active learning to control the labeling process to gather training data in the context of a semantic segmentation application. In particular, authors define actions, states and rewards, and slightly modify DQN, to learn a policy to select informative image regions to obtain pixel labels. The proposed strategy contains several novelties related to the model and the application domain. Furthermore, the paper is clear and well written. \n\nOne weak part of the proposed method is the predefined split of each image into 24 regions that are used to define the population of sampling candidates. I believe that, for the case of semantic segmentation, the arbitrary definition of the boundary of each region might degrade performance. This because for this application the real object boundary is highly informative. Actually, this could be a major reason to explain that the proposed method only presents a modest boost in performance with respect to baselines, which is clear only for the under-represented categories. \n\nA point that it is not clear to me is the initial training of the segmentation network. Also, it is not clear how is the process to train the segmentation network after the inclusion of each new batch of labeled data (do the segmentation network is trained incrementally with just the new data?). Finally, I believe it will be good to include information about the computational complexity of resulting model.\n\nIn summary, the idea of using AL and RL to control the labeling process in a semantic segmentation application is interesting and particularly relevant for this application. While the proposed method has some weaknesses, it is novel and it will be of interest to people interested in semantic segmentation. I rate the paper as weak accept.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Authors propose a novel scheme to perform active learning on image segmentation. This structured task is highly time consuming for humans to perform and challenging to model theoretically as to potentially apply existing active learning methods. Reviewers have remaining concerns over computation and that the empirical evaluation is not overwhelming (e.g., more comparisons). Nevertheless, the paper appears to bring new ideas to the table for this important problem.    
Decision: Accept (Poster)