Active Learning for Semantic Segmentation with Multi-class Label Query
Authors: Sehyun Hwang
Sohyun Lee
Hoyoung Kim
Minhyeon Oh
Jungseul Ok
Suha Kwak
Venue: NeurIPS 2023 poster (2023)
Keywords: semantic segmentation; active learning; partial label learning
TL;DR: This paper enhances active learning for semantic segmentation by a new query design that requests labels for all classes in a local region.

Abstract: This paper proposes a new active learning method for semantic segmentation. The core of our method lies in a new annotation query design. It samples informative local image regions ($\textit{e.g.}$, superpixels), and for each of such regions, asks an oracle for a multi-hot vector indicating all classes existing in the region. This multi-class labeling strategy is substantially more efficient than existing ones like segmentation, polygon, and even dominant class labeling in terms of annotation time per click. However, it introduces the class ambiguity issue in training as it assigns partial labels ($\textit{i.e.}$, a set of candidate classes) to individual pixels. We thus propose a new algorithm for learning semantic segmentation while disambiguating the partial labels in two stages. In the first stage, it trains a segmentation model directly with the partial labels through two new loss functions motivated by partial label learning and multiple instance learning. In the second stage, it disambiguates the partial labels by generating pixel-wise pseudo labels, which are used for supervised learning of the model. Equipped with a new acquisition function dedicated to the multi-class labeling, our method outperforms previous work on Cityscapes and PASCAL VOC 2012 while spending less annotation cost. Our code and results are available at [https://github.com/sehyun03/MulActSeg](https://github.com/sehyun03/MulActSeg).

Active Learning for Semantic Segmentation with Multi-class Label Query
Sehyun Hwang
Sohyun Lee Hoyoung Kim Minhyeon Oh Jungseul Ok Suha Kwak
Pohang University of Science and Technology (POSTECH), South Korea (sehyun03, Ishig96, eskhy 16, minhyeonoh, jungseul.ok, kwak. @ postech.ac.kr
Abstract
This paper proposes a new active learning method for semantic segmentation. The core of our method lies in a new annotation query design. It samples informative local image regions (e.g., superpixels), and for each of such regions, asks an oracle for multi-hot vector indicating all classes existing in the region. This multi-class labeling strategy substantially more efficient than existing ones like segmentation, polygon, and even dominant class labeling in terms of annotation time per click. However, introduces the class ambiguity issue in training as it assigns partial labels (i.e., set of candidate classes) to individual pixels. We thus propose new algorithm for learning semantic segmentation while disambiguating the partial labels in two stages. In the first stage, it trains segmentation model directly with the partial labels through two new loss functions motivated by partial label learning and multiple instance learning. In the second stage, it disambiguates the partial labels by generating pixel-wise pseudo labels, which are used for supervised learning of the model. Equipped with a new acquisition function dedicated to the multi-class labeling, our method outperforms previous work on Cityscapes and PASCAL VOC 2012 while spending less annotation cost. Our code and results available at
Introduction
Supervised learning of deep neural networks has driven significant advances of semantic segmentation for decade. At the same time, however, has limited practical applications of the task demands as supervision pixel-level class labels that are prohibitively expensive general. To address this limitation, label-efficient learning approaches such weakly supervised learning [2, 11, 20, 28, 28,34,39, 40,58, 63], semi-supervised learning (4, 12, 25, 23, 36, 37,41-44, 48, (8,, self-supervised learning [24, 69), and active learning (AL.) [9, 10, 14,23,31, 47,55,56 68] have been investigated.
This paper studies AL for semantic segmentation, where a training algorithm selects informative samples from training data and asks oracle to label them limited budget. In AL, the design of annotation query, i.e., the granularity of query samples and the annotation format, of vital importance for maximizing the amount and quality of supervision provided by the oracle within a given budget. Early approaches consider an entire image as a sample and ask for its pixel-wise class labels [56, 68), select individual pixels and query the oracle for their class labels [54]; they turned out to be suboptimal since the former lacks the diversity of samples [47] and the latter is less budget-efficient query provides supervision for only single pixel.
As compromise between these two directions, recent AL. methods treat non-overlapped local image regions individual samples [9, 10, 23, 17,, These region-based methods guarantee the diversity of samples by selecting local regions from numerous images with diverse contexts. Also, their queries are designed to obtain region-wise segmentation labels efficiently. For instance, they
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Comparison on a region query example
Labeling cost and accuracy on overage
Dominant class labeling
10.0
"Select dominant
12.12 sec
(Pole) click
Time per click (s)
class within the red border"
7.5
(Query) Multi-class labeling
(Annotation click)
(Dominant)
5.0
Building
Car
Pole
"Select all classes within the red border" (Query)
17.13 sec
(Building, Car, Pole) clicks
2.5
Dom. cassss
Dom.
Classes
Mul. 3 classes
Mui. 2 classes
Rider
0.0
60
80
100
(Annotation clicks)
(Multi-class)
Accuracy (96
Figure 1: Dominant class labeling [9] versus our multi-class labeling. (left) Given a local region as query, an oracle is asked to select the most dominant class by single click in dominant class labeling, and all existing classes by potentially more than one click in multi-class labeling. As shown here, multi-class labeling often takes less annotation time per click because, to determine the dominant one, the oracle has to infer every class in the region after all and sometimes should very carefully investigate the region when the classes occupy areas of similar sizes. (right) We conducted user study to compare the two strategies in terms actual labeling cost and accuracy versus the number of classes in region queries; the results are summarized in the right plot with one standard deviation. Multi-class labeling took less time per click on average due to the above reason. Furthermore, resulted in more accurate labels by annotating non-dominant classes ignored in dominant class labeling additionally. Details of this user study are given in Appendix A.
ask the oracle to draw segmentation masks in the form of polygons within an image patch [14, [14,47], or estimate the dominant class label of a superpixel so that the label is assigned the entire superpixel [9]. Although the existing region-based based methods have achieved great success, argue that there is still large room for further improvement their query designs: Polygon labeling [14, 47] still requires large number of clicks per query, and dominant class labeling [9] provides wrong supervision for part of a multi-class region. Note that the latter issue cannot be resolved even using superpixels since they frequently violate object boundaries and include multiple classes.
In this context, we first introduce a new query design for region- based AL of semantic segmentation. The essence of our proposal is to ask the oracle for multi-hot vector that indicates all classes existing in the given region. This multi-class labeling strategy enables to prevent annotation errors for local regions capturing multiple classes, and works the same as dominant class labeling (and thus inherits its advantages) for single-class region queries. Moreover, our user study revealed that multi-class labeling demands less annotation time per click and results in more accurate labels compared with dominant class labeling as demonstrated in Fig. 1. However, such region-wise multi-class labels introduce new challenge in training, known as the class ambiguity issue, since they assign partial labels [16,29]
29] (i.e., a set of candidate classes) to individual pixels of the selected regions.
To address the ambiguity issue, we propose new AL method tailored to learning semantic segmen- tation with partial labels. Fig. 2 illustrates the overall pipeline of the proposed method. Given a set of local regions and their multi-class labels, our method trains segmentation network in two stages. the first stage, the network is trained directly with the region -wise multi-class labels. To this end, we propose two new loss functions for the label disambiguation based on the notions of partial-label learning [16, 29] and multiple instance learning [19], respectively. the second stage, our method disambiguates the partial labels through pseudo segmentation labels, which are used to train the segmentation network in the supervised learning fashion. To be specific, it finds a set of class prototype features from each local region using the model of the first stage, and employs the prototypes as a region-adaptivy classifier to predict pixel-wise pscudo labels within the region. In addition, we propose to propagate the pseudo labels to neighboring local regions to increase the amount of supervision given per query; this strategy benefits by multi-class labeling that enables to propagate pseudo labels of multiple classes, leading to larger expansion of pseudo labels.
Last but not least, we introduce an acquisition function designed to maximize the advantage of multi-class labels in the region-based AL. Our acquisition function considers both uncertainty [32,62] and class balance [7,66, 67] of sampled regions so that local regions where the model finds difficult and containing underrepresented classes chosen more frequently. It shares similar motivation

---

with an existing acquisition function [9], but different in that it considers multiple classes of a region and thus better aligns with multi-class labeling.
The proposed framework achieved the state of the art on both Cityscapes [15] and PASCAL VOC 2012 [21]. Especially, it achieved 95% of the fully supervised learning performance on Cityscapes with only 4% of the full labeling cost. In addition, we verified the efficacy and efficiency of multi-class labeling through extensive empirical analyses: Its efficacy was demonstrated by experiments with varying datasets, model architectures, acquisition functions, and budgets, while its efficiency was examined in real-world annotation scenarios by measuring actual labeling time across large number of human annotators. In short, the main contribution of this paper is five-fold:
We introduce a new query design for region-based AL in semantic segmentation, which asks the oracle for a multi-hot vector indicating all classes existing within a particular region.
We propose novel AL framework that includes supervision of multi-class labels and a method for generating pseudo segmentation labels from the multi-class labels, resulting in enhanced supervision.
loss functions effectively utilizing the
To maximize the advantage of multi-class labels, design acquisition function that considers multiple classes of a local region when examining its uncertainty and class balance.
The effectiveness of multi-class labeling was demonstrated through extensive experiments and user study in real-world annotation scenarios.
The proposed framework achieved the state of the art on both two public benchmarks, Cityscapes and PASCAL VOC 2012, with a significant reduction in annotation cost.
2 Related Work
Active learning (AL). In AL, training algorithm samples informative data and asks oracle to label them on a limited budget so as to maximize performance of a model trained with the labeled data. To this end, AL methods have suggested various sampling criteria such as uncertainty (5,26,50) 50), diversity [53,56), or both [6,30, 64, 65]. Also, since most of existing AL methods for vision tasks have focused on image classification, the granularity of their annotation queries has been an entire image in general. However, for structured prediction tasks like semantic segmentation, queries should be more carefully designed optimize cost-effectiveness of annotation.
Active learning for semantic segmentation. Most AL methods for semantic segmentation can be categorized into mmage-based [17,56, 68] and region-based methods [47]. The image-basee based methods consider an entire image as the sampling unit and query an oracle for pixel-wise labels of sampled images. These methods have been known to be less cost-effective due to the limited diversity of sampled data; adjacent pixels in an image largely overlap in their receptive fields and thus fail to provide diverse semanties during training. On the other hand, the region- basod methods divide each image into non-overlapping local regions, which are considered as individual samples to be selected; As such local regions, image patches [10, 14, 14,47] and superpixels 19,55,55) 55] have been employed. Our paper proposes a new cost-effective region query that allows more accurate and faster annotation, and new training algorithm taking full advantage of the query design.
Partial label learning. Partial label learning [8, 16, 29,46)4 is a branch of weakly supervised learning where set of candidate classes is assigned to each of training data, leading to ambiguous supervision. One primitive yet common approach to partial label learning is to train a model while regarding its 1 prediction true labels [8, 46). However, this approach could be vulnerable class imbalance [61] and strong correlation between different classes [52]. In contrast, our two-stage training algorithm addresses these issues by disambiguating partial labels by pseudo labeling.
3 Proposed Method
We consider an AL process with R rounds. At each round, local regions of a batch are selected using an acquisition function, and then a multi-hot vector (i.e., multi-class label) is assigned to each of them by an oracle. Given the labeled regions, our training algorithm operates in two stages as illustrated in Fig. 2. In the first stage, a segmentation model is trained directly with the region-wise multi-class labels by two loss functions specialized to handle the ambiguity of the labels (Sec. 3.2). In the second
3

---

Stage Learning with region-wise multi-class labels
Stage 2: Learning with pixel-wise pseudo labels
Input
Merged posítive loss
Input
Frozen
I
CE
loss
(Rider, Bicycle)
C a Other
R-C+B Other Merging positives
Predictive probability
Features of the region
Intra-region label localization
CE: Cross Entropy
Final Model
Model
(b) Prototypical pixel loss
CE
CE loss with Features of adjacent regions
Label expansion
(Rider,
ArgMax per class
Prototypical pixals
Car, Bicycle)
pseudo label
Figure 2: Our two-stage training algorithm using partial labels. (left) the first stage, model trained using region-wise multi-class labels through losses: the merged positive loss that encourages the model to predict any of the annotated classes for each pixel of the region, and the prototypical pixel loss that ensures at least one pixel in the region corresponds each annotated class. (right) The second stage disambiguates the region-wise multi-class labels by generating pixel-wise pseudo labels, which are then used for training the final model. To this end, it first assigns pseudo class labels to individual pixels within the region (i.e., intra-region label localization), and then propagates the pseudo labels adjacent regions (i.e., label expansion).
stage, the ambiguity is mitigated by generating pixel-wise pseudo labels and using them for training the model further (Sec. 3.3). The remainder of this section describes details of our framework.
3.1 Acquisition of region-wise multi-class labels
For unlabeled image set I, partition each image I E I into of non-overlapping regions,
denoted by S()), such that pixel æ E / belongs to only a unique region E S(I). Such a non- be obtained by superpixel algorithm
overlapping partition be the set of all the partitions for Z. For each round 6, we issue batch of regions, denoted by B, C C, each of which is queried to acquire a multi-class label Y C 21 ... C). where |Y] 2 and C' is the number of classes. Then the model 0, is trained using the labeled regions obtained so far, denoted as D == U, Pr, where D, consists of pairs of region and associated multi-class label, D. == ((s, Y) E B.). The model 0r includes feature extractor fe(-) and classifier with weight matrix [Wr,1, We,2, ... Wt,c] E _dxC The predictive probability of pixel x being class c is computed by
in [9]. Let S = Urez S(I)
We. -I].f(=)] [[we.el])
fe(x) T
Pa, (y cz)) softmax
where 7 temperature term.
Acquisition function. We introduce acquisition function for selecting batch of regions B, C S round that aims optimize the benefits of multi-class labels, while adhering the principles of previous studies [7, 7.9, 66] for uncertain and bass-balanced region selection. We adopt best-versus- second-best (BvSB) [32, 62] as an uncertainty measure, defined as
ue (x) = Po. Csb (x)
(2)
= <,|=)
where % and csb are the classes with the largest and second -argeet predictive probabilities for under 0,, respectively. For Mlass-bllance sampling, first estimate the label distribution Po, (y)
Po. -
1
Σ ZEX
X
where X = (x: Es E S, x E s). Our acquisition function, favoring uncertain regions of rare classes, is defined
us, (z)
a(s;0,) (1+ (co))2

---

where is hyperparameter regulating the class balancing effect. Distinct from an existing acquisition function [9] that considers the dominant class only, our function considers classes of all pixels in region and thus better aligns with multi-class labeling. For the remainder of this section, we will omit the round index & from Dj and O, for simplicity.
3.2 Stage 1: Learning with region-wise multi-class labels
During training of the segmentation model, regions labeled with a single class are used for the conventional supervised learning using the pixel-wise cross-entropy (CE) loss. The set of local regions equipped with single-class labels is defined as
D3 = {s,, {c}) B(s, Y) E D, | -1cc))
(5)
The pixel-wise CE loss is then given by
CE Be((c)))D
1 8
Σ ZES
log Po(y = clx))
On the other hand, regions labeled with multiple classes, denoted as Dm := D DD, cannot be used for training using the pixel-wise CE loss, since a multi-class label lacks precise correspondence between each pixel and class candidates, making it a weak label [16, 19, To effectively utilize the supervision of Dm. we introduce two loss functions.
Merged positive loss. Each pixel in region is assigned with partial labels [16,29], i.e., set of candidate classes. The per per-piee prediction in each region should be one of these candidate classes. This concept is directly incorporated into the merged positive loss, which is defined as
CMP == B(,.Y)~D.
Σ log 2
Po(u--caa)
CEY
This loss encourages to predict any class from the candidate set since the predictive probability of every candidate class is considered positive.
Prototypical pixel loss. Learning with regions assigned with multi-class labels can be considered as an example of multiple instance learning (MIL) [19], where each region is a bag, each pixel in the region is an instance, and at least one pixel in the region must positive for each candidate class. We call such pixel prototypical pixel, and the pixel with the most confident prediction for each candidate class within the region chosen as a prototypical pixel:
F max Po c/x)
8
where c Y and (s, Y) Dm. The segmentation model trained by applying the CE loss each prototypical pixel with the assumption that the class associated with is true. To be specific, the loss defined as
1 Σ -log Paly fa(y=c)=?,e) Y
Cpp B
As reported in the literature of MIL. [19], although the prototypical pixels may always match the ground truth, it is expected that training with numerous prototypical pixels from diverse regions enables the model to grasp the underlying concept of each class. Moreover, this loss mitigates the class imbalance issue as it ensures that every candidate class equally contributes to training via single prototypical pixel in region, leading a balanced class representation.
In summary, the total training loss of the first stage is given by
L = >eE Len + AMP Lxp Lpp
(10)
where ÀCE and AM are balancing hyperparameters.
3.3 Stage 2: Learning with pixel-wise pseudo labels
In the second stage, we disambiguate the partial labels by generating and exploiting pixel-wise one- hot labels. The pseudo label generation process comprises two steps: intra-region label localization that assigns pscudo class labels to individual pixels within each labeled region, and label expansion
5

---

Pseudo label generation process
Results
cos: cosine similarity
Features of the region
index
(a) Label localization
Class-wise
adaptive
GT (a)+(b)
Prototypes
threshold
(Rider, Car, Bicycle)
Prototypical pixel indexing
B ae B
COS
ArgMaax per class
Prototypical pixels
Features
(b) Label
adjacent regions
expansion
Figure 3: The pseudo label generation process (left) and its qualitative results (right). In each of the labeled regions, the feature vector located at the prototypical pixel of annotated class is considered the prototype of the class, and the set of such prototypes is used as a region-adaptive classifier for pixel-wise pseudo labeling within the region (label localization). The pseudo labels of the region propagated to adjacent unlabeled regions similarly (label expansion). but for conservative propagation, only relevant pixels that are close least one prototype will be assigned pseudo labels.
that spreads the pseudo labels to unlabeled regions adjacent to the labeled one. This process is illustrated in Fig. 3, and described detail below.
Intra-region label localization. For each of the labeled regions, we define prototype for each annotated class as the feature vector located at the prototypical pixel the class, which is estimated by Eq. (8) using the model of the first stage. The set of such prototypes is then used as aegiinnadaptive classifier, which is dedicated to pixel-level classification within the region. To be specifie, we assign each pixel the class of its nearest prototype in feature space; the assigned pseudo label for æ E where (s, Y)= E D, is estimated by
j(x) == arg max cos (ool
(11)
cEY
where xh is the prototypical pixel of class e and cos(f. = IAII between two feature vectors f and f".
TT is the cosine similarity
Label expansion. The rationale behind the label expansion step is that the class composition Y of region (s, Y) E 'D may provide clues about classes of its adjacent regions s' E N(s), where N(-) denotes a of unlabeled regions adjacent to s, i.e. N(s) D 0. Similar to label localization. the label expansion step aims to assign pixels in N(s) the class labels of their nearest prototypes. However, since N(s) may contain irrelevant classes, propagating the labels to all pixels in N(s) could cause incorrect pseudo labels, leading to performance degradation of the final model. Hence, the pseudo labels are proposed only relevant pixels that are sufficiently close to at least one prototype in the feature space. More specifically, to compute the relevance in a region- and class-adaptive manner, we propose to use prototype-adaptive thresholds: the prototype-adaptive threshold for class E Y in (s, Y) 'D is defined as
(s) med (foo)
Es, ((x)= ((x)=
(12)
where med(-) yields the median value of a set, x², is the prototypical pixel of class (Eq. (8)), and ((x) is the pseudo label of x (Eq. (11)). We propagate pseudo labels of the labeled region in D to pixels of an adjacent region {x 3s' E N(s),
s'} b by
((x) = arg max cos (os(()) fo(x),)) only if 1 (x) 2 1,
(13)
cx)(2)
where (xx) fc: cos(fo(z)) fo()) fo(==_))) > (s)),, CE Y):
x is relevant pixel if (x)) By using the prototype-adee aedatti threshold for filtering, we can adjust the amount of label expansion in each region without the need for hand-tuned hyperparameters.
The segmentation model is then further trained using the pixel-wise CE loss with pseudo segmentation labels generated by both of the label localization and expansion steps.

---

Dom- Random
Dom+Bu56
Dom-ClascBal
Dom PodBal Mui+Puuall
15% Fully Supervised
Mul-Random
Mur+Ev5ü
Nul+Classtal
75.0
75.0
72.5
72.5
72
miou (8)
70.0
mlou (09)
77.0
mlou (%)
mlou (%)
70
67.5
67.5
66
65.0
62.5
62.5
66
100K 200K 300K 400K 500K Number of clicks (ResNet50)
100K 200K 300K 400K 500K Number of clicks (ResNet101)
10K 20K 30K 40K 50K Number of clicks (ResNet50)
10K 20K
40K 50K
Number of clicks (ResNet101)
(a) Clyscapes
(b) PASCAL VOC
Figure 4: Accuracy in mloU (%) versus the number of clicks (budget) for dominant class labeling (Dom) [9] and multi-class labeling (Mul) equipped with four different acquisition functions (Random, BvSB, ClassBal, PixBal). The reported accuracy scores are averaged across three trials.
Experiments
4.1 Experimental setup
Datasets. Our method is evaluated on two semantic segmentation datasets, Cityscapes [15] and PASCAL VOC 2012 (VOC) [21]. The former contains 2975 training, 500 validation, and 1525 test images with 19 semantic classes. The latter consists of 1464 training and 1449 validation images with semantic classes. We evaluated models on validation splits of these datasets.
Implementation details. We adopt DeepLabv3+ [13] with ResNet-50/101 pretrained on Ima- geNet [18] our segmentation models, AdamW [45] for optimization. The balancing hyper- parameters ÀCE and AMP of Eq. (10) are set to 16 and 8, respectively, and the temperature was fixed by 0.1. both datasets we utilize 32 32 superpixel regions given by SEEDS [59]. For Cityscapes, initial learning rates are set to 3 (stage 1) and 4c-3 (stage 2), and in Eq. (4) is set to 6. The models are trained for 80K iterations with mini-batches of four 769 x 769 images. We assign an extra undefined class for pixels not covered by the original 19 classes. For VOC, we configure to 12 and train the models for 30K iterations using a learning rate of le-3 both stages. Each mini-batch consists of twelve 513 x 513 images. More details are given in the Appendix B.
Active learning protocol. Following the previous work [9], we consider the number of clicks as the labeling cost. While this protocol assumes a uniform cost per click, it does not hold in reality as shown in Fig. but is adverse to the proposed method since our multi-class labeling takes less cost per click than dominant class labeling. We conduct rounds of consecutive data sampling and model updates, with budget of 100K and 10K clicks per round on Cityscapes and VOC, respectively. The models are evaluated each round mean Itersection-overt OvvrrUnnion (mloU) [21] on the validation sets. At first round, regions are selected at random, and the models are reinitialized with ImageNet pretrained weights per round. We conduct all experiments three times and report the average performance.
adopted for comparisons with the prior art using dominant class labeling [9],
Baseline methods. We compare our multi-class labeling (Mul) with the dominant class labeling (Dom) combination with various data selection strategies. Following the established strategies in the previous study [9], we employ Random, which randomly selects superpixels, and the uncertainty- based BvSB given in Eq. (2). ClassBal is BvSB sampling with additional class balancing term proposed in the previous work [9], and PixBal is our sampling method based on Eq. (4).
4.2 Experimental results
Impact multi-class labeling. In Fig. 4, we evaluate the performance of multi-class and dominant class labeling across varying budgets, using ResNet50 and ResNet101 backbones, on both Cityscapes and VOC with different acquisition functions. Multi-class labeling constantly outperforms dominant class labeling in every setting across all the compared architectures and datasets. In particular, the

---

Table I: The ratio of clicks (%) needed to reach 95% mloU the sully-super model, relative to full supervision. Results with i are from the prior work [9] using Xception-65 backbone.
Number of regions
Cumulative density
10
Histogram CDF
0.8
10a
0.6
Query
Sampling
Clicks (%)
Patch + Polygon
EntropyBox+! [14]
4
5
8
MetaBox+ [14]
10.3
Number of classes
ClassBall [9]
7,9
Figure 5: Histogram and cumulative distribu- tion function (CDF) for the number of classes regions selected at round-5 using PixBal sam- pling on Cityscapes.
Spx Dominant
ClassBal [9]
9.8
PixBal
7.9
Spx Multi-class PixBal (Ours)
Table 2: Contribution of each component of our method in accuracy (mloU, %) at each round.
Loss function
Pseudo labeling
Rnd-1
Rnd-2
Rnd-3
Rnd-4
Rnd-5
Avg
LMP
Lpp
Localization Expansion
/
66.6
73.6
74.5
75.6
75.7
73 73.2
(b)
/
/
x
65.2
72.2
73.5
74.3
74.7
72.040.3
(c)
/
/
63.8
70.3
71.8
72.6
73.3
70.4
(d)
/
X
X
x
60.8
69.7
71.0
72.3
72.3
69.2.1..
(c)
X
/
X
X
63.1
69.9
70.9
71.2
71.9
69.440.
multi-class labeling model, with just 200K clicks, outperforms the dominant class labeling counterpart that uses 500K clicks on Cityscapes. When using ResNet50, the multi-class labeling model equipped with PixBal sampling, achieves 95% mloU of the fully-supervised model using only 200K clicks on Cityscapes and 20K clicks on VOC, respectively.
Impact of the proposed sampling method. Fig. also demonstrates the efficacy of PixBal. On Cityscapes, PixBal consistently outperforms all the other sampling methods regardless of budget size. also enhances the performance of dominant class labeling. On VOC, PixBal generally surpasses the baselines, although its improvement over BvSB, which lacks class balancing, is marginal at times since VOC less suffers from class imbalance than Cityscapes. Further analysis on these sampling methods are provided in Appendix C.4. Comparison with various query designs. In Table 1, evaluate multi-class labeling against baseline methods employing different query designs: drawing polygon mask within an image patch (Patch+Polygon), clicking dominant class within superpixel (Spx+Dominant), and clicking all classes within superpixel (Spx+Multi-laas)) Following the prior work [9], in this experiment, we measure the ratio of clicks used, relative to the total number of clicks required draw polygon masks on all images (i.e., full supervision). We then measure the ratio of clicks each method needs to achieve 95% mloU of the fully-supervised model. As indicated in Table 1, superpixel based methods typically outperform the baselines using patch or polygon queries. Among these, our multi-class labeling stands out, achieving 95% mloU of the aally-supervised model using only 4% of required data.
4.3 In-depth analysis on the proposed method
The number of classes in selected regions. The histogram and cumulative distribution Fig. summarize the number of classes within regions selected at round-5 using our PixBal sampling method on Cityscapes. We observe that more than 50% of the selected regions contain two or more classes, explaining the necessity of multi-class labeling. This also suggests that, regarding labeling cost in reality (i.e., actual annotation time), multi-class labeling holds potential for further improvement in efficiency as it requires less labeling time for multi-class regions (Fig. 1).
Contribution of each component. Table quantifies the contribution of each component in our method over five rounds: merged positive loss in Eq. (7), prototypical pixel loss in Eq. (9), intra-region label localization, and label expansion. The results show that all components improve performance at every round. The performance gap between (c) and (c) in the table verifies the efficacy of merged

---

(a)
(b)
(c)
(d)
Figure 6: Qualitative comparisons between different labeling strategies. (a) Dominant class labels. (b) Label localization. (c) Label localization + label expansion. (d) Ground-truth.
positive loss for learning with multi-class labels. Meanwhile, the gap between (c) and (d) in the table shows the efficacy of prototypical pixel loss, particularly at the initial round with severe class imbalance due to random sampling. The largest mIoU gain is achieved by intra-region label localization, as shown by the gap between (b) and (c) of the table. Lastly, label expansion further boosts the performance by 1.2%p on average.
Qualitative analysis on pseudo labels. Fig. qualitatively compares region- dominant class labels and pixel-wise pseudo labels given by the proposed method show the label disambiguation ability of our method. Regions labeled with dominant classes overlook areas occupied by minor classes. In contrast,
intra-region localization accurately identifies various classes within each region as shown in the second column of Fig. 6. Moreover, label expansion augments the amount of supervision by referencing the class composition within the region. Notably, minor classes within region often significantly enhance the quality of pseudo labels via label expansion.
Ablation study of pseudo labeling. Fig. 7
of la labell localization (%%
Gain of label expansion (%)
2.0
presents our ablation study on pseudo label- ing with ResNet50 Cityscapes. Fig. 7(a)
1.5
compares performance improvement by the intra-region label localization (w/ proto-
1.0
0.5
types) and that by a baseline assigning the most confident class among multi-class la- bels as pixel-wise pscudo label (w/o pro- totypes). The result suggests that using prototypes consistently surpasses the base- line across different budgets due their adaptability to local regions. In Fig. 7(b), we investigate the improvement when la- bel expansion is applied to multi-class and dominant class labels across varying bud- gets. It turns out that label expansion is more effective with multi-class labeling, as enables the propagation of pscudo labels belonging to multiple classes, leading to more extensive expansion of pseudo labels.
0.5
0.0
0.0
wit patntype
Dom
prilulyue
-0.5
-0.5 -0.5 100K 200K 300K 400K 500K
100K 200K 300K 400K 500K Number of clicks
Number of clicks
(a) Label localization
(b) Label expansion
Figure 7: mloU gain (%) from intra-region label local- ization and label expansion varying budgets. (a) Gain of label localization with vs. without prototype. (b) Gain of label expansion on dominant and multi-class labels.
Impact of region generation algorithm. In Fig. 8(a), we evaluate both dominant class labeling (Dom) and multi-class labeling (Mul) across two superpixel generation algorithms: SLIC [1] and SEEDS [59]. Both labeling methods show better performance when combined with SEEDS. This is because SEEDS is better than SLIC in terms of boundary recall [57] and thus regions generated by SEEDS better preserves the class boundary. Note that the dominant class labeling shows significant performance degradation when combined with SLIC, while the multi-class labeling only shows modest performance drop. This result suggests that the proposed multi-class labeling is more robust to the quality of the superpixel generation algorithm.

---

75.0
75.0
72.5
70.0
70.0
mloU (%)
67.5
65.0
E 65.0
Dam-SP64
Dom-SP32
62.5
Dom+SUic
Dam-SP16
Dom+SEEDS
Mul+
60.0
Mul+SUC
60.0
Mul+SP32
Mui+ SEEDS
Mul+
57.5
100K
200K
300K
400K
500K
100K
200K
300K
500K
Number of clicks
Number of clicks
(a)
(b)
Figure 8: Accuracy in mloU (%) versus the number of clicks (budget) for dominant class labeling (Dom) and multi-class labeling (Mul) equipped with PixBal evaluated on Cityscapes using ResNet50. (a) Impact of superpixel generation algorithm: SEEDS [59] and SLIC [1]. (b) Impact of superpixel size: 16x 16 (SP16), 32x32 (SP32), and 64 x64 (SP64).
71.5
71.5
71.5
mlou (%)
71.0
mlou (%)
71.0
70.5
71.0
70.5
70.0
69.5
69.5
70.0
69.0
10
69.5
12
16
20
68.5
16
ACE
AMP
(a)
(b)
Figure 9: Average accuracy of our stage model over rounds, in mIoU (%), as a function of varying hyperparameters. The model is evaluated on Cityscapes using ResNet50 backbone with PixBal sampling. (a) The class balancing regulation term v. (b) Loss balancing term AMP (c) Loss balancing term ÅCE. The blue diamond marker indicates the value selected for our final model.
Impact of region size. In Fig. 8(b), we evaluate dominant class labeling (Dom) and multi-class labeling (Mul) across different superpixel sizes: 16x 16 (SP16), 32x32 (SP32), and 64x64 (SP64). Both methods achieve their best with the 32x32 region size. Note that dominant class labeling leads large performance drop when the region size increases from 32x32 64 since a larger region is more likely to contain multiple classes, which violates the key assumption of dominant class labeling. In contrast, multi-class labeling is more robust against such region size variations.
Impact of perpaaarreterss. In Fig. 9, we evaluate the sensitivity of our stage model to variations in the hyperparameters: v, AMP. and Acn. This evaluation conducted Cityscapes using ResNet50 backbone and combined with PixBal sampling. Our model demonstrates robustness to these hyperpa- rameter changes, with accuracy fluctuations of less than 1.5%. It's noteworthy that final model does not use the optimal hyperparameter values. This indicates that we did not exhaustively tune these parameters using the validation set.
Conclusion
We have introduced novel multi-class label query for AL in semantic segmentation. Our two- stage training method and new acquisition function enabled an AL framework with the multi-class label query to achieve the state of the art on two public benchmarks for semantic segmentation by improving label accuracy
reducing labeling cost. Some issues still remain for further exploration. As it stands, our method depends on off-the-shelf superpixels and does not transfer the pseudo labels the next rounds. Next on our agenda is to address these issues by learning image over-segmentation and developing advanced AL pipeline.
10

---

Acknowledgments and Disclosure of Funding
We sincerely appreciate Moonjeong Park for fruitful discussions. This work was supported by the NRF grant and the ITTP grants funded by Ministry of Science and ICT, Korea (NRF-2021R A JTTP-2020-0-00844 ITT-2022-0-00290, LTTP-2021-0002666 IITP-2019-0-01906, IITP-2021-0- 00739).
References
[1] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions pattern analysis and machine intelligence, 34(11):2274 2282, 2012.
[2] Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly supervised learning of instance segmentation with inter-pixel relations. In Proc. JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[3] Jiwaon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In Proc. IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2018.
[4] Inigo Alonso, Alberto Sabater, David Ferstl, Luis Montesano, and Ana C Murillo. Semi-supervised semantic segmentation with pixel-level contrastive learning from a class-wise memory bank. In Proc. IEEE/CVF International Conference Computer Vision (ICCV), 2021.
[5] Nabiha Asghar, Pascal Poupart, Xin Jiang, and Hang Li. Deep active learning for dialogue generation. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics ( (*SEM), 2017.
[6] Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In Proc. International Conference on Learning Representations (ICLR), 2020.
[7] Clemens-Alexander Brust, Christoph Käding, and Joachim Denzler. Active learning for deep object
detection, arXiv preprint arXiv: 09875, 2018.
[8] Vivien Cabannnes, Alessandro Rudi, and Francis Bach. Structured prediction with partial labelling through the infimum loss. In Proc. International Conference on Machine Learning (ICML), 2020.
[9] Lile Cai, Xun Xu, Jun Hao Liew, and Chuan Sheng Foo. Revisiting superpixels for active learning in semantic segmentation with realistic annotation costs. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[10] Arantxa Casanova, Pedro O Pinheiro, Negar Rostamzadeh, and Christopher J Pal. Reinforced active learning for image segmentation. In Proc. International Conference on Learning Representations (ICLR), 2020.
[11] Liyi Chen, Weiwei Wu, Chenchen Fu, Xiao Han, and Yuntao Zhang. Weakly supervised semantic segmentation with boundary exploration. In Proc. European Conference on Computer Vision (ECCV), 2020.
[12] Liang-Chich Chen, Raphael Gontijo Lopes, Bowen Cheng, Maxwell D. Collins, Ekin D. Cubuk, Barret Zoph. Hartwig Adam, and Jonathon Shlens. Naive-student: Leveraging sumi-supervised learning in video sequences for urban scene segmentation. In Proc. European Conference on Computer Vision (ECCV), 2020.
[13] Liang- Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. Proc. European Conference on Computer Vision (ECCV), 2018.
[14] Pascal Colling. Lutz Roese-Koerner, Hanno Gottschalk, and Matthias Rottmann. Metabox+: A new region based active learning method for semantic segmentation using priority maps. International Conference on Pattern Recognition Applications and Methods (ICPRAM), 2021.
[15] Marius Cordts, Mohamed Omran. Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson. Uwe Franke, Stefan Roth, Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[16] Timothee Cour, Ben Sapp. and Ben Taskar. Learning from partial labels. In Journal Machine Learning Research (JMLR), 2011.
[17] Chengliang Dai, Shuo Wang. Yuanhan Mo. Kaichen Zhou, Elsa Angelini, Yike Guo, and Wenjia Bai. Suggestive annotation of brain tumour images with gradient-guided guided sampling. In Proc. Medical Image Computing and Computer-Assien
Intervention (MICCAI), 2020.
[18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: a large-scake hierarchical image database. In Proc. IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). 2009.
[19] Thomas G. Dietterich, Richard H. Lathrop. and Tomás Lozano- Pérez. Solving the multiple instance problem with axis- parallel rectangles. Artificial Intelligence, 1997.
11

---

[20]
Zhang Dong. Zhang Hanwang. Tang Jinhui, Hua Xiansheng, and Sun Qianru. Causal intervention for weakly supervised semantic segmentation. In Proc. Neural Information Processing Systems (NeurIPS), 2020.
[21] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) Challenge. International Journal of Computer Vision (IJCV), 2010.
[22] Lei Feng. Jiaqi Lv. Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. Proc. Neural Information Processing Systems (NeurlPS). (0998-1999600 2020.
[23] S Alireza Golestanch and Kris M Kitani. Importance of self-consistency in active learning for semantic segmentation. In Proc. British Machine Vision Conference (BMVC), 2020.
[24] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. Unsuper- vised semantic segmentation by distilling feature correspondences. In Proc. International Conference on Learning Representations (ICIR), 2022.
[25] Ruifei He. Jihan Yang. and Xiaojuan Qi. Re-distributing biased pseudo labels for semi-supervise' semantic segmentation: A baseline investigation. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV),
[26] Tao He, Xiaoming Jin, Guiguang Ding, Lan Yi, and Chenggang Yan. Towards better uncertainty sampling: Active learning with multiple views for deep convolutional neural network. In IEEE International Conference on Multimedia and Expo (ICME), 2019.
[27] Tong He, Zhi Zhang. Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proc. IEEF/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), pages 558-567, 2019.
[28] Zilong Huang, Xinggang Wang. Jiasi Wang. Wenyu Liu, and Jingdong Wang. suaklyysdperrised semantic segmentation network with deep seeded region growing. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018
[29] Eyke Hüllermeier and Jürgen Beringer. Learning from ambiguously labeled examples. Intelligent Data Analysis (IDA). 2006.
[30] Sehyun Hwang. Sohyun Lee, Sungycon Kim, Jungseul Ok, and Suha Kwak. Combating label distribution shift for active domain adaptation. In Proc. European Conference on Computer Vision (ECCV), pages 549-566. Springer, 2022.
[31] Suyag Dutt Jain and Kristen Grauman. Active image segmentation propagation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2016.
[32] Ajay J Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classifi- cation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009. [33] Tarun Kalluri, Girish Varma, Manmohan Chandraker, and C V Jawahar. Universal semi-supervised semantic segmentation. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[34] Dahyun Kang, Piotr Koniusz, Minsu Cho, and Naila Murray. Distilling self-supervised vision transformers for weakly-superrisee few-shot classification & segmentation, In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19627-19638, 2023.
[35] Tejaswi Kasarla, Gattigorla Nagendar, Guruprasad M Hegde, Vinecth Balasubramanian, and CV Jawahar. Region-base based active learning for efficient labeling in semantic segmentation. In Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2019.
[36] Rihuan Ke, Angelica Aviles-Rivero, Saurabh Pandey, Saikumar Reddy, and Carola- Schönlieb.
A three-stage self-training framework for semi-supervised semantic segmentation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[37] Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson W.H. Lau. Guided collaborative training for pixel-wise semi-supervised learning. In Proc. European Conference on Computer Vision (ECCV), 2020.
[38] Hoyoung Kim, Minhyeon Oh. Sehyun Hwang, Suha Kwak. and Jungseul Ok. Adaptive superpixel for active learning in semantic segmentation. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[39] Namyup Kim, Sehyun Hwang, and Suha Kwak. Learning to detect semantic boundaries with image-level
class labels. International Journal of Computer Vision (IJCV), 1300))2111-22448 2022.
[40] Suha Kwak, Seunghoon Hong, and Bohyung Han. Weakly supervised semantic segmentation using superpixel pooling network. In Proc. AAAI Conference on Artificial Intelligence (AAAI), 2017.
[41] Doughyeon Kwon and Suha Kwak. Semi-supervised semantic segmentation with error localization network. In Proc. JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
[42] Xin Lai, Zhuotao Tian, Li Jiang, Shu Liu, Hengshuang Zhao, Liwei Wang, and Jiaya Jia. Semi-supervised semantic segmentation with directional context-aware consistency. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[43] Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon. Ficklenet: Weakly and semi- supervised semantic image segmentation using stochastic inference. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
12

---

[44] Daiqing Li. Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In Proc. IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
[45] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proc. International Conference on Learning Representations (ICLR), 2019.
[46] Jiaqi Lv, Miao Xu, Lei Feng. Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In Proc. International Conference on Machine Learning (ICML). 2020.
[47] Radek Mackowiak, Philip Lenz, Omair Ghori, Ferran Diego, Oliver I. Lange, and Carsten Rother. Cereals- cost-effective region-based active learning for semantic segmentation. In Proc. British Machine Vision Conference (BMVC), 2018.
[48] Robert Mendel, Luis Antonio de Souza Jr, David Rauber, João Paulo Papa, and Christoph Palm. Semi- supervised segmentation based on crror-correcting supervision. In Proc. Europeann Conference on Computer Vision (ECCV), 2020.
[49] Sudhanshu Mittal, Maxim Tatarchenko, and Thomas Brox. Semi-super semantic segmentation with high- and low level consistency. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2019.
[50] Natalia Ostapuk, Jie Yang, and Philippe Cudré-Mauroux, Activelink: deep active learning for link prediction knowledge graphs. In The World Wide Web Conference (WWW). 2019.
[51] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Zeming Lin, Alhan Desmaison, I Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In AutoDiff NIPS Workshop, 2017.
[52] Congyu Qiao, Ning Xu, and Xin Geng. Decompositional generation process for ristane--dpenddee partial label learning. In Proc. International Conference on Learning Representations (ICLR), 2023.
[53]
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In Proc. International Conference on Learning Representations (ICLR), 2018.
[54] Gyungin Shin, Weidi Xie, and Samuel Albanic. All you need are a few pixels: semantic segmentation with pixelpick. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[55] Yawar Siddiqui, Julien Valentin, and Matthias Nießner. Viewal: Active learning with viewpoint entropy for semantic segmentation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[56] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active lcarning. In Proc. IEEE/CVF International Conference Computer Vision (ICCV), 2019.
[57] David Stutz, Alexander Hermans, and Bastian Leibe. Superpixels: An evaluation of the state-of-the-art. Computer Vision and Image Understunding, 166: 166,1-27,22 2018.
[58] Guolei Sun, Wenguan Wang, Jifeng Dai, and Luc Van Gool. Mining cross-image semantics for weakly supervised semantic segmentation. In Proc. European Conference on Computer Vision (ECCV), 2020. [59] Michael Van den Bergh, Xavier Boix, Gemma Roig, Benjamin de Capitani, and Luc Van Gool. Seeds: Superpixels extracted via energy-driven sampling. In Proc. European Conference on Computer Vision (ECCV), 2012.
[60] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Unsupervised semantic segmentation by contrasting object mask proposals. In Proc. JEEE/CVF International Conference Computer Vision (ICCV), 2021.
[61] Haobo Wang, Mingxuan Xia, Yixuan Li, Yuren Mao, Lei Feng, Gang Chen, and Junbo Zhao. Solar: Sinkhorn label refinery for imbalanced partial-label lcarning. In Proc. Neural Information Processing Systems (NeurIPS), 2022.
[62]
Keze Wang. Dongyu Zhang, Ya Li, Ruimao Zhang, and Liang Lin. Cost-effectivee active learning for deep image classification. IEEE Transactions on Circaits and Systems for Video Technology (TCSVT), 2016.
[63] Xiang Wang, Shaodi You, Xi Li, and Huimin Ma. Weakly-supervised semantic segmentation hy iteratively mining common object features. In Proc. IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[64] Zengmao Wang. Bo Du. Weiping Tu, Lefei Zhang, and Dacheng Tao. Incorporating distribution matching into uncertainty for multiple kernel active learning, IEEE Transactions on Knowledge ond Data Engineering (TKDE), 2019.
[65] Zheng Wang and Jieping Ye. Querying discriminative and representative samples for batch mode active learning. ACM Transactions on Knowledge Discovery from Data (TKDD). 2015.
[66] Tsung-Han Wu, Yi-Syuan Liou, Shao-Ji Yuan, Hsin-Ying Lee, Tung-I Chen, Kuan-Chih Huang, and Winston H Hsu. D2ada: Dynamic density-aware active domain adaptation for semantic segmentation. In Proc. European Conference on Computer Vision (ECCV), pages 449-467, 2022.
[67] Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, and Xinjing Cheng. Towards fewer annotations:
Active learning via region impurity and prediction uncertainty for domain adaptive semantic segmentation.
In Proc. IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
13

---

[68] Lin Yang, Yizhe Zhang. Jianxu Chen, Siyuan Zhang, and Danny Z Chen. Suggestive annotation: A deep active learning framework for biomedical image segmentation. In Proc. Medical Image Computing and Computer-Asistte Intervention (MICCAI). 2017.
[69] Adrian Ziegler and Yuki M Asano. Self- learning of object parts for semantic segmentation. In Proc. IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). 2022.
14

---

Active Learning for Semantic Segmentation with Multi-class Label Query Appendix
This appendix provides additional experimental details and findings that have been omitted in the main paper due to the page limit. Sec. presents thorough explanation of the user study conducted to compare the cost of dominant class labeling and multi-class labeling, In Sec. B, we explain further details of configurations of our implementation. Sec. C presents additional experiments including the effect of the budget size (Sec. C.I), comparison with partial label learning loss baselines (Sec. C.2), a comparison with pseudo labeling baselines (Sec. C.3), a comparison with baseline acquisition functions (Sec. C.4), and a qualitative result of our final model (Sec. C.5).
A Details of user study
We conducted a user study to compare the dominant class labeling and multi-class labeling in terms of actual labeling cost and accuracy versus the number of classes in region queries. The examples of the questionnaire are illustrated in Fig. 10 and the results are summarized in Table 3. As shown in Fig. 10(a), for each question, annotators received an instruction, an image patch along with marked local region, and class options. They were requested to select the relevant class options as directed by the instruction. The instructions for dominant class labeling and multi-clas, labeling were as follows:
"Select the dominant class that corresponds to the inside of the red boundary.". "Select the all classes that exist within the red boundary.".
Prior to the survey, we ensured that every participant reviewed the pre-survey instructional material. This material covered the class composition of Cityscapes, offered the definition of the dominant class and the multi-class labeling, and provided example questions. The pre-survey instructional material is included in the supplementary materials under the name "pre-survey.pdf.
As shown in Fig. 10(b), each image patch was a 360-pixel square mostly centered on a local region. Using ground-truth segmentation mask, divided regions into three groups based the number of classes (from 1 to 3) present in each region. Twenty regions were then randomly selected from each group for each survey, excluding those containing pixels irrelevant the original 19 classes, referred to as the 'undefined" class.
Select the all classes that exist within the red boundary.
class
classes
classes
Road
Bailing
Thaffc sign
Tralfeligt
M21 48:02
Tenoa
Royole
Track
Sdowwikk
Sky
Trane
Monoroyle
Bide
Feece
(a) Questionnair for multi-class labeling
(b) Example queries with different number of classes
Figure 10: Questionnaire and local region examples used in the user study. (a) Questionnaire of multi-class labeling survey, consisting of instruction, image patch along with local region marked with red boundary, and class options allowing multiple selections. (b) Examples of local regions used in the user study according to the number of classes present in each region.
15

---

Table 3: The result of user study showing the labeling time (second) and accuracy (%) of dominant class labeling and multi-class labeling according to the number of classes within each region.
Query
# of classes
Total time (s)
Total clicks
Time per click (s)
Accuracy (%)
1
127.6 3.7.4
20.0
6.38
95.63:1.00
2
160.5+3330
20.0
8.02:11.65
:2.05
Dominant
3
172. 1335.8
20.0
8.60#1.79
65.83+1..7
average
153.4
20.0+0.0
7.67=2.005
17.84
1
145.5+11.8
21.6+2.11
6.75=11.65
95.97=11.10
2
191.6 101.6615.8
39.113.7
1.89
87.14
Multi-class
3
295.8165..
49.018.6
6.37 11.51
71.52188.2
average
211.0186.3
36.5112.5
6.01 11.75
84.88
total 45 volunteers participated in the survey. We report the results excluding five cases considered outliers in terms of time and accuracy. A unique survey prepared for each group of regions, categorized by the number of classes. Given three groupings and two labeling methods, a prepared. If annotator annotates the same region twice, there would
total of six unique forms be a risk of memorizing the image during the first annotation. To avoid this, we asked each participant answer three out of the six forms, ensuring region annotated twice by the same person.
The responses from annotators are evaluated by calculating the Jaceard Similarity (IS) between the ground-truth class set and the responded class set. We define the JS of annotator follows:
G Yial
u =
Σ EXX
(14)
X
G.UYM
where X is a set of regions, G; is ground truth multi-class label and Y is the set of classes selected by annotator for region i. Note that for dominant class labeling, |G,| [Yi,ul 1. We compute the final accuracy as the average IS of all annotators, given by:
Accuracy
Σ JS(u). DEU
(15)
TUT
As shown in Table 3, multi-class labeling demonstrates comparable efficiency to dominant class labeling for regions with single class. Moreover, when it comes to regions with multiple classes, multi-class labeling requires less annotation time per click compared to the dominant class labeling.
B Implementation details
Configurationss We implement our method using the PyTorch framework [51]. Following the previous literature [27], we make slight modification to the original ResNet architecture by replacing the initial 7x7 convolutional layer with two 3x 3 convolutional layers. The output stride of the network is set to 16. We set the learning rate of the backbone to be ten times lower than the standard rate, and we apply a weight decay of 1c-5. During the training phase, we incorporate several data augmentation techniques, including random scaling ranging from 0.5 to 2.0, random cropping, and random horizontal flipping. To ensure reproducibility and test the robustness of our approach, we conduct three independent experiments, each initialized with different seed values: 0, 1, and 2.
Label generation. Following previous work [9], we assign both dominant class labels and multi-class labels to each region using the ground-truth mask labels. For the dominant class label, we assign the class that dominates the majority of pixels within each region, in accordance with its definition. For the multi-class labels, we attribute all existing classes present within each region. Notably, we disregard classes that appear minimally along the region's boundary during this process. This procedure reflects realistic scenarios where a labeler might fail to recognize classes represented insignificantly on the boundary. More specifically, we implement binary dilation operation with 5x5 kernel along the region boundaries, consequently excluding classes that appear on these expanded boundaries. the user study, we reflect this by marking each local region with a thick, translucent boundary.
16

---

68.0
75.0
75.0
65.5
63.0
72.5
60.5
mlou (%
58.0
70.0
55.5
67.5
53.0
65.0
48.0
Dominant label
625
Dominant label
62.5
Dominant label
45.5
Mult-cllass label
60.0
Multi-class label
Multi-class label
10K
20K
30K
40K
50K
50K www.ko0...oc..ooooooooo www..a/...........o......
100K
200K 300K 400K 500K
Number of clicks
Number of clicks
Number of clicks (c)
(a)
(b)
Figure 11: Accuracy in mloU (%) versus the number of clicks (budget) evaluated on Cityscapes using ResNet50 with PixBal. We evaluate three budget scenarios: (a) 10K clicks over 5 rounds, (b) 50K clicks over 10 rounds, and (c) 100K clicks over 5 rounds.
Handling undefined class. In the Cityscapes dataset [15], pixels covered by the original 19 semantic classes are typically ignored when training segmentation models. On the other hand, in multi-class labeling setting, the precise locations of such uncovered pixels remain unspecified since the multi-class label only provide partial labels. Treating such pixels as belonging to one of the 19 semantic classes naively can misguide the model by providing confusing supervision. Furthermore, active sampling methods like BvsB, ClsBal, PixBal tend to prefer uncertain regions, often leading to the selection of regions containing these uncovered pixels, despite their lack of utility. To address this, we assign an additional undefined class for pixels not covered by the initial 19 classes and train the model to predict these undefined classes. As for active sampling, we introduce an extra condition to exclude regions where the predicted dominant class is the undefined class. This undefined class handling strategy implemented for both dominant class labeling and multi-class labeling.
Insight for choosing typerparameters. We further describe the detailed insights to determine the importance of each loss term; λMP. and λCE. We roughly determined them by considering the scale of the loss signal paok-propagated per pixel. Lpp is applied to only a tiny subset of pixels in an image, i.e., pixels whose number the same as the number of classes per region (i.e., superpixel). Considering that the final loss value is computed as an average over the pixels, the limited pixel domain of Lpp implies overly stronger loss signals compared with other losses. To counterbalance this side effect, we strategically assigned larger weights to λCE and AMP. which are associated with larger number of pixels. The values of λCE and AMP were estimated by grid search on the Cityscapes dataset and were used as-is on PASCAL VOC 2012 as well.
C Additional experiments
C.1 Impact of budget size
In Fig. 11, we evaluate the accuracy of our final model and dominant class labeling with different budget scenarios: 10K clicks over 5 rounds, 50K clicks over 10 rounds, and 100K clicks over rounds. This analysis is performed on Cityscapes, using ResNet50 backbone combined with PixBal sampling. As depicted in Fig. 1 la))), even in the tiny budget of 10K clicks, the multi-class labeling outperforms the dominant class labeling baseline. This highlights the utility our method in extremely constrained budget scenarios, where the need for active learning is emphasized. As shown Fig. 11(b) and (c), when the total budget kept constant, increasing the frequency of active sampling and model training enhances performance. This improvement can be attributed to the more frequent interactions between the model and the Oracle, leading more informative active sampling. Multi-class labeling consistently outperforms dominant-class labeling in a wide range of budget sizes, demonstrating the efficacy of our method.
C.2 Comparison with partial label learning loss baselines
In Fig. 12(a), we compare our proposed losses with baseline partial label learning losses: Infimum loss [8], and RC loss [22, 46]. This comparison is performed on Cityscapes, using ResNet50 backbone combined with PixBal sampling, As shown in Fig. 12(a), the model with proposed losses
17

---

74
72
mloU (%
68
70
mlou
66
68
Local+Expand
64
Lus Lae
w/ proto
Local
Infimum loss
w/o proto
Dom+Expand
62
64
Stage
100K
300K 400K 500K
100K 200K 300K 400K 500K
100K
300K
40DK 500K
Number of clicks (a)
Number of clicks (b)
Number of clicks (C)
Figure 12: Accuracy in mloU (%) versus the number of clicks (budget) evaluated Cityscapes using ResNer50 backbone, with PixBal sampling. (a) The accuracy of our stage model trained with proposed losses (CMP + Lrp), compared with baseline partial label learning losse: Infimum loss [8], and RC loss [22,46)- (b) The accuracy of stage multi-class labeling model (stage 1), stage multi-class labeling model solely employing intra-region label localization (w/ proto), and baseline pseudo labeling method without prototype (w/o proto). (c) The accuracy of stage 2 multi-class labeling model with (Xxaal+Expand) and without (Local) label expansion, compared with dominant class labeling model with (Dom+Expand) and without (Dom) label expansion.
76.0
miou (%)
71.0
Random
KNN
Margin
ClassBal
PixBal
100K
200K
300K
400K
500K
Number clicks (ResNet50)
Figure 13: Accuracy in mloU (%) versus the number of clicks (budget) for multi-class labeling evaluated Cityscapes equipped with six different acquisition function: Random, KNN, Margin, BvSB, ClassBal, and our sampling strategy (PixBal). The reported accuracy scores are averaged across three trials.
(LMP + Lpp) demonstrates superior performance the models using the baseline losses, particularly in the early rounds.
C.3 Comparison with pseudo labeling baselines
In Fig. 12(b), we conduct an ablation study comparing our proposed intra-region label localization method (denoted as 'w/ proto') with baseline intra-region pseudo labeling method that assigns the most confident class among multi-class labels as the pixel-wise pseudo label (denoted as "w/o proto") ) This comparison takes place on the Cityscapes dataset, utilizing a ResNet50 backbone in conjunction with PixBal sampling. While both of the intra-region pscudo-labeling methods improve upon the label localization demonstrates superior performance
stage model. the proposed prototype- over the baseline, specifically in the initial round, where the stage model may lack accuracy.
In Fig. 12(c), we compare the performance improvement brought by label expansion when applied to both the multi-class labeling model (denoted as 'Local+Expand") and the dominant class labeling model (denoted as 'Dom+Expandd), This comparison also conducted on the Cityscapes dataset, using a ResNet50 backbone paired with PixBal sampling. As shown in Fig. 12(c), label expansion proves to be more beneficial when used with multi-class labeling, as allows the spread of pseudo labels across multiple classes, resulting in broader expansion of pscudo labels.
18

---

C.4 Additional comparison with baseline acquisition functions
In Fig. 13, we evaluate multi-class labeling combined with 6 different acquisition functions: Random, wewsus-Seeeond-eeeeeeee Best (BvSB), Class Balanced sampling
K K-Nearest Neighbors (KNN), Margin, (ClassBal), and our sampling strategy (PixBal). PixBal consistently outperforms all the others regardless of budget size. Margin shows decent performance with large budget, e-g., it surpasses ClassBal when using 500K clicks. On the other hand, KNN offers modest improvements, marginally surpassing the performance of Random.
C.5 Qualitative results of our final model
Fig. 14 provides a qualitative result of the predictions of our final model at different rounds. As illustrated in Fig. 14, the quality of the predictions markedly improves as the rounds progress. Notably, the predictions produced by our final model round exhibit impressive quality, especially when taking into account that it requires only 9.8% of the labeling cost associated with a fully supervised model.
19

---

يجا
(a)
(b)
(c)
(d)
(e)
Figure 14: Qualitative results of our final model on the Cityscapes dataset. (a) Inputs. (b) Prediction
round (c) Prediction in round 3. (d) Prediction round 5. (e) Ground Truth.
20


Reviews:
[{"review_id": "LYEJRXwfbH", "reviewer": "Reviewer_DZFu", "summary": "This paper proposes an active learning approach for semantic segmentation using multi-class label queries. Different from dominant class labeling methods, where an oracle is asked to select the most dominant class by a single click, this paper instead designs a multi-class labeling approach that asks the oracle for a multi-hot vector that indicates all classes existing in the given region. Experiments on two benchmarks, Cityscapes and PASCAL VOC 2012, demonstrate promising performance with a significant reduction in annotation cost.", "questions": "As mentioned in Weaknesses.", "limitations": "The authors have stated the limitations and provided the future research directions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Reducing annotation costs for segmentation tasks is a long-standing and crucial task. This paper considers active learning to select informative regions from training data and asks an oracle to label them on a limited budget, which is a potential research direction to allow large-scale and annotation-efficient training for segmentation tasks.\n2. The proposed method is reasonable and technically sound.\n3. The experimental results are comprehensive and verify the effectiveness of the proposed method.\n4. The overall paper is well-written and easy to follow.", "weaknesses": "1. In stage 1, the total training objective comprises three loss terms. While the sensitivity analysis of the balanced terms $\\lambda$ is provided in the supplementary material, I am still wondering whether any findings or insights to determine the importance of each loss term.  More insights or clarifications are encouraged.\n2. In Figure 7(a), why does the gain of label localization (%) from the method w/ prototype decrease when the number of clicks increases? More clarifications are suggested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback and constructive comments that help improve our paper substantially. We will incorporate all the suggestions into the paper. Please find our detailed responses to the comments below.\n___\n**Q1. Insights to determine the importance of each loss term**\n\nA1. We appreciate your detailed comment. We roughly determined the significance of the loss terms by considering the scale of the loss signal back-propagated per pixel. $L_{PP}$ is applied to only a tiny subset of pixels in an image, i.e., pixels whose number is the same as the number of classes per region (i.e., superpixel). Considering that the final loss value is computed as an average over the pixels, the limited pixel domain of $L_{PP}$ implies overly stronger loss signals compared with other losses. To counterbalance this side-effect, we strategically assigned larger weights to $L_{CE}$ and $L_{MP}$, which are associated with a larger number of pixels.\n\nThe values of $\\lambda_{CE}$ and $\\lambda_{MP}$ were estimated by a grid search on the Cityscapes dataset and were used as-is on PASCAL VOC 2012 as well. Figure 2 of the supplementary material demonstrates that the values of $\\lambda_{CE}$ and $\\lambda_{PP}$ in the main paper are not optimal, which indicates that their values are not exhaustively tuned and that our method is not sensitive to the hyper-parameters.\n\nWe will elaborate on the process of determining $\\lambda_{CE}$ and $\\lambda_{MP}$ in the manuscript.\n\n___\n**Q2. In Figure 7(a), why does the gain of label localization (%) from the method w/ prototype decrease when the number of clicks increases?**\n\nA2. Thank you for the insightful comment. We first would like to clarify that the values in Figure 7(a) denote relative improvements over the performance of our stage 1 model. Absolute values are in Figure 4(b) of the supplementary material, confirming that our prototype-based pseudo labeling consistently outperformed the baseline across all budgets.\n\nThe diminishing performance gains with more clicks in the 'w/ prototype' method can be linked to its rapid convergence due to its high efficacy. This behavior is evident from Figure 4(b) of the supplementary material, where the 'w/ prototype' performance converges rapidly as clicks increase. As models receive more supervision, it is natural for them to converge to the upper bound of supervised learning performance. This holds true for both the 'w/ prototype' and 'w/o prototype' methods. However, the 'w/ prototype' method seems to reach this upper bound faster, probably due to its superior performance over the 'w/o prototype' method, resulting in a reduction in the rate of performance gain as clicks increase.\n"}, {"review_id": "WhXJpYoeea", "reviewer": "Reviewer_9EZK", "summary": "This work introduces a new active learning framework for semantic segmentation, involving a novel query design that uses a multi-hot vector for class representation with in a region, two novel loss functions for effective multi-class label supervision, and an acquisition function considering class uncertainty and balance in a local region. This approach optimizes label usage, enhancing supervision and efficiency in semantic segmentation tasks. ", "questions": "- In line 150, Why is Dm represented as D minus Ds? Shouldn't Dm include Ds.\n- In Figure 1, the Annotation Click for Dominant Class Labeling and Multi-class Labeling are 12.12s and 17.13s, respectively. The author should cite relevant references to make this data convincing to the reader.\n- In Figure 4, we see that all methods based on multi-class labeling perform better than those based on dominant class labeling. In my opinion, this comparison is not quite fair because multi-class labeling inherently provides more information than dominant class labeling, and thus better performance is to be expected.", "limitations": "Please see the weakness.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The motivation of this work is clear and interesting. In my opinion, the main strength of this paper lies in the proposed query based on multi-class labels. To be specific, the traditional dominant class labeling suffer from more annotation time per click and bad performance. Therefore, this paper proposed a novel multi-class labeling query and customized two loss and acquisition function for multi-class labeling to handle this problem.\n2) The quantitative experimental results are impressive. Only 4% clicks can reach 95% mIoU of the fully-supervised model.\n3) Most of this paper are easy to understand.\n", "weaknesses": "- In my view, the Merged Positive Loss and Prototypical Pixel Loss proposed in this paper are essentially minor modifications to the Cross Entropy Loss from a technical standpoint. The Acquisition Function proposed in the paper simply introduces a hyper-parameter to the existing Acquisition Function, incorporating the concept of multi-class. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful feedback and suggestions that helped improve our paper substantially. All the suggestions will be incorporated into the main text and the appendix. Please find our detailed responses to the comments below.\n___\n**Q1. Clarification of our contributions from a technical standpoint**\n\nA1. Thank you for the valuable comment. Our primary contribution is that we are the first to introduce multi-class labeling to active learning for semantic segmentation. Also, the proposed loss and acquisition functions are essential and carefully designed for learning with multi-class labels as you kindly mentioned in the 'Strengths' section. Below, we elaborate on the technical contributions of our work.\n\n**The proposed loss functions**\n\nFirst, we would like to emphasize that the standard cross-entropy loss cannot be used for learning semantic segmentation using region-wise multi-class labels since such a label cannot be converted to pixel-wise annotation.\n\nWhile the proposed losses are based on the standard cross-entropy as commented, we newly propose to transform multi-class label information into a trainable supervisory signal compatible with cross-entropy (e.g., the merged positive loss takes the sum of predictive probabilities for candidate classes as positive predictions, and the prototypical pixel loss dynamically selects the most likely pairs of a candidate class and a pixel). As reviewer FaJW recognized, we believe these losses could be insightful. \n\nMoreover, our losses are designed to complement each other. The merged positive loss supplies a weak yet consistent training signal across all pixels in a region, ensuring that any possible noise from the prototypical pixel loss does not misguide the overall training. On the other hand, the prototypical pixel loss gives a stronger signal to a few pixels and is applied uniformly among classes, which helps avoid class imbalance.\n\nTable 1 of the main paper empirically shows that both of the two losses consistently enhance segmentation performance across 5 rounds. In Section D.1 and Figure 4(a) of the supplementary material, we also demonstrate the superiority of the losses over baseline partial-label learning losses such as infimum loss and RC loss.\n\n**The proposed acquisition function**\n\nThe proposed acquisition function is also aligned with multi-class labels by accounting for all classes present in a region, in contrast to the prior acquisition function (Cai et al., CVPR 2021 [8]) targeting only the dominant class. While it is true that our acquisition function introduces an additional hyper-parameter, denoted as $\\nu$, Figure 2(a) of the supplementary material demonstrates that the performance of our method is insensitive to changes in $\\nu$.\n\n___\n**Q2. Clarifying the definition of $D_m$**\n\nA2. We define $D_m$ as the set of regions labeled with multiple classes. Specifically,\n$$  D_m := \\\\{(s,Y): \\exists (s,Y) \\in D, |Y| > 1\\\\}\\\\;\\\\;,$$\nwhere $s$ represents a region, and $Y$ denotes the set of classes present within $s$. Thus, $D_m$ does not contain $D_s$.\nThe rationale behind this exclusion is that regions in $D_s$ already have precise pixel-wise annotations, making $L_{MP}$ and $L_{PP}$ unnecessary. We will clarify the definitions of $D_m$ and multi-class labels in the revision.\n\n___\n**Q3. The annotation time in Figure 1**\n\nA3. Figure 1 of the main paper presents the results of a user study. 12.12s and 17.13s are the actual average times it took for three people to annotate the region on the left-hand side of the figure with dominant and multi-class labels, respectively. Meanwhile, the right-hand side summarizes the average annotation times across all surveyed regions. Details of this survey can be found in Section A of the supplementary material, and example questions of the survey are presented in 'pre-survey.pdf'. We will make sure to include these details in the caption of Figure 1 of the main paper.\n\n___\n**Q4. Fair comparisons with dominant class labeling in FIgure 4**\n\nA4. We emphasize that the number of clicks is adopted as our budget metric in Figure 4 of the main paper for fair comparisons with dominant class labeling. Please note that a multi-class label involving $n$ classes requires $n$ clicks to be annotated, i.e., given the same number of clicks, multi-class labeling leads to fewer labeled regions compared with those of dominant class labeling. Hence, multi-class labeling does not provide more information given the same number of clicks, and accordingly, the comparisons in Figure 4 are not favorable to multi-class labeling.\n\nActually, the comparisons are rather favorable to dominant class labeling since it in general demands more time per click as demonstrated in Figure 1 of the main paper. Since annotation time more closely reflects practical labeling cost, our multi-class labeling will demand less human resources than dominant class labeling in practice.\n\nThank you for the valuable comment and we will discuss the above in the revision.\n"}, {"review_id": "0jPzx7irHZ", "reviewer": "Reviewer_RB6d", "summary": "This manuscript proposes a new issue on active segmentation from multi-class label query. Concretely, the authors are motivated from the shortcomings of existing active learning paradigm, i.e., [1][2] query the whole image pixel annotations, which costs annotation; [3] selects pixels for annotation, which performs with less generalization. To this end, this manuscirpt follows the compromis that actively annotating the uncertain regions with annotating region-level categories and propose a new query design from multi-hot response. To tackle the issues including uncertain region selecting, category ambiguity, and the absence of fine-grained pixel supervision, the manuscript deviced: a region selection strategy (a.k.a.,  acquisition function), training with class-ambiguity loss functions (stage 1 training), self-train learning (stage 2 training). Then, the authors conducted experiments on various benchmarks of semantic segmentation to show their effectiveness. \n\n[1] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2019.\n[2] Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z Chen. Suggestive annotation: A deep active learning framework for biomedical image segmentation. In Proc. Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2017.\n[3] Gyungin Shin, Weidi Xie, and Samuel Albanie. All you need are a few pixels: semantic segmentation with pixelpick. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2021.", "questions": "I have no questions in evaluating the manuscript. Code is clear.", "limitations": "The authors did not array the limitations. As for the issues proposed in the Sec. Intro., the adopted methods could work intuitively, and experimental results partly support that. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The draft is beautiful and clear to view. The figures arrayed in Sec. 3 indeed assist me a lot in understanding the detailed methdology and  pipeline.\n2. The idea of annotate region-level omni-class is novel. Intuitively, it makes a compromis between image-level with pixel-level as authors claimed in the manuscript. \n3. The experiments show that the proposed method is effective in saving annotation budget and promise segmentation performance.", "weaknesses": "1. The manuscripts required double-check in some details. For example, in L28, \"Early approaches consider an entire image as a sample and ask for its pixel-wise class labels [48, 60], or select individual pixels and query the oracle for their class labels [46]; they turned out to be suboptimal since the former lacks the diversity of samples [40] and the latter is less 30 budget-efficient as a query provides supervision for only a single pixel.\" It is obvious that indivdual pixel-annotation lacks diversity, while whole image annotation costs heavily. But the author happened to claim the opposite of these two points. \n2. Despite that the authors deviced a novel issue in segmenting from region-level multi-class query, the method presented seems to be A+B+C style to me. Moreover, the composition seems not to be novel. Firstly, learning from multi-labels for pre-defined sample-level (i.e., region in this paper, image in learning from multi-class classification) via merging pixel loss is not novel. Secondly, Prototypical Pixel Loss and learning from pseudo labels are very common treatment in self-train paradigm, as cited by authors. Actually, I cannot grab the novelty of the manuscript in methdology design. If I misunderstood some points, it will be better to highlight the contributions and add necessary discussion with previous works.\n3. The comparison experiments on region selection (a.k.a., acquisition function) with other classic strategy, e.g., KNN, Margin, et. al are missing. Sample selection is of vital importance for active learning, while the authors seems to narrow down the importance to this part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback and constructive suggestions, which help improve our paper substantially. We will address all the comments and include additional experiments in the revision. Please find our responses to the comments below.\n___\n**Q1. Clarifying Line 28 in our manuscript**\n\nA1. Thank you for the valuable comment.\n\nWhole image annotation demands labeling every pixel of a selected image, which is extremely expensive. Therefore, only a small number of images can be annotated in general, and accordingly such labeled images are significantly limited in terms of scene diversity. \nOn the other hand, pixel annotation aims to label a few pixels of an image. Thus, a substantially larger number of images will be (partly) labeled and accordingly a greater variety of scenes will be covered. However, it is less budget-efficient than the image- and region-based strategies since pixel annotation requires a click per pixel while the image- and region-based counterparts can label a group of pixels (e.g., superpixel or polygon) by a single click. \n\nWe will more clearly describe the prior work in the main paper as discussed above and also will proofread the manuscript carefully and thoroughly. \n\n___\n**Q2. The novelty of our method**\n\nA2. Thank you for the insightful feedback. Our primary contribution is that we are the first to introduce multi-class labeling to active learning for semantic segmentation, as you kindly acknowledged in the 'Strengths' section. The proposed loss and self-training are essential and carefully designed to effectively utilize the multi-class labels. Below, we elaborate on the novelty of our method.\n\n**The proposed loss functions**\n\nFirst, we would like to emphasize that training a segmentation model with multi-class labels is not straightforward. As these labels lack pixel-wise annotations, the conventional pixel-wise cross-entropy loss for classification is inapplicable. The binary cross-entropy loss commonly used for multi-label classification is also unsuitable here since each pixel is associated with a specific class; this loss assumes the coexistence of multiple labels, which does not hold for individual pixels in semantic segmentation.\n\nOur loss functions resolve this issue by transforming multi-class labels into supervisory signals seamlessly coupled with the standard cross-entropy. For example, the merged positive loss takes the sum of predictive probabilities for candidate classes as positive predictions, and the prototypical pixel loss dynamically selects the most likely pairs of a candidate class and a pixel. These losses have not been employed for active learning to the best of our knowledge, and as reviewer FaJW recognized, we believe these losses could be insightful.\n\nMoreover, our losses are designed to complement each other to effectively utilize multi-class labels. The merged positive loss supplies a weak yet consistent training signal across all pixels in a region, ensuring that any possible noise from the prototypical pixel loss does not misguide the overall training. On the other hand, the prototypical pixel loss gives a stronger signal to a few pixels and is applied uniformly among classes, which helps avoid class imbalance.\n\nTable 1 empirically shows that both losses consistently improve performance across 5 rounds. In Section D.1 and Figure 4(a) of the supplementary material, we also demonstrate the superiority of our losses over baseline partial-label learning losses: infimum loss [7] and RC loss [39].\n\n**The proposed pseudo-labeling approach**\n\nAs commented, pseudo labeling is a common treatment in label-efficient learning [2, 10, 20, 24, 32, 41]. However, it is new to use multi-class labels to identify class-representative features within each region and utilize such features as region-adaptive classifiers for precise pseudo labeling.\n\nFigure 7 of the main paper shows that our pseudo-labeling method consistently surpasses the baseline confidence-based approach and is in particular effective when coupled with multi-class labeling.\n\nWe will discuss our contributions and compare the proposed method with prior work more clearly and comprehensively in the revision. Thank you for the valuable comment, and we are open to any further suggestions to clarify the contribution of our method.\n\n___\n**Q3. Additional comparisons with baseline acquisition functions**\n\nA3. We appreciate the valuable suggestion. In Figure 1 of the attached document available at the global response, we evaluate multi-class labeling combined with 6 different acquisition functions: **Random**, K-Nearest Neighbors (**KNN**), **Margin**, Best-versus-Second-Best (**BvSB**), Class Balanced sampling (**ClassBal**), and our sampling strategy (**PixBal**). **PixBal** consistently outperforms all the others regardless of budget size. **Margin** shows decent performance with a large budget, e.g., it surpasses **ClassBal** when using 500K clicks. On the other hand, **KNN** offers modest improvements, marginally surpassing the performance of **Random**. These observations will be included in the revision.\n\n___\n**Q4. Limitations of our work**\n\nA4. We have stated the limitations and potential future research directions of our study in Lines 300 - 303 of the manuscript, as quoted below, but will present them through a separate paragraph in the revision.\n\n>Some issues still remain for further exploration. As it stands, our method depends on off-the-shelf superpixels and does not transfer the pseudo labels to next rounds. Next on our agenda is to address these issues by learning image over-segmentation and developing an advanced AL pipeline."}, {"review_id": "dMWWPcIJOp", "reviewer": "Reviewer_FaJw", "summary": "The paper studied active learning for semantic segmentation. Instead of using each pixel as a query for annotation, this paper used a super-pixel as a query. Since each super-pixel may contain multi-class labels, thus a multi-class labeling scheme is used instead of dominant class labeling. However, it introduced the class ambiguity issue in training since it assigns multi-class labels to each super-pixel region.  The authors proposed a two-step learning method, effectively utilizing the supervision of multi-class labels.  Extensive experiments were conducted to valid the proposed method, where promising results were reported.  ", "questions": "-- In the right figure of Fig.1,  for the 1-class case, multi-class labeling and dominant class labeling should produce the same accuracy and cost the same amount of time. Why they are slightly different in this figure ? \n\n-- The paper says \"we utilize 32 × 32 superpixel regions given by SEEDS\". I am wondering how the superpixel generation method and superpixel size affect the performances. This is an important ablation study to conduct.  If the superpixel size is the same as the image size, the method turns to be an image level weakly supervised semantic segmentation task.  If the superpixel size is 1*1, the method is the same as using a pixel as the query unit.  \n\n\n ", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "++ The idea is quite interesting and practical. Using superpixels as active learning query unit is sensible. Assigning multi-class labels to the superpixels is also quite effective with the proposed two-step learning scheme. \n\n++ The paper is well written and easy to follow. I like the figures provided in the paper, they are quite informative. \n\n++ Good performance, clearly performance gain is obtained comparing with that of dominant class labeling. \n\n++  The training scheme includes merged positive loss and prototypical pixel loss, both of which are well designed loss functions. Both are insightful from my perspective. \n\n   ", "weaknesses": "\n--  It would be interesting to see how the two hyperparameters in Eq. (10) affect the performance. \n\n--  It would be better if the number of clicks in Fig.4 can start from 0 instead of 100k. Then we can study the extreme case where very limited number of clicks are provided. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback and valuable suggestions that helped improve our paper substantially. All the suggestions and experiments will be incorporated into the paper. Please find our responses to the comments below.\n___\n**Q1. Effect of hyperparameters**\n\nA1. Thank you for the detailed comment. We evaluated the sensitivity of our model to the hyperparameters: $\\\\nu$, $\\\\lambda_{MP}$, and $\\\\lambda_{CE}$ in Figure 2 of the supplementary materials. The results demonstrate that our model is insensitive to these hyperparameters; its accuracy fluctuation was less than 1.5%.\n\n___\n**Q2. Impact of budget size**\n\nA2. We appreciate your suggestion for an interesting experiment. In Figure 2 of the attached document available at the global response, we evaluate multi-class labeling and dominant class labeling with different budget scenarios: 10K clicks over 5 rounds, 50K clicks over 10 rounds, and 100K clicks over 5 rounds. Multi-class labeling consistently outperforms dominant-class labeling in a wide range of budget sizes, demonstrating the efficacy of our method. Even in the tiny budget of 10K clicks, the multi-class labeling outperforms the dominant class labeling baseline. This highlights the utility of our method in extremely constrained budget scenarios, where the need for active learning is emphasized. These observations will be included in the revision.\n\n___\n**Q3. Clarifying the results of the user study**\n\nA3. Thank you for the detailed comment. In Table 1 of the supplementary materials, we provide detailed results of the user study illustrated in Figure 1 of the main paper. In a 1-class scenario, the difference in time per click (s) and accuracy (%) between the two labeling methods are 0.37s and 0.34%p, respectively. This difference is marginal when considering the numerical value of the result, which is 6.75s and 95.97% for multi-class labeling. Given that the survey was conducted among 45 participants, such minor variations could exist. Thus, we assume that this slight discrepancy is not significantly meaningful. One possible interpretation for the increased time per click in multi-class labeling is the added time users might need to ensure all classes in the region have been labeled. Examples of the questions from the survey are available in 'pre-survey.pdf' of the supplementary material. We will incorporate this discussion into the paper.\n\n___\n**Q4. Impact of superpixel generation algorithm and superpixel size**\n\nA4. Thank you for suggesting an important experiment idea!\n\nIn Figure 3(a) of the attached document available at the global response, we evaluate both multi-class and dominant class labeling across two superpixel generation algorithms: SLIC [a] and SEEDS [50]. Both of the labeling methods show better performance when combined with SEEDS compared to SLIC. This is likely because SEEDS gives higher boundary recall than SLIC [b], which indicates that the generated region from SEEDS better preserves the class boundary. Note that the dominant class labeling shows significant performance degradation when combined with SLIC, while the multi-class labeling only shows a modest performance drop. This result suggests that the proposed multi-class labeling is more robust to the quality of the superpixel generation algorithm.\n\nIn Figure 3(b) of the attached document available at the global response, we evaluate multi-class labeling and dominant class labeling across different superpixel sizes: 16×16 (sp16), 32×32 (sp32), and 64×64 (sp64). Both methods show the best performance at the 32×32 region size. Note that dominant class labeling experiences a significant performance drop when the region size increases from 32x32 to 64x64. This is likely because a larger region is more likely to contain multiple classes, which challenges the assumption of dominant class labeling. In contrast, the multi-class labeling exhibits smaller degradation in the same scenario. This shows that multi-class labeling is more robust to the size of the superpixel.\n\nThese experiments will be added to our paper to show additional strengths of our method.\n\n[a] Achanta, Radhakrishna, et al. \"SLIC superpixels compared to state-of-the-art superpixel methods.\" IEEE transactions on pattern analysis and machine intelligence 34.11 (2012): 2274-2282.\n\n[b] Stutz, David, Alexander Hermans, and Bastian Leibe. \"Superpixels: An evaluation of the state-of-the-art.\" Computer Vision and Image Understanding 166 (2018): 1-27."}]

Author rebuttal:
We sincerely thank all the reviewers for their constructive comments. All reviewers appreciated the novelty of our multi-class labeling, comprehensive experiments, and the clarity of the paper. In addition, they recognized that our training method is both reasonable and sound (DZFu), well suited to the proposed query design (9EZK), and insightful (FaJw). On the other hand, valuable feedback was also provided: they asked for more discussions on our results (DZFu) and details about our experimental setup (9EZK), questioned the technical novelty of our training method compared to previous work (9EZK, RB6d), and requested more ablation studies (RB6d, FaJw).  We will make every effort to integrate all feedback in the revision.

The attached document includes additional comparisons with classic acquisition functions (RB6d), experiments on extremely low-budget settings (FaJw), and ablation studies on the superpixel size and its generation algorithm (FaJw).

**Potential of multi-class labeling for real applications**

Figure 1 of the main paper shows that multi-class labeling requires less annotation time per click. As multi-class labeling shows superiority over dominant class labeling even under the evaluation settings based on the number of clicks (Figure 4 of the main paper), its advantage will be even more evident if the annotation time per click is further considered for evaluation. Regarding that the total annotation time more closely reflects actual labeling cost, we believe multi-class labeling has greater potential in real-world active learning scenarios.

Meta-review: This paper proposes new active learning method for labeling semantic segmentation dataset. All reviewers recommend to accept this paper and agree that the proposed labeling method could have potentially high impact for the community to build large scale semantic segmentation dataset. Some questions regarding the loss function and hyper-parameters has been addressed through rebuttal. The AC congratulates the authors on the acceptance of their paper and encourage the authors to actually apply the proposed method to create datasets that could benefit the research community.
Decision: Accept (poster)