SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\hat{Y}(x) \coloneqq \left\{c : \cos\big(f_{\theta}(x), f_{\theta}(x_{s,c}^{*})\big) > \alpha_{c}(s), c \in Y\right\}$ ;  $x$  is a relevant pixel if  $|\hat{Y}(x)| \geq 1$ . By using the prototype-adaptive threshold for filtering, we can adjust the amount of label expansion in each region without the need for hand-tuned hyperparameters.
The segmentation model is then further trained using the pixel-wise CE loss with pseudo segmentation labels generated by both of the label localization and expansion steps.
Figure 4: Accuracy in mIoU (\%) versus the number of clicks (budget) for dominant class labeling (Dom) [9] and multi-class labeling (Mul) equipped with four different acquisition functions (Random, BvSB, ClassBal, PixBal). The reported accuracy scores are averaged across three trials.
Datasets. Our method is evaluated on two semantic segmentation datasets, Cityscapes [15] and PASCAL VOC 2012 (VOC) [21]. The former contains 2975 training, 500 validation, and 1525 test images with 19 semantic classes. The latter consists of 1464 training and 1449 validation images with 20 semantic classes. We evaluated models on validation splits of these datasets.
Implementation details. We adopt DeepLabv3+ [13] with ResNet-50/101 pretrained on ImageNet [18] as our segmentation models, AdamW [45] for optimization. The balancing hyperparameters  $\lambda_{\mathrm{CE}}$  and  $\lambda_{\mathrm{MP}}$  of Eq. (10) are set to 16 and 8, respectively, and the temperature  $\tau$  was fixed by 0.1. In both datasets we utilize  $32 \times 32$  superpixel regions given by SEEDS [59]. For Cityscapes, initial learning rates are set to  $2\mathrm{e} - 3$  (stage 1) and  $4\mathrm{e} - 3$  (stage 2), and  $\nu$  in Eq. (4) is set to 6. The models are trained for 80K iterations with mini-batches of four  $769 \times 769$  images. We assign an extra undefined class for pixels not covered by the original 19 classes. For VOC, we configure  $\nu$  to 12 and train the models for 30K iterations using a learning rate of  $1\mathrm{e} - 3$  in both stages. Each mini-batch consists of twelve  $513 \times 513$  images. More details are given in the Appendix B.
Active learning protocol. Following the previous work [9], we consider the number of clicks as the labeling cost. While this protocol assumes a uniform cost per click, it does not hold in reality as shown in Fig. 1. It is adopted for comparisons with the prior art using dominant class labeling [9], but is adverse to the proposed method since our multi-class labeling takes less cost per click than dominant class labeling. We conduct 5 rounds of consecutive data sampling and model updates, with a budget of 100K and 10K clicks per round on Cityscapes and VOC, respectively. The models are evaluated for each round in mean Intersection-over-Union (mIoU) [21] on the validation sets. At the first round, regions are selected at random, and the models are reinitialized with ImageNet pretrained weights per round. We conduct all experiments three times and report the average performance.
**Baseline methods.** We compare our multi-class labeling (Mul) with the dominant class labeling (Dom) in combination with various data selection strategies. Following the established strategies in the previous study [9], we employ Random, which randomly selects superpixels, and the uncertainty-based BvSB given in Eq. (2). ClassBal is BvSB sampling with additional class balancing term proposed in the previous work [9], and PixBal is our sampling method based on Eq. (4).
Impact of multi-class labeling. In Fig. 4, we evaluate the performance of multi-class and dominant class labeling across varying budgets, using ResNet50 and ResNet101 backbones, on both Cityscapes and VOC with different acquisition functions. Multi-class labeling constantly outperforms dominant class labeling in every setting across all the compared architectures and datasets. In particular, the
Table 1: The ratio of clicks  $(\%)$  needed to reach  $95\%$  mIoU of the fully-supervised model, relative to full supervision. Results with  $\dagger$  are from the prior work [9] using Xception-65 as backbone.
Query	Sampling	Clicks (%)
Patch + Polygon	EntropyBox+†[14]	10.5
MetaBox+†[14]	10.3
Spx + Dominant	ClassBal†[9]	7.9
ClassBal[9]	9.8
PixBal	7.9
Spx + Multi-class	PixBal(Ours)	4.0
Figure 5: Histogram and cumulative distribution function (CDF) for the number of classes in regions selected at round-5 using PixBal sampling on Cityscapes.
Table 2: Contribution of each component of our method in accuracy (mIoU, %) at each round.
Loss function	Pseudo labeling	Rnd-1	Rnd-2	Rnd-3	Rnd-4	Rnd-5	Avg
\( \mathcal{L}_{\text{MP}} \)	\( \mathcal{L}_{\text{PP}} \)	Localization	Expansion
(a)	✓	✓	✓	✓	66.6	73.6	74.5	75.6	75.7	\( 73.2_{\pm0.3} \)
(b)	✓	✓	✓	✘	65.2	72.2	73.5	74.3	74.7	\( 72.0_{\pm0.3} \)
(c)	✓	✓	✘	✘	63.8	70.3	71.8	72.6	73.3	\( 70.4_{\pm0.2} \)
(d)	✓	✘	✘	✘	60.8	69.7	71.0	72.3	72.3	\( 69.2_{\pm0.3} \)
(e)	✘	✓	✘	✘	63.1	69.9	70.9	71.2	71.9	\( 69.4_{\pm0.4} \)
multi-class labeling model, with just 200K clicks, outperforms the dominant class labeling counterpart that uses 500K clicks on Cityscapes. When using ResNet50, the multi-class labeling model equipped with PixBal sampling, achieves  $95\%$  mIoU of the fully-supervised model using only 200K clicks on Cityscapes and 20K clicks on VOC, respectively.
Impact of the proposed sampling method. Fig. 4 also demonstrates the efficacy of PixBal. On Cityscapes, PixBal consistently outperforms all the other sampling methods regardless of budget size. It also enhances the performance of dominant class labeling. On VOC, PixBal generally surpasses the baselines, although its improvement over BvSB, which lacks class balancing, is marginal at times since VOC less suffers from class imbalance than Cityscapes. Further analysis on these sampling methods are provided in Appendix C.4.
Comparison with various query designs. In Table 1, we evaluate multi-class labeling against baseline methods employing different query designs: drawing polygon mask within an image patch (Patch+Polygon), clicking dominant class within superpixel (Spx+Dominant), and clicking all classes within superpixel (Spx+Multi-class). Following the prior work [9], in this experiment, we measure the ratio of clicks used, relative to the total number of clicks required to draw polygon masks on all images (i.e., full supervision). We then measure the ratio of clicks each method needs to achieve  $95\%$  mIoU of the fully-supervised model. As indicated in Table 1, superpixel-based methods typically outperform the baselines using patch or polygon queries. Among these, our multi-class labeling stands out, achieving  $95\%$  mIoU of the fully-supervised model using only  $4\%$  of its required data.
4.3 In-depth analysis on the proposed method
The number of classes in selected regions. The histogram and cumulative distribution of Fig. 5 summarize the number of classes within regions selected at round-5 using our PixBal sampling method on Cityscapes. We observe that more than  $50\%$  of the selected regions contain two or more classes, explaining the necessity of multi-class labeling. This also suggests that, regarding labeling cost in reality (i.e., actual annotation time), multi-class labeling holds potential for further improvement in efficiency as it requires less labeling time for multi-class regions (Fig. 1).
Contribution of each component. Table 2 quantifies the contribution of each component in our method over five rounds: merged positive loss in Eq. (7), prototypical pixel loss in Eq. (9), intra-region label localization, and label expansion. The results show that all components improve performance at every round. The performance gap between (c) and (e) in the table verifies the efficacy of merged
Figure 6: Qualitative comparisons between different labeling strategies. (a) Dominant class labels. (b) Label localization. (c) Label localization + label expansion. (d) Ground-truth.
positive loss for learning with multi-class labels. Meanwhile, the gap between (c) and (d) in the table shows the efficacy of prototypical pixel loss, particularly at the initial round with severe class imbalance due to random sampling. The largest mIoU gain is achieved by intra-region label localization, as shown by the gap between (b) and (c) of the table. Lastly, label expansion further boosts the performance by  $1.2\%$  on average.
Qualitative analysis on pseudo labels. Fig. 6 qualitatively compares region-wise dominant class labels and pixel-wise pseudo labels given by the proposed method to show the label disambiguation ability of our method. Regions labeled with dominant classes overlook areas occupied by minor classes. In contrast, our intra-region localization accurately identifies various classes within each region as shown in the second column of Fig. 6. Moreover, label expansion augments the amount of supervision by referencing the class composition within the region. Notably, minor classes within a region often significantly enhance the quality of pseudo labels via label expansion.
Ablation study of pseudo labeling. Fig. 7 presents our ablation study on pseudo labeling with ResNet50 on Cityscapes. Fig. 7(a) compares performance improvement by the intra-region label localization (w/ prototypes) and that by a baseline assigning the most confident class among multi-class labels as a pixel-wise pseudo label (w/o prototypes). The result suggests that using prototypes consistently surpasses the baseline across different budgets due to their adaptability to local regions. In Fig. 7(b), we investigate the improvement when label expansion is applied to multi-class and dominant class labels across varying budgets. It turns out that label expansion is
(a) Label localization
Figure 7: mIoU gain  $(\%)$  from intra-region label localization and label expansion varying budgets. (a) Gain of label localization with vs. without prototype. (b) Gain of label expansion on dominant and multi-class labels.
(b) Label expansion
more effective with multi-class labeling, as it enables the propagation of pseudo labels belonging to multiple classes, leading to a more extensive expansion of pseudo labels.
Impact of region generation algorithm. In Fig. 8(a), we evaluate both dominant class labeling (Dom) and multi-class labeling (Mul) across two superpixel generation algorithms: SLIC [1] and SEEDS [59]. Both labeling methods show better performance when combined with SEEDS. This is because SEEDS is better than SLIC in terms of boundary recall [57] and thus regions generated by SEEDS better preserves the class boundary. Note that the dominant class labeling shows significant performance degradation when combined with SLIC, while the multi-class labeling only shows a modest performance drop. This result suggests that the proposed multi-class labeling is more robust to the quality of the superpixel generation algorithm.
(b)
Figure 8: Accuracy in mIoU (\%) versus the number of clicks (budget) for dominant class labeling (Dom) and multi-class labeling (Mul) equipped with PixBal evaluated on Cityscapes using ResNet50. (a) Impact of superpixel generation algorithm: SEEDS [59] and SLIC [1]. (b) Impact of superpixel size:  $16 \times 16$  (SP16),  $32 \times 32$  (SP32), and  $64 \times 64$  (SP64).
(c)
Figure 9: Average accuracy of our stage 1 model over 5 rounds, in mIoU (\%), as a function of varying hyperparameters. The model is evaluated on Cityscapes using ResNet50 backbone with PixBal sampling. (a) The class balancing regulation term  $\nu$ . (b) Loss balancing term  $\lambda_{\mathrm{MP}}$ . (c) Loss balancing term  $\lambda_{\mathrm{CE}}$ . The blue diamond marker indicates the value selected for our final model.
Impact of region size. In Fig. 8(b), we evaluate dominant class labeling (Dom) and multi-class labeling (Mul) across different superpixel sizes:  $16 \times 16$  (SP16),  $32 \times 32$  (SP32), and  $64 \times 64$  (SP64). Both methods achieve their best with the  $32 \times 32$  region size. Note that dominant class labeling leads to a large performance drop when the region size increases from  $32 \times 32$  to  $64 \times 64$  since a larger region is more likely to contain multiple classes, which violates the key assumption of dominant class labeling. In contrast, multi-class labeling is more robust against such region size variations.
Impact of hyperparameters. In Fig. 9, we evaluate the sensitivity of our stage 1 model to variations in the hyperparameters:  $\nu$ ,  $\lambda_{\mathrm{MP}}$ , and  $\lambda_{\mathrm{CE}}$ . This evaluation is conducted on Cityscapes using ResNet50 backbone and combined with PixBal sampling. Our model demonstrates robustness to these hyperparameter changes, with accuracy fluctuations of less than  $1.5\%$ . It's noteworthy that our final model does not use the optimal hyperparameter values. This indicates that we did not exhaustively tune these parameters using the validation set.
We have introduced a novel multi-class label query for AL in semantic segmentation. Our two-stage training method and new acquisition function enabled an AL framework with the multi-class label query to achieve the state of the art on two public benchmarks for semantic segmentation by improving label accuracy and reducing labeling cost. Some issues still remain for further exploration. As it stands, our method depends on off-the-shelf superpixels and does not transfer the pseudo labels to the next rounds. Next on our agenda is to address these issues by learning image over-segmentation and developing an advanced AL pipeline.
We sincerely appreciate Moonjeong Park for fruitful discussions. This work was supported by the NRF grant and the IITP grants funded by Ministry of Science and ICT, Korea (NRF-2021R1A2C3012728, IITP-2020-0-00842, IITP-2022-0-00290, IITP-2021-0-02068, IITP-2019-0-01906, IITP-2021-0-00739).
[68] Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z Chen. Suggestive annotation: A deep active learning framework for biomedical image segmentation. In Proc. Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2017.
[69] Adrian Ziegler and Yuki M Asano. Self-supervised learning of object parts for semantic segmentation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
—Appendix—
This appendix provides additional experimental details and findings that have been omitted in the main paper due to the page limit. Sec. A presents a thorough explanation of the user study conducted to compare the cost of dominant class labeling and multi-class labeling. In Sec. B, we explain further details of configurations of our implementation. Sec. C presents additional experiments including the effect of the budget size (Sec. C.1), a comparison with partial label learning loss baselines (Sec. C.2), a comparison with pseudo labeling baselines (Sec. C.3), a comparison with baseline acquisition functions (Sec. C.4), and a qualitative result of our final model (Sec. C.5).
A Details of user study
We conducted a user study to compare the dominant class labeling and multi-class labeling in terms of actual labeling cost and accuracy versus the number of classes in region queries. The examples of the questionnaire are illustrated in Fig. 10 and the results are summarized in Table 3. As shown in Fig. 10(a), for each question, annotators received an instruction, an image patch along with a marked local region, and class options. They were requested to select the relevant class options as directed by the instruction. The instructions for dominant class labeling and multi-class labeling were as follows:
"Select the dominant class that corresponds to the inside of the red boundary.", "Select the all classes that exist within the red boundary".
Prior to the survey, we ensured that every participant reviewed the pre-survey instructional material. This material covered the class composition of Cityscapes, offered the definition of the dominant class and the multi-class labeling, and provided example questions. The pre-survey instructional material is included in the supplementary materials under the name 'pre-survey.pdf'.
As shown in Fig. 10(b), each image patch was a 360-pixel square mostly centered on a local region. Using ground-truth segmentation mask, we divided regions into three groups based on the number of classes (from 1 to 3) present in each region. Twenty regions were then randomly selected from each group for each survey, excluding those containing pixels irrelevant to the original 19 classes, referred to as the 'undefined' class.
Figure 10: Questionnaire and local region examples used in the user study. (a) Questionnaire of multi-class labeling survey, consisting of instruction, image patch along with local region marked with red boundary, and class options allowing multiple selections. (b) Examples of local regions used in the user study according to the number of classes present in each region.
Table 3: The result of user study showing the labeling time (second) and accuracy  $(\%)$  of dominant class labeling and multi-class labeling according to the number of classes within each region.
Query	# of classes	Total time (s)	Total clicks	Time per click (s)	Accuracy (%)
Dominant	1	127.6±39.4	20.0±0.0	6.38±1.97	95.63±6.00
2	160.5±33.0	20.0±0.0	8.02±1.65	72.05±5.36
3	172.1±35.8	20.0±0.0	8.60±1.79	65.83±6.07
average	153.4±41.1	20.0±0.0	7.67±2.05	77.84±14.24
Multi-class	1	145.5±41.8	21.6±2.1	6.75±1.65	95.97±4.10
2	191.6±65.8	39.1±3.7	4.89±1.54	87.14±5.21
3	295.8±65.3	49.0±8.6	6.37±1.51	71.52±8.42
average	211.0±86.3	36.5±12.5	6.01±1.75	84.88±11.76
A total of 45 volunteers participated in the survey. We report the results excluding five cases considered outliers in terms of time and accuracy. A unique survey was prepared for each group of regions, categorized by the number of classes. Given three groupings and two labeling methods, a total of six unique forms were prepared. If an annotator annotates the same region twice, there would be a risk of memorizing the image during the first annotation. To avoid this, we asked each participant to answer three out of the six forms, ensuring no region was annotated twice by the same person.
The responses from annotators are evaluated by calculating the Jaccard Similarity (JS) between the ground-truth class set and the responded class set. We define the JS of annotator  $u$  as follows:
$$
\operatorname {J S} (u) = \frac {1}{| X |} \sum_ {i \in X} \frac {\left| G _ {i} \cap Y _ {i , u} \right|}{\left| G _ {i} \cup Y _ {i , u} \right|}, \tag {14}
$$
where  $X$  is a set of regions,  $G_{i}$  is ground-truth multi-class label and  $Y_{i,u}$  is the set of classes selected by annotator  $u$  for region  $i$ . Note that for dominant class labeling,  $|G_{i}| = |Y_{i,u}| = 1$ . We compute the final accuracy as the average JS of all annotators, given by:
$$
\text {A c c u r a c y} = \frac {1}{| U |} \sum_ {u \in U} \mathrm {J S} (u). \tag {15}
$$
As shown in Table 3, multi-class labeling demonstrates comparable efficiency to dominant class labeling for regions with a single class. Moreover, when it comes to regions with multiple classes, multi-class labeling requires less annotation time per click compared to the dominant class labeling.
B Implementation details
Configurations. We implement our method using the PyTorch framework [51]. Following the previous literature [27], we make a slight modification to the original ResNet architecture by replacing the initial  $7 \times 7$  convolutional layer with two  $3 \times 3$  convolutional layers. The output stride of the network is set to 16. We set the learning rate of the backbone to be ten times lower than the standard rate, and we apply a weight decay of  $1\mathrm{e} - 5$ . During the training phase, we incorporate several data augmentation techniques, including random scaling ranging from 0.5 to 2.0, random cropping, and random horizontal flipping. To ensure reproducibility and to test the robustness of our approach, we conduct three independent experiments, each initialized with different seed values: 0, 1, and 2.
Label generation. Following previous work [9], we assign both dominant class labels and multi-class labels to each region using the ground-truth mask labels. For the dominant class label, we assign the class that dominates the majority of pixels within each region, in accordance with its definition. For the multi-class labels, we attribute all existing classes present within each region. Notably, we disregard classes that appear minimally along the region's boundary during this process. This procedure reflects realistic scenarios where a labeler might fail to recognize classes represented insignificantly on the boundary. More specifically, we implement a binary dilation operation with a  $5 \times 5$  kernel along the region boundaries, consequently excluding classes that appear on these expanded boundaries. In the user study, we reflect this by marking each local region with a thick, translucent boundary.
(b)
Figure 11: Accuracy in mIoU (\%) versus the number of clicks (budget) evaluated on Cityscapes using ResNet50 with PixBal. We evaluate three budget scenarios: (a) 10K clicks over 5 rounds, (b) 50K clicks over 10 rounds, and (c) 100K clicks over 5 rounds.
Handling undefined class. In the Cityscapes dataset [15], pixels not covered by the original 19 semantic classes are typically ignored when training segmentation models. On the other hand, in multi-class labeling setting, the precise locations of such uncovered pixels remain unspecified since the multi-class label only provide partial labels. Treating such pixels as belonging to one of the 19 semantic classes naively can misguide the model by providing confusing supervision. Furthermore, active sampling methods like BvsB, ClsBal, PixBal tend to prefer uncertain regions, often leading to the selection of regions containing these uncovered pixels, despite their lack of utility. To address this, we assign an additional undefined class for pixels not covered by the initial 19 classes and train the model to predict these undefined classes. As for active sampling, we introduce an extra condition to exclude regions where the predicted dominant class is the undefined class. This undefined class handling strategy is implemented for both dominant class labeling and multi-class labeling.
Insight for choosing hyperparameters. We further describe the detailed insights to determine the importance of each loss term;  $\lambda_{\mathrm{MP}}$ , and  $\lambda_{\mathrm{CE}}$ . We roughly determined them by considering the scale of the loss signal back-propagated per pixel.  $\mathcal{L}_{\mathrm{PP}}$  is applied to only a tiny subset of pixels in an image, i.e., pixels whose number is the same as the number of classes per region (i.e., superpixel). Considering that the final loss value is computed as an average over the pixels, the limited pixel domain of  $\mathcal{L}_{\mathrm{PP}}$  implies overly stronger loss signals compared with other losses. To counterbalance this side effect, we strategically assigned larger weights to  $\lambda_{\mathrm{CE}}$  and  $\lambda_{\mathrm{MP}}$ , which are associated with a larger number of pixels. The values of  $\lambda_{\mathrm{CE}}$  and  $\lambda_{\mathrm{MP}}$  were estimated by a grid search on the Cityscapes dataset and were used as-is on PASCAL VOC 2012 as well.
C Additional experiments
C.1 Impact of budget size