SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sehyun Hwang Sohyun Lee Hoyoung Kim Minhyeon Oh Jungseul Ok Suha Kwak
Pohang University of Science and Technology (POSTECH), South Korea {sehyun03, lshig96, cskhy16, minhyeonoh, jungseul.ok, suha.kwak} @postech.ac.kr
This paper proposes a new active learning method for semantic segmentation. The core of our method lies in a new annotation query design. It samples informative local image regions (e.g., superpixels), and for each of such regions, asks an oracle for a multi-hot vector indicating all classes existing in the region. This multi-class labeling strategy is substantially more efficient than existing ones like segmentation, polygon, and even dominant class labeling in terms of annotation time per click. However, it introduces the class ambiguity issue in training as it assigns partial labels (i.e., a set of candidate classes) to individual pixels. We thus propose a new algorithm for learning semantic segmentation while disambiguating the partial labels in two stages. In the first stage, it trains a segmentation model directly with the partial labels through two new loss functions motivated by partial label learning and multiple instance learning. In the second stage, it disambiguates the partial labels by generating pixel-wise pseudo labels, which are used for supervised learning of the model. Equipped with a new acquisition function dedicated to the multi-class labeling, our method outperforms previous work on Cityscapes and PASCAL VOC 2012 while spending less annotation cost. Our code and results are available at https://github.com/sehyun03/MulActSeg.
Supervised learning of deep neural networks has driven significant advances of semantic segmentation for a decade. At the same time, however, it has limited practical applications of the task as it demands as supervision pixel-level class labels that are prohibitively expensive in general. To address this limitation, label-efficient learning approaches such as weakly supervised learning [2, 3, 11, 20, 28, 34, 39, 40, 58, 63], semi-supervised learning [4, 12, 25, 33, 36, 37, 41–44, 48, 49], self-supervised learning [24, 60, 69], and active learning (AL) [9, 10, 14, 23, 31, 38, 47, 55, 56, 68] have been investigated.
This paper studies AL for semantic segmentation, where a training algorithm selects informative samples from training data and asks an oracle to label them on a limited budget. In AL, the design of annotation query, i.e., the granularity of query samples and the annotation format, is of vital importance for maximizing the amount and quality of supervision provided by the oracle within a given budget. Early approaches consider an entire image as a sample and ask for its pixel-wise class labels [56, 68], or select individual pixels and query the oracle for their class labels [54]; they turned out to be suboptimal since the former lacks the diversity of samples [47] and the latter is less budget-efficient as a query provides supervision for only a single pixel.
As a compromise between these two directions, recent AL methods treat non-overlapped local image regions as individual samples [9, 10, 14, 23, 31, 47, 55]. These region-based methods guarantee the diversity of samples by selecting local regions from numerous images with diverse contexts. Also, their queries are designed to obtain region-wise segmentation labels efficiently. For instance, they
Comparison on a region query example
Figure 1: Dominant class labeling [9] versus our multi-class labeling. (left) Given a local region as query, an oracle is asked to select the most dominant class by a single click in dominant class labeling, and all existing classes by potentially more than one click in multi-class labeling. As shown here, multi-class labeling often takes less annotation time per click because, to determine the dominant one, the oracle has to infer every class in the region after all and sometimes should very carefully investigate the region when the classes occupy areas of similar sizes. (right) We conducted a user study to compare the two strategies in terms of actual labeling cost and accuracy versus the number of classes in region queries; the results are summarized in the right plot with one standard deviation. Multi-class labeling took less time per click on average due to the above reason. Furthermore, it resulted in more accurate labels by annotating non-dominant classes ignored in dominant class labeling additionally. Details of this user study are given in Appendix A.
Dominant class labeling
"Select dominant class within the red border"
{Pole} 1 click
(Dominant)
Multi-class labeling
"Select all classes within the red border" (Query)
{Building, Car, Pole} 3 clicks
(Annotation clicks)
(Multi-class)
Labeling cost and accuracy on average
ask the oracle to draw segmentation masks in the form of polygons within an image patch [14, 47], or to estimate the dominant class label of a superpixel so that the label is assigned to the entire superpixel [9]. Although the existing region-based methods have achieved great success, we argue that there is still large room for further improvement in their query designs: Polygon labeling [14, 47] still requires a large number of clicks per query, and dominant class labeling [9] provides wrong supervision for part of a multi-class region. Note that the latter issue cannot be resolved even using superpixels since they frequently violate object boundaries and include multiple classes.
In this context, we first introduce a new query design for region-based AL of semantic segmentation. The essence of our proposal is to ask the oracle for a multi-hot vector that indicates all classes existing in the given region. This multi-class labeling strategy enables to prevent annotation errors for local regions capturing multiple classes, and works the same as dominant class labeling (and thus inherits its advantages) for single-class region queries. Moreover, our user study revealed that multi-class labeling demands less annotation time per click and results in more accurate labels compared with dominant class labeling as demonstrated in Fig. 1. However, such region-wise multi-class labels introduce a new challenge in training, known as the class ambiguity issue, since they assign partial labels [16, 29] (i.e., a set of candidate classes) to individual pixels of the selected regions.
To address the ambiguity issue, we propose a new AL method tailored to learning semantic segmentation with partial labels. Fig. 2 illustrates the overall pipeline of the proposed method. Given a set of local regions and their multi-class labels, our method trains a segmentation network in two stages. In the first stage, the network is trained directly with the region-wise multi-class labels. To this end, we propose two new loss functions for the label disambiguation based on the notions of partial-label learning [16, 29] and multiple instance learning [19], respectively. In the second stage, our method disambiguates the partial labels through pseudo segmentation labels, which are used to train the segmentation network in the supervised learning fashion. To be specific, it finds a set of class prototype features from each local region using the model of the first stage, and employs the prototypes as a region-adaptive classifier to predict pixel-wise pseudo labels within the region. In addition, we propose to propagate the pseudo labels to neighboring local regions to increase the amount of supervision given per query; this strategy benefits by multi-class labeling that enables to propagate pseudo labels of multiple classes, leading to larger expansion of pseudo labels.
Last but not least, we introduce an acquisition function designed to maximize the advantage of multi-class labels in the region-based AL. Our acquisition function considers both uncertainty [32,62] and class balance [7,66,67] of sampled regions so that local regions where the model finds difficult and containing underrepresented classes are chosen more frequently. It shares a similar motivation
with an existing acquisition function [9], but different in that it considers multiple classes of a region and thus better aligns with multi-class labeling.
The proposed framework achieved the state of the art on both Cityscapes [15] and PASCAL VOC 2012 [21]. Especially, it achieved  $95\%$  of the fully supervised learning performance on Cityscapes with only  $4\%$  of the full labeling cost. In addition, we verified the efficacy and efficiency of multi-class labeling through extensive empirical analyses: Its efficacy was demonstrated by experiments with varying datasets, model architectures, acquisition functions, and budgets, while its efficiency was examined in real-world annotation scenarios by measuring actual labeling time across a large number of human annotators. In short, the main contribution of this paper is five-fold:
- We introduce a new query design for region-based AL in semantic segmentation, which asks the oracle for a multi-hot vector indicating all classes existing within a particular region.
- We propose a novel AL framework that includes two new loss functions effectively utilizing the supervision of multi-class labels and a method for generating pseudo segmentation labels from the multi-class labels, resulting in enhanced supervision.
- To maximize the advantage of multi-class labels, we design an acquisition function that considers multiple classes of a local region when examining its uncertainty and class balance.
- The effectiveness of multi-class labeling was demonstrated through extensive experiments and user study in real-world annotation scenarios.
- The proposed framework achieved the state of the art on both two public benchmarks, Cityscapes and PASCAL VOC 2012, with a significant reduction in annotation cost.
Active learning (AL). In AL, a training algorithm samples informative data and asks an oracle to label them on a limited budget so as to maximize performance of a model trained with the labeled data. To this end, AL methods have suggested various sampling criteria such as uncertainty [5,26,50], diversity [53, 56], or both [6, 30, 64, 65]. Also, since most of existing AL methods for vision tasks have focused on image classification, the granularity of their annotation queries has been an entire image in general. However, for structured prediction tasks like semantic segmentation, queries should be more carefully designed to optimize cost-effectiveness of annotation.
Active learning for semantic segmentation. Most AL methods for semantic segmentation can be categorized into image-based [17, 56, 68] and region-based methods [47]. The image-based methods consider an entire image as the sampling unit and query an oracle for pixel-wise labels of sampled images. These methods have been known to be less cost-effective due to the limited diversity of sampled data; adjacent pixels in an image largely overlap in their receptive fields and thus fail to provide diverse semantics during training. On the other hand, the region-based methods divide each image into non-overlapping local regions, which are considered as individual samples to be selected; As such local regions, image patches [10, 14, 47] and superpixels [9, 35, 55] have been employed. Our paper proposes a new cost-effective region query that allows more accurate and faster annotation, and a new training algorithm taking full advantage of the query design.
Partial label learning. Partial label learning [8, 16, 29, 46] is a branch of weakly supervised learning where a set of candidate classes is assigned to each of training data, leading to ambiguous supervision. One primitive yet common approach to partial label learning is to train a model while regarding its top-1 prediction as true labels [8, 46]. However, this approach could be vulnerable to class imbalance [61] and strong correlation between different classes [52]. In contrast, our two-stage training algorithm addresses these issues by disambiguating partial labels by pseudo labeling.
3 Proposed Method
We consider an AL process with  $R$  rounds. At each round, local regions of a batch are selected using an acquisition function, and then a multi-hot vector (i.e., multi-class label) is assigned to each of them by an oracle. Given the labeled regions, our training algorithm operates in two stages as illustrated in Fig. 2. In the first stage, a segmentation model is trained directly with the region-wise multi-class labels by two loss functions specialized to handle the ambiguity of the labels (Sec. 3.2). In the second
Stage 1: Learning with region-wise multi-class labels
Stage 2: Learning with pixel-wise pseudo labels
Figure 2: Our two-stage training algorithm using partial labels. (left) In the first stage, a model is trained using region-wise multi-class labels through two losses: the merged positive loss that encourages the model to predict any of the annotated classes for each pixel of the region, and the prototypical pixel loss that ensures at least one pixel in the region corresponds to each annotated class. (right) The second stage disambiguates the region-wise multi-class labels by generating pixel-wise pseudo labels, which are then used for training the final model. To this end, it first assigns pseudo class labels to individual pixels within the region (i.e., intra-region label localization), and then propagates the pseudo labels to adjacent regions (i.e., label expansion).
stage, the ambiguity is mitigated by generating pixel-wise pseudo labels and using them for training the model further (Sec. 3.3). The remainder of this section describes details of our framework.
3.1 Acquisition of region-wise multi-class labels
For an unlabeled image set  $\mathcal{I}$ , we partition each image  $I \in \mathcal{I}$  into a set of non-overlapping regions, denoted by  $S(I)$ , such that a pixel  $x \in I$  belongs to only a unique region  $s \in S(I)$ . Such a non-overlapping partition can be obtained by a superpixel algorithm as in [9]. Let  $\mathcal{S} := \bigcup_{I \in \mathcal{I}} S(I)$  be the set of all the partitions for  $\mathcal{I}$ . For each round  $t$ , we issue a batch of regions, denoted by  $\mathcal{B}_t \subset \mathcal{S}$ , each of which is queried to acquire a multi-class label  $Y \subset \{1, 2, \dots, C\}$ , where  $|Y| \geq 1$  and  $C$  is the number of classes. Then the model  $\theta_t$  is trained using the labeled regions obtained so far, denoted as  $\mathcal{D} := \bigcup_{t} \mathcal{D}_t$ , where  $\mathcal{D}_t$  consists of pairs of region and associated multi-class label,  $\mathcal{D}_t := \{(s, Y) : s \in \mathcal{B}_t\}$ . The model  $\theta_t$  includes a feature extractor  $f_t(\cdot)$  and a classifier with a weight matrix  $[\mathbf{w}_{t,1}, \mathbf{w}_{t,2}, \dots, \mathbf{w}_{t,C}] \in \mathbb{R}^{d \times C}$ . The predictive probability of pixel  $x$  being class  $c$  is computed by
$$
P _ {\theta_ {t}} (y = c | x) = \operatorname {s o f t m a x} \left(\frac {f _ {t} (x) ^ {\top} \mathbf {w} _ {t , c}}{\tau \| f _ {t} (x) \| \| \mathbf {w} _ {t , c} \|}\right), \tag {1}
$$
where  $\tau$  is a temperature term.
Acquisition function. We introduce an acquisition function for selecting a batch of regions  $\mathcal{B}_t\subset S$  at round  $t$  that aims to optimize the benefits of multi-class labels, while adhering to the principles of previous studies [7, 9, 66] for uncertain and class-balanced region selection. We adopt best-versus-second-best (BvSB) [32, 62] as an uncertainty measure, defined as
$$
u _ {\theta_ {t}} (x) := \frac {P _ {\theta_ {t}} (y = c _ {\mathrm {s b}} | x)}{P _ {\theta_ {t}} (y = c _ {\mathrm {b}} | x)}, \tag {2}
$$
where  $c_{\mathrm{b}}$  and  $c_{\mathrm{sb}}$  are the classes with the largest and second-largest predictive probabilities for  $x$  under  $\theta_t$ , respectively. For class-balanced sampling, we first estimate the label distribution  $P_{\theta_t}(y)$  as
$$
P _ {\theta_ {t}} (y = c) = \frac {1}{| X |} \sum_ {x \in X} P _ {\theta_ {t}} (y = c | x), \tag {3}
$$
where  $X \coloneqq \{x \colon \exists s \in S, x \in s\}$ . Our acquisition function, favoring uncertain regions of rare classes, is defined as
$$
a (s; \theta_ {t}) := \frac {1}{| s |} \sum_ {x \in s} \frac {u _ {\theta_ {t}} (x)}{\left(1 + \nu P _ {\theta_ {t}} \left(c _ {b}\right)\right) ^ {2}}, \tag {4}
$$
where  $\nu$  is a hyperparameter regulating the class balancing effect. Distinct from an existing acquisition function [9] that considers the dominant class only, our function considers classes of all pixels in a region and thus better aligns with multi-class labeling. For the remainder of this section, we will omit the round index  $t$  from  $\mathcal{D}_t$  and  $\theta_t$  for simplicity.
3.2 Stage 1: Learning with region-wise multi-class labels
During training of the segmentation model, regions labeled with a single class are used for the conventional supervised learning using the pixel-wise cross-entropy (CE) loss. The set of local regions equipped with single-class labels is defined as
$$
\mathcal {D} _ {\mathrm {s}} := \left\{\left(s, \{c \}\right): \exists (s, Y) \in \mathcal {D}, | Y | = 1, c \in Y \right\}. \tag {5}
$$
The pixel-wise CE loss is then given by
$$
\mathcal {L} _ {\mathrm {C E}} = \hat {\mathbb {E}} _ {(s, \{c \}) \sim \mathcal {D} _ {\mathrm {s}}} \left[ \frac {1}{| s |} \sum_ {x \in s} - \log P _ {\theta} (y = c | x) \right]. \tag {6}
$$
On the other hand, regions labeled with multiple classes, denoted as  $\mathcal{D}_{\mathrm{m}} \coloneqq \mathcal{D} - \mathcal{D}_{\mathrm{s}}$ , cannot be used for training using the pixel-wise CE loss, since a multi-class label lacks precise correspondence between each pixel and class candidates, making it a weak label [16, 19, 29]. To effectively utilize the supervision of  $\mathcal{D}_{\mathrm{m}}$ , we introduce two loss functions.
Merged positive loss. Each pixel in a region is assigned with partial labels [16, 29], i.e., a set of candidate classes. The per-pixel prediction in each region should be one of these candidate classes. This concept is directly incorporated into the merged positive loss, which is defined as
$$
\mathcal {L} _ {\mathrm {M P}} := \hat {\mathbb {E}} _ {(s, Y) \sim \mathcal {D} _ {\mathrm {m}}} \left[ \frac {1}{| s |} \sum_ {x \in s} - \log \sum_ {c \in Y} P _ {\theta} (y = c | x) \right]. \tag {7}
$$
This loss encourages to predict any class from the candidate set since the predictive probability of every candidate class is considered as positive.
Prototypical pixel loss. Learning with regions assigned with multi-class labels can be considered as an example of multiple instance learning (MIL) [19], where each region is a bag, each pixel in the region is an instance, and at least one pixel in the region must be positive for each candidate class. We call such a pixel prototypical pixel, and the pixel with the most confident prediction for each candidate class within the region is chosen as a prototypical pixel:
$$
x _ {s, c} ^ {*} := \max  _ {x \in s} P _ {\theta} (y = c | x), \tag {8}
$$
where  $c \in Y$  and  $(s, Y) \in \mathcal{D}_{\mathrm{m}}$ . The segmentation model is trained by applying the CE loss to each prototypical pixel with the assumption that the class associated with it is true. To be specific, the loss is defined as
$$
\mathcal {L} _ {\mathrm {P P}} := \hat {\mathbb {E}} _ {(s, Y) \sim \mathcal {D} _ {\mathrm {m}}} \left[ \frac {1}{| Y |} \sum_ {c \in Y} - \log P _ {\theta} (y = c \mid x _ {s, c} ^ {*}) \right]. \tag {9}
$$
As reported in the literature of MIL [19], although the prototypical pixels may not always match the ground truth, it is expected that training with numerous prototypical pixels from diverse regions enables the model to grasp the underlying concept of each class. Moreover, this loss mitigates the class imbalance issue as it ensures that every candidate class equally contributes to training via a single prototypical pixel in a region, leading to a balanced class representation.
In summary, the total training loss of the first stage is given by
$$
\mathcal {L} = \lambda_ {\mathrm {C E}} \mathcal {L} _ {\mathrm {C E}} + \lambda_ {\mathrm {M P}} \mathcal {L} _ {\mathrm {M P}} + \mathcal {L} _ {\mathrm {P P}}, \tag {10}
$$
where  $\lambda_{\mathrm{CE}}$  and  $\lambda_{\mathrm{MP}}$  are balancing hyperparameters.
3.3 Stage 2: Learning with pixel-wise pseudo labels
In the second stage, we disambiguate the partial labels by generating and exploiting pixel-wise one-hot labels. The pseudo label generation process comprises two steps: intra-region label localization that assigns pseudo class labels to individual pixels within each labeled region, and label expansion
Pseudo label generation process
Results
Figure 3: The pseudo label generation process (left) and its qualitative results (right). In each of the labeled regions, the feature vector located at the prototypical pixel of an annotated class is considered the prototype of the class, and the set of such prototypes is used as a region-adaptive classifier for pixel-wise pseudo labeling within the region (label localization). The pseudo labels of the region are propagated to adjacent unlabeled regions similarly (label expansion), but for conservative propagation, only relevant pixels that are close to at least one prototype will be assigned pseudo labels.
that spreads the pseudo labels to unlabeled regions adjacent to the labeled one. This process is illustrated in Fig. 3, and described in detail below.
Intra-region label localization. For each of the labeled regions, we define a prototype for each annotated class as the feature vector located at the prototypical pixel of the class, which is estimated by Eq. (8) using the model of the first stage. The set of such prototypes is then used as a region-adaptive classifier, which is dedicated to pixel-level classification within the region. To be specific, we assign each pixel the class of its nearest prototype in a feature space; the assigned pseudo label for  $x \in s$  where  $(s,Y) \in \mathcal{D}$ , is estimated by
$$
\hat {y} (x) := \underset {c \in Y} {\arg \max } \cos \left(f _ {\theta} (x), f _ {\theta} \left(x _ {s, c} ^ {*}\right)\right), \tag {11}
$$
where  $x_{s,c}^{*}$  is the prototypical pixel of class  $c$  and  $\cos(f, f') = \frac{f^{\top} f'}{\|f\| \|f'\|}$  is the cosine similarity between two feature vectors  $f$  and  $f'$ .
Label expansion. The rationale behind the label expansion step is that the class composition  $Y$  of a region  $(s, Y) \in \mathcal{D}$  may provide clues about classes of its adjacent regions  $s' \in \mathrm{N}(s)$ , where  $\mathrm{N}(\cdot)$  denotes a set of unlabeled regions adjacent to  $s$ , i.e.,  $\mathrm{N}(s) \cap \mathcal{D} = \emptyset$ . Similar to label localization, the label expansion step aims to assign pixels in  $\mathrm{N}(s)$  the class labels of their nearest prototypes. However, since  $\mathrm{N}(s)$  may contain irrelevant classes, propagating the labels to all pixels in  $\mathrm{N}(s)$  could cause incorrect pseudo labels, leading to performance degradation of the final model. Hence, the pseudo labels are proposed only to relevant pixels that are sufficiently close to at least one prototype in the feature space. More specifically, to compute the relevance in a region- and class-adaptive manner, we propose to use prototype-adaptive thresholds: the prototype-adaptive threshold for class  $c \in Y$  in  $(s, Y) \in \mathcal{D}$  is defined as
$$
\alpha_ {c} (s) = \operatorname {m e d} \left(\left\{\cos \left(f _ {\theta} (x), f _ {\theta} \left(x _ {s, c} ^ {*}\right)\right): x \in s, \hat {y} (x) = c \right\}\right), \tag {12}
$$
where  $\operatorname{med}(\cdot)$  yields the median value of a set,  $x_{s,c}^{*}$  is the prototypical pixel of class  $c$  (Eq. (8)), and  $\hat{y}(x)$  is the pseudo label of  $x$  (Eq. (11)). We propagate pseudo labels of the labeled region  $s$  in  $\mathcal{D}$  to pixels of an adjacent region  $\{x : \exists s' \in \mathrm{N}(s), x \in s'\}$  by
$$
\hat {y} (x) := \underset {c \in \hat {Y} (x)} {\arg \max } \cos \left(f _ {\theta} (x), f _ {\theta} \left(x _ {s, c} ^ {*}\right)\right) \quad \text {o n l y i f} | \hat {Y} (x) | \geq 1, \tag {13}
$$