SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Binhui Xie
Beijing Institute of Technology binhuixie@bit.edu.cn
Shuang Li
Beijing Institute of Technology shuangli@bit.edu.cn
Qingju Guo
Beijing Institute of Technology qingjuguo@bit.edu.cn
Chi Harold Liu
Beijing Institute of Technology chiliu@bit.edu.cn
Xinjing Cheng
Tsinghua University & Inceptio Technology cnorbot@gmail.com
Active learning, a label-efficient paradigm, empowers models to interactively query an oracle for labeling new data. In the realm of LiDAR semantic segmentation, the challenges stem from the sheer volume of point clouds, rendering annotation labor-intensive and cost-prohibitive. This paper presents Annotator, a general and efficient active learning baseline, in which a voxel-centric online selection strategy is tailored to efficiently probe and annotate the salient and exemplar voxel girls within each LiDAR scan, even under distribution shift. Concretely, we first execute an in-depth analysis of several common selection strategies such as Random, Entropy, Margin, and then develop voxel confusion degree (VCD) to exploit the local topology relations and structures of point clouds. Annotator excels in diverse settings, with a particular focus on active learning (AL), active source-free domain adaptation (ASFDA), and active domain adaptation (ADA). It consistently delivers exceptional performance across LiDAR semantic segmentation benchmarks, spanning both simulation-to-real and real-to-real scenarios. Surprisingly, Annotator exhibits remarkable efficiency, requiring significantly fewer annotations, e.g., just labeling five voxels per scan in the SynLiDAR  $\rightarrow$  SemanticKITTI task. This results in impressive performance, achieving  $87.8\%$  fully-supervised performance under AL,  $88.5\%$  under ASFDA, and  $94.4\%$  under ADA. We envision that Annotator will offer a simple, general, and efficient solution for label-efficient 3D applications.
3D perception and understanding have become indispensable for machines to effectively interact with the real world. LiDAR (Light Detection And Ranging) [50, 52] is a widely-used methodology for capturing precise geometric information about the environment, spurring significant advancements in areas like autonomous vehicles and robotics [16, 25]. However, semantic segmentation of LiDAR presents an enormous challenge. The high-speed collection of millions of point clouds per second by on-board sensors sharply contrasts with the laborious and cost-prohibitive nature of annotating them. Consider, for instance, the vast number of outdoor scenes an autopilot can encounter, which is practically limitless. Yet, acquiring annotations for these large-scale point clouds entails intensive human labor. This underscores the urgency of establishing a label-efficient learning mechanism capable of boosting performance in the low-data regime [19, 76, 90, 97] or facilitating the adaptation of models to new domains [55, 64, 74, 85].
$\text{忍}$  Corresponding author. Project page: https://binhuixie.github.io/annotator-web/
Extensive solutions encompass semi-supervised [8, 11, 29, 31, 83], weakly-supervised [22, 36, 76, 99] or self-supervised [5, 14, 63, 73, 101] learning. Semi- and weakly-supervised learning methods
aim to alleviate the annotation burden by harnessing partially labeled or weakly labeled data. In contrast, self-supervised ones learn representations from point clouds via pretext tasks and then transfer to downstream tasks for weight initialization. Although these works offer scalability and practicality for real-world utility, they also confront new challenges, such as variations in LiDAR configurations, sensor biases, and environmental conditions. That is, the majority of prior works has endeavored to in-distribution scenarios, with limited consideration for label-efficient paradigms in out-of-distribution scenarios, especially for sparse outdoor point clouds. Recent efforts turn to large-scale auxiliary datasets [56, 84] and delve into domain adaptation (DA) algorithms [1], [28, 51, 92] to significantly reduce the annotation workload under a domain
Figure 1: Performance vs. annotated proportion on SemanticKITTI val [3] of existing label-efficient LiDAR segmentation paradigms including domain adaptation (●) [55], [83], [84], [105], weakly- (▲) [22], [76] and semi-supervised (◆) [8], [29], [72] learning. As a reference, fully supervised counterpart (★) is reported as well. Annotator (■) attains excellent balance between performance and annotation cost.
shift. Nevertheless, the performance of these methods still lags behind the fully-supervised approaches. In Figure  $\boxed{1}$  we provide an intuitive comparison of results across various paradigms. It becomes evident that there is ample room for improvement in the performance of these methods.
To surmount these obstacles and promote performance in the domain of interest, active learning (AL) is being an optimal paradigm [27,35,60,82]. Given the limited annotation budget, a common scenario is that only an unlabeled target domain of large amounts of point clouds is available with the goal to interactively select a minimal subset of data to be annotated to maximally improve the segmentation performance. In reality, this setting faces a significant hurdle known as cold start problem: the lack of prior information to guide the initial selection of annotated data. A recent work has explored the impact of seeding strategies on the performance of AL methods [58]. Differently, we put forward a new path to access an auxiliary model via pre-training on the open-access auxiliary (source) dataset. This auxiliary model serves as a warm-up stage, allowing for smart target data selection for initial annotation. We formulate this new setting as active source-free domain adaptation, termed ASFDA. Take a further step, drawing inspiration from recent trends in 2D images [38,44,87,88], we delve into the third setting, active domain adaptation (ADA) for semantic segmentation of 3D point clouds. In this setting, a labeled auxiliary dataset is available, and the objective is to select target instances for annotating and learn a model with higher segmentation performance on the target test set.
Overall, in this work, we benchmark three distinct active learning settings for LiDAR semantic segmentation and deliver a simple and general baseline, Annotator, as illustrated in Figure 2. Borrowing the idea of modeling and computational techniques in geometry processing, we introduce a voxel-centric selection strategy dedicated to point clouds. Specifically, an input LiDAR scene is first voxelized into voxel grids, with a large voxel size to expand the local areas during the selection process. After obtaining final network predictions, importance estimation is carried out for each voxel grid using several common strategies such as Random, the softmax entropy (Entropy), and the margin between highest softmax scores (Margin). But considering only uncertainty for selection would be suboptimal [2,44,86]. Therefore, we introduce the concept of voxel confusion degree (VCD), which takes into account nearby predictions, capturing diversity and redundancy within a voxel grid. VCD enables the exploitation of local topology relations and point cloud structures. As a result, VCD can represent both uncertainty and diversity of a voxel grid in the LiDAR scene. In each active round, we query the top one voxel grid within each scan for annotation until the budget is exhausted. Despite the simplicity of our Annotator, it achieves performance on par with the fully-supervised counterpart requiring  $1000 \times$  fewer annotations and significantly outperforms all prevailing acquisition strategies.
Figure 2: An illustration of Annotator. Annotator is a new active learning baseline with broad applicability, capable of interactively querying a tiny subset of the most informative new (target) data points based on available inputs without task-specific designs. This includes (i) only unlabeled new (target) data being available (active learning, AL); (ii) access to an auxiliary (source) pre-trained model (active source-free domain adaptation, ASFDA); and (iii) availability of labeled source data and unlabeled target data (active domain adaptation, ADA). Remarkably, Annotator attains excellent results not only in in-domain settings but also manifests adaptive transfer to out-of-domain settings.
The contribution of this paper can be summarized in three aspects. First, we present a voxel-centric active learning baseline that significantly reduces the labeling cost and effectively facilitates learning with a limited budget, achieving near performance to that of fully-supervised methods with  $1000 \times$  fewer annotations. Second, we introduce a label acquisition strategy, the voxel confusion degree (VCD), which is more robust and diverse to select point clouds under a domain shift. Third, Annotator is generally applicable for various network architectures (voxel-, range- and bev-views), settings (in-distribution and out-of-distribution), and scenarios (simulation-to-real and real-to-real) with consistent gains. We hope this work could lay a solid foundation for label-efficient 3D applications.
2 A Generic Baseline
2.1 Preliminaries and overview
Problem setup. In the context of LiDAR semantic segmentation, a LiDAR scan is made of a set of point clouds and let  $X \in \mathbb{R}^{N \times 4}$ ,  $Y \in \mathbb{K}^N$  respectively denote  $N$  points and the corresponding labels.  $\mathbb{K}$  is a predefined semantic class vocabulary  $\mathbb{K} = \{1, \dots, K\}$  of  $K$  categorical labels. Each point  $x_i$  in  $X$  is a  $1 \times 4$  vector with a 3D Cartesian coordinate relative to the scanner  $(a_i, b_i, c_i)$  and an intensity value of returning laser beam. Our baseline works in the following settings: active learning (AL), active source-free domain adaptation (ASFDA), and active domain adaptation (ADA). First, we are given an unlabeled target domain  $\mathcal{D}^t = \{X^t \cup X^a\}$ , where  $X^t$  denotes unlabeled target point clouds and  $X^a$  denotes the selected points to be annotated and is initialized as empty set, i.e.,  $X^a = \emptyset$ . Next, for ASFDA and ADA, a labeled source domain  $\mathcal{D}^s = \{X^s, Y^s\}$  can be utilized only in pre-training stage and anytime respectively. Ultimately, given a limited budget, our goal is to iteratively select a subset of data points from  $\mathcal{D}^i$  to annotate until the budget is exhausted, all the while catching up with the performance of the fully-supervised model.
Overview. Figure 2 displays an overview of Annotator, which is a label-efficient baseline for LiDAR semantic segmentation. It is composed of two parts: 1) a generalist Annotator which contains a voxelization process to get voxel grids and an active function with online selection for picking the most valuable voxel grid of each input scan in each active round; 2) the pipelines of distinct active
learning settings are described. For AL, we interactively select a subset of voxels from the current scan to be annotated and train the network with these sparse annotated voxel grids. In the case of ASFDA, we begin by pre-training a network on the source domain through standard supervised learning. This warm-up network then serves as a strong initialization to aid the initial selection. As for ADA, except for the pre-training stage, we also make use of annotated source domain to promote the selection in each round and facilitate domain alignment. In the following, we will detail why we select salient and exemplar data points from a voxel-centric perspective and how to address cold start problem via an auxiliary model. After that, overall objectives for all three settings are elaborated.
2.2 A generalist Annotator
In this section, we proposed a general active learning baseline called Annotator. The core idea is to select salient and exemplar voxel grids from each LiDAR scan. It's important to note that previous researches have proposed frame-based [13, 94], region-based [82], and point-based [35] selection strategies. The first two usually require an offline stage, which may be infeasible at large scales. The last one is costly due to the sparsity of outdoor point clouds. By contrast, our voxel-centric selection focuses on querying salient and exemplar areas and annotating all points within those areas. This approach is more efficient and flexible. Moreover, it can be seamlessly applied to various network architectures, including voxel-, range- and bev-views, as demonstrated in the experiment section.
To implement it, we begin with the voxelization process as introduced in [9, 102]. Each input LiDAR scan  $X$  is transformed into a 3D voxel grid set  $V$ . This process involves sampling the continuous 3D input space into discrete voxel grids, where points falling into the same grid are merged. Each voxel grid serves as a selection unit. Mathematically, for a point  $x_{i} \in X$ , the corresponding voxel grid coordinate is  $(a_{i}^{v}, b_{i}^{v}, c_{i}^{v}) = \lfloor (a_{i}, b_{i}, c_{i}) / \triangle \rfloor$ , with  $\triangle$  denoting predefined voxel size. In our experiments, we have found that using a large voxel grid is more robust against noise and sparsity. Unless otherwise specified, we use  $\triangle_1 = 0.05$  for training and  $\triangle_2 = 0.25$  for the selection process.
Selection strategies. For each voxel grid  $v_{j} \in V$ , we assess its importance and select the best voxel grid per LiDAR scan in each active round. Initially, we employ a Random selection strategy. Subsequently, we explore softmax entropy (Entropy) and the margin between highest softmax scores (Margin). It's essential to note that while these common selection strategies are not technical contributions, they are necessary to build our baseline. Detailed calculations are provided below.
- Random: randomly select a target voxel grid  $v_{j}$  from  $V$  to be annotated in each round.
- Entropy: first calculate the softmax entropy of each point  $x_{i} \in v_{j}$  and then adopt the maximum value as the Entropy score of this grid, i.e.,  $\text{Entropy}(v_{j}) = \max_{x_{i} \in v_{j}} - p_{i} \log p_{i}$ , where  $p_{i}$  is the softmax score of point  $x_{i}$ . The voxel grid with the highest Entropy score is selected in each scan.
- Margin: first calculate the margin between highest softmax score of each point  $x_{i} \in v_{j}$  and then adopt the maximum value as the Margin score of this grid, i.e.,  $\mathrm{Margin}(v_j) = \max_{x_i \in v_j} (\max(p_i) - \max 2(p_i))$ , where  $\max 2(\cdot)$  is the second-largest value operator. In each scan, the voxel grid with the lowest Margin score is chosen.
The VCD strategy. Our voxel confusion degree (VCD) is motivated by an important observation: the previously mentioned selection strategies become less effective when models are applied in new domains due to mis-calibrated uncertainty estimation. Therefore, the VCD is designed to estimate category diversity within a voxel grid rather than uncertainty, making it more robust under domain shift. Here's how it works: we begin by obtaining pseudo label  $\hat{y}_i$  for each point  $x_i$ . Next, we divide points within  $v_j$  into  $K$  clusters:  $v_j^{<k>} = \{x_i^{<k}| x_i \in v_j, \hat{y}_i = k\}$ . This allows us to collect statistical information about the categories present in the voxel grid. With this information, we calculate VCD to assess the significance of voxel grids as follows:
$$
\operatorname {V C D} (v _ {j}) = - \sum_ {k = 1} ^ {K} \frac {| v _ {j} ^ {<   k >} |}{| v _ {j} |} \log \frac {| v _ {j} ^ {<   k >} |}{| v _ {j} |},
$$
where  $|\cdot|$  denotes the number of points in a set. Finally, voxel grid with the highest VCD score is selected in each scan. The insight is that a higher score indicates a greater category diversity within a voxel, which would be beneficial for model training once being annotated. In all experiments, Annotator is equipped with VCD by default, and the results indicate the superiority of VCD strategy.
Making a good first impression. To avoid the cold start problem mentioned before, we introduce a warm start mechanism that pre-trains an auxiliary model with an auxiliary (source) dataset, and then it is used to select voxel grids in the first round. This warm start stage is applied in ASFDA and ADA.
Discussion: balancing annotation cost and computation cost. Our primary focus is on reducing annotation cost while maintaining performance comparable to fully-supervised approaches. Let's consider simulation-to-real tasks as an example. The simplest setup involves active learning within the real dataset. However, this setup yields less satisfactory results due to the cold-start problem: the lack of prior information for selecting an initial annotated set. To address this, we utilize a synthetic dataset to train an auxiliary model in a brief warm-up stage, enabling smarter data selection in the first round. Importantly, this warm-up process is short, conducted only once, and results in minimal costs (both annotation and computation). For a detailed analysis, please refer to Appendix B.1.
2.3 Optimization
The overall loss function is the standard cross-entropy loss, which is defined as:
$$
\mathcal {L} _ {c e} (X) = \frac {1}{| X |} \sum_ {x _ {i} \in X} \sum_ {k = 1} ^ {K} - y _ {i} ^ {k} \log p _ {i} ^ {k},
$$
where  $K$  is the number of categories,  $y_{i}$  is the one-hot label of point  $x_{i}$  and  $p_i^k$  is the predicted probability of point  $x_{i}$  belonging to category  $k$ . Hereafter, for AL, the objective is  $\min_{\theta} \mathcal{L}_{ce}(X^a)$ ; for ASFDA, the objective is  $\min_{\theta_s} \mathcal{L}_{ce}(X^a)$ ; for ADA, the objective is  $\min_{\theta_s} \mathcal{L}_{ce}(X^s) + \mathcal{L}_{ce}(X^a)$ . Here,  $\theta$  and  $\theta_s$  denote training from scratch and training from the source pre-trained model, respectively.
In this section, we conduct extensive experiments on several public benchmarks under three active learning scenarios: (i) AL setting where all available data points are from unlabeled target domain; (ii) ASFDA setting where we can only access a pre-trained model from the source domain; (iii) ADA setting where all data points from source domain can be utilized and a portion of unlabeled target data is selected to be annotated. We first introduce the dataset used in this work and experimental setup and then present experimental results of baseline methods and extensive analyses of Annotator.
3.1 Experiment setup
Datasets. We build all benchmarks upon SynLiDAR [84], SemanticKITTI [3], SemanticPOSS [42], and nuScenes [4], constructing two simulation-to-real and two real-to-real adaptation scenarios. SynLiDAR [84] is a large-scale synthetic dataset, which has 198,396 LiDAR scans with point-level segmentation annotations over 32 semantic classes. Following [84], we use 19,840 point clouds as the training data. SemanticKITTI (KITTI) [3] is a popular LiDAR segmentation dataset, including 2,9130 training scans and 6,019 validation scans with 19 categories. SemanticPOSS (POSS) [42] consists of 2,988 real-world scans with point-level annotations over 14 semantic classes. As suggested in [42], we use the sequence 03 for validation and the remaining sequences for training. nuScenes [4] contains 19,130 training scans and 4,071 validation scans with 16 object classes.
Class mapping. To ensure compatibility between source and target labels across datasets, we perform class mapping. Specifically, we map SynLiDAR labels into 19 common categories for SynLiDAR  $\rightarrow$  KITTI and 13 classes for SynLiDAR  $\rightarrow$  POSS. Similarly, we map labels into 7 classes for KITTI  $\rightarrow$  nuScenes and nuScenes  $\rightarrow$  KITTI. We refer readers to Appendix A.1 for detailed class mappings.
Implementation details. We primarily adopt MinkNet [9] and SPVCNN [71] as the segmentation backbones. Note that, all experiments share the same backbones and are within the same codebase, which are implemented using PyTorch [43] on a single NVIDIA Tesla A100 GPU. We use the SGD optimizer and adopt a cosine learning rate decay schedule with initial learning rate of 0.01. And the batch size for both source and target data is 16. For additional details, please consult Appendix A.2. Finally, we evaluate the segmentation performance before and after adaptation, following the typical evaluation protocol [47] in LiDAR domain adaptive semantic segmentation [29, 31, 55, 82, 83].
3.2 Experimental results
Quantitative results summary. We initially evaluate Annotator on four benchmarks and two backbones while adhering to a fixed budget of selecting and annotating five voxel grids in each scan.
Method
Source-/Target-Only	Simulation-to-Real
SynLiDAR 19→KITTI SynLiDAR 13→POSS	Real-to-Real
KITTI 7→nuScenes nuScenes 7→KITTI
22.0 / 61.1	30.4 / 56.7	28.4 / 82.5	34.6 / 83.3
Random	35.3 / 36.3 / 45.3	27.4 / 30.9 / 43.4	66.0 / 67.5 / 71.9	70.9 / 69.7 / 74.7
Entropy [78]	39.8 / 49.6 / 50.1	42.8 / 45.5 / 49.9	59.7 / 60.3 / 73.1	70.7 / 69.1 / 74.0
Margin [26]	46.9 / 44.3 / 49.0	41.6 / 44.1 / 46.9	60.2 / 59.2 / 71.4	73.1 / 70.3 / 76.7
Annotator	53.7 / 54.1 / 57.7	44.9 / 48.2 / 52.0	70.4 / 72.4 / 75.9	76.8 / 75.3 / 81.8
Table 1: Quantitative summary of all baselines' performance based on MinkNet [9] over various LiDAR semantic segmentation benchmarks using only 5 voxel grids. Source-/Target-Only correspond to the model trained on the annotated source/target dataset which are considered as lower/upper bound. Note that results are reported following the order of AL / ASFDA / ADA in each cell.
Method
Source-/Target-Only	Simulation-to-Real
SynLiDAR 19→KITTI SynLiDAR 13→POSS	Real-to-Real
KITTI 7→nuScenes nuScenes 7→KITTI
24.2 / 63.7	37.0 / 51.9	21.3 / 81.3	47.1 / 85.0
Random	40.9 / 41.7 / 51.0	35.5 / 37.8 / 42.3	65.0 / 66.9 / 64.3	70.4 / 68.1 / 75.8
Entropy [78]	52.7 / 52.1 / 52.8	35.2 / 40.5 / 46.8	61.3 / 66.0 / 66.3	69.5 / 67.4 / 72.6
Margin [26]	47.1 / 49.9 / 50.7	42.9 / 44.8 / 47.1	57.8 / 60.3 / 63.2	72.3 / 73.0 / 75.3
Annotator	52.8 / 54.6 / 55.6	44.9 / 47.5 / 50.9	71.4 / 72.1 / 72.3	79.5 / 80.5 / 78.4