SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaoyu Tian $^{1*}$  Tao Jiang $^{1,3*}$  Longfei Yun $^{1}$  Yucheng Mao $^{1}$  Huitong Yang $^{4}$  Yue Wang $^{2}$  Yilun Wang $^{1}$  Hang Zhao $^{1,3,4\dagger}$
$^{1}$ IIIS, Tsinghua University  $^{2}$ University of Southern California  $^{3}$ Shanghai AI Lab  $^{4}$ Shanghai Qi Zhi Institute
Figure 1: Our Occ3D dataset demonstrates rich semantic and geometric expressiveness. (a) Diversity of scenes in the Occ3D dataset; (b) Out-of-vocabulary objects, also known as General Objects (GOs), that cannot be extensively enumerated in the real world; (c) Irregularly-shaped objects that 3D bounding boxes fail to represent their accurate geometry.
Robotic perception requires the modeling of both 3D geometry and semantics. Existing methods typically focus on estimating 3D bounding boxes, neglecting finer geometric details and struggling to handle general, out-of-vocabulary objects. 3D occupancy prediction, which estimates the detailed occupancy states and semantics of a scene, is an emerging task to overcome these limitations. To support 3D occupancy prediction, we develop a label generation pipeline that produces dense, visibility-aware labels for any given scene. This pipeline comprises three stages: voxel densification, occlusion reasoning, and image-guided voxel refinement. We establish two benchmarks, derived from the Waymo Open Dataset and the nuScenes Dataset, namely Occ3D-Waymo and Occ3D-nuScenes benchmarks. Furthermore, we provide an extensive analysis of the proposed dataset with various baseline models. Lastly, we propose a new model, dubbed Coarse-to-Fine Occupancy (CTF-Occ) network, which demonstrates superior performance on the Occ3D benchmarks. The code, data, and benchmarks are released at https://tsinghua-mars-lab.github.io/Occ3D/.
*Authors contributed equally.
† Corresponding to: hangzhao@mail.tsinghua.edu.cn
3D perception is a crucial component in vision-based robotic systems like autonomous driving. One of the most popular visual perception tasks is 3D object detection, which estimates the 3D locations and dimensions of objects defined in a pre-determined ontology tree [48, 22]. While the resulting 3D bounding boxes are compact, the level of expressiveness they provide is restricted, as illustrated in Figure 1: (1) 3D bounding box representation erases the geometric details of objects, a construction vehicle has a mechanical arm that protrudes from the main body; (2) uncommon categories, like trash cans on the streets, are often ignored and not labeled in the datasets [4, 43] since object categories in the open world cannot be extensively enumerated.
These limitations call for a general and coherent representation that can model the detailed geometry and semantics of objects both within and outside of the ontology tree. 3D Occupancy Prediction, i.e. understanding every voxel in the 3D space, is an important task to achieving this goal. We formalize the 3D occupancy prediction task as follows: a model needs to jointly estimate the occupancy state and semantic label of every voxel in the scene from images [2, 24, 5]. The occupancy state of each voxel can be categorized as free, occupied, or unobserved. For occupied voxels, semantic labels are assigned. For objects that are not in the predefined categories, they are labeled as General Objects (GOs). Although GOs are rare, they are essential for perception tasks with safety considerations since they are typically undetected by 3D object detection with predefined categories.
Despite recent advancements in 3D occupancy prediction [5, 16, 53], there is a notable absence of high-quality datasets together with benchmarks. Constructing such a dataset is challenging due to three major issues: sparsity, occlusion and 3D-2D misalignment. To overcome these hurdles, we create a semi-automatic label generation pipeline that consists of three steps: voxel densification, occlusion reasoning, and image-guided voxel refinement. Each step within our pipeline is validated through a 3D-2D consistency metric, demonstrating that our proposed label generation pipeline effectively generates dense and visibility-aware annotations.
Building upon the public Waymo Open Dataset [43], nuScenes [4] and Panoptic nuScenes [11] Dataset, we produce two benchmarks for our task accordingly, Occ3D-Waymo and Occ3D-nuScenes. Compared to conventional datasets such as SemanticKITTI [2] and KITTI-360 [24], our Occ3D is the first dataset to offer the surround-view images and high-resolution 3D voxel occupancy representation with the most diverse scenarios.
A series of recent occupancy prediction models are reproduced and benchmarked on Occ3D. Additionally, we propose CTF-Occ, a transformer-based Coarse-To-Fine 3D Occupancy prediction network. CTF-Occ achieves superior performance by aggregating 2D image features into 3D space via cross-attention in an efficient coarse-to-fine fashion.
The contributions of this work are as follows: (1) We introduce Occ3D, a high-quality 3D occupancy prediction benchmark to facilitate research in this emerging area; (2) We put forward a rigorous automatic label generation pipeline for constructing the Occ3D benchmark, with comprehensive validation of the effectiveness of the pipeline; (3) We benchmark existing model and propose a new CTF-Occ network that achieves superior 3D occupancy prediction performance.
3D detection. The goal of 3D object detection is to estimate the locations and dimensions of objects within a predefined ontology. 3D object detection is often performed in LiDAR point clouds [55, 18, 51, 52, 34, 36, 10, 39]. More recently, vision-based 3D object detection has gained more attention due to its low cost and rich semantic content [41, 46, 48, 22, 26, 33, 28, 13, 27, 15, 21, 31, 25]. Several LiDAR-camera fusion methods are also proposed [35, 8, 28].
3D occupancy prediction. A related task of 3D occupancy prediction is Occupancy Grid Mapping (OGM) [30, 44, 47], a classical task in mobile robots that aims to generate probabilistic maps from sequential noisy range measurements. OGM can be solved within a Bayesian framework, some recent works further combine semantic segmentation with OGM for downstream tasks [17, 42, 37]. Note that OGM requires range sensors, and also makes the assumption that the scene is static over time. The 3D occupancy prediction task does not have these constraints and can be applied in
Table 1: Dataset comparison. Comparing Occ3D Datasets with other occupancy prediction datasets. Surround = √ represents surround-view image inputs. C, D, L denote camera, depth and LiDAR.
Dataset	Type	Surround	Modality	# Classes	# Sequences	# Frames	Volume Size	Resolution (m)
NYUv2 [40]	Indoor	X	C & D	11	464	1449	[240, 240, 14]	-
ScanNet [9]	Indoor	X	C & D	11	1513	1513	[62, 62, 31]	-
SemanticKITTI [2]	Outdoor	X	C & L	28	22	4,3000	[256, 256, 32]	[0.2, 0.2, 0.2]
KITTI-360 [23]	Outdoor	Fisheye	C & L	19	11	90,960	[256, 256, 32]	[0.2, 0.2, 0.2]
Occ3D-nuScenes	Outdoor	✓	C & L	16+GO	1000	40,000	[200, 200, 16]	[0.4, 0.4, 0.4]
Occ3D-Waymo	Outdoor	✓	C & L	14+GO	1000	200,000	[3200, 3200, 128]	[0.05, 0.05, 0.05]
vision-only robotic systems in dynamic scenes. Recently, TPVFormer [16] proposes a tri-perspective view method to predict 3D occupancy. However, its output is sparse due to LiDAR supervision.
Semantic scene completion. Another related task is Semantic Scene Completion (SSC) [1, 6, 9, 3, 24, 50, 38, 49, 7, 19, 50, 32, 20, 29, 54], whose goal is to estimate a dense semantic space from partial observations. SSC differs from 3D occupancy prediction in two ways: (1) SSC focuses on inferring occluded regions given visible parts, while occupancy prediction does not intend to estimate the invisible regions; (2) existing SSC task typically deals with static scenes, whereas occupancy prediction works with dynamic ones.
3 Occ3D Dataset
Given a sequence of sensor inputs, the goal of 3D occupancy prediction is to estimate the state of each voxel in the 3D scene. Specifically, the input of the task is a  $T$ -frame historical sequence of  $N$  surround-view camera images  $\{\mathcal{I}_{i,t} \in \mathbf{R}^{H_i \times W_i \times 3}\}$ , where  $i = 1, \dots, N$  and  $t = 1, \dots, T$ . We also assume known sensor intrinsic parameters  $\{K_i\}$  and extrinsic parameters  $\{[R_i|t_i]\}$  in each frame. The ground truth labels are the voxel states, including occupancy state ("occupied", "free", or "unobserved") and semantic label (category, or "unknown"). For example, a voxel on a vehicle is labeled as ("occupied", "vehicle"), and a voxel in the free space is labeled as ("free", None). Note that the 3D occupancy prediction framework also supports extra attributes as outputs, such as instance IDs and motion vectors; we leave them as future work.
We generate two 3D occupancy prediction datasets, Occ3D-nuScenes and Occ3D-Waymo. Occ3D-nuScenes contains 600 scenes for training, 150 scenes for validation, and 150 for testing, totaling 40,000 frames. It has 16 common classes with an additional general object (GO) class. Each sample covers a range of  $[-40\mathrm{m}, -40\mathrm{m}, -1\mathrm{m}, 40\mathrm{m}, 40\mathrm{m}, 5.4\mathrm{m}]$  with a voxel size of  $[0.4\mathrm{m}, 0.4\mathrm{m}, 0.4\mathrm{m}]$ . Occ3D-Waymo contains 798 sequences for training, 202 sequences for validation, accumulating 200,000 frames. It has 14 known object classes with an additional GO class. Each sample covers a range of  $[-80\mathrm{m}, -80\mathrm{m}, -1\mathrm{m}, 80\mathrm{m}, 80\mathrm{m}, 5.4\mathrm{m}]$ , with an extremely fine voxel size of  $[0.05\mathrm{m}, 0.05\mathrm{m}, 0.05\mathrm{m}]$ .
Occ3D stands out when compared with other datasets, as shown in Table 1. The indoor datasets NYUv2 and ScanNet lack surround images and consist of fewer sequences and frames. SemanticKITTI and KITTI-360, the two other outdoor datasets, also lack surround images, with the exception of KITTI-360's fisheye images. For the safety of autonomous driving, the general object class is particularly important, a feature that is not available in the SemanticKITTI and KITTI-360 datasets. Furthermore, Occ3D-Waymo is currently the 3D occupancy dataset with the most diverse scenarios, comprehensive labels, and the highest resolution among all open-source datasets.
3.3 Dataset Construction Pipeline
Annotating 3D occupancy from images is impossible due to the lack of accurate depth and geometry. Therefore, we take advantage of LiDAR scans and their annotations to construct high-quality occupancy labels. However, there are three primary hurdles: sparsity, occlusion, and 3D-2D misalignment. Sparsity refers to the fact that LiDAR scans are sparse, thereby hindering the acquisition
Figure 2: Overview of the label generation pipeline. The pipeline consists of three main steps: voxel densification, occlusion reasoning, and image-guided voxel refinement. Voxel densification consists of object segmentation, multi-frame aggregation, and label assignment.
of dense voxels. Occlusion, on the other hand, is concerned with the identification of voxels that, once densified, become invisible in the current image view due to occlusion. 3D-2D misalignment pertains to the disparities when projecting the 3D voxels onto 2D images, often induced by sensor noises or pose errors.
Our proposed label generation pipeline addresses the above challenges, an overview is shown in Figure 2. Initially, in voxel densification, we increase the density of the point clouds by performing multi-frame aggregation for both static and dynamic objects separately. Then we employ a K-nearest neighbor algorithm to assign labels to unlabeled points and utilize mesh reconstruction to perform hole-filling. Subsequently, we carry out occlusion reasoning from both LiDAR and camera perspectives, utilizing a ray-casting operation to label the occupancy state of each voxel. Finally, misaligned voxels are eliminated through an image-guided voxel refinement process. We provide pseudo-code and the hyper-parameters of each step in the Appendix.
3.3.1 Voxel Densification
LiDAR data is inherently sparse, to acquire dense point clouds: 1) We aggregate all points throughout the frames, treating dynamic objects and static background points separately; 2) We take advantage of unlabeled frames (which we'll refer to as non-keyframes) and use a K-Nearest Neighbors (KNN) algorithm to assign semantic labels; 3) In spite of frame aggregation, there persist holes on the object surfaces, we fill these holes with mesh reconstruction.
Dynamic and static objects segmentation. Point clouds derived from individual frames are categorized into "dynamic objects" and "static scenes". The static scenes contain entities such as ground, buildings, and road signs that do not exhibit positional change over time. Dynamic objects, such as cars and pedestrians need to be segregated since naive temporal aggregation results in motion blur.
Multi-frame aggregation. After segregating dynamic objects from static scenes, multi-frame aggregation is conducted separately on them. For dynamic objects, we extract the points located within the annotated or tracked box and subsequently transform them from sensor coordinates to box coordinates. By concatenating these transformed points, we densify the point cloud of dynamic objects. For the static scene, we simply aggregate its points across time in the global coordinate system. The static scene is then fused with the aggregated dynamic objects in the current frame, thereby generating a single-frame dense point cloud.
(a) Occlusion Reasoning for Visibility Mask
(b) Image-guided Voxel Refinement
Figure 3: Visibility and refinement. (a) LiDAR visibility: a voxel is "occupied" if it reflects LiDAR (red voxels), or "free" if it is traversed through by a ray (white voxels); Camera visibility: Any voxel not scanned by camera rays is set to "unobserved" (blue and yellow voxels). (b) Image-guided voxel refinement: during ray casting, when the first voxel with the same semantic label as the pixel label is encountered, we set the previously traversed voxel states to "free" (green voxels).
KNN for label assignment. The task of directly annotating each point in every frame is labor-intensive. Current datasets only annotate a selected portion of the frames - for instance, the Waymo dataset proceeds at a rate of  $2\mathrm{Hz}$ , whereas Lidar scans operate at a  $10\mathrm{Hz}$  frequency. To utilize the unlabeled frames, we employ the K-nearest neighbors (KNN) algorithm to assign semantic labels to each unlabeled point. Specifically, for each point in the unlabeled frame, we find the K nearest keyframe points and assign the majority semantic label.
Mesh reconstruction. After multi-frame aggregation, the density of point clouds is still not enough to produce high-quality dense voxels: a smaller voxel size may lead to objects with many holes, while a larger voxel size could induce excessive smoothness. To mitigate these issues, we perform mesh reconstruction. For non-ground categories, we optimize surfaces through VDBFusion [45], an approach for volumetric surface reconstruction based on truncated signed distance functions (TSDF). The flexibility and efficacy of VDBFusion surpass traditional methods such as Poisson surface reconstruction. For the ground, VDBFusion fails as small ray angles result in incorrect TSDF values. We instead establish uniform virtual grid points and fit each local surface mesh using points within a small region. After reconstructing the meshes, dense point sampling is performed, and KNN is further adopted to assign semantic labels to the sampling points.
3.3.2 Occlusion Reasoning for Visibility Mask
We perform occlusion reasoning and introduce LiDAR visibility mask and camera visibility mask to further enhance our 3D occupancy prediction benchmark.
Aggregated LiDAR visibility mask. To obtain a 3D occupancy grid from aggregated LiDAR point clouds, a straightforward way is to set the voxels containing points to be "occupied" and the rest to "free". However, since LiDAR points are sparse, some occupied voxels are not scanned by LiDAR beams, and can be mislabeled as "free". To avoid this issue, we perform a ray casting operation to determine the visibility of each voxel, as shown in Figure 3a. Concretely, we cast a ray from the sensor origins to each LiDAR point. A voxel is considered visible if it either reflects LiDAR points, or if it is traversed through by a ray. If neither condition is met, the voxel is classified as "unobserved".
Camera visibility mask. We connect each occupied voxel center with the camera origin, thereby forming a ray. Along each ray, we set the first occupied voxel as "observed", and the remaining as "unobserved". Any voxel not scanned by camera rays is set to "unobserved" as well. Determining the visibility of a voxel is crucial for the evaluation of the 3D occupancy prediction task: evaluation is only performed on the "observed" voxels in both the LiDAR and camera views.
3.3.3 Image-guided Voxel Refinement
Influences such as LiDAR noise and pose drifts can cause the 3D shape of objects to appear larger than their actual physical dimensions. To rectify this, we further refine the dataset by eliminating
(a) 2D ROI
(b) 2D pixel semantic label
Figure 4: 3D-2D consistency (a) 2D ROI within single-frame LiDAR scan range. (b) Semantic labels of a single image within the 2D ROI. (c) The reprojection of 3D voxel semantic labels onto the image within the 2D ROI.
(c) 3D voxel semantic label
incorrectly occupied voxels, guided by semantic segmentation masks of images. As shown in Figure 3b, to obtain the correspondence between 3D voxels and 2D pixels, we adopt a ray casting operation similar to the one in the previous section: connecting each occupied voxel center with the camera center to form a ray, and traverse the voxel that this ray passes through from near to far from the pixel origin. When the first voxel with the same semantic label as the pixel label is encountered, we set the previously traversed voxel states to "free". This step greatly improves the shape at object boundaries.
4 Quality Check
Acquiring an occupancy representation that adheres to the complete shape of all objects is challenging. Therefore, evaluating the quality of the dataset and ensuring the effectiveness of each step in our pipeline is critical. To this end, we propose a method that evaluates the quality of occupancy by checking semantic consistency between 2D pixels and their corresponding voxel.
4.1 3D-2D consistency
Compared to 3D occupancy semantic labels obtained through aggregation and reconstruction, 2D semantic masks manually annotated by humans are highly accurate. Thus, we assess the quality of the dataset by verifying the 3D-2D consistency between semantic labels of 3D voxels and their corresponding 2D image pixels. We calculate 3D-2D consistently in three steps: filtering the 2D pixel region involved in consistency calculation for the current frame, identifying the corresponding 3D voxels of this pixel region, and finally, computing their 3D-2D semantic consistency.
2D ROI. 2D images contain objects that are beyond the scanning range of the LiDAR sensor. When calculating 3D-2D consistency, we use the maximum range covered by a single LiDAR frame as the 2D Region of interest (ROI). Specifically, we project single-frame LiDAR points onto the 2D image coordinate system using LiDAR-to-camera transformation. Then, our algorithm traverses in the horizontal coordinate direction and selects the highest vertical coordinate of the projected points in each vertical column as the height of that column. As shown in Figure 4a, all pixels below this height are treated as the 2D valid region involved in the consistency calculation.
3D label query. After determining the 2D ROI in each image, we identify its corresponding 3D voxels for these regions. Since each voxel has a certain volume, directly projecting them onto a 2D image poses a multi-pixel association issue. Moreover, when the projection overlap occurs, determining the corresponding occlusion relationship becomes complicated. We instead query corresponding 3D voxels for each 2D image pixel. Specifically, for each pixel in the selected region, we perform ray traversal and find the closest 3D voxel to the ray.
Metrics. To evaluate the dataset quality, for each pixel in an image, we compare its semantic label with the semantic prediction of its corresponding 3D voxel. We adopt the standard Precision, Recall, Intersection-over-Union(IoU), and mean Intersection-over-Union(mIoU) metric.
Table 2: Quantitative results for design choices. SFP, single frame points; MFP, aggregating points from unlabeled frames; VS, short for voxel size; Mesh, showcasing mesh reconstruction; and IGR, denoting image-guided voxel refinement. The three numbers from top to bottom in each choice are IoU, recall, and precision for the specific class.
SFP	MFP	VS	Mesh	IGR	vehicle	bicyclist	ped	sign	road	pole	cone	bicycle	building	mIOU
✓	-	5.87	5.12	3.65	3.47	0.33	0.10	0.09	0.11	0.34	13.32
8.53	6.61	4.81	3.85	0.33	0.10	0.09	0.11	0.34
95.38	58.66	60.13	61.44	92.78	34.90	25.35	60.12	66.93
✓	✓	-	37.89	37.99	28.25	12.57	11.70	5.48	3.51	6.01	15.49	17.65
40.02	58.77	37.21	14.80	12.06	6.32	3.99	6.45	17.69
87.48	51.79	53.98	45.45	79.72	29.25	22.37	46.76	55.49
✓	✓	0.1	75.23	38.66	30.78	33.77	56.30	30.58	24.03	31.36	49.85	41.17
91.20	87.00	60.90	56.80	67.35	55.97	42.22	37.00	68.66
81.12	41.03	38.37	45.45	95.53	40.27	35.81	67.28	64.53
✓	✓	0.1	✓	75.13	37.97	30.36	32.88	82.79	16.63	17.48	32.46	52.27	41.99
90.98	83.02	55.07	54.93	85.34	63.57	58.13	40.77	77.81
81.17	41.17	40.36	45.02	96.52	18.38	19.99	61.43	61.43
✓	✓	0.05	✓	78.76	46.33	34.04	34.85	64.56	18.28	20.57	42.59	52.27	43.58
89.20	84.22	55.42	52.02	67.43	63.17	58.89	53.71	84.73
87.06	50.74	46.87	51.36	93.80	20.46	24.02	67.28	57.71
✓	✓	0.05	✓	✓	88.82	76.89	47.54	50.18	71.97	31.77	32.09	66.40	60.90	58.50
91.34	92.12	63.78	61.20	75.11	73.80	62.44	77.67	94.73
96.98	82.31	65.11	73.59	94.51	35.80	39.76	82.07	63.04
Table 2 shows the performance gain of each proposed step of our auto-labeling pipeline. The 3D-2D consistency is evaluated in a subset of Occ3D-Waymo. Single-frame points (SFP) means that we only use a single-frame point cloud to calculate its 3D-2D consistency using the previously proposed method. As shown in the table, our method achieves high SFP precision and low recall. In addition to SFP, we aggregate points from multiple frames (MFP). Compared to SFP, MFP sees a significant improvement in recall, but its precision decreases to a certain extent, which is caused by the LiDAR noise and/or pose errors. Based on MFP, we study the effect of voxelization, which leads to better precision and recall. This further validates the effect of correction on pose inaccuracies. As mentioned before, a small voxel size results in objects containing many holes, while a larger voxel size leads to over smoothness. The former results in low recall, while the latter results in low precision. We use mesh reconstruction to alleviate the hole issue in objects caused by a small voxel size, which is reflected by the comparison between third row and fifth row in the table. Finally, we demonstrate that our proposed image-guided refinement indeed promotes the 3D-2D semantic consistency, shown in the last row.
5 Coarse-to-Fine Occupancy Network
To deal with the challenging 3D occupancy prediction problem, we present a new transformer-based model named Coarse-to-Fine Occupancy (CTF-Occ) network. An overview of CTF-Occ network is shown in Figure 5. First, 2D image features are extracted from multi-view images with an image backbone. Then, 3D voxel queries aggregate 2D image features into 3D space via a cross-attention operation. Our approach involves using a pyramid voxel encoder that progressively improves voxel feature representation through incremental token selection and spatial cross-attention in a coarse-to-fine fashion. This approach enhances the spatial resolution and refines the detailed geometry of objects, ultimately leading to more accurate 3D occupancy predictions.
Incremental token selection. The task of predicting 3D occupancy requires a detailed representation of geometry, but this can result in significant computational and memory costs if all 3D voxel tokens are used to interact with regions of interest in the multi-view images. Given that most 3D voxel grids