SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 5: The architecture of CTF-Occ network. CTF-Occ consists of an image backbone, a coarse-to-fine voxel encoder, and an implicit occupancy decoder.
in a scene are empty, we propose an incremental token selection strategy that selectively chooses foreground and uncertain voxel tokens in cross-attention computation. This strategy enables adaptive and efficient computation without sacrificing accuracy. Specifically, at the beginning of each pyramid level, each voxel token is fed into a binary classifier to predict whether this voxel is empty or not. We use the binary ground-truth occupancy map as supervision to train the classifier. In our approach, we select the K-most uncertain voxel tokens for the subsequent feature refinement.
Spatial cross attention. At every level of the pyramid, we first select the top-K voxel tokens and then aggregate the corresponding image features. In particular, we apply 3D spatial cross-attention [22] to further refine the voxel features.
Convolutional feature extractor. Once we apply deformable cross-attention to the relevant image features, we proceed to update the features of the foreground voxel tokens. Then, we use a series of stacked convolutions to enhance feature interaction throughout the entire 3D voxel feature maps. At the end of the current level, we upsample the 3D voxel features using trilinear interpolation.
Occupancy decoder. The CTF voxel encoder generates voxelized feature output  $V_{out} \in \mathbb{R}^{W \times H \times L \times C}$ . Then the voxel features  $V_{out}$  are fed into several MLPs to obtain the final occupancy prediction  $O \in \mathbb{R}^{W \times H \times L \times C'}$ , where  $C'$  is the number of the semantic classes. Furthermore, we introduce an implicit occupancy decoder that can offer arbitrary resolution output by utilizing implicit neural representations. The implicit decoder is implemented as an MLP that outputs a semantic label by taking two inputs: a voxel feature vector extracted by the voxel encoder and a 3D coordinate inside the voxel.
To benchmark our proposed Occ3D datasets and our CTF-Occ model, we evaluate existing 3D occupancy prediction methods on Occ3D-nuScenes and Occ3D-Waymo.
Dataset and Metrics. Occ3D-Waymo contains 1,000 publicly available sequences in total, where 798 scenes are for training and 202 scenes are for validation. The scene range is set from -40m to 40m along X and Y axis, and from -5m to 7.8m along Z axis. Occ3D-nuScenes contains 700 training scenes and 150 validation scenes. The occupancy scope is defined as -40m to 40m for X and Y axis, and -1m to 5.4m for the Z axis. We choose a voxel size of 0.4m to conduct our experiments on both two datasets. We adopt the metrics of Intersection-over-Union (IoU) and mean Intersection-over-Union(mIoU) to evaluate performance.
Table 3: 3D occupancy prediction performance on the Occ3D-nuScenes dataset. Cons. Veh represents construction vehicle and Dri. Sur is for driveable surface.
Method	others	barrier	bicycle	bus	car	Cons. Veh	motorcycle	pedestrian	traffic cone	trailer	truck	Dri. Sur	other flat	sidewalk	terrain	mammade	vegetation	mIoU
MonoScene [5]	1.75	7.23	4.26	4.93	9.38	5.67	3.98	3.01	5.90	4.45	7.17	14.91	6.32	7.92	7.43	1.01	7.65	6.06
TPVFormer [16]	7.22	38.90	13.67	40.78	45.90	17.23	19.99	18.85	14.30	26.69	34.17	55.65	35.47	37.55	30.70	19.40	16.78	27.83
BEVDet [14]	4.39	30.31	0.23	32.26	34.47	12.97	10.34	10.36	6.26	8.93	23.65	52.27	24.61	26.06	22.31	15.04	15.10	19.38
OccFormer [53]	5.94	30.29	12.32	34.40	39.17	14.44	16.45	17.22	9.27	13.90	26.36	50.99	30.96	34.66	22.73	6.76	6.97	21.93
BEVFormer [22]	5.85	37.83	17.87	40.44	42.43	7.36	23.88	21.81	20.98	22.38	30.70	55.35	28.36	36.0	28.06	20.04	17.69	26.88
CTF-Occ (Ours)	8.09	39.33	20.56	38.29	42.24	16.93	24.52	22.72	21.05	22.98	31.11	53.33	33.84	37.98	33.23	20.79	18.0	28.53
Table 4: 3D occupancy prediction performance on the Occ3D-Waymo dataset. Cons. Cone represents the construction cone.
Method	GO	vehicle	bicyclist	pedestrian	sign	traffic light	pole	Cons. Cone	bicycle	motorcycle	building	vegetation	tree trunk	road	sidewalk	mIoU
BEVDet [14]	0.13	13.06	2.17	10.15	7.80	5.85	4.62	0.94	1.49	0.0	7.27	10.06	2.35	48.15	34.12	9.88
TPVFormer [16]	3.89	17.86	12.03	5.67	13.64	8.49	8.90	9.95	14.79	0.32	13.82	11.44	5.8	73.3	51.49	16.76
BEVFormer [22]	3.48	17.18	13.87	5.9	13.84	2.7	9.82	12.2	13.99	0.0	13.38	11.66	6.73	74.97	51.61	16.76
CTF-Occ (Ours)	6.26	28.09	14.66	8.22	15.44	10.53	11.78	13.62	16.45	0.65	18.63	17.3	8.29	67.99	42.98	18.73
LiDAR-Only	1.01	57.41	35.31	20.33	11.7	13.01	36.21	7.81	0.13	0.0	57.83	54.71	27.07	69.15	54.47	29.74
BEVFormer-Fusion	5.11	64.61	52.35	21.52	32.74	17.1	42.62	27.75	13.36	0.05	63.65	60.51	35.64	81.89	66.84	39.05
Architecture. We extend two main-stream BEV models - BEVDet [14] and BEVFormer [22] to the 3D occupancy prediction task. We replace their original detection decoders with the occupancy decoder adopted in our CTF-Occ network and remain their BEV feature encoders. We employ ResNet-101 [12] pretrained on FCOS3D [46] as the image backbone and the image size is resized to  $(640 \times 960)$  for Occ3D-Waymo and  $(928 \times 1,600)$  for Occ3D-nuScenes. We also evaluate three existing 3D occupancy prediction methods - MonoScene [5], TPVFormer [16], and OccFormer [53] on our proposed Occ3D datasets. Additionally, we conduct experiments using LiDAR as an input on the Waymo dataset. "LiDAR-Onl" refers to adopting single frame LiDAR as input. Voxelization is applied with a voxel size of  $[0.1, 0.1, 0.4]$  on the x, y, and z axes respectively. Subsequently, a ResNet is employed to extract dense voxel features, which are then fed to the occupancy prediction head. The "BEVFormer-Fusio" method incorporates both camera and LiDAR inputs. We extract features from the same LiDAR branch and fuse them with the camera features captured by BEVFormer in the BEV space.
Our proposed CTF-Occ adopts a learnable voxel embedding with a shape of  $200 \times 200 \times 256$ . The voxel embedding will first pass through four encoder layers without token selection. There are three pyramid stage levels for the Occ3D-Waymo dataset, and the resolution of the z-axis in each stage is 8, 16, and 32. The resolution of the z-axis in each stage for the Occ3D-nuScenes dataset is 8 and 16 for the two pyramid stages. Each stage contains one SCA layer and an incremental token selection module to choose K non-empty voxels with the highest scores. The top-k ratio for the incremental token selection strategy is set to 0.2 for all pyramid stages.
Loss function. To optimize the occupancy prediction, we use the OHEM loss for model training  $\mathcal{L}_{occ} = \sum_{k}W_{k}\mathcal{L}(g_{k},p_{k})$ , where  $W_{k}$ ,  $g_{k}$ , and  $p_k$  represent the loss weight, the label, and the prediction result for the  $k$ -th semantic class. In addition, we supervise the binary classification head in each pyramid level with binary voxel masks. The binary voxel masks are generated by processing the semantic occupancy label at each spatial resolution  $s_i$  using  $f(g,s_i)$ , and the output of the binary classification head in the  $i$ -th level is denoted as  $p_i$ . The loss for the binary classification is defined as  $\mathcal{L}_{bin} = \sum_{i}\mathcal{L}(f(g,s_{i}),p_{i})$ , where  $i$  represents the  $i$ -th pyramid level.
6.2 Comparing with previous methods
Occ3D-nuScenes. Table 3 shows the performance of 3D occupancy prediction compared to related methods on the Occ3D-nuScenes dataset. It can be observed that our method performs better in all classes than previous baseline methods under the IoU metric. Our CTF-Occ surpass BEVFormer by  $1.65\mathrm{mIoU}$ . The observations are consistent with those in the Occ3D-Waymo dataset.
Occ3D-Waymo. We compare the performance of our CTF-Occ network with state-of-the-art models on our newly proposed Occ3D-Waymo dataset. Results are shown in Table 4. Our method outperforms previous methods by remarkable margins, increasing the mIoU by 1.97. Especially for some objects such as traffic cone and vehicle, our method surpasses the baseline method by 2.88 and 10.23 IoU respectively. This is because we capture the features in the 3D voxel space without compressing the heigh, which will preserve the detailed geometry of objects. The results indicate the effectiveness of our coarse-to-fine voxel encoder.
6.3 Ablation study
In this section, we ablate the choices of incremental token selection and OHEM loss. Table 5 shows the results. CC represents traffic cones and PED represents pedestrians. We focus on CC and PED to verify the effectiveness of our implementation on small objects. Both techniques improve performance. Using OHEM loss and top-k token selection produces the best performance. Without the OHEM loss, we only get  $14.06\mathrm{mIoU}$ . Combining the OHEM loss with a random token selection strategy achieves  $16.62\mathrm{mIoU}$ . Using an uncertain token selection strategy with OHEM loss achieve  $17.37\mathrm{mIoU}$ . For token selection, uncertain selection and top-k selection are on par and they significantly outperform the random selection as expected.
Table 5: Ablation study on our model components, performed on the Occ3D-Waymo dataset.
OHEM Loss	Token Selection Strategy	IoU	mIoU
random	uncertain	top-k	PED	CC
✓	4.16	10.03	14.06
✓	✓	5.07	12.95	16.62
✓	✓	6.27	13.85	17.37
✓	✓	7.04	14.16	18.43
We present Occ3D, a large-scale high-quality 3D occupancy prediction benchmark for visual perception. Meanwhile, we present a rigorous label generation protocol and a new model CTF-Occ network for the 3D occupancy prediction task. They are publicly released to facilitate future research.
Limitations. Although we meticulously design the dataset generation pipeline to significantly enhance its quality, there are several ways to achieve further improvement:
i. Sensor Calibration Error: Since we use LiDAR scans to construct high-quality occupancy labels for camera perception, the calibration between LiDAR and cameras becomes critical. Conducting multi-frame aggregation also relies on precise sensor calibration.
ii. Dynamic and Deformable Objects: For dynamic objects, we extract the points located within the box and aggregate them. However, some dynamic objects may not have box annotations, such as running animals, and some objects may not satisfy the rigid body assumption, like a person swinging their arms. There will be motion blur problems in these cases.
iii. General Objects: Both the nuScenes and Waymo datasets only annotate limited categories. Out-of-vocabulary objects such as trash cans and traffic cones are all regarded as general objects. Further human annotation to provide fine-grained details will help in reproducing an intelligence with unbounded understanding and benefit auto-driving research.
Acknowledgments. This work is supported by the National Key R&D Program of China (2022ZD0161700).
[40] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, pages 746-760. Springer, 2012.
[41] Andrea Simonelli, Samuel Rota Bulo, Lorenzo Porzi, Manuel López-Antequera, and Peter Kontschieder. Disentangling monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1991-1999, 2019.
[42] Liat Sless, Bat El Shlomo, Gilad Cohen, and Shaul Oron. Road scene understanding by occupancy grid learning from sparse radar clusters using semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
[43] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, pages 2446-2454, 2020.
[44] Sebastian Thrun. Probabilistic robotics. Communications of the ACM, 45(3):52-57, 2002.
[45] Ignacio Vizzo, Tiziano Guadagnino, Jens Behley, and Cyrill Stachniss. Vdbfusion: Flexible and efficient tsdf integration of range sensor data. Sensors, 22(3):1296, 2022.
[46] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. arXiv preprint arXiv:2104.10956, 2021.
[47] Xiaofeng Wang, Zheng Zhu, Wenbo Xu, Yunpeng Zhang, Yi Wei, Xu Chi, Yun Ye, Dalong Du, Jiwen Lu, and Xingang Wang. Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception. arXiv preprint arXiv:2303.03991, 2023.
[48] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning, pages 180–191. PMLR, 2022.
[49] Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu. Surroundocc: Multi-camera 3d occupancy prediction for autonomous driving. arXiv preprint arXiv:2303.09551, 2023.
[50] Xu Yan, Jiantao Gao, Jie Li, Ruimao Zhang, Zhen Li, Rui Huang, and Shuguang Cui. Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion. In AAAI, volume 35, pages 3101-3109, 2021.
[51] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10): 3337, 2018.
[52] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3D Object Detection and Tracking. arXiv preprint arXiv:2006.11275, 2020.
[53] Yunpeng Zhang, Zheng Zhu, and Dalong Du. Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction. arXiv preprint arXiv:2304.05316, 2023.
[54] Zaibin Zhang, Lijun Wang, Yifan Wang, and Huchuan Lu. Bev-io: Enhancing bird's-eye-view 3d detection with instance occupancy. arXiv preprint arXiv:2305.16829, 2023.
[55] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4490-4499, 2018.