SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To evaluate the superior performance and generalization capability of our method for scene understanding, we conduct experiments on both indoor and outdoor public datasets, namely ScanNet [91], nuScenes [92] and nuImages [93], and compare with current state-of-the-art approaches [20, 56, 17] on both 2D and 3D semantic segmentation tasks. Critical operations and losses of our method are also verified effectiveness by extensive ablation studies. In the following, we introduce datasets, implementation, comparison results, and ablation studies in detail.
Datasets. ScanNet [91] consists of 1,603 indoor scans, collected by RGB-D camera, with 20 classes, where 1,201 scans are allocated for training, 312 scans for validation, and 100 scans for testing. Additionally, we utilize 25,000 key frame images to train the 2D network. The nuScenes [92] dataset,
Table 1: Comparison (mIoU) with current state-of-the-art label-free methods for semantic segmentation tasks on the ScanNet [91], nuImages [93] and nuScenes [94] dataset.
Methods	Publication	ScanNet	nuImages	nuScenes
2D	3D	2D	3D
MaskCLIP [20]	ECCV 2022	17.3	14.2	14.1	12.8
MaskCLIP+ [20]	ECCV 2022	20.3	21.6	17.3	15.3
OpenScene [56]	CVPR 2023	14.2	16.8	12.4	14.6
CLIP2Scene [17]	CVPR 2023	23.7	25.6	18.6	20.8
Ours	-	28.4	33.5	22.1	26.8
Table 2: Ablation study on the ScanNet [94] val set for label-free 2D and 3D semantic segmentation.
Ablation Target	Setting	2D mIoU (%)	3D mIoU (%)
Cross-modality Noisy Supervision	Baseline	17.3	14.2
w/o CNS	24.5 (+7.20)	19.4 (+5.20)
Base+CLIP	20.4 (+3.10)	23.8 (+9.60)
Base+CLIP+LSC	22.5 (+5.20)	28.7 (+14.5)
Label Refinement	w/o Refine	23.7 (+6.40)	30.6 (+16.4)
Post-Refine	28.1 (+10.8)	33.2 (+19.0)
Prediction Consistency Regularization	w/o CT	27.7 (+10.1)	31.0 (+16.8)
w/o SCT	24.7 (+7.40)	30.6 (+16.4)
w/o CLIP	4.7 (-12.6)	5.1 (-9.1)
w 2D	6.9 (-10.4)	8.8 (-5.4)
w 3D	4.8 (-12.5)	6.4 (-7.8)
Latent Space Consistency Regularization	CLIP	27.3 (+10.0)	32.4 (+18.2)
SAM+CLIP	27.8 (+10.5)	32.5 (+18.3)
None	27.1 (+9.80)	32.0 (+17.8)
Image Numbers	8	27.9 (+10.6)	33.0 (+18.8)
16	28.4 (+11.1)	33.5 (+19.3)
32	28.3 (+11.0)	33.3 (+19.1)
Full Configuration	Ours	28.4 (+11.1)	33.5 (+19.3)
collected in traffic scenarios by LiDAR and RGB camera, comprises 700 scenes for training, 150 for validation, and 150 for testing, focusing on LiDAR semantic segmentation with 16 classes. To be more specific, we leverage a total of 24,109 sweeps of LiDAR scans for training and 4,021 sweeps for validation. Each sweep is accompanied by six camera images, providing a comprehensive 360-degree view. Note that the dataset does not provide image labels. Thus the quantitative evaluation of the paired images is not presented. The nuImages [93] dataset provides 93,000 2D annotated images sourced from a significantly larger dataset. This includes 67,279 images for training, 16,445 for validation, and 9,752 for testing. The dataset consists of 11 shared classes with nuScenes.
Implementation Details. We utilize MinkowskiNet34 [95] as the 3D backbone and DeeplabV3 [96] as the 2D backbone in our approach, where DeeplabV3 model is pre-trained on the ImageNet dataset. Based on the MaskCLIP framework, we modify the attention pooling layer of CLIP's image encoder to extract dense pixel-text correspondences. Our framework is developed using PyTorch and trained on two NVIDIA Tesla A100 GPUs. During training, both CLIP and SAM are kept frozen. For prediction consistency regularization, we transition to stage two after ten epochs of stage one. To enhance the robustness of our model, we apply various data augmentations, such as random rotation along the z-axis and random flip for point clouds, as well as random horizontal flip, random crop, and random resize for images. For the ScanNet dataset, the training process takes approximately 10 hours for 30 epochs, with the image number set to 16. In the case of the nuScenes dataset, the training time is 40 hours for 20 epochs, with the image number set to 6.
Figure 3: Qualitative results of label-free semantic segmentation on the ScanNet 25k images dataset. From the left to the right column are the input, prediction by SAM, CLIP, ours w/o CNS, our full method, and ground truth, respectively.
4.1 Comparison Results
Our method was compared with several state-of-the-art methods, namely MaskCLIP, MaskCLIP+, OpenScene, and CLIP2Scene, on the ScanNet and nuScenes datasets. Note that their codes are publicly available. For a fair comparison, we share all experiment settings with the above methods, including data augmentations, training epochs, learning rate, and network backbones. The results, as depicted in Table 1, demonstrate that our method surpasses the performance of other approaches by a significant margin. In terms of label-free semantic segmentation, our 2D and 3D networks achieve remarkable results, achieving  $28.4\%$  and  $33.5\%$  mIoU on the ScanNet dataset, corresponding to improvements of  $4.7\%$  and  $7.9\%$ , respectively. Furthermore, on the nuScenes dataset, our method achieves a mIoU of  $26.8\%$  for 3D semantic segmentation, exhibiting a  $6\%$  improvement. Note that the nuScenes dataset does not provide image labels. Thus the quantitative evaluation of 2D semantic segmentation is not presented.
We conduct a series of ablation experiments to evaluate and demonstrate the effectiveness of different modules within our framework, as shown in Table. 2. In this section, we present the ablation configuration and provide an in-depth analysis of our experimental results.
Effect of Cross-modality Noisy Supervision. In 2D scene understanding, the Baseline is CLIP. For 3D scene understanding, we directly project the 2D prediction to the 3D point cloud. We also conducted w/o CNS that the Cross-modality Noisy Supervision was not applied, i.e., build upon Baseline, the 2D label is refined by SAM, and projected to 3D points. Besides, we impose the baseline method self-training with the CLIP pseudo-labels, termed base+CLIP. Building upon base+CLIP, we also impose Latent Space Consistency Regularization, termed base+CLIP+ LSCR. Experiments
Figure 4: Qualitative results of label-free point cloud-based 3D semantic segmentation on the ScanNet dataset. From the left to the right are the input, prediction by CLIP, ours w/o CNS, our full method, and ground truth, respectively.
show that SAM refinement promotes pseudo-label quality. Moreover, our Cross-modality Noisy Supervision can effectively supervise networks under extremely noisy pseudo labels. We also evaluate CNS qualitatively using the ScanNet dataset, as illustrated in Fig. 3 and 4. Our comprehensive approach demonstrates remarkable performance in label-free scene understanding, encompassing both 2D and 3D scenarios. We observed that our results are comparable to human annotations in numerous instances. Interestingly, the 2D and 3D networks exhibit the remarkable ability to "segment anything," which can be attributed to the knowledge transferred from SAM.
Effect of Label Refinement. To verify how the label refinement affects the performance, we do not use SAM to refine the pseudo-labels during training (w/o Refine); we perform prediction refinement in the final result (Post-Refine). Notably, we observed a significant drop in accuracy when SAM refinement was omitted during training. This suggests that SAM is crucial in enhancing the model's performance. However, we also noticed that post-refinement did not contribute to the performance. This can be attributed to the fact that the trained networks already produced semantic-consistent regions similar to those obtained with SAM refinement.
Effect of Latent Space Consistency Regularization. We utilize two main features to evaluate the latent space consistency regularization: the CLIP feature and the  $\mathbf{CLIP} + \mathbf{SAM}$  feature. In addition, we have experimented without any Latent Space Regularization (None). Results demonstrate that our full method incorporating the SAM feature performs best. On the other hand, we have observed that the CLIP feature alone is less effective in our task, primarily because the feature space it provides is not specifically designed for segmentation purposes.
Effect of Prediction Consistency Regularization. We conducted five configurations to examine the effects of prediction consistency regularization: 1) w/o CT: In this experiment, the 2D and 3D networks were not allowed to cross-train each other in the second training stage, i.e., the 2D network's labels were randomly switched between CLIP and 2D prediction. Meanwhile, the 3D network's labels were randomly switched between CLIP and 3D prediction; 2) w/o SCT: This experiment eliminates self- and cross-training in 2D and 3D networks, i.e., 2D and 3D network labels are derived solely from CLIP; 3) w/o CLIP: This experiment excludes the usage of CLIP's labels during the second training stage, i.e., both the 2D and 3D network's labels were randomly switched between 2D and 3D prediction; And 4) w 2D, 5) w 3D: we employ either 2D or 3D network prediction to
synchrony train 2D and 3D networks, respectively. Experiments show that SAM refinement promotes pseudo-label quality. Besides, The experimental results indicate that cross-training, self-training, and label-switching effectively improve performance. Importantly, we observed that the method fails when CLIP's supervision is absent during the second training stage. This failure can be attributed to the tendency of the 2D and 3D networks to "co-corrupt" each other with extremely noisy labels.
Effect of Images Numbers. We also show the performance when restricting image numbers to 8, 16, and 32, respectively. However, we observe that the performances are comparable. Therefore, for efficient training, we set the image number to 16.
In this work, we study how vision foundation models enable networks to comprehend 2D and 3D environments without relying on labelled data. To address noisy pseudo-labels, we introduce a cross-modality noisy supervision method to simultaneously train 2D and 3D networks, which involves the label refinement strategy, prediction consistency regularization, and latent space consistency regularization. Our method achieves state-of-the-art performance in understanding open worlds, as demonstrated by extensive experiments on indoor and outdoor datasets.
Yuexin Ma was partially supported by NSFC (No.62206173), MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech University), and Shanghai Frontiers Science Center of Human-centered Artificial Intelligence (ShangHAI). Tongliang Liu was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031.
[1] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9939-9948, 2020.
[2] Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition-based pooling. In Advances in Neural Information Processing Systems, pages 33330-33342, 2022.
[3] Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen. Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12475-12485, 2020.
[4] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 652-660, 2017.
[5] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9939-9948, 2021.
[6] Yuenan Hou, Xinge Zhu, Yuexin Ma, Chen Change Loy, and Yikang Li. Point-to-voxel knowledge distillation for lidar semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8479–8488, 2022.
[7] Jianyun Xu, Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun, and Shiliang Pu. Rpvnet: A deep and efficient range-point-voxel fusion network for lidar point cloud segmentation. In IEEE/CVF International Conference on Computer Vision, pages 16024-16033, 2021.
[8] Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu. (af)2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. In IEEE Conference on Computer Vision and Pattern Recognition, pages 12547-12556, 2021.
[9] Lingdong Kong, Youquan Liu, Runnan Chen, Yuexin Ma, Xinge Zhu, Yikang Li, Yuenan Hou, Yu Qiao, and Ziwei Liu. Rethinking range view representation for lidar segmentation. In IEEE/CVF International Conference on Computer Vision, pages 228-240, 2023.
[86] Yingbin Bai, Erkun Yang, Zhaoqing Wang, Yuxuan Du, Bo Han, Cheng Deng, Dadong Wang, and Tongliang Liu. Msr: Making self-supervised learning robust to aggressive augmentations. arXiv preprint arXiv:2206.01999, 2022.
[87] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
[88] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[89] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213-229, 2020.
[90] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
[91] Angela Dai, Angel Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5828-5839, 2017.
[92] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11621-11631, 2020.
[93] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11621-11631, 2020.
[94] Whye Kit Fong, Rohit Mohan, Juana Valeria Hurtado, Lubing Zhou, Holger Caesar, Oscar Beijbom, and Abhinav Valada. Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking. IEEE Robotics and Automation Letters, 7:3795-3802, 2022.
[95] Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3075-3084, 2019.
[96] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.