SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $N_{k}$  represents the number of points within the corresponding segment. The final optimization objective is to minimize the aforementioned semantic spatial consistency loss  $\mathcal{L}^{vfm}$ , temporal consistency loss  $\mathcal{L}^{tmp}$ , and the point-to-segment regularization loss  $\mathcal{L}^{p2s}$ .
Role in Our Framework. Our semantic superpoint temporal consistency resorts to the accurate geometric information from the point cloud and exploits the different views of an instance across different timestamps to learn a temporally consistent representation. Considering the worst-case scenario that the 2D-3D correspondence between the LiDAR and camera sensors becomes unreliable, this geometric constraint can still effectively mitigate the potential errors that occur in inaccurate cross-sensor calibration and synchronization. Besides, our point-to-segment regularization mechanism can serve to aggregate the spatial information thus contributing to the effect of better-distinguishing instances in the LiDAR-acquired scene, e.g., "car" and "truck". As we will show in the following sections, our experimental results are able to verify the effectiveness and superiority of the proposed consistency regularization objectives, even under certain degrees of perturbation in-between sensors.
Table 1: Comparisons of different pretraining methods pretrained on nuScenes [26] and fine-tuned on nuScenes [7], SemanticKITTI [3], Waymo Open [88], and Synth4D [82]. LP denotes linear probing with frozen backbones. Symbol  $\dagger$  denotes fine-tuning with the LaserMix augmentation [55]. Symbol  $\ddagger$  denotes fine-tuning with semi-supervised learning. All mIoU scores are given in percentage (\%).
Method & Year	nuScenes	KITTI1%	Waymo1%	Synth4D1%
LP	1%	5%	10%	25%	Full
Random	8.10	30.30	47.84	56.15	65.48	74.66	39.50	39.41	20.22
PointContrast [ECCV'20] [103]	21.90	32.50	-	-	-	-	41.10	-	-
DepthContrast [ICCV'21] [116]	22.10	31.70	-	-	-	-	41.50	-	-
PPKT [arXiv'21] [65]	35.90	37.80	53.74	60.25	67.14	74.52	44.00	47.60	61.10
SLidR [CVPR'22] [85]	38.80	38.30	52.49	59.84	66.91	74.79	44.60	47.12	63.10
ST-SLidR [CVPR'23] [66]	40.48	40.75	54.69	60.75	67.70	75.14	44.72	44.93	-
Seal (Ours)	44.95	45.84	55.64	62.97	68.41	75.60	46.63	49.34	64.50
\( Seal^† \) (Ours)	-	48.41	57.84	65.52	70.80	77.13	-	-	-
\( Seal^‡ \) (Ours)	-	49.53	58.64	66.78	72.31	78.28	-	-	-
Data. We verify the effectiveness of our approach on eleven different point cloud datasets.  $^{1}nu$ Scenes [7, 26],  $^{2}SemanticKITTI$  [3], and  $^{3}Waymo$  Open [88] contain large-scale LiDAR scans collected from real-world driving scenes; while the former adopted a Velodyne HDL32E, data from the latter two datasets are acquired by 64-beam LiDAR sensors.  $^{4}ScribbleKITTI$  [95] shares the data with [3] but are weakly annotated with line scribbles.  $^{5}RELLIS-3D$  [49] is a multimodal dataset collected in an off-road campus environment.  $^{6}SemanticPOSS$  [75] is a small-scale set with an emphasis on dynamic instances.  $^{7}SemanticSTF$  [102] consist of LiDAR scans from adverse weather conditions.  $^{8}SynLiDAR$  [100],  $^{9}Synth4D$  [82], and  $^{10}DAPS-3D$  [51] are synthetic datasets collected from simulators. We also conduct extensive robustness evaluations on the  $^{11}nu$ Scenes-C dataset proposed in the Robo3D benchmark [53], a comprehensive collection of eight out-of-distribution corruptions that occur in driving scenarios. More details on these datasets are in the Appendix.
Implementation Details. We use MinkUNet [19] as the 3D backbone which takes cylindrical voxels of size  $0.10\mathrm{m}$  as inputs. Similar to [85, 66], our 2D backbone is a ResNet-50 [33] pretrained with MoCoV2 [12]. We pretrain our segmentation network for 50 epochs on two GPUs with a batch size of 32, using SGD with momentum and a cosine annealing scheduler. For fine-tuning, we follow the exact same data split, augmentation, and evaluation protocol as SLidR [85] on nuScenes and SemanticKITTI, and adopt a similar procedure on other datasets. The training objective is to minimize a combination of the cross-entropy loss and the Lovász-Softmax loss [4]. We compare the results of prior arts from their official reporting [85, 66]. Since PPKT [65] and SLidR [85] only conducted experiments on nuScenes and SemanticKITTI, we reproduce their best-possible performance on the other nine datasets using public code. For additional details, please refer to the Appendix.
Metrics. We follow the conventional reporting of Intersection-over-Union (IoU) on each semantic class and the mean IoU (mIoU) across all classes. For robustness probing, we follow the Robo3D protocol [53] and report the mean Corruption Error (mCE) and mean Resilience Rate (mRR) scores calculated by using the MinkUNet<sub>18</sub> (torchsparse) implemented by Tang et al. [89] as the baseline.
4.2 Comparative Study
Comparison to State-of-the-Arts. We compare Seal with random initialization and five existing pretraining approaches under both linear probing (LP) protocol and few-shot fine-tuning settings on nuScenes [26] in Table 1. We observe that the pretraining strategy can effectively improve the accuracy of downstream tasks, especially when the fine-tuning budget is very limited (e.g.  $1\%$ ,  $5\%$ , and  $10\%$ ). Our framework achieves  $44.95\%$  mIoU under the LP setup – a  $4.47\%$  mIoU lead over to the prior art ST-SLidR [66] and a  $6.15\%$  mIoU boost compared to the baseline SLidR [85]. What is more, Seal achieves the best scores so far across all downstream fine-tuning tasks, which verifies the superiority of VFM-assisted contrastive learning and spatial-temporal consistency regularization. We also show that recent out-of-context augmentation [55] could enhance the feature learning during fine-tuning, which establishes a new state of the art on the challenging nuScenes benchmark.
Table 2: Comparisons of different pretraining methods pretrained on nuScenes [26] and fine-tuned on different downstream point cloud datasets. All mIoU scores are given in percentage (\%).
Method	ScribbleKITTI	RELLIS-3D	SemanticPOSS	SemanticSTF	SynLiDAR	DAPS-3D
1%	10%	1%	10%	Half	Full	Half	Full	1%	10%	Half	Full
Random	23.81	47.60	38.46	53.60	46.26	54.12	48.03	48.15	19.89	44.74	74.32	79.38
PPKT [65]	36.50	51.67	49.71	54.33	50.18	56.00	50.92	54.69	37.57	46.48	78.90	84.00
SLidR [85]	39.60	50.45	49.75	54.57	51.56	55.36	52.01	54.35	42.05	47.84	81.00	85.40
Seal (Ours)	40.64	52.77	51.09	55.03	53.26	56.89	53.46	55.36	43.58	49.26	81.88	85.90
Table 3: Robustness evaluations under eight out-of-distribution corruptions in the nuScenes-C dataset from the Robo3D benchmark [53]. All mCE, mRR, and mIoU scores are given in percentage  $(\%)$ .
Initial	Backbone	mCE↓	mRR↑	Fog	Wet	Snow	Move	Beam	Cross	Echo	Sensor
LP	PPKT [65]	MinkUNet	183.44	78.15	30.65	35.42	28.12	29.21	32.82	19.52	28.01	20.71
SLidR [85]	MinkUNet	179.38	77.18	34.88	38.09	32.64	26.44	33.73	20.81	31.54	21.44
Seal (Ours)	MinkUNet	166.18	75.38	37.33	42.77	29.93	37.73	40.32	20.31	37.73	24.94
Full	Random	PolarNet	115.09	76.34	58.23	69.91	64.82	44.60	61.91	40.77	53.64	42.01
Random	CENet	112.79	76.04	67.01	69.87	61.64	58.31	49.97	60.89	53.31	24.78
Random	WaffleIron	106.73	72.78	56.07	73.93	49.59	59.46	65.19	33.12	61.51	44.01
Random	Cylinder3D	105.56	78.08	61.42	71.02	58.40	56.02	64.15	45.36	59.97	43.03
Random	SPVCNN	106.65	74.70	59.01	72.46	41.08	58.36	65.36	36.83	62.29	49.21
Random	MinkUNet	112.20	72.57	62.96	70.65	55.48	51.71	62.01	31.56	59.64	39.41
PPKT [65]	MinkUNet	105.64	76.06	64.01	72.18	59.08	57.17	63.88	36.34	60.59	39.57
SLidR [85]	MinkUNet	106.08	75.99	65.41	72.31	56.01	56.07	62.87	41.94	61.16	38.90
Seal (Ours)	MinkUNet	92.63	83.08	72.66	74.31	66.22	66.14	65.96	57.44	59.87	39.85
Downstream Generalization. To further verify the capability of Seal on segmenting any automotive point clouds, we conduct extensive experiments on eleven different datasets and show the results in Table 1 and Table 2. Note that each of these datasets has a unique data collection protocol and a diverse data distribution - ranging from diverse sensor types and data acquisition environments to scales and fidelity - making this evaluation a comprehensive one. The results show that our framework constantly outperforms prior arts across all downstream tasks on all eleven datasets, which concretely supports the effectiveness and superiority of our proposed approach.
Semi-Supervised Learning. Recent research [55, 59] reveals that combining both labeled and unlabeled data for semi-supervised point cloud segmentation can significantly boost the performance on downstream tasks. We follow these works to implement such a learning paradigm where a momentum-updated teacher model is adopted to assign pseudo-labels for the unlabeled data. The results from the last row of Table 1 show that Seal is capable of providing reliable supervision signals for data-efficient learning. Notably, our framework with partial annotation is able to surpass some recent fully-supervised learning methods. More results on this aspect are included in the Appendix.
Robustness Probing. It is often of great importance to assess the quality of learned representations on out-of-distribution data, especially for cases that occur in the real-world environment. We resort to the recently established nuScenes-C in the Robo3D benchmark [53] for such robustness evaluations. From Table 3 we observe that the self-supervised learning methods [65, 85] in general achieve better robustness than their baseline [19]. Seal achieves the best robustness under almost all corruption types and exhibits superiority over other recent segmentation backbones with different LiDAR representations, such as range view [16], BEV [115], raw points [78], and multi-view fusion [89].
Qualitative Assessment. We visualize the predictions of each pertaining method pretrained and fine-tuned on nuScenes [26] in Fig. 5. A clear observation is the substantial enhancement offered by all pretraining methods when juxtaposed with a baseline of random initialization. Diving deeper into the comparison among the trio of tested techniques, Seal stands out, delivering superlative segmentation outcomes, especially in the intricate terrains of driving scenarios. This is credited to the strong spatial and temporal consistency learning that Seal tailored to excite during the pretraining. We have cataloged additional examples in the Appendix for more detailed visual comparisons.
Foundation Model Comparisons. We provide the first study on adapting VFMs for large-scale point cloud representation learning and show the results in Table 4. We observe that different VFMs exhibit
Table 4: Ablation study on pretraining frameworks (ours vs. SLidR [85]) and the knowledge transfer effects from different vision foundation models. All mIoU scores are given in percentage (\%).
Method	Superpixel	nuScenes	KITTI1%	Waymo1%	Synth4D1%
LP	1%	5%	10%	25%	Full
Random	-	8.10	30.30	47.84	56.15	65.48	74.66	39.50	39.41	20.22
SLidR	SLIC [1]	38.80	38.30	52.49	59.84	66.91	74.79	44.60	47.12	63.10
SAM [50]	41.49	43.67	55.97	61.74	68.85	75.40	43.35	48.64	63.15
X-Decoder [122]	41.71	43.02	54.24	61.32	67.35	75.11	45.70	48.73	63.21
OpenSeeD [111]	42.61	43.82	54.17	61.03	67.30	74.85	45.88	48.64	63.31
SEEM [123]	43.00	44.02	53.03	60.84	67.38	75.21	45.72	48.75	63.13
Seal	SLIC [1]	40.89	39.77	53.33	61.58	67.78	75.32	45.75	47.74	63.37
SAM [50]	43.94	45.09	56.95	62.35	69.08	75.92	46.53	49.00	63.76
X-Decoder [122]	42.64	44.31	55.18	62.03	68.24	75.56	46.02	49.11	64.21
OpenSeeD [111]	44.67	44.74	55.13	62.36	69.00	75.64	46.13	48.98	64.29
SEEM [123]	44.95	45.84	55.64	62.97	68.41	75.60	46.63	49.34	64.50
Table 5: Ablation study of each component pretrained on nuScenes [26] and fine-tuned on nuScenes [26], SemanticKITTI [3], and Waymo Open [88]. C2L: Camera-to-LiDAR distillation. VFM: Vision foundation models. STC: Superpoint temporal consistency. P2S: Point-to-segment regularization.
#	C2L	VFM	STC	P2S	nuScenes	KITTI1%	Waymo1%
LP	1%	5%	10%	25%	Full
(1)	✓	38.80	38.30	52.49	59.84	66.91	74.79	44.60	47.12
(2)	✓	✓	40.45	41.62	54.67	60.48	67.61	75.30	45.38	48.08
(3)	✓	✓	43.00	44.02	53.03	60.84	67.38	75.21	45.72	48.75
(4)	✓	✓	✓	44.01	44.78	55.36	61.99	67.70	75.00	46.49	49.15
(5)	✓	✓	✓	43.35	44.25	53.69	61.11	67.42	75.44	46.07	48.82
(6)	✓	✓	✓	✓	44.95	45.84	55.64	62.97	68.41	75.60	46.63	49.34
diverse abilities in encouraging contrastive objectives. All VFMs show larger gains than SLIC [1] with both frameworks, while SEEM [123] in general performs the best. Notably, SAM [50] tends to generate more fine-grained superpixels and yields better results when fine-tuning with more annotated data. We conjecture that SAM [50] generally provides more negative samples than the other three VFMs which might be conducive to the superpixel-driven contrastive learning. On all setups, Seal constantly surpasses SLidR [85] by large margins, which verifies the effectiveness of our framework.
Cosine Similarity. We visualize three examples of feature similarity across different VFMs in Fig. 4. We observe that our contrastive objective has already facilitated distinction representations before fine-tuning. The semantically-rich VFMs such as X-Decoder [122], OpenSeeD [111], and SEEM [123] offers overt feature cues for recognizing objects and backgrounds; while the unsupervised (SLIC [1]) or too fine-grained (SAM [50]) region partition methods only provide limited semantic awareness. Such behaviors have been reflected in the linear probing and downstream fine-tuning performance (see Table 4), where SEEM [123] tends to offer better consistency regularization impacts during the cross-sensor representation learning.
Component Analysis. Table 5 shows the ablation results of each component in the Seal framework. Specifically, direct integration of VFMs (row #3) or temporal consistency learning (row #2) brings  $4.20\%$  and  $1.65\%$  mIoU gains in LP, respectively, while a combination of them (row #4) leads to a  $5.21\%$  mIoU gain. The point-to-segment regularization (row #5) alone also provides a considerable performance boost of around  $4.55\%$  mIoU. Finally, an integration of all proposed components (row #6) yields our best-performing model, which is  $6.15\%$  mIoU better than the prior art [85] in LP and also outperforms on all in-distribution and -out-of-distribution downstream setups.
Sensor Misalignment. An accurate calibration is crucial for establishing correct correspondences between LiDAR and cameras. In most cases, those sensors should be well-calibrated for an autonomous car. It is unusual if the calibration is completely unknown, but it is possible to be imprecise due to a lack of maintenance. Hence, we conduct the following experiments to validate the robustness of our method. For each point coordinate  $\mathbf{p}_i = (x_i,y_i,z_i)$  in a LiDAR point cloud, its corresponding pixel  $\hat{\mathbf{p}}_i = (u_i,v_i)$  in the camera view can be found via Eq. 1. To simulate the misalignment between LiDAR and cameras, we insert random noises into the camera extrinsic matrix  $\Gamma_K$ , with a relative proportion of  $1\%$ ,  $5\%$ , and  $10\%$ . Table 6 shows the results of PPKT [65], SLidR [85], and Seal under such noise perturbations. We observe that the possible calibration errors between LiDAR
Figure 5: The qualitative results of different point cloud pretraining approaches pretrained on the raw data of nuScenes [26] and fine-tuned with  $1\%$  labeled data. To highlight the differences, the correct / incorrect predictions are painted in gray / red, respectively. Best viewed in color.
Table 6: Ablation study on the possible misalignment between the LiDAR and camera sensors. The perturbation is randomly generated and inserted. All mIoU scores are given in percentage  $(\%)$ .
Method	1% Misalignment	5% Misalignment	10% Misalignment
1%	5%	10%	25%	1%	5%	10%	25%	1%	5%	10%	25%
PPKT [65]	34.94	51.11	58.54	65.01	33.69	51.40	58.00	64.11	33.35	50.98	57.84	63.52
SLidR [85]	37.92	53.08	59.89	66.90	38.00	52.36	60.01	64.10	37.30	51.11	58.50	64.50
Seal (Ours)	45.23	55.71	62.62	68.13	45.66	55.42	62.77	68.01	44.80	54.45	61.80	68.29
and cameras will cause performance degradation for different pretraining approaches (compared to Table 1). The performance degradation for PPKT [65] is especially prominent; we conjecture that this is because the point-wise consistency regularization of PPKT [65] relies heavily on the calibration accuracy and encounters problems under misalignment. Both SLidR [85] and Seal exhibit certain robustness; we believe the superpixel-level consistency is less sensitive to calibration perturbations. It is worth mentioning that Seal can maintain good performance under calibration error, since: i) Our VFM-assisted representation learning tends to be more robust; and ii) We enforce superpoint temporal consistency during the pertaining which does not rely on the 2D-3D correspondence.
5 Concluding Remark
In this study, we presented Seal, a versatile self-supervised learning framework capable of segmenting any automotive point clouds by encouraging spatial and temporal consistency during the representation learning stage. Additionally, our work pioneers the utilization of VFMs to enhance 3D scene understanding. Extensive experimental results on 20 downstream tasks across eleven different point cloud datasets verified the effectiveness and superiority of our framework. We aspire for this research to catalyze further integration of large-scale 2D and 3D representation learning endeavors, which could shed light on the development of robust and annotation-efficient perception models.
Potential Limitations. Although our proposed framework holistically improved the point cloud segmentation performance across a wide range of downstream tasks, there are still some limitations that could hinder the scalability. i) Our model operates under the assumption of impeccably calibrated and synchronized LiDAR and cameras, which might not always hold true in real-world scenarios. ii) We only pretrain the networks on a single set of point clouds with unified setups; while aggregating more abundant data from different datasets for pretraining would further improve the generalizability of this framework. These limitations present intriguing avenues for future investigations.
Acknowledgements. This study is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOE-T2EP20221-0012), NTU NAP, and under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). This study is also supported by the National Key R&D Program of China (No. 2022ZD0161600). We thank Tai Wang for the insightful review and discussion.
[113] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 1
[114] Tunhou Zhang, Mingyuan Ma, Feng Yan, Hai Li, and Yiran Chen. Pids: Joint point interaction-dimension search for 3d point cloud. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1298-1307, 2023. 3
[115] Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, and Hassan Foroosh. Polarnet: An improved grid representation for online lidar point clouds semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9601-9610, 2020. 3, 8
[116] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. In IEEE/CVF International Conference on Computer Vision, pages 10252-10263, 2021. 3, 7
[117] Junhao Zhao, Weijie Huang, Hai Wu, Chenglu Wen, Bo Yang, Yulan Guo, and Cheng Wang. Semanticflow: Semantic segmentation of sequential lidar point clouds from sparse frame annotations. IEEE Transactions on Geoscience and Remote Sensing, 61:1-11, 2023. 3
[118] Yiming Zhao, Lin Bai, and Xinming Huang. Fidnet: Lidar point cloud semantic segmentation with fully interpolation decoding. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4453-4458, 2021. 3
[119] Zixiang Zhou, Yang Zhang, and Hassan Foroosh. Panoptic-polarnet: Proposal-free lidar point cloud panoptic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13194-13203, 2021. 3
[120] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9939-9948, 2021. 3
[121] Zhuangwei Zhuang, Rong Li, Kui Jia, Qicheng Wang, Yuanqing Li, and Mingkui Tan. Perception-aware multi-sensor fusion for 3d lidar semantic segmentation. In IEEE/CVF International Conference on Computer Vision, pages 16280-16290, 2021. 3
[122] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, Nanyun Peng, Lijuan Wang, Yong Jae Lee, and Jianfeng Gao. Generalized decoding for pixel, image, and language. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 1, 2, 3, 6, 9, 21, 22, 23, 31, 32, 33
[123] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718, 2023. 1, 2, 3, 6, 9, 21, 22, 23, 24, 25, 31, 32, 33
In this appendix, we supplement the following materials to support the findings and observations in the main body of this paper:
- Section A elaborates on additional implementation details to facilitate reproduction.
- Section B provides the complete quantitative results of our experiments.
- Section C includes more qualitative results to allow better visual comparisons.
- Section D acknowledges the public resources used during the course of this work.
A Additional Implementation Detail
In this work, we conduct extensive experiments from a wide range of point cloud datasets to verify the effectiveness of the proposed Seal framework. A summary of the detailed configurations and emphases of these datasets is shown in Table A.