SlowGuess/NeurIPS_2023_OCR huggingface dataset:
-  ${}^{1}$  nuScenes [26]: The nuScenes  ${}^{4}$  dataset offers a substantial number of samples collected by the LiDAR, RADAR, camera, and IMU sensors from Boston and Singapore, allowing machine learning models to learn useful multi-modal features effectively. For the point cloud segmentation task, it consists of 1000 scenes of a total number of 1.1B annotated points, collected by a Velodyne HDL32E LiDAR sensor. It also includes image data from six cameras, which are synchronized with the LiDAR sensor. In this work, we use the LiDAR point clouds and image data from nuScenes for model pretraining. We also conduct detailed fine-tuning experiments to validate the effectiveness of representation learning. More details of this dataset can be found at https://www.nuscenes.org/nuscenes.
-  $^{2}$ SemanticKITTI [3]: The SemanticKITTI dataset is a comprehensive dataset designed for semantic scene understanding of LiDAR sequences. This dataset aims to advance the development of algorithms and models for autonomous driving and scene understanding using LiDAR data. It provides 22 densely labeled point cloud sequences that cover urban street scenes, which are captured by a Velodyne HDL-64E LiDAR sensor. In this work, we use the LiDAR point clouds from SemanticKITTI as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at http://semantic-kitti.org.
-  $^3$ Waymo Open [88]: The Waymo Open dataset is a large-scale collection of real-world autonomous driving data. The 3D semantic segmentation subset contains 1150 scenes, split into 798 training, 202 validation, and 150 testing scenes. This subset contains 23691 training scans, 5976 validation scans, and 2982 testing scans, respectively, with semantic segmentation labels from 23 classes. The data are captured by five LiDAR sensors: one mid-range LiDAR sensor truncated to a maximum of 75 meters, and four short-range LiDAR sensors truncated to a maximum of 20 meters. In this work, we use the LiDAR point clouds from Waymo Open as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://waymo.com/open.
-  $^{4}$ ScribbleKITTI [95]: The ScribbleKITTI dataset is a recent variant of the SemanticKITTI dataset, with weak supervisions annotated by line scribbles. It shares the exact same amount of training samples with SemanticKITTI, i.e., 19130 scans collected by a Velodyne HDL-64E LiDAR sensor, where the total number of valid semantic labels is  $8.06\%$  compared to the fully-supervised version. Annotating the LiDAR point cloud in such a way corresponds to roughly a  $90\%$  time-saving. In this work, we use the LiDAR point clouds from ScribbleKITTI as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://github.com/ouenal/scribblekitti.
<sup>4</sup>Here we refer to the lidarseg subset of the nuScenes dataset. Know more details about this dataset at the official webpage: https://www.nuscenes.org/nuscenes.
Table A: The sensor configuration and data statistics for the eleven datasets used in our experiments.
Dataset	Illustration	Sensor Setup	Statistics	Type
nuScenes [26]	1× LiDAR (32-beam) 6× RGB Camera 5× RADAR 1× IMU & GPS	16 semantic classes 29130 training samples 6019 validation samples 6008 testing samples	Real-world Low-resolution point cloud Dense annotation Multi-modality
SemanticKITTI [3]	1× LiDAR (64-beam) 1× Stereo Camera 1× IMU & GPS	19 semantic classes 19130 training samples 4071 validation samples 20351 testing samples	Real-world High-resolution point cloud Dense annotation Multi-modality
Waymo Open [88]	1× LiDAR (64-beam) 5× RGB Camera 1× IMU & GPS	23 semantic classes 23691 training samples 5976 validation samples 2982 testing samples	Real-world High-resolution point cloud Dense annotation Multi-modality
ScribbleKITTI [95]	1× LiDAR (64-beam) 1× Stereo Camera 1× IMU & GPS	19 semantic classes 19130 training samples 4071 validation samples 20351 testing samples	Real-world High-resolution point cloud Sparse annotation Weakly-supervised learning
RELLIS-3D [49]	1× LiDAR (64-beam) 1× LiDAR (32-beam) 1× Stereo Camera 1× IMU & GPS	20 semantic classes 7800 training samples 2413 validation samples 3343 testing samples	Real-world High-resolution point cloud Dense annotation Multi-modality
SemanticPOSS [75]	1× LiDAR (40-beam) 1× RGB Camera 1× IMU & GPS	14 semantic classes 2488 training samples 500 validation samples	Real-world High-resolution point cloud Dense annotation Dynamic instance
SemanticSTF [102]	1× LiDAR (64-beam) 1× LiDAR (32-beam) 1× Stereo Camera 1× RADAR	21 semantic classes 1326 training samples 250 validation samples 500 testing samples	Real-world High-resolution point cloud Dense annotation Adverse weather
SynLiDAR [100]	1× LiDAR (64-beam) 1× simulation suite	32 semantic classes 198396 total samples	Synthetic High-resolution point cloud Dense annotation Transfer learning
Synth4D [82]	1× LiDAR (64-beam) 1× LiDAR (32-beam) 1× simulation suite	22 semantic classes 10000 training samples 10000 validation samples	Synthetic Low-resolution point cloud Dense annotation Transfer learning
DAPS-3D [51]	3× LiDAR (64-beam) 1× simulation suite	4 semantic classes 19061 training samples 3995 validation samples	Semi-synthetic High-resolution point cloud Dense annotation Transfer learning
nuScenes-C [53]	1× LiDAR (32-beam) 6× RGB Camera 5× RADAR 1× IMU & GPS	16 semantic classes 144456 validation samples	Synthetic Low-resolution point cloud Dense annotation Robustness
-  $5^{\text{RELLIS-3D}}$  [49]: The RELLIS-3D dataset is a multimodal dataset collected in an off-road environment from the Rellis Campus of Texas A&M University. It consists of 13556 LiDAR scans from 5 traversal sequences. The point-wise annotations are initialized by using the camera-LiDAR calibration to project the more than 6000 image annotations onto the point clouds. In this work, we use the LiDAR point clouds from RELLIS-3D as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at http://www.unmannedlab.org/research/RELLIS-3D.
- 6SemanticPOSS [75]: The SemanticPOSS dataset is a relatively small-scale point cloud dataset with an emphasis on dynamic instances. It includes 2988 scans collected by a Hesai Pandora LiDAR sensor, which is a 40-channel LiDAR with 0.33 degree vertical resolution, a forward-facing color camera, 4 wide-angle mono cameras covering 360 degrees around the ego-car. The data in this dataset was collected from the campus of Peking University. In this work, we use the LiDAR point clouds from SemanticPOSS as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://wwwposs.pku.edu.cn/semanticposss.
- 7SemanticSTF [102]: The SemanticSTF dataset is a small-scale collection of 2076 scans, where the data are borrowed from the STF dataset [5]. The scans are collected by a Velodyne HDL64 S3D LiDAR sensor and covered various adverse weather conditions, including 694 snowy, 637 dense-foggy, 631 light-foggy, and 114 rainy scans. The whole dataset is split into three sets: 1326 scans for training, 250 scans for validation, and 500 scans for testing. All three splits have similar proportions of scans from different weather conditions. In this work, we use the LiDAR point clouds from SemanticSTF as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://github.com/xiaooraan/SemanticSTF.
-  $^{8}$ SynLiDAR [100]: The SynLiDAR dataset contains synthetic point clouds captured from constructed virtual scenes using the Unreal Engine 4 simulator. In total, this dataset contains 13 LiDAR point cloud sequences with 198396 scans. As stated, the virtual scenes in SynLiDAR are constituted by physically accurate object models that are produced by expert modelers with the 3D-Max software. In this work, we use the LiDAR point clouds from SynLiDAR as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://github.com/xiaoaoran/SynLiDAR.
-  $^9$ Synth4D [82]: The Synth4D dataset includes two subsets with point clouds captured by simulated Velodyne LiDAR sensors using the CARLA simulator. We use the Synth4D-nuScenes subset in our experiments. It is composed of around 20000 labeled point clouds captured by a virtual vehicle navigating in town, highway, rural area, and city. The label mappings are mapped to that of the nuScenes dataset. In this work, we use the LiDAR point clouds from Synth4D-nuScenes as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://github.com/saltoricristiano/gipso-sfouda.
-  $^{10}$ DAPS-3D [51]: The DAPS-3D dataset consists of two subsets: DAPS-1 and DAPS-2; while the former is a semi-synthetic one with a larger scale, the latter is recorded during a real field trip of the cleaning robot to the territory of the VDNH Park in Moscow. Both subsets are with scans collected by a 64-line Ouster OS0 LiDAR sensor. We use the DAPS-1 subset in our experiments, which contains 11 LiDAR sequences with more than 23000 labeled point clouds. In this work, we use the LiDAR point clouds from DAPS-1 as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://github.com/subake/DAPS3D.
-  $^{11}$ nuScenes-C [53]: The nuScenes-C dataset is one of the corruption sets in the Robo3D benchmark, which is a comprehensive benchmark heading toward probing the robustness of 3D detectors and segmentors under out-of-distribution scenarios against natural corruptions that occur in real-world environments. A total number of eight corruption types, stemming from severe weather conditions, external disturbances, and internal sensor failure, are considered, including 'fog', 'wet ground', 'snow', 'motion blur', 'beam missing', 'crosstalk', 'incomplete echo', and 'cross-sensor' scenarios. These corruptions are simulated with rules constrained by physical principles or engineering experiences. In this work, we use the LiDAR point clouds from nuScenes-C as a downstream task to validate the robustness of
Table B: The statistics of superpixels (for front-view cameras) generated by SLIC [1] and different vision foundation modes [50, 122, 111, 123]. The horizontal axis denotes the number of superpixels per image. The vertical axis denotes the frequency of occurrence.
Method	Front Left	Front	Front Right
SLIC [1]	1400
80 90 100 110 120 130 140	1600
1400
1200
1000
800
700
600
500
400
300
200
100
0 50 100 150 200 250	1400
1200
1000
800
700
600
500
400
300
200
100
0 50 100 150 200 250
SAM [50]	700
600
500
400
300
200
100
0 50 100 150 200 250	800
600
500
400
300
200
100
0 50 100 150 200 250	800
700
600
500
400
300
200
100
0 50 100 150 200 250
X-Decoder [122]	4000
3500
3000
2500
1500
1000
0 5 10 15 20	3000
2500
2000
1500
1000
0 5 10 15 20 25	4000
3000
2000
1000
0 5 10 15 20
OpenSeeD [111]	3000
2500
2000
1500
1000
0 10 20 30 40	2000
1750
1500
1250
1000
750
500
0 10 20 30 40 50	3500
3000
2500
1500
1000
500
0 10 20 30 40 50
SEEM [123]	4000
3000
2000
1500
1000
0 10 20 30 40 50	1600
1400
1200
1000
800
600
400
200
0 10 20 30 40 50	3000
2500
2000
1500
1000
500
0 10 20 30 40
pertaining methods under out-of-distribution scenarios. More details of this dataset can be found at https://github.com/1dkong1205/Robo3D.
A.2 Vision Foundation Models
In this work, we conduct comprehensive experiments on analyzing the effects brought by different vision foundation models (VFMs), compared to the traditional SLIC [1] method. Some statistical analyses of these different visual partition methods are shown in Table B and Table C.
- SLIC [1] (traditional method): The SLIC model, which stands for 'simple linear iterative clustering', is a popular choice for visual partitions of RGB images. It adapts a k-means clustering approach to generate superpixels, in an efficient manner, and offers good unsupervised partition abilities for many downstream tasks. The pursuit of adherence to boundaries and computational and memory efficiency allows SLIC to perform well on different image collections. In this work, we follow SLidR [85] and use SLIC to generate
Table C: The statistics of superpixels (for back-view cameras) generated by SLIC [1] and different vision foundation modes [50, 122, 111, 123]. The horizontal axis denotes the number of superpixels per image. The vertical axis denotes the frequency of occurrence.
Method	Back Left	Back	Back Right
SLIC [1]	1600	1750	2500
1400	1500	2000
800	1250	1500
600	750	1000
400	500	500
300	200	200
200	150	150
150	120	120
100	90	100
70	80	90
50	100	110
30	120	130
SAM [50]	700	800	700
600	600	600
500	400	500
400	300	300
200	200	200
150	150	150
100	50	50
0	50	100
0	50	150
X-Decoder [122]	4000	4000	5000
3000	3000	4000
2000	2000	3000
1000	1000	1000
0	5	10
0	5	15
OpenSeeD [111]	3500	2500	4000
3000	2000	3500
2500	1500	3000
1500	1000	2500
1000	500	1500
0	10	20
0	10	20
0	10	15
SEEM [123]	3000	2000	3000
2500	1500	2500
1500	1000	2000
1000	500	1500
0	10	20
0	10	20
superpixels, with a fixed quota of 150 superpixels per image, and compare our framework with previous ones using SLIC superpixels. More details of this model can be found at https://github.com/valeoai/SLidR.
- SAM [50]: The Segment Anything Model (SAM) is a recent breakthrough towards zero-shot transferable visual understanding across a wide range of tasks. This model is trained on SA1B, a large-scale dataset with over 1 billion masks on 11M licensed and privacy-respecting images. As a result, SAM is able to segment images, with either point, box, or mask prompts, across different domains and data distributions. In this work, we use a fixed SAM model with the ViT-H backbone (termed as ViT-H-SAM model) to generate superpixels. We use this pretrained model directly without any further fine-tuning. More details of this model can be found at https://github.com/facebookresearch/segment-anything.
- X-Decoder [122]: The X-Decoder model is a generalized decoding framework that can predict pixel-level segmentation and language tokens seamlessly. This model is pretrained on three types of data, including panoptic segmentation, image-text pairs, and referring segmentation. For the panoptic segmentation task, the model is trained on COCO2017,
which includes around 104k images for model training. In this work, we use a fixed X-Decoder model termed BestSeg-Tiny to generate superpixels. We use this pretrained model directly without any further fine-tuning. More details of this model can be found at https://github.com/microsoft/X-Decoder.
- OpenSeeD [111]: The OpenSeeD model is designed for open-vocabulary segmentation and detection, which jointly learns from different segmentation and detection datasets. This model consists of an image encoder, a text encoder, and a decoder with foreground, background, and conditioned mask decoding capability. The model is trained on COCO2017 and Objects365, under the tasks of panoptic segmentation and object detection, respectively. In this work, we use a fixed OpenSeeD model termed openseed-swint-lang to generate superpixels. We use this pretrained model directly without any further fine-tuning. More details of this model can be found at https://github.com/IDEA-Research/OpenSeeD.
- SEEM [123]: The SEEM model contributes a new universal interactive interface for image segmentation, where 'SEEM' stands for 'segment everything everywhere with multi-modal prompts all at once'. The newly designed prompting scheme can encode various user intents into prompts in a joint visual-semantic space, which possesses properties of versatility, compositionality, interactivity, and semantic awareness. As such, this model is able to generalize well to different image datasets, under a zero-shot transfer manner. Similar to X-Decoder, SEEM is trained on COCO2017, with a total number of 107k images used during model training. In this work, we use a fixed SEEM model termed SEEM-oq101 to generate superpixels. We use this pretrained model directly without any further fine-tuning. More details of this model can be found at https://github.com/UX-Decoder/ Segment-Everything-Everywhere-A11-At-Once.
The quality of superpixels directly affects the performance of self-supervised representation learning. Different from the previous paradigm [85, 66], our Seal framework resorts to the recent VFMs for generating superpixels. Compared to the traditional SLIC method, these VFMs are able to generate semantically-aware superpixels that represent coherent semantic meanings of objects and backgrounds around the ego-vehicle.
As been verified in our experiments, these semantic superpixels have the ability to ease the "over-segment" problem in current self-supervised learning frameworks [85, 66] and further improve the performance for both linear probing and downstream fine-tuning.
The histograms shown in Table B and Table C verify that the number of superpixels per image of VFMs is much smaller than that of SLIC. This brings two notable advantages: i) Since semantically similar objects and backgrounds are grouped together in semantic superpixels, the "self-conflict" problem in existing approaches is largely mitigated, which directly boosts the quality of representation learning. ii) Since the embedding length  $D$  of the superpixel embedding features  $\mathbf{Q} \in \mathbb{R}^{M \times D}$  and superpoint embedding features  $\mathbf{K} \in \mathbb{R}^{M \times D}$  directly relates to computation overhead, a reduction (e.g., around 150 superpixels per image in SLIC [1] and around 25 superpixels per image in X-Decoder [122], OpenSeeD [111], and SEEM [123]) on  $D$  would allow us to train the segmentation models in a more efficient manner.
Some typical examples of our generated superpixels and their corresponding superpoints are shown in Fig. A, Fig. B, Fig. C, and Fig. D. As will be shown in Section B, our use of semantic superpixels generated by VFMs brings not only performance gains but also a much faster convergence rate during the model pretraining stage.
A.3 Implementation Detail
A.3.1 Data Split
For model pertaining, we follow the SLidR protocol [85] in data splitting. Specifically, the nuScenes [26] dataset consists of 700 training scenes in total, 100 of which are kept aside, which constitute the SLidR mini-val split. All models are pretrained using all the scans from the 600 remaining training scenes. The 100 scans in the mini-val split are used to find the best possible hyperparameters. The trained models are then validated on the official nuScenes validation set, without any kind of test-time augmentation or model ensemble. This is to ensure a fair comparison with previous works and also in line with the practical requirements.
Figure A: Illustration of the semantic superpixel-to-superpoint transformation in the proposed Seal framework. [Row 1 & 2] The raw data captured by the multi-view camera and LiDAR sensors. [Row 3 & 4] The semantic superpixel on the camera images and superpoint formed by projecting superpixel into the point cloud via camera-LiDAR correspondence. The superpixels are generated using SEEM [123]. Each color represents one distinct segment. Best viewed in color.
Figure B: Illustration of the semantic superpixel-to-superpoint transformation in the proposed Seal framework. [Row 1 & 2] The raw data captured by the multi-view camera and LiDAR sensors. [Row 3 & 4] The semantic superpixel on the camera images and superpoint formed by projecting superpixel into the point cloud via camera-LiDAR correspondence. The superpixels are generated using SEEM [123]. Each color represents one distinct segment. Best viewed in color.
For linear probing, the pretrained 3D network  $F_{\theta_p}$  is frozen with a trainable point-wise linear classification head which is trained for 50 epochs on an A100 GPU with a learning rate of 0.05, and batch size is 16 on the nuScenes train set for all methods.
For downstream fine-tuning tasks, we stick with the common practice in SLidR [85] whenever possible. The detailed data split strategies are summarized as follows.
- For fine-tuning on nuScenes [26], we follow the SLidR protocol to split the train set of nuScenes to generate  $1\%$ ,  $5\%$ ,  $10\%$ ,  $25\%$ , and  $100\%$  annotated scans for the training subset.
- For fine-tuning on DAPS-3D [51], we take sequences '38-18_7_72_90' as the training set and '38-18_7_72_90', '42-48_10_78_90', and '44-18_11_15_32' as the validation set.
Figure C: Illustration of the semantic superpixel-to-superpoint transformation in the proposed Seal framework. [Row 1 & 2] The raw data captured by the multi-view camera and LiDAR sensors. [Row 3 & 4] The semantic superpixel on the camera images and superpoint formed by projecting superpixel into the point cloud via camera-LiDAR correspondence. The superpixels are generated using SEEM [123]. Each color represents one distinct segment. Best viewed in color.
Figure D: Illustration of the semantic superpixel-to-superpoint transformation in the proposed Seal framework. [Row 1 & 2] The raw data captured by the multi-view camera and LiDAR sensors. [Row 3 & 4] The semantic superpixel on the camera images and superpoint formed by projecting superpixel into the point cloud via camera-LiDAR correspondence. The superpixels are generated using SEEM [123]. Each color represents one distinct segment. Best viewed in color.
- For fine-tuning on SynLiDAR [100], we use the sub-set which is a uniformly downsampled collection from the whole dataset.
- For fine-tuning on SemanticPOSS [75], we use sequences 00 and 01 as half of the annotated training scans and use sequences 00 to 05, except 02 for validation to create full of the annotated training samples.
- For fine-tuning on SemanticKITTI [3], Waymo Open [88], ScribbleKITTI [95], RELLIS-3D [49], SemanticSTF [102], and Synth4D [82], we follow the SLidR protocol to create  $1\%$ ,  $10\%$ , half, or full split of the annotated training scans, e.g., one scan is taken every 100 frame from the training set to get  $1\%$  of the labeled training samples. Notably, the point
cloud segmentation performance in terms of IoU is reported on the official validation sets for all the above-mentioned datasets.
A.3.2 Experimental Setup
In our experiments, we fine-tune the entire 3D network on the semantic segmentation task using a linear combination of the cross-entropy loss and the Lovász-Softmax loss [4] as training objectives on a single A100 GPU. For the few-shot semantic segmentation tasks, the 3D networks are fine-tuned for 100 epochs with a batch size of 10 for the SemanticKITTI [3], Waymo Open [88], ScribbleKITTI [95], RELLIS-3D [49], SemanticSTF [102], SemanticPOSS [75], DAPS-3D [51], SynLiDAR [100], and Synth4D [82] datasets.
For the nuScenes [26] dataset, we fine-tune the 3D network for 100 epochs with a batch size of 16 while training on the  $1\%$  annotated scans. The 3D network train on the other portions of nuScenes is fine-tuned for 50 epochs with a batch size of 16. We adopt different learning rates on the 3D backbone  $F_{\theta_p}$  and the classification head, except for the case that  $F_{\theta_p}$  is randomly initialized. The learning rate of  $F_{\theta_p}$  is set as 0.05 and the learning rate of the classification head is set as 2.0, respectively, for all the above-mentioned datasets except nuScenes. On the nuScenes dataset, the learning rate of  $F_{\theta_p}$  is set as 0.02.
We train our framework using the SGD optimizer with a momentum of 0.9, a weight decay of 0.0001, and a dampening ratio of 0.1. The cosine annealing learning rate strategy is adopted which decreases the learning rate from its initial value to zero at the end of the training.
A.3.3 Data Augmentation
For the model pretraining, we apply two sets of data augmentations on the point cloud and the multi-view image, respectively, and update the point-pixel correspondences after each augmentation by following SLidR [85].