Title: Segment Any Point Cloud Sequences by Distilling Vision Foundation Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: autonomous driving

point cloud segmentation

self-supervised learning

3D scene understanding
Abstract: Recent advancements in vision foundation models (VFMs) have opened up new possibilities for versatile and efficient visual perception. In this work, we introduce Seal, a novel framework that harnesses VFMs for segmenting diverse automotive point cloud sequences. Seal exhibits three appealing properties: i) Scalability: VFMs are directly distilled into point clouds, obviating the need for annotations in either 2D or 3D during pretraining. ii) Consistency: Spatial and temporal relationships are enforced at both the camera-to-LiDAR and point-to-segment regularization stages, facilitating cross-modal representation learning. iii) Generalizability: Seal enables knowledge transfer in an off-the-shelf manner to downstream tasks involving diverse point clouds, including those from real/synthetic, low/high-resolution, large/small-scale, and clean/corrupted datasets. Extensive experiments conducted on eleven different point cloud datasets showcase the effectiveness and superiority of Seal. Notably, Seal achieves a remarkable 45.0% mIoU on nuScenes after linear probing, surpassing random initialization by 36.9% mIoU and outperforming prior arts by 6.1% mIoU. Moreover, Seal demonstrates significant performance gains over existing methods across 20 different few-shot fine-tuning tasks on all eleven tested point cloud datasets. The code is available at this link.

Segment Any Point Cloud Sequences by Distilling Vision Foundation Models

Abstract

Recent advancements in vision foundation models (VFMs) have opened up new possibilities for versatile and efficient visual perception. In this work, we introduce Seal , a novel framework that harnesses VFMs for segmenting diverse automotive point cloud sequences. Seal exhibits three appealing properties: i) Scalability: VFMs are directly distilled into point clouds, obviating the need for annotations in either 2D or 3D during pretraining. ii) Consistency: Spatial and temporal relationships are enforced at both the camera-to-LiDAR and point-to-segment regularization stages, facilitating cross-modal representation learning. iii) Generalizability: Seal enables knowledge transfer in an off-the-shelf manner to downstream tasks involving diverse point clouds, including those from real/synthetic, low/highresolution, large/small-scale, and clean/corrupted datasets. Extensive experiments conducted on eleven different point cloud datasets showcase the effectiveness and superiority of Seal. Notably, Seal achieves a remarkable 45 . 0% mIoU on nuScenes after linear probing, surpassing random initialization by 36 . 9% mIoU and outperforming prior arts by 6 . 1% mIoU. Moreover, Seal demonstrates significant performance gains over existing methods across 20 different few-shot fine-tuning tasks on all eleven tested point cloud datasets. The code is available at this link 2 .

1 Introduction

Inspired by the achievements of large language models (LLMs) [87, 35, 113, 73, 18], a wave of vision foundation models (VFMs), such as SAM [50], X-Decoder [122], and SEEM [123], has emerged. These VFMs are revolutionizing the field of computer vision by facilitating the acquisition of pixellevel semantics with greater ease. However, limited studies have been conducted on developing VFMs for the 3D domain. To bridge this gap, it holds great promise to explore the adaptation or extension of existing 2D VFMs for 3D perception tasks.

As an important 3D perception task, accurately segmenting the surrounding points captured by onboard LiDAR sensors is crucial for ensuring the safe operation of autonomous vehicles [92, 94, 81]. However, existing point cloud segmentation models rely heavily on large annotated datasets for training, which poses challenges due to the labor-intensive nature of point cloud labeling [2, 95]. To address this issue, recent works have explored semi- [55, 59] or weakly-supervised [95, 60] approaches to alleviate the annotation burden. While these approaches show promise, the trained point segmentors tend to perform well only on data within their distribution, primarily due to

SensorData

SLIC

SAM

X-Decoder

SEEM

Pose

Camera

Pose

GT

Seal

Seal

Seal

Seal

LiDAR

significant configuration differences (such as beam number, camera angle, emit rate) among different sensors [101, 28, 92]. This limitation inevitably hampers the scalability of point cloud segmentation.

To address the aforementioned challenges, we aim to develop a framework that can learn informative features while addressing the following objectives: i) Utilizing raw point clouds as input, thereby eliminating the need for semi or weak labels and reducing annotation costs. ii) Leveraging spatial and temporal cues inherent in driving scenes to enhance representation learning. iii) Ensuring generalizability to diverse downstream point clouds, beyond those used in the pretraining phase. Drawing inspiration from recent advancements in cross-modal representation learning [50, 122, 111, 123] and building upon the success of VFMs [50, 122, 111, 123], we propose a methodology that distills semantically-rich knowledge from VFMs to support self-supervised representation learning on challenging automotive point clouds. Our core idea is to leverage the 2D-3D correspondence inbetween LiDAR and camera sensors and construct high-quality contrastive samples for cross-modal representation learning. As shown in Fig. 1, VFMs are capable of generating semantic superpixel groups from the camera views 3 and provide off-the-shelf semantic coherence for distinct objects and backgrounds in the 3D scene. Such consistency can be distilled from 2D to 3D and further yields promising performance on downstream tasks, which we will revisit more formally in later sections.

Compared to previous framework [85, 66], our VFM-assisted contrastive learning has three notable advantages: i) The semantically-aware region partition mitigates the severe 'self-conflict' problem in driving scene contrastive learning. ii) The high-quality contrastive samples gradually form a more coherent optimization landscape, yielding a faster convergence rate than previous configurations. iii) A huge reduction in the number of superpixels also extenuates the overhead during pretraining.

Since a perfect calibration between LiDAR and cameras is often hard to meet, we further design a superpoint temporal consistency regularization to mitigate the case that there are potential errors from the sensor synchronization aspect. This objective directly resorts to the accurate geometric information from the point cloud and thus improves the overall resilience and reliability. As will be shown in the following sections, our framework (dubbed Seal ) is capable of se gmentation a ny point c l oud sequences across a wide range of downstream task on different automotive point cloud datasets.

To summarize, this work has the following key contributions:

To the best of our knowledge, this study represents the first attempt at utilizing 2D vision foundation models for self-supervised representation learning on large-scale 3D point clouds.

We introduce Seal , a scalable, consistent, and generalizable framework designed to capture semantic-aware spatial and temporal consistency, enabling the extraction of informative features from automotive point cloud sequences.

Our approach demonstrates clear superiority over previous state-of-the-art (SoTA) methods in both linear probing and fine-tuning for downstream tasks across 11 different point cloud datasets with diverse data configurations.

2 Related Work

Vision Foundation Models . Recent excitement about building powerful visual perception systems based on massive amounts of training data [80, 50] or advanced self-supervised learning techniques [8, 74] is revolutionizing the community. The segment anything model (SAM) [50] sparks a new trend of general-purpose image segmentation which exhibits promising zero-shot transfer capability on diverse downstream tasks. Concurrent works to SAM, such as X-Decoder [122], OpenSeeD [111], SegGPT [99], and SEEM [123], also shed lights on the direct use of VFMs for handling different kinds of image-related tasks. In this work, we extend this aspect by exploring the potential of VFMs for point cloud segmentation. We design a new framework taking into consideration the off-the-shelf semantic awareness of VFMs to construct better spatial and temporal cues for representation learning.

Point Cloud Segmentation . Densely perceiving the 3D surroundings is important for autonomous vehicles [94, 41]. Various point cloud segmenters have been proposed, including methods based on raw points [42, 90, 43, 114, 78], range view [69, 106, 93, 118, 20, 16, 52], bird's eye view [115, 119, 9], voxels [19, 120, 37, 36], and multi-view fusion [62, 89, 107, 121, 17, 79]. Despite the promising results achieved, existing 3D segmentation models rely on large sets of annotated data for training, which hinders the scalability [28]. Recent efforts seek semi [55, 56, 59], weak [95, 40, 86, 63, 60], and active [64, 44, 117] supervisions or domain adaptation techniques [48, 47, 54, 76, 82, 68] to ease the annotation cost. In this work, we resort to self-supervised learning by distilling foundation models using camera-to-LiDAR associations, where no annotation is required during the pretraining stage.

3D Representation Learning . Stemmed from the image vision community, most 3D self-supervised learning methods focused on object-centric point clouds [84, 77, 83, 14, 97, 45, 110, 29, 91] or indoor scenes [38, 15, 22, 108, 10, 39, 57, 58, 109] by either pretext task learning [21, 70, 71, 112, 30], contrastive learning [11, 32, 12, 34, 27, 13, 46, 98, 104] or mask modeling [105, 31, 24, 61], where the scale and diversity are much lower than the outdoor driving scenes [67, 6]. PointContrast [103], DepthContrast [116], and SegContrast [72] are prior attempts aimed to establish contrastive objectives on point clouds. Recently, Sautier et al. [85] proposed the first 2D-to-3D representation distillation method called SLidR for cross-modal self-supervised learning on large-scale point clouds and exhibits promising performance. The follow-up work [66] further improves this pipeline with a semantically tolerant contrastive constraint and a class-balancing loss. Our framework also stems from the SLidR paradigm. Differently, we propose to leverage VFMs to establish the cross-modal contrastive objective which better tackles this challenging representation learning task. We also design a superpoint temporal consistency regularization to further enhance feature learning.

3 Seal: A Scalable, Consistent, and Generalizable Framework

In this section, we first revisit the 2D-to-3D representation distillation [85] as preliminaries (Sec. 3.1). We then elaborate on the technical details of our framework, which include the VFM-assisted spatial contrastive learning (Sec. 3.2) and the superpoint temporal consistency regularization (Sec. 3.3).

3.1 Preliminaries

Given a point cloud P t = { p t i , e t i | i = 1 , .., N } consists of N points collected by a LiDAR acquisition at timestamp t , where p i ∈ R 3 denotes the point coordinates and e i ∈ R L is the feature embedding (intensity, elongation, etc. ), our goal is to transfer the knowledge from an image set I t = {{ I t 1 , ..., I t j }| j = 1 , ..., V } captured by V synchronized cameras at t to point cloud P t , where I ∈ R 3 × H × W is a single image with height H and width W . Prior works [85, 66] achieve this goal by first aggregating image regions that are similar in the RGB space into a superpixel set Φ S = {{ s 1 , ..., s m }| m = 1 , ..., M } , via the unsupervised SLIC [1] algorithm. The corresponding

Time

Sensor Data

Super Pixel &Point

2D&3DNet

Grouping

ContrastiveObjective

RANSAC

HDBSCAN

pt

F

LiDAR

个

Fθ

Hop

2D-to-3D

Framet

Group

:......

gt

Spatial

Temporal

VFM

Camera

Gθ

Hwi

Group

2D-to-3D

Frame

u+1

pt+n

RANSAC

HDBSCAN

LiDAR

Hwp

superpoint set Φ O = {{ o 1 , ..., o m }| m = 1 , ..., M } can be obtained by leveraging known sensor calibration parameters to establish correspondence between the points and image pixels. Specifically, since the LiDAR and cameras usually operate at different frequencies [7, 26], we transform each LiDAR point cloud p i = ( x i , y i , z i ) at timestamp t l to the pixel ˆ p i = ( u i , v i ) ∈ R 2 in the image plane at timestamp t c via the coordinate transformation. The point-to-pixel mapping is as follows:

where symbol Γ K denotes the camera intrinsic matrix. Γ camera ← ego t c , Γ ego t c ← global , Γ global ← ego t l , and Γ ego t l ← lidar are the extrinsic matrices for the transformations of ego to camera at t c , global to ego at t c , ego to global at t l , and LiDAR to ego at t l , respectively.

3.2 Semantic Superpixel Spatial Consistency

Superpixel Generation . Prior works resort to SLIC [1] to group visually similar regions in the image into superpixels. This method, however, tends to over-segment semantically coherent areas (as shown in Fig. 1) and inevitably leads to several difficulties for contrastive learning. One of the main impediments is the so-called 'self-conflict', where superpixels belonging to the same semantics become negative samples [96]. Although [66] proposed a semantically-tolerant loss to ease this problem, the lack of high-level semantic understanding still intensifies the implicit hardness-aware property of the contrastive loss. We tackle this challenge by generating semantic superpixels with VFMs. As shown in Fig. 1 and Fig. 4, these VFMs provide semantically-rich superpixels and yield much better representation learning effects in-between near and far points in the LiDAR point cloud.

VFM-Assisted Contrastive Learning . Let F θ p : R N × (3+ L ) → R N × C be a 3 D encoder with trainable parameters θ p , that takes a LiDAR point cloud as input and outputs a C -dimensional per-point feature. Let G θ i : R H × W × 3 → R H s × W s × C be an image encoder with parameters θ i , which is initialized from a set of 2D self-supervised pretrained parameters. The goal of our VFMassisted contrastive learning is to transfer the knowledge from the pretrained 2D network to the 3D network via contrastive loss at the semantic superpixel level. To compute this VFM-assisted contrastive loss, we first build trainable projection heads H ω p and H ω i which map the 3D point features and 2D image features into the same D -dimensional embedding space. The point projection head H ω p : R N × C → R N × D is a linear layer with ℓ 2 -normalization. The image projection head

H ω i : R H s × W s × C → R H s × W s × D is a convolution layer with a kernel size of 1 , followed by a fixed bilinear interpolation layer with a ratio of 4 in the spatial dimension, and outputs with ℓ 2 -normalization.

We distill the knowledge from the 2D network into the 3D network which is in favor of a solution where a semantic superpoint feature has a strong correlation with its corresponding semantic superpixel feature than any other features. Concretely, the superpixels Φ S are used to group pixel embedding and point embedding features. Then, an average pooling function is applied to each grouped point and pixel embedding features, to extract the superpixel embedding features Q ∈ R M × D and superpoint embedding features K ∈ R M × D . The VFM-assisted contrastive loss is formulated as:

̸

where ⟨ q i , k j ⟩ denotes the scalar product between superpoint embedding features and superpixel embedding features to measure the similarity. τ is the temperature term.

Role in Our Framework . Our VFM-assisted contrastive objective exhibits superiority over previous methods from three aspects: i) The semantically-rich superpixels provided by VFMs mitigate the 'self-conflict' problem in existing approaches. ii) As we will show in the following sections, the high-quality contrastive samples from VFMs gradually form a much more coherent optimization landscape and yield a faster convergence rate than the unsupervised superpixel generation method. iii) Using superpixels generated by VFMs also helps our framework run faster than previous works, since the embedding length of Q and K has been reduced from a few hundred (SLIC [1]) to dozens (ours).

3.3 Semantic Superpoint Temporal Consistency

The assumption of having perfectly synchronized LiDAR and camera data might become too ideal and cannot always be fulfilled in actual deployment, which limits the scalability. In this work, we resort to accurate geometric information from point clouds to further relieve this constraint.

Implicit Geometric Clustering . To group coarse instance segments in a LiDAR frame, we first partition non-ground plane points G t by eliminating the ground plane of a point cloud P t at timestamp t in an unsupervised manner via

q1

q2

q3

O

k

k

k

RANSAC [25]. Then, we group G t to yield a set of M k segments K t = {K t 1 , ..., K t M k } with the help of HDBSCAN [23]. To map different segment views at different timestamps, we transform those LiDAR frames across different timestamps to the global frame and aggregate them with concatenations. The aggregated point cloud is denoted as ˜ P = { ˜ P t , ..., ˜ P t + n } . Similarly, we generate non-ground plane ˜ G = { ˜ G t , ..., ˜ G t + n } from ˜ P via RANSAC [25]. In the same manner as the single scan, we group ˜ G to obtain M k segments ˜ K = { ˜ K 1 , ..., ˜ K M k } . To generate the segment masks for all n +1 scans at n consecutive timestamps, i.e. , ˜ K = { ˜ K t , ..., ˜ K t + n } , we maintain the point index mapping from the aggregated point cloud ˜ P to the n +1 individual scans.

Superpoint Temporal Consistency . We leverage the clustered segments to compute the temporal consistency loss among related semantic superpoints. Here we assume n = 1 ( i.e. the next frame) without loss of generalizability. Specifically, given a sampled temporal pair ˜ P t and ˜ P t +1 and their corresponding segments ˜ K t and ˜ K t +1 , we compute the point-wise features ˆ F t ∈ R N × D and ˆ F t +1 ∈ R N × D from the point projection head H ω p . As for the target embedding, we split the point features ˆ F t and ˆ F t +1 into M k groups by segments ˜ K t and ˜ K t +1 . Then, we apply an average pooling operation on ˆ F t +1 to get M k target mean feature vectors F t +1 = {F t +1 1 , F t +1 2 , ..., F t +1 M k } , where F t +1 M k ∈ R 1 × D . Let the split point feature ˆ F t be F t = {F t 1 , F t 2 , ..., F t M k } , where F t M k ∈ R k × D and

SLIC

SAM

X-Decoder

OpenSeeD

SEEM

Query: "car"

Query: "manmade"

Query:"truck"

k is the number of points in the corresponding segment. We compute the temporal consistency loss L t → t +1 to minimize the differences between the point features in the current frame (timestamp t ) and the corresponding segment mean features from the next frame (timestamp t +1 ) as follows:

̸

Since the target embedding for all points within a segment in the current frame serves as the mean segment representation from the next frame, this loss will force points from a segment to converge to a mean representation while separating from other segments, implicitly clustering together points from the same instance. Fig. 3 provides the positive feature correspondence in our contrastive learning framework. Furthermore, we swap ˆ F t when generating the target mean embedding features to form a symmetric representation. In this way, the correspondence is encouraged from both t → t +1 and t +1 → t , which leads to the following optimization objective: L tmp = L t → t +1 + L t +1 → t .

Point to Segment Regularization . To pull close the LiDAR points belonging to the same instance at timestamp t , we minimize the distance between the point feature F t and the corresponding mean cluster feature C t . To implement this, we leverage a max-pooling function to pool F t according to the segments to obtain C t = {C t 1 , C t 2 , ..., C t M k } , where C t M k ∈ R 1 × D . The point-to-segment regularization is thus achieved via the following loss function:

̸

where N k represents the number of points within the corresponding segment. The final optimization objective is to minimize the aforementioned semantic spatial consistency loss L vfm , temporal consistency loss L tmp , and the point-to-segment regularization loss L p 2 s .

Role in Our Framework . Our semantic superpoint temporal consistency resorts to the accurate geometric information from the point cloud and exploits the different views of an instance across different timestamps to learn a temporally consistent representation. Considering the worst-case scenario that the 2D-3D correspondence between the LiDAR and camera sensors becomes unreliable, this geometric constraint can still effectively mitigate the potential errors that occur in inaccurate crosssensor calibration and synchronization. Besides, our point-to-segment regularization mechanism can serve to aggregate the spatial information thus contributing to the effect of better-distinguishing instances in the LiDAR-acquired scene, e.g. , 'car' and 'truck'. As we will show in the following sections, our experimental results are able to verify the effectiveness and superiority of the proposed consistency regularization objectives, even under certain degrees of perturbation in-between sensors.

4 Experiments

4.1 Settings

Data . We verify the effectiveness of our approach on eleven different point cloud datasets. 1 nuScenes [7, 26], 2 SemanticKITTI [3], and 3 Waymo Open [88] contain large-scale LiDAR scans collected from real-world driving scenes; while the former adopted a Velodyne HDL32E, data from the latter two datasets are acquired by 64-beam LiDAR sensors. 4 ScribbleKITTI [95] shares the data with [3] but are weakly annotated with line scribbles. 5 RELLIS-3D [49] is a multimodal dataset collected in an off-road campus environment. 6 SemanticPOSS [75] is a small-scale set with an emphasis on dynamic instances. 7 SemanticSTF [102] consist of LiDAR scans from adverse weather conditions. 8 SynLiDAR [100], 9 Synth4D [82], and 10 DAPS-3D [51] are synthetic datasets collected from simulators. We also conduct extensive robustness evaluations on the 11 nuScenes-C dataset proposed in the Robo3D benchmark [53], a comprehensive collection of eight out-of-distribution corruptions that occur in driving scenarios. More details on these datasets are in the Appendix.

Implementation Details . We use MinkUNet [19] as the 3D backbone which takes cylindrical voxels of size 0 . 10 mas inputs. Similar to [85, 66], our 2D backbone is a ResNet-50 [33] pretrained with MoCoV2 [12]. We pretrain our segmentation network for 50 epochs on two GPUs with a batch size of 32, using SGD with momentum and a cosine annealing scheduler. For fine-tuning, we follow the exact same data split, augmentation, and evaluation protocol as SLidR [85] on nuScenes and SemanticKITTI , and adopt a similar procedure on other datasets. The training objective is to minimize a combination of the cross-entropy loss and the Lovász-Softmax loss [4]. We compare the results of prior arts from their official reporting [85, 66]. Since PPKT [65] and SLidR [85] only conducted experiments on nuScenes and SemanticKITTI , we reproduce their best-possible performance on the other nine datasets using public code. For additional details, please refer to the Appendix.

Metrics . We follow the conventional reporting of Intersection-over-Union (IoU) on each semantic class and the mean IoU (mIoU) across all classes. For robustness probing, we follow the Robo3D protocol [53] and report the mean Corruption Error (mCE) and mean Resilience Rate (mRR) scores calculated by using the MinkUNet 18 (torchsparse) implemented by Tang et al. [89] as the baseline.

4.2 Comparative Study

Comparison to State-of-the-Arts . We compare Seal with random initialization and five existing pretraining approaches under both linear probing (LP) protocol and few-shot fine-tuning settings on nuScenes [26] in Table 1. We observe that the pretraining strategy can effectively improve the accuracy of downstream tasks, especially when the fine-tuning budget is very limited ( e.g. 1% , 5% , and 10% ). Our framework achieves 44 . 95% mIoU under the LP setup - a 4 . 47% mIoU lead over to the prior art ST-SLidR [66] and a 6 . 15% mIoU boost compared to the baseline SLidR [85]. What is more, Seal achieves the best scores so far across all downstream fine-tuning tasks, which verifies the superiority of VFM-assisted contrastive learning and spatial-temporal consistency regularization. We also show that recent out-of-context augmentation [55] could enhance the feature learning during fine-tuning, which establishes a new state of the art on the challenging nuScenes benchmark.

Downstream Generalization . To further verify the capability of Seal on segmenting any automotive point clouds, we conduct extensive experiments on eleven different datasets and show the results in Table 1 and Table 2. Note that each of these datasets has a unique data collection protocol and a diverse data distribution - ranging from diverse sensor types and data acquisition environments to scales and fidelity - making this evaluation a comprehensive one. The results show that our framework constantly outperforms prior arts across all downstream tasks on all eleven datasets, which concretely supports the effectiveness and superiority of our proposed approach.

Semi-Supervised Learning . Recent research [55, 59] reveals that combining both labeled and unlabeled data for semi-supervised point cloud segmentation can significantly boost the performance on downstream tasks. We follow these works to implement such a learning paradigm where a momentum-updated teacher model is adopted to assign pseudo-labels for the unlabeled data. The results from the last row of Table 1 show that Seal is capable of providing reliable supervision signals for data-efficient learning. Notably, our framework with partial annotation is able to surpass some recent fully-supervised learning methods. More results on this aspect are included in the Appendix.

Robustness Probing . It is often of great importance to assess the quality of learned representations on out-of-distribution data, especially for cases that occur in the real-world environment. We resort to the recently established nuScenes-C in the Robo3D benchmark [53] for such robustness evaluations. From Table 3 we observe that the self-supervised learning methods [65, 85] in general achieve better robustness than their baseline [19]. Seal achieves the best robustness under almost all corruption types and exhibits superiority over other recent segmentation backbones with different LiDAR representations, such as range view [16], BEV [115], raw points [78], and multi-view fusion [89].

Qualitative Assessment . We visualize the predictions of each pertaining method pretrained and fine-tuned on nuScenes [26] in Fig. 5. A clear observation is the substantial enhancement offered by all pretraining methods when juxtaposed with a baseline of random initialization. Diving deeper into the comparison among the trio of tested techniques, Seal stands out, delivering superlative segmentation outcomes, especially in the intricate terrains of driving scenarios. This is credited to the strong spatial and temporal consistency learning that Seal tailored to excite during the pretraining. We have cataloged additional examples in the Appendix for more detailed visual comparisons.

4.3 Ablation Study

Foundation Model Comparisons . We provide the first study on adapting VFMs for large-scale point cloud representation learning and show the results in Table 4. We observe that different VFMs exhibit

diverse abilities in encouraging contrastive objectives. All VFMs show larger gains than SLIC [1] with both frameworks, while SEEM [123] in general performs the best. Notably, SAM [50] tends to generate more fine-grained superpixels and yields better results when fine-tuning with more annotated data. We conjecture that SAM [50] generally provides more negative samples than the other three VFMs which might be conducive to the superpixel-driven contrastive learning. On all setups, Seal constantly surpasses SLidR [85] by large margins, which verifies the effectiveness of our framework.

Cosine Similarity . We visualize three examples of feature similarity across different VFMs in Fig. 4. We observe that our contrastive objective has already facilitated distinction representations before fine-tuning. The semantically-rich VFMs such as X-Decoder [122], OpenSeeD [111], and SEEM [123] offers overt feature cues for recognizing objects and backgrounds; while the unsupervised (SLIC [1]) or too fine-grained (SAM [50]) region partition methods only provide limited semantic awareness. Such behaviors have been reflected in the linear probing and downstream fine-tuning performance (see Table 4), where SEEM [123] tends to offer better consistency regularization impacts during the cross-sensor representation learning.

Component Analysis . Table 5 shows the ablation results of each component in the Seal framework. Specifically, direct integration of VFMs (row #3) or temporal consistency learning (row #2) brings 4 . 20% and 1 . 65% mIoU gains in LP, respectively, while a combination of them (row #4) leads to a 5 . 21% mIoU gain. The point-to-segment regularization (row #5) alone also provides a considerable performance boost of around 4 . 55% mIoU. Finally, an integration of all proposed components (row #6) yields our best-performing model, which is 6 . 15% mIoU better than the prior art [85] in LP and also outperforms on all in-distribution and -out-of-distribution downstream setups.

Sensor Misalignment . An accurate calibration is crucial for establishing correct correspondences between LiDAR and cameras. In most cases, those sensors should be well-calibrated for an autonomous car. It is unusual if the calibration is completely unknown, but it is possible to be imprecise due to a lack of maintenance. Hence, we conduct the following experiments to validate the robustness of our method. For each point coordinate p i = ( x i , y i , z i ) in a LiDAR point cloud, its corresponding pixel ˆ p i = ( u i , v i ) in the camera view can be found via Eq. 1. To simulate the misalignment between LiDAR and cameras, we insert random noises into the camera extrinsic matrix Γ K , with a relative proportion of 1% , 5% , and 10% . Table 6 shows the results of PPKT [65], SLidR [85], and Seal under such noise perturbations. We observe that the possible calibration errors between LiDAR

Random

PPKT

SLidR

Seal (Ours)

and cameras will cause performance degradation for different pretraining approaches (compared to Table 1). The performance degradation for PPKT [65] is especially prominent; we conjecture that this is because the point-wise consistency regularization of PPKT [65] relies heavily on the calibration accuracy and encounters problems under misalignment. Both SLidR [85] and Seal exhibit certain robustness; we believe the superpixel-level consistency is less sensitive to calibration perturbations. It is worth mentioning that Seal can maintain good performance under calibration error, since: i) Our VFM-assisted representation learning tends to be more robust; and ii) We enforce superpoint temporal consistency during the pertaining which does not rely on the 2D-3D correspondence.

5 Concluding Remark

In this study, we presented Seal , a versatile self-supervised learning framework capable of segmenting any automotive point clouds by encouraging spatial and temporal consistency during the representation learning stage. Additionally, our work pioneers the utilization of VFMs to enhance 3D scene understanding. Extensive experimental results on 20 downstream tasks across eleven different point cloud datasets verified the effectiveness and superiority of our framework. We aspire for this research to catalyze further integration of large-scale 2D and 3D representation learning endeavors, which could shed light on the development of robust and annotation-efficient perception models.

Potential Limitations . Although our proposed framework holistically improved the point cloud segmentation performance across a wide range of downstream tasks, there are still some limitations that could hinder the scalability. i) Our model operates under the assumption of impeccably calibrated and synchronized LiDAR and cameras, which might not always hold true in real-world scenarios. ii) We only pretrain the networks on a single set of point clouds with unified setups; while aggregating more abundant data from different datasets for pretraining would further improve the generalizability of this framework. These limitations present intriguing avenues for future investigations.

Acknowledgements . This study is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOE-T2EP20221-0012), NTU NAP, and under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). This study is also supported by the National Key R&D Program of China (No. 2022ZD0161600). We thank Tai Wang for the insightful review and discussion.

References

Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence , 34(11):2274-2282, 2012. 2, 3, 4, 5, 6, 9, 21, 22, 23, 31, 32, 33

Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Jürgen Gall, and Cyrill Stachniss. Towards 3d lidar-based semantic scene understanding of 3d point cloud sequences: The semantickitti dataset. International Journal of Robotics Research , 40:959-96, 2021. 1

Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Juergen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In IEEE/CVF International Conference on Computer Vision , pages 9297-9307, 2019. 7, 9, 18, 19, 25, 26, 27, 29

Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: a tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4413-4421, 2018. 7, 26

Mario Bijelic, Tobias Gruber, Fahim Mannan, Florian Kraus, Werner Ritter, Klaus Dietmayer, and Felix Heide. Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11682-11692, 2020. 20

Alexandre Boulch, Corentin Sautier, Björn Michele, Gilles Puy, and Renaud Marlet. Also: Automotive lidar self-supervision by occupancy estimation. arXiv preprint arXiv:2212.05867 , 2022. 3

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11621-11631, 2020. 4, 7

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE/CVF International Conference on Computer Vision , pages 9650-9660, 2021. 3

Qi Chen, Sourabh Vora, and Oscar Beijbom. Polarstream: Streaming lidar object detection and segmentation with polar pillars. In Advances in Neural Information Processing Systems , volume 34, 2021. 3

Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7020-7030, 2023. 3

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning , pages 1597-1607, 2020. 3

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020. 3, 7, 26

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In IEEE/CVF International Conference on Computer Vision , pages 9640-9649, 2021. 3

Ye Chen, Jinxian Liu, Bingbing Ni, Hang Wang, Jiancheng Yang, Ning Liu, Teng Li, and Qi Tian. Shape self-correction for unsupervised point cloud understanding. In IEEE/CVF International Conference on Computer Vision , pages 8382-8391, 2021. 3

Yujin Chen, Matthias Nießner, and Angela Dai. 4dcontrast: Contrastive learning with dynamic correspondences for 3d scene understanding. In European Conference on Computer Vision , pages 543-560, 2022. 3

Huixian Cheng, Xianfeng Han, and Guoqiang Xiao. Cenet: Toward concise and efficient lidar semantic segmentation for autonomous driving. In IEEE International Conference on Multimedia and Expo , pages 1-6, 2022. 3, 8

Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu. Af2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12547-12556, 2021. 3

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. 1

Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3075-3084, 2019. 3, 7, 8, 26

Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy. Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds. In International Symposium on Visual Computing , pages 207-222, 2020. 3

Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In IEEE/CVF International Conference on Computer Vision , pages 1422-1430, 2015. 3

Yuhao Dong, Zhuoyang Zhang, Yunze Liu, and Li Yi. Complete-to-partial 4d distillation for selfsupervised point cloud sequence representation learning. arXiv preprint arXiv:2212.05330 , 2022. 3

Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 226-231, 1996. 5

Christoph Feichtenhofer, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal learners. In Advances in Neural Information Processing Systems , volume 35, 2022. 3

Martin A. Fischler and Robert C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM , 24(6):381-395, 1981. 5

Whye Kit Fong, Rohit Mohan, Juana Valeria Hurtado, Lubing Zhou, Holger Caesar, Oscar Beijbom, and Abhinav Valada. Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking. IEEE Robotics and Automation Letters , 7:3795-3802, 2022. 4, 7, 8, 9, 10, 18, 19, 23, 24, 26, 27, 28, 29, 30, 34, 35, 36

Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Unsupervised semantic segmentation by contrasting object mask proposals. In IEEE/CVF International Conference on Computer Vision , pages 10052-10062, 2021. 3

Biao Gao, Yancheng Pan, Chengkun Li, Sibo Geng, and Huijing Zhao. Are we hungry for 3d lidar data for semantic segmentation? a survey of datasets and methods. IEEE Transactions on Intelligent Transportation Systems , 2021. 2, 3

Siddhant Garg and Mudit Chaudhary. Serp: Self-supervised representation learning using perturbed point clouds. arXiv preprint arXiv:2209.06067 , 2022. 3

Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations , 2018. 3

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022. 3

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9729-9738, 2020. 3

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016. 7, 26

Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International Conference on Machine Learning , pages 4182-4192, 2020. 3

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen

Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022. 1

Fangzhou Hong, Lingdong Kong, Hui Zhou, Xinge Zhu, Hongsheng Li, and Ziwei Liu. Unified 3d and 4d panoptic segmentation via dynamic shifting network. arXiv preprint arXiv:2203.07186 , 2022. 3

Fangzhou Hong, Hui Zhou, Xinge Zhu, Hongsheng Li, and Ziwei Liu. Lidar-based panoptic segmentation via dynamic shifting network. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13090-13099, 2021. 3

Ji Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15587-15597, 2021. 3

Ji Hou, Saining Xie, Benjamin Graham, Angela Dai, and Matthias Nießner. Pri3d: Can 3d priors help 2d representation learning? In IEEE/CVF International Conference on Computer Vision , pages 5693-5702, 2021. 3

Qingyong Hu, Bo Yang, Guangchi Fang, Yulan Guo, Aleš Leonardis, Niki Trigoni, and Andrew Markham. Sqn: Weakly-supervised semantic segmentation of large-scale 3d point clouds. In European Conference on Computer Vision , pages 600-619, 2022. 3

Qingyong Hu, Bo Yang, Sheikh Khalid, Wen Xiao, Niki Trigoni, and Andrew Markham. Towards semantic segmentation of urban-scale 3d point clouds: A dataset, benchmarks and challenges. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4977-4987, 2021. 3

Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11108-11117, 2020. 3

Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Learning semantic segmentation of large-scale point clouds with random sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(11):8338-8354, 2022. 3

Zeyu Hu, Xuyang Bai, Runze Zhang, Xin Wang, Guangyuan Sun, Hongbo Fu, and Chiew-Lan Tai. Lidal: Inter-frame uncertainty based active learning for 3d lidar semantic segmentation. In European Conference on Computer Vision , pages 248-265, 2022. 3

Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu. Spatio-temporal self-supervised representation learning for 3d point clouds. In IEEE/CVF International Conference on Computer Vision , pages 6535-6545, 2021. 3

Olivier J. Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron Van den Oord, Oriol Vinyals, and Joao Carreira. Efficient visual pretraining with contrastive detection. In IEEE/CVF International Conference on Computer Vision , pages 10086-10096, 2021. 3

Maximilian Jaritz, Tuan-Hung Vu, Raoul De Charette, Émilie Wirbel, and Patrick Pérez. Cross-modal learning for domain adaptation in 3d semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(2):1533-1544, 2023. 3

Maximilian Jaritz, Tuan-Hung Vu, Raoul de Charette, Emilie Wirbel, and Patrick Pérez. xmuda: Crossmodal unsupervised domain adaptation for 3d semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12605-12614, 2020. 3

Peng Jiang, Philip Osteen, Maggie Wigness, and Srikanth Saripallig. Rellis-3d dataset: Data, benchmarks and analysis. In IEEE International Conference on Robotics and Automation , pages 1110-1116, 2021. 7, 19, 20, 25, 26

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In IEEE/CVF International Conference on Computer Vision , pages 4015-4026, 2023. 1, 2, 3, 6, 9, 21, 22, 31, 32, 33

Alexey Klokov, Di Un Pak, Aleksandr Khorin, Dmitry Yudin, Leon Kochiev, Vladimir Luchinskiy, and Vitaly Bezuglyj. Daps3d: Domain adaptive projective segmentation of 3d lidar point clouds. Preprint , 2023. 7, 19, 20, 24, 26

Lingdong Kong, Youquan Liu, Runnan Chen, Yuexin Ma, Xinge Zhu, Yikang Li, Yuenan Hou, Yu Qiao, and Ziwei Liu. Rethinking range view representation for lidar segmentation. In IEEE/CVF International Conference on Computer Vision , pages 228-240, 2023. 3

Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robo3d: Towards robust and reliable 3d perception against corruptions. In IEEE/CVF International Conference on Computer Vision , pages 19994-20006, 2023. 7, 8, 19, 20, 27, 30

Lingdong Kong, Niamul Quader, and Venice Erin Liong. Conda: Unsupervised domain adaptation for lidar segmentation via regularized domain concatenation. In IEEE International Conference on Robotics and Automation , pages 9338-9345, 2023. 3

Lingdong Kong, Jiawei Ren, Liang Pan, and Ziwei Liu. Lasermix for semi-supervised lidar semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2170521715, 2023. 1, 3, 7, 8, 26

Lingdong Kong, Jiawei Ren, Liang Pan, and Ziwei Liu. Semi-supervised lidar semantic segmentation with spatial consistency training. In International Conference on Learning Representations Workshop , 2023. 3

Shamit Lal, Mihir Prabhudesai, Ishita Mediratta, Adam W. Harley, and Katerina Fragkiadaki. Coconets: Continuous contrastive 3d scene representations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12487-12496, 2021. 3

Lanxiao Li and Michael Heizmann. A closer look at invariances in self-supervised pre-training for 3d vision. In European Conference on Computer Vision , pages 656-673, 2022. 3

Li Li, Hubert PH Shum, and Toby P. Breckon. Less is more: Reducing task and model complexity for 3d point cloud semantic segmentation. arXiv preprint arXiv:2303.11203 , 2023. 1, 3, 8

Rong Li, Raoul de Charette, and C. A. O. Anh-Quan. Coarse3d: Class-prototypes for contrastive learning in weakly-supervised 3d point cloud segmentation. In British Machine Vision Conference , 2022. 1, 3

Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling languageimage pre-training via masking. arXiv preprint arXiv:2212.00794 , 2022. 3

Venice Erin Liong, Thi Ngoc Tho Nguyen, Sergi Widjaja, Dhananjai Sharma, and Zhuang Jie Chong. Amvnet: Assertion-based multi-view fusion network for lidar semantic segmentation. arXiv preprint arXiv:2012.04934 , 2020. 3

Kangcheng Liu, Yuzhi Zhao, Qiang Nie, Zhi Gao, and Ben M. Chen. Weakly supervised 3d scene segmentation with region-level boundary awareness and instance discrimination. In European Conference on Computer Vision , pages 37-55, 2022. 3

Minghua Liu, Yin Zhou, Charles R. Qi, Boqing Gong, Hao Su, and Dragomir Anguelov. Less: Labelefficient semantic segmentation for lidar point clouds. In European Conference on Computer Vision , pages 70-89, 2022. 3

Yueh-Cheng Liu, Yu-Kai Huang, Hung-Yueh Chiang, Hung-Ting Su, Zhe-Yu Liu, Chin-Tang Chen, Ching-Yu Tseng, and Winston H. Hsu. Learning from 2d: Contrastive pixel-to-point knowledge transfer for 3d pretraining. arXiv preprint arXiv:2104.0468 , 2021. 7, 8, 9, 10, 27

Anas Mahmoud, Jordan SK Hu, Tianshu Kuai, Ali Harakeh, Liam Paull, and Steven L. Waslander. Selfsupervised image-to-point distillation via semantically tolerant contrastive loss. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023. 2, 3, 4, 7, 23

Björn Michele, Alexandre Boulch, Gilles Puy, Maxime Bucher, and Renaud Marlet. Generative zero-shot learning for semantic segmentation of 3d point clouds. In International Conference on 3D Vision , pages 992-1002, 2021. 3

Björn Michele, Alexandre Boulch, Gilles Puy, Tuan-Hung Vu, Renaud Marlet, and Nicolas Courty. Saluda: Surface-based automotive lidar unsupervised domain adaptation. arXiv preprint arXiv:2304.03251 , 2023. 3

Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss. Rangenet++: Fast and accurate lidar semantic segmentation. In IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 4213-4220, 2019. 3

Ishan Misra, C. Lawrence Zitnick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In European Conference on Computer Vision , pages 527-544, 2016. 3

Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision , pages 69-84, 2016. 3

Lucas Nunes, Rodrigo Marcuzzi, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. Segcontrast: 3d point cloud feature representation learning through self-supervised segment discrimination. IEEE Robotics and Automation Letters , 7:2116-2123, 2022. 3

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023. 1

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023. 3

Yancheng Pan, Biao Gao, Jilin Mei, Sibo Geng, Chengkun Li, and Huijing Zhao. Semanticposs: A point cloud dataset with large quantity of dynamic instances. In IEEE Intelligent Vehicles Symposium , pages 687-693, 2020. 7, 19, 20, 25, 26

Duo Peng, Yinjie Lei, Wen Li, Pingping Zhang, and Yulan Guo. Sparse-to-dense feature matching: Intra and inter domain cross-modal learning in domain adaptation for 3d semantic segmentation. In IEEE/CVF International Conference on Computer Vision , pages 7108-7117, 2021. 3

Omid Poursaeed, Tianxing Jiang, Han Qiao, Nayun Xu, and Vladimir G. Kim. Self-supervised learning of point clouds via orientation estimation. In International Conference on 3D Vision , pages 1018-1028, 2020. 3

Gilles Puy, Alexandre Boulch, and Renaud Marlet. Using a waffle iron for automotive point cloud semantic segmentation. In IEEE/CVF International Conference on Computer Vision , pages 3379-3389, 2023. 3, 8

Haibo Qiu, Baosheng Yu, and Dacheng Tao. Gfnet: Geometric flow network for 3d point cloud semantic segmentation. Transactions on Machine Learning Research , 2022. 3

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763, 2021. 3

Giulia Rizzoli, Francesco Barbato, and Pietro Zanuttigh. Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives. Technologies , 10(4), 2022. 1

Cristiano Saltori, Evgeny Krivosheev, Stéphane Lathuiliére, Nicu Sebe, Fabio Galasso, Giuseppe Fiameni, Elisa Ricci, and Fabio Poiesi. Gipso: Geometrically informed propagation for online adaptation in 3d lidar segmentation. In European Conference on Computer Vision , pages 567-585, 2022. 3, 7, 19, 20, 25, 26, 27, 29

Aditya Sanghi. Info3d: Representation learning on 3d objects using mutual information maximization and contrastive learning. In European Conference on Computer Vision , pages 626-642, 2020. 3

Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by reconstructing space. In Advances in Neural Information Processing Systems , volume 32, 2019. 3

Corentin Sautier, Gilles Puy, Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, and Renaud Marlet. Image-to-lidar self-supervised distillation for autonomous driving data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9891-9901, 2022. 2, 3, 7, 8, 9, 10, 21, 23, 24, 26, 27

Hanyu Shi, Jiacheng Wei, Ruibo Li, Fayao Liu, and Guosheng Lin. Weakly supervised segmentation on outdoor 4d point clouds with temporal matching and spatial graph propagation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11840-11849, 2022. 3

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Training compute-optimal large language models. arXiv preprint arXiv:1909.08053 , 2019. 1

Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2446-2454, 2020. 7, 9, 18, 19, 25, 26, 27, 29

Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point-voxel convolution. In European Conference on Computer Vision , pages 685-702, 2020. 3, 7, 8

Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In IEEE/CVF International Conference on Computer Vision , pages 6411-6420, 2019. 3

Bach Tran, Binh-Son Hua, Anh Tuan Tran, and Minh Hoai. Self-supervised learning with multi-view rendering for 3d point cloud analysis. In Asian Conference on Computer Vision , pages 3086-3103, 2022. 3

Larissa T. Triess, Mariella Dreissig, Christoph B. Rist, and J. Marius Zöllner. A survey on deep domain adaptation for lidar perception. In IEEE Intelligent Vehicles Symposium Workshop , pages 350-357, 2021. 1, 2

Larissa T Triess, David Peter, Christoph B Rist, and J Marius Zöllner. Scan-based semantic segmentation of lidar point clouds: An experimental study. In IEEE Intelligent Vehicles Symposium , pages 1116-1121, 2020. 3

Marc Uecker, Tobias Fleck, Marcel Pflugfelder, and J. Marius Zöllner. Analyzing deep learning representations of point clouds for real-time in-vehicle lidar perception. arXiv preprint arXiv:2210.14612 , 2022. 1, 3

Ozan Unal, Dengxin Dai, and Luc Van Gool. Scribble-supervised lidar semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2697-2707, 2022. 1, 3, 7, 18, 19, 25, 26

Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2495-2504, 2021. 4

Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J. Kusner. Unsupervised point cloud pre-training via occlusion completion. In IEEE/CVF International Conference on Computer Vision , pages 9782-9792, 2021. 3

Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for selfsupervised visual pre-training. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3024-3033, 2021. 3

Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284 , 2023. 3

Aoran Xiao, Jiaxing Huang, Dayan Guan, Fangneng Zhan, and Shijian Lu. Transfer learning from synthetic to real lidar point cloud for semantic segmentation. In AAAI Conference on Artificial Intelligence , pages 2795-2803, 2022. 7, 19, 20, 25, 26

Aoran Xiao, Jiaxing Huang, Dayan Guan, Xiaoqin Zhang, Shijian Lu, and Ling Shao. Unsupervised point cloud representation learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023. 2

Aoran Xiao, Jiaxing Huang, Weihao Xuan, Ruijie Ren, Kangcheng Liu, Dayan Guan, Abdulmotaleb El Saddik, Shijian Lu, and Eric Xing. 3d semantic segmentation in the wild: Learning generalized models for adverse-condition point clouds. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9382-9392, 2023. 7, 19, 20, 25, 26

Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European Conference on Computer Vision , pages 574-591, 2020. 3, 7

Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16684-16693, 2021. 3

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9653-9663, 2022. 3

Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. In European Conference on Computer Vision , pages 1-19, 2020. 3

Jianyun Xu, Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun, and Shiliang Pu. Rpvnet: A deep and efficient range-point-voxel fusion network for lidar point cloud segmentation. In IEEE/CVF International Conference on Computer Vision , pages 16024-16033, 2021. 3

Mingye Xu, Mutian Xu, Tong He, Wanli Ouyang, Yali Wang, Xiaoguang Han, and Yu Qiao. Mm-3dscene: 3d scene understanding by customizing masked modeling with informative-preserved reconstruction and self-distilled consistency. arXiv preprint arXiv:2212.09948 , 2022. 3

Ryosuke Yamada, Hirokatsu Kataoka, Naoya Chiba, Yukiyasu Domae, and Tetsuya Ogata. Point cloud pre-training with natural 3d structures. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21283-21293, 2022. 3

Siming Yan, Zhenpei Yang, Haoxiang Li, Li Guan, Hao Kang, Gang Hua, and Qixing Huang. Implicit autoencoder for point cloud self-supervised representation learning. arXiv preprint arXiv:2201.00785 , 2022. 3

Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. arXiv preprint arXiv:2303.08131 , 2023. 2, 3, 6, 9, 21, 22, 23, 31, 32, 33

Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In European Conference on Computer Vision , pages 649-666, 2016. 3

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022. 1

Tunhou Zhang, Mingyuan Ma, Feng Yan, Hai Li, and Yiran Chen. Pids: Joint point interaction-dimension search for 3d point cloud. In IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1298-1307, 2023. 3

Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, and Hassan Foroosh. Polarnet: An improved grid representation for online lidar point clouds semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9601-9610, 2020. 3, 8

Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. In IEEE/CVF International Conference on Computer Vision , pages 10252-10263, 2021. 3, 7

Junhao Zhao, Weijie Huang, Hai Wu, Chenglu Wen, Bo Yang, Yulan Guo, and Cheng Wang. Semanticflow: Semantic segmentation of sequential lidar point clouds from sparse frame annotations. IEEE Transactions on Geoscience and Remote Sensing , 61:1-11, 2023. 3

Yiming Zhao, Lin Bai, and Xinming Huang. Fidnet: Lidar point cloud semantic segmentation with fully interpolation decoding. In IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 4453-4458, 2021. 3

Zixiang Zhou, Yang Zhang, and Hassan Foroosh. Panoptic-polarnet: Proposal-free lidar point cloud panoptic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13194-13203, 2021. 3

Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9939-9948, 2021. 3

Zhuangwei Zhuang, Rong Li, Kui Jia, Qicheng Wang, Yuanqing Li, and Mingkui Tan. Perception-aware multi-sensor fusion for 3d lidar semantic segmentation. In IEEE/CVF International Conference on Computer Vision , pages 16280-16290, 2021. 3

Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, Nanyun Peng, Lijuan Wang, Yong Jae Lee, and Jianfeng Gao. Generalized decoding for pixel, image, and language. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023. 1, 2, 3, 6, 9, 21, 22, 23, 31, 32, 33

Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718 , 2023. 1, 2, 3, 6, 9, 21, 22, 23, 24, 25, 31, 32, 33

Appendix

In this appendix, we supplement the following materials to support the findings and observations in the main body of this paper:

Section A elaborates on additional implementation details to facilitate reproduction.

Section B provides the complete quantitative results of our experiments.

Section C includes more qualitative results to allow better visual comparisons.

Section D acknowledges the public resources used during the course of this work.

A Additional Implementation Detail

A.1 Datasets

In this work, we conduct extensive experiments from a wide range of point cloud datasets to verify the effectiveness of the proposed Seal framework. A summary of the detailed configurations and emphases of these datasets is shown in Table A.

1 nuScenes [26]: The nuScenes 4 dataset offers a substantial number of samples collected by the LiDAR, RADAR, camera, and IMU sensors from Boston and Singapore, allowing machine learning models to learn useful multi-modal features effectively. For the point cloud segmentation task, it consists of 1000 scenes of a total number of 1 . 1 B annotated points, collected by a Velodyne HDL32E LiDAR sensor. It also includes image data from six cameras, which are synchronized with the LiDAR sensor. In this work, we use the LiDAR point clouds and image data from nuScenes for model pretraining. We also conduct detailed fine-tuning experiments to validate the effectiveness of representation learning. More details of this dataset can be found at https://www.nuscenes.org/nuscenes .

2 SemanticKITTI [3]: The SemanticKITTI dataset is a comprehensive dataset designed for semantic scene understanding of LiDAR sequences. This dataset aims to advance the development of algorithms and models for autonomous driving and scene understanding using LiDAR data. It provides 22 densely labeled point cloud sequences that cover urban street scenes, which are captured by a Velodyne HDL-64E LiDAR sensor. In this work, we use the LiDAR point clouds from SemanticKITTI as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at http://semantic-kitti.org .

3 Waymo Open [88]: The Waymo Open dataset is a large-scale collection of real-world autonomous driving data. The 3D semantic segmentation subset contains 1150 scenes, split into 798 training, 202 validation, and 150 testing scenes. This subset contains 23691 training scans, 5976 validation scans, and 2982 testing scans, respectively, with semantic segmentation labels from 23 classes. The data are captured by five LiDAR sensors: one mid-range LiDAR sensor truncated to a maximum of 75 meters, and four short-range LiDAR sensors truncated to a maximum of 20 meters. In this work, we use the LiDAR point clouds from Waymo Open as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://waymo.com/open .

4 ScribbleKITTI [95]: The ScribbleKITTI dataset is a recent variant of the SemanticKITTI dataset, with weak supervisions annotated by line scribbles. It shares the exact same amount of training samples with SemanticKITTI, i.e. , 19130 scans collected by a Velodyne HDL64E LiDAR sensor, where the total number of valid semantic labels is 8 . 06% compared to the fully-supervised version. Annotating the LiDAR point cloud in such a way corresponds to roughly a 90% time-saving. In this work, we use the LiDAR point clouds from ScribbleKITTI as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://github.com/ouenal/scribblekitti .

5 RELLIS-3D [49]: The RELLIS-3D dataset is a multimodal dataset collected in an off-road environment from the Rellis Campus of Texas A&M University. It consists of 13556 LiDAR scans from 5 traversal sequences. The point-wise annotations are initialized by using the camera-LiDAR calibration to project the more than 6000 image annotations onto the point clouds. In this work, we use the LiDAR point clouds from RELLIS-3D as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at http://www.unmannedlab.org/research/RELLIS-3D .

6 SemanticPOSS [75]: The SemanticPOSS dataset is a relatively small-scale point cloud dataset with an emphasis on dynamic instances. It includes 2988 scans collected by a Hesai Pandora LiDAR sensor, which is a 40-channel LiDAR with 0 . 33 degree vertical resolution, a forward-facing color camera, 4 wide-angle mono cameras covering 360 degrees around the ego-car. The data in this dataset was collected from the campus of Peking University. In this work, we use the LiDAR point clouds from SemanticPOSS as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://www.poss.pku.edu.cn/semanticposs .

7 SemanticSTF [102]: The SemanticSTF dataset is a small-scale collection of 2076 scans, where the data are borrowed from the STF dataset [5]. The scans are collected by a Velodyne HDL64 S3D LiDAR sensor and covered various adverse weather conditions, including 694 snowy, 637 dense-foggy, 631 light-foggy, and 114 rainy scans. The whole dataset is split into three sets: 1326 scans for training, 250 scans for validation, and 500 scans for testing. All three splits have similar proportions of scans from different weather conditions. In this work, we use the LiDAR point clouds from SemanticSTF as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://github.com/xiaoaoran/SemanticSTF .

8 SynLiDAR [100]: The SynLiDAR dataset contains synthetic point clouds captured from constructed virtual scenes using the Unreal Engine 4 simulator. In total, this dataset contains 13 LiDAR point cloud sequences with 198396 scans. As stated, the virtual scenes in SynLiDAR are constituted by physically accurate object models that are produced by expert modelers with the 3D-Max software. In this work, we use the LiDAR point clouds from SynLiDAR as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://github.com/xiaoaoran/SynLiDAR .

9 Synth4D [82]: The Synth4D dataset includes two subsets with point clouds captured by simulated Velodyne LiDAR sensors using the CARLA simulator. We use the Synth4DnuScenes subset in our experiments. It is composed of around 20000 labeled point clouds captured by a virtual vehicle navigating in town, highway, rural area, and city. The label mappings are mapped to that of the nuScenes dataset. In this work, we use the LiDAR point clouds from Synth4D-nuScenes as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://github.com/ saltoricristiano/gipso-sfouda .

10 DAPS-3D [51]: The DAPS-3D dataset consists of two subsets: DAPS-1 and DAPS-2; while the former is a semi-synthetic one with a larger scale, the latter is recorded during a real field trip of the cleaning robot to the territory of the VDNH Park in Moscow. Both subsets are with scans collected by a 64-line Ouster OS0 LiDAR sensor. We use the DAPS-1 subset in our experiments, which contains 11 LiDAR sequences with more than 23000 labeled point clouds. In this work, we use the LiDAR point clouds from DAPS-1 as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://github.com/subake/DAPS3D .

11 nuScenes-C [53]: The nuScenes-C dataset is one of the corruption sets in the Robo3D benchmark, which is a comprehensive benchmark heading toward probing the robustness of 3D detectors and segmentors under out-of-distribution scenarios against natural corruptions that occur in real-world environments. A total number of eight corruption types, stemming from severe weather conditions, external disturbances, and internal sensor failure, are considered, including 'fog', 'wet ground', 'snow', 'motion blur', 'beam missing', 'crosstalk', 'incomplete echo', and 'cross-sensor' scenarios. These corruptions are simulated with rules constrained by physical principles or engineering experiences. In this work, we use the LiDAR point clouds from nuScenes-C as a downstream task to validate the robustness of

Method

Front Left

Front

Front Right

SLIC

[1]

SAM

[50]

X-Decoder

[122]

OpenSeeD

[111]

SEEM

[123]

1600

1400

1400

1400

1200

1200

1200

1000

1000

1000

800

800

800

600

600

600

400

400

400

200

200

200

0

80

90

100

110

120

130

140

90

100

110

120

130

140

0

80

90

100

110

120

130

140

800

700

800

700

600

600

500

600

500

400

400

300

400

300

200

200

200

100

100

50

100

150

200

250

50

100

150

200

250

50

100

150

200

250

4000

3500

0000

3000

2500

2500

2000

3000

2000

1500

1500

2000

1000

1000

1000

500

500

20

15

20

25

10

15

20

2000

3000

1750

3500

2500

1500

3000

2000

1250

2500

1500

1000

2000

750

1500

1000

500

1000

00S

250

500

10

20

30

40

0

10

20

30

40

50

10

20

30

40

50

1600

3000

4000

1400

2500

3000

1200

2000

1000

2000

800

1500

600

1000

1000

400

500

200

10

20

30

40

50

0

10

20

30

40

50

10

20

30

40

pertaining methods under out-of-distribution scenarios. More details of this dataset can be found at https://github.com/ldkong1205/Robo3D .

A.2 Vision Foundation Models

In this work, we conduct comprehensive experiments on analyzing the effects brought by different vision foundation models (VFMs), compared to the traditional SLIC [1] method. Some statistical analyses of these different visual partition methods are shown in Table B and Table C.

SLIC [1] (traditional method): The SLIC model, which stands for 'simple linear iterative clustering', is a popular choice for visual partitions of RGB images. It adapts a k-means clustering approach to generate superpixels, in an efficient manner, and offers good unsupervised partition abilities for many downstream tasks. The pursuit of adherence to boundaries and computational and memory efficiency allows SLIC to perform well on different image collections. In this work, we follow SLidR [85] and use SLIC to generate

Method

Back Left

Back

Back Right

SLIC

[1]

SAM

[50]

X-Decoder

[122]

OpenSeeD

[111]

SEEM

[123]

1600

1750

1400

2500

1500

1200

2000

1250

1000

1000

1500

800

600

750

1000

400

500

500

200

250

0

70

80

90

100

110

120

130

140

90

100

110

120

130

140

0

70

80

90

100

110

120

130

140

700

700

800

600

600

500

600

500

400

400

300

400

300

200

200

200

100

100

100

150

200

250

50

100

150

200

25(

50

100

150

200

250

4000

4000

5000

4000

3000

3000

3000

2000

2000

2000

1000

1000

1000

20

25

10

20

15

20

3500

4000

2500

3500

3000

2500

2000

3000

2500

2000

1500

2000

1500

1000

1500

1000

1000

500

500

500

20

80

10

20

OE

40

10

20

25

35

3000

3000

2000

2500

2500

2000

1500

2000

1500

1000

1500

1000

1000

500

500

500

10

20

30

40

50

0

10

20

30

40

D

10

20

OE

40

superpixels, with a fixed quota of 150 superpixels per image, and compare our framework with previous ones using SLIC superpixels. More details of this model can be found at https://github.com/valeoai/SLidR .

SAM [50]: The Segment Anything Model (SAM) is a recent breakthrough towards zero-shot transferable visual understanding across a wide range of tasks. This model is trained on SA1B, a large-scale dataset with over 1 billion masks on 11 Mlicensed and privacy-respecting images. As a result, SAM is able to segment images, with either point, box, or mask prompts, across different domains and data distributions. In this work, we use a fixed SAM model with the ViT-H backbone (termed as ViT-H-SAM model) to generate superpixels. We use this pretrained model directly without any further fine-tuning. More details of this model can be found at https://github.com/facebookresearch/segment-anything .

X-Decoder [122]: The X-Decoder model is a generalized decoding framework that can predict pixel-level segmentation and language tokens seamlessly. This model is pretrained on three types of data, including panoptic segmentation, image-text pairs, and referring segmentation. For the panoptic segmentation task, the model is trained on COCO2017,

which includes around 104 k images for model training. In this work, we use a fixed XDecoder model termed BestSeg-Tiny to generate superpixels. We use this pretrained model directly without any further fine-tuning. More details of this model can be found at https://github.com/microsoft/X-Decoder .

OpenSeeD [111]: The OpenSeeD model is designed for open-vocabulary segmentation and detection, which jointly learns from different segmentation and detection datasets. This model consists of an image encoder, a text encoder, and a decoder with foreground, background, and conditioned mask decoding capability. The model is trained on COCO2017 and Objects365, under the tasks of panoptic segmentation and object detection, respectively. In this work, we use a fixed OpenSeeD model termed openseed-swint-lang to generate superpixels. We use this pretrained model directly without any further fine-tuning. More details of this model can be found at https://github.com/IDEA-Research/OpenSeeD .

SEEM [123]: The SEEM model contributes a new universal interactive interface for image segmentation, where 'SEEM' stands for 'segment everything everywhere with multi-modal prompts all at once'. The newly designed prompting scheme can encode various user intents into prompts in a joint visual-semantic space, which possesses properties of versatility, compositionality, interactivity, and semantic awareness. As such, this model is able to generalize well to different image datasets, under a zero-shot transfer manner. Similar to X-Decoder, SEEM is trained on COCO2017, with a total number of 107 k images used during model training. In this work, we use a fixed SEEM model termed SEEM-oq101 to generate superpixels. We use this pretrained model directly without any further finetuning. More details of this model can be found at https://github.com/UX-Decoder/ Segment-Everything-Everywhere-All-At-Once .

The quality of superpixels directly affects the performance of self-supervised representation learning. Different from the previous paradigm [85, 66], our Seal framework resorts to the recent VFMs for generating superpixels. Compared to the traditional SLIC method, these VFMs are able to generate semantically-aware superpixels that represent coherent semantic meanings of objects and backgrounds around the ego-vehicle.

As been verified in our experiments, these semantic superpixels have the ability to ease the 'oversegment' problem in current self-supervised learning frameworks [85, 66] and further improve the performance for both linear probing and downstream fine-tuning.

The histograms shown in Table B and Table C verify that the number of superpixels per image of VFMs is much smaller than that of SLIC. This brings two notable advantages: i) Since semantically similar objects and backgrounds are grouped together in semantic superpixels, the 'self-conflict' problem in existing approaches is largely mitigated, which directly boosts the quality of representation learning. ii) Since the embedding length D of the superpixel embedding features Q ∈ R M × D and superpoint embedding features K ∈ R M × D directly relates to computation overhead, a reduction ( e.g. , around 150 superpixels per image in SLIC [1] and around 25 superpixels per image in XDecoder [122], OpenSeeD [111], and SEEM [123]) on D would allow us to train the segmentation models in a more efficient manner.

Some typical examples of our generated superpixels and their corresponding superpoints are shown in Fig. A, Fig. B, Fig. C, and Fig. D. As will be shown in Section B, our use of semantic superpixels generated by VFMs brings not only performance gains but also a much faster convergence rate during the model pretraining stage.

A.3 Implementation Detail

A.3.1 Data Split

For model pertaining , we follow the SLidR protocol [85] in data splitting. Specifically, the nuScenes [26] dataset consists of 700 training scenes in total, 100 of which are kept aside, which constitute the SLidR mini-val split. All models are pretrained using all the scans from the 600 remaining training scenes. The 100 scans in the mini-val split are used to find the best possible hyperparameters. The trained models are then validated on the official nuScenes validation set, without any kind of test-time augmentation or model ensemble. This is to ensure a fair comparison with previous works and also in line with the practical requirements.

FRONT_LEFT

FRONT

FRONT_RIGHT

BACK_LEFT

BACK

BACK_RIGHT

LIDAR (RAWUNLABELEDPOINT)

FRONT_LEFT

FRONT

FRONT_RIGHT

BACK_LEFT

BACK

BACK_RIGHT

LIDAR (SEMANTICSUPERPOINT)

FRONT_LEFT

FRONT

FRONT_RIGHT

BACK_LEFT

BACK

BACK_RIGHT

LIDAR(RAWUNLABELEDPOINT)

FRONT_LEFT

FRONT

FRONT_RIGHT

BACK_LEFT

BACK

BACK_RIGHT

LIDAR(SEMANTIC SUPERPOINT)

For linear probing , the pretrained 3D network F θ p is frozen with a trainable point-wise linear classification head which is trained for 50 epochs on an A100 GPU with a learning rate of 0 . 05 , and batch size is 16 on the nuScenes train set for all methods.

For downstream fine-tuning tasks, we stick with the common practice in SLidR [85] whenever possible. The detailed data split strategies are summarized as follows.

For fine-tuning on nuScenes [26], we follow the SLidR protocol to split the train set of nuScenes to generate 1% , 5% , 10% , 25% , and 100% annotated scans for the training subset.

For fine-tuning on DAPS-3D [51], we take sequences '38-18_7_72_90' as the training set and '38-18_7_72_90', '42-48_10_78_90', and '44-18_11_15_32' as the validation set.

FRONT_LEFT

FRONT

FRONT_RIGHT

BACK_LEFT

BACK

BACKRIGHT

LIDAR(RAWUNLABELEDPOINT)

A

FRONT_LEFT

FRONT

FRONT_RIGHT

BACK_LEFT

BACK

BACK_RIGHT

LIDAR (SEMANTICSUPERPOINT)

FRONT_LEFT

FRONT

FRONT_RIGHT

BACK_LEFT

BACK

BACK_RIGHT

LIDAR (RAWUNLABELEDPOINT)

FRONT_LEFT

FRONT

FRONT_RIGHT

BACK_LEFT

BACK

BACK_RIGHT

LIDAR(SEMANTIC SUPERPOINT)

For fine-tuning on SynLiDAR [100], we use the sub-set which is a uniformly downsampled collection from the whole dataset.

For fine-tuning on SemanticPOSS [75], we use sequences 00 and 01 as half of the annotated training scans and use sequences 00 to 05 , except 02 for validation to create full of the annotated training samples.

For fine-tuning on SemanticKITTI [3], WaymoOpen [88], ScribbleKITTI [95], RELLIS3D [49], SemanticSTF [102], and Synth4D [82], we follow the SLidR protocol to create 1% , 10% , half , or full split of the annotated training scans, e.g. , one scan is taken every 100 frame from the training set to get 1% of the labeled training samples. Notably, the point

cloud segmentation performance in terms of IoU is reported on the official validation sets for all the above-mentioned datasets.

A.3.2 Experimental Setup

In our experiments, we fine-tune the entire 3D network on the semantic segmentation task using a linear combination of the cross-entropy loss and the Lovász-Softmax loss [4] as training objectives on a single A100 GPU. For the few-shot semantic segmentation tasks, the 3D networks are fine-tuned for 100 epochs with a batch size of 10 for the SemanticKITTI [3], Waymo Open [88], ScribbleKITTI [95], RELLIS-3D [49], SemanticSTF [102], SemanticPOSS [75], DAPS-3D [51], SynLiDAR [100], and Synth4D [82] datasets.

For the nuScenes [26] dataset, we fine-tune the 3D network for 100 epochs with a batch size of 16 while training on the 1% annotated scans. The 3D network train on the other portions of nuScenes is fine-tuned for 50 epochs with a batch size of 16 . We adopt different learning rates on the 3D backbone F θ p and the classification head, except for the case that F θ p is randomly initialized. The learning rate of F θ p is set as 0 . 05 and the learning rate of the classification head is set as 2 . 0 , respectively, for all the above-mentioned datasets except nuScenes. On the nuScenes dataset, the learning rate of F θ p is set as 0 . 02 .

We train our framework using the SGD optimizer with a momentum of 0 . 9 , a weight decay of 0 . 0001 , and a dampening ratio of 0 . 1 . The cosine annealing learning rate strategy is adopted which decreases the learning rate from its initial value to zero at the end of the training.

A.3.3 Data Augmentation

For the model pretraining , we apply two sets of data augmentations on the point cloud and the multi-view image, respectively, and update the point-pixel correspondences after each augmentation by following SLidR [85].

Regarding the point cloud, we adopt a random rotation around the z -axis and flip the x -axis or y -axis with a 50% probability. Besides, we randomly drop the cuboid where the length of each side covers no more than 10% range of point coordinates on the corresponding axis, and the cuboid center is located on a randomly chosen point in the point cloud. We also ensure that the dropped cuboid retains at least 1024 pairs of points and pixels; otherwise, we select another new cuboid instead. For the temporal frame, we apply the same data augmentation to it.

Regarding the multi-view image, we apply a horizontal flip with a 50% probability and a cropped resize which reshapes the image to 416 × 224 . Before resizing, the random crop fills at least 30% of the available image space with a random aspect ratio between 14 : 9 and 17 : 9 . If this random cropping does not preserve at least 1024 or 75% of the pixel-point pairs, a different crop is chosen.

For the downstream fine-tuning tasks, we apply a random rotation around the z -axis and flip the x -axis or y -axis with a 50% probability for all the points in the point cloud. If the results are mentioned with LaserMix [55] augmentation, we augment the point clouds via the LaserMix before the above data augmentations. We report both results in the main paper so that we can fairly compare the point cloud segmentation performance with previous works.

A.3.4 Model Configuration

For the model pretraining , the 3D backbone F θ p is a Minkowski U-Net [19] with 3 × 3 × 3 kernels; while the 2D image encoder G θ i is a ResNet-50 [33] initialized with 2D self-supervised pretrained model of MoCoV2 [12]. These configurations are kept the same as SLidR [85]. The channel dimension of G θ i head and F θ p head is set to 64 . For the linear probing task, we adopt a linear classification head, as mentioned in previous sections. For the downstream fine-tuning tasks, the same 3D backbone F θ p , i.e. , the Minkowski U-Net [19] with 3 × 3 × 3 kernels, is used.

SLidR

8

Seal(Ours)

6

5

4

3

2

5,000

10k

15k

20k

25k

30k

35k

B Additional Quantitative Result

B.1 Self-Supervised Learning

Wereport the complete results ( i.e. , the class-wise IoU scores) for the linear probing and downstream fine-tuning tasks shown in the main paper. We report the official results from previous works whenever possible. We also report our reproduced results for random initialization, PPKT [65], and SLidR [85]. Specifically, the complete results on the nuScenes [26], SemanticKITTI [3], Waymo Open [88], and Synth4D [82] datasets are shown in Table D, Table E, Table F, and Table G, respectively. We observe that Seal constantly outperforms prior methods for most semantic classes on all datasets.

We also compare the convergence rate of Seal with SLidR [85] in Fig. E. As can be seen, our framework is able to converge faster with the use of semantic superpixels, where these higher qualitative contrastive samples gradually form a much more coherent optimization landscape.

B.2 Robustness Probing

We report the complete results of the robustness evaluation experiments in the main paper, including the per-corruption CE scores and the per-corruption RR scores. Specifically, the complete results on the nuScenes-C [53] dataset are shown in Table H and Table I. We observe that Seal is superior to previous methods in terms of both CE and RR metrics.

C Additional Qualitative Result

C.1 Cosine Similarity

We provide more examples for the cosine similarity study in Fig. F, Fig. G, and Fig. H.

C.2 Downstream Tasks

We provide more qualitative results for downstream fine-tuning tasks in Fig. I, Fig. J, and Fig. K.

Query: "car" (near, left)

Query: “car"(near, middle)

Query: "car" (near, right)

SLIC

SAM

X-Decoder

OpenSeeD

SEEM

Query:"car" (near, left)

Query: “car"(near, middle)

Query: "car"(near, right)

SLIC

SAM

X-Decoder

OpenSeeD

SEEM

Query:"car"(near, left)

Query:"car"(near, middle)

Query: "vegetation"(near, right)

SLIC

SAM

X-Decoder

OpenSeeD

SEEM

Prediction

Error Map

Random

SLidR

Seal (Ours)

Random

SLidR

Seal (Ours)

Prediction

ErrorMap

Random

SLidR

Seal (Ours)

Random

SLidR

Seal (Ours)

Prediction

Error Map

Random

SLidR

Seal (Ours)

Random

SLidR

Seal (Ours)

D Public Resources Used

We acknowledge the use of the following public resources, during the course of this work:

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study represents the pioneering effort to utilize 2D vision foundation models for self-supervised representation learning on large-scale 3D point clouds. They introduce Seal, an innovative framework specifically designed to extract informative features from sequences of automotive point clouds. With its emphasis on scalability, consistency, and generalizability, Seal effectively captures semantic-aware spatial and temporal consistency, enabling the generation of highly valuable insights. Finally, they clearly demonstrate their superiority over previous state-of-the-art (SoTA) methods in both linear probing and fine-tuning for various downstream tasks.\n\nSTRENGTHS:\n1. The paper is well-written and easily comprehensible.\n2. To my knowledge, this is the first attempt to utilize large-scale vision models for aiding 3D point cloud segmentation. The authors have also put in tremendous effort, completing this work within just a short span of one to two months.\n3. Conducted extensive experiments and achieved satisfactory results on various segmentation datasets.\n\nWEAKNESSES:\n1. Strictly speaking, the approach in this paper is not considered unsupervised pretraining as it utilizes large models that rely on additional data. Most methods in Table 1, however, are unsupervised pretraining. Nevertheless, this distinction is not crucial, as the primary concern is to improve model performance, which is achieved through effective methods.\n2. I believe that the approach presented in the paper has not fully distilled the knowledge of large 2D models. When using the complete dataset for both pretraining and fine-tuning, the information gain from distilling large models is not significant. Why do I say this? In the method employed in this paper, the large models primarily provide semantic supervision signals. If full semantic labels are given during fine-tuning, the information gain is limited. Although Table 2 shows significant improvements for some full dataset cases, this is mainly due to the use of semantic features in distillation rather than simple semantic pseudo-labels. The advantage of distilling large 2D models is that I can use more image-lidar pairs during pretraining. I believe that if additional image-lidar pairs are used, such as on the nuscenes dataset, there will be further improvements. I also hope to see similar experiments, such as pretraining with ten times the number of image-lidar pairs compared to nuscenes.\n3. The method proposed in this paper is expected to be very general, so it should yield better results for other 3D perception tasks (e.g., 3D object detection).\n\nQUESTIONS:\nThe questions are all in the weaknesses section.\n\nLIMITATIONS:\nLimitations are all in the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a novel framework named Seal. The Seal distills VFMs into point clouds, enabling achieves efficient segmentation of various automotive point cloud sequences without requiring extensive annotation during the pre-training stage. It exhibits excellent performance across multiple datasets.\n\nSTRENGTHS:\nThe Seal distills the feature extraction capability of visual VFMs to point clouds. This paper promotes cross-modal representation learning through the proposed spatiotemporal consistency compared to previous works. The results obtained by The Seal on 11 different point cloud datasets demonstrate its effectiveness and superiority, highlighting its significant potential for 3D feature learning.\n\nWEAKNESSES:\nThe innovative contributions of the paper can be summarized into two points compared to previous works: 1. Distilling the representation learning capability from VFMs into point cloud data processing. 2. Introducing semantic superpoint temporal consistency to promote cross-modal learning. However, it is noted that the level of innovation in the paper might not be particularly strong.\n\nQUESTIONS:\n1. I have some questions regarding the semantic superpoint temporal consistency. Firstly, according to Figure 4, the segmentation results provided by VFMs are at the instance level, while this paper aims to obtain a pretraining model for semantic segmentation tasks. According to Equation 2, this paper treats instances of the same object as positive samples and the rest as negative samples. Therefore, there is a high possibility that negative samples may include instances of the same class. This could potentially hurt the segmentation task. On the other hand, how is the correspondence between moving objects in point cloud data at different time steps obtained?\n2. The paper mentions the ability to achieve good performance even in cases where camera and LiDAR calibration information is unavailable (on page 6 line 199). However, without approximate intrinsic and extrinsic parameters, establishing the 2D-3D spatial correspondence would be challenging. Therefore, I am curious about the performance in such scenarios. However, it appears that the paper does not provide specific experiments or results for these particular cases.\n\nLIMITATIONS:\nThe statement in the limitation section that the assumption of obtaining calibrated and synchronized data between LiDAR and cameras is overly idealistic contradicts the description in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework, called *Seal*, that leverages VFMs for self-supervised representation learning on automotive point cloud sequences. The main idea is to leverage the 2D-3D correspondence between LiDAR and camera sensors and construct high-quality contrastive samples for cross-modal representation learning. The paper also proposes a superpoint temporal consistency regularization to enforce geometric stability across different timestamps. The paper evaluates the proposed framework on 11 different point cloud datasets and shows that it outperforms previous state-of-the-art methods in both linear probing and few-shot fine-tuning settings.\n\nSTRENGTHS:\n**Originality:**\n - the **first** work utilizing 2D VFMs for self-supervised representation learning on large-scale 3D point clouds\n - a novel VFM-assisted contrastive learning objective that transfers the knowledge from the pretrained 2D to the 3D network at the semantic superpixel level\n - a superpoint temporal consistency regularization\n\n**Quality:**\n - well-written and organized, with clear motivation, problem formulation, related work, methodology, experiments, conclusion and discussion & limitations\n - extensive experiments on 11 different point cloud datasets and sufficient ablation studies\n - sufficient mathematical formulations and derivations to support the proposed methods\n - well-written and sufficient Supplementary Material with demo visualizations\n\n**Clarity:**\n - clear and easy to follow\n - uses consistent notations and symbols throughout the text and equations\n - helpful figures and tables to illustrate the main ideas and results\n\n**Significance:**\n - significant for both research and practice in 3D perception and representation learning\n - addresses an important problem of segmenting diverse automotive point clouds in a scalable, consistent, and generalizable manner\n\nWEAKNESSES:\nThe datasets conducted on are all **outdoor** point cloud dataset for autonomous driving. Unless the authors conduct more experiments on **indoor** point cloud datasets, they'd better just state that \"Segment Any **Automotive** Point Cloud Sequences ...\" in their title.\n\nMinior Issue:\n- Undefined variable $L^{tmp}$ in Equ. (3)\n\nQUESTIONS:\nIn Table 3 (Full), it seems *Seal* performs not well in `Cross`, `Echo` and `Sensor` (especially `Sensor`, 39.85 vs 49.21). Could you please provide some possible reasons about that?\n\nLIMITATIONS:\nAuthors should be rewarded that the limitations are explicitly mentioned in *Discussion & Limitation*.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript presents Seal, a framework that leverages VFMs (Visual Foundation Models) to segment diverse point cloud sequences in autonomous driving scenarios. The proposed approach employs VFMs to initially segment superpixels in 2D camera images and subsequently projects them to 3D superpoints. Then, it incorporates two specific pre-training schemes: spatial contrastive learning, which involves training cross-modally from pre-trained 2D features to 3D representations, and temporal consistency regularization, which ensures the consistency of 3D point features across different timestamps. Extensive experimentation validates the effectiveness and robustness of the proposed framework.\n\nSTRENGTHS:\n1. The manuscript exhibits clear and concise writing, making it easy to comprehend. The mathematical explanations and figures are presented with high clarity, facilitating understanding.\n2. Through extensive experiments, the study demonstrates the effectiveness and robustness of the proposed method, establishing its reliability.\n3. The main contribution of the manuscript is the development of the framework that can serve as a valuable resource for future reference and practical implementation.\n\nWEAKNESSES:\nNovelty: The novelty of the manuscript may be somewhat limited. While the authors claim that the main novelty lies in leveraging VFM models for superpixel set segmentation and introducing two specific pretraining schemes, it is worth noting that the use of VFM models for segmentation is already a widely discussed topic and is short of novelty. Additionally, the proposed pretraining schemes closely follow common cross-modal contrastive learning approaches. This may give the impression that the paper is riding on the popularity of VFM models without truly exploring their potential for feature-level distillation.\n\nQUESTIONS:\n1. What is the parameterization of a superpixel? If a superpixel is selected as a representative pixel from an area with the same class in VFM outputs, how is the pixel specifically chosen?\n2. What is the motivation behind enforcing the convergence of 3D point features to a single mean representation? In the 2D scenario, it is common for features at different pixels of the same object to be distinct, as they capture diverse parts and information about the object. Shouldn't this principle apply to the 3D case?\n\nLIMITATIONS:\nBesides the limitations the authors discuss in the manuscript, I've seen no additional specific limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}