Segment Any Point Cloud Sequences by Distilling Vision Foundation Models
Authors: Youquan Liu
Lingdong Kong
Jun CEN
Runnan Chen
Wenwei Zhang
Liang Pan
Kai Chen
Ziwei Liu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: autonomous driving
point cloud segmentation
self-supervised learning
3D scene understanding
TL;DR: Seal is a spatial and temporal consistent framework that leverages vision foundation models for self-supervised learning on large-scale point clouds.

Abstract: Recent advancements in vision foundation models (VFMs) have opened up new possibilities for versatile and efficient visual perception. In this work, we introduce Seal, a novel framework that harnesses VFMs for segmenting diverse automotive point cloud sequences. Seal exhibits three appealing properties: i) Scalability: VFMs are directly distilled into point clouds, obviating the need for annotations in either 2D or 3D during pretraining. ii) Consistency: Spatial and temporal relationships are enforced at both the camera-to-LiDAR and point-to-segment regularization stages, facilitating cross-modal representation learning. iii) Generalizability: Seal enables knowledge transfer in an off-the-shelf manner to downstream tasks involving diverse point clouds, including those from real/synthetic, low/high-resolution, large/small-scale, and clean/corrupted datasets. Extensive experiments conducted on eleven different point cloud datasets showcase the effectiveness and superiority of Seal. Notably, Seal achieves a remarkable 45.0% mIoU on nuScenes after linear probing, surpassing random initialization by 36.9% mIoU and outperforming prior arts by 6.1% mIoU. Moreover, Seal demonstrates significant performance gains over existing methods across 20 different few-shot fine-tuning tasks on all eleven tested point cloud datasets. The code is available at this link.

Segment Any Point Cloud Sequences by Distilling Vision Foundation Models
Youquan Liul:* Lingdong Kong .2,- Jun Cen
Runnan Chen
Wenwei Zhang 1,5
Liang Pan Kai Chen Ziwei Liu
Shanghai AI Laboratory Natioona University of Singapore
The Hong Kong University of Science and Technology "The University Hong Kong
S--Lab, Nanyang Technological University
{ (liuyouquan, kontelannnooniiinn konglingdong zhangwenwei, chenkai Hapjaab. org. cn cenaa&connect. ust.hk (liang, pan, pan,iiwei. -ntuuutuuuu.u.. edu sg
Abstract
Recent advancements in vision foundation models (VFMs) have opened up new possibilities for versatile and efficient visual perception. In this work, we introduce Seal, a novel framework that harnesses VFMs for segmenting diverse automotive point cloud sequences. Seal exhibits three appealing properties: i) Scalability: VFMs are directly distilled into point clouds, obviating the need for annotations in either 2D or 3D during pretraining. ii) Consistency: Spatial and temporal relationships are enforeed at both the samera-to-LiDAE and peintto-segmmet regularization stages, facilitating cross-modal representation learning. iii) General- izability: Seal enables knowledge transfer in off-the-shelf manner downstream tasks involving diverse point clouds, including those from real/synthetic, low/high- resolution, large/small-scale, and clean/corrupted datasets. Extensive experiments conducted on eleven different point cloud datasets showcase the effectiveness and superiority of Seal. Notably, Seal achieves remarkable 45.0% mloU nuScenes after linear probing, surpassing random initialization by 36.9% mloU and out- performing prior arts by 6.1% mloU. Moreover, Seal demonstrates significant performance gains over existing methods across 20 different few-shot fine-tuning tasks on all eleven tested point cloud datasets. The code is available at this link?.
Introduction
Inspired by the achievements of large language models (LLMs) [87, 35, 113, 73, 18], wave of vision foundation models (VFMs), such as SAM [50], X-Decoder [122], and SEEM [123], has emerged. These VFMs are revolutionizing the field of computer vision by facilitating the acquisition of pixel- level semantics with greater ease. However, limited studies have been conducted on developing VFMs for the 3D domain. To bridge this gap, it holds great promise explore the adaptation or extension of existing 2D VFMs for 3D perception tasks.
As important 3D perception task, accurately segmenting the surrounding points captured by onboard LiDAR sensors is crucial for ensuring the safe operation of autonomous vehicles [92, 94, 81]. However, existing point cloud segmentation models rely heavily on large annotated datasets for training, which poses challenges due to the labor-intensive nature of point cloud labeling [2, 95]. To address this issue, recent works have explored semi- [55, 59] weakly-supervised [95, 60] approaches to alleviate the annotation burden. While these approaches show promise, the trained point segmentors tend perform well only on data within their distribution, primarily due to
Youqqun and Lingdong contributed equally to this work. Ex Ziwei serves as the corresponding author. Guttlub Repo: bettps///gitae wwcpaauu...oo////eeeeeeeeeeeeeee connyooqquan/Segmeet -arttt-Ann--P-----iiaee Point-Cloud.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

2D-3D Correspondence
Sensor Data
SLIC
SAM
X-Decoder
SEEM
Pose
Camera
Pase
GT
Seal
Seal
Seal
Seal
LIDAR
Figure I: The proposed Seal distills semantic awareness on cameras views from VFMs to the point cloud via superpixel-driven contrastive learning. [1st row] Semantic superpixels generated by SLIC [1] and recent VFMs [50, 122, 123], where each color represents one segment. [2nd row] Semantic superpoints grouped by projecting superpixels 3D via camera-LiDAR correspondence. [3rd row] Visualizations of the linear probing results of our framework driven by SLIC and different VFMs.
significant configuration differences (such as beam number, camera angle, emit rate) among different sensors [101, 28, 92]. This limitation inevitably hampers the scalability of point cloud segmentation.
To address the aforementioned challenges, we aim develop a framework that can learn informative features while addressing the following objectives: i) Utilizing raw point clouds as input, thereby eliminating the need for semi or weak labels and reducing annotation costs. ii) Leveraging spatial and temporal cues inherent in driving scenes to enhance representation learning. iii) Ensuring generalizability to diverse downstream point clouds, beyond those used in the pretraining phase. Drawing inspiration from recent advanceeents in cross- moosl representation learning [50, 122, 111, 123] and building upon the success of VFMs [50, 122, 111, 123], we propose a methodology that distills semantically-rie knowledge from VFMs to support self-supervised representation learning on challenging automotive point clouds. Our core idea is to leverage the 2D-3D correspondence in- between LiDAR and camera sensors and construct high-quality contrastive samples for cross modal representation learning. As shown in Fig. 1, VFMs are capable of generating semantic superpixel groups from the camera views' and provide off-the-shelf semantic coherence for distinct objects and backgrounds in the 3D scene. Such consistency can be distilled from 2D to 3D and further yields promising performance on downstream tasks, which will revisit more formally in later sections.
Compared to previous framework [85, 66], our VFM-assisted contrastive learning has three notable advantages: i) The semantically-aware region partition mitigates the severe "self-conflict" problem in driving scene contrastive learning. üi) The high-quality contrastive samples gradually form a more coherent optimization landscape, yielding faster convergence rate than previous configurations. íi) huge reduction in the number superpixels also extenuates the overhead during pretraining.
Since a perfect calibration between LiDAR and cameras is often hard to meet, we further design superpoint temporal consistency regularization mitigate the case that there are potential errors from the sensor synchronization aspect. This objective directly resorts to the accurate geometric information from the point cloud and thus improves the overall resilience and reliability. As will be shown in the following sections, our framework (dubbed Seal) is capable of segmentation any point cloud sequences across a wide range of downstream task on different automotive point cloud datasets.
To summarize, this work has the following key contributions:
To the best of our knowledge, this study represents the first attempt at utilizing 2D vision foundation models for self supf-supe representation learning large-scale 3D point clouds.
We show the front view for simplicity; more common setups have surrounding views from multiple cameras.

---

We introduce Seal, scalable, consistent, and generalizable framework designed to capture semantic-aware spatial and temporal consistency, enabling the extraction of informative features from automotive point cloud sequences.
Our approach demonstrates clear superiority over previous state-of-the-art (SoTA) methods in both linear probing and fine-tuning for downstream tasks across different point cloud datasets with diverse data configurations.
2 Related Work
Vision Foundation Models. Recent excitement about building powerful visual perception systems
based on massive amounts of training data [80, 50] or advanced self-supervised learning techniques
[8, 74] is revolutionizing the community. The segment anything model (SAM) [50] sparks new trend of general-purpose image segmentation which exhibits promising zero-shot transfer capability on diverse downstream tasks. Concurrent works to SAM, such as X-Decoder [122], OpenSeeD [111], SegGPT [99], and SEEM [123], also shed lights on the direct use of VFMs for handling different kinds of image-related tasks. In this work, we extend this aspect by exploring the potential of VFMs for point cloud segmentation. We design a new framework taking into consideration the off-the-shelff semantic awareness of VFMs to construct better spatial and temporal cues for representation learning.
Point Cloud Segmentation. Densely perceiving the 3D surroundings important for autonomous vehicles [94, 41]. Various point cloud segmenters have been proposed, including methods based on raw points [42, 90, 43, 114, 78], range view [69, 106, 93, 118, 20, 16, 52], bird's eye view [115, 1 119, 9I. voxels [19, 120, 37, 36], and multi-view fusion [62, 89, 107, 121, 17, 79J. Despite the promising results achieved, existing 3D segmentation models rely on large sets annotated data for training, which hinders the scalability [28]. Recent efforts seek semi [55, 56, 59], weak [95, 40, 86, 63, 60],
and active [64, 44, 117] supervisions or domain adaptation techniques [48, 47, 54, 76, 82, 68] to ease the annotation cost. In this work, resort to self-supervisee learning by distilling foundation models using camera-to-LiDAe associations, where no annotation is required during the pretraining stage.
3D Representation Learning. Stemmed from the image vision community, most 3D self supervisdd learning methods focused on object-centric point clouds [84, 77, 83, 14, 97,45, 110, 29, 91] or indoor scenes [38, 15, 22, 108, 10, 39, 57, 58, 109] by either pretext task learning [21, 70, 71, 112, 30], contrastive learning 32, 12, 34, 27, 13, 46, 98, 104] or mask modeling [105, 31, 21J61J, where the scale and diversity are much lower than the outdoor driving scenes [67, 6]. PointContrast [103], DepthContrast [116], and SegContrast [72] are prior attempts aimed to establish contrastive objectives on point clouds. Recently, Sautier et al. [85] proposed the first 2D>to-33 representation distillation method called SLidR for cross-modal self-supervised learning on large-scale point clouds and exhibits promising performance. The follow-up work [66] further improves this pipeline with semantically tolerant contrastive constraint and a balss-balancing loss. Our framework also stems from the SLidR paradigm. Differently, we propose to leverage VFMs to establish the cross-modal contrastive objective which better tackles this challenging representation learning task. We also design superpoint temporal consistency regularization to further enhance feature learning.
3 Seal: A Scalable, Consistent, and Generalizable Framework
In this section, first revisit the 2D-to-3D representation distillation [85] preliminaries (Sec. 3.1). We then elaborate on the technical details of our framework, which include the VFM-assisted spatial contrastive learning (Sec. 3.2) and the superpoint temporal consistency regularization (Sec. 3.3).
3.1 Preliminaries
Given a point cloud P' = (p/, e/li = 1,.., N} consists of N points collected by LiDAR acqui- sition at timestamp t, where P, E R3 denotes the point coordinates and e, E F is the feature embedding (intensity, elongation, etc.), our goal is to transfer the knowledge from an image set I' (TT 1 V} captured by V synchronized cameras at f to point cloud Pt, where I E R3xHxW is single image with height H and width W. Prior works [85, 66] achieve this goal by first aggregating image regions that are similar in the RGB space into a superpixel set & {(s). .... Sm Hm L,..., M. via the unsupervised SLIC [1] algorithm. The corresponding
3

---

Time
Sensor Data
Super Pixel Point
2D 3D3D Net
Grouping
Contrastive Objective
RWSSC
O UIAR
Fg
Holp
2D-to-3D
Frame
Group
g
VFM
Camera
Go
Hor
Group
2D-to-3D
Frame 2
NYC20000
LDAR
Fg
Hap
Figure 2: Overview of the Seal framework. We generate, for each [LiDAR, camera) pair (P',** at timestamp & and another LiDAR frame pt-n at timestamp & + n, the semantic superpixel and superpoint by vision foundation models (VFMs). Two pertaining objectives are then formed, including spatial contrastive learning between paired LiDAR and camera features (Sec. 3.2) and temporal consistency regularization between point segments at different timestamps (Sec. 3.3).
superpoint set Φo {[o,,..., Jm Hm 1,... .M can be obtained by leveraging known sensor calibration parameters to establish correspondence between the points and image pixels. Specifically, since the LiDAR and cameras usually operate at different frequencies 26]. we transform each LiDAR point cloud p; = (I,, Mos 2.) at timestamp t to the pixel p. = (u, ex) E IR? in the image plane timestamp to via the coordinate transformation. The point-to-pixel mapping is as follows:
[ug, 02 1] xIx X
xT XTego
ego.. gloohal X glabale-ngo, xI ego,, + lidar x [K; Vis 21, 1
camerat-ggy Dgo
(1)
where symbol F denotes the camera intrinsic matrix. I raamerat-ogo cgo rege global gglobale- cgo, and I eego, liidar are the extrinsic matrices for the transformations of ego to camera at fer global to ego at le. ego to global at 4, and LiDAR to ego at 4, respectively.
3.2 Semantic Superpixel Spatial Consistency
Superpixel Generation. Prior works resort to SLIC [1] to group visually similar regions in the image into superpixels. This method, however, tends to over-segment semantically coherent areas (as shown in Fig. 1) and inevitably leads to several difficulties for contrastive learning. One of the main impediments is the so-called "self-conflict", where superpixels belonging to the same semantics become negative samples [96]. Although [66] proposed semantically-tolese
loss ease this problem, the lack of high-level semantic understanding still intensifies the implicit hardness-aware property of the contrastive loss. We tackle this challenge by generating semantic superpixels with VFMs. As shown in Fig. 1 and Fig. 4, these VFMs provide semantically-riet superpixels and yield much better representation learning effects in-between near and far points in the LiDAR point cloud.
VFM-Assisted Contrastive Learning. Let Foz, Nx+3L) + NNC be a 3D encoder with trainable parameters Op. that takes LiDAR point cloud as input and outputs a C-dimensional per-point feature. Let Go, H x3 + R # xO be an image encoder with parameters 6,, which is initialized from a set of 2D self-supervised pretrained parameters. The goal of our VFM- assisted contrastive learning is to transfer the knowledge from the pretrained 2D network to the 3D network via contrastive loss the semantic superpixel level. To compute this VFM-assisted contrastive loss, we first build trainable projection heads Hu,, and Hu, which map the 3D point features and 2D image features into the same D-dimensional embedding space. The point projection head Hwp B NCC + xDD is linear layer with l f--ormalization. The image projection head

---

Hw, R * is a convolution layer with a kernel size of 1, followed by a fixed bi- linear interpolation layer with ratio of 4 the spatial dimension, and outputs with fg-normalization
We distill the knowledge from the 2D network into the 3D network which is in favor of a solu- tion where a semantic superpoint feature has strong correlation with its corresponding semantic superpixel feature than any other features. Concretely, the superpixels 2g are used to group pixel em- bedding and point embedding features. Then, an average pooling function is applied point and pixel embedding features, to extract the superpixel embedding features Q E IR x and superpoint embedding features K E RD The
each grouped
contrastive loss is formulated as:
M Σ log
f=mm= L(Q.
Ejtt cl(q.,k)/77) + el(a.k)//T)
where (q:, kj) denotes the scalar product between super- point embedding features and superpixel embedding features to measure the similarity. 7 the temperature term.
Role in Our Framework. Our VFM -asssssited contrastive objective exhibits superiority over previous methods from three aspects: i) The semantically-rich superpixels provided by VFMs mitigate the "self-conflict" problem in existing ap- proaches. ii) As we will show in the following sections, the high-quality contrastive samples from VFMs gradually form a much more coherent optimization landscape and yield a faster convergence rate than the unsupervised superpixel gen- eration method. iii) Using superpixels generated by VFMs also helps our framework run faster than previous works, since the embedding length of Q and K has been reduced from a few hundred (SLIC (1] to dozens (ours).
42
43
q1
ki
k-
K
k2 tn
kg tn
kjtn
3.3 Semantic Superpoint Temporal Consistency
The assumption of having perfectly synchronized LiDAR and camera data might become too ideal and cannot always be fulfilled in actual deployment, which limits the scalability. In this work, we resort to accurate geometric information from point clouds to further relieve this constraint.
The positive feature corre- spondences in the contrastive learning objective in our contrastive learning framework. The circles and triangles represent the instance-level and the point-level features, respectively.
Figure 3:
Implicit Geometric Clustering. To group coarse instance segments in LiDAR frame, we first partition non-ground plane points G by eliminating the ground plane of a point cloud Pr at timestamp t in an unsupervised manner via RANSAC [25]. Then, we group G to yield set of Ms segments K (KG, .... Kha, } with the help of HDBSCAN [23]. To map different segment views at different timestamps, we transform those LiDAR frames across different timestamps to the global frame and aggregate them with concatenations. The aggregated point cloud is denoted as P = {Pi (P',...,?P) pr.n). Similarly, we generate non-ground plane ğ = {
; ...,, Getn } from P. via RANSAC [25]. In the same manner as the single scan, we group & to obtain Mr segments K = masks for all + 1 scans at n consecutive timestamps, i.e., K. == (KC , ,..,, [+,,), we maintain the point index mapping from the aggregated point cloud P to the n + 1 individual scans.
K.M }- To generate the segment
Superpoint Temporal Consistency. We leverage the clustered segments to compute the temporal consistency loss among related semantic superpoints. Here we assume n 1 (i.e. the next frame) without loss of generalizability. Specifically, given sampled temporal pair pe and pt++1 and their corresponding segments KC' and K we compute the point-wise wise features F E R NDD and fit+1 E N D from the point projection head Hup- As for the target embedding, we split the point features f and fi+ into Mj groups by segments Kct and Æt+1, Then, we apply an average pooling operation on fi+1 to get M2 target mean feature vectors F++ ( Ft+1 ,..., Ft+11, where Me &RIxD, Let the split point feature f be F F Fle, J. where Fh., E R and
5

---

SLIC
SAM
X-Decoder
OpenSeeD
SEEM
Query: 'car'
Query: "manmade"
Query: "truck"
Figure 4: The cosine similarity between a query point (denoted as the red dot) and the feature learned with SLIC [1] and different VFMs [50, 122, 111, 123]. The queried semantic classes from top to bottom examples are: "car", "manmade", and "truck". The color goes from violet to yellow denoting low and high similarity scores, respectively. Best viewed in color.
l: is the number of points in the corresponding segment. We compute the temporal consistency loss f--t+1 to minimize the differences between the point features in the current frame (timestamp !) and the corresponding segment mean features from the next frame (timestamp + 1) as follows:
(1 Ejet C (( /7)) + e(((t,t))))))
C
log
MR
Since the target embedding for all points within a segment in the current frame serves as the mean segment representation from the next frame, this loss will force points from segment to converge to mean representation while separating from other segments, implicitly clustering together points from the same instance. Fig. provides the positive feature correspondence in contrastive learning framework. Furthermore, we swap f when generating the target mean embedding features to form symmetric representation. In this way, the correspondence is encouraged from both + f 1 and ++1 + f, which leads the following optimization objective: Ltmp - + L
Point to Segment Regularization. To pull close the LiDAR points belonging to the same instance at timestamp I, we minimize the distance between the point feature F and the corresponding mean cluster feature C To implement this, we leverage max-pooling function to pool F according to the segments to obtain C = [C C3 -., where CLa RIID The point-to-segment regularization thus achieved via the following loss function:
Mr N
((cl.1to))77) E + te e((cttia)
C
log
M&N
where N2 represents the number of points within the corresponding segment. The final optimization objective to minimize the aforementioned semantic spatial consistency loss Lefm, temporal consistency loss Limp, and the point-to-segment regularization loss L22e.
Role in Our Framework. Our semantic superpoint temporal consistency resorts the accurate geometric information from the point cloud and exploits the different views of an instance across different timestamps to learn temporally consistent representation. Considering the worst-case scenario that the 2D-3D correspondence between the LiDAR and camera sensors becomes unreliable, this geometric constraint can still effectively mitigate the potential errors that occur in inaccurate cross- sensor calibration and synchronization. Besides, our point-to-ae to-segment regularization mechanism can serve to aggregate the spatial information thus contributing to the effect of better-distinguishing instances in the LiDAR-acquired scene, e.g., "car" and "truck".. As we will show in the following sections, our experimental results are able to verify the effectiveness and superiority of the proposed consistency regularization objectives, even under certain degrees of perturbation in tn-between sensors.

---

Table 1: Comparisons of different pretraining methods pretrained on nuScenes [26] and fine-tume line-tuned on nuScenes [7], SemanticKIITI [3], Waymo Open [88], and Synth44 [82]. LP denotes linear probing
with frozen backbones. Symbol i denotes fine-tuning with the LaserMix augmentation [55]. Symbol
t denotes fine-tuning with supe-suueee learning. All mloU scores are given in percentage (%)
nuScenes
KITTI
Waymo
Synth4D
Method & Year
LP
1%
10%
25%
Full
1%
1%
1%
Random
8.10
30.30
47.84
56.15
65.48
74.66
39.50
39.41
20:22
PointContrast [ECCVV20] [103]
21.90
32.50
41.10
DepthContrast [ICCV"21] [116]
22.10
31.70
11.50
PPKT [arXiv221] [65]
35.90
37.80
53.74
00:25
67.14
74.32
44.00
47.60
61.10
SLidR [CVPR:22] [85]
38.80
38.30
52.19
59.81
66.91
71..9
41.60
47.12
63.10
ST-SL.idR [CVPR [66]
40.48
40.75
54.69
60.75
67.70
75.14
44.72
44.93
Seal (Ours)
44.95
45.84
55.64
62.97
68.41
75.60
46.63
49.34
64.50
Seal (Ours)
48.41
57.84
65.52
7U.80
77.13
Seal (Ours)
49.53
58.64
66.78
72.31
78:28
4 Experiments
4.1 Settings
Data. verify the effectiveness of our approach on eleven different point cloud datasets. 'nuScenes
762. "SemanticKITTI [3], and Waymo Open [88] contain large-scale LiDAR scans collected from real-world driving scenes; while the former adopted Velodyne HDL32E, data from the latter two datasets are acquired by 64-beam LiDAR sensors. 'ScribbleKITTT [95] shares the data with [3] but are weakly annotated with line scribbles. *RELLIS-DD [49] is multimodal dataset collected in an off-road campus environment. "SemanticPOSS [75] is a small-scale set with an emphasis dynamic instances. "SemanticSTF [102] consist of LiDAR scans from adverse weather conditions. "SynLiDAR [100]. Synth44 [82], and DAPS-33 [51] are synthetic datasets collected from simulators. We also conduct extensive robustness evaluations on the 11 "uSScenesSC dataset proposed in the Robo3D benchmark [53], a comprehensive collection of eight out-of distirbiin corruptions that occur in driving scenarios. More details on these datasets are in the Appendix.
Implementation Details. We use MinkUNet [19] as the 3D backbone which takes cylindrical voxels of size 0.10m inputs. Similar to [85, 66]. our 2D backbone is ResNet-50 [33] pretrained with MoCoV2 [12]. We pretrain our segmentation network for 50 epochs on two GPUs with a batch size of 32, using SGD with momentum and a cosine annealing scheduler. For fine-tuning, we follow the exact same data split, augmentation, and evaluation protocol as SLidR [85] on nuScenes and SemanticKITTI, and adopt a similar procedure a combination of the cross-entropy loss and the Sovász-Softmax loss [4]. We compare the results of prior arts from their official reporting [85, 66). Since PPKT [65] and SLidR [85] only conducted experiments on nuScenes and SemanticK/ITI we reproduce their best-possible performance on the other nine datasets using public code. For additional details, please refer the Appendix.
other datasets. The training objective is to minimize
Metrics. We follow the conventional reporting of Intersection-over- Union (IoU) on each semantic class and the mean IoU (mIoU) across all classes. For robustness probing, we follow the Robo3D protocol [53] and report the mean Corruption Error (mCE) and mean Resilience Rate (mRR) scores calculated by using the MinkUNet1s (torchsparse) implemented by Tang et al. [89] as the baseline.
4.2 Comparative Study
Comparison to State-of-the-Arts. We compare Seal with random initialization and five existing pretraining approaches under both linear probing (LP) protocol and few-shot fine-tuning settings on nuScenes [26] in Table 1. We observe that the pretraining strategy can effectively improve the accuracy of downstream tasks, especially when the fine-tuning budget is very limited (e.g. 1%, 5%, and 10%). Our framework achieves 44. .55%%% mloU under the LP setup - 4.47% mloU lead over to the prior art ST-SLidR [66] and 6.15% mloU boost compared to the baseline SLidR [85]. What is more, Seal achieves the best scores so far across all downstream fine-tuning tasks, which verifies the superiority of VFM-assisted contrastive learning and spatial-tempora consistency regularization. We also show that recent out-of-context augmentation [55] could enhance the feature learning during fine-tuning, which establishes new state of the art the challenging nuScenes benchmark.
7

---

Table 2: Comparisons of different pretraining methods pretrained on muScenes [26] and tine-tuned on different downstream point cloud datasets. All mloU scores are given in percentage (%).
ScribbleKITTI
RELLIS-3D
SemanticPOSS Full
SemanticSTF Full
SynLiDAR
DAPS-3D
Method
1%
10%
1%
10%
Half
Half
1%
10%
Half
Full
Random
23.81
47.60
38.46
53.60
46.26
54.12
48.03
48.15
19.89
44.74
74.32
79.38
PPKT [65]
36.50
51.67
49.71
54.33
50.18
56.00
50.92
54.69
37.57
46.48
78.90
84.00
51.d [85]
39.60
50..45
49.75
54.57
51.56
55.96
52.01
54.35
42.05
47.84
81.00
85.40
Seal (Ours)
40.64
52.77
51.09
55.03
53.26
56.89
53.46
55.36
43.58
49.26
81.88
85.90
Table 3: Robustness evaluations under eight out-of-distribution corruptions in the nuScenes-C dataset
from the Robo3D benchmark [53]. All mCE, mRR, and mloU scores are given in percentage (%).
Initial
Backbone
mCE. L
mRR
Fog
Wet
Snow
Move
Beam
Cross
Echo
Sensor
PPKT [65]
MinkUNet
183.44
78.
30.65
35. 35.42
28. 12
29. 21
32.
19.32
28.01
20.71
SLidR [85]
MinkU/Net
179.38
77.18
34.88
58.09
32.65
26i.44
33.77
20.81
31.54
.44
Seal (Ours)
MinkUNet
166.18
75.38
37 37.33
42.7 77
29.93
73
40.32
20.31
37.73
24.94
Random
PolarNet
115.09
76.34
58.23
63.91
64.82
44.60
61.91
40.77
53.66
42.01
Random
CENet
112.79
76.04
67.01
09.87
61.64
58.31
49.97
60.89
53.31
24.78
Random
Waffletron
106.73
72.78
56.07
73.93
49.59
59.46
65.19
33.12
61.51
44.01
Random
Cy 3yl
105.56
78.08
61.42
71.02
58. 40
56.02
64.15
45.36
59.97
43.03
Random
SPVCNN
106.65
74170
59.01
72.16
11.08
38.36
65.36
36.83
62.29
49.21 19.21
Random
MinkUNet
112.20
72.57
62.96
70.65
55.48
51.71
62.01
31.56
59.64
39.41
PPKT [65]
MinkUNet
105.64
76.06
6.1.01
72.18
59.08
57.17
63.88
36.34
60.59
39.57
SLidR [85]
MinkUNett
106.08
75.99
65.41
72.31
56.01
56.00
62.87
41.94
61.16
38.90
Seal (Ours)
MinkUNet
92.63
83.08
72.66
74.31
66.22
66.14
65.96
57.44
59.87
39.85
Downstream Generalization. To further verify the capability of Seal on segmenting any automotive point clouds, conduct extensive experiments eleven different datasets and show the results in Table 1 and Table 2. Note that each of these datasets has a unique data collection protocol and diverse data distribution ranging from diverse sensor types and data acquisition environments to scales and fidelity making this evaluation a comprehensive one. The results show that our framework constantly outperforms prior arts across all downstream tasks on all eleven datasets, which concretely supports the effectiveness and superiority of our proposed approach.
Semi-Supervised Learning. Recent research [55, 59] reveals that combining both labeled and unlabeled data for semi-supervised point cloud segmentation can significantly boost the performance on downstream tasks. We follow these works to implement such learning paradigm where momentum upduted teacher model is adopted to assign pseudo-labels for the unlabeled data. The results from the last row of Table show that Seal is capable of providing reliable supervision signals for data-efficient learning. Notably, our framework with partial annotation is able to surpass some recent fully-supervised learning methods. More results this aspect are included in the Appendix.
Robustness Probing. is often of great importance to assess the quality of learned representations on out-of-distribution data, especially for cases that occur in the real-world environment. We resort to the recently established nuScenes-C in the Robo3D benchmark [53] for such robustness evaluations. From Table 3 we observe that the self-supervised learning methods [65, 85] in general achieve better robustness than their baseline [19]. Seal achieves the best robustness under almost all corruption types and exhibits superiority over other recent segmentation backbones with different LiDAR representations, such as range view [16], BEV [115], raw points [78], and multi-view fusion [89].
Qualitative Assessment. We visualize the predictions of each pertaining method pretrained and fine-tuned on nuScenes [26] in Fig. 5. clear observation is the substantial enhancement offered by all pretraining methods when juxtaposed with a baseline of random initialization. Diving deeper into the comparison among the trio of tested techniques, Seal stands out, delivering superlative segmentation outcomes, especially in the intricate terrains of driving scenarios. This is credited to the strong spatial and temporal consistency learning that Seal tailored to excite during the pretraining. We have cataloged additional examples in the Appendix for more detailed visual comparisons.
4.3 Ablation Study
Foundation Model Comparisons. We provide the first study on adapting VFMs for large-scale point cloud representation learning and show the results in Table 4. We observe that different VFMs exhibit
8

---

Table 4: Ablation study on pretraining frameworks (ours vs, SLidR [85]) and the knowledge transfer effects from different vision foundation models. All mloU scores are given in percentage (%).
nuScenes
KITTI
Waymo
Syuth4ID 1%
Method
Superpixel
LP
5%
10%
25%
Full
1%
1
Random
8.10
30.30
47.84
56.15
65.48
74.66
39.50
39.41
20.22
SLIC [1]
38,80
38.30
52.49
59.84
66.91
74.79
41.00
17.12
63.10
SAM [50]
41.49
43.67
55.91
61.74
68.85
75.40
43.35
48.64
63.15
SLidR
D-Decdder [122]
41.71
54.24
67.35
45.70
OpenSeel [111]
42.61
43.82
54.17
61.03
67.30
74.85
45. 88
48.64
63.31
SEEM [123)
43.00
44.02
53.03
60.84
67.38
75.21
45.72
48.75
63.13
SLIC [1]
40.89
39.77
53.33
61.58
67.78
75.32
45.75
47.74
63.37
SAM [50]
43.94
45.09
56.95
62.35
69.08
75.92
46.53
49.00
63.76
Seal
X-Decoder [122]
42.64
44.31
55.18
62.03
68.24
75.56
46.02
49.11
61.21
Open [111]
44.67
44.74
55.13
62.36
69.00
75.64 65.644
46.13
48.98
64.229
SEEM [123]
44.95
45.84
55.64
62.97
68.41
75.60
46.63
49.34
64.50
Table 5: Ablation study of each component pretrained on nuScenes [26] and fine-tuned on nuScenes [26], SemanticK/777 [3], and Waymo Open [88]. C2L. Camera-to-LiDAd distillation. VFM: Vision foundation models. STC: Superpoint temporal consistency. P2S: Point-to-segment regularization.
nuSeenes
KITTI
Waymo
C2L
VFM
STC
P2S
L.P
1%
5%
10%
25 %
Full
1%
1%
())
35.80
38,30
52-49
59.84
66.91
74.79
44.60
47.12
(2)
40.45
41.62
54.67
60.48
67.61
75.30
45.38
(3)
43.00
44.02
53.03
60.84
67.38
75.21
45.72
48.75
(4)
44.01
44.78
55.36
61.99
67.70
75.00
46.49
49.15
43.35
44.25
53.69
61.11
67.42
75.44
46.07
48.82
(6)
44.95
45.84
55.64
62.97
68.41
75.60
46.63
diverse abilities in encouraging contrastive objectives. All VFMs show larger gains than SLIC [i] with both frameworks, while SEEM [123] in general performs the best. Notably, SAM [50] tends to generate more fine-grained superpixels and yields better results when fine-tuning with more annotated data. We conjecture that SAM [50] generally provides more negative samples than the other three VFMs which might be conducive to the superpixel-driven contrastive learning. On all setups, Seal constantly surpasses SLidR [85] by large margins, which verifies the effectiveness of our framework.
Cosine Similarity. We visualize three examples of feature similarity across different VFMs in Fig. 4. We observe that our contrastive objective has already facilitated distinction representations before fine-tuning. The semantically-rich VFMs such as X-Decoder [122], OpenSeeD [111], and SEEM [123] offers overt feature cues for recognizing objects and backgrounds; while the unsupervised (SLIC [1]) or too fine-grained (SAM [50]) region partition methods only provide limited semantic awareness. Such behaviors have been reflected in the linear probing and downstream fine-tuning performance (see Table 4), where SEEM [123] tends to offer better consistency regularization impacts during the cross-sensor representation learning.
Component Analysis. Table shows the ablation results of each component in the Seal framework. Specifically, direct integration of VFMs (row #3) or temporal consistency learning (row #2) brings 4.20% and 1.65% mIoU gains in LP, respectively, while combination of them (row #4) leads to a 5.21% mloU gain. The point-to-segment regularization (row #5) alone also provides considerable performance boost of around 4.55% mloU. Finally, an integration of all proposed components (row #6) yields our best rest-performing model, which is 6. 1.15% mloU better than the prior art [85] in I.P and also outperforms on all in in-distribution and -out-of cout-ofddistrbutiinn downstream setups.
Sensor Misalignment. An accurate calibration is crucial for establishing correct correspondences be- tween LiDAR and cameras. most cases, those sensors should be well-calibrated for an autonomous car. It is unusual if the calibration is completely unknown, but it is possible to be imprecise due to a lack of maintenance. Hence, we conduct the following experiments to validate the robustness of our method. For each point coordinate P: = (Iz, V/. 2;) in LiDAR point cloud, its corresponding pixel P. = (u, v;) in the camera view can be found via Eq. 1. To simulate the misalignment between LiDAR and cameras, we insert random noises into the camera extrinsic matrix I K, with relative proportion of 1%, 5%, and 10%. Table shows the results of PPKT [65], SLidR [85], and Seal under such noise perturbations. We observe that the possible calibration errors between LiDAR

---

Groundtruth
Random
PPKT
SLidR
Seal OOrrs)
Figure 5: The qualitative results of different point cloud pretraining approaches pretrained on the raw data of nuScenes [26] and fine-tuned with 1% labeled data. To highlight the differences, the correct / incorrect predictions are painted in gray / red, respectively. Best viewed in color.
Table 6: Ablation study on the possible misalignment between the LiDAR and camera sensors. The perturbation is randomly generated and inserted. All mloU scores are given in percentage (%)
1% Misalignment
5% Misalignment
10% Misalignment
Method
1%
5%
10%
25%
1%
5%
10%
25% %
1%
5%
10%
25%
PPKT [65]
34.94
51.11
58.54
65.01
33.69
51.40
58.00
64.11
33.35
50.98
57.
63.52
SL.dR [85]
37.92
53.08
59.89
66.90
38.00
52.36
60.01
64.10
37.30
51.11
58.50
64.50
Scal (Ours)
45.23
55.71
62.62
68.13
45.66
55.42
62. 77
68.01
44.80
54.45
61.80
68.29
and cameras will cause performance degradation for different pretraining approaches (compared to Table 1). The performance degradation for PPKT [65] is especially prominent; we conjecture that this because the point-wise consistency regularization of PPKT [65] relies heavily on the calibration accuracy and encounters problems under misalignment, Both SLidR [85] and Seal exhibit certain robustness; we believe the superpixel-level consistency less sensitive calibration perturbations. It is worth mentioning that Seal can maintain good performance under calibration error, since: i) Our VFM-assisted representation learning tends to more robust; and ii) We enforce superpoint temporal consistency during the pertaining which does not rely on the 2D-3D correspondence.
5 Concluding Remark
In this study, we presented Seal, a versatile self- supervseed learning framework capable of segmenting any automotive point clouds by encouraging spatial and temporal consistency during the representa- tion learning stage. Additionally, our work pioneers the utilization of VFMs to enhance 3D scene understanding. Extensive experimental results on 20 downstream tasks across eleven different point cloud datasets verified the effectiveness and superiority of our framework. We aspire for this research to catalyze further integration of large-scale 2D and 3D representation learning endeavors, which could shed light on the development of robust and unnotation-efficica perception models.
Potential Limitations. Although our proposed framework holistically improved the point cloud segmentation performance across a wide range of downstream tasks, there are still some limitations that could hinder the scalability. i) Our model operates under the assumption of impeccably calibrated and synchronized LiDAR and cameras, which might not always hold true in real-world scenarios. ii) single set of point clouds with unified setups; while aggregating
We only pretrain the networks more abundant data from different datasets pretraining would further improve the generalizability of this framework. These limitations present intriguing avenues for future investigations.
10

---

Acknowledgements. This study is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOE-T2EP20221 01172002110012), NTU NAP, and under the RIE2020 Industry Alignment Fund- Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). This study also supported by the National Key R&D Program of China (No. 20222/D0161600). We thank Tai Wang for the insightful review and discussion.
References
[1] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk.
Slic superpixels compared to state-of state-of-the-ar the-art superpixel methods. JEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11):2274 2282, 2012. 2, 3, 5, 6, 9, 21, 22, 23, 31, 32, 32, 33
[2] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Jürgen Gall, and Cyrill Stachniss. Towards 3d bidar-based semantic scene understanding of 3d point cloud sequences: The semantickitti dataset. International Journal of Robotics Research, 40:959-96, 2021. 1
[3] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Juergen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In /EEE/CVF International Conference on Computer Vision, pages 9297-9307, 2019. 9, 18, 19, 25, 26, 27, 29
[4] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The softmax loss: tractable surrogate the optimization of the -nneruuuonnno measure neural networks. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4413 4421, 2018. 7.26
[5] Mario Bijelie, Tobias Gruber, Fahim Mannan, Florian Kraus, Werner Ritter, Klaus Dietmayer, and Felix Heide. Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1682-1 1692, 2020. 20
[6] Alexandre Boulch, Corentin Sautier, Bjöm Michele, Gilles Puy. and Renaud Marlet. Also: Automotive lidar self-supervision by occupancy estimation. arXiv preprint 2r 2205867, 2022. 3
[7] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora. Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 11621-11631, 2020. 4,
[8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE/CVF International Conference on Computer Vision, pages 9650 (000)6660,,22 2021. 3
[9] Qi Chen, Sourabh Vora, and Oscar Beijbom. Polarstream: Streaming lidar object detection and seg- mentation with polar pillars. In Advances in Neural Information Processing Systems, volume 34, 2021.
3
[10] Runnan Chen, Youquan Liu. Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou. Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. In IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 7020-7030, 2023. 3
[11] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1697-1607, 2020. 3
[12) Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.0422 2020. 3, 7, 26
[13] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training sulf-supervised vision transformers. In IEEE/CVF International Conference Computer Vision, pages 9640 9649, 2021. 3
[14] Ye Chen, Jinxian Liu, Bingbing Ni. Hang Wang, Jiancheng Yang. Ning Liu, Teng Li, and Qi Tian. Shape self-correction for unsupervised point cloud understanding. In IEEE/CVF International Conference on Computer Vision, pages 8382-8391, 2021.
[15] Yujin Chen, Matthias Nießner, and Angela Dai. 4dcontrast: Contrastive learning with dynamic correspon- dences for 3d scene understanding. In European Conference on Computer Vision, pages 543-560, 2022.
3
[16] Huixian Cheng, Xianfeng Han, and Guoqiang Xiao. Cenet: Toward concise and efficient lidar semantic segmentation for autonomous driving. In IEEE International Conference on Multimedia and Expo, pages
1 1-6, 2022. 3,8 8
[17] Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu. A12-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12547-12556 2021. 3
11

---

[18] Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer. Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Bremnan Saeta, Mark Diaz. Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Douglas Eck, Jeff Dean. Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. arXiv preprint urXiv:220.003311, 2022.
[19] Christopher Choy, Jun Young Gwak. and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3075-3084, 2019. 3. 7, 26
[20] Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy. Salsanext: Fast, uncertainty aware semantic segmentation of lidar point clouds. In International Symposium Visual Computing, pages 207- 222, 2020.
[21] Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by
context prediction. In IEEE/CVF International Conference on Computer Vision, pages 1422-1430, 2015.
3
[22] Yuhao Dong, Zhuoyang Zhang, Yunze Liu, and Li Yi. Complete-to to partial 4d distillation for self- supervised point cloud sequence representation learning. arXiv preprint arXiv:2212.05330, 2022. 3
[23] Martin Ester, Mlans-Peter Kriegel. Jörg Sander, and Xiaowei Xu. A density-bass based algorithm for discovering clusters in large spatial databases with noise. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 226-231, 1996. 5
[24] Christoph Feichtenhofer, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal leamners. In Advances in Neural Information Processing Systems, volume 35, 2022. 3
[25] Martin A. Fischler and Robert C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381-395 395. 1981. 5
[26] Whye Kit Fong, Rohit Mohan, Juana Valeria Hurtado. Lubing Zhou, Holger Caesar, Oscar Beijbom. and Abhinav Valada. Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking. IEEE Robotics and Automation Letters, 7:3795-3880 -3095,, 2022. 4, 7, 8, 9, 10, 18, 19, 23, 24, 26, 27, 28, 29, 30, 34, 35, 36
[27] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Unsupervised semantic segmentation by contrasting object mask proposals, In /EEE/CVF International Conference on Computer Vision, pages 10052-10062, 2021. 3
[28] Biao Gao, Yancheng Pan, Chengkun Li, Sibo Geng, and Huijing Zhao. Are we hungry for 3d lidar data for semantic segmentation? survey of datasets and methods. IEEE Transactions on Intelligent Transportation Systems, 2021. 2,
[29] Siddhant Garg and Mudit Chaudhary. Serp: supervuue representation learning using perturbed point clouds. arXiv preprint T20:22200000667 2022. 3
[30] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
[31] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022. 3
[32] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[33] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016. 7, 26
[34] Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International Conference Machine Learning, pages 4182 4192, 2020. 3
[35] Jordan Hoffmmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen
12

---

Simonyan. Erich Elsen, Jack W. Rae. Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv preprint 2Xiv:2203.155566 2022. 1
[36]
Fangzhou Hong, Lingdong Kong, Hui Zhou, Xinge Zhu, Hongsheng Li, and Ziwei Liu. Unified 3d and 4d panoptic segmentation via dynamic shifting network. arXiv preprint arXiv:2203.07188 2022.
[37] Fangzhou Hong, Hui Zhou, Xinge Zhu, Hongsheng Li, and Ziwei Liu. Lidar-based panoptic segmentation via dynamic shifting network. In IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pages 13090-13099, 2021. 3
[38] Ji Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts, In JEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 15587-13597, 2021.
[39] Ji Hou, Saining Xie, Benjamin Graham, Angela Dai, and Matthias Nicßner. Pri3d: Can 3d priors help 2d representation lcarning? In /EEE/CVF International Conference on Computer Vision, pages 5693-5702, 2021. 3
[40] Qingyong Hu, Bo Yang, Guangchi Fang, Yulan Guo, Aleš Leonardis, Niki Trigoni, and Andrew Markham. Sqn: Weakly-supervised semantic segmentation of large-scale 3d point clouds. In European Conference on Computer Vision, pages 600-619, 2022. 3
[41] Qingyong Hu, Bo Yang, Sheikh Khalid, Wen Xiao, Niki Trigoni, and Andrew Markham. Towards semantic segmentation of urban-scale 3d point clouds: dataset, benchmarks and challenges. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4977-4987, 2021. 3
[42] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation large-scale point clouds. In /EER/CVF
Conference on Computer Vision and Pattern Recognition, pages 11 11108-11117, 2020. 3
[43] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Learning semantic segmentation of large-scale point clouds with random sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(11):8338-8354 2022. 3
[44] Zeyu Hu, Xuyang Bai, Runze Zhang, Xin Wang. Guangyuan Sun, Hongbo Fu, and Chiew-Lan Tai, Lidal: Inter-frame uncertainty based active learning for 3d lidar semantic segmentation. In Eumpean Conference
on Computer Vision, pages 248-265, 2022. 3
[45] Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu. Spatio-temporal self-supervised represen- tation learning for 3d point clouds. In /EEF/CVF International Conference on Computer Vision, pages 6535-6545, 2021. 3
[46] Olivier J. Hénaff, Skanda Koppula, Jean- Saan-Baptiste Alayrac, Aaron Van den Oord, Oriol Vinyals, and Joao Carreira. Efficcintt visual pretraining with contrastive detection. In IEFF/CVF International Conference on Computer Vision, pages 10086-10096, 2021. 3
[47] Maximilian Jaritz, Tuan Hung Vu, Raoul De Charette, Émilie Wirbel, and Patrick Pérez. Cross-mode moosl learning for domain adaptation in 3d semantic segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(2):1553-155 1544, 2023. 3
[48] Maximilian Jaritz. Tuan-Hung Vu, Raoul de Charette, Emilie Wirbel, and Patrick Pérez. xmuda: Cross- modal unsupervised domain adaptation for 3d semantic segmentation. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12605 12614, 2020. 3
[49] Peng Jiang, Philip Osteen, Maggie Wigness, and Srikanth Saripallig. Rellis-3d dataset: Data, beuchmarks and analysis. In IEEE International Conference on Robotics and Automation, pages 1110-1116 2021. 7, 19, 20, 25,26
[50] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg. Wan-Yer Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In /EEE/CVF International Conference Computer Vision, pages 4015-4026, 2023. 2. 6, 9, 21, 22,
31, 33
[51] Alexey Klokov, Di Un Pak, Aleksandr Khorin, Dmitry Yudin, Leon Kochiev, Vladimir Luchinskiy. and Vitaly Bezuglyj. Daps3d: Domain adaptive projective segmentation of 3d lidar point clouds. Preprint, 2023. 7, 19, 20, 24, 26
[52) Lingdong Kong. Youquan Liu, Runnan Chen, Yuexin Ma, Xinge Zhu, Yikang Li, Yuenan Hou, Yu Qiao, and Ziwei Liu. Rethinking range view representation for lidar segmentation. In IEEE/CVF International Conference on Computer Vision, pages 228 220, 2023. 3
[53] Lingdong Kong. Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robo3d: Towards robust and reliable 3d perception against corruptions. In IEEE/CVF International Conference Computer Vision, pages
2023. 7, 8. 19, 20, 27, 30
13

---

[54] Lingdong Kong, Niamul Quader, and Venice Erin Liong. Conda: Unsupervised domain adaptation for lidar segmentation via regularized domain concatenation. In IEEE International Conference on Robotics and Automation, pages 9338-9345, 2023.
[55] Lingdong Kong, Jiawei Ren, Liang Pan, and Ziwei Liu. Lasermix for semi-supervised lidar semantic segmentation. In IREE/CVP Conference on Computer Vision and Pattern Recognition, pages 21 21705- 21715, 2023. 1, 2,
[56] Lingdong Kong, Jiawei Ren, Liang Pan, and Ziwei Liu. semi-supervised lidar semantic segmentation with spatial consistency training. In International Conference on Learning Reprexentations Workshop.
2023. 3
[57] Shamit Lal, Mihir Prabhudesai, Ishita Mediratta, Adam W. Harley, and Katerina Fragkiadaki. Coconets: Continuous contrastive 3d scene representations. In IEEF/CVF Conference on Computer Vision and
Pottern Recognition, pages 12487-12496, 2021, 3
[58] Lanxiao Li and Michacl Heizmann. A closer look at invariances in self- self-supervised pre-training for 3d vision. In European Conference on Computer Vision, pages 656-65 673, 2022. 3
[59] Li Li, Hubert PH Shum, and Toby P. Breckon. Less is more: Reducing task and model complexity for 3d
point cloud semantic segmentation. arXiv preprint arXiv:2303. 12203, 2023. 1,
[60] Rong Li. Raoul de Charelte, and C. A. O. Anh-Quan. Coarse3d: Class-prototypes for contrastive learning weakly-supervised 3d point cloud segmentation. British Muchine Vision Conference, 2022. 1,3
[61] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-
image pre-training via masking. arXiv preprint arXiv:2212.00794 2022.
[62] Venice Erin Liong. Thi Ngoc Tho Nguyen, Sergi Widjaja. Dhananjai Sharma, and Zhuang Jie Chong. Amvnet: Assertion- based multi-view fusion network for lidar semantic segmentation. arXiv preprint arXiv:2012.04934, 2020. 3
[63] Kangcheng Liu, Yuzhi Zhao, Qiang Nie, Zhi Gao, and Ben M. Chen. Weakly supervised 3d scene segmentation with region-level boundary awareness and instance discrimination. In European Conference on Computer Vision, pages 37-55, 2022. 3
[64] Minghua Liu, Yin Zhou, Charles R. Qi, Boqing Gong, Hao Su, and Dragomir Anguelov. Less: Label- efficient semantic segmentation for lidar point clouds. In European Conference on Computer Vision, pages 70-89, 2022.
Hueg-Yueh [65] Yuch-Cheng Liu, Yu-Kai Huang, Hung- Yueh Chiang, Hung-Ting Su, Zhe- Yu Liu, Chin-Tang Tang Chen, Ching-Yu Yu Tseng, and Winston H. Hsu. Learning from 2d: Contrastive pixel-to-point knowledge transfer for 3d pretraining, arXiv preprint 2xXx::21440046. 2021. 7, 8, 9, 10, 27
[66] Anas Mahmoud, Jordan SK Hu, Tianshu Kuai, Ali Harakeh, Liam Paull, and Steven L. Waslander. Self- supervised image-to-point distillation via semantically tolerant contrastive loss. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 2, 3, 7,22
[67] Bjöm Michele, Alexandre Boulch, Gilles Puy, Maxime Bucher, and Renaud Marlet. Generative zero-shot learning for semantic segmentation of 3d point clouds. International Conference on 3D Vision, pages 992-1002, 2021. 3
[68] Bjöm Michele, Alexandre Boulch, Gilles Puy, Tuan- Hung Vu, Renaud Marlet, and Nicolas Courty. Saluda: Surface-based automotive lidar unsupervised domain adaptation. arXiv preprint arXiv:2304.03255 2023.
3
[69] Andres Milioto, Ignacio Vizzo. Jens Behley, and Cyrill Stachniss. Rangenet++: Fast and accurate lidar semantic segmentation. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4213-4220. 2019. 3
[70] Ishan Misra, C. Lawrence Zitnick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In European Conference on Computer Vision, pages
2016. 3
[71] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pages 69-84, 2016. 3
[72] Lucas Nunes, Rodrigo Marcuzzi, Xicyuanli Chen, Jens Behley, and Cyrill Stachniss. Segcontrast: 3d point cloud feature representation learning through self-supervised segment discrimination. IEEE Robotics and Automation Letters, 7:21 16-2123, 2022. 3
[73] OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.088. 087774444 2023. 1
[74] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Woj- ciech Galuba, Russell Howes, Po- Yao Huang. Shang Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bo- janowski. Dinov2: Learning robust visual features without supervision. arXiv preprint 234 2023.
14

---

[75] Yancheng Pan, Biao Gao, Jilin Mei, Sibo Geng, Chengkun Li, and Huijing Zhao. Semanticposs: A point cloud dataset with large quantity of dynamic instances. In IEEE Intelligent Vehicles Symposium, pages 687-693, 2020. 7. 19, 20, 25, 26
[76] Duo Peng. Yinjie Lei, Wen Li, Pingping Zhang, and Yulan Guo. Sparse-to-dente feature matching: Intra and inter domain cross- modal learning in domain adaptation for 3d semantic segmentation. In IEEE/CVF International Conference on Computer Vision, pages 7 7108-7117, 30
[77] Omid Poursaced, Tianxing Jiang, Han Qiao, Nayun Xu, and Vladimir G. Kim. Self-supervised learning of point clouds via orientation estimation. In International Conference on 3D Vision, pages 101 1018-1022 8-1288, 2020.
[78] Gilles Puy, Alexandre Boulch, and Renaud Marlet. Using a waffle iron for automotive point cloud semantic segmentation. In /EEE/CVF International Conference on Computer Vision, pages 5379-3389
2023. 3, 8
[79] Haibo Qiu. Baosheng Yu, and Dacheng Tao. Gfnet: Geometric flow network for 3d point cloud semantic segmentation. Transactions on Machine Learning Research, 2022. 3
[80] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763, 2021. 3
[81] Giulia Rizzoli, Francesco Barbato, and Pietro Zanuttigh. Multimodal semantic segmentation in au- tonomous driving: A review of current approaches and future perspectives. Technologies, 10(4). 2022.
[82] Cristiano Saltori, Evgeny Krivoshcev, Stéphane Lathuiliére, Nicu Sebe, Fabio Galasso, Giuseppe Fiameni, Elisa Ricci, and Fabio Poicsi. Gipso: Geometrically informed propagation for online adaptation in 3d lidar segmentation. In European Conference on Computer Vision, pages 567-585, 2022. 3, 7, 19, 20, 25, 26, 27,27
[83] Aditya Sanghi. Info3d: Representation learning on 3d objects using mutual information maximization and contrastive learning. In European Conference on Computer Vision, pages 626-642, 2020. 3
[84] Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by reconstructing space. In Advances in Neural Information Processing Systems, volume 32, 2019. 3
[85] Corentin Sautier, Gilles Puy, Spyros Gidaris. Alexandre Boulch, Andrei Bursuc, and Renaud Marlet. Image-to-lidar supervvid distillation for autonomous driving data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9891-9901, 2022. 2, 3, 7, 9, 10, 21, 23, 24, 26, 27
[86] Hanyu Shi, Jiacheng Wei, Ruibo Li, Fayao Liu, and Guosheng Lin. Weakly supervised segmentation on outdoor 4d point clouds with temporal matching and spatial graph propagation. In IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pages 1 11800-1889, 2022. 3
[87] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley. Jared Casper, and Bryan Catanzaro. Training compute- optimal large language models. arXiv preprint arXiv: 190:::9999000300 2019.
[88] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofcev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open datasct. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2446 2454, 2020. 7, 9, 18, 19, 25, 26, 27, 29
[89] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point-voxel convolution. In European Conference on Computer Vision, pages 685-702, 2020. 3, 7,8
[90] Hugues Thomas, Charles R Qi. -eaannel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In IEEE/CVF International Conference on Computer Vision, pages 144100,40, 2019. 3
[91] Bach Tran, Binh-Son Hua, Anh Tuan Tran, and Minh Hoai. Self-supervised learning with multi-view rendering for 3d point cloud analysis. In Asian Conference on Computer Vision, pages 3086-3103, 2022.
3
[92] Larissa T. Triess, Mariella Dreissig, Christoph B. Rist, and J. Marius Zöllner. A survey on deep domain adaplation for lidar perception. In IEEE Intelligent Vehicles Symposium Workshop, pages 350-357, 2021.
2
[93] Larissa T Triess, David Peter, Christoph B Rist, and Marius Zöllner. Bean-based semantic segmentation of lidar point clouds: An experimental study. In IEEE Intelligent Vehicles Symposium, pages 1121, 2020.
15

---

[94] Mare Uecker, Tobias Fleck, Marcel Pflugfelder, and J. Marius Zöllner. Analyzing deep learning represen- tations of point clouds for real-time in-vehicle lidar perception. arXiv preprint arXiv:2210.16612, 2022. 1.3
[95] Ozan Unal, Dengxin Dai, and Lue Van Gool. Scribble-supervieed lidar semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2697-2707, 2022. 1, 1,3,7 7, 18, 19, 25, 26
[96] Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2504,, 2021. 4
[97] Hanchen Wang. Qi Liu, Xiangyu Yuc, Joan Lasenby, and Matt J. Kusner. Unsupervised point cloud pre-training via occlusion completion. In IEEE/CVF International Conference on Computer Vision, pages 9782-9792, 2021. 3
[98] Xinlong Wang. Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self- supervised visual pre-training. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3024-3033, 2021. 3
[99] Xinlong Wang. Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284, 2023. 3
[100] Aoran Xiao, Jiaxing Huang, Dayan Guan, Fangneng Zhan, and Shijian Lu. Transfer learning from synthetic to real lidar point cloud for semantic segnenttttnnioooonn AAAI Conference on Artificial Intelligence. pages 2795-2803, 2022. 7, 19, 20, 25, 26
[101] Aoran Xiao, Jiaxing Huang. Dayan Guan, Xiaoqin Zhang. Shijian Lu, and Ling Shao. Unsupervised point cloud representation learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 2
[102] Aoran Xiao, Jiaxing Huang. Weihao Xuan, Ruijie Ren, Kangcheng Liu, Dayan Guan, Abdulmotaleb El Saddik, Shijian Lu, and Eric Xing. 3d semantic segmentation in the wild: Leaminng generalized models for adverse-condition point clouds. In /EEE/CVF Conference on Computer Vision and Pattern Recognition,
pages 9382 9392, 2023. 7, 19, 20, 25, 26
[103] Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European Conference on Computer Vision, pages 574-591, 2020. 3,7
[104] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16684 16684-16693, 2021. 3
[105] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9653 9663, 2022. 3
[106] Chenfeng Xu, Bichen Wu, Zining Wang. Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. European Conference on Computer Vision, pages 1-19, 2020.
[107] Jianyun Xu. Ruixiang Zhang. Jian Dou, Yushi Zhu, Jie Sun, and Shiliang Pu. Rpvnet: A deep and efficient sange-point-voxe fusion network for lidar point cloud segmentation. In IEEF/CVF International Conference on Computer Vision, pages (6024-16033, 2021. 3
[108] Mingye Xu, Mutian Xu, Tong He, Wanli Ouyang, Yali Wang. Xiaoguang Han, and Yu Qiao. Mm ids-dde 3d scene understanding by customizing masked modeling with self-distileed consistency. arXiv preprint vrxiv:2212.09944 2022. 3
reconstruction and
[109] Ryosuke Yamada, Hirokatsu Kataoka, Naoya Chiba, Yukiyasu Domae, and Tetsuya Ogata. Point cloud pre-training with natural 3d structures. In JEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 21283-21293, 2022. 3
[110] Siming Yan, Zhenpei Yang, Haoxiang Li, Li Guan, Hao Kang, Gang Hua, and Qixing Huang, Implicit autoencoder for point cloud self-supervised representation learning. arXiv preprint arXiv:2001.00785. 2022.3
[111] Hao Zhang, Feng Li. Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang. and Lei Zhang. A simple framework for open- vocabbuayyl segmentation and detection. arXiv preprint arXiv:2303.0813 2023. 2, 3, 6, 9, 21, 22, 23, 31, 32, 33
[112] Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In European Conference Computer Vision, pages 649 666, 2016. 3
16

---

[113] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov. Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Pumit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zeulemoyer. Opt: Open pre-trained transformer language models, arXiv preprint 1exXv:2205.01066 2022.
[114] Tunhou Zhang, Mingyuan Ma, Feng Yan, Hai Li, and Yiran Chen. Pids: Joint point search for 3d point cloud. IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2298-1307, 2023. 3
[115] Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, and Hassan Foroosh. Polarnet: improved grid representation for online lidar point clouds semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9601-9610, 2020. 8
[116] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. In IEEE/CVF International Conference on Computer Vision, pages 10252-10263, 2021. 3.7
[117] Junhao Zhao, Weijie Huang. Hai Wu, Chenglu Wen, Bo Yang, Yulan Guo. and Cheng Wang. Semanticflow: Semantic segmentation of sequential lidar point clouds from sparse frame annotations. IEEE Transactions on Geoscience and Remote Sensing, 61:1-11, 2023.
[118] Yiming Zhao, Lin Bai, and Xinming Huang. Fidnet: Lidar point cloud semantic segmentation with fully interpolation decoding. In IEEE/RSJ International Conference Intelligent Robots and Systems, pages
4453 4458, 2021. 3
[119] Zixiang Zhou, Yang Zhang, and Hassan Foroosh. Panoptic-polarnet Proposal-free lidar point cloud panoptic segmentation, In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13194-13203, 2021.
[120] Xinge Zhu, Hui Zhou, Tai Wang. Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In /EEE/CVF Conference
Computer Vision and Pattern Recognition, pages 9939-9948, 2021. 3
[121] Zhuangwei Zhuang. Rong Ii, Kui Jia, Qicheng Wang. Yuanqing Li, and Mingkui Tan. Perception-aware multi-sensor fusion for 3d lidar semantic segmentation. In JEEE/CVF International Conference on Computer Vision, pages 16280-16290 2021. 3
[122] Xueyan Zou, Zi-Yi Dou, Jianwei Yang. Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, Nanyun Peng, Lijuan Wang. Yong Jae Lee, and Jianfeng Gao. Generalized decoding for pixel, image, and language. In IEEE/CVF Conference Computer Vision and Pattenn Recognition, 2023. 2, 3, 6, 9, 21, 22, 23, 32, 32.
[123] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jac Lee. Segment everything everywhere all once. arXiv preprint arXh::3304.06718 18, 2023. 1, 2, 3, 6, 9, 21, 22, 23, 24, 25, 31, 32, 33
17

---

Appendix
In this appendix, we supplement the following materials to support the findings and observations in the main body of this paper:
Section A elaborates on additional implementation details to facilitate reproduction.
Section provides the complete quantitative results of our experiments.
Section includes more qualitative results to allow better visual comparisons.
Section D acknowledges the public resources used during the course of this work.
A
Additional Implementation Detail
A.1
Datasets
In this work, we conduct extensive experiments from a wide range of point cloud datasets to verify the effectiveness of the proposed Seal framework. A summary of the detailed configurations and emphases of these datasets is shown in Table A.
'nSScenes [26]: The nuScenes" dataset offers substantial number of samples collected by the LiDAR, RADAR, camera, and IMU sensors from Boston and Singapore, allowing machine learning models to learn useful multi-modal features effectively. For the point cloud segmentation task, it consists 1000 scenes of a total number of 1.1B annotated points, collected by a Velodyne HDL32E LiDAR sensor. It also includes image data from six cameras, which are synchronized with the LiDAR sensor. In this work, we use the LiDAR point clouds and image data from nuScenes for model pretraining. We also conduct detailed fine-tuning experiments to validate the effectiveness of representation learning. More details of this dataset can be found https: http://wwwwwwseeeeee wwwcenes.ee............ org/ httg//uscenss..
"SemanticKITTI [3]: The SemanticKITTI dataset a comprehensive dataset designed for semantic scene understanding of LiDAR sequences. This dataset aims to advance the development of algorithms and models for autonomous driving and scene understanding using LiDAR data. It provides 22 densely labeled point cloud sequences that cover urban street scenes, which are captured by a Velodyne HDL-64E LiDAR sensor. In this work, we use the LiDAR point clouds from SemanticKITTI as downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at http://smantic-kits tiittnnnttikkiii...... .tr...............oo............ org.
Waymo Open [88]: The Waymo Open dataset large-scale collection of real-world autonomous driving data. The 3D semantic segmentation subset contains 1150 scenes, split into 798 training. 202 validation, and 150 testing scenes. This subset contains 23691 training scans, 5976 validation scans, and 2982 testing scans, respectively, with semantic segmentation labels from 23 classes. The data are captured by five LiDAR sensors: one mid-range LiDAR sensor truncated to maximum of 75 meters, and four short-range LiDAR sensors truncated to a maximum of 20 meters. In this work, we use the LiDAR point clouds from Waymo Open methods. More detaits of this dataset can be found at ht /ttp::::::/aaee /waymo. com/ cwm/opem open.
downstream task validate the generalizability of pertaining
'SriibbleKTTTI [95]: The ScribbleKITTY dataset is recent variant of the SemanticKITTI dataset, with weak supervisions annotated by line scribbles. It shares the exact same amount of training samples with SemanticKITTI, i.e., 19130 scans collected by Velodyne HDL- 64E LiDAR sensor, where the total number of valid semantic labels is 8.06% compared to the supervisdd version. Annotating the LiDAR point cloud in such way corresponds to roughly a 90% time-saving. In this work, we use the LiDAR point clouds from Scrib- bleKITTI a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https: / gi thub. com/ ouenal www.sbliebbiiiiccccccccccc.eeeie tti.
'Here we refer to the lidarseg subset of the nuScenes dataset. Know more details about this dataset at the official webpage: ht https: //::://wwww nuscenes org/ nuscenes.
18

---

Table A: The sensor configuration and data statistics for the eleven datasets used in our experiments.
Dataset
Ilustration
Sensor Setup
Statisties
Type
1x LiDAR (32-beam)
16 semantic clusses training samples 6019 validation samples 6008 testing samples
Real-world
nuScenes [26]
RGB Camera
resolutoottnn point cloud
5x RADAR
Dense annotation Multi-modality
1x IMU & GPS
1x LiDAR (64-beam)
19 semantic classes
Real-workd
SemanticKITTI 33
1x Stereo Camera
191 30 training samples
nigh-ressolutinn point cloud Dense annotation Multi-modality
1x IMU GPS
4071 validation samples
20351 testing samples
1x LiDAR (64-beam)
23 semantic classes
Real-world
Waymo Open [88]
RGB Camera
training samples
High-resolution point cloud Dense annotation
1x IMU & GPS
5976 validation samples
2982 testing samples
Multi-modality
1x LiDAR (64-bcam)
19 semantic classes
Real-workd
SeribbleKITII [95]
1x Stereo Camera
191 30 training samples
nigh-resoluttinn point cloud Sparse annotation Weaakyyssupervve learning
1x IMU GPS
4071 validation samples
20351 testing samples
1x LiDAR (64-beam)
20 semantic classes
Real-world
RELLIS-3D [49]
1x LiDAR (32-beam)
7800 training samples
High-resolution point cloud Dense annotation Multi-modality
1x Steren Camera
2413 validation sumples
1x IMU & GPS
3343 testing samples
LiDAR beam)
14 semantic
world nigh-ressolution point cloud Dense annotation
SemanticPOSS [75]
1x RGB Camera
2488 training sumples
1x IMU GPS
500 validation samples
Dynamic instance
1x LiDAR (64-beam)
21 semantic classes
Real-world
SemanticSTF [102]
1x LiDAR (32-bcam)
1326 training samples
High-resolution point cloud Dense annotation Adverse weather
1x Stereo Camera
250 validation samples
1x RADAR
500 testing samples
LiDAR (el-beam) Ix simulation suite
32 semantic 198396 total samples
Synthetic High-resolation point cloud Dense annotation learning
SyaLiDAR [100]
1x LiDAR (64-beam)
22 semantic classes 10000 training samples
Synthetic
Synth4D [82]
1x LiDAR (32-bcam)
Low-resolution point cloud Dense annotation Transfer learning
1x simulation suite
10000 validation samples
3x LiDAR (64-beam) Ix simulation suite
semantic classes
Semi-synthetic High-resolution point cloud Dense annotation Transfer learning
DAPS-3D [51]
19061 training samples
3995 validation samples
1x LiDAR (32-beam)
16 semantic classes 144456 validation samples
Synthetic
muScenes C [53]
6x RGB Camera
Low-resolution point cloud Dease annotation
5x RADAR
1x IMU & GPS
Robustness
19

---

"RELLIS33D [49]: The RELLIS-30 3D dataset is a multimodal dataset collected in an off-road environment from the Rellis Campus of Texas A&M University. consists of 13556 LiDAR scans from 5 traversal sequences. The point-wise annotations are initialized by using the camera-LiDDA calibration to project the more than 6000 image annotations onto the point clouds. In this work, we use the LiDAR point clouds from RELLIS-3) as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at http: //www. unmannedlab httg//eeaarcc/BLLLSS----
"SemanticPOSS [75]: The SemanticPOSS dataset is a relatively small-scale point cloud dataset with emphasis on dynamic instances. It includes 2988 scans collected by Hesai Pandora LiDAR sensor, which is a 40-channel LiDAR with 0.33 degree vertical resolution, forward-facine color camera, 4 wide-angle mono cameras covering 360 degrees around the ego-car. The data in this dataset was collected from the campus of Peking University. In this work, we use the LiDAR point clouds from SemanticPOSS as downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found https: //www poss p cn/ semanti cposs.
"SemanticSTF [102]: The SeManticSTF dataset is small-scale collection of 2076 scans, where the data are borrowed from the STF dataset [5]. The scans are collected by a Velodyne HDL66 S3D LiDAR sensor and covered various adverse weather conditions, including 694 snowy, 637 dense-foggy. 631 light-foggy, and 11 rainy scans. The whole dataset is split into three sets: 1326 scans for training. 250 scans for validation, and 500 scans for testing. All three splits have similar proportions of scans from different weather conditions. work, we use the LiDAR point clouds from SeManticSTF as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https://gitaie thub. wwt.cooo///o./o. ht//xiaaaaa araoaanoaa///S /ataa//iimmmm..........
this
"SynLiDAR [100]: The SynLiDAR dataset contains synthetic point clouds captured from constructed virtual scenes using the Unreal Engine 4 simulator. In total, this dataset contains 13 LiDAR point cloud sequences with 198396 scans. As stated, the virtual scenes in SynLiDAR are constituted by physically accurate object models that produced by expert modelers with the 3D-Max software. In this work, we use the LiDAR point clouds from SynLiDAR as a downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https: //gi thub. com/ x aiaaooran///// /
4D [82]: The Synth4D dataset includes two subsets with point clouds captured by simulated Velodyne LiDAR sensors using the CARLA simulator. We use the Synth4D- nuScenes subset in our experiments. It is composed of around 20000 labeled point clouds captured by a virtual vehicle navigating in town, highway, rural area, and city. The label mappings are mapped to that of the nuScenes dataset. In this work, we use the LiDAR point clouds from Synth4D-nuSeenes pertaining methods. More details of this dataset can be found at https: //gt tigh..thh..... com/ saltoricristi www.rriiisiiiittoiiioooooooooree gip-s-sss sf ouda.
downstream task to validate the generalizability of
10DPS--3 [51]: The DAPS-3D dataset consists of two subsets: DAPS-1 and DAPS-2 2; while the former is semi-synthee one with larger scale, the latter is recorded during a real field trip of the cleaning robot to the territory of the VDNH Park in Moscow. Both subsets are with scans collected by 64-line Ouster OSO LiDAR sensor. We use the DAPS-1 subset in our experiments, which contains 11 LiDAR sequences with more than 23000 labeled point clouds. In this work, we use the LiDAR point clouds from DAPS-1 as downstream task to validate the generalizability of pertaining methods. More details of this dataset can be found at https: //gi com/ haamm/uuaae//ine /DASSD..
11 "uSSeeessCC C [53]: The nuScenes-C dataset is one of the corruption sets in the Robo3D benchmark, which is a comprehensive benchmark heading toward probing the robustness of 3D detectors and segmentors under ditt----distiiutioon scenarios against natural corruptions that occur in real-world environments. A total number of eight corruption types, stemming from severe weather conditions, external disturbances, and internal sensor failure, are considered, including 'fog', 'wet ground', 'snow", 'motion blur", 'beam missing', 'crosstalk', 'incomplete echo', and 'cross-sensor' scenarios. These corruptions are simulated with rules constrained by physical principles or engineering experiences. In this work, we use the LiDAR point clouds from nuScenes-C downstream task to validate the robustness of
20

---

Table B: The statistics of superpixels (for front-view cameras) generated by SLIC [1] and different vision foundation modes [50, 122, 111, 123]. The horizontal axis denotes the number of superpixels image. The vertical axis denotes the frequency of occurrence.
Method
Front Left
Front
Front Right
SLIC
200
111
SAM [50]
200
X- D-Decoder [122]
300
OpenSceD [111]
7M
120-
SEEM [123]
pertaining methods under out-of-distribution scenarios. More details of this dataset can be found at https: // github. cww (oom//dkoogg125/Roobb]
A.2 Vision Foundation Models
In this work, we conduct comprehensive experiments on analyzing the effects brought by different vision foundation models (VFMs), compared to the traditional SL.I [1] method. Some statistical analyses of these different visual partition methods are shown in Table B and Table C.
SLIC [1] (traditional method): The SLIC model, which stands for 'simple linear iterative clustering', is popular choice for visual partitions of RGB images. It adapts k-means clustering approach to generate superpixels, in an efficient manner, and offers good un- supervised partition abilities for many downstream tasks. The pursuit of adherence to boundaries and computational and memory efficiency allows SLIC to perform well on different image collections. In this work, we follow SLidR [85] and use SLIC to generate
21

---

Table C: The statistics of superpixels (for back-view cameras) generated by SLIC [1] and different vision foundation modes [50, 122, 111, 123]. The horizontal axis denotes the number of superpixels image. The vertical axis denotes the frequency of occurrence.
Method
Back Left
Back
Back Right
SLIC
200
14
111
SAM [50]
130-
200
X- D-Decoder [122]
OpenSceD [111]
200
SEEM [123]
3000
300
superpixels, with fixed quota of 150 superpixels per image, and compare framework with previous ones using SLIC superpixels. More details of this model can be found at wtt.ggtth......allleeeeeeeoo
https://gi wwt.coo///aaaaaaee commvaaeoai//Lidtt.
SAM [50]: The Segment Anything Model (SAM) is recent breakthrough towards zero-shot transferable visual understanding across a wide range of tasks. This model trained on SA- IB, large-scale dataset with over billion masks 1 11M licensed and privacy-respecting images. As a result, SAM is able to segment images, with either point, box, or mask prompts, across different domains and data distributions. In this work, with the ViT-H backbone (termed as ViT-H-SA model) to generate superpixels. We use this pretrained model directly without any further fine-tuning, More details of this model can be found at https:::... / //g/ tigu........ com wtt.://co/ooccrreeeeeeeeeea http///acebooorseercchc htt.kkressearccsseeeeeeeete arggeent-anythinn
fixed SAM model
X-Decoder [122]: The X-Decoder model is a generalized decoding framework that can predict pixel-level segmentation and language tokens seamlessly. This model is pretrained on three types of data, including panoptic segmentation, image-text pairs, and referring segmentation. For the panoptic segmentation task. the model is trained COCO2017,
22

---

which includes around 104k images for model training. In this work, we use fixed X- Decoder model termed BestSeg-Tiie to generate superpixels. We use this pretrained model directly without any further fine-tuning, More details of this model can be found at https:/ github. wwt.///coroooottooooooooo DDecooer..
The OpenSeeD model is designed for open-vocabulary segmentation and detection, which jointly learns from different segmentation and detection datasets. This model consists of an image encoder, a text encoder, and a decoder with foreground, background, and conditioned mask decoding capability. The model is trained on COC02017 and Objects365, under the tasks of panoptic segmentation and object detection, respectively, In this work, we use fixed OpenSeeD model termed openseed seee-sswittttttnee superpixels. We use this pretrained model directly without any further fine-tuming More details of this model can be found at https: : //g github....... com IDEA- (eweearch/ppeSeeeel
OpenSeeD [111]:
generate
SEEM [123]: The SEEM model contributes a new universal interactive interface for image segmentation, where 'SEEM' stands for 'segment everything everywhern with multi-modal prompts all at once'. The newly designed prompting scheme can encode various user intents into prompts in joint visual-semantic space, which possesses properties of versatility, compositionality, interactivity, and semantic awareness. As such, this model is able to generalize well to different image datasets, under zero-shot transfer manner. Similar to X-Decoder, SEEM is trained on COCO2017, with a total number of 107k images used during model training. In this work, we use a fixed SEEM model termed SEEM-oq101 to generate superpixels. We use this pretrained model directly without any further fine- tuning. More details of this model can be found at https: //tgt/ggtttt/t////////e//t com /UX--UUXDDeee Regment-Everyy Everything-e--eeee Seaiinn-vveryeereee All-At-
The quality of superpixels directly affects the performance of self-supervised representation learning. Different from the previous paradigm [85, 66], our Seal framework resorts to the recent VFMs for generating superpixels. Compared to the traditional SLIC method, these VFMs are able to generate semantically-aware superpixels that represent coherent semantic meanings objects and backgrounds around the ego-vehicle.
As been verified in our experiments, these semantic superpixels have the ability to ease the "over- segment" problem in current self-supervised learning frameworks [85, 66] and further improve the performance for both linear probing and downstream fine-tuning.
The histograms shown in Table and Table C verify that the number of superpixels per image of VFMs is much smaller than that of SLIC. This brings two notable advantages: i) Since semantically similar objects and backgrounds are grouped together in semantic superpixels. the "self-conflict" problem in existing approaches is largely mitigated, which directly boosts the quality of representation learning. ii) Since the embedding length of the superpixel embedding features Q & M x and superpoint embedding features K E R x directly relates computation overhead, reduction (e.g., around 150 superpixels per image in SLIC [i] and around 25 superpixels per image in X- Decoder [122], OpenSeeD [111], and SEEM [123]) on D would allow us train the segmentation models in more efficient manner.
Some typical examples of our generated superpixels and their corresponding superpoints shown in Fig. A, Fig. B, Fig. C, and Fig. D. As will be shown in Section B, our use of semantic superpixels generated by VFMs brings not only performance gains but also much faster convergence rate during the model pretraining stage.
A.3
Implementation Detail
A.3.1 Data Split
For model pertaining. we follow the SLidR protocol [85] in data splitting. Specifically, the nuScenes [26] dataset consists 700 training scenes in total, 100 of which are kept aside, which constitute the SLidR mini-val split. All models are pretrained using all the scans from the 600 remaining training scenes. The 100 scans the mini-val split are used to find the best possible hyperparameters. The trained models are then validated on the official nuScenes validation set, without any kind of test-time augmentation or model ensemble. This is to ensure a fair comparison with previous works and also in line with the practical requirements.
23

---

LEFT
FRONT
PRONT_RIGHT
LEFT
BACK
BIGHTRIGHT
LIDAR (RAW UNLABELED POINT)
FRONT_ LEFT
FRONT
FRONT RIGHT
BAFKLEFFT
BACK
BACK RIGHT
LIDAR (SEMANTIC SUPERPOINT)
Figure Illustration the semantic superpiiel-toosspperpinn transformation in the proposed Seal framework. [Row & 2] The raw data captured by the multi-view camera and LiDAR sensors. [Row 4] The semantic superpixel on the camera images and superpoint formed by projecting superpixel into the point cloud via camera-LiDAR correspondence. The superpixels are generated using SEEM [123]. Each color represents one distinct segment. Best viewed in color.
FRONT_LEFT
FRONT
TRONT_RIGHT
BAFK_LEFT
BACK
BACK_RIGHT
LIDAR (RAW UNLABELED POINT)
FRONT LEFT
FRONT
FRONT RIGHT
LEFT
BACK
RIGHTRIGHT
LIDAR (SEMANTIC SUPERPOINT)
Figure B: Illustration of the semantic superpixel-to-superppiee transformation in the proposed Seal framework. [Row 2] The raw data captured by the multi-vies view camera and LiDAR sensors. [Row & 4] The semantic superpixel the camera images and superpoint formed by projecting superpixel into the point cloud via camera-LiDAE correspondence. The superpixels are generated using SEEM [123]. Each color represents one distinct segment. Best viewed in color.
For linear probing, the pretrained 3D network Fo, is frozen with trainable point-wise linear classification head which is trained for 50 epochs batch size is 16 on the nuScenes train set for all methods.
A100 GPU with learning rate of 0.05, and
For downstream fine-tuning tasks, we stick with the common practice in SLidR [85] whenever possible. The detailed data split strategies are summarized follows.
For fine-tuning nuScenes [26]. follow the SLidR protocol to split the train set of nuScenes to generate 1%. 5%, 10%, 25%, and 100% annotated scans for the training subset.
For fine-tuning on DAPS-3D [51], we take sequences "38-18_7_7" 7__22_9_99" as the training set and '38-18 9_ *42-48 10_78 78 90', and '44-18 11 15 32' as the validation set.

---

FRONT
FRONT
FRONT RIGHT
RIGHT
LEFT
BACK
RIGHT
LIDAR (RAW UNLABELED POINT)
FRONT LEFT
FRONT
FRONT RIGHT
LEFK
BACK
BACK RIGHT
LIDAR SEMANNIGSSPEEPPIIN
Illustration the semantic tuperpiicltoo-suppppontt transformation in the proposed Seal framework. [Row & 2] The raw data captured by the multi- camera and LiDAR sensors. [Row 4] The semantic superpixel on the camera images and superpoint formed by projecting superpixel into the point cloud via camera-LiDAR correspondence. The superpixels generated using SEEM [123]. Each color represents one distinct segment. Best viewed in color.
Figure C:
FRONT_LEFT
FRONT
FRONT_RIC RIGHT
BACKLEEFT LEFT
BACK
BACK RIGHT
BACK_RIGHT
LIDAR (RAW UNLABELED POINT)
FRONT LEFT
FRONT
FRONT RIGHT
BACK
BACK
BACK_RIGHT
LIDAR (SEMANTIC SUPERPOINT)
Figure D: Illustration of the semantic poperpixel-to-supeppoiii transformation in the proposed Seal framework. [Row 2] The raw data captured by the multi- view camera and LiDAR sensors. [Row & 4] The semantic superpixel the camera images and superpoint formed by projecting superpixel into the point cloud via camera- LiDAR correspondence. The superpixels are generated using SEEM [123]. Each color represents one distinct segment. Best viewed in color.
For fine-tuning on SynLiDAR [100], we use the sub-set which a uniformly downsampled collection from the whole dataset.
For fine-tuning on SemanticPOSS [75], we use sequences 00 and 01 as half of the annotated training scans and sequences 00 05, except 02 for validation to create full of the annotated training samples.
For fine-tuning on SemanticKITTI [3], Waymo Open [88], ScribbleKrTTI [95], RELLIS. 3D [49], SemantieSTF [102), and Synth4D [82], we follow the SLidR protocol to create 1%, 10%, half, or full split of the annotated training scans, e.g., one scan taken every 100 frame from the training set to get 1% of the labeled training samples. Notably, the point
25

---

cloud segmentation performance in terms of IoU is reported on the official validation sets for all the chove-mentioned datasets.
A.3.2 Experimental Setup
In our experiments, we fine-tune the entire 3D network on the semantic segmentation task using linear combination of the cross-entropy loss and the Lovász-Sofinet Softmax loss [4] as training objectives on single A100 GPU. For the few-shot semantic segmentation tasks, the 3D networks are fine-tuned for 100 epochs with a batch size of 10 for the SemanticKITTI [3], Waymo Open [88], ScribbleKITTI [95], RELLIS-3D [49], SemanticSTF [102], SemanticPOSS [75], DAPS-3D [51], SynLiDAR [100), and Synth4D [82] datasets.
For the nuScenes [26] dataset, we fine-tune the 3D network for 100 epochs with batch size of 16 while training on the 1% annotated scans. The 3D network train on the other portions of nuScenes fine-tuned for 50 epochs with batch size of 16. We adopt different learning rates on the 3D backbone Fe, and the classification head, except for the case that Fe,, is randomly initialized. The learning rate of Fo, is set 0.05 and the learning rate of the classification head is set as 2.0, respectively. for all the mentione datasets except nuScenes. On the nuScenes dataset, the learning rate of Fo, is set as 0.02.
We train our framework using the SGD optimizer with a momentum of 0.9, a weight decay of 0.0001, and dampening ratio of 0.1. The cosine annealing learning rate strategy is adopted which decreases the learning rate from its initial value to zero at the end of the training.
A.3.3 Data Augmentation
For the model pretraining, we apply two sets of data augmentations on the point cloud and the multi-view image, respectively, and update the point-pixel correspondences after each augmentation by following SLidR [85].
Regarding the point cloud, we adopt random rotation around the z-axis and flip the z-axis or y-axis with a 50% probability. Besides, we randomly drop the cuboid where the length of each side covers no more than 10% range of point coordinates on the corresponding axis, and the cuboid center located on a randomly chosen point in the point cloud. We also ensure that the dropped cuboid retains at least 1024 pairs of points and pixels; otherwise, we select another new cuboid instead. For the temporal frame, we apply the same data augmentation to it.
Regarding the multi-view image, we apply horizontal flip with a 50% probability and cropped resize which reshapes the image to 416 224. Before resizing, the random crop fills at least 30% of the available image space with random aspect ratio between 14 9 and 9. If this random cropping does not preserve at least 1024 or 75% of the pixel-point pairs, different crop is chosen.
For the downstream fine-tuning tasks, we apply a random rotation around the z-axis and flip the z- axis y-axis with 50% probability for all the points in the point cloud. If the results mentioned with LaserMix [55] augmentation, we augment the point clouds via the LaserMix before the above data augmentations. We report both results in the main paper that we can fairly compare the point cloud segmentation performance with previous works.
A.3.4 Model Configuration
For the model pretraining, the 3D backbone Fo, is a Minkowski U-Net [19] with 3 x 3 kernels; while the 2D image encoder Go, is a ResNet-50 [33] initialized with 2D self-supervised pretrained model of MoCoV2 [12]. These configurations are kept the same as SLidR [85]. The channel dimension of Go, head and Fe, head is set to 64. For the linear probing task, we adopt linear classification head, as mentioned previous sections. For the downstream fine-tuning tasks, the same 3D backbone Fe,, i.e., the Minkowski U-Net [19] with 3 x 3 x 3 kernels, is used.
26

---

SLidR
Seal (Ours)
5003
33k
35k
28x
28
Figure E: The convergence rate comparison between SLidR [85] and the proposed Seal framework.
B
Additional Quantitative Result
B.1
Self-Supervised Learning
We report the complete results (i.e., the class-wise IoU scores) for the linear probing and downstream fine-tuning tasks shown in the main paper. We report the official results from previous works whenever possible. We also report our reproduced results for random initialization, PPKT [65], and SLidR [85]. Specifically, the complete results the nuScenes [26], SemanticKITTI [3], Waymo Open [88], and Synth4D [82] datasets are shown in Table D, Table E, Table F, and Table G, respectively. We observe that Seal constantly outperforms prior methods for most semantic classes on all datasets.
We also compare the convergence rate of Seal with SLidR [85] in Fig. E. As can be seen, our framework is able converge faster with the use of semantic superpixels, where these higher qualitative contrastive samples gradually form much more coherent optimization landscape.
B.2 Robustness Probing
We report the complete results of the robustness evaluation experiments in the main paper, including the per-corruption CE scores and the per-corruption RR scores. Specifically, the complete results on the nuScenes-C [53] dataset are shown in Table and Table I. We observe that Seal is superior to previous methods in terms of both CE and RR metrics.
C
Additional Qualitative Result
C.1 Cosine Similarity
We provide more examples for the cosine similarity study in Fig. F, Fig. G, and Fig. H.
C.2
Downstream Tasks
We provide qualitative results for downstream fine-tuning tasks in Fig. I, Fig. and Fig. K.
27

---

Table D: The per-class IoU scores of different pretraining methods pretrained nuScenes [26] and linear probed or fine-tuned on different proportions (1%, 5%, 10%, 25%, and Full) the nuScenes [26] data. Symbol denotes our reproduced results and the remaining are reported scores. All IoU scores are given in percentage (%). The best mloU score is highlighted in bold.
mloU
bariier
bicycle
con veth
utI. cone
cruck
other Bar
siddwalk
tarain
munmack
bax
ped
dri
vog
Method
Linear Probing
Random 1
0.5
0.0
PoiniContrust
21.9
DepthContrast
22.1
PPKT
35.9
SLidR
39.2
44.2
30.8
00.2 15.1
22.4
47.2
27.7
34.3
80.6
21.8
35.2
48.1
71.0
71.9
ST-SI ARR
40.5
Seal (Ours)
45.0
54.7
3.6
61.7
18.9
28.8
48.1
31.0
22.1
39.5
83.8
35.4
46.7
56.9
74.7
74.7
Fine- Luning (1 (
Random
30.3
0.0
8.1
65.0
0.1
6.6
21.0
9.3
25.8
80.5
14.8
41.7
48.7
72.4
73.3
PoiniContrusi
32.5
0.0
5.8
67.4
0.0
3.3
31.8
5.6
12.1
30.8
91.7
21.9
48.4
50.8
75.0
74.6
DepthContrast
31.7
0.0
0.6
6.5
64.7
0.2
5.1
29.0
9.5
12.1
29.9
90.33
17.8
44.4
49.5
13.5
74..0
PPKT
37.8
20.7
38.4
92.5
19.2
56.8
SLidR
38.8
0.00
1.8
15.4
73.1
1.9
19.9
47.2
17.1
14.5
34.5
92.0
27.1
531.6
61.0
79.8
82.3
ST:SI aRR
40.8
0.0
16.0
74.5
3.2
25.4
50.9
20.0
40.2
30.7
54.2
61.1
82.9
Seal (Ours)
45.8
0.0
9.4
32.6
77.5
10.4
28.0
53.0
25.0
30.9
49.7
94.0
33.7
60.1
59.6
$3.9
83.4
Fine-tuning (5 %)
Random
47.8
Random 9
14.5
50.1
57.3
39.1
11.8
77.9
PPKT
62.7
56.1
8.2
05.3
79.0
9.1
15.5
54.3
34.5
26.7
58.6
03.2
44.1
63.1
64.8
85.1
83.6
SLidR
52.5
ST-SLidR
64.7
Seal (Ours)
55.6
61.0
70.4
62.4
11.9
30.4
58.2
34.0
33.6
61.1
94.7
46.2
63.4
63.9
85.7
84.9
Fine-tuning (10%)
Random
56.2
Random
53.2
56.8
5.2
66.3
74.5
5.7
36.1
49.5
38.2
29.2
54.4
91.0
47.7
61.4
67.6
82.8
81.1
PPKT* 4
67.8
56.3
62.7
68.7
85.4
SLidR
59.8
ST-SLidR
60.8
Seal (Ours)
63.0
61.9
15.1
78.9
83.5
22.5
61.0
63.0
51.5
37.4
65.2
95.2
50.4
67.7
69.6
86.2
86.4
Fine-tuning (25%)
Random
65.5
Random
22.5
65.2
69.6
PPKT
67.1
63.7
12.1
87.4
85.2
42.0
61.2
69.6
54.7
50.1
74.9
95.8
64.6
69.9
70.4
87.4
85.1
SLidR
ST-SI .dR
67.7
Seal (Ours)
68.4
67.4
15.5
90.5
85.1
40.0
62.3
68.1
58.3
54.5
76.0
95.8
65.6
69.8
71.0
67.9
86.7
Fine-tuning (Full)
Random
74.7
Random
74.9
77.4
34.4
90.4
66.5
51.1
78.4
77.0
64.9
67.8
77.7
96.6
72.2
74.3
73.2
89.6
86.7
1
74.5
75.4
91.7
45.3
75.0
65.1
65.0
79.9
96.6
72.1
73.4
86.6
SLidR
74.8
STd
75.1
Seal (Ours)
75.6
76.7
26.0
93.4
86.1
55.8
82.7
77.3
66.0
87.9
83.9
96.5
73.6
74.3
73.2
80.3
87.1
28

---

Table E: The per-class ToU scores of different pretraining methods pretrained nuScenes [26] and fine-tuned on 1% of the GemanticKITTI [3] data. Symbol denotes our reproduced results and the remaining reported scores. All IoU scores are given in percentage (%). The best mloU score is highlighted in bold.
persos
ENgyslat
parking
sidewalk
vegetation
mlot
100d
Sencee
Method
Random
PFIC
43.9
39.5
91.3
91.2
0.0
9.4
8.0
10.7
21.2
31.4
0.0
0.0
89.4
27.0
21.4
74.6
74.7
73.0
1.1
86.5
86.3
41.1
38.2
65.0
84.9
68.0
50.1
68.5
71.8
71.4
37.7
55.4
37.6
ST:SLid
Scal
46.6
92.3
14.9
18.7
16.1
23.7
43.0
34.4
0.0
91.3
27.2
70.3
0.7
86.7
33.8
86.1
61.9
71.3
57.7
47.7
Table The per-class IoU scores of different pretraining methods pretrained on nuScenes [26] and fine-tuned on 1% of the Waymo Open [88] data. Symbol 1 denotes our reproduced results and the
remaining are reported scores. All loU scores are given in percentage (%). The best mloU score is highlighted in bold.
midee light
bolding
aleeal
C00
Line
odht
Random
20.4
PPKT
47.6
92.3
48.8
51.8
69
0.0
22.0
75.0
59.8
17.6
56.8
20.4
25.6
788
91.5
80.1
698
65.1
89.1
58.6
54.7
71.4
62.6
SLidk
47.1
SESL
$3.9
49.3
Table G: The per-class IoU scores of different pretraining methods pretrained on nuScenes [26] and fine-tuned on 1% of the Symth4D [82] data. Symbol denotes our reproduced results and the remaining reported scores. All IoU scores given in percentage (%). The best mloU score is
highlighted in bold.
polcanis
bull Raho
mlbal
mith
stalee
krmin
Method
Mandom PPKT
30.2
33.2
13.8
0.0
13.9
85.6
48.9
96.7
19.4
0.0
0.2
0.0
36.2
61.1
63.5
Scal
29

---

Table H: The Corruption Error (CE) scores of different pretraining methods pretrained nuScenes [26] and probed under the eight out-of-distribution corruptions in the nuScenes- dataset from the Robo3D benchmark [53]. All CE scores given in percentage (%). The best CE score for each corruption type is highlighted bold.
Incompler Echo
Bean Missing
Wet Ground
Motion Blur
Sensor
Crossalk
Snow
Fog
Initial
Backbone
PPKT SLidR
MinkUNetta
183.44
149.50
247.53
120.50
266.03
213.54
109.62
199.03
161.65
MinkUNeets
179.38
140.47
237.29
112.95 93
276.44
210.65
107.86
189.27
160.16
Scal (Ours)
MinkUNetia
166.18
135.18
219.36
117.447
234.01
189.70 70
108.54
172.16
153.03
Random
PolarNet
115.09
90.10
115.33
58. 98
208.19
121.07
80.67
128.17
118.23
Random
FIDNet
68.78
16484
Random
CENet
112.79
71.16
115.48
61.31
156.67
159.03
53.27
129.08
153.35
Random
Wallletron
Random
Cylinder3D
105.56
83.22
111.08
69.71
165.28
113.95
71.12
110.67
116.15
Full
Random
SPVCNN,s
106.65
88.42
105.56
98.78
156.48
110.11
86.04
104.26
103.55
Random
SPVCNN,:
97.15
95.21
99.50
97.32
95.34
98.75
97.922
96. 88
98.74
Random
MinkUNetus
112.20
79.90
112.50
74.64
181.47
120.76
111.58
123.53
PPKT
MinkUNoths
105.63
77.63
101.22
68.60
160.95
111.81
86.71
108.96
123.20
SLidR
MinkUNetts
106.08
74.61
73.75
165.09
118.02
107.38
124.57
Seal (Ours)
MinkUNetia
92.63
58.97
98.17
56.63
127.25
108.20
37.97
110.95
122.63
Table I: The Resilience Rate (RR) scores of different pretraining methods pretrained nuScenes
[26] and probed under the eight out-of-distribution corruptions in the nuScenes-CC dataset from the Robo3D benchmark [53]. All RR scores given in percentage (%). The best RR score for each corruption type is highlighted bold.
Incomplere Echo
Beam Missing
Wet Ground
Motion Blur
Cross Closs-ensce Sensor
Cosstalk
mRR
Fog
Snow
Initial
Backbone
PPKT
MinkUNehas
78.15
85.38
98.66
78.33
81.36
91.42
54.37
78.02
57.69
SLidR
MinkUNeths
77.18
89.90
98.17
84.12
68.14
86.93
33.63
81.29
55.26
Seal (Ours)
MinkUNeha
75.38
83.05
95.15
66.50
83.94
89.70
45.18
83.94
55.48
Random
PolarNet
76.34
81.59
97.95
90.82
62.49
86.75
37.12
75.16
58.8 86
Random
FIDNet
73.33
90.78
67.41
33.20
Random
CENet
76.04
91.44
95.35
84.12
79.57
68.19
83.00
72.75
33.82
Random
Wafflelron
65.19
85.70
80.86
57.85
Random
Cylinder3D
78.08
83.52
96.67
79.41
76.18
87.23
61.68
81.65
58.51
Random
SPVCNN1s
35.22
Random
SPVCNN&
75.10
72.95
96.70
54.79
97.
90.04
36.71
84.84
67.35
Random
MinkUNetia
72.57
84.33
74.31
83.06
42.27
79.88
52.79
PPKT
MinkUNetia
76.06
85.90
97.71
79.28
76.72
85.72
46.77
81.31
53.10
SL.dR
MinkUNetta
75.99
87.46
96.68
74.80
84.06
56.08
81.78
52.01
Seal (Ours)
MinkUNotis
83.08
96.11
98.29
87.09
87.49
87.25
76.98
79.19
52.71
30

---

Query: "car" (near, left)
Query: "car" (near, middle)
Query: "car" (near, right)
SLIC
SAM
X-Decoder
OpenSeeD
SEEM
Figure F: The cosine similarity between the query point (denoted the red dot) and the feature learned with SLIC [1] and different VFMs [50, 122, 111, 123]. The color goes from violet to yellow denoting low and high similarity scores, respectively. Best viewed in color.
31

---

Query: "car" (near, left)
Query: "car" (near, middle)
Query: "car" (near, right)
SLIC
SAM
X-Decoder
OpenSeeD
SEEM
Figure The cosine similarity between the query point (denoted as the red dot) and the feature learned with SLIC [1] and different VFMs [50, 122, 111, 123]. The color goes from violet to yellow denoting low and high similarity scores, respectively. Best viewed in color.
32

---

Query: "car" (near, left)
Query: "car" (near, middle)
Query: "vegetation" (near, right)
SLIC
SAM
X-Decoder
OpenSeeD
SEEM
Figure H: The cosine similarity between the query point (denoted as the red dot) and the feature learned with SLIC [1] and different VFMs [50, 122, 111, 123]. The color goes from violet to yellow denoting low and high similarity scores, respectively. Best viewed in color.
33

---

Groundtruth
Prediction
Error Map
Random
SLidR
(Ours)
Random
SLidR
Seal (Ours)
Figure The qualitative results of different point cloud pretraining approaches pretrained on the raw data of nuScenes [26] and fine-tuned with 1% labeled data. To highlight the differences, the correct incorrect predictions are painted gray / red, respectively. Best viewed in color.
34

---

Groundtruth
Prediction
Error Map
Random
SLidR
Seal (Ours)
Random
SLiR
Seal (Ours)
Figure J: The qualitative results of different point cloud pretraining approaches pretrained on the raw data of nuScenes [26] and fine-tuned with 1% labeled data. To highlight the differences, the correct incorrect predictions are painted gray / red, respectively. Best viewed in color.
35

---

Groundtruth
Prediction
Error Map
Random
SLidR
(Ours)
Random
SLidR
Seal (Ours)
Figure K: The qualitative results different point cloud pretraining approaches pretrained on the raw data of nuScenes [26] and fine-tuned with 1% labeled data. To highlight the differences, the correct incorrect predictions are painted gray / red, respectively. Best viewed in color.
36

---

D Public Resources Used
We acknowledge the use of the following public resources, during the course of this work:
nuScenes
CC BY-NC-SA 4.0
nuScenes-devkit
Apache License 2.0
SemanticKITTI
CC BY-NC-SA 4.0
SemanticKITTI-AP!
MIT License
Waymo Open Dataset
Waymo Dataset License
ScribbleKITTI
Unknown
RELLIS-3D
CC B B--NC-SA 3.0
SemanticPOSS
Unknown
SemanticSTFI
CC BY-NC-SA 4.0
SynLiDAR
MIT License
Synth4D
GNU General Public License 3.0
DAPS-3D
MIT License
nuScenes-C"
CC BY-NC-SA 4.0
MinkowskiiEngine
MIT License
19
Apache License 2.0
spvnas
MIT License
Cylinder3D
Apache License 2.0
LaserMix
CC BY-NC-SA 4.0
mean-teacher"
Atribution-NonCCommrrii c.t..........iiiiiiiooooooooooaa 4.0 International
PyTorch-Lightning
Apache License 2.0
mmdetection3dd
Apache License 2.0
"nctp://semantic-kittii.om
Puttpe://saamoo.com//pee.
"nttp://uww.unner
24
37


Reviews:
[{"review_id": "ebGiu0WbIq", "reviewer": "Reviewer_EPuy", "summary": "The manuscript presents Seal, a framework that leverages VFMs (Visual Foundation Models) to segment diverse point cloud sequences in autonomous driving scenarios. The proposed approach employs VFMs to initially segment superpixels in 2D camera images and subsequently projects them to 3D superpoints. Then, it incorporates two specific pre-training schemes: spatial contrastive learning, which involves training cross-modally from pre-trained 2D features to 3D representations, and temporal consistency regularization, which ensures the consistency of 3D point features across different timestamps. Extensive experimentation validates the effectiveness and robustness of the proposed framework.", "questions": "1. What is the parameterization of a superpixel? If a superpixel is selected as a representative pixel from an area with the same class in VFM outputs, how is the pixel specifically chosen?\n2. What is the motivation behind enforcing the convergence of 3D point features to a single mean representation? In the 2D scenario, it is common for features at different pixels of the same object to be distinct, as they capture diverse parts and information about the object. Shouldn't this principle apply to the 3D case?", "limitations": "Besides the limitations the authors discuss in the manuscript, I've seen no additional specific limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The manuscript exhibits clear and concise writing, making it easy to comprehend. The mathematical explanations and figures are presented with high clarity, facilitating understanding.\n2. Through extensive experiments, the study demonstrates the effectiveness and robustness of the proposed method, establishing its reliability.\n3. The main contribution of the manuscript is the development of the framework that can serve as a valuable resource for future reference and practical implementation.", "weaknesses": "Novelty: The novelty of the manuscript may be somewhat limited. While the authors claim that the main novelty lies in leveraging VFM models for superpixel set segmentation and introducing two specific pretraining schemes, it is worth noting that the use of VFM models for segmentation is already a widely discussed topic and is short of novelty. Additionally, the proposed pretraining schemes closely follow common cross-modal contrastive learning approaches. This may give the impression that the paper is riding on the popularity of VFM models without truly exploring their potential for feature-level distillation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer EPuy for devoting time to this review and providing valuable comments.\n\n---\n> ***Q1:** \"The use of VFM models for segmentation is widely discussed. The proposed pretraining schemes closely follow common cross-modal contrastive learning approaches. The paper is riding on the popularity of VFM models without truly exploring their potential for feature-level distillation.\"*\n\n**A:** We thank the reviewer for the comment. Our response to this question is as follows:\n- Using vision foundation models (VFMs) for segmentation is among one of the hottest research topics in the computer vision community. Several recent attempts have been made to leverage VFMs (e.g. SAM [R1]) for tackling different vision tasks, such as remote sensing [R2], open-set object detection [R3], tracking [R4], image tagging [R5], medical imaging [R6], and many others.\n- It is worth noting that, different from all the above pursuits, we present the first study on utilizing  VFMs for self-supervised representation learning on large-scale 3D point clouds. We believe this work could enlighten follow-up research on designing more robust and scalable LiDAR segmentation frameworks.\n- Seal adopts the camera-to-LiDAR contrastive objective [R7] during representation learning. The knowledge from VFMs is leveraged, in the form of semantic superpixels and superpoints, to help better enforce cross-modal consistency. We agree with the reviewer that exploring feature-level distillation is also a promising solution. However, there are several limitations to conducting such a distillation using existing VFMs:\n  - *Computational overhead*. Current VFMs are often constrained by large model sizes and are struggled with high latency. Conducting feature-level distillation would require significantly more computing resources compared to the original LiDAR segmentation task.\n  - *Domain gap and representational differences*. VFMs are primarily trained for specific tasks; the features learned might not be directly transferable to the LiDAR modality due to the inherent differences in sensor data and environmental characteristics.\n  - *Loss of modality-specific information*. When using VFMs' features for distillation, modality-specific information present in LiDAR data might be lost. VFMs' features are optimized for vision-related tasks and might not capture the intricate details and characteristics unique to LiDAR data.\n\n---\n> ***Q2:** \"What is the parameterization of a superpixel? If a superpixel is selected as a representative pixel from an area with the same class in VFM outputs, how is the pixel specifically chosen?\"*\n\n**A:** Thanks for asking. In our framework, a superpixel denotes a group of semantically cohesive pixels segmented by VFMs, which is similar to the concept of a semantic mask. That is to say, a superpixel is not just one representative pixel from an area but a group of pixels that tend to belong to the same semantic class. Some examples of superpixels are shown in the first row of Figure 1, where each color represents a distinct superpixel generated by either SLIC or VFMs.\n\n---\n> ***Q3:** \"What is the motivation behind enforcing the convergence of 3D point features to a single mean representation? In the 2D scenario, it is common for features at different pixels of the same object to be distinct, as they capture diverse parts and information about the object. Shouldn't this principle apply to the 3D case?\"*\n\n**A:** We thank the reviewer for asking this insightful question. Our response is as follows:\n- The motivation for enforcing the convergence of 3D point features to a single mean representation is rooted in the distinct nature of LiDAR data. Unlike 2D scenarios, where pixel features at different locations of the same object can encapsulate various object parts and details, the complexities of LiDAR point clouds pose unique challenges:\n  - Different from 2D scenarios which contain relatively stable shape and texture cues, 3D objects and backgrounds are characterized by their spatial coherence and intricate arrangements of points. Treating each individual point as an independent entity might lead to fragmented or noisy representations that fail to capture the object's underlying structure.\n  - Here we enforce the convergence of 3D point features to a single mean representation to synthesize a cohesive depiction of the objects and backgrounds, while mitigating noise and variability intrinsic to the raw LiDAR data. This could facilitate, to a certain extent, a more stable and informative 3D representation.\n  - We validate through experiments that this single mean representation exhibits better performance than the point-wise contrastive approaches, such as PointContrast [R8] and PPKT [R9], under the linear probing, few-shot fine-tuning, and robustness evaluations (i.e. Tables 1 to 3 in the main text).\n- What is more, the per-point contrastive learning objectives, which are more popular for indoor scenes captured by RGB-D cameras, are computationally more expensive than the mean representation for handling outdoor LiDAR scenes; the automotive point clouds are both larger in scale and richer in diversity than the indoor scenes.\n\n---\n**References:**\n- [R1] Segment anything. ICCV, 2023.\n- [R2] RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Model. arXiv, 2023.\n- [R3] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. arXiv, 2023.\n- [R4] Segment and Track Anything. arXiv, 2023. \n- [R5] Recognize Anything: A Strong Image Tagging Model. arXiv, 2023.\n- [R6] Segment Anything in Medical Images. arXiv, 2023.\n- [R7] Image-to-LiDAR Self-Supervised Distillation for Autonomous Driving Data. CVPR, 2022.\n- [R8] PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding. ECCV, 2020.\n- [R9] Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining. arXiv, 2021."}, {"review_id": "SsWV47rfot", "reviewer": "Reviewer_3rsk", "summary": "This paper introduces a novel framework, called *Seal*, that leverages VFMs for self-supervised representation learning on automotive point cloud sequences. The main idea is to leverage the 2D-3D correspondence between LiDAR and camera sensors and construct high-quality contrastive samples for cross-modal representation learning. The paper also proposes a superpoint temporal consistency regularization to enforce geometric stability across different timestamps. The paper evaluates the proposed framework on 11 different point cloud datasets and shows that it outperforms previous state-of-the-art methods in both linear probing and few-shot fine-tuning settings.", "questions": "In Table 3 (Full), it seems *Seal* performs not well in `Cross`, `Echo` and `Sensor` (especially `Sensor`, 39.85 vs 49.21). Could you please provide some possible reasons about that?", "limitations": "Authors should be rewarded that the limitations are explicitly mentioned in *Discussion & Limitation*.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "**Originality:**\n - the **first** work utilizing 2D VFMs for self-supervised representation learning on large-scale 3D point clouds\n - a novel VFM-assisted contrastive learning objective that transfers the knowledge from the pretrained 2D to the 3D network at the semantic superpixel level\n - a superpoint temporal consistency regularization\n\n**Quality:**\n - well-written and organized, with clear motivation, problem formulation, related work, methodology, experiments, conclusion and discussion & limitations\n - extensive experiments on 11 different point cloud datasets and sufficient ablation studies\n - sufficient mathematical formulations and derivations to support the proposed methods\n - well-written and sufficient Supplementary Material with demo visualizations\n\n**Clarity:**\n - clear and easy to follow\n - uses consistent notations and symbols throughout the text and equations\n - helpful figures and tables to illustrate the main ideas and results\n\n**Significance:**\n - significant for both research and practice in 3D perception and representation learning\n - addresses an important problem of segmenting diverse automotive point clouds in a scalable, consistent, and generalizable manner", "weaknesses": "The datasets conducted on are all **outdoor** point cloud dataset for autonomous driving. Unless the authors conduct more experiments on **indoor** point cloud datasets, they'd better just state that \"Segment Any **Automotive** Point Cloud Sequences ...\" in their title.\n\nMinior Issue:\n- Undefined variable $L^{tmp}$ in Equ. (3)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 3rsk for devoting time to this review and providing valuable comments.\n\n---\n> ***Q1:** \"Unless the authors conduct more experiments on indoor point cloud datasets, they'd better just state that 'Segment Any Automotive Point Cloud Sequences …' in their title.\"*\n\n**A:** Thanks for your suggestion. We will seek a more proper title description in the revision. In this work, we mainly targeted segmenting point cloud *sequences*, where such sequences are often in the form of *automotive point clouds*. Nevertheless, the proposed spatial and temporal consistency regularization objectives tend not constrained by current use cases. We seek further extension of this framework to other point cloud datasets as well, such as those from indoor scenes.\n\n---\n> ***Q2:** \"Undefined variable $L^{tmp}$  in Equ. (3).\"*\n\n**A:** We thank the reviewer for pointing this typo out. We have addressed this issue in the revised paper. Here the symbol $\\mathcal{L}^{tmp}$ means to denote the temporal consistency loss, which is used to encourage consistency between the mean representation of a segment across different timestamps.\n\n---\n> ***Q3:** \"In Table 3, it seems Seal performs not well in Cross, Echo, and Sensor. Can you provide some possible reasons for that?\"*\n\n**A:** Thanks for your question. We would like to highlight that the performance discrepancy of Seal under the *'Crosstalk'*, *'Incomplete Echo'*, and *'Cross-Sensor'* scenarios in Table 3 is mainly attributed to the use of different LiDAR segmentation backbones. \n- When comparing different pretraining approaches with *the same LiDAR segmentation backbone*, i.e., MinkUNet for the last four rows in Table 3, Seal exhibits superiority over all corruption types except *‘Incomplete Echo’*, which is slightly lower than that of SLidR (59.87 vs. 61.16).\n- When comparing among backbones of different modalities, e.g., range view (CENet [R2]), raw points (WaffleIron [R3]), and point-voxel fusion (SPVCNN [R4]), the robustness against different corruption types becomes different. As been validated in Robo3D [R1], different LiDAR modalities often show diverse robustness effects. on nuScenes, the sparse voxel-based backbones like MinkUNet show sub-par performance under point cloud density changes, such as *'Crosstalk'* (extra noisy points within the mid-range areas in between two or multiple LiDAR sensors), *'Incomplete Echo'* (point miss detection for instances with dark colors), and *'Cross-Sensor'* (point clouds captured by sensors of different configurations).\n- To further solidify this observation, we will explore the robustness of Seal and other pretraining approaches (e.g. PPKT and SLidR) using LiDAR segmentation backbones from other modalities. Due to the time limit, we are only able to supplement this result in the updated paper.\n\n---\n**References:**\n- [R1] L. Kong, et al. “Robo3D: Towards Robust and Reliable 3D Perception against Corruptions.” arXiv, 2023.\n- [R2] H.-X. Cheng, et al. “CENet: Toward Concise and Efficient LiDAR Semantic Segmentation for Autonomous Driving.” ICME, 2022.\n- [R3] G. Puy, A. Boulch, and R. Marlet. “Using A Waffle Iron for Automotive Point Cloud Semantic Segmentation.” arXiv, 2023.\n- [R4] H. Tang, et al. “Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution.” ECCV, 2020."}, {"review_id": "h584Tw3Do2", "reviewer": "Reviewer_1xK9", "summary": "This paper proposed a novel framework named Seal. The Seal distills VFMs into point clouds, enabling achieves efficient segmentation of various automotive point cloud sequences without requiring extensive annotation during the pre-training stage. It exhibits excellent performance across multiple datasets.", "questions": "1. I have some questions regarding the semantic superpoint temporal consistency. Firstly, according to Figure 4, the segmentation results provided by VFMs are at the instance level, while this paper aims to obtain a pretraining model for semantic segmentation tasks. According to Equation 2, this paper treats instances of the same object as positive samples and the rest as negative samples. Therefore, there is a high possibility that negative samples may include instances of the same class. This could potentially hurt the segmentation task. On the other hand, how is the correspondence between moving objects in point cloud data at different time steps obtained?\n2. The paper mentions the ability to achieve good performance even in cases where camera and LiDAR calibration information is unavailable (on page 6 line 199). However, without approximate intrinsic and extrinsic parameters, establishing the 2D-3D spatial correspondence would be challenging. Therefore, I am curious about the performance in such scenarios. However, it appears that the paper does not provide specific experiments or results for these particular cases.", "limitations": "The statement in the limitation section that the assumption of obtaining calibrated and synchronized data between LiDAR and cameras is overly idealistic contradicts the description in the main text.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The Seal distills the feature extraction capability of visual VFMs to point clouds. This paper promotes cross-modal representation learning through the proposed spatiotemporal consistency compared to previous works. The results obtained by The Seal on 11 different point cloud datasets demonstrate its effectiveness and superiority, highlighting its significant potential for 3D feature learning.", "weaknesses": "The innovative contributions of the paper can be summarized into two points compared to previous works: 1. Distilling the representation learning capability from VFMs into point cloud data processing. 2. Introducing semantic superpoint temporal consistency to promote cross-modal learning. However, it is noted that the level of innovation in the paper might not be particularly strong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 1xK9 for devoting time to this review and providing valuable comments.\n\n---\n> ***Q1:** \"(i) The results provided by VFMs are at the instance level, while this paper aims to obtain a pretraining model for semantic segmentation tasks. (ii) The negative samples may include instances of the same class, which could potentially hurt the segmentation task. (ii) how is the correspondence between moving objects in point cloud data at different time steps obtained?\"*\n\n**A:** Thank you for the comment. \n- For question (i): The reason for performing instance-level temporal consistency has two aspects:\n  - Unlike 2D scenarios with stable shape and texture cues, 3D objects and backgrounds have spatial coherence and complex point arrangements. Treating each point separately and applying point-level temporal consistency regularization could result in fragmented or noisy representations that don't capture the object's inherent structure.\n  - The class distribution of LiDAR scenes is severely long-tailed, with a substantial portion of points belonging to static classes like 'driveable surface,' 'manmade,' and 'vegetation.' For instance, in the nuScenes training set, approximately ~90.6% of points correspond to *static* classes, leaving only around ~9.4% for *dynamic* (instance) classes. Adequate representations for static classes can be acquired through fine-tuning.\n- For question (ii): We agree that it is possible to include instances of the same class into negative samples.  In fact, it could be a common problem (i.e., negative samples), but our method is more reliable than the prior works. For one thing, the semantic superpixels generated by VFMs are semantically richer than the conventional method and also provide more complete instances. This reduces, to a certain extent, the possibility of \"self-conflict\". Besides, our method utilizes much fewer segments (\\~30) in a frame than the prior work (\\~300). We verify through experiments that our approach outperforms the prior art.\n- For question (iii): We use the following steps to associate the moving objects in different timestamps:\n  - *Coordinate transformation*. The point cloud $P^{t+n}$​ at timestamp $t+n$ is first transformed into the coordinate system of $P^t$ at $t$. This alignment is established using the sensor extrinsic and intrinsic matrices. The transformed $P^{t+n}$ is concatenated with $P^t$, creating a composite point cloud $P$.\n  - *RANSAC segmentation*. The composite point cloud $P$ is divided into ground and non-ground point groups using the RANSAC algorithm, enabling differentiation between static and moving points.\n  - *HDBSCAN clustering.* The non-ground points are then used for clustering. This provides distinct segments representing different instances in the composite point cloud $P$.\n  - *Instance labeling.* Each segment is assigned a unique instance ID, where the ground plane points are labeled as background.\n  - Consequently, we separate $P$ into $P^{t+n}$ and $P^t$ via reverse coordinate transformation. The correspondence of the same moving object across different timestamps can thus be extracted using the instance IDs.\n\n---\n> ***Q2:** \"I am curious about the performance in cases where camera and LiDAR calibration information is unavailable.\"*\n\n**A:** Thanks for the comment.\n- An accurate calibration could be important for establishing correct correspondences between LiDAR and cameras. In most cases, those sensors should be well-calibrated for an autonomous car. It is unusual if the calibration is completely unknown, but it is possible to be imprecise due to a lack of maintenance. Hence, we conduct the following experiments to validate the robustness of our method.\n\n-  For each point coordinate $(x_i, y_i, z_i)$ in a LiDAR point cloud, the corresponding pixel $(u_i, v_i)$ can be found by:\n$$[u_i, v_i, 1]^T = \\frac{1}{z_i}  \\cdot \\Gamma_K\\cdot  \\Gamma \\cdot [x_i, y_i, z_i, 1]^T,$$\nwhere $\\Gamma \\in \\mathbb{R}^{4\\times4}$ is the camera extrinsic matrix that consists of a rotation matrix and a translation matrix, and $\\Gamma_K\\in \\mathbb{R}^{3\\times4}$ is the camera intrinsic matrix. To simulate the misalignment between LiDAR and cameras, we add random noises to the camera extrinsic matrix $\\Gamma$.\n\n- **[Table 1]** Add $\\pm$1% random noises to $\\Gamma$:\n|Method|1%|5%|10%|25%\n|-|-|-|-|-\n|Random|30.30|47.84|56.15|65.48\n|PPKT|34.94|51.11|58.54|65.01\n|SLidR|37.92|53.08|59.89|66.90\n|Seal|45.23|55.71|62.62|68.13\n- **[Table 2]** Add $\\pm$5% random noises to $\\Gamma$:\n|Method|1%|5%|10%|25%\n|-|-|-|-|-\n|Random|30.30|47.84|56.15|65.48\n|PPKT|33.69|51.40|58.00|64.11\n|SLidR|38.00|52.36|60.01|64.10\n|Seal|45.66|55.42|62.77|68.01\n- **[Table 3]** Add $\\pm$10% random noises to $\\Gamma$:\n|Method|1%|5%|10%|25%\n|-|-|-|-|-\n|Random|30.30|47.84|56.15|65.48\n|PPKT|33.35|50.98|57.84|63.52\n|SLidR|37.30|51.11|58.50|64.50\n|Seal|44.80|54.45|61.80|68.29\n- **[Table 4]** Performance w/o calibration noises (from Table 1 in the paper):\n|Method|1%|5%|10%|25%\n|-|-|-|-|-\n|Random|30.30|47.84|56.15|65.48\n|PPKT|37.80|53.74|60.25|67.14\n|SLidR|38.30|52.49|59.84|66.91\n|Seal|45.84|55.64|62.97|68.41\n- From the above results we observe:\n  - The possible calibration errors in-between LiDAR and camera sensors will cause performance degradation for different pretraining approaches, i.e., Tables 1 to 3 compared to that of Table 4.\n  - The performance degradation for PPKT is especially prominent; we conjecture that this is because the point-wise consistency regularization of PPKT relies heavily on the calibration accuracy and encounters problems under misalignment.\n  - Both SLidR and Seal exhibit certain robustness; we believe the superpixel-level consistency is less sensitive to calibration perturbations. It is worth mentioning that Seal can maintain good performance under calibration error, since i) our VFM-assisted representation learning tends to be more robust; ii) we enforce also superpoint temporal consistency which not relies on the 2D-3D correspondence.\n\n"}, {"review_id": "Iz77fSrRH5", "reviewer": "Reviewer_7qT5", "summary": "This study represents the pioneering effort to utilize 2D vision foundation models for self-supervised representation learning on large-scale 3D point clouds. They introduce Seal, an innovative framework specifically designed to extract informative features from sequences of automotive point clouds. With its emphasis on scalability, consistency, and generalizability, Seal effectively captures semantic-aware spatial and temporal consistency, enabling the generation of highly valuable insights. Finally, they clearly demonstrate their superiority over previous state-of-the-art (SoTA) methods in both linear probing and fine-tuning for various downstream tasks.", "questions": "The questions are all in the weaknesses section.", "limitations": "Limitations are all in the weaknesses section.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easily comprehensible.\n2. To my knowledge, this is the first attempt to utilize large-scale vision models for aiding 3D point cloud segmentation. The authors have also put in tremendous effort, completing this work within just a short span of one to two months.\n3. Conducted extensive experiments and achieved satisfactory results on various segmentation datasets.", "weaknesses": "1. Strictly speaking, the approach in this paper is not considered unsupervised pretraining as it utilizes large models that rely on additional data. Most methods in Table 1, however, are unsupervised pretraining. Nevertheless, this distinction is not crucial, as the primary concern is to improve model performance, which is achieved through effective methods.\n2. I believe that the approach presented in the paper has not fully distilled the knowledge of large 2D models. When using the complete dataset for both pretraining and fine-tuning, the information gain from distilling large models is not significant. Why do I say this? In the method employed in this paper, the large models primarily provide semantic supervision signals. If full semantic labels are given during fine-tuning, the information gain is limited. Although Table 2 shows significant improvements for some full dataset cases, this is mainly due to the use of semantic features in distillation rather than simple semantic pseudo-labels. The advantage of distilling large 2D models is that I can use more image-lidar pairs during pretraining. I believe that if additional image-lidar pairs are used, such as on the nuscenes dataset, there will be further improvements. I also hope to see similar experiments, such as pretraining with ten times the number of image-lidar pairs compared to nuscenes.\n3. The method proposed in this paper is expected to be very general, so it should yield better results for other 3D perception tasks (e.g., 3D object detection).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 7qT5 for devoting time to this review and providing valuable comments.\n\n---\n> ***Q1:** \"The approach in this paper is not considered unsupervised pretraining as it utilizes large models that rely on additional data.\"*\n\n**A:** Thanks for the comment. We agree with the reviewer that this approach is not strictly unsupervised, since the vision foundation models (VFMs) used are trained with image annotations. We also believe that this distinction is not crucial as the use of VFMs is becoming prevailing and can be considered as, to a certain extent, *\"off-the-shelf\"*. To avoid any potential misunderstanding, we will revise the tone of the elaboration and try to avoid using *\"unsupervised\"* in the revised manuscript.\n\n---\n> ***Q2:** \"I believe that if additional image-lidar pairs are used, there will be further improvements. I hope to see similar experiments, such as pretraining with ten times the number of image-lidar pairs compared to nuScenes.\"*\n\n**A:** Thanks for asking this insightful question. Our response to this question is as follows:\n- We agree with the reviewer that if sufficient semantic labels are given during the fine-tuning, the gain from model pretraining would become limited. In fact, this is also the case for many 2D pretraining tasks. Nevertheless, as verified in the experiments, the use of pretraining still exhibits advantages over random initialization, such as higher final performance, faster convergence rate, and better out-of-training-distribution robustness.\n- The improvements over some full dataset cases in Table 2 are mainly credited to the use of the large-scale pretraining dataset, i.e., nuScenes, compared to these small-scale downstream datasets (SemanticPOSS, SemanticSTF, and DAPS-3D).\n- An experiment on pretraining with much more image-LiDAR pairs is indeed desirable. Since nuScenes is already one of the largest LiDAR datasets, we seek the following two resources to consolidate the above observation:\n  - The *sweeps* data of nuScenes, which contains unannotated image-LiDAR pairs around nine times larger (around a total of 260k) than the used keyframes.\n  - The *Waymo Open* dataset, consisting of around a total of 150k image-LiDAR pairs from 64-beam LiDAR and five camera sensors. The point clouds are denser than that of nuScenes.\n- Due to the time limit, we are only able to include these results in the revised manuscript. We will update our progress (if any) during the Author-Reviewer discussion section.\n\n---\n> ***Q3:** \"The method proposed in this paper is expected to be very general, so it should yield better results for other 3D perception tasks (e.g., 3D object detection).\"*\n\n**A:** Thanks for your suggestion. Our response to this question is as follows:\n- We agree with the reviewer that the pretrained weights should be general for related 3D perception tasks as well. We consolidate this statement by extending Seal (and existing LiDAR representation approaches [R1,R2]) to 3D panoptic segmentation -- a task that requires predicting both semantics and instance identities of LiDAR points.\n- The results of different pretraining approaches pretrained on nuScenes (the same configuration as LiDAR semantic segmentation in the paper) and fine-tuned on the Panoptic nuScenes dataset [R3] under 5%, 10%, and 20% splits are as follows:\n|Method|PQ (5%)|mIoU (5%)|PQ (10%)|mIoU (10%)|PQ (20%)|mIoU (20%)|\n|-|:-:|:-:|:-:|:-:|:-:|:-:\n|Random|35.60|45.56|44.10|54.80|51.29|62.29\n|PPKT|47.73|53.16|53.63|58.95|56.25|64.42\n|SLidR|44.68|52.92|51.44|60.00|55.64|65.67\n|Seal|49.20|55.33|55.01|62.74|57.73|66.94\n- We use the pretrained MinkUNet as the semantic extractor in the 3D panoptic segmentation pipeline. The instance extractor consists of an instance head, followed by a clustering step. This step leverages the semantic predictions to filter out *stuff* points, retaining only those belonging to *thing* instances, such as ‘pedestrian’, ‘car’, and ‘bicyclist’. Subsequently, the remaining points are subjected to clustering to identify the instances based on the features from the instance head. Similar to [R4], the mean shift clustering is employed for this purpose, where the bandwidth is set to 2.5, and the minimum number of points per cluster is set to 20.\n- Following the conventional reporting of PQ (panoptic quality) and mIoU, we observe from the above table that Seal also shows superiority over existing LiDAR pretraining approaches in the 3D panoptic segmentation task. The results verify that the pretrained weights can be general across different 3D perception tasks.\n- To further solidify this claim, we seek another extension to the 3D object detection task. Due to the time limit, we will include the results in the revised manuscript.\n\n---\n**References:**\n- [R1] Y.-C. Liu, et al. \"Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining,\" arXiv, 2021.\n- [R2] C. Sautier, et al. \"Image-to-LiDAR Self-Supervised Distillation for Autonomous Driving Data,\" CVPR, 2022.\n- [R3] W. K. Fong, et al. \"Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Tracking,\" RA-L, 2022.\n- [R4] F. Hong, et al. \"LiDAR-based Panoptic Segmentation via Dynamic Shifting Network,\" CVPR, 2021.\n"}]

Author rebuttal:

Meta-review: The paper received 4 detailed reviews, and the authors did a serious job in addressing all the reviewer comments and suggestions. This includes running new experiments (e.g. performance analysis and generalization to other 3D perception tasks). Overall, the general view of the reviewers was quite positive and leaning to accept. The rebuttal addressed the majority of their concerns.
Decision: Accept (spotlight)