Fine-grained Image-to-LiDAR Contrastive Distillation with Visual Foundation Models
Authors: Yifan Zhang
Junhui Hou
Venue: NeurIPS 2024 poster (2024)
Keywords: self-supervised learning
cross-modal distillation
3D representation learning
Primary area: machine_vision

Abstract: Contrastive image-to-LiDAR knowledge transfer, commonly used for learning 3D representations with synchronized images and point clouds, often faces a self-conflict dilemma. This issue arises as contrastive losses unintentionally dissociate features of unmatched points and pixels that share semantic labels, compromising the integrity of learned representations. To overcome this, we harness Visual Foundation Models (VFMs), which have revolutionized the acquisition of pixel-level semantics, to enhance 3D representation learning. Specifically, we utilize off-the-shelf VFMs to generate semantic labels for weakly-supervised pixel-to-point contrastive distillation. Additionally, we employ von Mises-Fisher distributions to structure the feature space, ensuring semantic embeddings within the same class remain consistent across varying inputs. Furthermore, we adapt sampling probabilities of points to address imbalances in spatial distribution and category frequency, promoting comprehensive and balanced learning. Extensive experiments demonstrate that our approach mitigates the challenges posed by traditional methods and consistently surpasses existing image-to-LiDAR contrastive distillation methods in downstream tasks. We have included the code in supplementary materials.

Fine-grained Image-to-LiDAR Contrastive Distillation with Visual Foundation Models
Yifan Zhang and Junhui Hou"
Department of Computer Science, City University of Hong Kong yzhang3362 cêmy, ci cityu. edu edu. hh; jh. edu. hk
Abstract
Contrastive image-to-LiDDR knowledge transfer, commonly used for learning 3D representations with synchronized images and point clouds, often faces a self- conflict dilemma. This issue arises contrastive losses unintentionally dissociate features of unmatched points and pixels that share semantic labels, compromising the integrity of learned representations. To overcome this, we harness Visual Foun- dation Models (VFMs), which have revolutionized the acquisition of pixel-level semantics, enhance 3D representation learning. Specifically, pixel-to-point the-shelf VFMs to generate semantic labels for contrastive distillation. Additionally, employ Mises Fisher distributions structure the feature space, ensuring semantic embeddings within the same class remain consistent across varying inputs. Furthermore, we adapt sampling probabil- ities points to address imbalances in spatial distribution and category frequency, promoting comprehensive and balanced learning. Extensive experiments demon- strate that our approach mitigates the challenges posed by traditional methods and consistently surpasses existing image-to-LiDAR contrastive distillation methods downstream tasks. Code is available at https//gtthbb.om//aaaaaa/./...
utilize off-
Introduction
LiDAR sensors deliver critical information in the 3D environment, crucial for applications such as autonomous driving [78, 75, 79]. State-of-the-art neural networks have shown promising performance on point clouds processing, which rely extensive annotated datasets [29, 77, 17]. However, the process of annotating point clouds is not only time-consuming but also costly, presenting significant challenges in terms of scalability and practicality [36]. Self-supervision offers solution by leveraging vast quantities of unlabeled data to pre-train networks and subsequently fine-tuning them with a smaller, labeled dataset. This approach significantly reduces the reliance extensive annotated data sets [10].
A prevalent method for learning 3D representations involves contrastive pixel-to-point knowledge transfer, using synchronized and calibrated images and point clouds. PPKT [39] enables 3D network derive extensive knowledge from pre-trained 2D image backbone through pixel- to-point contrastive loss. This entire pre-training process necessitates no annotations for either images point clouds. Then SLidR [50] employs superpixels to cluster pixels and points from visually coherent regions, leading to more meaningful contrastive task. Building on this, Seal [38] utilizes semantically rich superpixels generated by visual foundation models and introduces temporal consistency regularization across point segments at different times. Meanwhile, HVDistill [73] innovates by implementing cross-modality contrastive distillation that integrates both image-plane and bird-eye views.
Cooresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Unfortunately, existing contrastive distillation methods are hindered by several critical limitations. Firstly, "self-conffict" issue arises during pre-training, where (super)pixels that belong to the same category as an anchor (super)point but do not directly correspond are simply treated as negative samples (see (a).. This approach neglects the inherent semantic connections within the same cat- egory, leading to conflicts in the learning process where beneficial relationships might be disregarded. This problem is magnified by the contrastive loss's intrinsic hardness-awe aware characteristic, which results in the most substantial gradient influences derived from negative samples that are semantically the most similar [59, 41J. While ST-SLidR [41] has introduced a semantically tolerant loss to mitigate this issue, the absence of robust high-level semantic understanding could not fundamentally change
the intrinsic hardness-aware nature of the con- trastive loss. Secondly, conventional sampling methods for point-pixel pairs fail to consider significant category imbalances or variations in point densities relative to their distance from sensors [39]. For example, bicycles comprise only 1.47% of annotations in the nuScenes- lidarseg dataset, whereas drivable surfaces make up 37. 37.66%. This oversight can result in a skewed representation of the environment, where dom- inant categories or densely populated areas are over-represented. impacting the model's effective- ness and fairness.
Anchor Point
Pull
Push
(a)
(b)
In this study, we address the "self-conffict" issue by leveraging supervised contrastive distillation enhanced with weak semantic labels generated by VFMs (Visual Foundation Models). VFMs like SAM (Segment Anything Model), trained on expansive datasets, has revolutionized computer vision by simplifying the acquisition of pixel-level semantics. These models are exceptionally adaptable, enabling the direct derivation of semantic labels through specified prompts without the need for retraining. As depicted in Fig.I(b), with these weak labels, we draw the embeddings of an anchor point and corresponding pixels of the same class closer together, while distancing the anchor from "negative" " pixels of differing classes. Furthermore, given that representation of samples in the same class can vary significantly across different batches, we introduce semantic-guides consistency regularization to enhance 3D representation learning. This approach structures the feature space by modeling each class with von Mises-Fisher distribution and making point features adhere closely to their respective distributions.
Figure 1: Illustration of (a) self-conflict that ex- ists in conventional pixel-to-point contrastive dis- tillation and (b) our weakly supervised contrastive distillation.
Considering the challenges posed by category imbalance and the non-uniform distribution of point clouds, we propose a density and category-aware sampling strategy. This method accounts for both the density of points and the frequency of their categories. By adjusting the sampling probabilities of different anchor points, we enhance the quality of learned 3D representations, particularly for points that fall into minority categories or are situated in areas of low density.
Extensive experiments reveal that our pre-training approach outperforms state-of-the-art 3D self- supervised methods on both the nuScenes and SemantickITTI datasets. The primary contributions of this work are summarized as follows: 1) To tackle the challenge of "self--onflict", we utilize off-the-shed VFMs to generate semantic labels for
pixel-to-point contrastive distillation. 2) We introduce semantic-guided consistency regularization to cultivate a meaningful and structured feature space. 3) We develop an innovative sampling strategy for point-pixel pairs that considers both the category frequency and spatial density of points.
2 Related Work
3D Representation Learning. Recent advancements in 3D self celf-supervised learning have closely paralleled innovations the image domain, extending these methods diverse 3D contexts such as object-level point clouds [49, 60, 23), indoor scenes [61, 12, 81, 68, 9, 32, 69, 58], and outdoor settings [34, 4, 71, 42, 18]. These techniques are grounded in contrastive learning [66, 71, 42], mask modeling [72], and other pretext tasks [4, 76]. PPKT [39] utilized the InfoNCE loss to facilitate the 3D network in distilling rich knowledge from the 2D image backbone. Sautier et al. pioneered
superpixelto-superppe contrastive loss for self-supervised 2D-to-3D representation distillation.

---

Ieputs
BacAbone
Flukkly Sww.hliuuddddddddddd Corpustre
DGAS Dently Canegey-anane
$20
Po anl Pairs
Sell Su Suporv Contrastie
Semantic Label Generatio by VPM
Prompt bus, road,
JaD
Segment Anyhhing Model
Semards Consissency Regutatioionn
Ieferwenoe
Keek Labe's
Point Festuress
Braege Finelune
Anchor
MMP
Magping
Tunad Weighta C
Frozen We ghis
Pull
Fush
Group by Class
Param
Updatn
The overall pipeline of our proposed OLIVINE. The pipeline starts with feature extraction via a trainable 3D backbone and pre-trained 2D backbone, followed by feature alignment in common space, The learning driven by weakly-supervised contrastive distillation with coarse semantic labels, self-supervised distillation of randomly sampled point-pixel pairs, and semantic consistency regularization through the von Mises-Fisher distribution. Besides, our approach also characterized by the novel sampling strategy of point-pixce pairs addressing spatial and category distribution imbalances.
Figure 2:
Building on this, Mahmoud et al. [41] enhanced the approach by incorporating semantically tolerant contrastive constraint and class-balas basssbbaanniin loss. Liu et al.[38] further refined these techniques through semantic-aware spatial and temporal consistency regularization, advancing feature learning. Moreover, Zhang et al. [73] explored cross-modallns contrastive distillation across not only the image plane but also the bird-eye views.
Visual Foundation Models. The advent of powerful visual neural networks, trained on extensive
datasets [46, 27] or through cutting-edge self-supervised learning techniques [7, 11, 21], catalyzed significant advancements within the community. Notably, the Segment Anything Model (SAM) [27]
initiated a new paradigm in general-purpose image segmentation, demonstrating remarkable zero-shot transfer capabilities across a variety of downstream tasks. Building on this, Grounded-SAa SAMM [47] enhanced the model by incorporating elements from Srounding-DINO [37], open-set object detector capable of recognizing and classifying previously unseen objects during training [37]. In our work, leverage the inherent semantic awareness of these VFMs [84] to generate weak semantic labels, which are crucial for our supervised contrastive distillation framework.
3D Scene Understanding. Traditional approaches to 3D scene understanding primarily utilize paradigms based on raw points [55, 16, 8, 62]. voxels [17, 13], range views [29, 56), and multi-view fusion [19, 67]. Despite the effectiveness of these methods in capturing detailed environmental fea- tures, they are significantly constrained by their dependence on large-scale annotated data. Acquiring and annotating this data is both time-consuming and costly, limiting the scalability of 3D segmentation models [36]. To reduce the reliance on large annotated datasets, recent studies have also turned to semi-supervised [31, 22], weakly-supervised [35, 15], and active learning techniques [40, 65].
3 Proposed Method
Overview. As depicted in Fig. 2, our method, namely OLIVINE, integrates visual foundation model for pre-training with paired point clouds and images. Feature extraction is conducted using a trainable 3D backbone for point clouds and pre-trained 2D backbone for images, with these features then mapped into a common feature space via decoupled projection heads for both point-pixel level and category-level contrastive distillation. The representation learning in OLIVINE is driven by three objectives: weakly-supervised contrastive distillation using coarse semantic labels to identify positive pairs by category, self-supervised contrastive distillation applied to randomly sampled point-pixes pairs, and a regularization framework based on the von Mises-Fisher distribution to ensure semantic consistency. Additionally, address imbalances in spatial distribution and category frequency through a targeted sampling strategy, ensuring a balanced representation in the learning process.
Notation. Let P = {1I, P2, ..., PN pp E R3 } be point cloud consisting of N points collected by LiDAR sensor, and I (I e == I,.... Neam) multi-view images captured by Neam synchronized cameras, where Le E R X x3 is single image with height H and width W.
3

---

3.1 Baseline Architecture
We follow the existing work [39] to perform the basic point-to-pixel contrastive distillation, based on which we build our whole pipeline. Starting with point cloud and image inputs, we employ distinct encoders for feature extraction. The 3D features are extracted using an encoder faD(-) NN3 + RN which processes the point clouds to produce features of dimension CaD per point. For encoder f-D(-) Hxxx>x3 + R # xW' xCaD initialized with weights from pre-trained image models. This setup facilitates knowledge transfer from the 2D domain to the 3D domain through contrastive learning. For the computation of contrastive loss, we design trainable projection heads, h2g for 2D features and hg for 3D features, both aligning the features into unified dimensional space. Specifically, the 3D projection head kgD is linear layer with lg fg-normalization, converting 3D features to a normalized C--dimensional space. Similarly, the 2D projection head h.p. a convolution layer with I x kernel followed by bi-linear interpolation layer adjusting the spatial dimensions by factor of 4, also applies Eg fz-normalization
image features,
Utilizing the calibration matrix, establish dense point-to- pixel correspondences { F?DDM where F.D and F.D represent the paired features of points and images for the i-th pair, with M denoting the total count of such valid pairs. Previous methods achieve cross-modal knowledge transfer by attracting positive pairs and repelling negative pairs within the feature space, employing the InfoNCE loss [43]. The point-pixel level contrastive loss is defined as
Ji=l-
1
M. Σ log
expl ( F?DD //2D)/T) M, j=l exp( F2D) //))//
LPPNCE M,
(1)
where 7 is the temperature factor, M, is the number of sampled corresponding point-pixel pairs, (:, denotes the scalar product measure the similarity between features.
3.2 Weakly-supervised Contrastive Distillation
Existing methods [39, 50, 38] often directly treat unmatched points and pixels that share semantic labels as negative pairs. This practice overlooks the intrinsic semantic connections within the same categories, leading to potential conflicts in the learning process where beneficial relationships are ignored. To address this, we utilize the Segment Anything Model, which adeptly interprets and translates semantic cues from text prompts into precise semantic segmentation of images. This application of SAM allows us to generate high-quality semantic labels without repetitive training, enhancing the learning process. We represent these labels as y = (0 l i==1 where each label corresponds to a specific point-pixel pair.
M
In point-pixe pixel level contrastive loss, the pixels that belong to the same category but do not correspond the given anchor point are taken as negative samples (see Eq. (I)). Therefore, we argue that the 2D and 3D features used for weakly-supervisee contrastive learning, which take the category information into consideration, should differ from the features F3D and FED that represent the individual points and pixels. To address this issue, we apply another two heads h IR xw' xC2n + RR HXCC and herm Naan + R xC extract the semantic-level feature embeddings C2D and GBD.
For the sampled points and pixels, we use their semantic labels to identify positive and negative pairs. Positive pairs are defined as point and pixel features that share the same semantic label, whereas negative pairs are those with different labels [26]. The
contrastive loss is defined
Caup
1
M, Σ log
Σ (E)(()
exp( G2D //T)/T)
M&
AA(I)
M, Σ exp( G.D)/T)
where A(i) denotes the set of indices of matched point-to-pin pairs in the batch that have the same class with i-th point-pixel pair, and [A(i) indicates its cardinality.
3.3 Semantic-guided Consistency Regularization
We advocate that the construction of latent semantic structures in feature space could enhance representation learning. By leveraging semantic labels derived from SAM inference, we organize points with identical semantic labels into coherent groups. This grouping promotes feature consistency within these semantic categories, thereby stabilizing the learning of feature representations across varied data instances and yielding structured feature space.

---

Distribution Assumption. Intuitively, the point features extracted by the projection head 3 from the same class should exhibit similarity in the feature space. For the purposes of contrastive learning. these features are normalized to exist on the unit hypersphere. Consequently, we model the point features of each class is a spherical adaptation of the normal distribution, suitable for data constrained to hypersphere [33]. Here, μA represents the mean direction, while Kş is the concentration parameter, indicating the degree to which category features are concentrated around ua.. The probability density function for the vMF distribution, applicable to a random C-dimensional unit vector 2, is formulated as follows:
von Mises-Fisher (vMF) distribution vMF jKk, K+))- This distribution
Jo(æ; (Kk N&) = (c((k)) exp(#x 2).
where k 2 0 and ||l|| 1. The normalization constant (c(K) is defined as:
C/2-1
Kk
Kc(ks) = (2x) (1/2-))
(4)
2m+-/22-1
[())-1)()== E mC 1) 2
(5)
m==
where I(C/2-1) is the modified Bessel function of the first kind at order C/2 1. The distribution exhibits a higher concentration around the mean direction 14 as K2 increases, and becomes uniform on the hypersphere when K3:- = 0.
Parameter Updating. Specifically, conduct the semantic-guided consistency regularization in a two-stage framework. First, we update the parameter of vMF(2; ua, kx)) with the features GDD extracted from point cloud branch. During training, we obtain the statistical value of feature embeddings via the EMA (Exponential Moving Average) algorithm, following:
= az + (1 - +)
where = Mr E. 1 -k)= G denotes the sample mean of class i at t-th mini-batch, is the fixed smoothness coefficient. The maximum likelihood estimates of the mean direction μ4 is simply the normalized arithmetic mean:
μL - Rs: where Rs = -| And the concentration parameter Ky could be obtained by the solving the equation:
Ac(R) RE:
(8)
where Ac(s) = Icc2(())/o//211ll Sra [52] proposed a simple method to estimate the Ky:
R4 Rs (C
R?
R2
And we model the features of each observed point as Spherical Dirac Delta distribution during training:
8(2 - 3D { 0, + G 3D
(10)
2 = 3D
D0,
Loss Function of Regularization. At the second step, could perform the semantic- semantic-guided consistency regularization by pulling the points features and distribution of its corresponding category vMF (ky Kk) with Kullback-Leibler (KI.) Divergence loss:
(2; Cu M (klz -G:D)) ([ (2: kk))
1
Σ
1 M
10f= log fo
3D : Pk KK)
M
K
( - log (x(())) - KR G
(11)
M
In summary, the overall loss function for pre-training written as L L1 + d2ZSupp Ng where X1, X2, and Ag are the weights to balance the three terms.
5

---

3.4 Density and Category-aware Sampling Strategy
Previous methods [39] neglect the spatial distribution and category frequency imbalances in the sampling of point-pixel pairs for contrastive distillation. To overcome these challenges, we introduce novel sampling strategy that utilizes both the distance of each point from the LiDAR sensor and the occurrence frequency of its category. First, we calculate the distance of each point in the point cloud from the LiDAR sensor. We then apply kernel density estimation (KDE) these distances to determine the probability density function of the spatial distribution of points, Given point, its density can be calculated using the formula based on its distance d from the LiDAR sensor:
(n(d) = Mh
Σ 1=1
Kh
(12)
h
where d, denotes the distance of point p; from the LiDAR sensor, h is the bandwidth, K2, is the kernel function. This density estimation helps us understand how densely points are distributed with respect to their distance from the sensor, which is crucial for addressing that may bias the learning process.
of high point concentration
Simultaneouslyy we assess the frequency of each category in the valid point-pixel pairs. By count- ing the occurrences of each category, we can identify which categories are underrepresented or overrepresented in the dataset.
Combining these two dimensions of analysis, we define the sampling probability for each point as inversely proportional to both its KDE-derived density and its category occurrence frequency. Mathematically. the sampling probability for point p, is given by:
(13) By implementing this sampling strategy, we aim to ensure a more uniform representation of both spatial and categorical dimensions in our contrastive learning setup. This method reduces the risk of overfitting to dense clusters of points or to frequently occurring categories, therehy fostering more robust and generalizable representation learning.
p(p:) 1/(f. (d.) IA(1)))
4
Experiments
4.1 Transfer on Semantic Segmentation
We evaluate the learned representations for semantic segmentation on nuScenes- lidarseg and SemanticKITTI datasets. The nuScenes-lidarse and SemanticKITTI datasets contain 16 and 19 semantic categories for validation, respectively. We modify the network by adding a 3D convolutional layer to the pre-trained backbone as the segmentation head. Basically, we finetune the whole network on different portions of annotated data. Following previous works [50, 41], we finetune network for 100 epochs with a batch size 10 and on SemantieKITTI and nuScenes- lidarseg, respectively. The initial learning rate of the backbone and the segmentation head are set to 0.05 and 2.0, respectively. When utilizing 1% of the annotated data, the network is fine-tuned for 100 epochs, whereas for other percentages, it is fine-tuned for 50 epochs. In another protocol, evaluate the quality of learned representation by linear probing. Different from the setting of finetuning, we optimize only the added segmentation head and keep the weights of backbone fSD frozen the
Evaluation Protocol.
Table 1: Comparison of various pre-training techniques for semantic segmentation tasks using either finetuning or linear probing (LP). This evaluation uses different proportions of accessible annotations from the nuScenes or SemanticKITTI datasets and presents the mean Intersection over Union (mloU) scores on the validation set.
nuScenes
KITTI
Initialization
Present at
1%
5%
10%
25%
100%
1%
Random
8.1
30.30
47.84
56.
65.48
74.66
39.50
PointContrast [66]
ECCV'20
21.90
32.50
41.10
DepthContrast [80]
ICCV'21
22.10
31.70
41.50
PPKT [39]
Arxiv'21
35.90
37.80
53.74
60.25
67.14
74.52
44.00
SLidR [50]
CVPR'22
38.80
38.30
52.49
59.84
66.91
74.79
44.60
ST-SLiaR [41]
CVPR'23
40.48
40.75
54.69
60.75
67.70
75.14
44.72
Seal [38]
NeurIPS'23
44.95
45.84
55,64
62.97
68.41
75.60
46.63
Ours
Neur
50.09
50.58
60.19
65.01
70.13
76.54
49.38

---

(e) Ground-Truth
Random
(c) PPKT
SLidR
(c) Ours
Figure 3: The visual results of various point cloud pretraining strategies, pre-trained nuScenes and fine-tuned using merely 1% of annotated data, are displayed. To illustrate the distinctions, we mark correctly predicted areas in gray color and incorrect ones in red.
Table Finetuning results on SemanticKITTI across various percentages of annotated data, The table compares the improvement achieved by our method relative the SLidR [50].
Initialization
1%
5%
10%
20%
100%
Random
39.5
45.7
51.5
56.0
56.9
SLidR [50]
44.6
+5.1
54.7
49.0
56.3
+4.8
56.7
+0.7
57.1
+0.2
Ours
49.4
+9.9
57.5
+11.8
60.3
+8.8
60.9
+4.9
61.1
+4.2
uuScenes-lidarsee dataset. For both protocols, the training objective is a linear combination of the cross-entropy loss and the Lovász-Softmnn loss [3].
Results of Linear Probing. Under the linear probing scenario, our method achieves the highest mloU of 50.09%, surpassing the previous state-of-the-arl method, Seal [38], which records a mloU of 44.95% (see Table 1). This performance indicates significant improvement in extracting useful features directly from pre-trained models without additional training of the 3D backbone.
Results of Fine-tuning. For the fine-tuning on nuScenes, our consistently excel, particularly at smaller data proportions. With only 1 % of the training data of the nuScenes, our method achieves mloU of 50.58%. This trend persists across other data proportions, with our method consistently leading or closely competing with the best results, particularly at 60. 19% mloU with 5% of the data and 65.01% % mloU with 10% of the data. The qualitative results are presented in Fig. 3.
We also fine-tuned various models on the SemanticKITTI dataset to assess their performance across a spectrum of annotated data availability, from as low as 1 % to full data utilization (see Table 2). Besides, as shown in Table 3, our method consistently achieves state-of oftth--f--heartt performance on another six datasets.
4.2 Transfer on 3D Object Detection
Evaluation Protocol. In evaluating our pre-trained model for 3D object detection, we utilized two prominent architectures: SECOND [70] and PV-RCNN [51]. Both are built the VoxeINet 3D backbone [82], which processes voxels via 3D sparse convolutions and includes a 2D backbone for bird's-eye-vies s-eee-vvewww encoding after BEV projection. The primary distinction between the architectures is in their detection heads. SECOND uses a region proposal network (RPN) directly on the 2D
Table 3: Evaluation of various pretraining methods initially trained the nuScenes dataset and subsequently fine-tuned on multiple downstream point cloud datasets. The mloU scores are presented percentages (%).
ScribbleKITTI
RELLIS-3D
SemanticPOSS
SemantieSTF
SynLiDAR
DAPS-3D
Method
1%
10%
1%
10%
50%
100%
50%
100%
1%
10%
50%
100%
Random
23.81
47.60
38.46
53.60
46.26
54.12
48.03
48.15
19.89
44.74
74.32
79.38
PPKT [39]
36.50
51.67
49.71
54.33
50.18
56.00
50.92
54.69
37.57
46.48
78.90
84.00
SLidR [50]
39.60
50.45
49.75
54.57
51.56
55.36
52.01
54.35
42.05
47.84
81.00
85.40
Seal [38]
40.64
52.77
51.09
55.03
53.26
56.89
55.36 53.46
43.58
49.26
81.88
85.90
Ours
44.91
54.96
53.47
58.21
55.70
58.51
56.65
60.42
46.34
52.78
83.63
86.84

---

Table Comparison of our method with other pre-training techniques through tnnettuning on the KITTI dataset. The results reflect the 3D object detection performance under moderate difficulty on the validation set.
Detectors
Initialization
Car
Pedestrian
Cyclist
mAP@40
Random
81.5
50.9
66.5
66.3
PPKT [39]
81.8
51.4
68.2
SECOND 170]
SLidR [50]
81.9
51.6
68.5
67.3
Ours
82.0
53.2
69.8
68.3
Random
84.5
57.9
71.3
71.3
STRL [24]
84.7
S7.8
71.9
71.5
PV-RCNN [51]
PPKT [39]
83.2
55.5
73.8
70.8
SLidR [50]
84.4
57.3
74.2
71.9
Ours
84.8
59.3
74.2
72.8
backbone, while PV-RCNN refines RPN predictions with fine-grained keypoint features, enhancing bounding box accuracy and confidence in estimations.
In the fine-tuning stage, integrate the detection head of SECOND or PV-RCNN with the pre-trained backbone (VoxelNet). This integrated detector is then fine-tuned on the train data of KITTI [20], which includes implementations of these detectors and follows the standard training parameters specified by OpenPCDet [54]. We conduct the fine-tuning three separate times, and report the highest mean Average Precision (mAP) recorded the KITTI validation set.
Results. The experimental results detailed Table 4 showcase the performance of various initializa- tion strategies. When using the SECOND architecture, our method outperforms other pre-training techniques. Starting from a baseline with random initialization, the performance improves consistently with more specialized pre-trained weights like PPKT and SLidR, and ultimately, our method achieves highest mAP at 68.3%. Significant gains are observed across all categories, with particularly notable improvements in detecting pedestrians and cyclists. Similarly, the PV-RCNN architecture benefits from refined initialization methods. Our method again yields the highest overall mAP@ at 72.8%, surpassing the performance of SLidR.
Remark. Compared the semantic segmentation task, the model architecture for object detection more complex. Besides the 3D backbone, 3D detectors typically project features to a BEV plane, followed by 2D convolutional network and Rol operations. These crucial components were not pre-trained, which may limit the overall performance gain from pre-training approach. It's important to note that semantic segmentation and object detection use different metrics and scales, making direct performance comparisons improper. The nature of these tasks and their evaluation criteria inherently lead to varying degrees of improvement when applying our proposed method.
Table 5: Ablation study of each component pre-trained on nuScenes and fine-tuned on nuScenes- lidarseg and SemanticKITTI. WCD: Weakly-supervised Contrastive Distillation. SCR: Sematic- guided Consistency Regularization. DCAS: Denstiy and Category Sampling.
nuScenes
SemanticKITTI
Exp.
WCD
SCR
DCAS
LP
1 1%
5%
10%
25%
100%
1%
1
X
X
X
36.87
37.89
53.15
60.33
67.03
74.59
44.12
2
V
X
42.10
42.33
55.22
61.53
67.70
75.06
45.58
3
X
V
X
40.58
41.99
54.49
60.98
67.69
74.88
45.67
4
V
V
44.76
44.91
56.01
63.12
68.74
75.48
46.60
5
V
X
V
44.15
43.96
55.75
62.49
68.13
75.07
46.07
6
V
V
47.30
46.12
57.51
63.04
69.39
76.13
47.35
4.3
Ablation Studies
Effect of Key Components. Table 5, we investigate the effect of each added component in our method. The integration of Wreakly-supervise Contrastive Distillation alone yields a significant increase performance, improving mloU by 5.23% for linear probing. Similarly, incorporating Sematic-guided Consistency Regularization also enhances model performance, delivering a 7. % increase in mloU. When these components are combined, they synergistically contribute further mloU gain of 7.89% for linear probing. Additionally, the application of Denstiy and Category-aware

---

Table Comprehensive ablation studies for the key components. We report the fine-tuned results on nuScenes- mideenne-lidarseg and SemanticKITTI (S.K.) datasets with 1% of the labeled data.
(a) Different types of supervision utilized for con- (c) Different distributtins to model semantic features.
trastive distillation.
Distribution
nuScenes
S.K.
Method Baseline
nuScenes
S.K.
Deterministic
44.15
45.98
37.89
44.12
vMF
46.12
47.35
w/ Weak label
46.12
47.35
Improvement
+1.97
+1.37
w/ GT label
57.72
51.56
(d) Different sampling strategies.
(b) Different architectures of projection heads.
Sampling
nuScenes
S.K.
Heads
nuScenes
S.K.
Random
44.91
46.60
Not Decoupled
42.59
45.46
Density-aware
45.33
46.77
Decoupled
46.12
47.35
Category-aware
45.74
46.82
Improvement
+3,53
+1,89
DCAS
46.12
47.35
Sampling independently provides substantial performance boost. The culmination of integrating all proposed components results in the optimal model, achieving a mloU improvement of 10.43% for linear probing. This comprehensive analysis underscores the effectiveness of each component and their collective impact in enhancing the model's segmentation capabilities.
Potential of Supervised Contrastive Distillation. As mentioned in Section. 3, we perform weakly- supervised contrastive distillation with the pseudo-labels predicted by SAM. With the free and available models, our method learns effective 31) representations. When with the ground truth provided in auScenes-lidarsee dataset, we can obtain a significant improvement downstream tasks (see Table 6a). The results further demonstrate the effectiveness of supervision for cross-modal contrastive distillation and the potential of our pipeline with stronger VFMs.
replace the weak labels
Effect of the Decoupled Projection Heads. The experimental results outlined in Table 6b demon- strate the effectiveness of employing decoupled projection heads in our model. These results highlight distinct performance enhancement when projection heads specialized for distinct tasks specif- ically, self-supervised and weakly-suppervise contrastive distillation. On the nuScenes dataset, the implementation of decoupled projection heads results in mloU improvement of 3.53%, indicating a robust enhancement in the model's ability to generalize from the training data. Similarly, for the SemanticKITTI dataset, gain of 1.89% in mloU is observed, further substantiating the benefits of this architectural modification.
Effect of the vMF Distribution. The ablation study in Table 6c compares the use of deterministic (Dirac delta) and von Mises-Fisher (vMF) distributions for modeling semantic features of each class, demonstrating clear advantages with vMF both nuScenes and SemanticKITTI datasets. Specifically, the vMF distribution, with its adjustable concentration parameter, provides a mloU improvement of 1.97% on nuScenes and 1.37% on SemanticKITTI compared the deterministic approach. The learned concentration parameter that represents the uncertainty helps in mitigating overfitting by providing robustness against inaccuracies in coarse semantic labels.
Effeet of Different Sampling Strategies. Table 6d, Category-awarr and density-aware sampling determine the sampling probability of point by its category frequency and distance information, respectively. These are part of hybrid strategy we refer to as density and category-awan sampling. We found that the density and sategory-awar sampling strategy consistently achieves the best performance on downstream tasks.
4.4 Further Discussions
We would like to emphasize the uniqueness and advantages of our approach over existing ones:
Previous works [39, 50, 41, 38, 73] have not solved the self-conflict problem properly. Especially, Seal [38] generates semantically coherent superpixels for distinct objects and backgrounds in the 3D scene. However, the superpoints and superpixels with the same category may still be simply considered negative pairs during contrastive learning. By

---

contrast, our method explicitly defines the points and pixels with the same semantic labels as positive pairs during weakly-supervisee contrastive learning.
Our pipeline performs knowledge distillation on two levels: self-supervised and weakly- supervised contrastive learning. To achieve this, we develop two different heads in both the image and point cloud branches decouple the learned representation. Previous methods [39, 50, 41, 38, 73] have only attempted self-supervised contrastive distillation and have not explored using labels to guide contrastive distillation.
The representation of samples in the same class vary significantly across different batches during the contrastive distillation, so the model will struggle to learn stable semantic features. By making point features of the same class closely aligned, our method aims to create a more consistent and structured feature space.
Existing methods [50, 41, 38, 73] are highly dependent on the generated superpixels. Super- pixels balance asymmetries between areas with denser coverage of points and sparser areas in the contrastive loss. However, we do not need this process at all and ensure a uniform representation of both spatial and categorical dimensions by employing novel sampling strategy.
ST-SLidR [41] reduces the contribution of false negative samples based on superpixel-to. superpixel similarity, using 2D self-supervised features to determine semantic similarities between superpixels. By contrast, our method directly estimates the semantic labels of images with VFMs, and defines pixels and points with the same label as positive pairs.
The purposes of using VFMs in Seal and our method are completely different. To avoid over- segmenting semantically coherent areas, Seal generates superpixels using visual foundation models (VFMs) to replace the traditional method SLIC [1]. In contrast, our method does not rely on superpixels. Although we also use VFMs, leverage them to obtain coarse semantic labels for fine-grained contrastive distillation.
4.5 Visualization and Analysis
The T-SNE visualization shown in Fig. 4 demonstrates the efficacy of our method in achieving a more discriminative and well-separated feature distribution for each category compared to the baseline model PPKT [39]. To some extent, each category forms distinct cluster with our method, with relatively clear boundaries separating differ- ent classes. This enhanced clustering ef- feet underscores the benefits of our approach, Figure 4: T-SNE visualization of the extracted point which incorporates semantic supervision and cloud features by PPKT and our OLIVINE (with head applies semantic- guided consistency regular- hig"), ,. Each feature is colorized based on its ground- truth semantic labels on nuScenes dataset.
PPKT
ization during pre-training.
Conclusion
We have introduced OLIVINE, novel method that leverages visual foundation models for fine- grained image-to-LiDDee knowledge transfer. The key ingredient of our method is utilizing the weak semantic labels generated from VFMs to avoid semantically similar negative samples and tackle the challenge of "self-conflict" challenges. We further exploit the semantic labels with semantic- guided consistency regularization, which makes embeddings of points in the same class remain consistent across varying inputs and yields structured feature space. Extensive experiments on various datasets confirm that method achieves superior performance in downstream tasks compared to existing image-to-LilDe LiDAR contrastive distillation methods.
10

---

Acknowledgement
This work was supported in part by the National Natural Science Foundation of China Excellent Young Scientists Fund 12422118, in part by the Hong Kong Research Grants Council under Grants 11219324 and 11219422, and in part by the Hong Kong Innovation and Technology Fund 14//164/23. This work used the computational facilities provided by the Computing Services Centre at the City University of Hong Kong. Besides, we thank the anonymous reviewers for their invaluable feedback that improved our manuscript.
References
[1] Radhakrishna Achanta, Appu Shaji. Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11):2274 2282, 2012.
[2] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitt:: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9297-9307, 2019.
[3] Maxim Berman, Amal Rannen Triki, and Matthew Blaschko. The lovász lovász-softmax softmax loss: A tractable surrogate for the optimization of the pneerecction-ove-nnee measure in neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4413 4421, 2018.
[4] Alexandre Boulch, Corentin Sautier, Björn Michele, Gilles Puy, and Renaud Marlet. Also: Automotive lidar self-supervision by occupancy estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13455-13465, 2023.
[5] S.R. Bowman, L. Vilnis, O. Vinyals, A.M. Dai, R. Jozefowicz, and S. Bengio. Generating sentences from continuous space. In Proceedings of The 20th SIGNLL Conference Computational Natural Language Learning, pages 10-21, 2016.
[6] Holger Caesar, Varun Bankiti, Alex H Lang. Sourahh Vora, Venice Frin Liong, Qiang Xu, Anush Krishnan, Pan, Giancarlo Baldan, and Oscar Beijhom. nuscenes: A multimodal dataset autonomous driving. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11621 11631, 2020.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[8] Chen Chen, Zhe Chen, Jing Zhang, and Dacheng Tao. Sasa: Semantics- aemmentcs-augmmetted set abstraction for point- based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36. pages 22 1219, 2022.
Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma. Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang- Clip2scene: Towards label-efficient 3d scene understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7020-7030, 2023.
[10] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint 2XX::2003.04299 2020.
[11] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[12] Yujin Chen, Matthias Nießner, and Angela Dai. 4dcontrast: Contrastive learning with dynamic corre- spondences for 3d scene understanding. In European Conference on Computer Vision, pages 543-560, 2022.
[13] Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21674-21683, 2023.
[14] Huixian Cheng. Xianfeng Han, and Guoqiang Xiao. Cenet: Toward concise and efficient lidar semantic segmentation for autonomous driving. In IEEE International Conference on Multimedia and Expo, pages 1-6 6, 2022.
11

---

[15] Julian Chibane, Francis Engelmann, Tuan Anh Tran, and Gerard Pons-Moll. Box2mask: Weakly supervised 3d semantic instance segmentation using bounding boxes. In European Conference on Computer Vision, pages 681 (81-699, 2022.
[16] Jaesung Choe, Chunghyun Park, Francois Rameau, Jaesik Park, and In So Kweon. Pointmixer: MIp-mixer for point cloud understanding. In European Conference on Computer Vision, pages 620-640. Springer, 2022.
[17] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnm: Towards high performance voxel- object detection. Proceedings of the AAAI Conference on Artificial Intelligence, pages
2021.
[18] Chaoqun Du, Yulin Wang, Shiji Song, and Gao Huang. Probabilistic contrastive learning for long-tailcd visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[19] Sudeep Fadadu, Shreyash Pandey, Darshan Hegde, Yi Shi, Fang-Chich Chou, Nemanja Djurie, and Carlos Vallespi-onzzaeez. Multi-view fusion sensor data for improved perception and prediction in autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pages 2349-2357, 2022.
[20] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3354-3361, 2012.
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 56000-16009, 2022.
[22] Cheng-Ju Ho, Chen-Hsuan Tai, Yen-Yr Yu Lin, Ming-Hsuan Yang, and Yi-Hsuan Tsai. Diffusion-ss3ae Diffusion model for semi-supervised 3d object detection. Advances in Neural Information Processing Systems, 36, 2024.
[23] Siyuan Huang, Yichen Xie. Song- Chgn Zhu, and Yixin Zhu. Spatio-temporal self- supervised representation learning for 3d point clouds. In Proceedingy of the IEEE/CVF International Conference on Computer Vision, pages 6535-6545, 2021.
[24] Siyuan Huang, Yichen Xie, Song- mig-Chun Zhu, and Yixin 7hu. Spatio-temporaa self-supervised representation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6535-6545, 2021.
[25] Peng Jiang, Philip Osteen, Maggie Wigness, and Srikanth Saripallig. Rellis-3d dataset: Data, benchmarks and analysis. In JEEE International Conference on Robotics and Automation, pages 1110-11 1111111,,7111111 2021.
[26] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems,
2020.
[27] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[28] Alexey Klokov, Un Pak, Aleksandr Khorin, Dmitry Yudin, Leon Kochiev, Vladimir Luchinskiy, and Vitaly Bezuglyj. Daps3d: Domain adaptive projective segmentation 3d lidar point clouds. Preprint, 2023.
[29] Lingdong Kong, Youquan Liu, Runnan Chen, Yuexin Ma, Xinge Zhu, Yikang Li, Yuenan Hou, Yu Qiao, and Ziwei Liu. Rethinking range view representation for lidar segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 228 24000 2023.
[30] Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robo3d: Towards robust and reliable 3d pereeption against corruptions. In Proceedings of the IEEE/CVF Internittonnll Conference Computer Vision, pages 19994-20006, 2023.
[31] Lingdong Kong, Jiawei Ren, Liang Pan, and Ziwei Liu. Lasermix for semi-supervised lidar semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 21705-21715, 2023.
[32] Lanxiao Li and Michael Heizmann. A closer look at invariances in self supervied pre-training for 3d vision. In European Conference on Computer Vision, pages 356-673, 2022.
12

---

[33] Shen Li, Jianqing Xu, Xiaqing Xu, Pengcheng Shen, Shaoxin Li, and Bryan Hooi. Spherical confidence learning for face recognition. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 15629-15637, 2021.
[34] Guibiao Liao, Jiankun Li, and Xiaoqing Ye. V1m2scene: Self-supervised image-text text-lidar learning with foundation models for autonomous driving scene understanding. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 3351 13519,99,, 2024.
[35] Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotalch El Saddik, Christian Theobalt, Eric Xing, and Shijian Lu. Weakly supervised 3d open-vocabulary segmentation. Advances in Neural Information Processing Systems, 36:53433-53456 2023.
[36] Minghua Liu, Yin Zhou, Charles R Qi, Boqing Gong. Hao Su, and Dragomir Anguelov. Less: Label- efficient semantic segmentation for lidar point clouds. In European Conference on Computer Vision, pages 70-89, 2022.
[37] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su. Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023.
[38] Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment any point cloud sequences by distilling vision foundation models, Advances in Neural Information Processing Systems, 36, 2024.
[39] Yueh-Cheng Liu, Yu-Kai Huang, Hung-Yude Yueh Chiang, Hung-Ting Su, Zhe-Yo Yu Liu, Chin-Tang Chen, Ching- Yu Tseng, and Winston H Hsu. Learning from 2d: Contrastive pixel-to-point knowledge transfer for 3d pretraining. arXiv preprint (tfl::2004.04mm 2021.
[40] Yadan Luo, Zhuoxiao Chen, Zijian Wang, Xin Yu, Zi Huang, and Mahsa Baktashmotlagh. Exploring active 3d object detection from generalization perspective. In The Eleventh International Conference on Learning Reprexentations, 2023.
[41] Anas Mahmoud, Jordan SK Hu, Tianshu Kuai, Ali Harakeh, Liam Paull, and Steven Waslander. Self- supervised image-to-pe pegeett-pport to-pnnnt distillation via semantically tolerant contrastive loss. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7102-711 1000, 2023.
[42] Lucas Nunes, Rodrigo Marcuzzi, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. Segcontrast: 3d point cloud feature representation learning through self-supervised segment discrimination. IEFE Robotics and Automation Letters, 7(2):21 16-2123, 2022.
[43] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 807.03748, 2018
[44] Yancheng Pan, Biao Gao, Jilin Mei, Sibo Geng, Chengkun Li, and Huijing Zhao. Semanticposs: A point cloud dataset with large quantity dynamic instances. In IEEE Intelligent Vehicles Symposium, pages 687-693, 2020.
[45] Gilles Puy. Alexandre Boulch, and Renaud Marlet. Using waffle iron for automotive point cloud semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3379-3389, 2023.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[47] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks, arXiv preprint 240..240011455 2024.
[48] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer Assisted Intervention, pages 234 241.. Springer, 2015.
[49] Jomathan Sauder and Bjarne Sievers. Self supervised deep learning on point clouds by reconstructing space. In Advances in Neural Information Processing Systems, volume 32, 2019.
13

---

[50] Corentin Sautier, Gilles Puy, Spyros Gidaris. Alexandre Boulch, Andrei Bursuc, and Renaud Marlet. Image-to-lidar self-supervised distillation for autonomous driving data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9891-9901, 2022.
[51] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-remn: Point-voxed feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10529 10538, 2020.
[52] Suvrit Sra. A short note on parameter approximation for von mises-fisher distributions: and fast implementation of (x). Computational Statistics, 27:177-190, 2012.
[53] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point-voxel convolution. In European Conference on Computer Vision, pages 685-702, 2020.
[54] OpenPCDet Developpent Team. Openpedet: An open-source toolbox for 3d object detection from point clouds. https://giihae com/ het///ppnmmmaaaae ab/ppenCCDpp,. 2020.
[55] Hugues Thomas, Charles R Qi, Feaannmmma Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF international conference on computer vixion, pages 6411-6420 2019.
[56] Zhi Tian, Xiangxiang Chu, Xiaoming Wang, Xiaolin Wei, and Chunhua Shen. Fully convolutional one- stage 3d object detection on lidar range images. Advances in Neural Information Processing Systems, 35:34899 34911, 2022.
[57] Ozan Unal, Dengxin Dai, and Luc Van Gool. Scribble superviiee lidar semantic segmentation. In IFFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2697-2707, 2022.
[58] Chengyao Wang, Li Jiang, Xiaoyang Wu, Zhuotao Tian, Bohao Peng, Hengshuang Zhao, and Jiaya
Jia. Groupeontrast: Semantic-aware sele-supervised representation learning for 3d understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[59] Feng Wang and Huaping Liu. Understanding the behaviour contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2495-2504, 2020.
[60] Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J. Kusner. Unsupervised point cloud pre-training via occlusion completion. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 9782-9792, 2021.
[61] Ziyi Wang, Yongming Rao, Xumin Yu, Jie Zhou, and Jiwen Lu. Point-to-pixel prompting for point cloud analysis with pre-trained image models. IEEE Transactions on Pattern Analysis and Muchine Intelligence, 46(06):4381 4397, 2024.
[62] Lintai Wu. Qijian Zhang, Junhui Hou, and Yong Xu. Leveraging single-view images for unsupervised 3d point cloud completion. IEEE Transactions on Multimedia, pages 14, 2023.
[63] Aoran Xiao, Jiaxing Huang, Dayan Guan, Fangneng Zhan, and Shijian Lu. Transfer learning from synthetic to real lidar point cloud for semantic segmentation. In AAAI Conference on Artificial Intelligence, pages 2795-2803, 2022.
[64] Aoran Xiao, Jiaxing Huang, Weihao Xuan, Ruijic Ren, Kangcheng Liu, Dayan Guan, Abdulmotaleb El Saddik, Shijian Lu, and Eric Xing. 3d semantic segmentation in the wild: Learning generalized models for adverse-condition point clouds. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9382-9392, 2023.
[65] Binhui Xie, Shuang Li, Qingju Guo, Chi Liu, and Xinjing Cheng. Annotator: A generic active learning baseline for lidar semantic segmentation. Advances in Neural Information Processing Systems, 36, 2023.
[66] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European Conference on Computer Vision, pages 574-591, 2020.
[67] Jianyun Xu, Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun, and Shiliang Pu. Rpvnet: A deep and efficient range range-point-vo voxel fusion network for lidar point cloud segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 16024 16033, 2021.
14

---

[68] Mingye Xu, Mutian Xu. Tong He. Wanli Ouyang, Yali Wang, Xiaoguang Han, and Yu Qiao. Mm- 3dscene: 3d scene understanding by customizing masked modeling with informative-preseree reconstruction and self-distilled consistency. Pruceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2380-4390, 2023.
[69] Ryosuke Yamada, Hirokatsu Kataoka, Naoya Chiba, Yukiyasu Domae, and Tetsuya Ogata. Point cloud pre- training with natural 3d structures. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21283-21293, 2022.
[70] Yan Yan. Yuxing Mao, and Bo Li. 18(10):3337, 2018
Second: Sparsely embedded convolutional detection. Sensors,
[71] Junbo Yin, Dingfu Zhou, Liangjun Zhang. Jin Fang. Cheng- Cheng-Zhong Xu, Jianbing Shen, and Wenguan Wang. Proposalcontrast: Unsupervised pre-training for addr-based 3d object detection. In European Conference Computer Vision, pages 17-33, 2022.
[72] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang. Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19313-19322, 2022.
[73] Sha Zhang, Jiajun Deng. Lei Bai, Houqiang Li, Wanli Ouyang. and Yanyong Zhang. Hvdistill: Transferring knowledge from images to point clouds via unsupervised hybrid-view distillation. International Journal of Computer Vision, pages 1-15, 2024.
[74] Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yuc, Zerong Xi, Boqing Gong, and Hassan Foroosh. Polamet: An improved grid representation for online lidar point clouds semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9601-9610, 2020.
[75] Yifan Zhang, Junhui Hou, and Yixuan Yuan, A comprehensive study of the robustness for lidar-based 3d object detectors against adversarial attacks. International Journal of Computer Vision, 132(5):1522-1624 2024.
[76] Yifan Zhang, Siyu Ren, Junhui Hou, Jinjian Wu, and Guangming Shi. Self-supervised learning of lidar 3d point clouds via 2d-3d neural calibration. arXiv preprint arXiv:2401. 2024.
177] Yifan Zhang, Qijian Zhang, Junhui Hou, Yixuan Yuan, and Guoliang Xing. Unleash the potential of image branch for cross-modal 3d object detection. In Advances in Neural Information Processing Systems, volume 36, pages 51562-51583, 2023.
[78] Yifan Zhang, Qijian Zhang, Zhiyu Zhu, Junhui Hou, and Yixuan Yuan. Glenet: Boosting 3d object detectors with generative label uncertainty estimation. International Journal of Computer Vision, 131((2):3332-3322 2023.
[79] Yifan Zhang, Zhiyu Zhu, Junhui Hou, and Dapeng Wu. Spatial-temporal graph enhanced detr towards fulti-frame 3d object detection. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024.
[80] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 10252-10263, 2021.
[81] Zhuoyang Zhang, Yuhao Dong, Yunze Liu, and Li Yi. Complete-to-partial 4d distillation for self-supervised point cloud sequence representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Patiern Recognition, pages 17661-17670, 2023.
[82] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4490-4499 2018
[83] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong. Yuexin Ma. Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks lidar segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9939999999999 (0099948,777 2021.
[84] Xueyan Zou, Jianwei Yang. Hao Zhang, Feng Li. Linjie Li. Jianfeng Wang. Lijuan Wang. Jianfeng Gao. and Yong Jae Lee. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36, 2023.
15

---

A Appendix
In this appendix, we present the details omitted from the manuscript due to space constraints. The appendix is organized as follows:
Section A.I: Theoretical justification for the application of the vMF distribution.
Section A.2: Details of the dataset and evaluation metrics.
Section A.3: Experimental setup of pre- pre-training.
Section A.4: More quantitative results.
Section A.5: More qualitative results.
Section A.6: Potential limitations of method.
Section A.7: Societal and environmental impact of our work.
Section A.8: Public resources used in this work.
A.1 Theoretical Analysis
The features of each class k can be modeled as von Fisses-Fisher (vMF) distribution. This means that for class kt the feature vectors di lie on a unit hypersphere and are centered around mean direction μ with a concentration parameter
Proposition 1:
Justification: To show that the features of each class can be effectively modeled by vMF distribution, we use maximum likelihood estimation (MLE) to determine that the parameters and K3 are optimal for the given of feature vectors.
For set of Mj feature vectors {g, 1 from class k, the likelihood function for the vMF distribution is:
L(pr, K&)
II f(e Us. RK) II exp(xa 9:)
(14)
Taking the natural logarithm of the likelihood function, we get the log-likelihood:
Me
Mx
log L(kk KR)) Substituting the expression for (N(N))) get:
Σ
log fgg Mby Ku) Ma log Ko(NK) Kg
T
gi
(15)
C/2-1
MR
Rg
Kk
log L( KK)) = Mr
log
uk Ji
(16)
1/1
Ma
Mr
C
log L(uu, RR) Mr (c/2-1))og) log Ka log 1
log(2
+
14
T
gi
(17)
MR
To maximize the log-likelihood, we normalize μr by setting it to the normalized sum the feature vectors:
Σ
(18)
The derivative of the log-likelihood with respect to Kik is:
i log L(pR, NK)
C/2
1c/2(KH)
M₁
Mv
Σ
u
(19)
0kk
Kk
(152
Mr
i=1
16

---

Setting this derivative to zero, we get:
C/2-1 (k)2(kk)
1
Mr
uk 9i
(20)
Kkk
1c/2-1(15x)
MR
Solving for KR, obtain:
12 Mr
9:ll(C - Σ Me 1=l
(21)
1-
- Me gill
This equation allows us to compute the concentration parameter &y based on the alignment of the feature vectors. The concentration parameter Ką is larger when the distribution is more tightly clustered around the mean direction, and smaller when the features are more uniformly spread across the hypersphere.
By maximizing the likelihood function for the vMF distribution, we have shown that the parameters H4 and кą can be estimated to model the distribution of feature vectors for each class. The mean direction μa denotes the central direction of the feature cluster, and the concentration parameter K& controls the tightness of this clustering. Moreover, the way estimate the parameters of vMF distribution in EMA is also consistent with the results of the above theoretical derivation.
The representation of samples in the same class can vary significantly across different batches during contrastive distillation, and semantic-guided consistency regularization helps to learn structured features.
Proposition 2:
Without regularization, the representation of samples within the same class can vary significantly across different batches during contrastive distillation. This variance arises due to random sampling and the influence of negative samples in different batches. The weakly-supervist contrastive loss is defined as:
Justification:
M. Σ log
G //T)
Σ
Saup
(22)
M,
[A(i))
dEA(() E (C //T)
The features of negative samples G vary across batches, leading to different optimization paths for each mini-batch. This introduces variability in the learned representations G for samples of the same class k.
When do not use monanticc-uided consistency regularization, the within class variance for class across different batches is:
M
1g
u 112
(23)
M
For ease of reading, we use 9: to refer to point feature G And fu is the mean feature vector for class in batch B. Due to the batch-wise variability in negative samples, 1e can differ significantly across batches, leading to high within-class variance.
By minimizing the KL divergence, we align feature vectors gi of class with the mean direction μ4. reducing the spread of feature vectors within the same class. The within-class variance with regularization
2 ow
9
2
(24)
K
M
k=
Since μą is consistent across batches due to the regularization, the within-class variance is significantly reduced. This results in structured feature representations, enhancing class separability and improving performance in downstream tasks.
17

---

Proposition 3: Learning structural representation during pretraining can benefit downstream tasks.
Justification: Structured features those well-aligned within the same class (low within-class variance ) and well-separated between different classes (high between-class variance Øg)
With semantic-gus guided consistency regularization, feature vectors g for class ls are closely aligned with the mean direction As. This alignment reduces the within-clase variance o Weakly-superviss contrastive learning pushes apart feature vectors different classes, increasing the separation between class means μgg- This increases the between-class variance B
Take the linear classifier as an example, the decision boundary is determined by the separation between class means. Higher o and lower ow result in clearer decision boundaries, reducing classification errors.
Consider a simple linear classifier with weight vector w and bias The decision function is:
[(x) = w x +
(25)
The decision boundary is given by:
T x + w
(26)
For well-structured features, the margin (distance between decision boundary and nearest samples) maximized. The margin y for class k can be expressed as:
w " - (1)
(27)
|w||
Higher between-class variance (o and lower within-class variance (oly) increase this margin, leading to better classification performance.
A.2 Dataset and Evaluation Metric
The NuScenes dataset, compiled from driving recordings in Boston and Singa- pore, utilizes a vehicle equipped with a 32-bbem LiDAR and additional sensing technologies [6]. This comprehensive dataset equipped with the typical sensor array found autonomous vehicles, including a 32-beam LiDAR setup, six cameras, and radar systems, ensuring full 360-degree envi- ronmental perception. It includes 850 driving scene snippets, with 700 designated for training and 150 for validation, each scene lasting 20 seconds with annotations provided every 0.5 seconds. The dataset features extensive annotations across several object categories, including vehicles, pedestrians, bicycles, and road barriers, with each object encapsulated in a 3D bounding box and supplemented with attributes detailing visibility, activity, and pose.
NuScenes Dataset.
MAScenes-lidarsser Dataset. The nuScenes dataset now encompasses features for semantic and panoptic segmentation through its extension, muScenes-lidaae lidarseg [6]. This enhanced dataset provides semantic labeling across 32 distinct categories, with each point in the dataset's 's keyframes meticulously annotated. We utilize the 700 training scenes equipped with segmentation labels for refining semantic segmentation models, and we assess model performance using the 150 scenes in the validation set.
SemanticKITTI Dataset. The SemanticKITTI (SK) dataset features paired RGB images and point cloud data derived from KITTI's urban scenes, specifically designed for semantic segmentation tasks [2]. This dataset gathered using sensors mounted on a vehicle, including more than 200,000 images alongside their corresponding point clouds across 21 distinct sequences. Both images and point clouds are aligned to maintain consistent relative transformation. Originally, the images are captured a resolution of 1241x376 pixels, and each point cloud is composed of roughly 40,000 3D points. In line with standard practices, the dataset is divided into training and validation sets, with 10 sequences designated for training and the eighth sequence reserved for validation.
KITTI Dataset. KITTI a crucial dataset for advancing 3D object detection in autonomous driving. With 7481 training and 7518 test point clouds, it covers diverse urban and suburban environments [20].
18

---

scenarios, using the official validation set to report The dataset includes 3D point clouds and RGB images captured using a Velodyne HDL-64E LiDAR sensor. Calibration information between the camera and LiDAR provided. essential for cross-modal knowledge transfer sensor fusion tasks. Annotated with 3D bounding boxes, it features common objects like cars, pedestrians, and cyclists. The dataset is split into training (3712 samples) and validation (3769 samples) subsets.
ScribbleKITTI Dataset. ScribbleKITTI is derived from the SemanticKITTI dataset but introduces weak supervision in the form of line scribbles rather than fully labeled point clouds [57]. It retains the same set of 19,130 LiDAR scans, captured by Velodyne HDL-64E sensor, but only about 8.06% of the semantic labels are provided compared to the fully-supervised SemanticKITTI dataset. This method of annotation drastically reduces the time required for labeling, offering around 90% time saving. We use this dataset to evaluate how well models pre-trained on other datasets generalize under weaker annotations. For our experiments, we follow the SLidR protocol to create different splits of the training set, e.g., one scan is selected from every 100 frames to generate 1% labeled samples. The model's performance evaluated on the official validation set.
RELLIS-3D Dataset. RELLIS-3D is a multimodal dataset collected from off-road environments on the Texas A&M University campus [25]. The dataset comprises 13,556 annotated LiDAR scans, providing a challenging scenario with complex terrain and class imbalance. It is valuable for assessing model performance outdoor enviromments with varying topographies and object densities.
SemanticPOSS Dataset. SemanticPOSS is a smaller dataset focused on dynamic objects, captured on the campus of Peking University [44]. It includes 2,988 LiDAR scans from Hesai Pandora 40-channel LiDAR sensor. The dataset is designed to challenge models with its focus on moving instances and dense environments, making it useful resource for evaluating the adaptability of models to dynamic scenes. In our setup, sequences 00 and 01 are used to create half of the annotated training samples, and sequences 00 to 05, excluding sequence 02, are used for validation.
SemantieSTF Dataset. This dataset features 2,076 LiDAR scans collected under adverse weather conditions such snow, fog, and rain, using Velodyne HDL64 S3D sensor [64]. The dataset is split into training, validation, and test sets, ensuring an even distribution of weather conditions across all subsets. SemanticSTF is particularly suited for testing the robustness of models in extreme environmental conditions.
SynLiDAR Dataset. The SynLiDAR dataset is composed of synthetic point clouds generated in virtual environments using Unreal Engine [63]. consists of 13 sequences with total of 198,396 scans. This synthetic dataset enables large-scale experimentation in scenarios that closely mimic real-world conditions, offering a controlled environment for model pre-training and testing. For our fine-tuning experiments, we use uniformly downsampled subset of the dataset.
DAPS-3D Dataset. DAPS-3D includes both semi-synthetic and real-world data, with the subset DAPS-1 consisting of over 23,000 labeled LiDAR scans across 11 sequences [28]. The data was collected in the context of an autonomous robot's deployment in a real-world scenario. This dataset helps evaluate the transferability of models pre-trained on synthetic data to real-world tasks. In our setup, we use the sequence "38-18 7 0______2___0 90" for training and validate the model on the sequences "38-18_7_11 100____22_99000 "0""2"00,,"""_ "42-48_ 1a__111___11_1 10""","""__0"0000" and "44-18_1 $$$11111_3222 This configuration helps in evaluating the model's performance on both synthetic and real-world data.
Robo3D (nuScenes-C) Benchmark. As a part of the Robo3D benchmark [30], nuScenes-C tests the robustness of models against various corruptions that simulate real-world challenges, such as severe weather and sensor malfunctions. These corruptions are categorized into different levels of severity (light, moderate, heavy), and the dataset includes eight types of disturbances like fog. snow, motion blur, and sensor interference. It is designed to assess how well models perform under diut----iitiiitooon conditions. We follow the standard protocol of the Robo3D benchmark to evaluate model robustness under these results.
Evaluation Metrics. In semantic segmentation tasks, performance is assessed through Intersection- over-Union (IoU) for individual classes and mean IoU (mloU) across all classes. In 3D object detection, the 3D detector's efficacy on the KITTI dataset is measured using Average Precision (AP) metrics at IoU thresholds of 0.7 for cars, 0.5 for pedestrians, and 0.5 for cyclists. Similarly, for the Waymo dataset, evaluation is based on 3D mean Average Precision (mAP).
19

---

A.3 Experimental Setup of 3D Pretraining
Network Architectures. For the image processing branch, we utilize the ResNet-50 structure the core architecture. This 2D backbone is initialized with weights that have been pre- pre-trained using MoCov2 [10] on the ImageNet dataset. To preserve the receptive field while keeping the spatial resolution intact, substitute the second and subsequent stridden convolutions with dilated convolutions, following established methodologies [50]. The up-sampling projection head includes 1xl convolutional layer that reduces the channel count from 2048 64, followed by bi-linear interpolation up-sampling layer that enlarges the scale by factor of 4. This up-sampling process effectively restores the resolution of the 2D feature map match that of the original input images, specifically to the size of 414x224.
In the point cloud processing branch, adopt two types of backbones. For the 3D semantic segmentation task, we employ the Sparse Residual 3D U-Net 34 (SR (SR-UNet34) [48], adhering to practices previously established in SLidR [50]. The output from SR-UNet34 offers 256 channels, whereas the image branch outputs 64-dimensi feature map. To align these dimensions, a 3D convolutional layer used in the projection head to reduce the channel count of the point features to 64. We process the 3D point data into voxels to serve as input for the SR-UNet. The voxels are formatted in Cartesian coordinates covering X-axis and Y-axis range of [-51.2m 1m,,m, 51.2m] and Z-axis range of [-5.0m, 3.0m]. with each voxel measuring (0. m, 0. m. 0m).) To fully evaluate our method, pre-train and transfer another VoxelNet [82] for the 3D object detection task.
Pre-training Details. We utilize momentum SGD for optimization, setting the initial learning rate at 0.5 and 0.01 for SR-UNet34 and VoxeINet respectively, with momentum of 0.9 and weight decay of le-4. To adjust the learning rate, we employ cosine annealing scheduler [5] that gradually reduces it from the initial value over 50 epochs. The 3D network is pre-tree pre-trained for these 50 epochs four NVIDIA-3090 GPUs, processing a total batch size of 16, unless specified otherwise. For data augmentation, we incorporate several techniques. For the point cloud data, we apply random rotations around the z-axis, randomly flip the x and y axes, and omit points within a randomly selected cuboid, following the method described in [80]. For image data, augmentations include random horizontal flips and random crop-resize operations. In terms of generating weak semantic labels, the prompts we provided the SEEM [84] encompass a total 16 object categories: barrier, bicycle, bus, car, truck, trailer. motorcycle, construction vehicle, pedestrian, traffic cone, road, sidewalk, terrain, vegetation, building, and other ground. Unless otherwise specified, the semantic labels used in the ablation study are inferred with Grounded-SAM.
10c
Count
105
104
103
driveable surface
manmade
vegetation
terrain
sidewalk
car
truck
other flat
barrier
bus
pedestrian
trailer
construction vehicle
traffic cone
motorcycle
bicycle
Category
Figure 5: Class distribution at the pixel level for nuScenes dataset.
20

---

107
109
Count
10a
10
road
terrain
sidewalk
building
car
fence
parking
trunk
other-vehicle
pole
truck
person
ather-ground
traffic-sign
motorcycle
bicyclist
bicycle
motorcyclist
Category
Figure 6: Class distribution at the pixel level for SemanticKITTI dataset.
A.4 More Quantitative Results
Effectiveness Pre-training on Stronger 3D Backbone. A key question arises: the reduced effectiveness with larger data due to the model's insufficient size inherent limitations in the proposed approach? When the available training data limited, the benefits of pre-trained model weights on downstream tasks become more apparent. This phenomenon, commonly observed in self-supervised learning, is particularly important when downstream task data is scarce. In such cases, pre-trained representations offer strong foundation, capturing crucial features that would otherwise remain unlearned from the limited labeled data. However, as the volume of labeled data increases, the model is capable of learning these features directly from the data itself, rendering the pre-trained representations less essential.
To further investigate this, we conducted experiments using more robust 3D backbone, Waffle- Iron [45] (see Table 7). The results demonstrate that the effect pre-training weights becomes less significant when sufficient training data is available for downstream tasks. This suggests that the reduced effectiveness with larger datasets is not due to the backbone's capability, but rather to the diminishing importance of pre-trained features the model learns directly from ample labeled data.
Table 7: Performance for 3D backbone WaffleIron.
Method
1%
10%
100%
Random
33.26
58.13
77.60
Ours
50.14
66.43
78.21
Results on OOD Datasets. Table presents the robustness evaluation of several state-of-the-art pretraining methods under eight out-of-distribution (OOD) corruption scenarios from the nuScenes-C dataset, part of the Robo3D benchmark. The corruptions include conditions such as fog, snow, motion blur, beam missing, cross-sensor interference, and more. The table reports three key metries: mean Corruption Error (mCE, lower is better), mean Recovery Rate (mRR. higher is better), and mean Intersection over Union (mloU, higher better). Random Initialization: Among the randomly initialized models, Cylinder3D demonstrates the best overall performance in terms of mRR (78.08%) and mloU (62.29%) indicating relatively better robustness across corruption scenarios. IaffleIron achieves the lowest mCE (106.73), but its performance in mRR and mloU is slightly lower than Cylinder3D. SPVCNN also shows competitive results in mloU (62.29% but slightly lags in mRR. Pretrained Methods: Pretraining methods, such as PPKT, SLidR, and Seal, show clear improvement in robustness compared to random initialization. Seal stands out with the second-bes mCE (92.63) and mRR (83.08), along with strong mloU (72.66%). However, method demonstrates the best
21

---

overall performance, achieving the lowest mCE (90.85) and the highest mRR (83.35), as well as competitive mloU (70.62). This indicates that our approach outperforms both random initialization and other pretraining techniques in most OOD corruption scenarios, especially in beam missing (67.28%) and cross-sensor interference (59.47%). These results confirm that pretraining, particularly with our method, enhances the model's resilience to OOD corruptions, leading to more robust performance across varying environmental disturbances.
Table 8: Robustness evaluation of state-of-the-art pretraining methods under eight out- distriibuuton corruptions in the muScenes-C C dataset from the Robo3I) benchmark. All mCE (L), mRR (t), and
mloU (1) scores are given as percentages (%)
Initial
Buckbone
mCE (()
mRR (*)
Fog
Wet
Snow
Motion
Beam
Cross
Echo
Sensor
Random
PolarNet [74]
115.09
76.34
58.23
69.91
64.82
44.60
61.91
40.77
53.64
42.01
Random
CENet [14]
112.79
76.04
67.01
69.87
61.64
58.31
49.97
60.89
53.31
24.78
Random
WaffleIron [45]
106.73
72.78
56.07
73.93
49.59
59.46
65.19
33.12
61.51
44.01
Random
Cylinder3D [83]
105.56
78.08
61.42
71.02
58.40
56.02
64.15
45.36
59.97
43.03
Random
SPVCNN [53]
106.65
74.70
59.01
72.46
41.08
58.36
65.36
36.83
62.29
49.21
Random
MinkUNet
112.20
72.57
62.96
70.65
55.48
51.71
62.01
31.56
59.64
39.41
PPKT
MinkUNet
105.64
76.06
64.01
72.18
59.08
57.17
63.88
36.34
60.39
39.57
SL.dRR
MinkUNet
106.08
75.99
65.41
72.31
56.01
56.07
62.87
41.94
61.16
38.90
Seal
MinkUNet
92.63
83.08
72.66
74.31
66.22
66. 14
65.96
57.44
59.87
39.85
Ours
MinkUNet
90.85
83.35
70.62
75.86
66.51
64.06
67.28
59.47
62.90
47.94
Computational Cost. Our approach, OLIVINE, focuses on providing pre-trained weights and does not impact the inference speed of the model on downstream tasks. As seen in the Table OLIVINE requires similar GPU memory and training time compared to other pre-training methods, demonstrating that our method does not significantly increase computational costs during pre-training.
Table 9: Comparison with other methods regarding the computational cost during pre- rre-training.
Method
GPU Memory (GB) Training Time (Hour)
PPKT
7.6
35.7
SLidR
10.7
38.9
OLIVINE
8.1
36.5
Class Unbalance in Datasets. The visualizations in Figures 5 and 6 illustrate the class distribution at the pixel level for the nuScenes and SemanticKITTI datasets, respectively. These figures reveal significant class imbalance, a common challenge in many real-world datasets, where some classes are overwhelmingly more frequent than others. Such imbalance can skew the training process, leading to models that perform well on frequent classes but poorly on rare ones. This disparity predominantly affects the model's ability to generalize effectively across different scenarios, particularly underrepre- sented ones, resulting in biased predictions and reduced overall accuracy. For instance, infrequent but critical objects like pedestrians or bicycles might not be detected reliably, which is particularly concerning in autonomous driving contexts where safety is paramount.
To mitigate these issues, our method incorporates an optimized sampling strategy. This strategy involves adjusting the probability of selecting samples from underrepresented classes during the training process. By increasing the likelihood of including that the model does not overlook these important but less frequent categories.
classes in the training set, ensure
Per-class Performance. In Tables 10 and 11, we showcase the per-class performance of various point cloud pretraining strategies, including our method and other baselines, fine-tuned using just 1% of labeled data from the suScenes-lidarsee and SemanticKITTI datasets. Our approach consistently surpasses other methods, achieving the highest mean Intersection over Union (mIoU) in nearly all categories. This marked superiority is also significant in complex categories like the bus and truck that demand more precise segmentation, highlighting our method's robustness in processing sparse and intricate data scenarios.
Effects of Different VFMs. The impact of different VFMs on the performance of OLIVINE is an important consideration. The precision of the semantic labels generated by these VFMs plays crucial role in the success of OLIVINE. Those VFMs that also enable text prompts can be applied in OLIVINE to further improve its performance. Stronger VFMs are able to produce more accurate
22

---

Table 10: Per-class results on the DeScenes-lidarsee dataset using only 1% of the labeled data for
fine-tuning. This chart displays the IoU scores for each category, with the highest and second second-highest
scores marked in dark blue and light blue, respectively.
const. reh.
pedestrian
traffic cone
driv. surf.
other flat
sidewalk
manmade
vegetation
barrier
bicycle
motor
tradler
truck
terrain
mloU
bus
car
Method
Random
0.0
0.0
8.1
65.0
0.1
6.6
21.0
9.0
9.3
25.8
89.5
14.8 41.7 48.7
72.4
73.3
30.3
PoinCContrast
0.0
1.0
5.6
67.4
0.0
3.3
31.6
5.6
12.1
30.8
91.7
21.9
48,4
50.8
75.0
74.6
32.5
DepthContrast
0.0
0.6
6.5
64.7
0.2
5.1
29.0
9.5
12.1
29.9 90.3 17.8 44.4 49.5
73.5 74.0
31.7
PPKT
0.0
2.2
20.7
75.4
13.2
45.6
8.5
17.5
38.4
92.5 19.2 52.3 56.8
80.1
80.9
37.8
SLidR
0.0
1.8
15.4
73.1
1.9
19.9
47.2
17.1
14.5
34.5
92.0 27.1 53.6
61.0
79.8
82.3
38.3
ST-SLidR
0.0
2.7
16.0
74.5
3.2
25.4
50.9
20.0
17.7
40.2
92.0 30.7 54.2
61.1
80.5
82.9
40.8
Ours
0.0 12.8
74.3
82.9
13.5
43.1
58.3
31.2
20.9 47.6 93.6 40.2 59.8 66.1 8T.9
82.6
50.5
Table 11: Per-class performance on SemanticKITTI with 1% of the labeled data utilized for fine- tuning. The figure shows the IoU for each class, where the top and second top scores are indicated by dark blue and light blue backgrounds, respectively.
other- vehiole
motorcyellott
other- ground
bicycle
motoreycle
truck
person
bicyelist
parking road
sidewalk
building
Fence
vegetation
trunk
tommii
Pole
traffic- traffic-sign
mioU
car
Method
Random
91.2
0.0
9.4
8.0
10.7
21.2
0.0
0.0 89.4
21.4
73.0
83.3
41.1 84,9 SO.1
71.4
35.4
37.6 39.5
PPKT
91.3
1.9
11.2
23.1
12.1
27.4
37.3
0.0 91.3
27.0
74.6
0.3
86.5
38.2
85.3
58.2
71.6
57.7
40.1
43,9
SLidR
92.2
3.0
17.0
22.4
14.3
36.0
22.1
0.0
91.3
30.0
74.7
0.2
87.7
41.2
85.0
58.5
70.4
58.3
42.4
44.6
Ours
93.1 17.5
28.1
45.2
18.7
47.4
31.4
0.0
91.8
32.3
75.5
1.8
88.1
47.2
85.7
59.0
71.4
59.8
43,9
49.4
semantic labels, which in turn lead to better learned representations during the pre-training process. As shown the table below, the use of a stronger VFM, such as SEEM [84], can improve performance, highlighting the potential of our method when paired with more advanced models.
Table 12: Effects of different VFMs for generating semantic labels.
VFMs
L.P
1%
5%
10%
25%
Grounded-SAM
47.30
46.12
57.51
63.04
69.39
Grounded-Soe -A
47.84
48.03
58.51
64.08
69.52
SEEM
50.09
50.58
60.19
65.01
70.13
A.5 More Qualitative Results
2D-3D Feature Similarities. The visualization presented in Figure 7 showcases the feature similari- ties between image and point cloud data as extracted through different projection heads. In the first column, the raw image is displayed alongside the projection of an anchor point within that image, setting the context for the comparison. The second column visualizes the feature similarities as extracted by the conventional projection heads, h and hpp used for point-pixel level contrastive distillation. Here, only the features of the pixel that directly correspond to the anchor point show a high degree of similarity, emphasizing tight, point-to-point correspondence.
In contrast, the third column introduces results from the extra projection heads, hap and Azon, which (category-awary contrastive distillation. This setup reveals broader similarity pattern, where points and pixels sharing the same category exhibit notably higher feature similarities. This suggests that newly proposed projection heads are effective in capturing and reinforcing category-level feature associations across the 3D and 2D domains, thus enhancing the model's ability recognize semantically similar but spatially disparate features.
designed for veakly-superriin
Visual Results on Downstream Tasks. In Figures 8, 9, 10, and 11, we present additional qualitative results from fine-tuning tasks on downstream datasets. The application of pre-training strategies markedly improves model accuracy over baselines that use random initialization. Notably, our proposed OLIVINE outperforms SLiDR [50], highlighting its superior segmentation capabilities.
23

---

Anchar Point
Low
High
Figure 7: Visualization of the similarities between image and point cloud feature. In the first column, we show the raw image and the projection of anchor point in the image. In second columns, we illustrate the similarities between 3ID query and 2D features extracted by the conventional projection heads he and PP for point-pixel level contrastive distillation. third columns, we illustrate the similarities between 3D query and 2D features extracted by the extra projection heads hsem and hep for seakly-superviied (category-aware) contrastive distillation.
24

---

Despite these advancements, we note the occurrence of false positive predictions in edge cases, which we aim to address future research.
Visualization of the Weak Semantic Labels. The weak labels generated by SEEM [84] using targeted prompts play vital role in our processing pipeline. While reviewing these labels, we observe instances of imprecision (see Fig. 12 and Fig. 13). Should future advancements in Segmentation Anything Models yield more robust and accurate results, the effectiveness our 3D pre-training strategy is likely to improve significantly. This progression would enhance our model's 's ability to interpret and learn from nuanced environmental data, ultimately leading to superior representation learning.
A.6 Potential Limitations
While our method, OLIVINE, effectively enhances the fine-grained image-to-LiDAR contrastive distillation process and demonstrates significant improvements in 3D scene understanding, there are technical and potential limitations that merit attention:
Semantic Label Accuracy. The accuracy of the weak semantic labels generated by VFMs is critical the success of our model. Any deficiencies in these labels could propagate through the learning process, potentially compounding errors in the learned representations.
Training Data Diversity. Currently, model is pre-trained using single dataset, which may limit its applicability to environments or scenarios not well-represented in the training data. Expanding the training include diverse datasets with varying characteristics could enhance the robustness and generalizability of our model.
Dependency on High-Quality Data Calibration. Our framework relies on the precise calibration and synchronization between LiDAR sensors and cameras. In real-world applications, perfect synchronization and calibration can be challenging maintain, potentially affecting the accuracy and reliability of the semantic labels generated and subsequently the distillation process [50, 38, 73].
These limitations highlight areas for future development and research, suggesting a path toward more robust, adaptable, and efficient systems for 3D scene understanding in diverse and dynamic environments.
A.7 Societal and Environmental Impact
Our method, OLIVINE, which enhances -mage-to-iDDAR contrastive distillation using Visual Foun- dation Models, significantly impacts society and the environment. Societally, it boosts the safety and reliability autonomous systems, increasing public trust and improving data analysis various industries. Environmentally. deploying advanced deep learning models requires increased computa- tional resource usage, which can lead to higher energy consumption and associated carbon emissions. This is particularly relevant during the intensive training phases that require performrrre GPUs and long training durations, especially as the model scales to larger datasets or more complex sce- narios. Conversely, by distributing our pre-trained models, we aim reduce the need for repetitive training across multiple downstream tasks, which can decrease the overall computational load and en- ergy consumption needed achieve high performance in various applications. This aspect potentially mitigates some of the environmental costs.
A.8 Public Resources Used
We acknowledge the use of the following public resources, during the course of this work:
Boounded-Segment-Anyiiin
Apache License 2.0
KITTI Dataset
CC BY-NC-SA 3.0
MinkowskiEnniine
MIT License
nuScenes
CC BY-NC-SA 4.0
www.IDE htwp:/DEA-Reeeaacc///rruueeeeee htwprrchhccrroeedeeeeeeeeeeee--e Segment- Anything.
Phttps://wee htgpahab...o////ooiaW/ooooeeaana
hnttps://ww.nusee www.wnus..eec....mmmm.. wwweaanee..rr//usscceeeee nuscenes..
25

---

ScribbleKITTI
Unknown
RELLIS-3D
CC BY-NC-SA 3.0
SemanticPOSS
Unknown
SemantieSTF
CC BC 4.0
SynL.iDA
MIT I License
DAPS-3D"
MIT License
nuScenes-C"
CC BY-N--S 4.0
nuScenes-devkit"
Apache License 2.0
OpenPCDet
Apache License 2.0
PyTorch-Lightning
Apache License 2.0
SemanticKITTII
CC BY-NC-SA 4.0
SLidR
Apache License 2.0
SEEM
Apache License 2.0
html. html.
htttgp//ggtthaaacccm////ggggg2gg
"httpp//eemmntio--ittcc...g
SatershhreeAAl----e All-At. A1-eee-nnce.
26

---

Ground-Trufe
Prediction
Error Map
Random
SLidR
Random
SLidR
Figure 8: Qualitative results of fine-tuning on 1% the Secennes-lddasee dataset with different pre-training strategies. Note that the results are shown as error maps on the right, where red points indicate incorrect predictions. Best viewed in color and zoom in for more details.
27

---

Ground-Truth
Prediction
Error Map
Random
SLidR
Ours
Random
SLidR
Ours
Figure 9: Qualitative results of fine-tuning on 1% of the muScenes-lidarseg dataset with different pre-training strategies. Note that the results are shown as error maps on the right, where red points indicate incorrect predictions. Best viewed in color and zoom in for more details.
28

---

Ground-Truth
Prediction
Error Map
Random
SLidR
Ours
Random
SLidR
Ours
Figure 10: Qualitative results of fine-tuning 1% of the SemanticKITTI dataset with different pre-training strategies. Note that the results are shown as error maps on the right, where points indicate incorrect predictions. Best viewed in color and zoom in for more details.
29

---

Ground-Truth
Prediction
Error Map
Random
SLidR
Ours
Random
SLidR
Ours
Figure 11: Qualitative results of fine-tuning 1% of the SemanticKITTI dataset with different pre-training strategies. Note that the results are shown as eиror maps on the right, where red points indicate incorrect predictions. Best viewed in color and zoom in for more details.
30

---

unknown
barrier
bicycle
bus
matorcycle
pedestrian
traffic cone
truck
road
sidewalk
terrain
vegetation
buildding
FRONT
LEFT
FRONT
FRONT RIGHT
BACK LEFT
BACK
BACK RIGHT
FRONT LEFT
FRONT
FRONT RIGHT
BACK LEFT
BACK
BACK RIGHT
Figure 12: Illustration of the weak semantic labels predicted by Grounded SAM. The top half of the figure displays the raw RGB images and LiDAR point clouds, while the bottom half presents the corresponding weak semantic labels applied to both images and point clouds, aligned using camera parameters. Each distinct segment represented by unique color. Best viewed in color.
unknown
barrier
bicycle
bus
car
motorcycle
pedestrian
traffic
truck
road
sidewalk
terrain
vegetation
building
FRONT LEFT
FRONT
FRONT RIGHT
BACK LEFT
BACK
BACK RIGHT
FRONT LEFT
FRONT
FRONT RIGHT
BACK LEFT
BACK
BACK RIGHT
Figure 13: Illustration of the weak semantic labels predicted by Grounded-SAMe The top half of the figure displays the raw RGB images and LiDAR point clouds, while the bottom half presents the corresponding weak semantic labels applied both images and point clouds, aligned using camera parameters. Each distinct segment is represented by unique color. Best viewed in color.
31

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
jusiifiaa)usiffcctttioo Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked use the checklist one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] " or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect the paper's ss contributions and scope. These sections effectively summarize the key aspects of the research. They clear expectations for the reader about the methodologies employed and the advancements over existing techniques.
Justification:
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the potential limitations of our method in Section A.6.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The focus of our paper is primarily on the application and empirical validation of VFMs for Image-to-LiDAR contrastive distillation, rather than on deriving new theoretical results. Consequently, our work does not introduce formal theorems or proofs that would necessitate detailed discussion of assumptions mathematical proof structure.
32

---

4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our paper provides detailed methodology and experimental settings necessary to reproduce the main results, supporting our core claims and conclusions.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We attach the code in supplementary materials, where we provide sufficient instructions to reproduce the main results. And we will release the source code on github in the future.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper meticulously details all training and testing parameters, including data splits, hyperparameters. selection criteria, and the type of optimizer used, ensuring complete understanding of the experimental results.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Error bars are not included due to the computational expense involved. How- ever, we generally ensure robustness against random seed variations by reporting results averaged over three independent experiments. For the fine-tuning on 3D detection task, we recorded the best results from the three experiments.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer the details in Section A.3.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https
Answer: [Yes]
Justification: Yes, we adhere to the NeurlPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the societal impacts of our paper in Section A.7.
11. Safeguards
33

---

Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Our paper does not involve the release of models or datasets that pose a high risk for misuse. Therefore, safeguards for responsible release and not necessary for this research. We will consider this issue when we release our pre-trained model.
Justification:
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the relevant papers and credit the term of used resource in Section A.8.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We attach the source with documentation in the supplementary materials anonymously.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
34


Reviews:
[{"review_id": "htDADWzZp1", "reviewer": "Reviewer_2WGA", "summary": "The paper addresses the \"self-conflict\" issue in contrastive image-to-LiDAR knowledge transfer, where features of semantically similar but unmatched points and pixels are unintentionally dissociated, compromising representation integrity. To solve this, Visual Foundation Models are employed to generate semantic labels for weakly-supervised pixel-to-point contrastive distillation. The method includes structuring the feature space with von Mises-Fisher distributions for consistency and adjusting sampling probabilities to handle spatial and category imbalances. Extensive experiments demonstrate that this approach significantly outperforms traditional methods in various downstream tasks.", "questions": "Please refer to the weaknesses section.", "limitations": "The paper discusses the limitations in the appendix.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper uses Visual Foundation Models to generate semantic labels, resolving the \"self-conflict\" issue and improving representation integrity.\n2. The paper proposes a density and category-aware sampling method, ensuring balanced learning and better representation of minority categories.", "weaknesses": "1. The overall architecture is similar to Seal [1], limiting its novelty except for the sampling strategy. Providing more clarification about the differences from Seal would be beneficial.\n2. The improvement in fine-tuning results compared to the state-of-the-art is marginal.\n\n\n[1] Segment any point cloud sequences by distilling vision foundation models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's time and effort in reviewing our paper. In the following, we will comprehensively address your concerns.\n\n---\n**Comment:** *The overall architecture is similar to Seal, limiting its novelty except for the sampling strategy. Providing more clarification about the differences from Seal would be beneficial.*\n\n**Response:** Our overall framework **significantly differs** from the existing method Seal [R1]. We would like to clarify the following points to highlight the novelty of our method:\n\n- The purposes of using VFMs in Seal [R1] and our method are completely different. To avoid over-segmenting semantically coherent areas, Seal [R1] generates superpixels using visual foundation models (VFMs) instead of the traditional method SLIC [R2]. In contrast, our method does not rely on superpixels. Although we also use VFMs, we leverage them to obtain coarse semantic **labels** for fine-grained contrastive distillation.\n- Although the more precise superpixels generated by VFMs could mitigate the self-conflict problems to some extent, such a method does not solve the problem thoroughly. The superpoints and superpixels with the same category may still be mistakenly considered negative pairs during contrastive learning. Our method explicitly defines the points and pixels with the same semantic labels as positive pairs during weakly-supervised contrastive learning. \n- Our pipeline performs knowledge distillation on two levels: self-supervised and weakly-supervised contrastive learning. To achieve this, we develop two different heads in both the image and point cloud branches to decouple the learned representation. Previous methods like Seal have **only attempted self-supervised** contrastive distillation and have not explored using labels to guide contrastive distillation.\n- We explicitly model the features of each class as a von Mises-Fisher (vMF) distribution, promoting feature consistency within the same category. This approach cultivates a meaningful and structured feature space, an aspect that Seal does not explore.\n- Existing methods like Seal [R1] are highly dependent on the generated superpixels. Superpixels balance asymmetries between areas with denser coverage of points and sparser areas in the contrastive loss. However, we do not need this process at all and ensure a uniform representation of both spatial and categorical dimensions by employing a novel sampling strategy.\n\nWe genuinely hope that these clarifications provide a clearer perspective on our research and its merits. Thanks again for your valuable time and feedback. We will further clarify the novelty and the differences with related methods like Seal [R1] detailedly in the revised manuscript.\n\n---\n**Comment:** *The improvement in fine-tuning results compared to the state-of-the-art is marginal.*\n\n**Response:** We agree with you that the improvement in downstream tasks compared to the state-of-the-art is not significant. But, we have to clarify the following points:\n- As stated in the manuscript, we believe that employing stronger visual foundation models for more precise semantic labels can lead to better 3D representations. Therefore, we obtained coarse labels with a stronger VFM, namely SEEM, and evaluated the learned 3D representation. As shown in the table below, our method outperforms Seal [R1] by a **significant** margin, achieving an improvement of 5.14\\% under the setting of linear probing.\n- We have **completely open-sourced** the code for OLIVINE, whereas existing state-of-the-art methods like HVDistill and Seal have **NOT** yet made their training code available. We believe this contributes positively to the image-to-point knowledge transfer community by promoting transparency and enabling further research.\n- Our method is **compatible** with existing techniques. For example, the semantic temporal consistency proposed in Seal [R1] and BEV-based contrastive distillation [R3] can also be integrated into our pipeline. We plan to explore these aspects further once the source code for these works is released.\n  \n[**Table A1**] Comparison of various pre-training techniques for semantic segmentation tasks using either finetuning or linear probing.\n| Method| LP | 1% | 5% | 10%| 25%| 100% |\n| :-------- | :----- | :----- | :----- | :----- | :----- | :----- |\n| Random | 8\\.10  | 30\\.30 | 47\\.84 | 56\\.15 | 65\\.48 | 74\\.66 |\n| PPKT | 35\\.90 | 37\\.80 | 53\\.74 | 60\\.25 | 67\\.14 | 74\\.52 |\n| SLidR| 38\\.80 | 38\\.30 | 52\\.49 | 59\\.84 | 66\\.91 | 74\\.79 |\n| ST-SLidR| 40\\.48 | 40\\.75 | 54\\.69 | 60\\.75 | 67\\.70 | 75\\.14 |\n| HVDistill| 39\\.50 | 42\\.70 | 56\\.60 | 62\\.90 | 69\\.30 | 76\\.60 |\n| Seal | 44\\.95 | 45\\.84 | 55\\.64 | 62\\.97 | 68\\.41 | 75\\.60 |\n| Ours | 50\\.09 | 50\\.60 | 60\\.25 | 65\\.07 | 70\\.15 | 76\\.69 |\n\n**References**:\\\n[R1] Liu et al. Segment Any Point Cloud Sequences by Distilling Vision Foundation Models. NeurIPS2023. \\\n[R2] Achanta et al. Slic superpixels compared to state-of-the-art superpixel methods. TPAMI2021. \\\n[R3] Zhang et al. HVDistill: Transferring Knowledge from Images to Point Clouds via Unsupervised Hybrid-View Distillation. IJCV2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eXqmayVjnA", "reviewer": "Reviewer_54Gn", "summary": "In this paper, the authors introduced a novel approach for improving 3D representation learning by leveraging VFMs to generate semantic labels for weakly-supervised pixel-to-point contrastive distillation. The proposed method addressed the self-conflict issue in traditional contrastive learning and presented a density and category-aware sampling strategy to ensure balanced learning. This approach showed the better performance over existing methods on nuScenes and SemanticKITTI datasets.", "questions": "There are some vague sentences and grammatical errors in the paper. I recommend that the author will revise the paper.", "limitations": "I mentioned all comments including reasons and suggestions in the above sections. I recommend that the author will provide all the concerns, and improve the completeness of the paper. If the rebuttal period resolves the above-mentioned concerns, I will gladly raise my score.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "First of all, the motivation of the paper seems to be meaningful and pragmatic in the perspective of the better semantic understanding by leveraging VFMs and balanced learning by using the density and category frequency. \n\nThe key idea is very intuitive how to integrate of VFMs with existing multi-modal SSL allowing for generating semantic labels to deal with self-conflict issues. More specifically, the model is trained by three objectives as weakly-supervised contrastive distillation using pseudo labels to identify positive pairs by category, self-supervised contrastive learning applied to randomly sampled point-pixel pairs, and lastly a regularization based on the von Mises-Fisher distribution to ensure semantic consistency.\n\nIn experimental section, the proposed method achieved SoTA results in two kinds of downstream tasks (segmentation and detection), demonstrating its effectiveness. The ablation study is highly analytical for each level module.", "weaknesses": "One concern is the validity of the proposed SSL method for better representation learning. First of all, it is not clear whether the reduced effectiveness with larger data is due to the model’s insufficient size (capability) or limitations in the proposed model itself. Also, an explanation is required to determine whether lower detection performance gain is due to the ineffectiveness of the proposed method, despite using object semantics. If necessary, the reasons for varying performance improvements across different downstream tasks should described in terms of the mechanism of the proposed learning pipeline.\n\nExperiment analysis and technical description are not specific and descriptive in some extent. The category-aware sampling is not specified in detail. There is not detailed description of performance variation based on sampled data groups or the extent of improvement over existing methods when learning from a sample of the entire dataset. (1%, 5%, 10%,…)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's time and effort in reviewing our paper. Thanks for your valuable comments and recognition of our work. In the following, we will comprehensively address your concerns.\n\n---\n**Comment:** *The validity of the proposed SSL method. It is not clear whether the reduced effectiveness with larger data is due to the model’s insufficient size or limitations in the proposed model.*\n\n**Response:** Thanks for your comments. We would like to clarify the following points to address your concerns:\n- When the available training data is limited, the benefits of pre-trained model weights on downstream tasks are more pronounced. This phenomenon is widely observed in self-supervised learning. When downstream task data is limited, these representations are crucial because they provide a strong starting point, capturing features that the model wouldn't learn from the small labeled dataset alone. As the amount of labeled data increases, the model can learn these features directly from the labeled data, making the initial representations from pretraining less critical.\n- We also conducted experiments with a stronger 3D backbone namely WaffleIron [R1] (see the Table B1). The effect of pre-training weights becomes less obvious when training downstream tasks on sufficient data. So the reduced effectiveness with larger data is not due to the capability of backbone.\n- We can further improve the performance with more accurate semantic labels generated by a stronger SAM like SEEM [R2]. As shown in the Table B2, our method achieves a 2.03\\% mIoU improvement on nuScenes with the full training data.\n- You might mean that the improvement compared to other pretraining methods is not obvious. We believe the main value of self-supervision methods is to improve the performance when the annotation resources are limited. And OLIVINE outperforms existing methods **significantly** when the annotation is limited. \n\n**Table B1**: Performance for 3D backbone WaffleIron\n| Method | 1% | 10%| 100% |\n| :----- | :----- | :----- | :----- |\n| Random | 33\\.26 | 58\\.13 | 77\\.60 |\n| Ours | 50\\.14 | 66\\.43 | 78\\.21 |\n\n**Table B2**: Comparison of various pre-training techniques.\n| Method| LP | 1% | 5% | 10%| 25%| 100% |\n| :-------- | :----- | :----- | :----- | :----- | :----- | :----- |\n| Random| 8\\.10| 30\\.30 | 47\\.84 | 56\\.15 | 65\\.48 | 74\\.66 |\n| Seal| 44\\.95 | 45\\.84 | 55\\.64 | 62\\.97 | 68\\.41 | 75\\.60 |\n| Ours| 50\\.09 | 50\\.60 | 60\\.25 | 65\\.07 | 70\\.15 | 76\\.69 |\n\n---\n**Comment:** *An explanation for lower detection performance gain is required. The reasons for varying performance improvements across different downstream tasks should described in terms of the mechanism of the proposed learning pipeline.*\n\n**Response:** Thanks for your insightful questions. We provide the following explanations to address your concerns:\n- We observed a 2.0\\% mAP improvement with the SECOND and a 1.5\\% mAP improvement with the PV-RCNN, surpassing previous pretraining methods. These improvements were achieved by fine-tuning on full training data, so the enhancements may appear less significant compared to using limited labels.\n- Compared to the semantic segmentation task, the model architecture for object detection is more complex. Besides the 3D backbone, 3D detectors typically project features to a BEV plane, followed by a 2D convolutional network and RoI operations. These crucial components were not pre-trained, which may limit the overall performance gain from our pre-training approach.\n- It's important to note that semantic segmentation and object detection use different metrics and scales, making direct performance comparisons improper. The nature of these tasks and their evaluation criteria inherently lead to varying degrees of improvement when applying our proposed method.\n\n---\n**Comment:** *Experiment analysis and technical description are not specific and descriptive in some extent. The category-aware sampling is not specified in detail. There is not detailed description of performance variation based on sampled data groups from a sample of the entire dataset.*\n\n**Response:** Thanks for pointing out this issue. Category-aware and density-aware sampling determine the sampling probability of a point by its category frequency and distance information, respectively. These are part of a hybrid strategy we refer to as density and category-aware sampling (DCAS). Following your suggestion, we have added a comparison of sampling strategies using 1\\%, 5\\%, 10\\%, 25\\%, and 100\\% of the annotated data from nuScenes. The results are presented in the table below. We found that the density and category-aware sampling strategy consistently achieves the **best** performance on downstream tasks, effectively leveraging both spatial distribution and category frequency.\n\n|Sampling|1%|5%|10%|25%|\n|:-------------|:-----|:-----|:-----|:-----|\n|Random|44\\.91|56\\.01|62\\.58|68\\.74|\n|Density-aware|45\\.33|56\\.6|62\\.74|68\\.96|\n|Category-aware|45\\.74|56\\.98|62\\.89|69\\.18|\n|DCAS (Density and Category-aware)|46\\.12|57\\.51|63\\.04|69\\.39|\n\n---\n**Comment:** *There are some vague sentences and grammatical errors in the paper.*\n\n**Response:** Thank you for your feedback. We appreciate your attention to detail. We have thoroughly reviewed the manuscript, revised the vague sentences, and corrected the grammatical errors.\n\nWe genuinely hope that these clarifications address your concerns. Thanks again for your valuable time and feedback. We will include the results and analysis in the revised manuscript.\n\n---\n**References**:\\\n[R1] Puy et al. Using a Waffle Iron for Automotive Point Cloud Semantic Segmentation. ICCV2023.\\\n[R2] Zou et al. SEEM: Segment Everything Everywhere All at Once. NeurIPS2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AFJIfKmtBH", "reviewer": "Reviewer_uKMo", "summary": "Annotating point clouds with semantic classes can be expensive and time consuming. The authors of this work propose a new pretraining strategy for weakly supervising point cloud segmentation using image-based supervision (i.e., image-to-LIDAR knowledge transfer). The proposed approach improves upon traditional contrastive pretraining strategies by leveraging visual foundation models (VFMs, e.g., SAM) to provide weak supervision for associating LiDAR points with corresponding pixels that have matching semantic classes. The authors also model features using von Mises-Fisher distributions to further encourage feature semantic clustering, and improve upon sampling by incorporating spatial distances of points and class frequency. This approach shows impressive state-of-the-art performance on pretraining across two benchmark datasets (SemanticKITTI and nuScenes). The ablation study also carefully highlights the impact of each of the contributions. This work will likely serve as a healthy addition to the image-to-point knowledge transfer community.", "questions": "1. What is the impact of using Grounded SAM vs other VFMs for this approach?\n2. Which existing methods make the semantic unmatching mistake mentioned in L148?", "limitations": "Yes, the authors clearly describe the limitations of the approach as it pertains to the (1) accuracy of the pseudo-labels derived from the VFM, (2) the diversity of the training data impacting environment adaptation, and (3) the dependency on highly calibrated cameras and LiDAR sensors to ensure knowledge transfer.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors identify an, evidently, common problem in point and pixel contrastive learning and address this issue with the proposed method. Namely, the authors (or perhaps Mahmoud et al. [36], see limitations section) recognize that prior works do not ensure semantic consistency when performing contrastive learning for image to point cloud knowledge transfer. This issue causes objects of the same class (e.g., car) to be pushed apart in feature space, simply because they are not part of the same super pixel. The authors address this using weakly contrastive distillation to ensure semantic consistency across anchor points.\n2. State-of-the-art results by pretraining on nuScenes and SemanticKITTI across a wide range of annotation data limitations.\n3. A detailed and thorough evaluation on multiple benchmarks and an extensive ablation study providing insightful results. The ablation study, in particular, shows the impact of weakly-supervised labels, separate projection heads, different distributions for modeling semantic features, and various sampling strategies.\n4. The authors are tackling an interesting and challenging problem of improving knowledge transfer across modalities. This research area is of particular importance given the decreased interest and investment in annotating campaigns by the community, and increased interest in self-supervised methods.", "weaknesses": "1. The related work section does not adequately differentiate this approach from prior works. \n- While the related work section does cite relevant works, it does not identify how the shortcomings of any of these works are addressed in this paper. Moreover, the related work section does not isolate how this paper is different, unique, or better than any of the existing approaches at image to point cloud knowledge transfer. \n- In particular, I found Mahmoud et al.'s [36] approach, for feature similarity from pixels to points and class balancing, sharing many commonalities with the proposed approach, hence, a detailed comparison may be warranted. \n- Liu et al. [33] also leverage VFMs like SAM for semantic segmentation to improve image to point cloud knowledge transfer, which is strikingly similar to the proposed approach. Detailed comparisons would greatly help clarify these commonalities, and it would strengthen the reader's confidence in the novelty of the proposed approach.\n\t\nMinor:\n- L148: Which existing methods make the semantic unmatching mistake? While this may have been briefly mentioned [36] in the introduction, there was no clear statement with multiple cited works to support this claim. Consider citing these (uncited) prior works to provide evidence for this claim.\n- Tables 2 and 3 could be combined, it seems somewhat unnecessary to keep them separated.\n- Typographical/grammatical errors: L90, L155, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and effort in reviewing our paper, the valuable comments, and the favorable recommendation.\n\n**Q:** *To differentiate this approach from prior works.*\n\n**A:** We agree with you that it's necessary to highlight the shortcomings of previous works and the novelty of OLIVINE in the related work section. Here, we would like to clarify the following points:\n- Previous works [R1-R5] have not solved the self-conflict problem properly. Especially, Seal [R4] generates semantically coherent superpixels for distinct objects and backgrounds in the 3D scene. However, the superpoints and superpixels with the same category may still be mistakenly considered negative pairs during contrastive learning. By contrast, our method explicitly defines the points and pixels with the same semantic labels as positive pairs during weakly-supervised contrastive learning.\n- Our pipeline performs knowledge distillation on two levels: self-supervised and weakly-supervised contrastive learning. To achieve this, we develop **two** different heads in both the image and point cloud branches to **decouple** the learned representation. Previous methods [R1-R5] have **only attempted self-supervised** contrastive distillation and have not explored using labels to guide contrastive distillation.\n- The representation of samples in the same class can vary significantly across different batches during the contrastive distillation, so the model will struggle to learn stable semantic features. By making point features of the same class closely aligned, our method aims to create a more consistent and structured feature space.\n- Existing methods [R2-R5] are highly dependent on the generated superpixels. Superpixels balance asymmetries between areas with denser coverage of points and sparser areas in the contrastive loss. However, we do not need this process at all and ensure a uniform representation of both spatial and categorical dimensions by employing a novel sampling strategy.\n\n**Q:** A detailed comparison with Mahmoud et al [36].\n\n**A:** Thanks for your suggestion. The main differences between ours and ST-SLidR [R3] are:\n- ST-SLidR [R3] reduces the contribution of false negative samples based on superpixel-to-superpixel similarity, using 2D self-supervised features to determine semantic similarities between superpixels. By contrast, our method directly estimates the semantic labels of images with VFMs, and defines pixels and points with the same label as positive pairs.\n- Regarding class balancing, ST-SLidR [R3] assigns higher weights to over-represented anchors that exhibit high similarities to most negative samples. By contrast, our approach directly adjusts the sampling probability of anchor points using easily accessible semantic labels.\n\nIn summary, our OLIVINE offers a more direct and effective way to mitigate the effect of false negative samples and class imbalance.\n\n**Q:** *Liu et al. [33] also leverage VFMs. Detailed comparisons help clarify these commonalities.*\n\n**A:** Thanks for your suggestions to compare Seal [R4] and OLIVINE. We would like to clarify the following points:\n- To avoid over-segmenting semantically coherent areas, Seal [R4] generates superpixels using VFMs instead of the traditional method SLIC. In contrast, our method does not rely on superpixels. Although we also use VFMs, we leverage them to obtain coarse semantic **labels** for fine-grained contrastive distillation.\n- In method Seal [R4], the superpoints and superpixels with the same category may still be mistakenly considered negative pairs during contrastive learning. Our method explicitly defines the points and pixels with the same semantic labels as positive pairs during weakly-supervised contrastive learning. \n- The semantic labels generated by VFMs, rather than superpixels, can be flexibly utilized in multiple aspects of the knowledge transfer process, such as weakly-supervised contrastive distillation, semantic-guided consistency regularization, and category-aware anchor point sampling. These aspects cannot be effectively addressed using only the class-agnostic superpixels.\n\n**Q:** *L148: Which methods make semantic unmatching mistake?*\n\n**A:** Thank you for your valuable feedback. Existing methods [R1, R2, R4, R5] may mistakenly treat unmatched (super)points and (super)pixels in the same category as negative pairs during contrastive distillation. We will cite these methods to support this claim in the revised manuscript to provide clear evidence.\n\n**Q:** *Tables 2 and 3 could be combined. Typo errors...*\n\n**A:** We appreciate your attention to detail. We have combined Tables 2 and 3 to streamline the presentation and carefully corrected the typo errors.\n\n**Q:** *Impact of Grounded SAM vs other VFMs for this approach?*\n\n**A:** Thanks for your question. Our response is as follows:\n- Grounded SAM supports text prompts by combining Grounding DINO and SAM. Other VFMs that enable text prompts can also be applied in OLIVINE.\n- The precision of the semantic labels significantly impacts the effectiveness of OLIVINE. Stronger VFMs provide more accurate semantic labels, leading to better learned representations. As shown in the table below, the potential of our method can be further unleashed by using a stronger VFM, namely SEEM [R6].\n\n|VFMs|LP|1%|5%|10%|25%|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|Grounded-SAM|47\\.30|46\\.12|57\\.51|63\\.04|69\\.39|\n|Grounded-SAM-HQ|47\\.84|48\\.03|58\\.51|64\\.08|69\\.52|\n|SEEM|50\\.09|50\\.60|60\\.25|65\\.07|70\\.15|\n\n**Ref**:\\\n[R1] Learning from 2d: Contrastive pixel-to-point knowledge transfer for 3d pretraining.\\\n[R2] Image-to-lidar self-supervised distillation for autonomous driving data.\\\n[R3] Self-supervised image-to-point distillation via semantically tolerant contrastive loss.\\\n[R4] Segment Any Point Cloud Sequences by Distilling Vision Foundation Models.\\\n[R5] HVDistill: Transferring Knowledge from Images to Point Clouds via Unsupervised Hybrid-View Distillation.\\\n[R6] Segment Everything Everywhere All at Once.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P8fCEoWdti", "reviewer": "Reviewer_svRz", "summary": "This work aims to tackle the image-to-LiDAR contrastive learning problem for LiDAR-based point cloud segmentation. Previous approaches designed the cross-modal contrastive learning objective for model pretraining, using superpixels and superpoints as guidance.\n\nIn this work, the authors observe that the superpixel-driven contrastive loss tends to involve ‘’self-conflict’’ issues during representation learning. A weakly-supervised contrastive distillation method is proposed, which generates semantic superpixels/superpoints using the Segment Anything Model (SAM). Additionally, to balance the imbalanced class distributions of LiDAR scene categories during representation, a density and category-aware sampling strategy is proposed to adjust the sampling probabilities of different anchor points using the weak semantic labels.\n\nThe overall framework is named OLIVINE, which adopts three optimization objectives:\n- Weakly-supervised contrastive distillation using coarse semantic labels to identify positive pairs by category.\n- Self-supervised contrastive distillation applied to randomly sampled point-pixel pairs.\n- A regularization framework based on the von Mises-Fisher (vMF) distribution to ensure semantic consistency.\n\nThe proposed OLIVINE method is evaluated on the nuScenes, SemanticKITTI, and KITTI object detection datasets. The results exhibit a consistent improvement of the proposed method compared to existing approaches.", "questions": "- **Q1:** As mentioned in Weakness 1, the semantic masks generated by the Segment Anything Model could inevitably involve errors (e.g., wrong segmentation results). How do the authors handle the propagated errors during image-to-LiDAR representation learning?\n\n- **Q2:** As mentioned in Weakness 2, could the authors provide more details on the hyperparameter settings for the vMF distribution and the reasoning behind their chosen values? Adding a more detailed explanation and theoretical justification would be even better.\n\n- **Q3:** As mentioned in Weakness 3, having more thorough experimental analyses on other LiDAR-based point cloud datasets, such as SemanticPOSS, Waymo, SynLiDAR, etc., could further consolidate the findings and conclusions drawn in the manuscript.\n\n- **Q4:** As most 2D and 3D representation learning approaches (MoCo, SimCLR, Seal, etc.) do, having empirical analyses of models under out-of-distribution datasets is recommended. \n\n- **[Minor]:** The computational cost of the proposed multi-modal contrastive distillation approach is not thoroughly analyzed, which is crucial for real-time applications in autonomous driving.\n\n- **[Minor]:** The generalizability of OLIVINE to other types of sensors (for example, hybrid-solid LiDARs) or environments (for example, off-board environments) beyond the evaluated datasets is not discussed.\n\n- **[Minor]:** “NuScenes” should be revised to “nuScenes”.", "limitations": "The authors mentioned \"Semantic Label Accuracy\" as one of their limitations. As also discussed in Weakness 1, more analyses are needed to address the impact of inaccuracies in the weak labels generated by the Segment Anything Model. These inaccuracies could propagate errors during the image-to-LiDAR representation learning process, potentially affecting the overall performance of the proposed method.\n\nAdditionally, while the von Mises-Fisher distribution is used for consistency regularization, the motivation and theoretical foundation for its use are not fully elaborated. A deeper exploration of its advantages and potential drawbacks in this context would be beneficial.\n\nThe computational cost associated with the multi-modal contrastive distillation approach is another important aspect that is not thoroughly analyzed. For practical applications, especially in real-time scenarios such as autonomous driving, it is crucial to understand the resource requirements and efficiency of the proposed method.\n\nLastly, the scalability and generalizability of OLIVINE to other sensor types and different environments have not been extensively discussed. Exploring its applicability in diverse settings and with various sensor configurations would provide a more comprehensive evaluation of its robustness and versatility.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(+) This work aims to improve the image-to-LiDAR self-supervised representation learning problem on LiDAR-based point cloud datasets, which is one of the current research hotspots, especially for applications related to autonomous driving and robotics.\n\n(+) The proposed method has exhibited promising performance on mainstream benchmarks, including nuScenes linear probing, nuScenes fine-tuning, SemanticKITTI fine-tuning, and KITTI object detection.", "weaknesses": "(-) The weakly-supervised contrastive distillation method has been used in previous literature, such as [R1] and [R2]. Adding semantic categories seems not to cause a major improvement over class-agnostic masks, as the Segment Anything Model is able to segment rather complete and semantically consistent objects and backgrounds. Additionally, using weak labels (which might be erroneous) could introduce additional errors during pretraining.\n\n(-) The motivation for using the von Mises-Fisher (vMF) distribution to enforce consistency regularization for image-to-LiDAR representation learning is not clear enough to demonstrate its superiority. A more detailed explanation and theoretical justification would strengthen this aspect of the work.\n\n(-) Compared to some of the most related works, for example, [R1] and [R3], the scale and depth regarding the experiments (for example, downstream fine-tuning on other datasets than SemanticKITTI) could be further enhanced.\n\n---\n\n### References:\n- [R1] Youquan Liu, et al. “Segment Any Point Cloud Sequences by Distilling Vision Foundation Models,” NeurIPS, 2023.\n- [R2] Ayça Takmaz, et al. “OpenMask3D: Open-Vocabulary 3D Instance Segmentation,” NeurIPS, 2023.\n- [R3] Gilles Puy, et al. “Revisiting the Distillation of Image Representations into Point Clouds for Autonomous Driving,” arXiv, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and effort in reviewing our submission and valuable comments. In the following, we will address your concerns and correct the potential misunderstandings.\n\n**Q:** *The weakly-supervised contrastive distillation method has been used in previous literature [R1, R2].*\n\n**A:** We believe there may be a **misunderstanding** regarding the mentioned methods.\n Seal [R1] generates semantically coherent superpixels for distinct objects and backgrounds in the 3D scene. However, it does not infer semantic labels or use them to supervise contrastive distillation. Consequently, superpoints and superpixels within the same category may still be mistakenly considered negative pairs during pretraining. [R2] is **NOT** relevant to contrastive distillation or weakly-supervised learning.\n\n**Q:** *Adding semantic categories seems not to cause a major improvement over class-agnostic masks, as the SAM is able to segment rather complete and semantically consistent objects and backgrounds.*\n\n**A:**  Although Seal [R1] uses VFMs to generate semantically coherent superpixels, it can still mistakenly treat superpoints and superpixels of the same category as negative pairs during contrastive distillation. Our method explicitly defines the points and pixels with the same semantic label as positive pairs during weakly-supervised contrastive learning. Besides, our method can achieve better performance with weak labels generated by stronger VFMs. We refer you to Table M1 of the uploaded PDF file for the additional results.\n\n**Q:** *Using weak labels could introduce additional errors during pretraining.*\n\n**A:** We acknowledge there is a trade-off. While the weak labels might be erroneous, they enable semantic-guided image-to-LiDAR contrastive distillation and **indeed** yield state-of-the-art performance on downstream tasks. Similarly, the superpixels widely used in previous image-to-LiDAR knowledge transfer methods can also be inaccurate but effective.\n\n**Q:** *The motivation for using the vMF distribution is not clear enough. A more detailed explanation and theoretical justification would strengthen this aspect.*\n\n**A:** Thank you for your valuable suggestion. We provide more explanations as follows.\n- The representation of samples in the same class can vary significantly across different batches during the contrastive distillation, so the model will struggle to learn stable semantic features. By making point features of the same class closely aligned, our method aims to create a more consistent and structured feature space. \n- The vMF distribution is defined on a hypersphere, making it well suited for directional data in feature space. The concentration parameter can be dynamically adapted during training to refine the feature alignment process. Early in training, a lower $\\kappa$ might allow for more exploration, while later stages can benefit from higher $\\kappa$ to solidify the learned representations.\n- Due to the space limitation of each response, we will provide more theoretical justification in another window.\n\n**Q:** *The experiments could be enhanced on more datasets.*\n\n**A:**  Following your valuable suggestion, we conducted experiments on more datasets. **We refer you to the Table M2 in the uploaded PDF file.**\n\n**Q:** *How do the authors handle the propagated errors during image-to-LiDAR representation learning?*\n\n**A:** Thank you for raising this insightful point. We acknowledge that the inaccuracy of the labels generated by the SAM is a limitation of the current pipeline. We are actively developing a new label disambiguation module for future work. This module will utilize the learned feature similarities to refine the coarse labels. We believe that weak labels can help mitigate self-conflict, while structured semantic representations can assist in refining these labels. Together, they mutually reinforce each other, ultimately leading to more robust and accurate representations.\n\n**Q:** *Could authors provide details on the hyperparameter settings for the vMF distribution and the reason behind chosen values?*\n\n**A:** Regarding the vMF distribution, we **did not** set many hyperparameters. The mean direction and the concentration parameter are learned from the statistical values of features via the EMA (Exponential Moving Average) algorithm. The smoothness coefficient for the moving average is empirically set to 0.0001.\n\n**Q:** *As most 2D and 3D representation learning approaches do, having empirical analyses of models under out-of-distribution datasets is recommended.*\n\n**A:** Following your valuable suggestion, we added experiments on nuScenes-C datasets. **We refer you to the Table M3 in the uploaded PDF file.**\n\n**Q:** *The computational cost of the proposed approach is not analyzed, which is crucial for real-time applications.*\n\n**A:** Thanks for your comments. Our approach only provides pre-trained weights, which **do not affect** the inference speed of the model on downstream tasks. As shown in the table below, our OLIVINE does not require obviously more GPU memory or training time compared to other pre-training methods.\n\n|Method|GPUMemory (GB)|TrainingTime (Hour)|\n|:-|:-:|:-:|\n|PPKT|7\\.6|35\\.7|\n|SLidR|10\\.7|38\\.9|\n|OLIVINE|8\\.1|36\\.5|\n\n**Q:** *The generalizability of OLIVINE to other types of sensors or environments is not discussed.*\n\n**A:** Following your suggestion, we have supplemented experiments on another six datasets, which demonstrate the generalizability of OLIVINE to some extent (**see Table M2 of the uploaded PDF file**). Since the computational resources are limited, we will further enhance the experiments part after the rebuttal.\n\n**Q:** *\"NuScenes\" should be \"nuScenes\".*\n\n**A:** Thanks for your careful review. We will correct the typo in the final version.\n\n**References**:\\\n[R1] Liu et al. Segment Any Point Cloud Sequences by Distilling Vision Foundation Models. NeurIPS2023.\\\n[R2] Takmaz et al. OpenMask3D: Open-Vocabulary 3D Instance Segmentation. NeurIPS2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for your time and constructive comments. 

---
We are glad that the reviewers see the value in our work:
1. "_The paper addresses the self-conflict issue in contrastive image-to-LiDAR knowledge transfer ... significantly outperforms traditional methods in various downstream tasks_" (Reviewer 2WGA);
2. "_the motivation of the paper seems to be meaningful and pragmatic  in the perspective of the better semantic understanding_" (Reviewer 54Gn);
3. "_The key idea is very intuitive how to integrate VFMs with existing multi-modal SSL_" (Reviewer 54Gn);
4. "_The ablation study is highly analytical for each level module_" (Reviewer 54Gn);
5. "_This work will likely serve as a healthy addition to the image-to-point knowledge transfer community_" (Reviewer uKMo).
---
We would like to emphasize the **uniqueness** and **advantages** of our approach over existing ones:
1. Previous works [R1-R5] have not solved the self-conflict problem properly. Especially, Seal [R4] generates semantically coherent superpixels for distinct objects and backgrounds in the 3D scene. However, the superpoints and superpixels with the same category may still be mistakenly considered negative pairs during contrastive learning. By contrast, our method explicitly defines the points and pixels with the same semantic labels as positive pairs during weakly-supervised contrastive learning.
2. Our pipeline performs knowledge distillation on two levels: self-supervised and weakly-supervised contrastive learning. To achieve this, we develop **two** different heads in both the image and point cloud branches to **decouple** the learned representation. Previous methods [R1-R5] have **only attempted self-supervised** contrastive distillation and have not explored using labels to guide contrastive distillation.
3. The representation of samples in the same class can vary significantly across different batches during the contrastive distillation, so the model will struggle to learn stable semantic features. By making point features of the same class closely aligned, our method aims to create a more consistent and structured feature space.
4. Existing methods [R2-R5] are highly dependent on the generated superpixels. Superpixels balance asymmetries between areas with denser coverage of points and sparser areas in the contrastive loss. However, we do not need this process at all and ensure a uniform representation of both spatial and categorical dimensions by employing a novel sampling strategy.
5. ST-SLidR [R3] reduces the contribution of false negative samples based on superpixel-to-superpixel similarity, using 2D self-supervised features to determine semantic similarities between superpixels. By contrast, our method directly estimates the semantic labels of images with VFMs, and defines pixels and points with the same label as positive pairs.
6. The purposes of using VFMs in Seal [R4] and our method are completely different. To avoid over-segmenting semantically coherent areas, Seal [R4] generates superpixels using visual foundation models (VFMs) instead of the traditional method SLIC [R6]. In contrast, our method does not rely on superpixels. Although we also use VFMs, we leverage them to obtain coarse semantic **labels** for fine-grained contrastive distillation.
---
Following the reviewers' valuable comments and suggestions, we have made these efforts:
1. We have achieved further improvements on downstream tasks using semantic labels generated by stronger VFMs, as suggested by Reviewer 2WGA.
2. We have highlighted the novelty of our method and clarified the differences from previous methods, as suggested by Reviewers 2WGA and uKMo.
3. We discussed the reasons for varying performance improvements across different downstream tasks, considering the mechanism of the proposed learning pipeline, as suggested by Reviewer 54Gn.
4. We have supplemented the experiment analysis and provided a technical description of sampling strategies, as suggested by Reviewer 54Gn.
5. We have added relevant citations to support our claim in L148, as suggested by Reviewer uKMo.
6. We have combined Tables 2 and 3 to streamline the presentation and carefully corrected typographical errors, as suggested by Reviewers uKMo and svRz.
7. We have compared the effects of different VFMs for generating semantic labels, as suggested by Reviewer uKMo.
8. We have provided a detailed explanation and theoretical justification for the application of the vMF distribution, as suggested by Reviewer svRz.
9. We have added experiments on six additional LiDAR-based point cloud datasets and one out-of-distribution dataset, as suggested by Reviewer svRz.
10. We have reported the computational cost of the proposed pretraining method, as suggested by Reviewer svRz.
---
Finally, we extend our gratitude to the PCs, ACs, and all the reviewers for their dedicated time and effort in this review process. We look forward to engaging in discussions with you over the next few days.

---
**References**:\
[R1] Liu et al. Learning from 2d: Contrastive pixel-to-point knowledge transfer for 3d pretraining. arxiv2021.\
[R2] Sautier et al. Image-to-lidar self-supervised distillation for autonomous driving data. CVPR2022.\
[R3] Mahmoud et al. Self-supervised image-to-point distillation via semantically tolerant contrastive loss. CVPR2023.\
[R4] Liu et al. Segment Any Point Cloud Sequences by Distilling Vision Foundation Models. NeurIPS2023.\
[R5] Zhang et al. HVDistill: Transferring Knowledge from Images to Point Clouds via Unsupervised Hybrid-View Distillation. IJCV2024.

Meta-review: After the discussion phase all reviewers recommend acceptance, noting the clear motivation/presentation, compelling results, and thorough evaluation. A substantial rebuttal was submitted that helped to address a majority of the reviewer concerns, including adding additional details, theoretical justifications, and experiments. As such, the ACs reached a decision to accept the paper. Please take the reviewer feedback into account when preparing the camera ready version.
Decision: Accept (poster)