CLAP: Unsupervised 3D Representation Learning for Fusion 3D Perception via Curvature Sampling and Prototype Learning
Authors: Runjian Chen
Hang Zhang
Avinash Ravichandran
Hyoungseob Park
Wenqi Shao
Alex Wong
Ping Luo
Venue: ICLR 2026 Poster (2026)
Keywords: Unsupervised 3D Representation Learning; Fusion Perception; Autonomous Driving
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Joint unsupervised 3D representation learning for fusion perception with Camera and LiDAR.

Abstract: Unsupervised 3D representation learning reduces the burden of labeling multimodal 3D data for fusion perception tasks. Among different pre-training paradigms, differentiable-rendering-based methods have shown most promise. However, existing works separately conduct pre-training for each modalities due to computational costs of processing large point clouds with images. As such, mutual benefit of high-level semantics (from image) and 3D structure (from point cloud) has not been exploited. To address this gap, we propose a joint unsupervised differentiable-rendering-based pre-training method for images and point clouds, termed CLAP, short for Curvature sampLing and leArnable Prototype. Specifically, our method overcomes the computational hurdle by Curvature Sampling to select the more informative points/pixels for pre-training. To uncover the performance benefits brought by their complementarity, we propose to use learnable prototypes to represent parts of the 3D scenes in a common feature space and an Expectation-Maximization training scheme to associate embeddings of each modality to prototypes. We further propose a swapping prediction loss that explores their interplay through prototypes along with a Gram Matrix Regularization term to maintain training stability. Experiments on NuScenes and Waymo datasets show that CLAP achieves up to 100% more performance gain as compared to previous SOTA pre-training methods. Codes and models will be released.

Published as a conference paper at ICLR 2026
CLAP: UNSUPERVISED 3D REPRESENTATION LEARN- ING FOR FUSION 3D PERCEPTION VIA CURVATURE SAMPLING AND PROTOTYPE LEARNING
Runjian Chen' Hang Zhang Avinash Ravichandran Hyoungseob Park2
Wenqi Shao Alex Wong Ping Luol.5x
I 'The University of Hong Kong
Cruise Yale University
"Shanghai AI Laboratory *HKU Shanghai Intelligent Computing Research Center
{richen, pluo}fes. hku.nk alex. wong3yale edd
ABSTRACT
Unsupervised 3D representation learning reduces the burden of labeling mul- timodal 3D data for fusion perception tasks. Among different pre-training paradigms, fffferentiible-renderiin based methods have shown most promise. However, existing works separately conduct pre-training for each modalities due to computational costs of processing large point clouds with images. As such, mutual benefit of high-level semantics (from image) and 3D structure (from point cloud) has not been exploited. To address this gap, we propose joint unsu- pervised differentiable--rnderigg ceapbleerenddiin-bbseeesee -based pre-training method for images and point clouds, termed CLAP, short for Curvature sampLing and leArnable Prototype. Specifically, our method overcomes the computational hurdle by Curvature Sam- pling to select the more informative points/pixels for pre-training. To uncover the performance benefits brought by their complementarity, we propose to learn- able prototypes to represent parts of the 3D scenes in a common feature space and an Expectation- Expectation-Muxiiizziin Maximization training scheme to associate embeddings of each modality to prototypes. We further propose swapping prediction loss that ex- plores their interplay through prototypes along with a Gram Matrix Regularization term to maintain training stability. Experiments on NuScenes and Waymo datasets show that CLAP achieves up to 100% more performance gain as compared to pre- vious SOTA pre-training methods. Codes and models will be available here
INTRODUCTION
3D perception facilitates spatial applications such as autonomous driving. The autonomous system, on which these applications are deployed, are typically equipped with multiple sensors including visual ones like cameras that produce RGB images, and range sensors like LiDAR Light-Detection
And-Ranging that generate point clouds. Fusion of these two modalities (Liu et al. 2023
Liang
ct al. 2022] Li et al. 2022b Liang et al. 2019 Li et al. 2022a Meyer et al.] 2019] Chen et al.
2017)
have generally improved over use of single modality. e.g., camera (Ding al. 2020 Chen et al.
2016 Simonelli et al.; 2019 Wang et al. 2021b 2021b:Zhang et al. 2021 Ku et al.
2019 or LiDAR (Shi
al. 2019 Yan al. 20188 Shi al. separately, in 3D perception performance.
2021
Yin et al. 2021 Fan
al. 2021 Bai al. 2022)
However, training multimodal 3D perception models is expensive labeling in 3D) space is no- wendddeeery-conummiiin Unsupervised 3D representation learning, which pre-trains backbones without any label and fine-tuned the pre-trained weight for downstream performance im- provement, has shown the potential to alleviate the labeling burden in 3D perception. Amongst the many unsupervised 3D pre- training methods (Liang et al. 2021 HHang al.) 2021 Chen et al.) 2022 Yang et al., [2023] et all 2023] Huang et al.) 2023 Zhu et al.] 2023 Yang et al. 2024), use of mask auto-encoding (reconstruction) and differntiable rendering has emerged as the most per- formant. However, encoding and processing high-dimensional multi-modal data (images and point
toriously
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
Curoniure Eatimarias
Uddr Reconstuction
Lidor Poin Cloud
Suapping Protogpe Ansigement
Fusion
Comera Encocer
For Recon Loss
Sumoundi Comora
Sumooudd ding Carrena
Predous 30 pre-imening wtt.eeeetttttteeeeeeinn deparste pra-tmining
3od Our propered method API jointty in peicptpti
Figure 1: Unlike previous SOTA unsupervised 3D representation learning method UniPAD (Yang et al. 2024) that separately pre-train LiDAR and camera encoders with differential rendering (a), our proposed method CLAP conducts joint pre-training for fusion perception.
clouds) is computationally expensive: If one were to pre-train using all points and pixels within the point cloud and image, even the most advanced GPU date only able to hold batch size of 1. Therefore, existing methods, e.g., UniPad (Yang et al. 2024), has conventionally pre- pre-trained each modality separately. The limitation of separate pre-training is that each encoder is restricted to its own modality. Recovering 3D information from images is an ill-posed problem; point clouds provide geometry cues but lack higher-level semantics.
To address this pain point, we propose joint unsupervised pre-training method of image and point cloud modalities based on differentiable rendering to exploit their complementarity. Our method, CLAP, short for Curvature sampLing and leArnalle Prototype, addresses the primary computational challenge by a curvature sampling strategy, which comes from the observation that there exists redundancy in information when sampling multiple points on the same flat surface. This is enabled by estimating the curvature of each point in the 3D space by taking second order derivative of the SDF (signed distance field) function. This sampling strategy captures variations in the point cloud and in turn provide more informative points than the conventional strategy in (Yang et al. 2024).
As our curvature sampling strategy has reduced the computational burden to allow both modalities to be simultaneously processed, we propose to model the interplay between image and point cloud modalities through set of Tearnable prototypes, which represents parts of the 3D scene and enables common feature space to bridge the two modalities. These prototypes are trained via an Expectation- Maximization (EM) algorithm that maximizes similarity between embeddings for each modality and the set of prototypes. Furthermore, we propose to use the swapping prediction loss to explore the interaction between the two modalities. Last but not least, we utilize Gram Matrix Regularization term to minimize similarity across prototypes, avoiding collapse of prototype when trained naively.
Our contributions are 4-folds: (1) We propose curvature sampling strategy identify informa- tive points (and pixels) for sampling, which enables the first joint differentiable-renderii beeiiibbleeeederrng-bbseedd pre-training method for fusion perception.
(2) Learnable prototypes are utilized to learn a com- Pnpectation-Maximizztionn approach is proposed to train the prototypes to represent parts of the 3D scene. (3) We further propose to use a swapping prediction loss for modality interplay and Gram Matrix Regularization loss that avoids collapse of prototype learning. (4) Through extensive experiments on the popular autonomous driving datasets NuScenes (Caesar) et al. 2020) and Waymo (Sun et al. 2020), we demonstrate the effectiveness of CLAP For exam- ple, CLAP achieves up to 100% more improvement than previous SOTA 3D pre-training methods and shows potential scaling property.
mon feature space and
RELATED WORK
Ranging (LiDAR) and camera important sensors for autonomous driving perception. Previous works mainly focus on single-modality 3D perception. For LiDeR-based 3D object detection, there three main streams with different em-
Fusion 3D Object Detection. Light-Detection-AAe Annnaainnn
2

---

Published as a conference paper at ICLR 2026
bedding schemes for point clouds inputs. 1) Point-bused methods (Shi et al., 2019: 2020b) utilize point-level embeddings for 3D object detection. 2) Voxel-based methods (an et al. 2018 Deng
et al.; 2021; Yin et al., 2021: Bai et al. 2022; Yang et al. 2018 Fan et al. 2021 voxelize the 3D
meth- ods (Shi et al. 2020a (02T) utilize both embeddings from 1) and 2). For camera-based 3D percep- tion. DDng et al. [2020; Chen et al., [016)] Simonelli et al., 2019 WWan et al.) 2021b] Zhang et al.) 2019) embed 2D features on image plane and project these 2D features into 3D space with estimated depth. Recently, research starts to focus on fusion 3D perception (Chen et al.,
scene and use sparse convolution or transformer for embedding. 3) foint-voxel-combiie
Chen
2021
Ku et al.
2017
Liu et al.] [023] Liang et al. [022] Li al. 20226 Liang et al. 2019 Li al. [022a] Meyer 2019) with camera images and LiDAR point clouds as inputs. These methods focus on inte-
et al.
grating embeddings of different modalities and train in an supervised manner. There are other works on fusion-based semantic segmentation (Wu et al. (024) and object tracking (L.iu et al.) 2024). As labeling in 3D space is costly, we explore unsupervised 3D pre-training for fusion 3D perception.
3D Pre-training Annotating for 3D data is notoriously time- and energy- and the emer- gence of unsupervised representation learning for 2D image (He et al. 2020 Tian et al. 2019]
Caron al.
2020; Grill al. 2020 Wang et al. 2021c He et al. (2022) provides promising way Liu et al.
to alleviate the annotation burden. Existing works (et a 2020
2020; Hou et al., [021]
Chen et al. 2022 Huang al. 2021 Liang et al.; 202 Liu et al. [2211a] (Sautier et al. 2022 Pang
iet al. [023 Yang et al.) 2023 et al 2023 Huang et al. 2023 Zhu et al. 2023) in unsupervised 3D
representation learning into scene-level 3D point clouds can be divided into two contrastive-basee
and masked-and wwwpoddrrooootttton-bbaaaee
paradigms. Contrastive-base works (et al 2020; Liu et al.
2020 HHu et al. 202 Chen et al. 2022 Huang et 202T Liang al., 2021] Pang et al. 2023
Liu et al. 202 propose various ways to build suitable views and conduct contrastive learning to
improve the performance in downstream perception task. Inspired by (He et al., 2022) in image
domain, (Yang et al. 2023] et all [023 Huang et al., [0233 Zhu et al. (023) propose to first mask the input point clouds and pre-train the 3D encoders with a shallow decoder for reconstructing the unmasked inputs. (Sun et al., 2023; Zhang et al. 2024 Li et al. (202cc) are pioneering works to in- troduce contrastive learning into fusion perception. They consider camera and LiDAR embeddings as different views of the scene and apply the contrastive loss between the two modalities. Inspired by the success of neural field in representing 3D scenes (Mildenhall et al., 2021 Wang et al.
2021a) and previous attempts to introduce neural rendering to 3D pre-training for point clouds Yann al. 2023
Huang et al.) 2023 Zhu et al. 2023), UniPAD (Yang et al.. 2024) proposes to use differentiable- rendering decoder for
pre-training and achieves SOTA performance for unsupervised 3D representation learning on fusion 3D perception. However, due to the high GPU memory consumption, UniPAD (Yang et al. (024) is only able to separately pre-train the image and point cloud encoders and fails to utilize the interaction between modalities during pre-training. In this paper, we explore unsupervised joint pre-training for 2D and 3D backbones via differentiable rendering with Curvature Sampling and Prototype Learning. Previous work like SWAV (Caron et al.) 2020) is related to our prototype learning part. SWAV (Caron et al. 2020) explores learnable proto- types to represent different categories in the same modality (image) and uses swapping assignment prediction loss for different views of the same image instance. On the contrary. learnable prototypes in CLAP are used to represent part of the 3D scenes and learn the interaction between modalities, which differs from the context in (Caron et al. 2020). Thus we propose Exxpecation-Maxiiizzae training scheme to maximize similarity between prototypes and 3D embeddings. Then swapping prediction loss is used to learn the modality interaction. Furthermore, to avoid prototypes collapsing to the same vector, we propose a Gram Matrix Regularization loss.
3 METHOD
In this section, we introduce CLAP for joint unsupervised 3D pre-training via differentiable ren- dering on fusion 3D perception. As described in Fig. 2 CLAP pre-trains the image, LiDAR and fusion encoders jointly with Neural Field Rendering. In order to enable joint pre-training, Curvature Sampling is proposed as shown in (a) of Fig. 2 To further make use of both modalities, we utilize learnable prototypes to represent parts of the 3D scenes shown in Fig. 2 (b). To optimize the learnable prototype, we train a common feature space with an Expectation Manittiion-ximzzaattionn training scheme and incorporate interaction between modalities by swapping prediction loss. Finally Gram Matrix Regularization loss is proposed to avoid collapse in prototype learning. We first dis- cuss the formulation and overall pipeline in Section 3.1. Then we introduce the details about neural
3

---

Published as a conference paper at ICLR 2026
[M] Curvature Sampling ey
UDAR Feotures P
Clouda:
Concec
Fusion Features F
Fradas
Protonype Learning
Coeeration
Image Faaturees ]
Lidkiig ancil
Dertpcing www.iigg.iiiioooooo bart wan Natiiii
Surounding Carmana tor lggacc
The pipeline of CLAP. In order to jointly pre-train the LiDAR, camera and fusion en- coders, we first embed the paired LiDAR point clouds and camera images with f f and fision Then based on the fusion features, CLAP applies differentiable rendering to predict both range and rgb with the SDF and RGB values of the sampled points along LiDAR/camera rays from fSDF and fRGB, with which compute loss against the observed LiDAR point cloud and camera images. To make joint pre-training feasible, we propose Curvature Sampling to sample informative parts of the 3D scene, as descriped in (a). Furthermore, we propose to use learnable prototypes represent parts of objects in common feature space and utilize an Expectation-Maxins Maximizztiinnnn approach to maxi- mize the similarity between prototypes and 3D embeddings of each modality. To delve deeper into the interplay of image semantics and LiDAR geometry, we use swapping prototype prediction loss. Finally, we propose a Gram Matrix Regularization loss to prevent collapse of prototype learning.
Figure 2:
field and differentiable rendering in Section
3.2
Finally, we describe the curvature sampling and
prototype learning separately in Section 3.3 and 3.4
3.1 FORMULATION AND PIPELINE
Notations. To begin with, we denote the input image set from Neam cameras as {I, E Nan x Nan and LiDAR point cloud as P E R, (3++d) H and W are the height and width of the images and each pixel the images has values for RGB. Np is the number of points in the LiDAR point cloud and each of them contains ryyclloattion and d feature channels. For example, in NuScenes (Caesar et al., 2020) dataset, d 2 represents the intensity and timestamp of each point and there are Nem = 6 surrounding cameras on the autonomous vehicle. For each pair of camera image and LiDAR point cloud, we have the transformation matrix Tn E R3x4 indicating the projection between the camera plane and LiDAR coordinate, where n = 1,2, Neam.
Encoding, The goal of unsupervised 3D representation learning for fusion perception is to pre-train the LiDAR, camera and fusion encoder in an unsupervised manner. Hence, we first voxelize and embed the raw LiDAR point cloud P with LiDAR encoder rene
(P),
where P E @xx is the embedded 3D features for LiDAR point cloud. D, á and W are spatial resolutions of the embedded features and dp is number of feature channels after encoding. Then for camera images I, gene encodes them with swin transformer (Liu et al. (021b) and uses T = (T,) Nan to project the 2D features to 3D space.
i= - (I,T.P):
where E xDx is the embedded 3D features for surrounding camera images with the similar dimensions as P except for di feature channels. The projection of 2D features to 3D space similar to (Liu et al. 2023): we transform LiDAR points back to image planes with T and use the projected ranges from LiDAR to project the 2D features to 3D space. With P and I, we further concatenate them along feature dimension and apply the fusion encoder frne get the fusion
4

---

Published as a conference paper at ICLR 2026
feature F R fx xxxxxxx with de feature dimensions,
F fusion (P,I))
(3)
Loss Function. To guide fp" si and fusion to learn good representations in an unsupervised manner, CLAP first embed the fusion features F with a shallow 3D convolution network fau get = (F) and we have F E Rxx Then a rendering loss Lend is applied on F for rrssec-rroo
both point clouds P and images I. Furthermore, prototype learning scheme Lposto is utilized in order to bridge the two modalities and incorporate interaction of image semantics and LiDAR geometry into pre-training. The overall loss function is below:
L W/ X Leod (P, F. + Wproco X Cpono (P,
(4)
with wi and wproto as weighting parameters to balance the losses.
3.2 NEURAL FIELD AND DIFFERENTIABLE RENDERING
Inspired by the success of UniPAD Yang et al. 2024). CLAP applies differentiable rendering de- coder with neural field to conduct the -nrr--andrrroostuuction pre-training. Different from (Yang et al. 2024), additional surface signed distance field loss is applied to better optimize the scene geometry. Here we first introduce Neural Field, which is the basis for camera images and point clouds rendering, and then discuss the differentiable rendering process range and RGB values.
Neural Field. Given a specific point p [x, y, a] ER the 3D space, the feature fp E Rdr at is queried from the fusion 3D embedding F by trilinear interpolation
f (", F),
(5)
where f is an built-in module implemented in Pytorch (Paszke et al. 2019). Taking the concatena- tion of location p and queried feature fp inputs, we predict the signed distance value R (Chan
& Zhu 2005; Malladi et al. 1995) and color value R3 (Wang et al. 202 at p with fsDr and RGB and fSDD are parameterized by Multi-layer Perceptron.
FRGB
s = } (SD,((p.)))
c= c=fRGB [RGB((P
(6)
Differentiable Rendering. Similar to (Mildenhall et al. 2021 Wang et al. 2021a), we first sample
Ni. or Ne rays at the LiDAR or camera sensor origin o, each of which is described by its normalized direction d and o. Next, we sample Nuy points following (Wang et al. 202 la along each ray. Here each point along the ray can be interpreted by p o rd, where r is the range from the sensor Nay origin to the point p. Thus the sampled point set can be annotated by {p.. = o + +rrrd r,,d) n=1 and we predict the estimated signed distance value 8, and color value ea for them with fRGB and fSDF; Following (Wang et al. 2021a). we estimate the occupancy value a, for each sampled point,
(sn) - (2((n)1) .0.
On max
(s,)
Here (z) = (1+ee e stands for the sigmoid function paired with learnable scalar h. After that, we predict the accumulated transmittance tn similar to (Wang et al. 202
II 1=1
az).
Based on l, compute unbiased and occlusion-aware weight wn, k, (Wagg et al. 202 and integrate all samples along the ray to predict the range i or color along this ray,
N..
N.
Σ
Wn
Tn
wn
C..
For the observed LiDAR points, it evident that signed distance value should be 0. The loss function for differentiable rendering is combination of L--1 loss surface SDF, range and color predictions.
Nc. 3
Lrend
1 Nr.
Σ (Ir Ful waun (sa)) + 3 No
We
ΣΣ
(10)
1=1
5

---

Published as a conference paper at ICLR 2026
where waur and we weighting parameters for the losses. T; is the observed range along the ith sampled ray and 8; is the predicted signed distance value at the observed points. cf is the value of 5th channel in the image pixel, where (1, corresponds to RGB channels.
3.3 CURVATURE SAMPLING
In order to make joint unsupervised representation learning feasible, we have to make Nj. < Np and Nc < H W Neam- Intuitively, uniform sampling with range can be used, same as "Memory-friendll Ray Sampling" in (Yang et al. 2024). But due to the relatively small sample number compared to the raw inputs (~ __ ). this sampling method brings little improvement against separate pre-training, which is contradictory to our motivation. Hence, need sample more infor- mative part of the scene for Lrend- As shown in Figure D we are inspired by the observation that surface with higher curvature (surface of vehicle) generally contains more information as compared to that with lower curvature (road plane). There- fore, we propose the Curvature Sampling for effec- tive sampling. In order to make it more intuitive, we show camera image here and the curvature estima-
T0O
Flat Ground (lower curvature)
Vehic Surface (higher survature)
Figure 3: Inspiration of Curvature Sampling. tion is actually computed in 3D with the SDF function. For each point in the LiDAR point cloud P. we first estimate surface normal by deriving the signed distance function with respect to
8f SDD ([p. óp
(11)
where n E R is the predicted normal. Then we normalized n to get the direction of the normal I Here || |l2 is the L-2 norm. Next we apply differential operator on ń with respect to p
(12)
óp
and get e E R3. Then we estimate the geodesic curvature (ooonogov 2006) of each point pa in LiDAR point cloud by computing the norm of C, and use it the sampling weights Wa, that is w. = c, 2. With Was we sample Nr. points with Multinomial Sampler implemented in PyTorch (Paszke al. 2019) for differentiable rendering. For pixels on image plane, we project the LiDAR point cloud back to image planes with T, assign of each point to the projected pixel and apply a gaussian blur kernel of size Kgaus densify the weights, with which we sample No pixels for finnd- As the curvature estimation is noisy especially in the first few epochs, we apply uniform sampling to warm-up for Nwammup epochs and after that Curvature Sampling is utilized. We implement curvature estimation within torch.no -grd(() context, so that it is computed only once and not stored. Thus the computational and GPU memory overhead less than 1% and negligible.
3.4 PROTOTYPE LEARNING
Curvature Sampling allows for joint pre-training, prompting us to explore how camera and LiDAR data can be used understand "objectness"" or "object parts" in an unsupervised way. We use learnable prototypes to represent segments of 3D scenes and establish a shared feature space that connects the two modalities. Firstly, we randomly initialize NR learnable prototypes each of which is dk vectors. Then an Expectation Maximiiziiaai training approach is proposed to maximize the similarity between these prototypes and 3D embeddings from the two modalities. To delve deeper into the interplay between image semanties and LiDAR geometry, use the swap- ping prototype prediction loss. Finally, to avoid the prototype collapsing into one same vector, we introduce a Gram Matrix Regularization Loss.
E NR
ponnctationnMaxiiizztte In order to guide the learnable prototypes to represent parts of the environment, we propose Exppecaation-Mxxiizzaa (Moon) 1996) training scheme to optimize

---

Published as a conference paper at ICLR 2026
the prototypes. We first project the LiDAR embeddings P and camera embeddings separately with two projection heads f and f! to the same dimension K
1o P= Jt i- 1T
(13)
where fp and fl are parameterized by Multi-Layer Perceptron and P xxxHxwx4 i e Dx xxxxxx We denote NaD D x H x W and then normalize and reshape the projected
embeddings into P E xdx and E IR xdx, After that, similarity scores Sp E R between 3D embeddings and prototypes are computed separately for LiDAR and camera branches
Sp P. S, -
(14)
In the Expectation step, we compute the probability Spa that each prototype is assigned to each embeddings by applying softmax operation Spa. Then for Maximization step, expect to maximize the probability of the assignment between one prototype to one specific part of the scene and this is equal to minimize the entropy of the similarity matrix. Thus, the EM loss is computed as
LEM
SSp
Sp
S.. logSS S.
(15)
NR
Swapping Prototype Prediction. To further explore interaction between modalities, we detach Spi and apply sinkhorn algorithm (Cuturi) 2013) approximate them to double stochastic matrix in Nsink iterations. We denote the updated matrix as codes Qpı E IR x Nx. The swapping prototype prediction loss is computed with a temperature parameter т, inspired by (Caron al. 2020).
Nap
Nk
exp(Sp (R.. 3/T Ek=1 exp (Sp 3/T
LSWAV =
Q
log
NAD
1=
(16)
e.p( ST m 1/T }.
+ Qp n.m log
Σ Ni
exp (S Sm
Gram Matrix Minimization. When training the randomly initialized prototypes, the network might learn a short cut with all prototypes being the same (Caron et al. 2020), which called collapse. To avoid this, estimate similarity between prototypes by the gram matrix G KKT of prototypes K, the dimension which G RR. Finally we minimize the average the non-diagonal elements of G in order to avoid collapse
LGMM
G
(17)
NK 1)
Overall Prototype Learning Loss. We apply weighting parameters WSwAV, WEM and WGMM bal- ance the three losses proposed above, which leads to the overall loss function for prototype learning,
Epmats SS + WEM LEM WCMM
(18)
EXPERIMENTS
Unsupervised 3D representation learning for fusion perception aims to pre-train both LiDAR and camera encoders and initialize downstream models with the pre-trained weights to gain performance improvement in downstream tasks. In this section, we design extensive experiments lar autonomous driving dataset NuScenes (Caesar et al. 2020) and Waymo (Sun et al. 2020 to demonstrate the effectiveness of CLAP. To begin with, describe experiment setups Section 4.1 Next, we show and analyze main results in Section 4.2 Finally, we provide ablation study and visualizations separately in Section 4.3 and 4.5
the popu-

---

Published as a conference paper at ICLR 2026
Table 1: Results for fine-tuning on 5% of training set in NuScenes. "Init." means the way initial- ize models. We guarantee convergence of from- scr-sch model and fix the training iteration for all fine-tuning experiments. We provide mAP and NDS as an evaluation of the overall performance of different models and highlight the best mAP and NDS with bold font. We also indicate the perfor- mance improvement by green color. VC. "Mot.", "Bic.", "Ped." and "T.C." are abbreviations for Construction Vehicle, Motorcycle, Bicycle, Pedestrian and Traffic Cone. Results are in %.
Init.
I8AP
NDS
Car
Trick
C.V.
Bus
Trailer
Berrier
Mot.
Bic
Fod.
T.C.
Read.
55.28
78.52
16.18
50.44
22.11
57.00
46.87
30.56
76.
62.40
ALSO
45.34
-3.35
49.19
77.61
41.98
14.61
36.09
12.55
58.10
42.89
30.56
74.09
64.98
OCC-MAE
47.30
-1.30
54.97
-0.31
77.43
46.25
15.36
50.6
19.35
55.35
43.24
30.92
74.60
61.34
SLidR
1:40
52.77
2.61
77.12
16.19
50.50
22.17
57.74
41.47
30.22
71.31
60.71
PPKT
19.58
+00.80
55.85
79.24
47.26
17.26
51.37
21,14
59.55
44.82
31.43
78.03
65.73
UsiPAD
49.81
+:1.22
55.20
-0.03
50.81
42.81
17.08
48.08
25.85
61.72
50.19
27.53
78:53
64.57
CLAP
12:48
1.70
79.56
18.84
56.34
23,98
60.60
48.87
34.11
78.08
62.87
4.1 SETTINGS
Datasets. We use the popular autonomous driving dataset NuScenes (Caesar et al. 2020) and Waymo (Sun et al. 2020) to evaluate the performance of CLAP. NuScenes (Caesar al.) 2020) uses one roof LiDAR and six surrounding cameras to collect data. The LiDAR is 32-beam Velo- dyne and collecting frequency 20Hz. The frequency of camera capturing is 12Hz. (Caesar al.) 2020) conducts the synchronization and provides paired data of LiDAR point cloud and camera im- ages. The whole NuScenes dataset contains 1000 scenes collected in Boston and Singapore. Each scene lasts for around 20 seconds and there are a total of 5.5 hours data. Following convention prac- tice from (Caesar et al., 2020, Team 2020), we divide the whole dataset into training set with 850 scenes and validation set with 150 scenes. Waymo (Sun et al., 2020) uses one top 64-beam LiDAR, corner LiDARs and 5 surrounding cameras to collect point clouds and camera images. Following the same practice in (Sun et al., 2020 Team 2020), the collected 1000 scenes split into training set (798 scenes) and validation set (202 scenes). All pre-trainings are conducted on the training set without labels and we conduct downstream 3D object detection training in few-shot setting via uniform sampling of the training data (NuScenes 5% and Waymo 1%).
Downstream 3D Object Detectors. For NuScenes, select the SOTA 3D object detector called BEVFusion (Liu et al. 2023) and for Waymo, we use CenterPoint (Yin et al. (021). Both BEVFu- sion and CenterPoint are implemented in the popular code repository for autonomous driving per- ception called OpenPCDet [Team] 2020). For evaluation metrics, we use average precisions of vari- ous categories (APs), mean average precision (mAP) and NuScenes Detection Score (NDS) (Caesar et al., 2020) for NuScenes and mAP (mean accurate precisions) and mAPH (mean accurate pre- cisions with headings) at different difficulty levels (Level-1 and 2) for Waymo (Sun et al.) 2020). We follow similar setting in (He et al. 2019: et all 2023) to gradually increase training iterations of the from-scratch model until convergence is observed. Here convergence means further increas- ing training iterations will not improve the performance. Then the number of training iterations is fixed for fine-tuning pre- pre-trained models. This setting avoids the case that pre-training only acceler- ates convergence and makes sure that pre-training indeed improve the performance of downstream models, that is improving the sample efficiency of the downstream task.
Baseline Pre-training Method for Fusion Perception. We incorporate three kinds of pre-training
baseline methods: 1) an occupancy estimation method called ALSO (Boulch et al. 2023). 2) oceu- pancy masked autoencoder called Occupancy-MAF: (Min al., 2023). 3) multi-modality methods including SL.d (Sautier et al.] 2022). PPKT (Liu et al. 2021a) and UniPAD (Yang et al.) 2024)
We use the official implementations to pre-train the backbones with same setting as CLAP.
Implementation Details of CLAP. The feature channels for embeddings P., i, F and prototypes K are respectively set to dp 256, d = 80, dr = 512 and dg = 128. Sampling number for point cloud and pixel are N₁ 8192 and Ne = 1024 x Neam The number of sample points along each ray is NNny = 96. Warm-up epochs for Curvature Sampling is Nwarmmp = 4. We set the number of learnable prototypes and sinkhorn update iterations to Nx 512 and N 3. The temperature for swapping prediction loss is т = 1.0. The loss weighting parameters are implemented as wi = 2.0, Wipreto 1.0, Wsur 0.05, We 0.05, WswAv 1.0, WEM 0.1 and WOMM 0.1. We use torch.auto grad() (Paszke et al. 2019) to implement the derivatives in the curvature estimation. More details about pre-training and inn-tuning are provided in Appendix
8

---

Published as a conference paper at ICLR 2026
Table Results for Waymo Sun al. p020
Table 3: Results on scaling property.
Level-1
Level-2
FT.
NDS
Init.
mAP
mAPII
mAP
mAPII
48.69
55.28
5%
Rand.
61.60
58.58
55.62
52.87
CLAP
51.17
57.04
AL
62.09
59.03
56.12
53.32
+-0.47
Random
39.12
40.01
2.5%
OCC-MAE
62.33
59.32
56.36
53.63
+0.74
CLAP
42.86
42.18
SLidR
62.10
59.09
56,10
53,36
+-0.49
Random
26.22
29.82
PPKT
62.32
59.22
56.37
53.55
+-0.69
CLAP
30.53
31,87
UniPAD
61.57
58.64
55.64
52,93
+-0.02
Random
0.5%
16.49
22.61
CLAP
62.87
59.88
56.88
54.16
+1.28
CLAP
23.71
27.32
Weight
High Weight
Table 4: Ablation Study. The second line joint pre-training with sampling method from UniPAD.
Joint Pre train
Cur. Sam.
Proto, Leuring
mAP
x
X
49.1
/
/
x
5081
/
/
/
51.17
Figure 4: Visualization of curvature estimation.
4.2 MAIN RESULTS
NuScenes Results. As shown in Table CLAP achieves 2.48% mAP improvement over randomly initialization at convergence, which is 100% more improvement for mAP than SOTA unsupervised 3D representation method UniPAD Yang et al. 2024) and the best among all initialization methods. For NDS metric, UniPAD (Yang al. 2024) only achieves comparable performance and PPKT (Liul et al.
2021a) makes a gain of 0.57% while CLAP surpasses the train-from-scrate model by 1.76% When turns to different categories, CLAP generally benefit the performance of all the categories and for Construction Vehicle, Bus, Barrier, Motorcycle and Bicycle, the improvement over random initialization are more than 2% AP.
Waymo Results. In Table 2 we provide the average performance difference of mAP and mAPH over different difficulty levels. can be found that CLAP achieves the best performance at conver- gence. Meanwhile, the performance gain brought by CLAP is approximately two times as the best (OCC-MAR (Min et al., (003)) of previous pre-training methods. This demonstrates the effective- ness and generalization ability of CLAP.
Potential Scaling Property. As we are not able to scale up the pre-training dataset at current stage, we explore potential scaling property by gradually decreasing the sample numbers (2.5%, 1% and 0.5%) for fine-tuning on NuScenes, which increases the ratio between pre-training data and fine- tuning data. The results shown in Table (F.T. is fine-tuning! It can be found that the ratio between pre- pre-training data and fine-tuning data gets larger, the performance improvement by CLAP increases and CLAP provides a gain up to 7.22% mAP and 4.71% NDS with 0.5% fine-tuning data. These results show that CLAP promising scaling property and in the future, we can scale up the pre-training dataset, CLAP might further improve current SOTA performance.
4.3 VISUALIZATIONS CURVATURE ESTIMATION
We use CLAP to estimate the curvature of point clouds and use heatmap color to indicate the weight computed in Section 3.3 As shown in Figure 4 (orange boxes for those regions with relatively correct estimation and green ones for those with noisy estimation), can be found that though some noise exists (because pre-training is conducted in unsupervised manner), CLAP is able to predict high weights for those highly informative region for sampling and meanwhile assign lower weights most of the background, which makes joint pre-training feasible.

---

Published as a conference paper at ICLR 2026
(a)
(b)
Figure 5: Visualization of the prototype learning results. Different color indicates different prototype
assignments.
4.4 VISUALIZATIONS ON PROTOTYPE LEARNING
We the model pre-trained by CLAP infer the 3D features and assign prototypes different LiDAR points in the 3D space. Then we use different random colors to indicate different prototypes and visualize them, shown in Figure 5 If can be found that the background road plane inside the same frame is generally assigned to the same prototype. And foreground vehicles are assigned to another prototype. This demonstrates that the proposed prototype learning scheme actually learns to represent parts of the scenes with prototypes in an unsupervised manner. However, our pre- training does not incorporate any label, can also be found that the prototype assignment has some noise, for example some of the road plane points are assigned to other prototypes.
4.5 ABLATION STUDY
We conduct ablation study to evaluate the effectiveness of different components. Results are in Table 4 The first line is seperate pre-training with UniPAD (Yang al., 2024). The second one is jointly pre-training using UniPAD (Yang et al. 2024) and uniform sampling guided by range ("Memory-friendi) Ray Sampling" in (Yang al. 2024)) to address the GPU memory limitation.
It can be found that using simple sampling method does not bring improvement over separate pre- training. Then add Curvature Sampling in third line and found that Curvature Sampling enables more effective sampling and improves the performance over uniform sampling guided by range and separate pre-trainin pre-training Finally, the Prototype Learning scheme (fourth line) learns a common feature space for segments of 3D scenes and introduces interaction of LiDAR and camera encoders, which achieves the best performance.
CONCLUSION
In this paper, we propose CLAP unsupervised fusion joint pre-training via differentiable render- ing. CLAP uses Curvature Sampling sample more informative parts and learnable prototypes to approach. To explore
represent parts of 3D scenes, optimized with an Dnrectation-Maxiiiaalll the interplay between LiDAR geometry and image semantics, swapping prediction loss is used with a regularization loss to avoid collapse. Experiment results demonstrate that CLAP is superior in unsupervised 3D representation learning and has the potential to scale up.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This paper is supported by the General Research Fund of Hong Kong No. No.1720990 17209324 and 17200622, and the Global Industrial Technology Cooperation Center (GITCC) through grant agreement with the Korea Institute for Advancement of Technology (KIAT), project number PO028922.
ETHICS STATEMENT
We further discuss potential negative social impact of CLAP in this section.
Job Displacement. Automation of tasks that require 3D perception, such as autonomous vehicles or robotics, might lead to job losses in sectors like transportation, warehousing, and manufacturing. While automation can create new jobs, the transition period can be challenging for those whose jobs are automated away.
Security Risks. The deployment of autonomous systems could introduce new vulnerabilities. Any failures in autonomous driving or robotics could lead to accidents or be exploited maliciously.
Accessibility and Inequality The benefits of advanced 3D perception technologies might not be evenly distributed across society. Wealthier regions or organizations may have earlier and better access to these technologies, potentially widening the gap between different socioeconomic groups.
REPRODUCIBILITY STATEMENT
We provide method details in Section and implementation details in Appendix A which are enough for reproducing the experiments. Besides, will release the code and models.
REFERENCES
Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. P Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1090-1099, 2022.
Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, J. Gall, and Stachniss. Towards 3D LiDAR-based semantic scene understanding of 3D point cloud sequences: The SemanticKITTI Dataset. The International Journal on Robotics Research, (0(8-9)9599999)) 967, 2021. doi: 10.1177/
0278364921 1006735.
Alexandre Boulch al. ALSO: Automotive lidar self-suppriision by occupancy estimation. In CVPR, 2023.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for au- tonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1621-11631, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9912-9924. Curran Asso- neurips wwt.riis.c.ppaae. httpp//.c/./pppe/////////ee ciates, Inc., 2020. URL http:: httttp:::pppoeeeaiiiiito Paper. Papeee.ppppppppp. TOFE6S226e3116002888774422-
Tony Chan and Wei Zhu. Level set based shape prior segmentation. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'OH '05), volume 2, pp. 1164-1170. IEEE, 2005.
Runjian Chen, Yao Mu, Runsen Xu, Wenqi Shao, Chenhan Jiang, Hang Xu, Zhenguo Li, and Ping Luo. Co" 3: Cooperative unsupervised 3d representation learning for autonomous driving. arXiv preprint arXiv:2206.04400 2022.
11

---

Published as a conference paper at ICLR 2026
Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monoc- ular 3d object detection for autonomous driving. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2147-2156, 2016.
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-viww 3d object detection network for autonomous driving. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 1907-1915, 2017.
MMDetection33 Contributors. MMDetecttion3D.Cm OpenMMaab next-generation platform for gen- eral 3D object detection. https://githu......./////// ctm optp://penn-eeeeee -mppennmmmabbbm. httplaaannnneeeetinnn33 2020.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Jiajun Deng. Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cmn: Towards high performance voxel-basee 3d object detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 1201-1209, 2021.
Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang. Jianping Shi, Zhiwu Lu, and Ping Luo. Learning depth-guided convolutions for monocular 3d object detection. In Proceedings the IEEE/CVF Conference on computer vision and pattern recognition workshops, pp. 1000-1001, 2020.
Xie Saining et al. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In ECCV, pp. 574-591, 2020.
Xu Runsen et al. Mv-jar: Masked voxel jigsaw and reconstruction for lidar-based self-supervised 2023.
pre-training. In CVPR, pp.
Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. arXiv preprint
2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21271-21288 Curran Associates, Inc., 2020. URL bttps: /ttpp:: /ttp://eee//oceeeeeeeeeee www.innn......nnnnneeee.......ee cc/ /aaa:e/ee22////eee/aaaaadddoooo 20leeceeeeee01111111 7b81 92 www.oaseebbbeeeeeee Papiee--apppppppp..
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4918 4927, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 16000~16009, 2022.
Ji Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 15587-15597, 2021.
Di Huang, Sida Peng, Tong He, Honghui Yang, Xiaowei Zhou, and Wanli Ouyang. Ponder: Point cloud pre- treitrain via neural rendering. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 16089-16098, 2023.
Siyuan Huang, Yichen Xie, Song- Chng Zhu, and Yixin Zhu. Spatio-temporal self-supervised repre- sentation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6535-65455 2021.
12

---

Published as a conference paper at ICLR 2026
Jason Ku, Alex D Pon, and Steven L Waslander. Monocular 3d object detection leveraging accurate proposals and shape reconstruction. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 1 11867-11876. 2019.
Yanwei Li, Xiaojuan Qi, Yukang Chen, Liwei Wang. Zeming Li, Jian Sun, and Jiaya Jia. Voxel field fusion for 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattern Recognition, pp. 1120-1129, 2022a.
Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu, Denny Zhou, Quoe Le, et Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 17 17182-17191, 2022b.
Zhenyu Li, Zehui Chen, Ang Li, Liangji Fang, Qinhong Jiang, Xianming Liu, Junjun Jiang, Bolei Zhou, and Hang Zhao, Simipu: Simple 2d image and 3d point cloud unsupervised pre-training for spatial-aware visual representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 1500-1508, 2022c.
Hanxue Liang, Chenhan Jiang, Dapeng Feng, Xin Chen, Hang Xu, Xiaodan Liang, Wei Zhang, Zhenguo Li, and Luc Van Gool. Exploring geometry-aware contrast and clustering harmonization for self suuerviueri 3d object detection. In Proceedings of the IEEE/CVF International Conference 2021.
on Computer Vision, pp.
Ming Liang, Bin Yang. Yun Chen, Rui Hu, and Raquel Urtasun. Multi-task multi-sensor fusion for 3d object detection. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7345-7353, 2019.
Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang. Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Systems, 35:10421-10434, 2022.
Jiaming Liu, Yue Wu, Maoguo Gong, Qiguang Miao, Wenping Ma, Cai Xu, and Can Qin. M3sot: Multi-frame, multi-field, multi-space 3d single object tracking. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 38, pp. 3630-3638, 2024.
Yueh-Cheng Liu Learning from 2d: Contrastive pixel-to poeett knowledge transfer for 3d 2021a.
pretraining, arXiv preprint
Yunze Liu, Li Yi, Shanghang Zhang, Qingnan Fan, Thomas Funkhouser, and Hao Dong, P4contrast: Contrastive learning with pairs of point-pixel pairs for rgb-d scene understanding. arXiv preprint arXiv:2012.13089, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021b.
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. In 2023 IEEE international conference on robotics and automation (ICRA), pp. 2774-2781. IEEE, 2023.
Ravi Malladi, James A Sethian, and Baba C Vemuri. Shape modeling with front propagation: A level set approach. IEEE transactions on paltern analysis and machine intelligence, 17(2):158-775 1995.
Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, al. One million scenes for autonomous driving: Once dataset. arXiv preprint arXiv: 106. 1037, 2021.
Gregory P Meyer, Jake Charland, Darshan Hegde, Ankit Laddha, and Carlos Vallespi-Gonzalez Sensor fusion for joint 3d object detection and semantic segmentation. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition workshops, pp. 0-0, 2019.
13

---

Published as a conference paper at ICLR 2026
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Chen Min et al. Occupancy-mas Self-supervised pre-training large-scale lidar point clouds with masked occupancy autoencoders. IEEE TIV, 2023.
Todd K Moon. The expectationn-maxiinzaaii algorithm. IEEE Signal processing magazine, 13(6):
47 47-60, 1996.
Bo Pang. Hongchi Xia, and Cewu Lu. Unsupervised 3d point cloud representation learning by tri- angle constrained contrast for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5229-5229233 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Corentin Sautier et al. Image-to-lidar self-supervised distillation for autonomous driving data. In CVPR, 2022.
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng
Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. (0529-10538, 2020a.
Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts:
3d object detection from point cloud with part-aware and part-aggregation network. IEEE trans- actions pattern analysis and machine intelligence, (3(8):26472666) 2020b.
Shaoshuai Shi, Li Jiang, Jiajun Deng. Zhe Wang. Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Py-rcnn++: Point-voxel feature set abstraction with local vector representation for 3d object detection. arXiv preprint
2021.
Andrea Simonelli, Samuel Rota Bulo, Lorenzo Porzi, Manuel López-Antequera, and Peter Kontschieder. Disentangling monocular 3d object detection. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1991-1999, 2019.
Jiachen Sun, Haizhong Zheng, Qingzhao Zhang, Atul Prakash, Z Morley Mao, and Chaowei Xiao. Calico: Self-supervised camera-lida contrastive pre-training for bev perception. arXiv preprint arXiv:2306.0003: 2023.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 2464,,44, 2020.
OpenPCDet Development Team. Openpedet: An open-source toolbox for 3d object detection from cwm/ open mmmlab/ppenni
point clouds. prEtps://gitiee
2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, arXiv preprint
(xXiv:990.0055000 2019.
Victor Toponogov. Differential geometry curves and surfaces. Springer, 2006.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2 10689, 2021a.
Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Feos3d: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 913-922, 2021b.
14

---

Published as a conference paper at ICLR 2026
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2021c.
Yue Wu, Jiaming Liu, Maoguo Gong, Qiguang Miao, Wenping Ma, and Cai Xu. Joint semantic segmentation using representations of lidar point clouds and camera images. Information Fusion, 108:102370, 2024.
Siming Yan, Zhenpei Yang, Haoxiang Li, Chen Song, Li Guan, Hao Kang, Gang Hua, and Qixing Huang. Implicit autoencoder for point-cloud self-supervised representation learning. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 14530-14542, 2023.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
Bin Yang. Wenjie Luo, and Raquel Urtasun. Pixor: Real-time object detection from point clouds. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7652- 7660, 2018
Honghui Yang, Tong He, Jiaheng Liu, Hua Chen, Boxi Wu, Binbin Lin, Xiaofei He, and Wanli Ouyang. Gd-mae: generative decoder for mae pre-training on lidar point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9403-9414, 2023.
Honghui Yang, Sha Zhang, Di Huang, Xiaoyang Wu, Haoyi Zhu, Tong He, Shixiang Tang, Heng- shuang Zhao, Qibo Qiu, Binbin Lin, al. Unipad: A universal pre-training paradigm for tonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15238-15250, 2024.
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking- In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp- 11784-11793, 2021.
Yifan Zhang, Siyu Ren, Junhui Hou, Jinjian Wu, Yixuan Yuan, and Guangming Shi. Self Self-super learning of lidar 3d point clouds via 2d-3d neural calibration. arXiv preprint arXiv:2401.12455 2024.
Yunpeng Zhang, Jiwen Lu, and Jie Zhou. Objects are different: Flexible monocular 3d object detec- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3289-32999 2021.
Hui Zhou, Xinge Zhu, Xiao Song, Yuexin Ma, Zhe Wang, Hongsheng Li, and Dahua Lin. Cylin- der3d: An effective 3d framework for driving- scene lidar semantic segmentation. arXiv preprint arXiv:2008.01550, 2020.
Haoyi Zhu, Honghui Yang, Xiaoyang Wu, Di Huang, Sha Zhang, Xianglong He, Tong He, Heng- shuang Zhao, Chunhua Shen, Yu Qiao, et al. Ponderv2: Pave the way for 3d foundataion model with a universal pre-training paradigm. arXiv preprint arXiv:2310008586 2023.
15

---

Published as a conference paper at ICLR 2026
A MORE IMPLEMENTATION DETAILS
Pre-training. We use learning rate of 0.00005 with cosine learning schedule for pre- pre-training and use mask augmentation for CLAP with a masking rate of 0.9. All the pre-trainings are conducted on 8-H100 clusters for similar time (~ 45 mins epoch). As pre-trained image backbones used for image feature extraction, the implementation of Yang et al. (2024) also use pre-trained image backbone from Contributors (2020) initiate their pre-training and the training set for this
broadly
backbone does not involve any data from NuScenes Caesar et al. (2020) and Waymo Sun et al.
(2220). We adopt this practice for training-from-scrate model and CLAP.
Downstream Training. We follow the common practice in OpenPCDet Team 2020) and only
change the training iterations in order observe the convergence of train-from-seee scrasch models, which avoids the case that pre-training only accelerate convergence and make sure that pre-training indeed improve the performance of downstream models. The training epoch 108 for BEVfusion
Liu et al. (2023) with 5% of NuScenes Caesar al. (2020) training data and 252 for CenterPoint
Yin et al. 2021 with 1% of Waymo Sun et 2020 training data.
B
MORE EXPERIMENT RESULTS
B. SEMANTIC SEGMENTATION
We fine-tune Cylinder3D Zhou et al. (2020) for LiDAR semantic segmentation on Semantic KITTI dataset Behley 1)21). ). We mloU as eval metric. Results 28.23% for random initializa- tion, 31.88(+3... (-555)) % for UniPAD and 34. 34.28(+6..0)) (+.00)))% for CLAP. It can be found that CLAP is able benefit different tasks and achieves 70% more improvement than UniPAD Yang al. (0244)
B.2 REPRATED EVALUATION
We use the same fixed random seed for all experiments in the main paper for reproducibility. As repeated evaluation can further reveal the training robustness, we repeat fine-tunings on NuScenes for 5 times with random initialization, UniPAD and CLAP. Mean and standard deviation of mAP are 48.55 + 0.18% (Rand.), 49. 66 + 0.29% (UniPAD) and 51.16 + 0.10% (CLAP). CLAP achieves the best average performance and robustness against random seeds.
B.3 TRANSFERRING TO OTHER DATASETS
We conduct further experiments evaluate the transferring ability of CLAP. Specifically, we lect LiDAR-based 3D object detector CenterPoint Yin et al. (2021) on Once Mao et al. (202T) I) for the downstream task. A 40-beam LiDAR utilized Once Mao et al. (2021) to collect 15k la- beled training data. We randomly sample 5% and also use all of the labeled training set to train the from-scratch model until convergence observed. Then we use pre-trained weights by CLAP on NuScenes Caesar et al.] (2020) to initialize the same model and fine-tune it with the same train- ing iterations as the randomly initialized model. Results are shown in Table [ It can be found that pre-training by CLAP also benefits LiDAR-based 3D object detection, even in a cross-dataset setting. And if we look at the performance of "Rand*" and "CLAP#", CLAP also accelerates the convergence in downstream task.
B.4 MORE FINE-TUNING DATA
We fine-tune on 100% NuScenes and Waymo with fixed random seed to provide more insights. Here, we show mAP and NDS for NuScenes 2nd and 3rd cols and Level-2 mAP, mAPH for Waymo. CLAP improves random initialization by -0.55 whereas UniPAD by +0.15 average, showing CLAP's effectiveness. With Table 3 in the paper, more gain can be expected with more unlabeled data in real scenario.
16

---

Published as a conference paper at ICLR 2026
Vehiele
Podestrian
Cyclist
Init.
FT.
mAP
0.30m
30.50m
50m
0.30m
30-50m
50m
0.30m
30.50m
50m
25.22
CLAP*
22.86 +2.38
58.37
26.38
14.07
12.60
9.50
7.88
30.08
10.50
5.39
Rand
46,07
76.71
51.15
31.84
37.53
20.12
9.84
62.00
42.61
24.18
+0
Rand*
64.00
86.21
70.20
58.20
$7.80
41.18
23.55
75.95
61.45
45.80
CLAP*
64.74 +0
88.14 14
72.59
57.37
42.24
24.22
77.11
61.91
45.63
100%
Rand
65.03
88.18
74.23
61.75
57.32
38.90
21.96
78.07
64.32
48.16
CLAP
65.56 +0
87.97
72.77
62.11
58.33
40.11
21.29
78.63
64.70
47.27
Table 5: Results for transferring experiments on Once Mao et al. (022) dataset. CenterPoint Yin
al. (2021) is used the downstream detector. "Init." indicates the initialization methods. "FT."" indicates the number of training samples in fine-tuning stage. Overall mAP and APs for different categories within different ranges shown in this table. "Rand"" means training the randomly initialized model with the original training iterations in OpenPCDet Team (202)). "Rand" indicates that increase the number of training iterations for randomly initialized model until convergence is observed. "CLAP*" indicates that we pre-train the backbones with CLAP on NuScenes Caesar al. (2020) and then fine-tune Once with the original iterations in Team (2020). "CLAP" uses the same tine-tuning iterations as "Rand". We use green color to highlight the performance improvement brought by CLAP. All the results are in %.
Init.
mAP
NDS
mAP
mAPH
Rand.
64.57
67.61
69.00
67.57
UniPAD
64.69
68.06
69.03
67.56
CLAP
65.11
68.38
69.49
68.04
Table 6: Fine-tuning on 100% NuScenes and Waymo with fixed random seed.
C COMPARISON TO "3D TRANS"
We respectfully clarify that methods in 3DTrans focus which requires labels during pre-training, and domain adaptation, which conducts supervised train- ing on one dataset and transfers to other datasets. The focus of 3DTrans is fundamentally different problem settings from our work. The baseline methods our experiments (ALSO, OCC-MAE, SLidR, PPKT, UniPAD) represent the current SOTA in unsupervised 3D pre-training with LiDAR and camera modalities, which the specific problem we address. Meanwhile, we agree that includ- ing supervised/semi-sueervvee pre-training methods would enrich analysis. We provide addi- tional experiments using supervised/semm wewerviddd/eemi--eee sui-ssperri-ede methods from 3DTrans (ADPT and SPOT).
wtwpriiidddsmmssuupririiii pre-training,
Init.
mAP
NDS
From-scratch
48.69
55.28
SPOT
50.98
57.20
ADPT
51.05
56.25
CLAP
51.17
57.04
D DISCUSSION ON PROTOTYPE LEARNING.
Although prototype learning is established the SSL community, we emphasize that our contribu- tion is not simply applying existing prototype methods, but rather novel adaptations for 3D scene understanding:
(1) Different purpose: Unlike SwAV Caron et al. 2020), where prototypes represent different cat- egories within the same modality (images), our prototypes represent parts of 3D scenes and bridge two different modalities (LiDAR camera).
17

---

Published as a conference paper at ICLR 2026
We propose maximizing similarity be- tween 3D embeddings and prototypes by minimizing entropy of assignment probabilities designed specifically for 3D scene part representation, not category clustering.
(2) Sopectation-Maxiiizzee training scheme (Section 3.4
(3) Gram Matrix Regularization: A novel regularization term to prevent prototype collapse in the
3D multimodal setting.
(4) Cross-modks swapping prediction: While inspired by SwAV, our swapping operates between different modalities (predicting camera prototypes from LiDAR embeddings and vice versa) rather than different views of the same modality.
Empirical validation: Table 4 shows that prototype learning provides improvement beyond curvature sampling alone, and the combination achieves 100% more improvement than SOTA (2.48% vs. 1.12% from UniPAD).
E DISCUSSION ON CLAP AND UNIPAD.
We would like clarify that CLAP represents substantial departure from UniPAD Yang et al. (2224), which employs differentiable rendering for pre-training. While both methods use neural- rendering decoders common approach in the field), our decoder differs by incorporating an ad- ditional surface signed distance field loss for better scene geometry optimization. The fundamental distinction is that UniPAD performs separate pre-training of each modality, while CLAP enables joint pre-training with Curvature Sampling and introduces cross-modal interactions via Prototype Learning, which are paradigm shift. CLAP's novel contributions (Curvature Sampling, Prototype Learning with EM training, Swapping prediction loss, and Gram Matrix Regularization) have counterparts UniPAD and enable capabilities that UniPAD cannot achieve. Our ablation study in Table 4 quantifies this difference.
F THE USE OF LARGE LANGUAGE MODELS.
We use Large Language Models to help and aid editing/polishing the paper. In details, polish the Introduction and Related Work sections with Large Language Models, which mostly focuses on grammar, spelling and word choice.
18


Reviews:
[{"review_id": "oKlaApywtC", "reviewer": "Reviewer_JtoM", "summary": "This paper proposes CLAP, a framework for unsupervised fused joint pre-training of LiDAR and camera data through differentiable rendering. CLAP incorporates two key components: a Curvature Sampling strategy to select geometrically informative regions and a set of learnable prototypes to represent distinct parts of a 3D scene, which are optimized using an Expectation-Maximization (EM) procedure.", "questions": "I recommend expanding the comparative analysis to include several recent methods from the '3DTrans' benchmark. A fair and thorough comparison is essential to validate the claim that CLAP represents a meaningful advancement over existing alternatives. The current evaluation feels selective and does not fully establish the method's state-of-the-art status.\n\nI would like the authors to report the mean and variance of five replicates in all experiments, as reproducibility has always been a problem in this field.\n\nI'd like to know how the prototype learning proposed in this paper differs from other existing prototype learning methods. If it's merely being borrowed from this field, then its value wouldn't be that great.\n\nHow much time does online differentiable rendering consume? As far as I know, it slows down the pre-training process. Please explain this in detail.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The manuscript is well-written, and the motivation for the work is clearly established.\n\nThe authors design a curvature sampling strategy to identify informative points and pixels for sampling.\n\nLearnable prototypes are utilized to establish a common feature space, and an Expectation-Maximization approach is employed to optimize these prototypes, enabling them to represent distinct parts of the 3D scene.", "weaknesses": "The coverage of related work and compared methods lacks comprehensiveness. Notably, several recent methods from the \"3DTrans\" GitHub repository, which reportedly achieve strong performance, are neither discussed nor included in the experimental comparisons.\n\nWhen compared to other outdoor self-supervised learning (SSL) methods that also employ differentiable rendering, the primary novelty of the proposed CLAP framework appears to be the Curvature Sampling strategy.\n\nThe use of prototype learning is a well-established technique in the SSL community. Consequently, positioning this as a major contribution is not fully persuasive, as its incremental advancement over existing approaches is not sufficiently demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8huwOmqlcV", "reviewer": "Reviewer_4dye", "summary": "CLAP is an unsupervised pre-training method for fusion 3D perception in autonomous driving, with the core idea of achieving joint unsupervised pre-training of images and LiDAR point clouds through **curvature sampling** and **learnable prototype learning**. \n\nFirstly, CLAP uses a curvature sampling strategy to select informative points and pixels, solving the computational cost problem of multi-modal data processing. Secondly, it randomly initializes learnable prototypes and combines the Expectation-Maximization (EM) algorithm to construct a cross-modal common feature space, associating image semantics with point cloud geometric information. Meanwhile, it designs a swapping prediction loss to explore inter-modal interaction and introduces Gram matrix regularization to avoid prototype collapse. \n\nExperiments on NuScenes and Waymo datasets show that CLAP performs excellently in downstream tasks such as 3D object detection (e.g., BEVFusion, CenterPoint) and semantic segmentation (e.g., Cylinder3D), with significant performance improvements compared to random initialization and existing SOTA methods (e.g., UniPAD). It also has outstanding scalability in few-shot fine-tuning scenarios and can effectively improve model performance during cross-dataset transfer.", "questions": "Given that learnable prototypes are employed to represent 3D scene segments, with similarity estimated via the Gram matrix $G$ to prevent collapse, have you conducted ablation studies on $N_{K}$? Does there exist an optimal $N_{K}$? beyond which the additional prototypes become noise sources rather than meaningful representations?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.Strong scalability of the method: In few-shot fine-tuning scenarios, CLAP’s performance improvement increases as the amount of training data decreases, demonstrating strong scalability potential.\n\n2.Practical engineering significance: CLAP reduces the computational cost of joint unsupervised pre-training for image and point cloud modalities, facilitating information interaction between different modalities.\n\n3.Experimental validation of effectiveness: On benchmarks (e.g., NuScenes, Waymo), CLAP outperforms existing baseline methods. Cross-task (3D object detection, semantic segmentation) transfer experiments further validate its effectiveness.", "weaknesses": "1.Insufficient analysis of results:Further analysis is needed to determine whether CLAP’s advantages lie in handling complex scenarios or achieving broad accuracy gains. Notably, its inferior performance on specific classes (vs. UniPAD/PPKT in Table 1) investigating the underlying causes.\n\n2.Memory efficiency requires further proof:UniPAD separately pre-train the image and point cloud encoders due to GPU memory constraints. Although curvature sampling is introduced to mitigate this issue, the absence of rigorous empirical validation (e.g., memory reduction metrics or training efficiency analysis) limits its persuasiveness. Future work should include quantitative evaluations to demonstrate its benefits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hQCDtBs2Ym", "reviewer": "Reviewer_brLm", "summary": "This paper proposes CLAP, an unsupervised 3D representation learning framework that enables joint pre-training of LiDAR and camera encoders via differentiable rendering. It builds upon the UniPAD framework but introduces two key innovations: Curvature Sampling – a memory-efficient sampling strategy that prioritizes points with higher geometric curvature to retain informative structures during rendering. Prototype Learning – a cross-modal feature alignment mechanism using learnable prototypes optimized via Expectation-Maximization (EM), swapping prediction loss, and Gram Matrix Regularization to bridge image and point cloud modalities. Experiments on NuScenes and Waymo demonstrate good improvements over previous SOTA methods, particularly under few-shot fine-tuning settings. Ablation studies further show the effectiveness of each proposed component.", "questions": "Please check the weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Clear motivation\n\n- Curvature Sampling is intuitive; it prioritizes complex regions (e.g., vehicles, edges) while maintaining low computational overhead in theory.\n\n- Prototype Learning sounds crucial to align and interact between LiDAR and image modalities, improving cross-modal understanding.\n\n- Experiments include both NuScenes (5%) and Waymo (1%), with detailed scaling analyses (0.5–5%) showing gains.\n\n- Ablation results show contributions of each component.", "weaknesses": "- The authors still focus mainly on few-shot downstream training (NuScenes 5%, Waymo 1%), which, while useful for showing sample efficiency, is non-standard in representation learning. Full-data fine-tuning results would better demonstrate scalability and practical performance benefits.\n\n- Although Curvature Sampling is motivated as memory-efficient, the paper does not explicitly quantify memory or runtime savings compared to UniPAD’s “Memory-friendly Ray Sampling.”\n\n- The prototype mechanism is interesting, but it’s unclear how well these prototypes correspond to meaningful physical parts or semantic regions. From figure 5, only the prototype assigned to the ground and the car can be figured out correctly.\n\n- The ablation table confirms effectiveness but doesn’t directly compare against UniPAD’s sampling strategies", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y0Zixf9WN4", "reviewer": "Reviewer_tkBv", "summary": "This paper presents a joint unsupervised differentiable rendering pre-training method for images and point clouds, called CLAP, curvature sampling and learnable prototype. Specifically, CLAP overcomes computational barriers through curvature sampling, thus selecting more informative points/pixels for pre-training. To reveal the complementary performance advantages of both, it is proposed to use learnable prototypes to represent the various parts of the 3D scene in the common feature space and associate the embedding of each modality with the prototype using an expectation-maximizing training scheme. This paper also proposes an exchange prediction loss that explores their interactions through the prototype and uses a Gram matrix regularization term to maintain training stability. Experiments on the NuScenes and Waymo datasets show performance gains of up to 100% for CLAP compared to previous SOTA pre-training methods.", "questions": "As a reviewer, I reviewed this work on an almost per-word-by-word basis. Because of my obvious interest in it, however, given the current presentation quality and experimental results (see **Weaknesses**) of this paper leave much to be desired, I can only give it a neutral rating.\n\nI very much hope that the authors can provide enough evidence to convince me in the limited time available. If the evidence is sufficient, I will not begrudge my score and may highlight it to AC, conversely, I may feel that it needs enough revisions to reject it.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The strengths of this paper are as follows:\n1. The research direction of this paper has obvious value. The joint multi-modal unsupervised representation learning has a wide range of application scenarios for real-world applications in 3D vision.\n2. The research content of this paper is quite challenging, and there is a shortage of work that can effectively solve the limitations in this field.\n3. From the perspective of methods and results, this paper is highly complete and the methods are sufficiently original. For example, the curvature sampling, expectation maximization training, and swap prediction loss are proposed for the first time.\n4. The performance quality of this paper is excellent. Although there are not many illustrations and result figures in the text due to content limitations, it is easy for readers to understand and recognize.", "weaknesses": "The weaknesses of this paper are as follows:\n1. Some elements in Figure 1 should be distinguished, including but not limited to: the sub-images corresponding to the point cloud and rendering loss are consistent in content, which seems to be unable to reflect the role of rendering loss. It is recommended to modify the sub-images corresponding to the rendering loss to have information with object supervision; (a) (b) The element repetition is high. It is recommended to reflect the transformation of the point cloud and rendering loss after using Swapping Prototype Assignment and Curvature Sampling.\n2. As the pipeline of the method, Figure 2 shows the depth integration and RGB integration, however, this is not explained in the subtitles or the text. In addition, Nuscenes and waymo do not have depth maps. How is the depth integration achieved in this paper?\n3. The rendering loss is mentioned many times in the paper, which should refer to the loss of rendering the point cloud as an image. However, the method introduces the projection of points to pixels through a given projection matrix. The rendering process seems redundant?\n4. In Formula 1, the author says that $\\hat{\\mathbf{P}} \\in \\mathbb{R}^{\\hat{D} \\times \\hat{H} \\times \\hat{W} \\times \\hat{d}_{\\mathrm{p}}}$ is the generated embedding feature, which is obviously not per-point features. For each superscript, the author should give a clear explanation, and it is recommended to explain it in text and in Figure 2.\n5. Formula 4 needs more introduction. How are $L_{rend}$ and $L_{proto}$ implemented, and what type of loss is used?\n6. The example of curvature sampling in Figure 3 is unconvincing to me. Because there is no curvature in a two-dimensional image, it is recommended that the author redraw the drawing as described in the text. Operate on LiDAR point cloud, reflect the signed distance function, surface normal, etc.\n7. The experiment section of this paper shows that a pre-trained model is used for fine-tuning, but it is not accurate to call this an unsupervised method.\n8. Figure 4 obviously needs to be improved. It is recommended that the author add sample sub-images in a row to fill the blanks. In addition, why are there fewer points with high sampling weights and concentrated at the bottom of the car object? According to theoretical analysis, shouldn't the area with large curvature transformation be concentrated above or around the car? The author must conduct sufficient experimental analysis, as I guess this may be a defect caused by the implicit signal function used.\n9. The method of this paper is abbreviated as CLAP (**C**urvature samp**L**ing and le**A**rnable **P**rototype). Is it to be as similar as possible to the classic work CLIP? I think it is easier to understand if CSLP appears in the title. Of course, it depends entirely on the author's ideas.\n10. The formulas and mathematical symbols in this paper seem to have flaws. It is recommended to give a complete explanation or separate them by pronunciation, for example, $f^{enc}$ can be changed to $f^{encoder}$ or $f^{en}$. Especially in formulas 13 and 14, there is only one dot above the mathematical symbols. It is recommended to change to a more obvious expression, such as a line or superscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)