CO3: Cooperative Unsupervised 3D Representation Learning for Autonomous Driving
Authors: Runjian Chen
Yao Mu
Runsen Xu
Wenqi Shao
Chenhan Jiang
Hang Xu
Yu Qiao
Zhenguo Li
Ping Luo
Venue: ICLR 2023 poster (2023)
Keywords: Cooperative Contrastive Learning
Contextual Shape Prediction
Unsupervised Representation Learning
Autonomous Driving

Abstract: Unsupervised contrastive learning for indoor-scene point clouds has achieved great successes. However, unsupervised representation learning on outdoor-scene point clouds remains challenging because previous methods need to reconstruct the whole scene and capture partial views for the contrastive objective. This is infeasible in outdoor scenes with moving objects, obstacles, and sensors. In this paper, we propose CO3, namely {Co}operative {Co}ntrastive Learning and {Co}ntextual Shape Prediction, to learn 3D representation for outdoor-scene point clouds in an unsupervised manner. CO3 has several merits compared to existing methods. (1) It utilizes LiDAR point clouds from vehicle-side and infrastructure-side to build views that differ enough but meanwhile maintain common semantic information for contrastive learning, which are more appropriate than views built by previous methods. (2) Alongside the contrastive objective, we propose contextual shape prediction to bring more task-relevant information for unsupervised 3D point cloud representation learning and we also provide a theoretical analysis for this pre-training goal. (3) As compared to previous methods, representation learned by CO3 is able to be transferred to different outdoor scene dataset collected by different type of LiDAR sensors. (4) CO3 improves current state-of-the-art methods on Once, KITTI and NuScenes datasets by up to 2.58 mAP in 3D object detection task and 3.54 mIoU in LiDAR semantic segmentation task. Codes and models will be released.

Published as a conference paper at ICLR 2023
CO3: COOPERATIVE UNSUPERVISED 3D REPRESENTA- TION LEARNING FOR AUTONOMOUS DRIVING
Runjian Chen Yao Mu Runsen Xu Wenqi Shao 2,, Chenhan Jiang
Hang Xu 3, Yu Qiao 2, Zhenguo Li Li Ping Luo 1+
(rjchen, ( hku. hk pluo. 1hi @ew.l.mmalllooo com
[shaowenqi, qiacyu]@pjlab. wttpgjlaa.rgg.ooooomoom org.en
1i zhenguo@huawee. com xb xbjxhelive.com
eun.enxuu cuwk..cuuuuuuuuuu. hk
cjlangao@cconnect. hkust.
The University of Hong Kong
2 Shanghai AI Laboratory
3 Huawei Noah's ss Ark Lab
4 The Chinese University of Hong Kong
Hong Kong University of Science and Technology
ABSTRACT
Unsupervised contrastive learning for indoor-scene point clouds has achieved great successes. However, unsupervised representation learning outdoor-scene point clouds remains challenging because previous methods need to reconstruct the whole scene and capture partial views for the contrastive objective. This is infeasible in outdoor scenes with moving objects, obstacles, and sensors. In this paper, we propose CO3, namely Cooperative Contrastive Learning and Contextual Shape Prediction, to learn 3D representation for outdoor- scene point clouds in an unsupervised manner. CO3 has several merits compared to existing methods. (1) It utilizes LiDAR point clouds from vehicle-side and infrastructure-side to build views that differ enough but meanwhile maintain common semantic information for contrastive learning, which more appropriate than views built by previous methods. (2) Alongside the contrastive objective, we propose contextual shape task-relevant information for unsupervised 3D point cloud representation learning and we also provide a theoretical analysis for this pre- training goal. (3) As compared to previous methods, representation learned by CO3 is able to be transferred to different outdoor scene datasets collected by different type of LiDAR sensors. (4) CO3 improves current state-of-the-ar methods on both Once, KITTI and NuScenes datasets by up to 2.58 mAP in 3D object detection task and 3.54 mloU in LiDAR semantic segmentation task. Codes and models will be released here). We believe CO3 will facilitate understanding LiDAR point clouds in outdoor scene.
prediction to bring
INTRODUCTION
LiDAR is an important sensor for autonomous driving outdoor environments and both of the machine learning and computer vision communities have shown strong interest on perception tasks
LiDAR point clouds, including 3D object detection, segmentation and tracking. Up to now, randomly initializing and directly training from scratch on detailed annotated data still dominates this field. On the contrary, recent research efforts (He et al. 20200] Tian et al. 2019 Caron et al., 2020 Grill al. 2020 Wang al. 202T) in image domain focus unsupervised representation learning with contrastive objective on different views built from different augmentation of images. They pre-train the 2D backbone with large-scale dataset like ImageNet (Deng et al., 2009) in an unsupervised manner and use the pre-trained backbone to initialize downstream neural networks on different datasets and tasks, which achieve significant performance improvement 2D object detection and semantic segmentation GGirshick al. 2014 Lin et al. 2017 Ren et al. 2015). Inspired by these
ccrrespoondig authors are Wenqi Shao and Ping Luo

---

Published as a conference paper at ICLR 2023
Vehicle
Vehicles
Vehicle
View2
Viewl
Sampled scene
View
View 2
Sampled frame
Linear augmentation
T-10 frame
(a) Indoor Scene
(b) Previous Methods Outdoor Scene
frastre
Vehicleside
Vehicle to collect
Vehicle
collect data
Illustration the soene
infrastructure
from vehicle
(c) Our CO3: Cooperative View from Infrastructure and Vehicle
Example views built by different methods in contrastive learning, including (a) previous indoor-scene methods (b) previous outdoor-scene methods and (c) the proposed CO3. Compared to previous methods, CO3 build two views that differ lot and share adequate common semanties.
Figure I:
successes, we explore unsupervised representation learning for outdoor scene point clouds to learn general representations for different architectures on various downstream datasets and tasks.
In the past decade, learning 3D representation from unlabelled data has achieved great success in indoor-scent point clouds. PointContrast (Xie et al., 2020) is the pioneering work and proposes to reconstruct the whole indoor scenes, collect partial point clouds from two different poses and utilize them as two views in contrastive learning to Iearn dense (point-level
voxel-level) representation.
More recent works such as (Hou et al. 2021) and (Liu et al.) 2020) also need the reconstruction and this naturally assumes that the environment is static. Fig. [ (a) shows example of views in PointContrast (Xie et al. 2020). We can see that the views differ lor because they are captured from different poses but meanwhile, they still contain enough common semantic information such as the same sofa and table. These are demonstrated important properties of views in contrastive learning in (Tian al. 2020).
However, outdoor scenes are dynamic and large-scale, making it impossible to reconstruct the whole scenes for building views. Thus, methods in (Xie et al. 2020 Hou al. 2021 Liu et 2020) cannot be directly transferred but there exists two possible ble alternatives to build views. The first idea, embraced by (Liang et al., [021]] Yin al. 2022), to apply data augmentation to single frame of point cloud and treat the original and augmented versions different views, which are indicated by the first and second pictures in Fig. (b). However, all the augmentation point clouds, including random drop, rotation and scaling, can be implemented in linear transformation and views constructed in this way do not differ enough. The second idea is to consider point clouds at different timestamps different views, represented by (Huang et al.) 2021). Yet the moving objects would make it hard to find correct correspondence for contrastive learning. See the first and third pictures in Fig. (b), while the autonomous vehicle is waiting at the crossing, other cars and pedestrians are moving around. The autonomous vehicle has no idea about how they move and is not able to find correct correspondence (common semantics). Due these limitations, it still challenging when transferring the pre-trained 3D encoders to datasets collected by different LiDAR sensors. Could we find better views learn general representations for outdoor-sce -ceeee LiDAR point clouds?
In this paper, we propose COoperative COntrastive Learning and COntextual Shape Prediction, namely CO3, to explore the potential of utilizing vehicle-infrastrucuure cooperation dataset for build- ing adequate views in unsupervised 3D representation learning. As shown in (c) in Fig. [ recently released frfraattecturr--viill---ooppppee
dataset called DAIR-V2X (Yu et al. 2022) is utilized to learn general 3D representations. Point clouds from both vehicle and infrastructure sides are captured at the same timestamp thus views share adequate common semantic. Meanwhile infrastructure and vehicle-side point clouds differ lot. These properties make views constructed in this way appro- priate in contrastive learning. Besides, as proposed in (Wang et al. (022), representations learned by pure contrastive learning lack task-relevant information. Thus further add pre-training goal
2

---

Published as a conference paper at ICLR 2023
Pre-train only once DAIR-V2X dataset with different methods
CD3
Orteffare on
an
Propossi Cantrast
PAROW
Littr Point Cleuds
3D Backbrne
Vovele with
Downstream to various arebitectures for different datasets and tasks
conu Deer in
PVN
detection
3D Object KITTI
LIAA
Once
C03
PropesalCantrast
(4) Pipeline for unsupervised 3D representation learning
(b) Performance change after initialized
different methods
Figure 2: (a) shows the unsupervised 3D representation learning pipeline. (b) presents the performance changes
after pre-training with different methods. Our CO3 achieves consistent improvement.
called contextual shape prediction reconstruct local point distribution, encouraging our proposed CO3 to capture more task-relevant information.
Our contributions can be summarized as follows. (1) CO3 is proposed to utilize the vehicle- dataset to build adequate views for unsupervised contrastive 3D rep- resentation learning on outdoor-se seene point clouds. (2) A shape-context prediction task is proposed alongside to inject task-relevant information, which is beneficial for downstream 3D detection and LiDAR semantic segmentation tasks. (3) The learned 3D representations is generic enough to be well transferred to datasets collected by different LiDAR sensors. (4) Extensive experiments demonstrate the effectiveness of CO3, For example, on 3D object detection task, CO3 improves Second, Center- Points on Once dataset by 1.07 and 2.58 mAPs respectively. As for LiDAR semantic segmentation task, CO3 improves Cylinder3D on NuScenes dataset by 3.54 mloUs.
2 RELATED WORKS
3D Perception Tasks. 3D object detection aims to predict 3D boundary boxes for different objects in the LiDAR point clouds. Current 3D detectors can be divided into three main streams due to the 3D backbones they use: (1) point-based methods (Shi et al. 2019: Chen al. 2017] Yang et al. 2018 use
point-based 3D backbone. (2)
methods (Zhou & Tuzel 2018 Lang et al. 2019 Su et al.) 2021 Fan et al. 2021) generally transform point cloud into voxel grids and process them using 3D volumetrie convolutions. (3) point-voxel-combnne methods (Shi jct al. 2020a; 2021 Deng et al. (02T) utilize features from both (1) and (2). LiDAR Semantic Segmentation aims predict per-point label for LiDAR point clouds. Cylinder3D (Zhu al. 2021
2015; Shi et al. 2020b Yin et al.
and PVKD (Hou et al. 2022) are current SOTA methods for LiDAR Semantic Segmentation.
Unsupervised 3D Representation Learning. As shown in (a) of Fig. unsupervised 3D representa- tion learning aims to pre-train only for one time and downstream to different architectures on various datasets and tasks achieve performance gain. PointContrast (Xie et al. (2020) the pioneering work for unsupervised contrastive learning on indoor-scene point clouds, which relies on the reconstructed point clouds for constructing adequate views. To extend their ideas outdoor-scene point clouds, GCC-3D and ProposalContras, (Liang et al. 202 Yin et al. 2022) augment single frame of point cloud build views and STRL (Huang al. 2021 utilizes point clouds different timestamps as views for contrastive learning. Alongside contrastive learning, authors in (Hu et al. 2021 propose to use safe space prediction as a pretext task for self supervieed representation learning. However, as discussed in Sec. [ previous works fail to build suitable views and their learned representations are unable to transfer to datasets collected by different LiDAR sensors. In this paper, we propose to use point clouds from vehicle and infrastructure to construct views in contrastive learning and also extend the idea of shape context to introduce task-relevant information with a distr-iittiittion prediction goal. The performance change after initialized with different methods are shown in (b) of Fig. 2 C03 generally improve the performance and achieve more performance gains than other pre-training methods for different detectors different datasets.
METHODS
In this section, we introduce the proposed CO3 for unsupervised representation learning on LiDAR point clouds in outdoor scenes. As detailed in Fig. 3. CO3 has two pre-training objectives: (a)
3

---

Published as a conference paper at ICLR 2023
Vehlelo-4ide Point Cloud/ Voxels
3D Rackbone
Vehiele-side Polnts/Vosels with features
(+) Cooperative Contrastive Loss
Sumpled Voticle PalatsYinel
Songled Fuudan
Potats
SharheWeiigtt
negative pair (+) positive pair
(b) Contextual Shape Prediction Loss
Sampled
Predictal
Grosed-traih
3D Rackhone
Fusion Point Cloud Voxels
Fusion Polnts/Voxels with features
Figure 3: The pipeline of CO3, With vehicle-side and fusion point clouds inputs, we first process them with the 3D backbone and propose two pre trairriinng objectives: (a) Cooperative Contrastive Loss (b) Contextual Shape Prediction Loss
cooperative contrastive learning goal on dense (point-level or voxel-level) representations between vehicle-side and fusion point clouds, which provides adequate views for contrastive learning. (b) a contextual shape prediction loss to bring more task-relevant information. To start with, we discuss the problem formulation and overall pipeline in Sec. 3.1 Then we respectively introduce the cooperative contrastive objective and contextual shape prediction goal in Sec. 3.2 and See. 3.3 3.3
3.1 PROBLEM FORMULATION AND PIPELINE
Notation. To begin with, denote LiDAR point clouds P RNx+3+d) the concatenation of the ryz-coordinate C E R Nx3 and the point features F E N which leads to P = [C F). Here N denotes the number of points (or voxels) and d is the number of point feature channels. For raw LiDAR point clouds, d = represents the intensity of each point. Moreover, we use the subscripts (e.g. 'v' and 'i' to indicate point clouds from different sources. For example, P, = [C., and P, = [C Fi] E NNN respectively represent Ny and N; points (or voxels) vehicle and infrastructure sides. Besides, cach pair of vehicle-side and infrastructure-side point clouds is associated with a transformation T mapping infrastructure-sidee coordinate to that of vehicle-side.
[C,, F.J R. (3+d)
Pre-processing and Encoding. For cooperative learning, we need to first align the point clouds from both sides the same coordinate. Hence, we transform the infrastructurr--in point clouds to P/=IC F,] where C₁ T(C;). However, empirically find that the contrastive learning built upon vehicle-side point cloud P. and transformed infrastructure-side point clouds P/ only achieve marginal performance gains than training from scratch on downstream ONCE dataset (0.53 mAPs vs 2.58 mAPs, also see Table 7 in Appendix D). We believe this stems from the sparsity of LiDAR point clouds, which sometimes make it difficult to find good positive pairs to perform contrastive learning. To mitigate this problem, we concatenate the LiDAR point clouds from both sides to fusion point clouds P. Pv, P.] E IR (3)+))))) which is used as the contrastive view of vehicle-side point can express the coordinate and feature of
clouds P, as shown Fig 3 By default of notations, fusion point clouds as C C.,, CJ] and F, = [F. F.l. respectively. Then P, and Py are embedded by the 3D encoder fene to obtain their 3D representations, We use subscript "v/f" to indicate that the same operation is applied respectively on both pointclouds.
P vt fane (P
where Pw E IR (3++d) indicates the vehicle point cloud and fusion point cloud respectively. Here use Now to denote the number of points after encoding because pooling operation often exists in 3D encoders (Graham al. 2018) and changes the number of voxels/points. Moreover, d the number of feature channels after encoding.
Loss Function. To guide the 3D encoder to learn good representations in an unsupervised manner, our proposed CO3 consists of a cooperative contrastive loss Lco, and contextual shape prediction loss Lesp. The overall loss function is given by:
L = iPwll
1
Σ
Lco, (P.,Pr) x Lesp Lesp Pr. Cr)
(2)
Pur(PPr)
4

---

Published as a conference paper at ICLR 2023
where Pvr denote batch of vehicle and fusion point clouds and [Purl indicates the batch size. Cco, applies contrastive learning on the encoded vehicle and fusion pointclouds. Meanwhile, LCsP introduces more task-relevant information into fone by using the encoded features to predict contextual shape obtained by the coordinate of fusion point clouds Cj. w is a weight to balance the losses.
3.2 COOPERATIVE CONTRASTIVE OBJECTIVE
Unsupervised contrastive learning has been demonstrated successful image domain (He al. 2020; Tian et al., 2019) and indoor-scene point clouds (Xie et al.) (020). However, when it turns to outdoor-scene LiDAR point clouds, building adequate views, which share common semantics while differing enough, for contrastive learning is difficult. To tackle this challenge, we utilize recently released
dataset called DAIR-V2L (Yu et al. 2022) and vehicle-side point clouds and fusion point clouds views for contrastive representation learning. Views built in this way differ lot because they are captured at different positions and they share enough information because they are captured the timestamp. More details about view building in contrastive learning can be found in Appendix
Contrastive Head. Following BYOL (Grill al., 2020), we construct contrastive head of cooperative (MLP) layer and a lz fy-normalizatio. Specifically, the embedded features of vehicle and fusion point clouds, F, and F,, are first projected by a MLP layer denoted as MLP, Then an fz-normalization applied along feature dimension. This process is described below, where Zyf E IR Norxd, and dg is the output feature dimension.
contrastive objective by Multi-Layer-Pereeppel
Z MLP (F
ZwI Zwr/ (Zrrl
Cooperative Contrastive Loss. For cooperative contrastive learning, we obtain positive pairs by exploring the correspondence between coordinates C, and Čj. In detail, uniformly sample Ni points from the vehicle point cloud and find their corresponding points the fusion point cloud to form N1 pairs features ([=WE) N == from Zur for contrastive learning, where w R R1xds We treat corresponding points (or voxels) positive pairs and otherwise negative pairs for contrastive
learning. Example pairs are shown in Fig. The final loss function formulated below,
Lco,
Σ N1 n=1 Σ exp(2* 1 =//T)
1
Ni
log(
exp(2") 27' /T)
(4)
where r is the temperature parameter. The bottom right box in Fig. shows examples for this loss. As ground points only contain background information that is not related to perception tasks, we mark those points with height value lower than threshold Zhd as ground points and filter them out when sampling.
3.3 CONTEXTUAL SHAPE PREDICTION
CO3 aims learn representations applicable to various downstream datasets. However, demon- strated in [Wang et al. 2022), pure contrastive loss in Eqn. (4) would result the lack of task-relevant information in the learned representations, making hard to generalize across different architectures and datasets. Meanwhile, an additional reconstruction objective could increase the mutual information between the representations and the input views, which would bring in task-relevant information. Please refer to detailed explanations about this in Appendix B For outdoor-scene point clouds, is difficult reconstruct the whole scene with point/voxel-leww representations. rerersstttn.eeerrr.psetettttto. To mitigate this issue, we propose to reconstruct the neighborhood of each point/voxel with its representation.
Local Distribution. Shape context has been demonstrated as useful point fea- ture descriptor in previous works (Hou
et al. 2 Belongie al. 2002 Kört- gen et al.] 22003] Xie et al. 2018). Fig.
4 showw two examples of shape context with 8 bins (the number of bins can be changed) around the query point, which
(a) Rta Point Cloud
Shaps Contest
Shape
Figure 4: Two examples of shape context.
5

---

Published as a conference paper at ICLR 2023
is marked as larger black point. Previous works the number of points in each bin the descriptor for the query point. However, it can be difficult for the neural net to learn to regress the exact number of points in each bin. Thus, we propose to the representation to predict local distribution built upon shape context. In practice, we divide the neighhorhood of each point in fusion point clouds into Nain 32 bins along zyz--lane with R1 0.5m and R2 4m. Then we compute the "raw" shape context Q' E IR NexNNN which describe the number of points in each bin around every fusion point. Next, we apply Eg-normalization Q' and consecutive softmax obtain the ground-trute tounh shape context Q E R Nrx Nos as described below, where Qi R Nui describes the ground-trun truth local distribution for i-th point.
(5) Prediction Loss. We use the encoded features of both vehicle and fusion point clouds, i.e. Fwr, to predict the local distribution. To be specific, For is first passed through MLP layer MLP2 and softmax operation is applied on the projected features to get the predicted local distribution.
Q = (ottmaxx0Q._/1100,,...
Pw voftmax(MLP2(P)
(6)
where Por E Nor... Then we uniformly sample N2 predictions ((pw))n=))
1.22, from Per and find their corresponding "ground truth" in Q by coordinate correspondence, where we have {g The dimensions of each prediction and "ground truth" are por E TR. 1ais and Lhe E RIINNin Finally, the prediction loss with KL-divergence given by,
1
N2
Nun
n.m
Pí n,
CCsP
(p
log
qr g.m. + Pt
n.m
log n.m. Of
N2 ==
m=1
EXPERIMENTS
The goal for unsupervised representation learning is to learn general representation that benefit different downstream architectures on different downstream datasets and tasks. In this section, we design experiments answer the question whether CO3 learns such representation as compared to previous methods. We first provide experiment setups in Sec. 4.1 and then discuss main results in Sec. 4.2
We also conduct ablation study and qualitative visualization in Sec. and 4.4
4.1
EXPERIMENT SETUP
dataset called DAIR-V2X (Yu et al. 2022) to pre-train the 3D encoder. DA/R-VE V2X the first real-world autonomous dataset for 40-beam while a 120-beam LiDAR is utilized at infrastructure (10084 in vehicle-side and 22325 in infrastructure-siin side for cooperative-detection task. The dataset contains around 7000 synchronized cooperative samples in total.
Pre-training Dataset. We utilize the recently released vehicce-iffrastuuturr-cooom
task. The LiDAR sensor at vehicle-side is
There are 38845 LiDAR frames
the 3D encoder which is 3D con- volutional network because it is widely used as 3D encoders in current state-of-the-art methods (Zhou Tuzel 2018 Yin et al. 202 Shi et al. (0200)). We set the number of feature channels dene 64, the temperature parameter in contrastive learning T = 0.07, the dimension of common feature space of vehicle-side and fusion point clouds d, 256 and the sample number in cooperative contrastive loss N1 2048. For contextual shape prediction, we set the number of bins Noi.. 32, the sample number N₂ 2048 and the weighting constant ws 10. The threshold for ground point filtering is 7th 1.6m. We empirically find that freezing the parameters of ML.P brings better results in detection task thus we them.
Implementation Details of CO3. We use Sparse-Convolutie
Baselines. We implement STRL (Huang et al. 2021) and use the official code of ProposalCon- trast (Yin et al. 2022) as baselines. Besides, as proposed in (Mao et al. 2021). several methods in image domain and indoor-scene point clouds can be transferred to outdoor scene point clouds, including Swav (Caron et al. 2020), Deep Cluster (Caron et al. 2018). BYO (Grill et al., (020) and Point Contrast (Xie et al.] (020). Note that in order to make fair comparisons, all the pre- trainngg methods are pre-traine trainained on DAIR- V2X. Thus there might exist number discrepancy between the results and previous benchmarks.

---

Published as a conference paper at ICLR 2023
Second
PVRCNN
CenterPoint
Inii.
Overall
0:30m
30-50m
Overall
0-30m
30.30m
Overall
0:30m
30.30m
Rand
52.21
60.71
47.31
63.05
50.00
55.92
66.39
50.16
Sway
53.03
61.38
48.34
$4.89
63.41
50.31
57.00
67.54
50.60
Deep Cluster
61.03
47.00
54,91
50.29
57.65
BYOL
45.24
53.84
40.22
49.41
58.04
45.19
52.17
63.53
45,00
PoiniContrast
47.64
56.91
42.54
50.49
58.94
46.30
54.17
65.57
46.53
ProposalContrast
50.70
59.42
46.09
52.79
60.97
48.91
56.02
66.26
48.70
STRL
31.57
59.80
46.59
54,25
63.03
49.31
57.44
67.90
50.47
Ours
53.28 1:007
62.16
49.33
55.17
63.94
50,29
58.50
58.50
69.09
51.51
Table 1: Results of 3D object detection on Once dataset (Mao et al. 2021).
We conduct experiments on 3
different detectors: Second (Zhou & Tuzel 2018) (short Sec.), PV-RCNN (Shi et al. 202T) (short as PV) and CenterPoint (Yin et al. (021) (short Cen.) and 8 different initialization methods including random (short Rand, i.e. training from scratch). Swav (Caron el al.) 2020). Deep Cluster (short as D. CL.) (Caron el al. 2018). BYOL (Grill et al. 2020), Point Contrast (short as P.C.) (Xic et al.) 2020), GCC-3D (Liang et al. 2002]) and STRL (Huang et al. (2021). Results are mAPs in %. "0-30m" and 30-55mm respectively indicate results for
objects in 0 30 meters and 30 to 50 meters. The "Overall" metric highlighted in red is the overall mAP, which
serves major metric for comparisons. We bold font for the best overall mAP of each detector for better
understanding.
Downstream Tasks. Two downstream tasks are selected for evaluation: 3D object detection and LiDAR semantic segmentation. 3D object detection task takes raw 3D point clouds inputs and aims to output 3D boundary boxes of different objects in the scene. LiDAR semantic segmentation assign each 3D point a category label, including Car, Pedestrian, Bicycle, Truck and so on.
3D Object Detection. We select two downstream datasets: Once (Mao et al. 2021) and KITTI (Geiger et al., 2012). Once has 15k fully annotated frames of LiDAR scans. A 40-beam LiDAR is used to collect the point cloud data. We adopt common practice, including point cloud range and voxel size, in their public code repository. mAPs (mean accurate precisions) in different ranges and overall mAP are presented. KITTI is widely used self-driving dataset, where point clouds are collected by LiDAR with 64 beams. It contains around 7k samples for training and another 7k for evaluation. All the results evaluated by mAPs with three difficulty levels: Easy, Moderate and Hard. We select several current state-of-tt aef-theeart methods implemented in the public repository of Once dataset (Mao) al. 2021 I and PCe to evaluate the quality of representations learned by CO3, including Second (Zhou & Tuzel 2018). CenterPoint (Yin et al. 2021) and PV-RCNN (Shi et al. (020a).
LiDAR Semantic Segmentation. We select NuScenes (Caesar et al.) 2020) downstream dataset. There are 1000 scenes each of which lasts 20 seconds in NuScenes. It is collected with 32- beam LiDAR sensor and the total number of frames is 40,000. We select Cylinder3D (Zhu et al. 202T) as downstream architecture. The full training is time-consuming (at least 4 days for one training) and we use a 1/8 training schedule setting evaluate whether CO3 is able to speed up training. We follow the conventional evaluation metrics. mAPs for detailed categories and mean intersection-over--nioo (mloU) for overall evaluation are presented. Per-class mloU first computed as mloU; = +P,+ where TP:, FP; and FN; respectively represent true positive, false positive and false negative for class i. We then average over classes and get the final mloU.
TP.
4.2 MAIN RESULTS
Once Detection. As shown Table [ when initialized by CO3, all the three detectors achieve the best performance the overall mAPs, which we value the most, and CenterPoint (Yin et al. 2021) achieves the highest overall mAP (58.50) with 2.58 improvement. The improvement on PV. RCNN (Shi et al. 2021) is 0.62 in mAP (similar lower improvement with other pre-training methods) because PV-RCNN (Shi et al. 2021) T) has both the point-based and voxel-based 3D backbones, among which CO3 only pre-trains the voxel-based branch. It can be found that other baselines are not able to learn general representation for different architectures. For example, STRL achieves +1 +1.52 mAPs improvements on CenterPoint but degrades the performance of Second and PV-RCNN, On the contrary, CO3 achieve consistent improvement over different detectors.
KITTI Detection. As shown in Table 2 when initialized by CO3, PV-RCNN Shi et al. 2021
achieves the best performance on Easy and Hard (+1.19) level and third place on Moderate level.
attps:
Benchmark
htpp:///ithub.com hbtp://nmmmmaaaaoaaaaaaaaaaa
7

---

Published as a conference paper at ICLR 2023
Second
PVRCNN Miderate
Initialization
Easy
Moderale
Iard
Euxy
Tani
73.29
63.16
60.34
67.23
63.68
Sway
73.23
64.05
60.90
78.43
67.91
64,60
Deep
73.19
63.37
BYOL
71.05
60,39
56.98
77.96
67.50
64,42
Poin:Contrast
72.67
62.74
59.21
77.62
67.79
63,31
ProposalContrat
74.23
64.22
60.88
77.28
66.47
63.22
STRL
73.95
63.90
60.90
77.10
66.21
62.90
Ours
74.40 1:11
64.38
60.90
78.84
00:30
67.75
0..75
64.87
11909
Table 2: Results of 3D object detection on KITTI dataset (Geiger et al. "Moderate" and "Hard" respectively indicate difficulty levels defined in best mAP each detector in each difficulty level for better understanding.
Results are mAPs in %.. "Easy", KITTI dataset. We use bold font for the
2012).
Initialization mloU
Car
Truck
Con. Veh.
Pod.
Trailer
Bic.
M.C.
S.W.
Terrain
Vog.
Random
63,34
84,32
70.25
29.29
64.46
41.35
0.00
60.44
69.76
71.21
83.32
84.70
STRI.
64.71
84,66
76.65
27.30
63.29
52,76
12.79
60.11
70.27
71.70
84,60
Ours
66.88 19:04
85.52
77.00
36.00
66.93
53.13
19.51
70.65
70.40
72.43
84.88
Table 3: LiDAR semantic segmentation on NuScenes (Caesar et al.
2020
Results mloU and mAPs in "M.C.", "S. W." and "Veg."
%. We shows details results in each category. "Con. Veh.". "Ped.". "Bic." abbreviations respectively for Construction Vehicle, Pedestrian, Bicycle, MotoCycle, Sidewalk and Vegetation. We use bold font for the best results in each column for better understanding.
Init.
Overall
Vehicle
Podestrian
Cyclist
Cyclist
Random
55.92
62.85
45.52
59.39
From Scratch
55,92
62.85
45.52
$9.39
Sup. 57.50
46.96
68
CO3 w. ground
$7.37
63.19
48.16
60.76
C03
58.50
64,60
48.83
62.17
CO3
Table 4: Comparison to supervised initialization
Table 5: Ablation study on filtering out ground points.
Meanwhile, when Second (Zhou & Tuzel 2018) is equipped with CO3, it achieves the highest mAPs on Easy level (+1.11), Moderate level (+1.22) and Hard level (+0.56). The lower gains on the KITTI dataset (Geiger et al. 2012) stem from the smaller number of training samples (half of that in Once (Mao et al. 2021)), which makes the detectors easily reach their capacity and improvement is hard achieve. Consistent results across different initialization methods demonstrate this.
NuScene Semantic Segmentation. As shown in Table 3 C03 improve Cylinder3D by 3.54 in mloU and also achieves the best performance among the four initialization methods. Meanwhile, when initialized by CO3, Cylinder3I achieves the best mAPs among all the categories. On truck and construction vehicle, CO3 improve the performance of random initialization by 6.75 and 7.71 mAPs, which is very important in autonomous driving because correct segmentation of different vehicles can help benefit control and avoid accidents.
Comparison to Supervised Pre-training. We find backbone pre-trained on 3D detection task from the official codebase of DAIR- V2X dataset and use it to initialize CenterPoint, after which we fine-tune it on Once dataset. Results are shown in Table It can be found that although supervised pre-training achieve improvement as compared to training from scratch, CO3 leads to the best results in all categories and the overall mAP. This is because supervised pre-training overfit DAIR-V2 dataset, making the improvement lower than CO3 in 3D detection task on Once dataset.
Overall Evaluation. To summarize, CO3 achieves consistent performance gains different architectures on different tasks (3D object detection and LiDAR semantic segmentation) and datasets (Once, KITTI and NuScenes) when pre-trained only DAIR-V22 dataset. In comparison, other baseline pre-training methods only occasionally improve the performance and sometimes even bring degradation. These demonstrate CO3 learns general 3D representations.
4.3 ABLATION STUDY
The influence of ground points in pre-training. We first conduct ablation experiments on filtering out ground points. We use CO3 pre-train 3D backbone without filtering ground points and downstream to 3D object detection with CenterPoint on Once. Results are shown Table When pre-training without filtering ground points, the performance of CO3 drop in each category and
8

---

Published as a conference paper at ICLR 2023
Once (CenterPoint)
KITTI (Seernd)
Overall
Vehicle
Pedestrian
Cyclisi
Overall
Vehicle
Pedestrian
Cychist
Random
55.92
48.71
63.32
Contextual Shape Prediction Ouly
57.30
62.86
49.17
59.86
63.36
77.75
49.16
63.18
Cooperative Contrastive Only
C03
58.50
64.50
48.83
62.17
64.38
77.95
49.59
65.60
Table 6: Results of ablation study Once (Mao et
2021 and KITTI (Geiger et al.
2012).
use
CenterPoint (Yin et al. 2021) on Once and Second (Zhou Tuzel 2018) on KITTL Results are mAPs in %. For Once, results are average across different ranges. For KITTI, results are all in moderate level. We highlight the best performance in each column for hetter understanding.
de Cer
Pre-tmined by 003
Presimincd by PointConimal
Pre-mincal CO3
Prestiiall by PuinCunined
(a) Case
(b) Case
Presimined
Prosimincally by PointConiaa
Presminall by
Prostminal by
(c) Case
(d) Case
Figure 5: Visualization for detection results. Green boxes predicted ones and red boxes are the ground truth.
the overall mAP. This demonstrate the effectiveness of filtering out ground points because ground points contain background information that is not useful for 3D perception tasks.
The effect of each component in CO3. We conduct ablation experiments to analyze the effectiveness of different components. We respectively pre-train the 3D encoder with cooperative contrastive objective and contextual shape prediction objective. As shown in Table it can be found that each objective alone can achieve slight improvement, which demonstrates the effectiveness of either part. Besides, when pre-trained by CO3, we achieve the best performance on the overall mAPs. A more detailed discussion on the ablation study provided in Appendix
4.4 QUALITATIVE EXPERIMENT
We use the 3D backbone pre-trained by CO3 and PointContrast to initialize CenterPoint and train on Once dataset. Then we visualize the detection results in Fig. 5 where predicted boxes are marked as green and the ground truth boxes are red. In Case and 2, when the detector initialized by CO3, the detection results are more correct in headings as shown in the zoom-in area. Correct heading prediction is important especially for control in autonomous driving. Case and 4, can found that CO3 helps the CenterPoint detect object with only a few points captured by the LiDAR sensor and meanwhile, PointContrast initialization fails to detect them. This is also essential autonomous driving because detection failure sometimes lead disaster.
5 CONCLUSION AND FUTURE WORK
In this paper, we propose CO3, namely Cooperative Contrastive Learning and Contextual Shape Prediction, for unsupervised 3D representation learning in outdoor scenes. The recently released aahidd-iinaasruuuur--coppattot dataset DAIR-V V2X utilized to build views for cooperative contrastive learning. Meanwhile the contextual shape prediction objective provides task-relevant information for the 3D encoders. Our experiments demonstrate that the representation learned by CO3 can be transferred to various architectures and different downstream datasets and tasks to achieve performance gain. Currently the size of the real cooperation dataset is relatively small and it will be interesting if larger cooperative datasets be collected for pre-training in the future.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
Ping Luo is supported by the General Research Fund of HK No.27208720, No.17212122 and No. No.17200622.
REFERENCES
Serge Belongie, Jitendra Malik, and Jan Puzicha. Shape matching and object recognition using shape contexts. IEEE transactions on pattern analysis and machine intelligence, 24(4):509-522, 2002.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11221-11611, 2020.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9912-9924. Curran Asso- ciates, Inc., 2020. URL httttp::/prroeeednne neurips cc cc/papprr htt.://2SSS..2222222Aaa2222222a2
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 1907-1915, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255 2009. doi: 10. 109/CVPR.2009 5506848.
Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1201-1209, 2021.
Lue Fan, Ziqi Pang. Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. arXiv preprint arXiv:21 2021.
Keinosuke Fukunaga. Introduction to statistical pattern recognition. Elsevier, 2013.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR). 2012.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 380-587,40 2014.
Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21271-21284, Curran Associates, Inc., 2020. URL URL.tttpps.///rreeeeiiiii http:////pppcccccnnnnnneoooooppp cc/ 2aper/2020/F11ee 2ttpe/00000.e///aaaaaaaadeeeeeee 4
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 15387-15597, 2021.
Yuenan Hou, Xinge Zhu, Yuexin Ma, Chen Change Loy, and Yikang Li. Point-to-voxel knowledge distillation for lidar semantic segmentation. In IEEE Conference Computer Vision and Pattern Recognition, pp. 8479-8488, 2022.
Peiyun Hu, Aaron Huang, John Dolan, David Held, and Deva Ramanan. Safe local motion planning with supe-supe freespace forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12732-12741, 2021.
Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu. Spatio-temporal self-supervised representation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6535-6545, 2021.
Marcel Körtgen, Gil-Joo Park, Marcin Novotni, and Reinhard Klein. 3d shape matching with 3d shape contexts. In The 7th central European seminar on computer graphics, volume 3, pp. 5-17. Citeseer, 2003.
J. Kreer. A question of terminology. IRE Transactions on Information Theory, 3(3):208 208, 1957. doi: 10.11099T...9571155555
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12697-12705, 2019.
Hanxue Liang, Chenhan Jiang, Dapeng Feng, Xin Chen, Hang Xu, Xiaodan Liang, Wei Zhang, Zhenguo Li, and Luc Van Gool. Exploring geometry-aware contrast and clustering harmonization for self-supervised 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3293-3302, 2021.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2117-2125, 2017.
Yunze Liu, Li Yi, Shanghang Zhang, Qingnan Fan, Thomas Funkhouser, and Hao Dong. P4contrast: Contrastive learning with pairs point-pixel pairs for rgb-d scene understanding. arXiv preprint arXiv:2012.1300% 2020.
Jiageng Mao, Minzhe Niu, Chenhan Jiang, Xiaodan Liang. Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, Jie Yu, Chunjing Xu, et al. One million scenes for autonomous driving: Once dataset. 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Claude Elwood Shannon. A mathematical theory communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3-55, 2001.
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and
detection from point cloud. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
11

---

Published as a conference paper at ICLR 2023
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2020a.
Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and aggr-egggnn network. IEEE transactions on pattern analysis and machine intelligence, 4348):2647-2664, 2020b.
Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-ren+++: Point-vort feature set abstraction with local vector representation for 3d object detection. arXiv preprint arXiv:2102.0000
2021.
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned- Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pp. 945-953, 2015.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05849, 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6827-6839. Curran Associates, Inc., 2020. URL BRLhttt::: httptttss///pccceeiiing neurips cc/ wt2002/0/11ee/ee htt.://lee/Aceeeeaeeeeeeeee papprr/200//LLee/cceeeesee
Haoqing Wang, Xun Guo, Zhi-Hong Deng, and Yan Lu. Rethinking minimal sufficient representation in contrastive learning. arXiv preprint 22X1v:2203.07004 2022.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2021.
Wikipedia contributors. Mutual information Wikipedia, the free encyclopedia, 2022. URL
http:///ewwwwiiiid.eee www.or/////...... owtpa//iewww///ixx.../.a inwex.iiixie.eee pttpdex.pppptitteee.. MutuuuuueeMuuu informations. oldid=108934362 [Online; accessed 21-June-2022).
Saining Xie, Sainan Liu, Zeyu Chen, and Zhuowen Tu. Attentional shapecontextnet for point cloud recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4606-4615, 2018.
Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European Conference on Computer Vision, pp. 574-591. Springer, 2020.
Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Real-time 3d object detection from point clouds. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7652-7660, 2018.
Junbo Yin, Dingfu Zhou, Liangjun Zhang, Jin Fang. Cheng-Zhong Xu, Jianbing Shen, and Wenguan Wang. Proposalcomtrast: Unsupervised pre- pre-training for lidar-based 3d object detection. 2022.
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11784-11793 2021.
Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, and Zaiqing Nie. Dair-v2x: A large-scale dataset for infialittffrsttucuuuree cooperative 3d object detection. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), June 2022.
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4490-4499, 2018.
12

---

Published as a conference paper at ICLR 2023
Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In IEEE Conference Computer Vision and Pattern Recognition, pp. 2021.
13

---

where N is the number of images in the batch) to get augmented images xi and zz. which are called different views of x². The main objective of contrastive learning is to pull together the representations of views of the same image in the feature space while pushing away representations of different Published as a conference paper at ICLR 2023
A BACKGROUND ABOUT VIEW BUILDING IN CONTRASTIVE LEARNING
In this section, we discuss how to build proper views in contrastive learning. Firstly, we introduce the formulation of contrastive learning and some important properties of good views in contrastive learning as proposed in (Tian et al. 2020). Then we discuss view building for LiDAR point clouds.
View Building Contrastive Learning. Unsupervised representation learning aims to pre-train 2D/3D backbones on a dataset without labels, which can be transferred to downstream datasets and tasks to achieve performance improvement over training from scratch (random initialization). Recently, unsupervised contrastive learning achieves great success in image domain (He ct al. [0200] Tian et al., 2019; Caron et al. 2020 Grill et al.) 2020 Wang et al., 202T). Given batch of images as inputs, these works first apply two kinds of random augmentations for each image x" E X (n = 1,2,....
images, indicated in the equation below:
exp(= =z/T²)
Lecon
N Σ
log(
with
E exp(=? =b//T)
(8)
n = 1, ....N N
where fi is the 2D backbone used to extract representations. 21" is the encoded representations for views 2 To apply contrastive loss in the first line in Eqn. (8). views of the same image are considered as positive pairs and other pairs are negative ones. The numerator indicates the similarity of positive pairs while the denominator sums up the positive similarity and the sum of similarity of negative pairs. is the temperature parameter. Minimizing this loss equals maximize the similarity of positive pairs and minimize the similarity of negative pairs.
Authors in (Tia et al. 2020) discuss what property views should have to benefit contrastive learning via Information Theory and propose that mutual information (Shannon 200T: Kreer 1957| Wikipedia contributors 2022) of views _I(x;: xx"), can indicate the quality of the learned representations. Mutual information formally quantifies "how much information about one random variable we can obtain when observing the other one". Experiments on images suggest that there exists "sweet spot" for [xP;:=)) where the pre-trained backbone can achieve the most significant performance improvement in downstream tasks. This means the mutual information between views can neither be too low (sharing little semantics) nor too high (differing little). Further experiments in (Tian augment al. 2020) indicate that the mutual information of different augmented images high and reducing [x]; 22) by applying stronger augmentations is effective. The performance on downstream tasks increases at the beginning and then decrease when the augmentations are too strong.
Views Building for LiDAR Point Clouds. As discussed in the main paper, it is impossible for us to reconstruct the whole outdoor-scene for constrative learning, which is demonstrated useful in indoor-scene (Xie et al., 2020, Hou et al., 2021 Liu et al. 2020). And there exists two alternatives to build views for outdoor-scent LiDAR point clouds. The first one (Liang et al. 2021: Yin et al. 2022) is to apply data augmentation to single frame of point cloud and treat the original and augmented versions as different views, which is similar to what previous works do in image domain. However, the augmentations in image domain are highly non-linear while all the augmentation of point clouds, including random drop, rotation and scaling, can be implemented in a linear transformation. As claimed in (Tian et al., 2020), the highly non-linear augmentations on images already bring high mutual information between views. Thus views of LiDAR point clouds built in this way would have higher mutual information, which is not adequate for learning representations. The second intuitive idea to build views is to utilize point clouds at different timestamps, embraced by (Huang et al. 202Tb. However, outdoor-scenes are dynamic and the autonomous driving vehicle has no idea about how other objects (cars, pedestrians, etc.) move. Thus observing one view (timestamp t) bring in little information about the other one (timestamp [+10 for example), meaning that [xf;; x/)) can be extremely low and this can be harmful to the learned representations. Due to these limitations, pre- trained 3D encoders in (Liang et al. 2021 Huang et al. (002)) cannot achieve noticeable improvement when transferring to datasets collected by different LiDAR sensors. Thus, in this paper, propose to utilize the from different view-point at the same timestamp, for contrastive representation learning. Views built
cnffrastructure-ccoperttio dataset (Yu et al. 2022). which capture the same scene
14

---

Published as a conference paper at ICLR 2023
with this dataset neither share too much information (captured from different view-points) nor share too little information (captured at the same time, easy to find correspondence), which is adequate for contrastive learning.
B RECONSTRUCTION OBJECTIVE FOR TASK-RELEVANT INFORMATION
In this section, borrow the ideas from (Wang et al. 2022 to explain why pure contrastive learning bring less improvements as shown in Table 3 in the main paper. Firstly, we give the definition of sufficient representation and minimal sufficient representation in contrastive learning. Then, present analysis on image classification problem as downstream task and we refer readers to (Wang al. 2022) for other downstream tasks. Finally, we propose our pre-training objective for LIDAR point clouds.
Sufficient Representation and Minimal Sufficient Representation. Sufficient Representation s of x contains all the information that is shared by of and z22 which means 2i, can be used to express common semantics shared by these two views. Among all the sufficient representations for 422 minimal sufficient representation 2i,minn contains the least information about x", The learned representation in contrastive learning is sufficient and almost minimal. Assuming that [sf the set of all possible sufficient representations of view T/, we can define these two concepts as belows,
Definition 1. "i, of view xf is sufficient for 422 if and only if [2 222) 1 1(x]
Definition 2. 2',min E Zi. of view af is minimal sufficient if and only if I2 z4) S (z 2T), V2i,uuf E Zi.su.-
Theorem. (1) 21. provides more information about the downstream task T than mini (2) The upper bound of error rates in downstream tasks using minimal sufficient representations are higher than that of sufficient representations. That is,
[2 7) 2 T [= T) Sup(Pan S sup PP
This gap stems from the missing task-relevant information in 21,min- To prevent this problem, authors in (Wang et al. 2022) propose to add reconstruction objective (reconstruct 21 using #1) alongside the contrastive loss to increase [ .I). which indirectly increases I(21, T(x2) and brings improvement in downstream classification problem over pure contrastive learning.
Proof for Image Classification Problem. We denote the downstream classification task as T and the task-relevant information in minimal sufficient representation 21,min be described as below:
[22 T) [221, T) + [T(=1, T mm, [ s(1,
(10)
2 [(21,min; T)
To begin with, 21. and mi, are sufficient representations and they contain two parts of infor- mation: shared information between xι and z2, and extra information about Thus the mutual information 1 s21,,utf: 2)) can be decomposed into 1 (221,min: T') and [x (.11,") 21..mm)))) ) _(x1 T st)),, where I (x1, 1minn) indicates the information about T we can obtain by observing 2 when 21,min is known. As 21, contains more information about I than 31,mins the second term is larger than zero and the hight-hand-side of the first line in Eqn. (10) is larger than T(21, (1.mm T). This indicates that sul contains more task-relevant information than 21, and thus would have better performance in T. Then consider using Bayes error rate pe (Fukunaga) 2013), which is the lower-bound of achievable error for the classifier, to analyze performance of 21, and Z) mi, on downstream classification problem. We have
pe < 1 cxp + [xx x2, T2,T) + 1 [22 T(22)
(11)
min < 1 - exp(-H(()) 1(2,, 22, T)]
where (T) is the entropy of the task. Since 1 e-
[[11, L2, +( T[x2)] 1- 1-
exp(-H(TT) + [221, 22, I),, the upper bound of Bayes error rate of minimal sufficient representation is larger than that of sufficient representations. This indicates that ideally 2 ouf can achieve better performance than min in classification problem.
15

---

Published as a conference paper at ICLR 2023
Contextual Shape Prediction Objective. As it is impossible to reconstruct the whole scene point cloud with point-level or voxel-level representations. We propose an additional pre-training objective to predict distribution of local neighborhood using point/voxel-eeeel representation. We use shape context to describe the local neighhorhood distribution of a point/voxel, which has been demonstrated useful local distribution descriptor in previous works (Hou al. [02T [ellngge al., 2002 Körtgen et al. 2003; Xie et al.) 2018). As demonstrated in our ablation study (Table in main paper), this additional pre-training objective bring more significant performance improvement over pre-trained by pure contrastive loss. We also provide python-style code for computing shape context as followings
Algorithm 1 Implementation of Contextual Shape Computation in Python Style.
class Contextual Shape (object):
def init (self, r1=0.125, r2-2, nbins xy-2, nbins zy=2)
self.rl rl
self.r2 = r2 self.nbins xy nbins xy
self.nbins zy nbins_ zy
self.partitions nbins_xy mbins_ey
def (rel_ [_tee_tttaaa::
D2 = torch. srrech..umrre__trnns return torch.sgrt (D2 1e-7)
3)
def compute_rel_trans batch (A, B): return (1)
unsqueezz (2)
def hash_batch (A, B, seed)
mask = (A >= 0) & (B >= 0)
- torch.zeros zekes_likell (A) -1
[[mask] Amask] seed [mask]
return
def compute_anglee htt.raannngggoatttttttteeeetaaae angles_Xy angles_xy torch. fmod (angles__y angles_zy torch, atanz torch.atan2 (rel_tranns [;, i, i, 1), rel_trans(:: i, ;, 2])
atach (rel_trane ;, :, 1), rel_trans :, 1, 01)
math.pi, 2 math.pi)
angles_zj torch. (angles_zy return angles_ argles_2y
math.pi, math.pi)
def compute_partttionn bartitiins_batchhiie (self, syz_batch):
rel rel_trans_batte batch
(xyz_batch,
xyz_batch)
compute angles from different points query ingles_xy_bate angles_zy_bated ShapeContext. htwp:ontteet.coootet__eeeaee compute_angles batch
rel zel_trans_batch
angles_xy_biie xy bht
batch = torch. floor (angles_ jaytsss_yytbathh / (2 math
self. self__abins_x))
angles_ay_bins__atte torch. floor (aygles___yybbattc) (math.pl self
abins_2y)]
angles_bins_batch
angles__y_bins__atte self self.nbins_ry)
compute distances between different points and the query one
seel_trras_baatoe aestance_matrix_bateee.o batch hagpcheeeeo__iiieaaggeeeeaa.aa (akelagges_ dist_bins_batco torch.zeros__
generate partitions for each points
mask_
ceaastance_matrrie battrix_batchhhee >= self.rl) &
distance_natriixbatteh dist dist_bins batchb/at///mmss batchl =
self.r2)
mask bask_batch distance matrix batch >= self.r2
dist_bins_batchh (mask_bachhhh =
pins batch ShapeContext hash batch (dist bins batch,
bins return bins_batch
bates__biss_bbtthhhe batch, self.nbins
_zyy]
16

---

Published as a conference paper at ICLR 2023
C DATASETS DETAILS
In this section, we introduce details about different datasets used in the main paper for evaluation and also two more datasets in additional experiments.
DAIR-V2X, DAIR-V2. V2X Yu et al., 2022) is the first real-world autonomous dataset for vehicle- infrastructure-ccoppraatt detection task. It covers various scenes, including cities and highways, and different whether condition including sunny, rainy and foggy days. A virtual world coordinate is used to align the vehicle LiDAR coordinate and infrastructure LiDAR coordinate. There are 38845 LiDAR frames (10084 in vehicle-side and 22325 in infrastructure-side for cooperative-detectin task. The dataset contains around 7000 synchronized cooperative samples in total and we utilize them to pre-train 3D encoder in unsupervised manner via the proposed CO3. The LiDAR sensor at vehicle-side is 40-beam while 20-beam LiDAR is utilized at infrastructure--tie
Once. Once (Mao al. 2021 is a large-scale autonomous dataset for evaluating self-supervised methods with Million LiDAR frames and only 15k fully annotated frames with 3 classes (Vehicle, Pedestrian, Cyclist). A 40-beam LiDAR is used in (Mao et al.) (02T) to collect the point cloud data. We adopt common practice, including point cloud range and voxel size, in their public code repository As for the evaluation metrics, IoU thresholds 0.7, 0.3, 0.5 are respectively adopted for vehicle, pedestrian, cyclist. Then 50 score thresholds with the recall rates ranging from 0.02 to 1.00 (step size if 0.02) are computed and the 50 corresponding values are used draw PR curve, resulting in the final mAPs (mean accurate precisions) for each category. We also further overage over the three categories and compute an 'Overall' mAP for evaluations.
KITTL KITT/ (Geiger et al. 2012) is widely used self-driving dataset, where point clouds are collected by LiDAR with 64 beams. It contains around 7k samples for training and another 7k for evaluation. For point cloud range and voxel size. adopt common practice in current popular codebase like MMDetection3DDaan epenPCDef All the results are evaluated by mAPs with three difficulty levels: Easy, Moderate and Hard. These three results are further average and an 'Overall' mAP is generated for comparisons.
D ADDITIONAL EXPERIMENT RESULTS
D.1 INFRASTRUCTURE AS VIEW IN CONTRASTIVE LEARNING
We also conduct ablation study on the fusion view. Instead of using fusion point clouds as view in contrastive learning, we directly use infrastructure side point cloud as another view. Results are shown in Table It can be found that if directly use infrastructure-sidee point cloud for contrastive learning, the performance improvement very marginal. This stems from the sparsity LiDAR point clouds, which sometimes make it difficult to find good positive pairs to perform contrastive learning.
Init.
Overall
Vehicle
Pedestrian
Cyclist
Random
55.92
62.85
45.52
59.39
Inf-view
56.45
62.71
46.63
60.02
C03
58.50
64.60
48.83
62.17
Table Resuls of directly using infrastructure- side point cloud as view in contrastive learning.
E IMPLEMENTATION DETAILS
In this section, we introduce some details about implementation in both pre-training stage and fine- tuning stage. Common settings of pre-training and fine-tuning are listed in Table 8 and we discuss other settings that vary in different detectors later.
on3d
17

---

Published as a conference paper at ICLR 2023
Configuration
Pre-training
KITTI
Once
optimizer
Adam
Adam
Adam
base learning rate
0,0001
0.003
0.003
weight decay batch size learning rate schedule
0.01
0.01
0.01
16
cyclic 8
cyclic
cyclic
GPU numbers training epochs
4
4
20
80
80
Table 8: Details about implementations. "Pre-training" means settings in DAIR- V2X Yu al. 2022). "KITTI" "Onee" respectively indicate settings for detection tasks KITTI (Geiger 2012 and Once (Mao 202T). We list all common settings and discuss those vary in different delectors below, which are marked " in this table.
Other Pre-training Settings. To accelerate the pre-training process, we utilize the "repeated dataset" in MMDetection3D and the schedule is set to 10 epochs, which equals 20 epochs without "repeated dataset". Thus the number 20 for training epochs in Table tilt.
Other Fine-tuning Settings. Batch size settings for different detectors different datasets shown in Table 9 We use different types of GPUs, different number of GPUs and different version of PyTorch (Paszke al., 2019) compared to those used in the codebases, which may lead to degrading when training from scratch. Thus these parameters are tuned based on the original settings from the codebases make the performance of training from scratch match even surpass the results they published.
Detectors
KITTI
Once
Second
48
48
PV-RCNN
16
48
CenterPoint
32
Table 9: Details about batchsize settings different detectors on different datasets. "." means there no configuration for the detector on the exact dataset in the codebase or we do not conduct the downstream experiments.
18

---

Published as a conference paper at ICLR 2023
F
TABLE OF NOTATION
Variable
Description
Note
Pwr
Raw vehicle/fusion LiDAR points.
Point/Voxxcl-eeet features after embedded by the backbone. Projected point/voxel-leve features in the common feature space.
Indicating same process applied vehicle/fusion point cloud.
Pwr
Zuit
An I.-2 normalization applied the feature dimension of Z.r. 20r is sampled point/voxel feature for contrastive learning.
Normalized point/voxel-leve features in the common feature space.
Zv
N1
The number of samples for contrastive learning.
Temperature parameter in contrastive learning.
A matrix in dimension B N indicating the number of points in each bin in the neighborhood of each fusion point.
Q'
Apply 22- 2-noomalization and softmax to Q then get Q E R x which describes local geometry distribution the neighborhood of each fusion point.
dur is sampled "ground truth"
Q
distribution from Q
Por is sampled distribution prediction from P.
Predicted local geometry distribution from P
N2 The number of samples contextual shape prediction.
Table 10: Detailed description of variables
G DISCUSSION ABOUT THE ABLATION STUDY
(a) Different Cases of Pedestrian Class
(b) Different Cases of Cyclist Class
Figure 6: Example samples of pedestrian and cyclist. LiDAR points are shown in red. It can be found that pedestrian category has consistent shape while the shape of cyclist varies across time and identities.
It can be found in Table 6 that using contextual shape prediction loss only brings improvement on Pedestrian class while cooperative contrastive loss introduce more gains on the Cyclist class. In this section, provide discussion this phenomenon.
First of all, the "ground truth" shape context is computed with point clouds and the contextual shape prediction goal is to predict the local point distribution with voxel-level representations. This
19

---

Published as a conference paper at ICLR 2023
enables the voxel-level representation predict the structure inside voxel. Meanwhile, cooperative contrastive only focus on contrast different voxel representation. For pedestrian class, there are usually only or two voxels for one pedestrian. Thus, with cooperative contrastive loss only, the representation fail to recognize the inner structure the voxel and thus cannot improve upon the redestrian category. On the contrary, when pre-train the encoder with contextual shape prediction loss only, the learned representation is able to express the structure inside the voxel and this helps with the downstream detection on the pedestrian category.
Second, as the contextual shape prediction goal aims to capture local shape distribution, it fails learn good representation with varying shape. Compared to pedestrians whose shape is always cylinder-like, cyclists with their bicycles are usually captured with different poses, leading to different shapes. visualization of different cases of the two categories are provided in Fig. ( Thus the representation fails to learn knowledge when predicting varying shape of the same semantic. When we look at the cyclist category, can be found that pre-training with contextual shape loss only brings little improvement.
Besides, as discussed Appendix B the two pre-training objectives are complementary. Pure cooperative contrastive learning makes the representation minimal sufficient, which lacks of task- relevant information. The contextual shape prediction loss brings more task-relevant information by increasing the mutual information between the representations and the inputs. Thus combining them leads to better performance.
H
MORE EXPERIMENTS.
H.1 ABLATION EXPERIMENTS SHAPE CONTEXT.
We conduct a ablation experiment on the contextual shape prediction loss. Previous works (Belongie al. 2002 Körtgen et al. 2003) use number of points in each bin as the feature of the query point. However, we think it is difficultin for the network to directly regress the exact number of points in the bins. Thus we propose construction a local distribution and use the representation to predict In this part, conduct ablation study this design. We use the exact number of points prediction goal and keep other parts of our method the same. The experiment results are shown in Table [T] and it can be found that directly transfer the idea of shape context only brings little improvement (0. (0.94mAP) as compared to the proposed CO3 (2.58 mAP). Thus our insight can also be transferred to indoor scene point clouds.
Initialization
Overall mAP
Vehicle
Pedestrian
Cyclist
Random
55.92
62.85
45.52
59,39
C03 with exact number of points prediction
56.86 (+0.94)
62.79
47.63
60.15
C03
58.50 (+2.58)
64.60
48.83
62.17
Table 11: Ablation experiments shape context.
H.2 EXPERIMENTS COMPARED TO SAFETY SPACE FORECASTING.
We also conduct experiments where freespace forecasting (Hu et al. 2021 used as pre-training goal. The pre- pre-training setting is kept the same all the other initialization methods. The results are shown in Table 12 It can be found that pre-training with safety space forecasting (Hu et al. (021) brings minor performance gain. We think this might stems from the objective of the pre-training goal. The pre-training goal of (Hu et al. 2021) is to predict freespace for safe driving and their downstream task is motion planning. However, 3D perception task requires semantic information and simply predicting freespace does not help distinguish the representation of different objects. This is why pre-training with freespace forecasting bring lower performance improvement on 3D perception task.
H.3 PARAMETER SENSITIVITY EXPERIMENTS
In this part, we conduct parameter sensitivity experiments on temperature parameter and radius in contextual shape prediction loss.
20

---

Published as a conference paper at ICLR 2023
Initialization From Scratch
Overall mAP
Vehicle
Pedestrian
Cyclist
55.92
62.85
45.52
59.39
Freespace forecasting
56.18 (+0.26)
63.33
46.19
59.01
C03
58.50 (+2.58)
64.60
48.83
62.17
Table 12: Experiment results on freespace forecasting pre-training.
We first fix all the other parameters in the main paper and change R1 and R2 in the contextual shape prediction loss. We pre-train the 3D encoder on the DAIR-V2X dataset and downstream dataset with CenterPoint as detector. Results are shown in Table [3 It can be found that for the same R2, an increasing RI brings more improvement on the downstream detection tasks. This might stem from that a larger inner radius can help capture more neighborhood information, making the contextual shape prediction goal more meaningful. It can also be found that when R2 relatively small, for example R2=3.5m and R1=1m, the performance drops. This might stem from the relatively small R2 which might fail to capture the local shape distribution a larger object like cars or trucks. Also, we did not scarch these parameters before and we surprisingly find that using R1=1.5m and R2=4m brings the best performance (58.86 86 mAP).
Once
R1/m
0.5
1.0
1.5
2.0
R2/m
3.5
57.55
57.31
58.00
58.42
4.0
57.53
58.50
58,86
58.48
Table 13: Results on parameter sensitivity experiments on temperature.
In the experiment on temperature, all other parameters are kept the same as those in the main experiment and we only change the temperature in the cooperative contrastive loss. We pre-train the 3D backbone on DAIR-V2X dataset with different parameter settings and fine-tune it on Once dataset with CenterPoint. The results are shown in Table 14 It can be found that as the temperature increases to 0.15, the downstream performance drop to 57.34 mAP. This is because a higher temperature brings smaller gap between negative pairs and this makes the representations hard to separate different objects. Also, as we keep decreasing the temperature, the overall downstream performance drop to 57.74 mAP. This is because too small temperature push representation of different objects too far away but ignore the similar semantic meaning for the same category. For example, it will make representation of two different cars far away and this will harm the performance of downstream detection task. It can also be found that using adequate temperature brings comparable performance (58.50 mAP 58.10 mAP).
Temperature
0.02
0.07
0.1
0.15
Overall mAP
57.74
58.50
58. 10
57.34
Table 14: Results parameter sensitivity experiments on radius in contextual shape prediction loss.
I DISCUSSION ON THE INFLUENCE TO V2X COMMUNITY.
The V2X community is developing rapidly and the original motivation of V2X setting is to alleviate occlusion and long-range sensing problems (Yu et al., (022). There are several V2X settings including vehicle-to-infraasrrucuue and vehicle-to-vehicle
In this paper, we propose to use cooperation dataset for unsupervised representation learning and achieve performance improvement on perception task using vehicle-side point clouds only. Although our experiments are conducted on rchicle-infrastrettue other V2X settings without any label. As labeling is the most intensive and time-consuming part in the collection of cooperation dataset, we believe larger scale of unlabeled cooperation dataset will be collected in the future for unsupervised 3D representation learning to introduce more performance gain. Also, it is expensive and difficult to deploy V2X settings everywhere. In our work, we pre- train such cooperation dataset and achieve improvements downstream tasks with vehicle-side
dataset, our method can also be applied on
21

---

Published as a conference paper at ICLR 2023
point clouds as inputs, which is new exciting finding about the V2X research. We believe the promising results will encourage more attempts in cooperative unsupervised representation learning and accelerate the development of the V2X community.
22


Reviews:
[{"review_id": "uKpFkm8jtV", "reviewer": "Reviewer_1HxB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose CO3, Cooperative Contrastive Learning, and Contextual Shape Prediction for unsupervised 3D representation learning in outdoor scenes. By using DAIR-V2X, a vehicle-infrastructure-cooperation dataset, the data mitigates challenges due to the moving objects that would make it hard to find correct correspondence for contrastive learning. The experiments demonstrate that the representation learned by CO3 can be transferred to various architectures and different downstream datasets and tasks to achieve performance gain.", "summary_of_the_review": "The reviewer thinks it is interesting to leverage DAIR-V2X for unsupervised 3D representation learning. However, significant concerns are found and discussed in the Weaknesses section. The reviewer would like to know the feedback on the questions.", "strength_and_weaknesses": "Strength\n\n1. As we know, it is non-trivial to annotate point cloud data in scale. Unsupervised 3D representation learning emerges. The paper proposes an interesting approach that leverages point cloud data collected in a V2X setting. The setting mitigates challenges due to the moving objects that would make it hard to find correct correspondence for contrastive learning.\n2. Empirically, the representation learned by the proposed **Co**operative **Co**ntrastive Learning and **Co**ntextual Shape Prediction (CO3) can be transferred to different outdoor scene datasets collected by different LiDAR sensors. Specifically, they show favorable performance compared with STRL (Huang et al., ICCV 2021) on the KITTI and nuScene datasets on LiDR 3D detection and semantic segmentation, respectively.\n\nWeaknesses\n\n1. LiDAR is indeed a valuable sensor for perception. However, it is too strong to claim that LiDAR is the most reliable sensor in outdoor environments. LiDAR has many issues, e.g., sparsity, visibility under occlusion, and false alarm due to smoke and fog. Please consider rephrasing the first paragraph.\n2. The key question the authors aim to answer is: how to learn 3D representation from unlabelled point cloud data for outdoor scenes? The authors argue that existing approaches are not suitable for outdoor scenes. However, they ignore a recent advance in Freespace forecasting (Hu et al., CVPR 2021). The work could be an alternative to be compared with the proposed one. Using point clouds from both the vehicle and infrastructure sides is okay because we want data captured simultaneously. Thus, views share adequate semantics. However, the motivation is not convincing by arguing the limitations of existing self-supervised learning for point cloud data without discussing Hu et al., CVPR 2021.\n- Hu et al., Safe Local Motion Planning with Self-Supervised Freespace Forecasting, CVPR 2021\n3. The proposed approach's novelty is limited: This paper's two ideas are cooperative contrastive objective and contextual shape completion. The cooperative contrastive objective is the contrastive loss proposed by He et al., 2020. The only change is the input features. It seems that the contribution (cooperative contrastive objective) is over-claimed. The enabler of the proposed cooperative contrastive objective is the DAIR-V2X dataset. As mentioned by the authors, many existing approaches for shape context modeling exist. Why Eq. (5) is a better choice? Without a clear justification, we cannot conclude the importance of Eq. (5).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "hEWs3UAt9C", "reviewer": "Reviewer_bVKL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for self-supervised representation learning of LIDAR point clouds, to initialise training of standard supervised methods for LIDAR 3D object detection and point cloud segmentation. The paper has two main contributions:\n\nThe self-supervised representation is learnt via contrastive learning on point clouds from LIDAR mounted on vehicles and LIDAR mounted on infrastructure, synchronised with respect to each other. This provides more diversity than state-of-the-art methods, that use ad-hoc augmentations or LIDAR point clouds collected by the same vehicle but at different timestamps (which is problematic due to moving objects)\nA shape-based loss as part of the unsupervised learning step that is meant to bring in task-specific information into this step\nThe paper is evaluated on standard 3D object detection and LIDAR semantic segmentation benchmarks, showing marginal (but consistent) improvements over alternative self-supervised initialisation methods.", "summary_of_the_review": "Overall, I recommend accepting the paper, and I'd like to invite the authors to address the points I listed in the weaknesses section.\n\nThere is enough novelty in the contributions (using a new source of point cloud data for self-supervised representation learning for point clouds and use of task-relevant losses based on shape during self-supervised learning). There is also enough signal showing the usefulness of the method. While the gains compared to previous methods are overall small, they are very consistent across datasets, supervised tasks, and supervised methods. Some additional analysis on the sensitivity to multiple runs (to get an estimate of the variance) would make the case stronger. The paper is clearly explained, with enough details for reproducibiltiy.", "strength_and_weaknesses": "Strengths:\n\nThe core ideas in the paper are interesting. I liked the concept of introducing a more task-relevant loss based on shape in the self-supervised stage\nThe evaluation procedure is mostly thorough and sound, using several public benchmarks, comparing against relevant related work, and with appropriate ablative analysis. Results are consistent across datasets, which is convincing (see however my concern under weaknesses on the magnitude of the deltas)\nThe paper is written with a level of clarity that is significantly above average (I commend the authors for this, it makes reviewing so much easier!)\nWeaknesses:\n\nIn my opinion, the main weakness is the magnitude of the improvements. I am looking for example at Table 2, where the difference between the various methods in <0.5 mAP. It would be valuable to have some sort of estimate of the variance, for example, how much does performance change upon different runs of training (e.g. with different random initialisation) and (less important) across the values of hyperparameters (e.g. the temperature, the shape descriptor radii) . However, it has to be said that, while small, the improvement can be observed consistently across various datasets, tasks, and networks used for supervised learning\nThe method to generate data for self-supervised training seems hard to scale, as it seems quite expensive and operationally hard to get to work (collaborative infra and vehicles), which might be the reason why the cooperation dataset is fairly small. Related to this, could the authors comment on the sensitivity of the method to the accuracy of synchronisation and relative point estimation between the vehicle and the infra point cloud? How is the relative pose computed?\nThe paper needs a more in depth overview of related work on two fronts: use of task-relevant losses as part of the self-supervised learning, and use of shape descriptors like the one in Sec. 3.3 in the context of learning from point clouds.\nMinor details for the authors for improvements in the next version of the paper:\n\nI would recommend to bring Table 7 from the Appendix into the main paper\nIt would be useful to add some rationale or ablative analysis on the choice of some parameters (e.g. the radii of the shape descriptors (Sec. 3.3)\nIt is hard to understand/see the negative and positive pairs in Fig. 3\nThe math notation is a bit cumbersome. For example, it took me a while to understand the v/f notation\nI have some questions on invariance to rotations: is the shape descriptor rotationally invariant? If yes, how does it work? Or it does not matter because the infra and vehicle point clouds are aligned first? If the point clouds are aligned, how do methods learn about rotational invariance: are there such kind of augmentations in the supervised learning stage?\nSome statements are fairly strong, such as: \"This stems from the sparsity of LiDAR point clouds, which sometimes make it difficult to find good positive pairs to perform contrastive learning\". Unless there is some strong evidence backing this claim, I would recommend it softening, e.g. \"We believe this stems from the sparsity...\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "V7RTUQu8zPp", "reviewer": "Reviewer_BVa4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Cooperative Contrastive Learning and Contextual Shape Prediction (CO3). It features 1) using vehicle-infrastructure-cooperation dataset to obtain proper views for unsupervised contrastive learning; 2) a shape pretext for enforcing the learning to be task-relevant; and 3) the learned representation can be generalized to other autonomous driving datasets. \n\nThe proposed CO3 has been evaluated over the ONCE, KITTI and nuScenes datasets. Results show that it is effective in boosting detection performance.  \n", "summary_of_the_review": "I like the idea of using vehicle-infrastructure dataset for contrastive learning. But I have some concerns that the improvement on object detection is not significant. ", "strength_and_weaknesses": "\nStrength\n1. The paper is well motivated and clearly explained. \n2. The idea of using vehicle-infrastructure-cooperation dataset for unsupervised contrastive learning is interesting.\n3. Experiments are extensive. Results on semantic segmentation are good.\n\nWeakness\n1. The overall improvement by the proposed CO3 is marginal for object detection. For example, the PV-RCNN results on the KITTI and ONCE datasets. If we have a larger supervised training set, will unsupervised contrastive learning still be helpful? \n2. In ablation study Table 6, the results on KITTI need more explanation. What is the reason that using either of proposed contextual shape prediction and cooperative contrastive alone gives worse or marginally better performance than the random baseline but the combined method CO3 is better than the baseline?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "1VhJjKt6ie", "reviewer": "Reviewer_XpxU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on self-supervised point cloud representation learning. The authors combine the contrastive loss with KL divergence between the predicted feature and manually crafted 3D shape context. \n\nTo construct better views for contrastive loss,  the authors utilize a recently proposed dataset where point clouds are captured from both vehicle sensors and infrastructure sensors. \n\nTo demonstrate the effectiveness of the proposed methods, the authors finetuned the pre-trained model on  3D object detection and segmentation tasks and have achieved better performance than other self-supervised learning methods or even supervised pre-trained models.", "summary_of_the_review": "Although the performance gain is significant, I prefer to reject the paper because it resolves the (positive) view construction of 3D point clouds by using an existing dataset, and the novelty and insights of the method itself are insufficient.\n\n\n[post-rebuttal] I appreciate the efforts the authors did on proving the effectiveness of the proposed method on indoor data. Although I maintain my original opinion that the paper is more about construct data on a very specific dataset and is less inspiring for further researches in general case, I updated the rating due to the additional effort the authors have made.", "strength_and_weaknesses": "Strength\n+ The motivation of construct views from infrastructure sensors and vehicle sensors is well introduced.\n+ The performance gain is significant on various downstream benchmarks.\n+ The evaluation is extensive.\n\nWeakness:\n+ The paper converts the view construction of outdoor scene to a case similar to indoor by using the new dataset. So the proposed solution should also works for indoor, but which is not demonstrated in the paper.\n+ The novelty and the insight of the method itself is limited.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster