Pre-training LiDAR-based 3D Object Detectors through Colorization
Authors: Tai-Yu Pan
Chenyang Ma
Tianle Chen
Cheng Perng Phoo
Katie Z Luo
Yurong You
Mark Campbell
Kilian Q Weinberger
Bharath Hariharan
Wei-Lun Chao
Venue: ICLR 2024 poster (2024)
Keywords: 3D object detection
LiDAR point cloud
pre-training
autonomous driving
self-supervised learning
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Accurate 3D object detection and understanding for self-driving cars heavily relies on LiDAR point clouds, necessitating large amounts of labeled data to train. In this work, we introduce an innovative pre-training approach, Grounded Point Colorization (GPC), to bridge the gap between data and labels by teaching the model to colorize LiDAR point clouds, equipping it with valuable semantic cues. To tackle challenges arising from color variations and selection bias, we incorporate color as "context" by providing ground-truth colors as hints during colorization.
Experimental results on the KITTI and Waymo datasets demonstrate GPC's remarkable effectiveness. Even with limited labeled data, GPC significantly improves fine-tuning performance; notably, on just 20% of the KITTI dataset, GPC outperforms training from scratch with the entire dataset. 
In sum, we introduce a fresh perspective on pre-training for 3D object detection, aligning the objective with the model's intended role and ultimately advancing the accuracy and efficiency of 3D object detection for autonomous vehicles.

Published as a conference paper at ICLR 2024
PRE-TRAINING LIDAR-BASED 3D OBJECT DETECTORS THROUGH COLORIZATION
Tai-Yu Yu Pan', Chenyang Ma', Tianle Chen', Cheng Perng Phoo", Katie Z Luo", Yurong You", Mark Campbell', Kilian Q. Weinberger'. Bharath Hariharan" and Wni Chao' 'The Ohio State University, Columbus, OH CCrnell University, Ithaca, NY
ABSTRACT
Accurate 3D object detection and understanding for self-driving cars heavily re- lies on LiDAR point clouds, necessitating large amounts of labeled data train. In this work, we introduce an innovative pre-training approach, Grounded Point Colorization (GPC), bridge the gap between data and labels by teaching the model to colorize LiDAR point clouds, equipping it with valuable semantic cues. To tackle challenges arising from color variations and selection bias, we incor- porate color "context" by providing ground-truth colors as hints during col- orization. Experimental results on the KITTI and Waymo datasets demonstrate GPC's remarkable effectiveness. Even with limited labeled data, GPC signif- icantly improves fine-tuning performance; notably, on just 20% of the KITTI dataset, GPC outperforms training from scratch with the entire dataset. In sum, we introduce a fresh perspective on pre-training for 3D object detection, aligning the objective with the model's intended role and ultimately advancing the accuracy and efficiency of 3D object detection for autonomous vehicles. Code available: https://githus www wwwp:////////CCCc///
INTRODUCTION
Detecting objects such as vehicles and pedestrians in 3D is crucial for self- delf-driving cars to operate safely. Mainstream 3D object detectors (Shi et al.] [2019] 2020b| Zhu et al., [2020) He et al.] (2020a) take LiDAR point clouds input, which provide precise 3D signals of the surrounding environment. However, training a detector needs a lot of labeled data. The expensive process of curating annotated data has motivated the community to investigate model pre-training using unlabeled data that can be
collected easily, Most of the existing pre- treiirainng methods are built upon contrastive learning (Yin) et al. 2022 Xie et al. 2020 Zhang et al. 2021 Huang et al. 2021 Liang et al. 2021), inspired by its success in 2D recognition (Chen et al. 2020a He et al. 2020b). The key novelties, however, are often limited to how the positive and negative data pairs constructed. This paper attempts to go beyond contrastive learning by providing a new perspective on pre-training 3D object detectors.
We rethink pre-training's role in how it could facilitate the downstream fine-tuning with labeled data. A labeled example of 3D object detection comprises point cloud and a set of bounding boxes. While the format typical, there is no explicit connection between the input data and labels. We argue that a pre-training approach should help bridge these two pieces of information to facilitate fine-tuning.
How can we go from point clouds bounding boxes without human supervision?? ? One intuitive way is to first segment the point cloud, followed by rules to read out the bounding box of each segment. The closer the segments match the objects of interest, the higher their qualities are. While a raw LiDAR point cloud may not provide clear semantic cues to segment objects, the color image collected along with it does. In color images, each object instance often possesses coherent color and has a sharp contrast to the background. This information has been widely used in image segmentation (Li) & Chen 2015; Liu et al., 2011 I Adams & Bischoi 1994) before machine learning- take over. We hypothesize that by learning to predict the pixel color each LiDAR point is projected to, the pre-trained LiDAR-
methods
model backbone will be equipped with the semantic cues that facilitate the subsequent fine-tuning for 3D object detection. To this end, we propose pre- traiirinn LiDAR- based 3D object detector by learning to colorize LiDAR point clouds. Such an objective makes the key strength. Taking models that
pre-training procedure fairly straightforward, which

---

Published as a conference paper at ICLR 2024
Figure I: Illustration of the proposed Grounded Point Colorization (GPC), We pre-train the detector backhone by colorization (left right), taking the hints seed set of points (middle) overcome the inherent color variation. We show that the middle step is the key to learning the semantic cues of objects from colors.
use point- baaed backbones like PointNet++ QQi et al. 2017b a) as an example, pre-training becomes solving point-wise regression problem (for predicting the RGB values).
That said, the ground-truth colors of LiDAR point cloud contain inherent variations. For example, the same car model (hence the same point cloud) can have multiple color options; the same street scene (hence the same point cloud) can be pictured with different colors depending on the time and weather. When selection bias occurs in the collected data (e.g., only few instances of the variations are collected), the model may learn to capture the bias, degrading its transferability. When the data is well collected to cover all variations without selection bias, the model may learn to predict the average color, downgrading the semantic cues (e.g., contrast). Such a dilemma seems to paint a grim picture of using colors to supervise pre- pre-training. In fact, the second issue is reminiscent of the "noise" term in the well-known bias-variance decomposition (Hastie et al. 2009). which cannot be reduced by machine learning algorithms. The solution must come from better way use the data.
Based on this insight, we propose a fairly simple but effective refinement to how we leverage color. Instead of using color solely as the supervised signal, we use it as "context" to ground the colorization process on the inherent color variation. We realize this idea by providing ground-truth colors to a seed set of LiDAR points as "hints" for colorization. (Fig. T gives illustration.) On the one hand, this contextual information reduces the color variations on the remaining points and. in turn, increases the mutual information between the remaining points and the ground-truth colors supervise pre- training. On the other hand, when selection bias exists, this contextual information directly captures it and offers as a "shortcut" to the colorization process. This could reduce the tendency of a model to (rer) this bias (Lu, 2022 Clark et al. 2019 Cadene et al. 2019), enabling it to focus on the semantic cues useful for 3D object detection. After all, what we hope the model learns is not the exact color of each point but which subset of points should possess the same color and be segmented together. By providing the ground-truth truth colors to a seed set of points during the colorization process, concentrate the model backbone on discovering with which points each seed point should share its color, aligning the pre-training objective to its desired role.
We implement our idea, which we name Grounded Point Colorization (GPC), by introducing another point-based model (e.g., PointNet++) on top of the detector backbone. This could be seen as the projection head commonly added in contrastive learning (Chen et al., 2020a): Fig. 2 provides an illustration. While the detector backbone takes only the LiDAR point cloud as input, the projection head takes both the output embeddings of the LiDAR points and the ground tround-truth colors of a seed set of points as input and is in charge of filling in the missing colors grounded on the seed points. This leaves the detector backbone with a more dedicated task producing output embeddings with clear semantic cues about which subset of points should be colored (i.e., segmented) together.
We conduct extensive experiments to validate GPC, using the KITTI (Geiger et al.] 2012) and Waymo (Sun et al.] (020) datasets. We show that GPC can effectively boost the fine-tuning perfor- mance, especially on the labeled-data-scarcee settings. For example, with GPC, fine-tuning on 20% KITTI is already better than training 100% from scratch. Fine-tuning using the whole dataset still brings a notable 1.2% gain on overall AP, outperforming existing basclines.
Our contributions are three-fold. First, we propose pre-training by colorizing the LiDAR points novel approach different from the commonly used contrastive learning methods. Second, we identify the intrinsic difficulty of using color as the supervised signal and propose simple yet well-founded and effective treatment. Third, we conduct extensive empirical studies to validate our approach.
RELATED WORK
LiDAR-bases 3D object detectors. Existing methods can be roughly categorized into two groups
based their feature extractors. Point-based methods (Shi et al. 2019 Zhhan et al. 2022b Qi
2

---

Published as a conference paper at ICLR 2024
et al. 2019 Fan al. 2022 Yang et al. 2020) extract features for each LiDAR point. Voxel-basee
methods (Shi et al. 2020a] Yin et al. 2022 1 Yan et al. 2018 Lang et al. 2019) extract features on grid locations, resulting in feature tensor. In this paper, we focus on point- based 3D object detectors.
SSL on point clouds by contrastive learning. Self-supervised learning (SSL) seeks to build feature representations without human annotations. The main branch of work is based contrastive learning (Chen et al. [2020a] [Chen & He 2021; He et al. 2020b] CCen et al. 2020b] Grill et al. 2020 Henaff
2020 Tian et al. 2020). For example. Zhang al. (2021); Huang et al. (202)) treat each point cloud as an instance and learn the representation using contrastive learning. Liang et al. (202 Xie et al.
(2020) Yin et al. (2022) build point features by contrasting them across different regions different
scenes, The challenge is how to construct data pairs to contrast, which often needs extra information or ad hoc design such as spatial-temporal information (Huang et al.. 202T) and ground planes (Yin)
et al. 2022). Our approach goes beyond contrastive learning for SSL on point clouds.
SSL on point clouds by reconstruction. Inspired by natural languages processing (Devlin et al.
2018 Raffel et al., 22200) and image understanding (Zhou et al., 2021 Assran et al., [022] Larsson
ct al.
2017; Pathak et al. 2016), this line of work seeks to build feature representations by filling up
missing data components (Wang et al. [021 Pang et al.) 2022: Yu et al. 2022 Liu et al. 2022 Zhang
cet al.
2022a Eckart et al. 2021). However, most methods are limited to object-level reconstruction
with synthetic data, e.g., ShapeNet55 (Chang al. 2015) and ModelNet40 (Wu et al.
2015). Only
few recent methods investigate outdoor driving scenes (Min et al. 2022 Lin & Wang. 2002). using
the idea of masked autoencoders (MAH) (He et al.) (022). Our approach is related to reconstruction but with fundamental difference - we do not reconstruct the input data (i.e., LiDAR data) but the data from an associated modality (i.e.. color images). This enables us to bypass the mask design and leverage semantic cues in pixels, leading to features that facilitate 3D detection in outdoor scenes.
Weak//elf-superviis from multiple modalities. Learning representations from multiple modalities has been explored in different contexts. One prominent example is leveraging large-scale Internet text such as captions supervise image representations (Radford et al.] 2021 Singh et al. 2022 Driess
et al. 2023). In this paper, we use images to supervise LiDAR based representations. Several recent
methods also use images but mainly to support contrastive learning (Sautier et al. 2022] [anda et al.,
2023 Liu et al. 2222 2020 Hou al., 202 Afham et al.] [022] Pang et al. 2023). Concretely, they form data pairs across modalities and learn two branches of feature extractors (one for each modality) to pull positive data pairs closer and push negative data pairs apart. In contrast,
work takes
approach. Our architecture is a single branch without pair construction.
Colorization. Image colorization has been active research area in computer vision (Levin et al. 2004| Zhang et al., 2016). predicting colorful versions of grayscale images. In our paper, we introduce colorization the pre-training task equip the point cloud representation with semantic cues useful for the downstream 3D LiDAR R-DeR- detectors. We provide more reviews in Appendix ST
3 PROPOSED APPROACH: GROUNDED POINT COLORIZATION (GPC)
Mainstream 3D object detectors take LiDAR point clouds as input. To localize and categorize objects accurately, the detector must produce internal feature representation that can properly encode the semantic relationship among points. In this section, we introduce Grounded Point Colorization (GPC), which leverages color images supervised signals to pre-train the detector's feature extractor (i.e., backbone), enabling it to output semantically meaningful features without human supervision.
3.1 PRE-TRAINING VIA LEARNING TO COLORIZE
Color images can be easily obtained in autonomous driving scenarios since most sensor suites involve color cameras. With well-calibrated and synchronized cameras and LiDAR sensors, we can get the color of LiDAR point by projecting it to the 2D image coordinate. As colors tend to be similar within object instance but sharp across object boundaries, we postulate that by learning to colorize the point cloud, the resulting backbone would be equipped with the knowledge of objects.
The first attempt. Let us denote point cloud by X = [ .NN] E R the corresponding
color information by Y = [y1.... y E R3xN The three dimensions in point a, encode the 3D location in the ego-car coordinate system; the three dimensions in yn encode the corresponding RGB pixel values. With this information, we can frame the pre- pre-training of a detector backbone as a supervised colorization problem, treating each pair of (X, Y) labeled example.
3

---

Published as a conference paper at ICLR 2024
deco
hints
concal
detecha PreB
backbone
feature
pre-training
fine tuning
Architecture of GPC. The key insight is grounding the pre-training colorization process the hints, allowing the model backbone to focus learning semantically meaningful representations that indicate which subsets (i.e., segments) of points should be colored similarly facilitate downstream 3D object detection.
Figure 2:
Taking a point-based backbone fe like PointNet++ (Qi et al. 2017b) as an example, the model takes X as imput and outputs a feature matrix E PDN in which each column vector &n E RP is meant to capture semantic cues for the corresponding point x, facilitate 3D object detection. One way to pre-train So using (X, Y) is to add another point-based model d6 (seen as the projection head or color decoder) on top of fe, which takes Ž as input and output Y of the same shape as Y. In this way, Y can be compared with Y to derive the loss, for example, the Frobenius norm. Putting things together, we come up with the following optimization problem.
min 0.0
Σ
11 o fo(() Y I| min
Σ
Σ
y
0.4
()
(X
where Duran means the pre-training dataset and Un the n-th column of Y 9 o fo(
The inherent color variation. At first glance, Eq. should be fairly straightforward to optimize using standard optimizers like stochastic gradient descent (SGD). In our experiments, we however found a drastically slow drop in training error, which results from the inherent color variation. That is, given X, there a huge variation of Y in the collected data due to environmental factors (e.g., when the image is taken) or color options (e.g., instances of the same object can have different colors). This implics a huge entropy in the conditional distribution P(Y x), í.e., a large H =
E|- log P(Y|X) ]. indicating small mutual information between X and Y. i.e., small / ((X;Y)
HY) X In other words, Eq. may not provide sufficient information for fo to learn.
Another way to look this issue is the well rell-known variance decomposition (Hastic et al. 2009
Bishop Nasrabadi 2006) in regression problems (suppose Y E I), which contains a "noise" term,
Noise: Ex [Y - x [T])
When X has high entropy, the noise term will be large, which explains the poor training error. Importantly, this noise term cannot be reduced solely by adopting a better model architecture. It can only be resolved by changing the data, or more specifically, how we leverage (X,Y).
One intuitive way to reduce H(Y) X) is to manipulate P(Y|X). e.g., to carefully sub-sample data to reduce the variation. However, this would adversely reduce the amount of training data and also lead to side effect of selection bias (Cawley & Talbot 2010). In the extreme case, the pre-trained model may simply memorize the color of each distinct point cloud via the color decoder Iø- preventing the backbone fe from learning a useful embedding space for the downstream task.
Grounded colorization with hints. To resolve these issues, we propose reduce the variance in P(Y|X) by providing additional conditioning evidence S. From the perspective of information theory (Cover 1999). adding evidence into XXY, i.e., P(Y|X, S), guarantees,
H(Y X) 2 H(Y X, S),
and thus I((X,)) Y) 2 I(X:Y),
unveiling more information for fe to learn
We realize this idea by proving ground-truth colors to seed set of points in X. Without loss of generality, we decompose X into two subsets Xs and X such that X = [Ks, Xu.. Likewise, decompose Y into Ys and Yo such that Y (Ys, Yu). We then propose inject Ys as the conditioning evidence S into the colorization process. In other words, we directly provide the ground-truth colors some of the points the model initially has to predict.
At first glance, this may controversially simplify the pre-training objective in Eq.. However, as the payback, we argue that this hint would help ground the colorization process the inherent color
4

---

Published as a conference paper at ICLR 2024
original image
128 colors
32 colors
Figure 3: Color quantization. We apply K- Means algorithm to cluster RGB pixel values into discrete bins. The resulting image with 128 bins is hardly distinguishable from the original image by human eyes.
variation, enabling the model to learn useful semantic cues for downstream 3D object detection. Concretely, to colorize a point cloud, the model needs to a) identify the segments that should possess similar colors and b) decide what colors to put on. With the hint that directly colors some of the points, the model can concentrate more on the first task, which aligns with the downstream task to identify which subset of points to segment together.
Sketch of implementation. We implement our idea by injecting Ys as additional input to the color decoder DΦ- We achieve this by concatenating Ys row-wise with the corresponding columns in Z, resulting in a (D + di-dimensi input to Ja- Let Oy denote an all-zero matrix with the same shape as Yu, we modify Eq. I as.
Σ ()
I (( Ys, Ys:
[Ys, Yoil s.t. 2 = fo ([ Xul),
min 0,0
where "." denotes row-wise concatenation. Importantly, [Eq. [Eq. 4jonly provides the hint Yg to the color decoder Da. leaving the input and output formats of the backbone model fo intact. We name our approach Grounded Point Colorization (GPC).
GPC could mitigate selection bias. Please see Appendix S2 for a discussion.
3.2 DETAILS OF GROUNDED POINT COLORIZATION (GPC)
Model architecture. For the backbone fo. we use the model architectures defined in existing 3D object detectors. We specifically focus on detectors that use point-based feature extractors, such as PointRCNN (Shi et al., 2019) and IA-SSD (Zhang et al. 2022b). The output of So a set of point-wise features denoted by Z. For the color decoder 96 that predicts the color of each input point, without loss of generality, use PointNet++ (Qi al. 2017b).
Color regression vs. color classification. In Sec. 3.1 we frame colorization as a regression problem. However, regressing the RGB pixel values is sensitive potential outlier pixels. In our implementa- tion, we instead consider a classification problem by quantizing real RGB values into discrete bins. We apply the K-means algorithm over pixels of the pre-training images to cluster them into K classes. The label of each LiDAR point 2,, then becomes Yn E {1,.... K). Fig. 3 shows the images with color quantization; each pixel is colored by the assigned cluster center. is hard to distinguish the images with K = 128 bins from the original image by human eyes.
We accordingly modify the color decoder GΦ. In the input, we concatenate a K-dimensional vector to the feature Σ, RDD of each LiDAR point ah resulting in (D -)ddimension input Jø- For seed points where we inject ground-truth colors, the K -dimensioonl vector is one-hot, whose yn element one and zero elsewhere. For other LiDAR points, the K-ddimensional vector is zero vector. In the output, IΦ generates K -dddimnniional logit vector ń). E IR for each input point nn. The predicted color class In can be obtained by arg max 9n [c].
Pre-training objective. The cross-entropy loss is the standard loss for classification. However, in driving scenes, most of the LiDAR points are on the ground, leading to an imbalanced distribution across color classes. We thus apply the Balanced Softmax (Ren et 2020 to re-weight each class,
Oyy X A..
E(E,, yn) = log
(5)
EK Ore efm(co]
where ae is the balance factor of class c. In the original paper, ae is proportional to the number of training examples of class Here, set a per matih, proportional to the number of points with class label c. A small e is added to ag for stability since an image may not contain all colors.
Optimization. In the pre- mre-training stage, we train fe and 9ø end-to-end without human supervision. In the fine-tuning stage with labeled 3D object detection data, I6 is disregarded; the pre- pre-trained fo is fine-tuned end-to-end with other components in the 3DD object detector.
5

---

Published as a conference paper at ICLR 2024
Table I: Pre-trained on KITTI and fine-tuned on subsets of KITTI. GPC is consistently better than scratch and PointContrast (Xie et al. 2020 especially when labeled data are extremely sparse.
Sub. Method
Detector
Car
Podestrian
Cyclist mod.
Overall
easy
mod.
hard
easy
mod.
hard
easy
hard
easy
mod.
hard
scratch
PointRCNN
66.8
51.6
45.0
61.5
55.4
48.3
81.4
58.6
54.5
69.9
55.2
49.3
PointCoutrast
PaintRNN
58.2
46.7
40.1
$2.5
47.5
42.0
73.1
49.7
46.6
61.2
47.9
42.9
5%
GPC (Ours)
PointRCNN
81.
65.1
57.7
63.7
57.4
50.3
88.2
65.7
61.4
77.7
62.7
56.7
scratch
IA-SDD
73.7
61.6
55.3
41.2
32.0
29.2
75.1
54.2
51.1
63.3
49.3
45.2
GPC (Ours)
IA-SSD
81.3
66.6
59.7
49.2
40.5
36.7
77.3
54.7
51.2
69.3
53.9
49.2
scratch
PoinRCNNN
87.6
74.2
71.1
68.2
60.3
52.9
93.0
71.8
67.2
82.9
68.8
63.7
PointCoutrast
PoinRCNN
88.4
73.9
70.7
71.0
63.6
56.3
86.8
69.1
64.1
82.1
68.9
63.7
20%
GPC (Ours)
PointRNN
88.7
76.0
72.0
70.6
63.0
55.8
92.6
73.0
68.1
84.0
70.7
65.3
scratch
SS-
86.1
74.0
70.6
52.2
45.6
40.1
84.5
64.0
59.8
74.3
61.2
56.8
GPC (Ours)
IA-SSD
87.6
76.4
71.5
54.1
47.3
42.0
86.2
69.5
64.9
76.0
64.4
59.5
scratch
PoinRRCNN
89.7
77.4
74.9
68.9
61.3
54.2
88.4 4
69.2
64.4
82.3
69.3
64.5
PointCoutrast
PoinRCNN
89.1
77.3
74.8
69.2
62.0
55.1
90.4
70.8
66.5
82.9
70.0
65.4
50%
GPC (Ours)
PointRNN
89.3
77.5
75.0
68.4
61.4
54.7
92.1
72.1
67.6
83.3
70.3
65.8
scratch
IAD
88.6
77.6
74.2
61.8
55.0
49.0
89.9
70.0
65.6
80.1
67.5
63.0
GPC (Ours)
IA-SSD
89.3
80.1
76.9
62.5
55.4
49.6
88.8
73.7
69.2
80.2
69.7
65.2
scratch
PoinRRCNN
90.8
79.9
77.7
68.1
61.1
54.2
87.3
68.1
63.7
82.0
69.7
65.2
PointContrast
PointRCN
91.1
80.0
77.5
66.9
61.1
54.7
91.3
70.7
66.4
83.1
70.6
66.2
100%
GPC (Ours)
PointRCNN
90.8
79.6
77.4
68.5
61.3
54.7
92.3
71.8
67.1
83.9
70.9
66.4
scratch
ISD
89.3
80.5
77.4
60.1
52.9
46.7
87.8
69.7
65.2
79.1
67.7
63.1
GPC (Ours)
IA-SSD
89.3
82.5
79.6
60.2
54.9
49.0
88.6
71.3
67.2
79.3
69.6
65.3
All pieces together. We propose a novel self-supervised learning algorithm GPC, which leverages colors to pre-train LiDAR-based 3D detectors. The model includes a backbone fe and a color decoder I6i fo learns to generate embeddings to segment LiDAR points while 96 learns to colorize LiDAR points. To mitigate the inherent color variation, GPC injects hints into Jø- To overcome the potential outlier pixels. GPC quantizes colors into class labels and treats colorization as a classification problem. Fig. I shows our main insight and Fig. 2 shows the architecture.
some LiDeRR-baed detectors encode LIDAR points into voxel-based feature representation. We investigate two ways of extension to voxel-based detectors. First, recent advancements, such as PV-RCNN (Shi al., 2020a). have incorporated a point-based branch alongside the voxel-based backbone. This design allows GPC to still be applicable, the color decoder P6 can be applied to the point branch, enabling the pre-training signal to be backpropagated to the voxel-based backbone. The pre- pre-trained backbone can then be utilized in any voxel-based detectors (see (Sec. 4.2). Second, we explore enriching input data by augmenting LiDAR point coordinates with output features from the pre-trained point-based backbone fo: which also yields promising results (see Appendix S4 for the results and discussion).
Extension to voxel-based detectors. As discussed Sec. 2,
4 EXPERIMENTS
4.1 SETUP
Datasets. We use two commonly used datasets in autonomous driving: KITTI (Geiger et al. 2012) and Waymo Open Dataset (Sun et al. 2020). KITTI contains 7, 481 annotated samples, with 3, 712 for training and 3, 769 for validation. Waymo dataset is much larger, with 158, 081 frames in 798 scenes for training and 40, 077 frames in 202 scenes for validation. We pre-train on each dataset with all training examples without using any labels in the pre-training. We then fine-tune on different amounts of KITTI data, 5%, 10%, 20%, 50%, and 100%, with 3D annotations to investigate data- efficient learning. We adopt standard evaluation on KITTI validation set. Experiments of KITTI + KITTI show the in-domain performance, Waymo- + KITTI provides the scalability and transfer ability of the algorithm, which is the general interest in SSL.
3D Detectors. As mentioned in Sec. 3 we focus on learning effective point representation with PointRCNN and IA-SSD. We see consistent improvement on both (Sec. 4.2). Additionally, extend GPC to Voxel-based detectors as discussed in Sec. 3.2 Sec. 4.2 and Appendix S4
Implementation. We sample images from the training set (3k for KITTI and 10k for Waymo) and 1k pixels for each image to learn the quantization. K-Means algorithm is adopted to cluster the colors into 128 discrete labels which are used for seeds and ground truths. The color labels in one hot concatenated with the features from the backbone are masked out 80% of colors (i.e., assigned to zeros). For 3D point augmentation, follow the standard procedure, random flipping along
6

---

Published as a conference paper at ICLR 2024
Table 2: Pre-trained on Waymo and fine-tuned on subsets of KITTL We compare with DepthContrast Zhang al.
2021
and ProposalContrast Yin et al. 2022 "directly taken from Yin et al. 2022
Pedestrian
Cyclist
Overall
Sah.
Method
Detector
easy
mod.
hard
easy
mod.
hard
easy
mod.
hard
easy
mod.
hard
scratch
PointRCNN
66.8
51.6
45.0
61.5
55.4
48.3
81.
58.6
54.5
69.9
55.2
49,3
GPC (Ours)
PoinRCNN
83,9
67.8
58.1
51.5
88.3
65.9
63,9
.5.55
5%
scratch
TASSSD
75.7
66.6
553
4C2
320
292
75.
34.1
51.1
63.3
39.3
"43.2"
GPC (Ours)
IA-SSD
81.5
67.4
60.1
47.2
39.5
34.5
81.2
61.4
57.4
70.0
56.1
50.7
scratch'
PointRCNN
88.6
75.2
72.5
55.5
48.9
42.2
85.4
66.4
71.7
63.5
ProposalContrass
PointRCNN
88.5
77.0
72.6
58.7
51.9
45.0
90.3
69.7
65.1
66.2
20%
GPC (Ours)
PointRCNN
90.9
76.6
73.5
70.5
62.8
55.5
89.9
71.0
66.6
83.8
70.1
65.2
scratch
TA-SSD
86.1
74.0
70.6
52.2
45.6
40.1
84.5
64.0
59.8
74.3
61.2
"56.8
GPC (Ours)
IA-SSD
83.1
76.8
50.9
44.6
86.1
64.6
59.5
scratch!
PointRCNN
89.1
77.9
75.4
61.8
54.6
47.9
86.3
67.8
63.3
66.7
ProposalContrast'
PointRCNN
89.3
80.0
77.4
62.2
54.5
46.5
92.3
73.3
68.5
69.2
50%
GPC (Ours)
PointRCNN
89.5
79.2
77.0
69.3
61.8
54.6
89.1
71.3
66.4
82.7
70.7 66.0
scralch
IA-SSD
88.6
77.6
74.2
61.8
55.0
49.0
89.9
70.0
65.6
80.1
67.5
'63.
GPC (Ours)
IA-SSD
88.4
79.8
76.4
63.4
56.2
49.6
88.3
70.5
66.0
80.1
68.9
64.0
scratch!
PointRCNN
90.0
80.6
78.0
62.6
55.7
48.7
89.9
72.1
67.5
69.5
DepdiContrast
PointRCNN
89.4
80.3
77.9
65.6
57.6
51.0
90.5
72.8
68.2
70.3
ProposalContrant
PointRCNN
89.5
80.2
78.0
66.2
58.8
52.0
91.3
73.1
70.7
100%
GPC (Ours)
PointRCNN
89.1
79.5
71.2
67.9
61.5
55.0
89.1
71.0
67.0
82.0
70.7 66.4
scratch
IA-SSD
89.3
80.5
71.4
60.1
52.9
46.7
87,8
69.7
65.2
79.1
67.7
63.1
GPC (Ours)
IA SSD
90.7
82.4
79.5
60.7
55.0
49.5
86.5
70.5
67.1
79.3
69.3
65.4
Table 3: GPC PV-RCNN (hi et al. 2020a) and SOTA comparison. The result shows that GPC can be applied on voxel-based detectors and it consistently performs better than existing methods.
Sub Mothod
Car
Pod.
Cyc.
Overall
scratch
82.52
53.33
64.28
66.71
20%
ProposalContrast Yin et al
2023
82.65
55.05
66.68
68.13
PanchContrast (Shrout GPC (Ours)
82.63
57.77
71.84
70.75
82.65
59.87
71.59
71.37
seratchh
82.68
57.10
69.12
69.63
50%
PreposalContrast Yin et al
2022
82.92
59.92
72.45
71.76
PatchContrast (Shrout of GPC (Ours)
2046
84.47
60.76
71.94
72.39
84.58
61.06
72.61
72.75
scratch
84.50
57.06
70.14
70.57
GCC-3D (Liane et al_ 2021
71.26
STRL [Hhing or R0M1
84.70
57,80
71 88
71.46
100%
PoiniContrar (X12
PruposalConiras Yin c m 2022
2020
84.72
84.18
60.36
57.74
72.72
73.69
71.55
72.92
ALSO (Boulch
84.68
60.16
74.04
72.96
PatchContrast (Shrout
2023
84.67
59.92
74.33
72.97
GPC (Ours)
84.68
62.06
73.99
the X-axis, random rotation ((4/TT, 4/7)), random scaling (00.95, 1.05)). random point sampling (N 16384 with range= 40), and random point shuffling. We pre-train the model for 80 and 12 epochs on KITTI and Waymo respectively with batch sizes 16 and 32. We fine-tune on subsets of KITTI for 80 epochs with batch size of 16 and the same 3D point augmentation above. The Adam & Hutter 2016) are adopted both stages which the maximum learning rate is 0.001 and 0.0004 respectively.
optimizer (Loshchilov & Hutter (017) and Cosine learning rate scheduler(Loshchie
4.2 RESULTS
Results on KITTI - KITTI. We first study in-domain data-efficient learning with different amounts of labeled data available. Our results outperform random initialization (scratch) by a large margin, particularly when dealing with limited human-annotated data, as shown in Tab. I For instance, with PoinRRNN,, on 5% KITTI, we improve +7.5% on overall moderate AP (55.2 2 to 7). Remarkably, even with just 20% KITTI, we already surpass the performance achieved by training on the entire dataset from scratch, reaching a performance of 70.7. This highlights the benefit of GPC in data- efficient learning, especially when acquiring 3D annotations is expensive. Moreover, even when fine-tuning on 100% of the data, we still achieve a notable gain, improving from 69.7 to 70.9 (+1.2). We re-implement recent method, PointContrast (Xie et al.) (020). as stronger baseline. GPC performs better across all subsets of data. Notice that PointContrast struggles when annotations are extremely scarce, as seen on 5% subset where it underperforms the baseline (47.9 v5.55.2).
Results on Waymo- + KITTL. We investigate the transfer learning ability on GPC as it's the general interest in SSL. We pre-train Waymo (20x larger than KITTI) and fine-tune subsets
7

---

Published as a conference paper at ICLR 2024
Table 4: Comparison with methods using images. PPKT [Lu et al.
2021).
SLidR (Sautier et al.
2022), and
TriCC (Pang al. 2023 pre-train on NuScenes (Caesar al. 2020) with UNet (Choy al. 2019
The results
shown in moderate AP. ddirectly taken from Pang et al. 2023).
Method
Arch
Pre. Dataset
5%
10%
20%
scratch'
UNet PoimRRCNN
56.6
58.8
63.7
PPKTT
UNet PointRCNN
NuScenes
59.6 (+3.0) 63.7
(+4.9) 64.8 (+1.1)
SLidRi
UNet PuintRCNN
NuScenes
58.8 (+2.2)
63.5
(+4.7)
63.8 (+0.1)
TriCC!
UNet + PointRCNN
NuScenes
61.3 (+4.7) 64.6
(+5.8)
65.9 (+2.2)
scrach
PointRCNN
55.2
65.2
68.8
GPC (Ours)
PointRCNN
KITTI
62.7 (+7.5)
66.4
(+1.2)
70.7
(+1.9)
GPC (Ours)
PoinRCNN
Waymo
63.9 (+8.7)
67.0 (+1.8)
70.1 (+1.3)
Table 5: Loss ablation for colorization. We explore two regression losses Mean Squared Error (MSE), SmoothL1 (SL1) and two classification losses CrossEntropy (CE), modified Balanced Softmax (BS). The results are on KITTI 5% KITTI. The best result is reported for each loss after a hyperparameter search.
Car
Pedestrian
Cyclist
Overall
Loss
Type
easy
mod.
hard
casy
mod.
hard
easy
mod.
hard
easy
mod.
hard
scratch
66.8
51.6
45.0
61.5
55.4
48.3
81.4
58.6
54.5
69.9
55.2
49,3
reg.
72.4
52.9
64.2
49.2
86.2
74.2
60.1
SLI
Teg
76.8
62.9
56.0
63.2
55.6
49.2
84.1
63.3
58.9
74.7
60.6
54.7
CE
cls
78.4
64.5
58.5
60.7
54.1
47.6
85.5
61.5
57.6
74.9
60.0
54.6
BS
81.2
65.1
57.7
63.7
57.4
50.3
88.2
65.7
61.4
77.7
62.7
56.5
of KITTI. In Tab. 2 our method shows consistent improvement upon scratch baseline. The overall moderate AP improves 55.2 + 63.9 on 5% KITTI. On 20%, it improves 63.5 - 70.1 which is again already better than 100% from scratch (69.5). Compared to the current state-of-the-ard (SOTA), ProposalContrast Yin al. 2022). GPC outperforms on 20% (66.2 vs. 10.1) and 50% (69.2 vs. 70.7).
Extension to Voxel-Based Detectors. We extend the applicability of GPC to PV-RCNN (Shi et al. 2020a) for Waymo - KITTI. We randomly select 16, 384 key points, whose features are interpolated from voxelized features, for colorization supervision (see/Sec. 3.2). Note that the learned voxelization backbone can be applied to any voxel-based detectors, not limited to PV-RCNN, As shown in Tab. 3) GPC consistently outperforms existing algorithms. The existing state-of-ther of- (SOTA) algorithms primarily utilize contrastive learning which requires specific designs to identify meaningful regions to contrast. GPC, in contrast, offers fresh perspective on pre-training 3D detectors. It is straightforward in its approach yet remarkably effective. Please see more results and discussion in Appendix S4
Learning with Images. We've noticed recent efforts in supe-supe learning using images (see Sec. 2). where most approaches use separate feature extractors for LiDAR and images to establish correspondences for contrastive learning. In contrast, GPC directly utilizes colors within a single trainable architecture. We strive to incorporate numerous baselines for comparison, but achieving a fair comparison is challenging due differences in data processing, model architecture, and training details. Therefore, we consider the baselines as references that demonstrate our approach's 's potential, rather than claiming SOTA status. In Tab. we compare to PPKT (Liu al. (021) 1) SL.dRR (Sautier et al.) 2022) and TriCC (Pang et al., (023) with moderate AP reported in |ang et al. (2023). Notably,
on 5% KITTI, we outperform the baselines with substantial gains of +7.5 and +8.7, which are significantly higher than their improvements (less than +4.7).
4.3 ABLATION STUDY
Colorization with regression or classification? In Tab. 5 we explore four losses for colorization. There is nearly no performance difference in using MSE, SL1, and CE, but BS outperforms other losses by clear margin. We hypothesize that it prevents the learning bias toward the ground (major pixels) and forces the model learn better on foreground objects that have much fewer pixels which facilitates the downstream 3D detection well.
Hint mechanism is the key design to overcome the inherent color variation (see Sec. 3). In Tab.
we directly regress RGB values (row 2 and 3) without hints and color decoders. The performance is better than scratch but limited. We hypothesize that the naive regression is sensitive to pixel values and can't 't learn well if there exists large color variation. The hint resolves the above issue and performs much better (row 4, 62.7 and row 5, 63.9). In Tab. 6) we ablate different levels of hint and find out providing only 20% seeds is the best (61.9), suggesting the task to learn can't be too easy
8

---

Published as a conference paper at ICLR 2024
(a) original image
(b) bias
(e) manual seeds
(c) 20% seeds
(d) colorized
(f) colorized
Figure 4: Qualitative results of colorization. (left) We use trained GPC to infer on validation LiDAR (b)
without any seeds and with (c) resulting (d). The model learned the bias about the driving scene: the ground gray and black, trees are green, and tail lights are red, but it predicts the average color all cars because they can variant. On the other hand, correctly colorizes the point cloud some hints are provided. We hypothesize such an ability to know where to pass the color is essential for downstream 3D detection. (right) (c) We further manipulate the seeds by hand. (f) GPC successfully colorize the regions with given colors in (e).
Table Seed ratios.
Table 7: Hint mechanism: with and with- Table 8: Number of epochs out color decoder.
pertaining on Waymo.
seed
easy
mod, hard
100%
51.7
38.7
33.8
Hint
Pre.
Dataset
casy
mod. hard
epochs easy mod. hard
80%
66.4
50.5
44.9
scratch
69.9
55.2
49.3
75.8
60.7
54.4
40%
75.1
59.7
53.4
KITTI
75.1
59.4
53.3
4
76.4
61.6
55.9
20%
77.1
61.9
56.2
Waymo
74.9
60.4
54.5
78.3
63.1
58.4
0%
57.9
44.1
38.4
KITTI
77.7
62.7
56.5
12
78.9
63.9
58.5
scratch 69.9
55.2
49.3
Waymo
78.9
63.9
58.5
(i.e., providing too many hints). However, providing hints falls into the issue of color variation again, deviating from learning a good representation (0%, 44.1 worse than scratch, 55.2).
Repeated experiments. To address potential performance fluctuations with different data sampled for fine-tuning, we repeat the 5% fine-tuning experiment for another five rounds, each with a distinctly sampled subset, as shown in Appendix S5
The results show the consistency and promise of GPC.
4.4 QUALITATIVE RESULTS
We validate GPC by visualization as shown in Fig. 4 (left). Impressively the model can learn common sense about colors in the driving scene in (b), even red tail lights cars. We further play with manual seeds to manipulate the object colors (Fig. 4 (right)) and it more or less recognizes which points should have the same color and stops passing the ground.
4.5 DISCUSSION
GPC outperforms the mainstream contrastive learning-based methods, suggesting our new perspective as promising direction to explore further. It is particularly advantageous when the labeled data are scarce. Regardless of the 3D model and the pre-training dataset, GPC 's performance of fine-tuning on just 20% of labeled data is consistently better than training from scratch with the entire dataset Tab.2 2 Tab. 3). This advantage holds the potential to reduce the annotation effort required.
Tab.
CONCLUSION
Color images collected along with LiDAR data offer valuable semantic cues about objects. We lever- age this insight to frame pre-training LiDAR- based 3D object detectors as a supervised colorization problem, novel perspective drastically different from the mainstream contrastive learning- approaches. Our proposed approach, Grounded Point Colorization (GPC), effectively addresses the inherent variation the ground-truth colors, enabling the model backbone to learn to identify LiDAR points belonging the same objects. As result, GPC could notably facilitate downstream 3D object detection, especially under label-scarce settings, as evidenced by extensive experiments on benchmark datasets. We hope work can vitalize the community with new perspective.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
This research is supported in part by grants from the National Science Foundation (IS72200077, OAC-2118240, OAC-21 12606, I1S-2107161, III-1526012, and IIS- (IS-1149882), NVIDIA research, and Cisco Research. We are thankful for the generous support of the computational resources by the Ohio Supercomputer Center. We also thank Mengdi Fan for her valuable assistance in creating the outstanding figures for this work.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
R. Adams and L. Bischof. Seeded region growing. TPAMI, 16(6))441-66 1994. doi: 00.1109/34.959111
Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In CVPR, 2022.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski. Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficie beel-efffcieet learning. In ECCV. Springer, 2022.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
Alexandre Boulch, Corentin Sautier, Bjöm Michele, Gilles Puy, and Renaud Marlet. Also: Automotive lidar self suserviperviion by occupancy estimation. In CVPR, 2023.
Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. In NeurfPS, 2019.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal datasct for autonomous driving. In CVPR, 2020.
Xu Cao and Katashi Nagao. Point cloud colorization based on densely annotated 3d shape dataset. In MultiMedia Modeling: 25th International Conference, MMM 2019, Thessaloniki, Greece, January 8 11, 2019, Proceedings, Part 1 25, pp. 436 436-446. Springer, 2019.
Gavin Cawley and Nicola LC Talbot. On over-fitting model selection and subsequent selection bias in performance evaluation. The Journal of Machine Learning Research, 11:2079-2101 21077,2017 2010.
Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012. 2015.
Zheng Chang, Shuchen Weng, Peixuan Zhang, Yu Li, Si Li, and Boxin Shi. L-coins: Language- hased colorization with instance awareness. In CVPR, 2023.
Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In JCML. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04299 042977 2020b.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-tempora convnets: Minkowski convolutional neural networks. In CVPR, 2019.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way Ensemble based methods avoiding known dataset biases. arXiv preprint (XXX::1999..06833 2019.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810004805, 2018.
Danny Driess, Fei Xia, Mchdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, al. Palm-e; An embodicd multimodal language model. arXiv preprint arXiv:2303.03378 2023.
Benjamin Eckart, Wentao Yuan, Chao Liu, and Jan Kautz, Self-supervised learning on 3d point clouds by learning discrete generative models. In CVPR, 2021.
Lue Fan, Feng Wang, Naiyan Wang, and ZHAO-XIANG ZHANG. Fully sparse 3d object detection. Advances Neural Information Processing Systems, 35:351-361 2022.

---

Published as a conference paper at ICLR 2024
Rongrong Gao, Tian-Zhu Xiang, Chenyang Lei, Jaesik Park, and Qifeng Chen. Scene-level point cloud colorization with semantics-ang saaeles--nndgggeeer-re geometry-awan networks. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2818-288 28244224. IEEE, 2023.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
Jean-Bastien Grill, Florian Strub, Floreat Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurIPS, 2020.
Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume Springer. 2009.
Chenhang He, Hui Zeng, Jianqiang Huang, Xian-Sheng Hua, and Lei Zhang. Structure aware single-stage 3d object detection from point cloud. In CVPR, 2020a.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020h.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Mingming He, Dongdong Chen, Jing Liao, Pedro V Sander, and Lu Yuan. Deep exemplar-hased colorization. ACM Transactions Graphics (TOG), 37(4))1-16, 2018.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In ICML. PMLR, 2020.
Ji Hou, Saining Xie, Benjamin Graham, Angela Dai, and Matthias NieBner. Pri3d: Can 3d priors help 2d representation learning' In JCCV, 2021.
Siyuan Huang, Yichen Xic, Song-Chun Cang-Chhn Zhu, and Yixin Zhu. Spatio-temporal self-supervised representation learning for point clouds. In ICCV, 2021.
Andrej Janda, Brandon Wagstaff, Edwin G Ng. and Jonathan Kelly. Contrastive learning for self supervied pre-training of point cloud segmentation networks with image data. arXiv preprint arXiv:2301.07286 2023.
Xiaozhong Ji, Boyuan Jiang, Donghao Luo, Guangpin Tao, Wenqing Chu, Zhifeng Xie, Chengjie Wang, and Ying Tai. Colorformer: Image colorization via color memory assisted hybrid-attention transformer. In ECCV, 2022.
Geonung Kim, Kyoungkook Kang, Seongtae Kim, Hwayoon Lee, Schoon Kim, Jonghyun Kim, Seung- Hwan Baek, and Sunghyun Cho. Bigcolor: colorization using generative color prior for natural images. In ECCV, 2022.
Manoj Kumar, Dirk Weissenborn, and Nal Kalchbrenner. Colorization transformer. arXiv preprint arXiv:2102.04433 2021.
Alex H I Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom, Pointpillars: Fast encoders for object detection from point clouds. In CVPR, 2019.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Leamingg representations for automatic colorization. In ECCV, 2016.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization a proxy task for visual understand- ing. CVPR, 2017.
Anat Levin, Dani Lischinski, and Yair Weiss, Colorization using optimization. In ACM SIGGRAPH 2004 Papers, pp. 689-694. 2004.
Zhengqin Li and Jiansheng Chen. Superpixel segmentation using linear spectral clustering. In CVPR, 2015.
Hanxue Liang, Chenhan Jiang, Dapeng Feng, Xin Chen, Hang Xu, Xiaodan Liang, Wei Zhang, Zhenguo Li, and Lue Van Gool. Exploring geometry-awer aware contrast and clustering harmonization for self-supervised 3d object detection. In ICCV, 2021.
Zhiwei Lin and Yongtao Wang. Bev-mae: Bird's eye view masked autoeneoders for outdoor point cloud pre-training. arXiv preprint a221v222/..05758, 2022.
12

---

Published as a conference paper at ICLR 2024
Haotian Liu, Mu Cai, and Yong Jae Lee. Masked discrimination for self-supervised learning on point clouds. In ECCV. Springer, 2022.
Jitao Liu. Songmin Dai, and Xiaoqiang Li. Peen: Point cloud colorization network. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 3716-3720. JEEE, 2019.
Ming-Yio Yu Liu, Oncel Tuzel, Srikumar Ramalingam, and Rama Chellappa. Entropy superpixel segmentation. In CVPR, 2011.
Yuch-Cheng Liu, Yu-Kai Huang, Hung-Yute Yueh Chiang, Hung-Ting Su, Zhe-Yu Yu Liu, Chin-Tang Chen, Ching- Yu Tseng, and Winston Hsu. Learning from 2d: Contrastive pixel-to-point knowledge transfer for 3d pretraining. arXiv preprint
2021.
Yunze Liu, Li Yi, Shanghang Zhang, Qingnan Fan, Thomas Funkhouser, and Hao Dong. P4contrast: Contrastive learning with pairs of point-pixel pairs for rgb-d scene understanding. arXiv preprint arXiv:2012.13095 2020.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, In ICLR, 2016.
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2017.
Zaiwei Lu. comprehensive survey on visual question answering debias. In 2022 IEEE International Conference on Advances in Electrical Engineering and Computer Applications (ALECA), pp. 470-475, 2022.
Chen Min, Xinli Xu, Dawei Zhao, I Liang Xiao, Yiming Nie, and Bin Dai. Occupancy-mac Self-supervised pre-
training large-scale lidar point clouds with masked occupancy autoencoders. arXiv preprint arXiv: 09900, 2022.
Bo Pang. Hongchi Xia, and Cewu Lu. Unsupervised 3d point cloud representation learning by triangle constrained contrast for autonomous driving. In CVPR, 2023.
Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autocncoders for point cloud sulf-supe lcarning. In ECCV. Springer, 2022.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In CVPR, 2017a.
Charles R Qi, Or Litany, Kaiming He, and I Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9277-9286. 2019.
Charles Ruizhongtai Qi. Li Yi, Hao Su, and Leonidas Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in metric space. NeurlPS, 2017b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21 (1):5485-555,, 2020.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long-tailed visual recognition. In NewrlPS, 2020.
Corentin Sautier, Gilles Puy. Spyros Gidaris, Alexandre Boulch. Andrei Bursuc, and Renaud Marlet, Image-to- lidar self- supervisdd distillation for autonomous driving data. In CVPR, 2022.
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In CVPR, 2019.
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. In CVPR, 2020a.
Shaoshuai Shi, The Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. TPAMI, 2020b.
13

---

Published as a conference paper at ICLR 2024
Takayuki Shinohara, Haoyi Xiu, and Masashi Matsuoka. Point2color: 3d point cloud colorization using conditional generative network and differentiable rendering for airborne lidar. In CVPR, 2021.
Oren Shrout, Ori Nitzan, Yizhak Ben- Shabat, and Ayellet Tal. Patchcontrast: Self-supervised pre-training for 3d object detection. arXiv preprint arXiv:2308. 00t.:23388.0685, 2023.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuha, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, 2022.
Jheng-Wor Wei Su, Hung-Kuo Kuo Chu, and Jia- Bin Huang. Instance-aware image colorization. In CVPR, 2020.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception autonomous driving: Waymo open dataset. In CVPR, 2020.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? NeurlPS, 2020.
Sandra Treneska, Eftim Zdravevski, Ivan Miguel Pires, Petre Lameski, and Sonja Gievska. Gan-based image colorization for self-supervised visual feature learning. Sensors, 1544):1599 2022.
Shaohua Wan, Yu Xia, Lianyong Qi, Yee-Hong Yang. and Mohammed Atiquzzaman, Automated colorization of grayscale image with seed points propagation. IEEE Transactions on Multimedia, 22(7):1756-11 1768, 2020.
Hanchen Wang. Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt Kusner. Unsupervised point cloud pre- pre-training via occlusion completion. In ICCV, 2021.
Shuchen Weng, Jimeng Sun, Li, Si Li, and Boxin Shi. Ct Colorization transformer via color tokens. In ECCV, 2022.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang. and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In CVPR, 2015.
Menghan Xia, Wenbo Hu, Tien-Tsin Wong, and Jue Wang. Disentangled image colorization via global anchors. ACM Transactions on Graphicx (TOG), 11(6):1-13, 2022.
Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In ECCV. Springer, 2020.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In CVPR, 2020.
Junbo Yin, Dingfu Zhou, Liangjun Zhang, Fang, Cheng- Xu, Jianbing Shen, Wenguan Wang. Proposalcontrast: Unsupervised pre-training for lidar-based 3d object detection. In ECCV. Springer. 2022.
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-hased 3d object detection and tracking. In CVPR, 2021.
Xumin Yu, Lulu Tang. Yongming Rao, Tiejun Huang. Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In CVPR, 2022.
Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2ae: Multi-scale masked autoencoders for hierarchical point cloud pre-training. In NeurlPS, 2022a.
Richard Zhang, Phillip Isola, and Alexci A Efros. Colorful image colorization. In ECCV, 2016.
Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, Yulan Guo. Not all points are equal: Learning highly efficient point- detectors for 3d lidar point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 18953-18962, 2022h.
Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. In ICCV, 2021.
Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832 2021.
Xinge 7hu, Yuexin Ma, Tai Wang, Yan Xu, Jianping Shi, and Dahua Lin. Ssn: Shape signature networks for multi-class object detection from point clouds. In ECCV, 2020.
14


Reviews:
[{"review_id": "j3r8BsL8ys", "reviewer": "Reviewer_Hnni", "summary": "This paper proposes a very interesting idea for boosting the performance of LiDAR-based 3D object detection by teaching the model to colorize LiDAR point clouds. The initial idea suffers from an inherent color variance issue, so the authors further propose the \"hints\" concept to directly provide ground-truth colors to some of the points that the model initially has to predict. This is a simple but effective idea grounded in both theory and experimental results. The proposed idea, namely GPC, demonstrates great performance in downstream 3D object detection tasks compared to benchmarks and has the potential to reduce the need for human annotations.", "questions": "Overall, this is a very good paper, but I still have several concerns and hope to get the authors' clarification.\n\n1. How can we ensure that the \"color\" assigned to each point is accurate? Color only exists in 2D images, and as mentioned by the authors, the color of a LiDAR point is retrieved by projecting it to the 2D image coordinate. One concern is that for those points out of the field of view of the camera (e.g., multiple points with different colors can be projected to the same 2D pixel), the color can be inaccurately labeled.\n\n2. It would be great to have an ablation study regarding the number of color bins used.\n\n3. The reweighted loss idea in Eq 5 sounds reasonable, but one concern is that it may make the pretrained model specific to a particular dataset, reducing its transferability since each dataset can have a different number of classes and class distribution. It would also be interesting to see how the pretrained model performs when it is trained on one dataset but fine-tuned on another (this experiment is not required in the rebuttal; providing insights into this is sufficient.)\n\n4. Please include more recent state-of-the-art works for comparison in Table 1 and Table 2. While I can gauge the effectiveness of the proposed idea through existing comparisons, a more extensive comparison with more recent works on the same benchmark is also necessary. The leaderboard of these two benchmarks has several entries showing better performance, so it would be great to discuss the advantages of the proposed method when compared to them.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The idea of introducing colorization to pre-train a LiDAR-based 3D object detection model sounds novel, and the experimental results also demonstrate its effectiveness on benchmark datasets.\n2. The method of providing ground-truth labels to seed points to reduce the inherent color variance issue also seems straightforward but very effective.\n3. I love the way the authors write this paper because it provides a clear motivational flow, explaining why they believe colorization can bridge the gap between point cloud and bounding box prediction. They also describe how their first attempt failed due to the color invariance issue and further propose their thinking process and solution.", "weaknesses": "Please check Questions section for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g5k248ykEK", "reviewer": "Reviewer_nb3N", "summary": "The paper presents a colorization-based pretraining approach for 3D LiDAR detectors. The idea is to use aligned RGB images to extract ground truth colors for a LiDAR point cloud and to then pretrain a network with the task of point cloud colorization. The paper empirically shows that a naive direct approach to this would actually harm the downstream detection performance, given that it would force the network to memorize specific scenes as the color of points could for example depend on the weather, the time of day. Furthermore, the color of many objects is not well-defined by the shape, e.g. cars come in different colors. To overcome this issue, the paper proposes to provide 20% of the points with the ground truth colors, thus allowing the network to focus on learning how to propagate these colors across semantically connected segments in a scene. Furthermore, instead of directly regressing RGB colors, all the colors in a dataset are clustered into 128 clusters and a class-balanced classification loss is used to train the network. Empirically, using the proposed pretraining, the paper shows that several downstream detectors can get significant improvements, often requiring only a fraction of the annotations to match the fully-supervised approach.", "questions": "- How did you sample the points that get a ground truth? Is this completely random? Or do you do some more informed sampling, ensuring that at least 1 sample is present for each cluster (if available in an image)? I guess this is not super crucial, but it could potentially reduce the amount of needed points and thus boost scores further since the network has to predict more.\n- Did you consider joint training? While we often see a strict pretraining and finetuning setup, think this could easily be trained jointly, maybe also improving results?\n- In the introduction you write \"A labeled example of 3D object detection comprises a point cloud and a set of bounding boxes. While the format is typical, there is no explicit connection between the input data and labels.\" I'm rather confused by this. To me there is definitely a direct connection between the input data and the labels and I don't understand how this is meant.\n\nIn general, while I enjoyed reading the paper, some sentences are a bit broken due to small grammar mistakes or missing articles. Please run this through a grammar checker or get a native speaker to proofread it.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a interesting and quite orthogonal approach to most other self-supervised pretraining methods for point cloud-based detection. Many design choices makes sense to me.\n- Empirical results look quite strong.\n- Important aspects are ablated and I very much like the fact that the paper first shows that naive colorization actually harms the performance.\n- The paper was a fun and refreshing read.", "weaknesses": "- The main weakness that I see is with the two baseline approaches, why did you focus on Point R-CNN and IA-SSD? There seem to be significantly better performing methods out there. These might not be point-based, but the paper also shows that the method transfers to voxel-based methods. So I'm mainly wondering what happens when we look at current state-of-the-art methods? Can we still gain something from colorization-based pretraining for those methods? Or is there some explicit reason why stronger methods cannot be considered?\n- Secondly, the related work is a bit lacking. While in general it covers the considered areas nicely, I'm missing related work on image colorization based pretraining, image colorization with seeds and general point cloud colorization. A quick google search reveals that all of these things exist. While I don't think they reveal major conflicts that should lead to the paper being rejected, I do think they are highly related and should be discussed. I'll name a few below, but obviously these papers are cited by many other papers and I'm sure for all three categories you can find some more relevant related work.\n  - Colorful Image Colorization by Zhang et al. looked at image colorization as a pretext task.\n  - Point2color: 3D Point Cloud Colorization Using a Conditional Generative Network and Differentiable Rendering for Airborne LiDAR by Shinohara et al. looks at point cloud colorization with a GAN setup. This could actually be a very interesting alternative to the seeding proposed here.\n  - Automated Colorization of a Grayscale Image With Seed Points Propagation by Wan et al. looks at seed-based image colorization. While different, it could be interesting to discuss such differences.\n- While you compare to ProposalContrast and DepthContrast, I find the discussion of the results quite lacking. Obviously they outperforms the proposed method in the case of cyclist and car, but only the strong performance of GPC for the pedestrian class pulls the overall score up. It would be good to at least talk about this a bit in the discussion of Table 2.\n- The ablation regarding the different loss variants is not that great. Again the discussion seems to be based on the overall score, but per class there are quite some significant differences. In general I am not completely convinced by this ablation, given these are completely different losses with potentially vastly different magnitudes. An actual fair comparison would require a new tuning of the learning rate, or potentially a normalization of the loss values with some theoretical maximum? Did you do something in this direction, or did you just replace the loss and assume that everything else stays the same? Also the SL1 regression loss seems better than CE loss in quite some cases, what about a balanced SL1? Could that be better?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WdoAD0HqQB", "reviewer": "Reviewer_cL42", "summary": "This paper focuses on an interesting topic - pretraining LiDAR networks by learning colorization.  To achieve LiDAR point colorization, this work proposes Grounded Point Colorization (GPC), which incorporates color as \"context\" by using GT colors during colorization.  The reported experiments seem to show it is effective to use colorization for LiDAR network pretraining.", "questions": "1. The camera and point cloud perspectives are inconsistent, which means that only a portion of points can find corresponding color information. It's somewhat surprising that this pretraining can significantly boost fine-tuning performance, especially in the case of KITTI, which only has a front-facing camera. Maybe the authors could provide a detailed analysis of why using colorization for pretraining is so effective.\n\n2. The authors claim that GPC + 20% fine-tuning can surpass 100% of training from scratch. However, it seems they haven't mentioned how they selected this 20%. If it's a random 20%, then the current LaserMix approach can achieve nearly the same level of fully supervised performance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. It is a new research idea - using colorization as a pre-train strategy for LiDAR networks.\n\n2. It is interesting to see that colorization might be effective for LiDAR-based network pretraining.", "weaknesses": "1. The authors use point-level, especially for the ground category, for pretraining, but lack validation of its effectiveness in instance-level 3D object detection tasks.\n\n2. They employ a multimodal pretraining approach (camera + LiDAR), while the compared methods are all unimodal (LiDAR-only).\n\n3. They did not conduct linear probing, so it's challenging to determine whether pretraining itself significantly improves performance or if enhancements were added during fine-tuning.\n\n4. Many details related to the proposed method and experiments have not been provided, which could confuse the readers.\nI have a few other questions listed in the next section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mTMEfsF7gX", "reviewer": "Reviewer_anv1", "summary": "This paper proposes a new pretraining method for LiDAR-based 3D object detection, by colorization. In particular, to mitigate the ambiguity in colorization, this paper proposes Grounded Point Colorization (GPC), introducing some seed points as hints.\nI believe the idea is good and makes sense to me.\nExperiments are mainly conducted in the KITTI dataset, showing promising results but reaching the SOTA performance.", "questions": "- According to my first concern in the weaknesses box, I wonder why not authors adopt segmentation as the downstream task? Or some other detection benchmarks relying on accurate semantics such as nuScenes? I believe the proposed method can obtain better performance on these tasks.\n- Authors should adopt SOTA detectors for experiments on KITTI, such as PVRCNN. The effectiveness should further be proven on Waymo Open Dataset with SOTA detectors such as PVRCNN, FSD, etc. The current performance is not competitive enough.\n\nPV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection \nFSD: Fully Sparse 3D Object Detection", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed method of Grounded Point Colorization (GPC) is reasonable. The use of seed points is also a good design.\n- This paper is well-written and easy to follow.\n- Though the experiments are mainly conducted in KITTI, they are thoughtful and complete.", "weaknesses": "- Authors use object detection as the downstream task, but I personally believe LiDAR-based segmentation is the best choice. Colorization-based pre-training mainly learns the semantics in my opinion, but object detection needs accurate locations and poses especially in the benchmark using IoU-based metrics such as KITTI and Waymo. \n- The ultimate performance is not that good. The results on the Waymo Open Dataset are too weak. Other results on KITTI are also not convincing because PointRCNN and IA-SSD are not SOTA detectors nowadays. \n- Although the writing is clear, there are some unnecessary parts like the decorative math on page 4, which do not make the paper better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)