Self-Supervised Pretraining for Large-Scale Point Clouds
Authors: Zaiwei Zhang
Min Bai
Li Erran Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep Learning
3D Computer Vision
Self-supervised Learning

Abstract: Pretraining on large unlabeled datasets has been proven to improve the down-stream task performance on many computer vision tasks, such as 2D object detection and video classification. However, for large-scale 3D scenes, such as outdoor LiDAR point clouds, pretraining is not widely used. Due to the special data characteristics of large 3D point clouds, 2D pretraining frameworks tend to not generalize well. In this paper, we propose a new self-supervised pretraining method that targets large-scale 3D scenes. We pretrain commonly used point-based and voxel-based model architectures and show the transfer learning performance on 3D object detection and also semantic segmentation. We demonstrate the effectiveness of our approach on both dense 3D indoor point clouds and also sparse outdoor lidar point clouds.

Self-Supervised Pretraining for Large-Scale Point Clouds
Zaiwei Zhang AWS AI Santa Clara, CA 95054
Min Bai AWS AI Santa Clara, CA 95054 baimi @dimin@amazon.com com
com
Erran Li AWS AI Santa Clara, CA 95054
lilimam ammzzon com
Abstract
Pretraining large unlabeled datasets has been proven to improve the down- stream task performance on many computer vision tasks, such as 2D object detec- tion and video classification. However, for large-scale 3D scenes, such outdoor LiDAR point clouds, pretraining is not widely used. Due to the special data char- acteristics of large 3D point clouds, approaches for 2D pretraining frameworks tend to not generalize well to this domain. In this paper, we propose a new self- supervised pretraining method that targets large-scale 3D scenes. We pretrain commonly used point-based and voxel-based model architectures and show the transfer learning performance on 3D object detection and semantic segmentation. We demonstrate the effectiveness of our approach on both dense 3D indoor point clouds and sparse outdoor LiDAR point clouds.
Introduction
Accurate and reliable perception is core to the capabilities and safety of autonomous and robotic systems such as self-driving vehicles and indoor robotics. To accurately measure structure of their surroundings, such platforms usually rely on LiDARs and depth cameras for outdoor and indoor scenes, respectively. Given the 3D point clouds, models are built for variety of perception tasks, such object detection, scene segmentation, localization, and mapping.
Modern deep learning models have shown great capabilities when large quantities of annotated data are given. However, generating the required amount of annotations is a laborious and expensive manual process [14; 57: 18). This is especially true for labeled datasets targeting object detection and semantic segmentation of large real-world scenes, which can contain dozens of independently moving objects and irregularly shaped background material. Moreover, the 3D data difficult to clearly visualize, manipulate, and annotate 2D screen. In the case of outdoor LiDAR scans, the annotation tasks are further complicated by the sparsity of the typical point cloud. However, the cost of acquiring a large quantity of unlabeled sensor data is usually much lower compared to manual annotation. Hence, developing algorithms to automatically discover useful patterns and strengthen the performance of models with few annotated examples is of great value.
Recently, researchers have explored self-supervised learning (SSL) bootstrap model training in various domains. In computer vision, [26; 8; 22; 25] make use of datasets such as ImageNet [50] with carefully designed pretext tasks to learn useful features for downstream tasks such as object detection and segmentation. These techniques rely on the enormous ImageNer dataset, which depicts
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Figure 1: Complexities of datasets. Other techniques pretrain on simpler scenes like ImageNet [50] (upper left) where each image prominently features one foreground object single ScanNet [14] frames (lower left). On the other hand, target highly complex indoor scenes like aggregated complete ScanNet scenes (middle) and LiDAR seans of road scenes like SemanticKITTI [5] (right) with dozens of different objects (colored).
tens of thousands of object types. Most images prominently feature single foreground object, which can be well-described by global feature vector for each sample. the 3D point cloud domain, several works [72; 65] have proposed pretraining techniques learn useful features for downstream tasks. However, these methods likewise primarily focus on much simpler scenes, such as single objects limited indoor scene captured by depth camera from a single viewpoint.
On the other hand, we target complex 3D scenes with a multitude of objects and background ma- terial. We visualize the differences between the subject datasets of existing approaches and our method in Figure We first consider large-scale, complete 3D indoor scene models generated from combined depth images. Next, we explore outdoor environments with LiDAR points up to 100m away on dozens of objects and complex background regions. We refer to our method as SSPL (self supervised pretraining for large-scale point clouds). Unlike most of the prior work, we propose technique which performs contrastive learning on local features extracted from sub-regions (vol- umes) within large-scale point clouds. Our algorithm makes use of local contrastive loss within scene and as well as a global contrastive loss across different scenes. As we will demonstrate, our method involving local feature reasoning greatly outperforms the existing methods in our settings.
In practice, [71] has noted that there are significant domain gaps attributable the LiDAR sensor due to its sparsity, even when the observed scenes and downstream tasks are highly similar. Motivated by this, we explore the setting of in-domain self-supervised pre-training for LiDAR point clouds. As we will demonstrate, existing 3D SSL methods [72; 65) are unsuitable for highly complex outdoor scenes, motivating the need for new algorithms.
In this paper, we make the following contributions:
We propose self-supervised pretraining method which leverages reasoning over local vol- ume level features for 3D point clouds.
We introduce novel ClusterInfoNCE loss to efficiently perform contrastive learning glob- ally for a large of local volume features.
We show that our pretraining approach outperforms baselines on multiple downstream tasks on both single-view outdoor LiDAR and large-scale indoor point clouds, with up to +6.2% higher AP25 in object detection and +3.6% mIoU in semantic segmentation.
2
Related work
Our method builds and extends the work from the self-supervised learning literature, with focus on large-scale 3D point clouds. In this section, we give an overview of the recent advances in self-supervised learning, as well as perception and representation learning in 3D.
Self-supervised learning for images Self-supervised learning has been studied extensively in ma- chine learning and computer vision [61; 44; 49; 51; 39]. There are many approaches for 2D repre- sentation learning, such as clustering [6; 32], GANs [16; 40J, various pretext tasks [15; 42; 62] etc. Recent advances [41; 26; 28; 8: 10; 59; 35; 20; 22; 25] have shown that self-supervised pre- training exhibits great advantages transfer learning, especially for benchmark with limited labels. [9; 4] both use similar local and global contrastive learning approaches but [9] uses domain specific

---

MxD
Auginnntation
Global Contraativ
Local Compastiv Lose
Global Festure
Loss
Augmentation
Ocoupled Voses
Figure Overview of SSPL- representation learning method for sargesccale point clouds pretraining. split& a 3D scene point cloud into occupied volumes and apply two different data augmentations for point clouds in each volume. In this figure, we show data augmentation examples of single volume. We use a neural network encoder and momentum encoder extract spatial features which pooled and projected to obtain local volume-level features. With paired volume features, SSPI. optimizes the local contrastive loss between paired features and optimizes a global contrastive loss between the local features and features in the global feature queue.
clustering for their global feature learning and [4] mainly studies the local feature affinities while we explicitly conduct similarity measures across different instances. [27] conducts data augmentation on local features but not global reasoning across different images. Our work adopts the momentum encoder from MoCo [26] to extract features for instance discrimination, instead of memory bank.
Self-supervised learning for 3D data Studies on unsupervised feature learning for 3D data have mainly been focused single 3D object representation with applications reconstruction, classi- fication or part segmentation [2: 17; 23; 24; 37: 52; 63; 69; 34; 1]. However, due the domain gap issues, features learned from single 3D object representations does not transfer well to scene level understanding tasks [65]. Recently, there has been a growing interest in designing self-supervised methods build representations of scene level point clouds [65; 31; 72]. Xie et al. [65] proposed self-supervieed method based on enforcing contrastive learning on point correspondences between be acquired after aligning
different views of 3D scenes. These point-wise correspondences different depth maps into a unified 3D scene, which requires time-consuming 3D registration algo- rithms specific human labeling efforts identifying static correspondences. In the case of sparse outdoor LiDAR point clouds, temporal correspondences are often not well defined. Similarly, an- other line of work [31; 19] rely on multiple scene scans' alignments. On the other hand, our method only requires single frame 3D scans, making it more generally applicable. Zhang et al. [72] con- ducted instance discrimination task on global scene-level features while our method focuses on reasoning across local patches.
Supervised 3D perception With the significant applications of roboties and self-driving vehicles, 3D perception has been a widely studied topic. Among the many perception tasks, supervised object detection [46; 54; 67; 53; 55; 36; 70; 11; 68; 73] and semantic segmentation [74; 60; 66; 29; 30; 33] have been explored by a large number of prior works. Because of the popularity and usefulness of these tasks, select them the downstream tasks to analyze the effectiveness of approach. However, we note that as our self-supervision is task agnostic, it can be applied to other downstream tasks. To achieve these tasks, number of neural network backbones have been proposed process 3D point cloud data, which is usually given as an unordered and irregular set of points [47; 21; 58]. In our work, we use both the popular PointNet++ [48] based on pointwise reasoning and 3D U-Net [66] based on sparse voxel convolutions to demonstrate the general effectiveness of our approach. Recently, [43] has proposed a similar local contrastive learning approach on clustered point clouds but they did not conduct global reasoning for those local point clusters.
3 Method
We introduce our self-supervised pretraining framework: SSPL as outlined Algorithm 1, and vi- sualized in Figure 2. As shown in Figure 1, we focus large-scale point clouds, which contain numerous objects and distinct structures within each scene. these cases, a global vector encod- ing requires a combinatorial and intractable level of expressiveness to describe the attributes and

---

Algorithm Training Framework for SSPL
Input Backhone architecture NN; Dataset (X)
Global feature queues Fg E R RCXD:
Output Pre-trained weight for the NN
for x; X do
Split into M occupied volumes:
[V [i=l
From Zi, generate two augmented versions zi and a Generate local crops of in .; and global crops of in 2
Compute volume features f² & RMx
Compute local contrastive loss , and global contrastive loss Lg( Fg)
- Upddte Fg with f² and update NN
end for
locations of the various objects. Therefore, we propose to perform representation learning on lo- cal features. We first describe local volume features extraction in Section 3.1, and introduce the local contrastive loss in Section 3.2. Finally, we discuss the formulation of the global contrastive reasoning in Section 3.3.
3.1 Local feature extraction
Given a large scale 3D point cloud instance 24, we split into M occupied volumes. Similar to patches in 2D images, can predefine volume size and sample up to M occupied volumes based on typical scale input scenes. x; then transformed into two augmented views x and x by applying random rotation and scaling. We then sample points in each occupied volume and generate two crops with different sizes for two augmented views. The smaller crops v/. generated from x;. defined as local crops while larger ones ef, generated from x7. are global crops. This scheme creates natural local-global correspondences, which enables the network to predict global features from local information in feature learning. x! and xi are mapped to feature vectors by applying a neural network. After acquiring the per-point/per-voxe features, we group the features by their corresponding volumes and apply max-pooling the last layer's features. We then apply two layer MLP as [10] and L2 normalization to compute the volume features f'l and f².
Momentum encoder Adapting the method of He el al. [26]. we use an encoder 0,, to extract features for of and momentum encoder O, to extract features for augmented view z?. The weights of the momentum encoder updated with an exponential moving average (EMA) from the encoder's weights. The update rule is 0g + 20,- + (1 x)0, with a fixed À 0.999.
3.2 Local volume reasoning
Traditionally, given a pair of volume features f² MxD from two augmented versions of I, instance, feature consistency is usually applied pre-text task. For contrastive learning ap- proaches, InfoNCE [26] is usually enforced, where volume features fi and fo are pushed to simi- lar and volume features are dissimilar to all other volume features from other instances. However, computation costs increase linearly as the number of feature embedding increases for each batch. Traditionally, InfoNCE assumes that there is one feature embedding for each data instance. Since in average there are about 300 occupied volumes for each scene in our experiments, is computa- tionally infeasible perform InfoNCE on each volume feature with large set of global features as negative examples.
Therefore, we adapt the PointInfoNCE [65] loss to conduct contrastive learning between different volumes the same point cloud. Compared to PointInfoNCE, we reason about volume-level infor- mation, where the loss is defined as:
L. = Σ log exp(s} 17/T)
JEM ExeM exp(f) f2/T)
Here, M the set all occupied volumes for the x; instance, and 7 temperature parameter [64]. The positive examples are the two augmented crops from the same volume, while negative examples

---

are from other volumes under the same instance. In practice, we find that performing this local volume reasoning alone as a pre-text task gives comparable better pretrained models compared to baseline methods. Please refer to Section 5.4 for experimental evidences.
Although applying BYOL [22] pretext task does not yield computational issues, find that it results in minimal performance gains in our settings. Please see Section 5.3 for more details.
3.3 Global volume reasoning
One natural objective for global contrastive reasoning is to use instance discrimination on volume features. As mentioned before, this objective does not scale well with large numbers of occupied volumes. Across different large scale scene datasets, we observe that there common objects appearing in almost every scene, egg. road and car in outdoor scenes or chair and sofa in indoor scenes. Inspired by this, we propose to conduct global volume reasoning through fixed number of global clusters. This reduces the computational burden and adds regularization during pretraining.
Preliminary on unsupervised feature clustering Traditional methods on feature clustering, such as k-means and DBSCAN have proven to be effective but consucnng to apply on large dataset. Recently, a line of methods [3; 8] show that pseudo-labed assignment can be formulated as optimal transport problem, where clustering can be done efficiently by the Baokhorn-Knopp algorithm [13]. We follow similar approach for our feature clustering steps.
Global feature queue We first build global feature queue Fg to store features of occupied volumes over multiple data instances. Adapting from He et al. [26]. the global feature queue is created with volume features f2 from the momentum encoder. The following steps are then used during training.
Global feature clustering For building global clusters, we follow the same method proposed by Asano el al. [3]. Given global feature queue Fg RCDD where C the queue size and D is the feature dimension, we apply a classification head hg : RD + R to convert the global feature vectors to class (cluster ID) scores. We then map them class probabilities via softmax operator: P = sof softmaz( Fg hg). We find the class label assignments Q by performing iterative Sinkhorn- Knopp updates [13].
Global contrastive loss Given a pair of volume features f², f2, we query the global feature queue with local features f to extract the corresponding cluster labels for each volume. We use kNN on 12 feature distances when querying, and we set = 1 for clustering.
With volume level class labels, it is natural to use classification pretext task for pretraining. How- ever, compared with pretraining on object-level features (e.g. ImageNet), our volume-level features usually lack class diversity and feature diversity. For indoor scenes, there are repeating wall and floor volumes, while majority of volumes are roads and buildings in outdoors scenes. We observe that such pretraining strategy provides minimal improvements. (Please see section 5.4 for details.) To directly apply feature representation learning across scenes, we propose a ClusterInfoNCE loss:
log exp(St (nn(FF,Fo)/ Ex-re F2//)
Lg
K
where K is the number of clusters and f and f. are drawn from f and f2 their corresponding class labels are knm(fe, F.2) extracts the most similar features of f? from F' which serve as positive examples. For negative examples, F/ labels are j. In our experiments, we find that using 20k negative examples works well. Since the runtime of ClusterInfoNCH loss depends only on number of clusters used, it scales well with more occupied volumes. By optimizing this loss, we are explicitly enforcing feature consistency across scenes and encouraging the volume features to form clusters globally. As shown in Section 4, observe that adding the ClusterInfoNCE loss results in improvements on various downstream tasks. As well, we compare with different negative sampling strategies in Section 5.4.
sampled from F; if their corresponding class
3.4 Implementation details
In our experiments, we set the size of global feature queue to be 300K. We use a temperature value of 0.1 while computing the non-paramttric softmax in Eq and 3, The local and global contrastive
5

---

Table 1: Fine-tuning performance ScanNet Dataset and SUN RGB-D Dataset (AP25)
% of Scan Used for Fine-Tuning
% of SUN Used for Fine- Tunnnn
Self-Supervision Method
20%
50%
70%
100%
20%
50%
70%
100%
None
46.1
52.3
54.0
58.6
45.2
55.2
55.6
57.4
BYOL [22]
48.2
54.0
55.5
59.1
47.7
55.6
56.4
58.0
PointContrast [65]
59.2
57.5
DepthContrast [72]
50.1
56.0
57.0
60.1
45.5
56.5
57.5
59.1
Ours
53.0
58.5
60.6
63.0
47.5
57.8
58.5
60.1
loss are equally weighted. For training, we use standard SGD optimizer with momentum 0.9, and we use cosine learning rate scheduler [38] which decreases from 0.06 0.00006 and train the model for 500 epochs with batch size of 96.
4 Experimental results
In this section, we thoroughly analyze the efficiency of our proposed method. First, we describe the tasks and datasets involved in our experiments. Then, we present quantitative results, demonstrating the increase in performance relative to our comparison baselines. Finally, we take a closer look at the contributions of each of our proposed components.
The key goal of sulf-super learning is automatically learn feature extractors that are useful for actual downstream tasks. Moreover, it is beneficial if the learned features are useful for different tasks, as this improves the usability of the technique and reduces the need for task-specific redesign. Our approach is chiefly designed for large scale scenes that are not dominated by a single, well de- fined foreground object. We demonstrate the effectiveness of our self-supervised learning approach by showing that downstream models can achieve significantly higher performance using small fraction of available annotations. Next, we dive into the features learned by our model gain addi- tional insights, followed by various ablation studies to justify the design choices in our approach. As mentioned in Section 1, for LiDAR point clouds, the sensor configuration has a tremendous impact on the structure of perceived 3D points. Therefore, we opt to pretrain and finetune our models in each domain individually for outdoor datasets.
4.1 Baseline methods
We limit our input data to measurements taken at single instants in time (instead of temporal se- quences) for generalization. As such, we use the recent BYOL [22] and DepthContrast [72] (based on MoCo [26]) as baselines. In both cases, we use the same full scene augmentation scheme as pro- posed in DepthContrast. For indoor scene pretraining, to introduce more scene complexity, we use the point clouds of fully reconstructed scenes built by aggregating multiple sequential measurements of the scene. Therefore, we also use PointContrast [65] comparison for the indoor setting.
4.2 Large scale dense point clouds
First, we evaluate our feature learning framework on point clouds produced with common depth cameras. In work, we choose the ScanNet [14] dataset for pretraining. ScanNet contains 1500 large scale indoor scenes, with each scene consisting of variety of objects. Additionally, it contains more than 10 different room types with diverse room scales. We believe that ScanNet covers diverse set of indoor object distributions.
Compared to DepthContrast [72] and PointContrast [65], only utilize the full scans from ScanNet training dataset for pretraining. We aim to learn sharp local features from large scale scenes, and thus are only interested in feature learning with full views instead of partial views. The pretraining dataset contains 1201 full view scans with 50k points per scan. We use a PointNet++ [48] backbone is widely used for different 3D perception benchmarks.
network for pretraining,
To evaluate the transfer learning performance of our pretrained models, we use two commonly used indoor 3D object detection benchmarks: ScanNet (SCAN) [14] and SUN RGB-D (SUN) [56]. We

---

finetune our pretrained model by using it as the backbone initialization of VoteNet [45] (a well known 3D object detector), and report the detection performance using the mean Average Precision at 1oU=0.25 (AP25) metric. To study the label efficiency of our pretrained models, we also subsam- ple different sets of the training data used for finetuning. We follow the setup VoteNet [45] for finetuning.
Based on the results in Table 1, our approach significantly improves downstream object detection performance across all settings, with up to 4.4% improvement on ScanNet and 2.7% on SUN RGB- D. Moreover, on ScanNet, our pretraining approach together with 20% of labeled data allows the downstream detection task to achieve the equivalent performance of using 50% of labeled data and training from scratch. Practically, this can lead significant savings labeling cost. Our ap- proach also outperforms the top-performing baseline method DepthContrast. (Note that we repro- duce DepthContrast with the original Votenet model and pretrain only on the 1200 instances for fair comparison.) We see a more prominent performance gain on ScanNet compared to on SUN RGB-D, as the former contains only 1201 training scenes while the latter contains more than 5k.
4.3 Large scale sparse point clouds
Next, we further increase the size and complexity of the 3D scene by using outdoor scenes collected for autonomous vehicles using LiDAR units. These scenes consist of large number of different objects and regions, spanning dozens of semantic classes. As we will show, critical consider local regions separately in the self-superrised learning framework.
Our algorithm aims learn useful features for all regions in point cloud. While detectable in- dividual objects cover a large fraction of the 3D structures present in the previous indoor datasets, compact and countable foreground objects suitable for detection such cars and pedestrians con- sist of only 10% of a typical outdoor scene. As such, we select the semantic segmentation task to validate our method, as it aims to give a meaningful label to every 3D point captured by the sensor.
We use two large scale, well-established public datasets for this task, which contain a large variety of scenes. SemanticKITTI (SK) [5] is based on the original KITTI [18] dataset, which is one of the original autonomous driving datasets that propelled a tremendous amount of progress in the perception field. This dataset contains 19k training frames over 10 sequences with approximately 122k points per scene. The more recent Waymo Open Dataset (WOD) [57] provides a further leap in the scale and variability of data by providing 24k training frames sampled from 850 driving sequences, with a higher point density of 161k points per scene. Both datasets define approximately 20 semantic classes. For these experiments, we use the sparse 3D convolution backbone based on U-Net [12] as it is used in current state of the art semantic segmentation models [66; 74]. As well, largely follow the training settings of [66].
To show that our approach extracts meaningful representations, we present semantic segmentation results on subsampled sets of the training data provided by SemanticKITTI and WOD. The samples are uniformly and randomly selected from the overall training datasets. To simulate the scenario where different levels of human annotation effort is available, the samples drawn such that any smaller fractional subset is contained within any larger subset. The results are shown in Table 2. We add a light-weight decoder with two convolutional layers to the pretrained point-wise feature encoding backbone network, and finetune until convergence. In each case, we report the average performance over the checkpoints of the last 15% of the training process to reduce noise.
Our approach significantly improves downstream semantic segmentation when the annotation is limited. For example, we observe 3.6% and 2.3% improvements in mloU on SemanticKITTI and WOD, respectively, when annotations for 1% subsets of the full datasets are provided. Morcover, these improvements are much more significant than those gained from pre-training with BYOL and DepthContrast, suggesting that beneficial pre-training on large scale point clouds require designs with specific considerations. On the other hand, we observe that the gains are reduced when addi- tional annotated data is provided, as seen in the case with 5% and 10% annotations, as expected.
4.4 Feature analysis
Here, we examine the features learned by our self-supervision scheme. As with the other analysis in our work, we compare with the BYOL and DepthContrast baselines, and use the SemanticKITTI
7

---

Table 2: Fine-tuning performance SemanticKITTI Dataset and Waymo Open Dataset (mloU)
% of SK Used for Fine- Tunnng % of WOD Used for Fine- Mune-Tuning
Self-Supervision Method
1%
2%
5%
10%
1%
2%
5%
10%
None
38.9
44.0
51.7
53.4
42.5
45,8
50.4
52.8
BYOL [22]
38.8
43.4
51.0
52.3
42.3
46.5
49.9
53.2
DepthContrast [72]
39.2
44.7
49.9
52.3
42.7
45.8
50.7
53.0
Ours
42.5
46.4
51.0
53.6
44.8
47.3
51.3
53.5
16
Figure 3: t-SNE visualization of features generated by backbone network with various weight settings.
dataset for the study. We visualize the final per-point features from the 3D UNet model using t-SNE dimensionality reduction and color the points according their semantic class. For clearer interpretability, we limit the semantic classes to the commonly seen and important classes road, person, and car. The points subsampled from the validation set.
We see that the embeddings produced using DepthContrast and our approach are both able sepa- rate the three classes in feature space to a much stronger degree than the BYOL baseline, as well as the random initialization which serves as a sanity check. In comparison with DepthContrast, our em- beddings have larger variance within the feature space while still being able to separate the semantic classes, suggesting that they can possibly preserve more distinctive geometric information while ex- tracting meaning. This hypothesis is supported by the added performance gain of pretraining using our method compared with DepthContrast as seen in Table 2.
5 Ablation study
In this section, we demonstrate the importance of each component in our training framework. We conduct ablation studies with models pretrained the widely accepted SemanticKITTI dataset with complex and large-scale scenes autonomous driving.
5.1 Volume cropping ratios
In the left side of Figure 4, we report the impact of applying different cropping ratios for the local crop. For the global crop, we randomly crop approximately 50% of the original volume so that most of the geometry feature is still preserved in the momentum encoder. We observe that smaller local crops produce marginal improvements in finetuning, and that 10% crop ratio is optimal.
41.0
40.5
39.5
Crop
Figure 4: Segmentation ablation study on SemanticKITI using 1% data for fine-tuning. Left: mloU vs number of occupied volumes, middle: mloU vs volume crop ratios, right: mloU vs number of global clusters.

---

Table BYOL contrastive methods
Table Different global volume reasoning strategies
Scratch
BYOL
Contrastive
Local
Cls
Hard
Rand
Our
1% SK
38.9
39.8
41.7
1% SK
41.7
40.8
41.0
40.5
42.9
5.2 Number occupied volumes
For local volume reasoning, the number of occupied volumes plays important role it directly affects the number of negatives used for local contrastive loss. Intuitively, by increasing the num- ber of occupied volumes, the local contrastive loss forces the network learn more distinet local features. However, with a fixed model size, the expressiveness of local features saturates after forms certain number of clusters, which directly correlates with number of occupied volumes in our training approach. As shown in the middle of Figure 4, we see strong performance with 400 occupied volumes with little gain for further increases.
5.3 BYOL vs contrastive method
Instead of contrastive methods, Grill al. [22] proposed new approach for self-supervised learn- ing BYOL, which directly enforces feature consistency between a student network and a teacher network. To better understand our framework, effort to replace the PointInfoNCE in the local volume reasoning. Formally, we use the following mean square error between the normalized predictions and target projections:
reproduce the BYOL loss formulation to our best
L
Σ - JEM
-
(3)
Similar to Equation 2, M is the set of all occupied volumes, and is predictor for the student network. Based on the results from Table 3, the features learned with BYOL performs significantly worse than the contrastive approach in 1 1% SemanticKITTI finetuning. We hypothesize that within the same instance, some volumes share very similar feature patterns.
Without explicitly encouraging dispersion within the feature space (e.g. negative examples), BYOL learns less distinguishing features in our setting.
5.4 Different strategies for global volume reasoning
As mentioned Section 3, natural pretext task for global volume reasoning is classfication. Given pair of volume features f1, f and their corresponding cluster labels Q", we apply linear repro- jection hy to map the volume features to class scores, and optimize the classification loss below:
Lute = Σ
Qj log P.
Σ JEM
Q log P
(4)
JEM
where P] = softmarr h1) and = so ffaa(f2 22ff.... h1). As shown in Table 4 where Cls repre- sents the model pretrained on the classification loss, we see that does not improve the finetuning performance, compared to using local contrastive loss alone (Local). We also compare different negative sampling strategies for the ClusterInfoNCE loss. our approach, after we find the positive feature example for cluster i, we sample negative feature examples from other clusters. In Table 4, we compare with only sampling negative examples from cluster i (Hard) (which can be seen hard negative sampling) as well as random sampling from all clusters (Rand). Based on the results, our approach (Our) provides largest performance boost.
5.5 Number of global clusters
We explore the impact the number global clusters used in our global feature queue to compute the contrastive loss (right plot of Figure 4). This hyperparameter is shown to be impactful, where 64 the best setting for the SemanticKITTI LiDAR seans. We believe that the optimal number influenced by the complexities of the 3D scenes in question.

---

Limitations, future work, and broader impact
Lastly, we discuss the limitations of our model. As the characteristics of 3D sensors vary widely (dense depth scans vs sparse LiDAR point clouds), our approach relies on in-domain unlabeled data as opposed to producing general purpose foundation model like those used in 2D images and language. Moreover, we do not make use of any temporal information, which can provide significant cues for feature consistency. We believe that these are possible areas for future research. We believe that increasing 3D perception capabilities of autonomous agents will allow them to operate more safely and enrich our lives. As self-supervised learning able to extract useful representation from unlabeled datasets, it can be less impacted by biases that may influence annotation processes, and possibly improve the reliability and fairness of perception models.
10

---

References
[1] Achituve, I,, Maron, H., Chechik, G.: Self-supervised learning for domain adaptation on point- clouds. arXiv preprint 1221: (2020)
[2] Achlioptas, P., Diamanti, O., Mitliagkas, I., Guibas, L.: Learning representations and genera-
tive models for 3d point clouds. In: Proceedings of the International Conference on Machine Learning (ICML). pp. 40-49. PMLR (2018)
[3] Asano, Y.M., Rupprecht, C., Vedaldi, A.: Self-Jabelling via simultaneous clustering and repre- sentation learning. International Conference Learning Representations (ICLR) (2020)
[4] Bai, Y., Chen, X., Kirillov,, A., Yuille, A., Berg, A.C.: Point-level region contrast for object detection pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16061-16070 (2022)
[5] Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., Gall, J.: Se- ManticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences. In: Proc. of the IEEE/CVF International Conf. on Computer Vision (ICCV) (2019)
[6] Bojanowski, P., Joulin, A.: Unsupervised learning by predicting noise. In: Proceedings of the International Conference on Machine Learning (ICML) (2017)
[7] Caron, M., Bojanowski, P. Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)
[8] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. In: NeurIPS (2020)
[9] Chaitanya, K., Erdil, E., Karani, N., Konukoglu, E.: Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Infor- mation Processing Systems 33, -2558 (2020)
[10] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 (2020)
[11] Chen, Y., Liu, S.. Shen, X., Jia, J.: Fast point r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 9775-9784 (2019)
[12] Cicek, O., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: Learning dense volumetric segmentation from sparse annotation. MICCAI (2016)
[13] Cuturi, M.: Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems 26 (2013)
[14] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Niessner, M.: Scannet: Richly- annotated reconstructions of indoor scenes. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017)
[15] Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by con- text prediction. In: Proceedings of the International Conference on Computer Vision (ICCV) (2015)
[16] Donahue, J., Krahenbühl, P.. Darrell, T.: Adversarial feature learning. In: International Con- ference on Learning Representations (ICLR) (2016)
[17] Gadelha, M., Wang, R., Maji, S.: Multiresolution tree networks for 3d point cloud processing. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 103-118 (2018)
[18] Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The kitti dataset. Interna- tional Journal of Robotics Research (URR) (2013)
[19] Gojeic, Z., Litany, O., Wieser, A., Guibas, L.J.,, Birdal, T.: Weakly supervised learning of rigid 3d scene flow. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5692-5703 (2021)

---

[20] Goyal, P., Mahajan, D., Gupta, A., Misra, I: Scaling and benchmarking self self-suuervised vi- sual representation learning. Proceedings of the International Conference on Computer Vision (ICCV) (2019)
[21] Graham, B., Engelcke, M., van der Maaten, L.:
3d semantic segmentation with submanifold sparse convolutional networks. In: Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 9224-9232 (2018)
[22] Grill, J.B., Strub, F. Altché, F, Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C.,, Avila Pires, B., Guo, Z.,, Gheshlaghi Azar, M., et...: Bootstrap your own latent-a new approach to self-supervised learning, Advances in Neural Information Processing Systems 33, 21271- -- 21284 (2020)
[23] Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: A papier-måché approach to learning 3d surface generation. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). pp. 216-224 (2018)
[24] Hassani, K., Haley, M.: Unsupervised multi-task feature learning on point clouds. In: Proceed- ings of the International Conference Computer Vision (ICCV). pp. 8160-8177 (2019)
[25] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. arXiv preprint 2Xiv:2111.0633) (2021)
[26] He, K., Fan, H., Wu, Y. Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. arXiv preprint urXiv:1911.05722 (2019)
[27] Hénaff, O..,, Koppula, S., Alayrac, J.B., Van den Oord, A., Vinyals, O., Carreira, J.: Efficient visual pretraining with contrastive detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10086-10096 (2021)
[28] Hénaff, O.J., Razavi, A., Doersch, C., Eslami, S., Oord, A.v.d.: Data-efficient image recogni- tion with contrastive predictive coding. arXiv preprint arXiv:1905 1905.092222 (2019)
[29] Hu, Q., Yang. B., Xie, L., Rosa, S., Guo, Y., Wang. Z., Trigoni, N., Markham, A.: Randla-net: Efficient semantic segmentation of large-scale point clouds. Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition (2020)
[30] Hu, Q. Yang, B., Xie, L., Rosa, S., Guo, Y., Wang, Z., Trigoni, N., Markham, A.: Learning semantic segmentation of large-scale point clouds with random sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)
[31] Huang, S., Xie, Y., Zhu, S.C., Zhu, Y.: Spatio-temporal self-supervised representation learning for 3d point clouds. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6535 6545 (2021)
[32] Ji, X., Henriques, J.F., Vedaldi, A.: Invariant information clustering for unsupervised image classification and segmentation. In: Proceedings of the International Conference on Computer Vision (ICCV) (2019)
[33] Jiang. L., Shi, S., Tian, Z., Lai, X., Liu, S., Fu, C.W., Jia, J.: Guided point contrastive learn- ing for semi supervipee point cloud segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2021)
[34] Jing, I.., Chen, Y., Zhang, L., He, M., Tian, Y.: Self-supervised modal and view invariant feature learning. arXiv preprint arXiv:2005. 14169 (2020)
[35] Kolesnikov, A., Zhai, X., Beyer, L.: Revisiting self-supervised visual representation learning. arXiv preprint arXiv: 100,0000544 (2019)
[36] Li, B., Ouyang. W. Sheng, L., Zeng, X., Wang. X.: Gs3d: An efficient 3d object detection framework for autonomous driving. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1019-1028 (2019)
12

---

[37] Li, J., Chen, B.M., Hee Lee, G.: So-net: Self-organizing network for point cloud analysis. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9397-9406 (2018)
[38] Loshchilov, I,, Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 2XXiv:600.00398 (2016)
[39] Masci, J., Meier, U., Cires, D., Schmidhuber, J.: Stacked convolutional auto-encoders for hierarchical feature extraction. In: ICANN. pp. 52-59 (2011)
[40] Mescheder, L., Nowozin, S., Geiger, A.: Adversarial variational Bayes: Unifying variational autoencoders and generative adversarial networks. In: Proceedings of the International Con- ference on Machine Learning (ICML) (2017)
[41] Misra, I, van der Maaten, L.: Sele-super learning of arXiv preprint 1911011912.0191 (2019)
representations.
[42] Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: Proceedings of the European Conference on Computer Vision (ECCV) (2016)
[43] Nunes, L., Marcuzzi, R., Chen, X., Behley, J., Stachniss, C.: Segcontrast: 3d point cloud feature representation learning through self-suuprrvied segment discrimination. IEEE Robotics and Automation Letters 7(2), 2116-2122 (2022)
[44] Olshausen, B.A., Field, DJJ.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature (8583),, 607 (1996)
[45] Qi, C.R., Litany, O., He, K., Guibas, L.J.: Deep hough voting for object detection in point clouds. arXiv preprint 1xXiv:190..0960. (2019)
[46] Qi, C.R., Liu, W., Wu, C., Su, H., Guibas, LJ:: Frustum pointnets for 3d object detection from rgb-d data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 918-999899 927 (2018)
[47] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classi- fication and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 652-660 (2017)
[48] Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++:: Deep hierarchical feature learning on point sets in a metric space. In: Advances in neural information processing systems, pp. 5099-5108 (2017)
[49] Ranzato, M., Huang, FJJ., Boureau, Y.L., LeCun, Y.: Unsupervised learning of invariant feature hierarchies with applications object recognition. In: CVPR (2007)
[50] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. IJCV 115 (2015)
[51] Salakhutdinov, R., Hinton, G.: Deep Boltzmann machines. In: AI-STATS. pp. 448-455 (2009)
[52] Sauder, J., Sievers, B.: Self-supervised deep learning on point clouds by reconstructing space. In: Advances in Neural Information Processing Systems (NeurlPS), pp. 12962-12972 (2019)
[53] Shi, S., Guo, C., Jiang, L.,, Wang, Z., Shi, J., Wang, X., Li, H.: Pv-renn: Point-voxel feature set abstraction for 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10529-10538 (2020)
[54] Shi, S., Wang, X., Li, H.: Pointrenn: 3d object proposal generation and detection from point cloud. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770-779 (2019)
[55] Shi, S., Wang, Z., Shi, J., Wang, X., Li, H.: From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. arXiv preprint (Tt.:1907003677 (2019)
13

---

[56] Song, S., Lichtenberg, S.P., Xiao, J.: Sun rgb-d: A rgb-d scene understanding benchmark suite. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 567-576 (2015)
[57] Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo. J., Zhou, Y., Chai, Y., Caine, B., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition. pp. 2446-2454 (2020), this publication was made using the Waymo Open Dataset, provided by Waymo LLC under license terms available vaymo.com/open.
[58] Thomas, H., Qi, C.R., Deschaud, J.E., Marcotegui, B., Goulette, F., Guibas, LJ:: Kpconv: Flexible and deformable convolution for point clouds. In: Proceedings of the IEEE Interna- tional Conference on Computer Vision. pp. 6411-6420 (2019)
[59] Tian, Y., Sun, C.,, Poole, B., Krishnan, D., Schmid, C., Isola, P.: What makes for good views for contrastive learning? arXiv preprint arXiv:2005.10243 (2020)
[60] Unal, O., Dai, D., Van Gool, L.: Scribble-supervised lidar semantic segmentation. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
[61] Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: ICML (2008)
[62] Wang, X., Gupta, A.: Unsupervised learning of visual representations using videos. In: Pro-
ceedings of the International Conference on Computer Vision (ICCV) (2015)
[63] Wang, Y., Solomon, J.M.: Deep closest point: Learning representations for point cloud reg- istration. In: Proceedings the International Conference on Computer Vision (ICCV). pp- 3523 3532-3332000 (2019)
[64] Wu, Z., Xiong, Y., Yu, S.X., Lin, D.: Unsupervised feature learning via non- con-parametric in- stance discrimination. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
[65] Xie, S., Gu, J.. Guo, D.,, Qi, C.R., Guibas, L.J., Litany, O.: Pointcontrast: Unsupervised pre- training for 3d point cloud understanding. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020)
[66] Yan, X., Gao, J., Li, J., Zhang, R., Li, 7., Huang, R., Cui, S.: Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion. In: Proceed- ings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 3101-3109 (2021)
[67] Yan, Y., Mao, Y., Li, B.: Second: Sparsely embedded convolutional detection. Sensors 18(10), 3337 (2018)
[68] Yang, B., Liang, M., Urtasun, R.: Hdnet: Exploiting hd maps for 3d object detection. In: Proceedings of Conference on Robot Learning (2018)
[69] Yang, Y., Feng, C., Shen, Y., Tian, D.: Foldingnet: Point cloud auto-encoder via deep grid deformation. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). pp. 206-215 (2018)
[70] Yang, Z., Sun, Y., Liu, S., Shen, X., Jia, J.: Std: tpeese- 3d object detector for point cloud. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1951- 1960 (2019)
[71] Yi. L.. Gong, B., Funkhouser, T.: Complete & label: domain adaptation approach to semantic segmentation of lidar point clouds. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2021)
[72] Zhang, Z.,, Girdhar, R., Joulin, A., Misra, I.: Self- Self-supervised pretraining of 3d features on any point-cloud. In: Proceedings the IEEE/CVF International Conference on Computer Vision. pp. 10252-10263 (2021)
14

---

[73] Zhang, Z., Sun, B., Yang, H.. Huang, Q.: H3dnet: 3d object detection using hybrid geometric primitives. arXiv preprint arXiv:2006.05682 (2020)
[74] Zhu, X., Zhou, H., Wang, T., Hong, F. Ma, Y., Li, W., Li, H., Lin, D.: Cylindrical and asym- metrical 3d convolution networks for lidar segmentation. arXiv preprint arXiv:2011. 10033 (2020)
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as URL)? [No] We provide the detailed information about our model architecture in the supplementary material. Our codes are still under internal review and we will release the codes as soon as it's approved internally.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provide the detailed
(c) Did you report enror bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] We try to fix random seed during pretraining, and we always measure the finetuning performance after the loss converges.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We have provided this information in the supplementary material.
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] We used widely tackled academic
datasets with clear license information their respective publications as cited in our work.
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 'e using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "yJh9R7Wc2Iv", "reviewer": "Reviewer_x6Sn", "summary": "This paper proposed an unsupervised pertaining technique to improve representation learning of point clouds that can be applied to popular models for various downstream tasks.\nThe key idea of this paper, compared to the baseline method (DepthContrast [67]), is to learn the features from local patch level.", "questions": "- PointNet++ gradually downsamples the point cloud by grouping local neighbourhood together. How do you get per-point feature?\n- L151, should the reference be [3] instead?\n", "limitations": "I didn't find discussions about limitations. ", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- the main idea, implemented through two different losses, i.e. local volume PointInfoNCE and global volume ClusterNCE, makes sense to me, since patch-based approach allows the feature encoding to learn from repetitive structures in large-scale point cloud scenes.\n- the method is evaluated on different types of data, including outdoor dataset, which is not done in [67].\n- the pretrained model can be valuable to the community.\n\n**Weakness**\n- The method seems to work well. But I have some concerns about the principles of this approach. \n  - As mentioned by the authors, the information in outdoor lidar scans is very sparse, \"important foreground often comprise few points\". Local crops may contain zero foreground. In this case, $f^1$ does not capture any unique information that can be sufficiently associated to its corresponding global crop. \n  - The pretraining takes place on local crop level and as suggested by Figure 4 the crop size matters. What happens when at test-time the scale of the input point cloud changes? In this case the receptive field of the network (e.g. the spatial span of KNN in PointNet++) also changes. How can the proposed method address such scale variation?\n- The paper is quite difficult to read for anyone not up-to-date with unsupervised learning. I find Section 3.3 particularly difficult to understand. IMHO, some basic concepts of unsupervised feature clustering should be given to make the paper complete.\n- Similarly, at several places the authors mentioned the advantage of local feature learning in terms of computational cost, e.g. L127-128, L141 and L146. However, from the paper itself, it's not obvious what the baseline is, why it won't scale well with the large number of occupied volumes. The authors should briefly summarise what would be the computational cost for standard practice.\n- The comparison with [67] was not very clear to me. In L227-229, the author mentioned they had to reproduce [67] using a different training set, which seems to result in a slightly worse result for [67] compared to the what's originally reported (Table 2 in [67]). The reason for this isn't very clearly conveyed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gK9uUGKyXsV", "reviewer": "Reviewer_4aUz", "summary": "The paper proposes a self-supervised pretraining method for large-scale point clouds (SSPL). SSPL is motivated by the observation that most 3D scenes don’t consist of a single, large, foreground object but instead contain many objects of equal scale and importance. To account for this, SSPL introduces a contrastive term over local crops in the same scene and couples this to a standard global contrastive loss. SSPL implements this algorithm using standard techniques from the SSL literature. The resulting algorithm outperforms previous SSL algorithms on several 3D datasets for object detection and semantic segmentation.\n", "questions": "- How dependent is the performance of the method on the augmentation used? Augmentation is the trick by which contrastive SSL methods of this kind are given information about the structure of the target domain. So additional ablations or analysis would be helpful for interpretation of how general this method is. In particular, it would be very useful to see an error analysis as a function of the augmentation used: what specific examples in the dataset benefit from using e.g. RandomScaling vs. RandomLocalDrop?\n\n- L273: “We see that the embeddings produced using DepthContrast and our approach are both able to separate the three classes in feature space to a much stronger degree than the BYOL baseline,...” The t-SNE plots shown in Figure 3 aren’t sufficient to support this claim. The separability exhibited in t-SNE is well-known to depend on details of the hyperparameters in unpredictable and unreliable ways (https://distill.pub/2016/misread-tsne/). I suggest reporting quantitative numbers on clustering or removing this claim. If using t-SNE plots to compare methods, it is better practice to sweep best hypers individually for each plot to avoid confirmation bias.\n\n- L137: “pre-text” -> “pretext”\n\n- The use of “SK” as an abbreviation for SemanticKITTI is a bit distracting and makes the paper harder to read. It makes complete sense to use this when space is severely constrained (e.g.  Table 2), but please write out SemanticKITTI in the main text if at all possible.\n\n- Table 1: mAP@0.25 -> AP25 to match the text (L219)\n", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper addresses an important problem: developing SSL methods beyond the simple, highly stereotyped settings (ImageNet, single-frame ScanNet) for which standard global SSL techniques are engineered.\n\n- The proposed method is straightforward and elegantly reuses features for both local and global contrastive losses, which in principle (and at adequate scale) should allow it to learn features that are strictly more discriminative than local- or global-only contrastive methods.\n\n- The approach is well-motivated both in terms of the literature and in terms of the limitations of existing SSL methods for representation learning on point-clouds.\n\n- The approach produces compellingly strong results against fair SSL baselines on several competitive benchmarks in the point cloud domain.\n\n\n**Weaknesses**\n- Table 1: only SSL methods are presented. These are problems that are well-addressed by existing supervised learning methods, so baseline and SotA supervised results are stronger baselines: SotA supervised on ScanNet is >70% AP25. These numbers should be reported to make it clear that while SSPL narrows the gap, it still performs significantly worse than supervised methods.\n\n- The proposed method makes few assumptions that are specific to point clouds (other than the augmentation strategy). It would be fantastic to see this work applied to other domains, such as images (where most SSL methods are currently evaluated), audio, text, multimodal, or other domains. Is there an algorithmic reason to focus on point clouds exclusively? If not, why is the evaluation limited to point clouds, which are less competitive than other SSL domains?\n\n- While the method is evaluated on two tasks (*detection* on ScanNet/SUN and semantic *segmentation* on SemanticKITTI and Waymo Open Dataset) it is not evaluated on multiple tasks on a single dataset. A key reason to investigate SSL at all (rather than directly optimize a task of interest, which works on this data) is that it allows direct application to multiple downstream tasks. The case for this method would be much stronger if it led to simultaneous improvements on several of the tasks supported by ScanNet, rather than just one.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eqlU-Wzij3L", "reviewer": "Reviewer_2B4N", "summary": "The paper proposes a method for unsupervised pretraining of models for large-scale point cloud processing. The learned representations can be used for finetuning across datasets for various tasks such as point cloud classification or segmentation. The experiments show that pretraining the representation with local and global context improves performance compared to training a model from scratch.\n", "questions": "* In the experiments, all datasets contain similar scenes, so the global features encode some common context between scenes. What happens if the context is changed drastically? For example, from urban outdoor environments to more rural ones with less man-made features and more natural ones? Would this method suffer in these cases?\n\n* Line 265: \"On the other hand, we observe that the gains are reduced when additional annotated data is provided, as seen in the case with 5% and 10% annotations, as expected.\" Why is this expected? Please elaborate.\n", "limitations": "As far as I can tell limitations of the approach are not discussed in the paper. Consider discussing limitations in the conclusion section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe idea of transfer learning has been shown to be highly effective for learning in the 2D image domain. Adapting this concept to 3D data is an exciting and promising research direction. The experiments also seem to validate that unsupervised pretraining can lead to better-separated feature spaces and improve performance on downstream tasks. \n\n* The paper is clearly written and easy to follow, well polished, and does not contain many errors. The paper also contains supplemental material with additional details for reproducibility and additional results. \n\n* The paper motivates the proposed approach by showing a gap in the literature, focusing mainly on data containing a single prominent object. In contrast, the paper proposes to study the problem of contrastive pretraining of large-scale point clouds containing multiple objects. \n\n* The provided ablations and visualizations of the t-SNE embeddings are very helpful in solidifying the point that the learned features are more informative and better separated in feature space than the features learned from scratch or by the baselines.\n\nWeaknesses:\n* As I am not intrinsically familiar with the literature on contrastive learning on point clouds, it was unclear what the intuition was behind global features. Here are some pointers towards making this clear: \n1. In Figure 2. what is the dimension of the global feature queue? \n2. In Section 3.3: What is the intuition behind global features? What do they represent? How is the global feature queue constructed and updated? Are random f2 features from the momentum encoder added to this queue or do some criteria choose them?\n\n* One point the paper should address is how the proposed method compares against methods that use diverse, labeled datasets for pretraining. I think it would be interesting to see how the proposed method performs against a method that, for example, uses SUN and ScanNet training sets for pretraining. I am especially interested if the proposed method's resulting feature space is still more informative (I think it might be).\n\n* Code and data are not released with this paper. Is there a reason why the paper does not contain this or will this be released upon acceptance?\n\nMinor issues:\n* Algorithm 1: \"Computer local contrastive\" -> Compute local contrastive\n* Line 275: \"the the\" -> the\n* Line 278: \"This is hypothesis\" -> This hypothesis", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept