Pre-Training by Completing Point Clouds
Authors: Hanchen Wang
Qi Liu
Xiangyu Yue
Joan Lasenby
Matt Kusner
Venue:  (2021)
Keywords: self-supervised learning
pre-training
point clouds

Abstract: There has recently been a flurry of exciting advances in deep learning models on point clouds. However, these advances have been hampered by the difficulty of creating labelled point cloud datasets: sparse point clouds often have unclear label identities for certain points, while dense point clouds are time-consuming to annotate. Inspired by mask-based pre-training in the natural language processing community, we propose a pre-training mechanism based point clouds completion. It works by masking occluded points that result from observations at different camera views. It then optimizes a completion model that learns how to reconstruct the occluded points, given the partial point cloud. In this way, our method learns a pre-trained representation that can identify the visual constraints inherently embedded in real-world point clouds. We call our method Occlusion Completion (OcCo). We demonstrate that OcCo learns representations that improve the semantic understandings as well as generalization on downstream tasks over prior methods, transfer to different datasets, reduce training time and improve label efficiency.

Under review as a conference paper at ICLR 2021
PRE-TRAININ BY COMPLETING POINT CLOUDS
Anonymous authors
Paper under double-blind review
ABSTRACT
There has recently been a flurry of exciting advances in deep learning models on point clouds. However, these advances have been hampered by the difficulty of creating labelled point cloud datasets: sparse point clouds often have unclear label identities for certain points, while dense point clouds are consuming to annotate. Inspired by mask-based pre-training in the natural language processing community, we propose pre-training mechanism based point clouds completion. It works by masking occluded points that result from observations at different camera views. It then optimizes a completion model that learns how to reconstruct the occluded points, given the partial point cloud. In this way, our method learns a pre-trained representation that can identify the visual constraints inherently embed- ded in real-world point clouds. We call our method Occlusion Completion (OcCo). We demonstrate that OcCo learns representations that improve the semantic un- derstandings as well as generalization on downstream tasks over prior methods, transfer to different datasets, reduce training time and improve label efficiency.
INTRODUCTION
Point clouds are a natural representation of 3D objects. Recently, there has been a flurry of exciting new point cloud models in areas such as segmentation (Landrieu & Simonovsky 2018 Yang et al.
2019a Hu et al. 2020a) and object detection (Zhou & Tuzel Lang et al. 2019 Wang et al. 2020b Current 3D sensing modalities (i.e., 3D scanners, stereo cameras, Tidars) have enabled
the creation of large repositories of point cloud data (Rusu & Cousins 2011 Hackel et al. 2017). However, annotating point clouds challenging as: (I) Point cloud data can sparse and low resolutions, making the identity of points ambiguous; (2) Datasets that are not sparse can easily reach hundreds of millions of points (e.g., small dense point clouds for object classification (Zhou & Neumann 2013) and large vast point clouds for 3D reconstruction (Zolanvari et al. 2019);; (3) Labelling individual points or drawing 3D bounding boxes are both more complex and time- consuming compared with annotating 2D images (Wang et al., 2019a). Since most methods require dense supervision, the lack of annotated point cloud data impedes the development of novel models.
On the other hand, because of the rapid development of 3D sensors, unlabelled point cloud datasets are abundant. Recent work has developed unsupervised pre- pre-training methods to learn initialization for point cloud models. These based designing novel generative adversarial networks (GANs) (Wul et al. al. 2018a Yang
2016 Han et al. 2019 Achlioptas et al.] 2018) and autoencoders (Hassani & Haley 2019
2018). However, completely unsupervised pre-training methods have been recently outperformed by the self-supervised pre-training techniques of (Sauder & Sievers 2019) and (Alliegro et al. 2020). Both methods work by first voxelizing point clouds, then splitting each axis into k parts, yielding 1- voxels. Then, voxels are randomly permuted, and model is trained to rearrange the permuted voxels back to their original positions. The intuition is that such a model learns the spatial configuration of objects and scenes. However, such random permutation destroys all spatial information that the model could have used to predict the final object point cloud.
Our insight is that partial point-cloud masking is good candidate for pre-training in point-clouds because of two reasons: (I) The pre-trained model requires spatial and semantic understanding of the input point clouds to be able to reconstruct masked shapes. (2) Mask-based completion tasks have become the de facto standard for learning pre- pre-trained representations in natural language processing (NLP) (Mikolov et al. 2013 Devlin et al. 2018 Peters et al. 2018). Different from random permutations. masking respects the spatial constraints that are naturally encoded in point clouds of real-world objects and scenes. Given this insight, we propose Occlusion Completion (OcCo)

---

Under review as a conference paper at ICLR 2021
occlusion
completion
downstream training
(e.g classification)
camera view-points
completed point point cloud P
OcCo weights as Initialization
x
accuracy
occluded point
original point cloud
P
random baselline Initialization
iterations
Figure 1: OcCo consists of two steps: (a) occlusion o(-) of point cloud P based on random camera view-point into partial point cloud P, and (b) a model c(.) that completes the occluded point cloud 'P so that P = P. We demonstrate that the completion model c(-) can be used as initialization for downstream tasks, leading to faster training and better generalization over existing methods.
a self-supervised pre-training method that consists of (a) a mechanism to generate occluded point clouds, and (b) completion task to reconstruct the occluded point cloud.
Specifically, in (a) point clouds are generated by determining what part of objects would be occluded if the underlying object was observed from particular view-point. In fact, many point clouds generated from a fixed 3D sensor will have occlusions exactly like this. Given an occluded point cloud, the goal of the completion task (b) is to learn a model that accurately reconstructs the missing parts of the point cloud. For model to perform this task well, it needs to learn to encode localized structural information, based on the context and geometry of partial objects. This is something that is useful for any point cloud model to know, even if used only for classification or segmentation.
We demonstrate that the weights learned by our pre-training method on a single unsupervised dataset can be used as initialization for models in downstream tasks (e.g., object classification, part and semantic segmentation) to improve them, even on completely different datasets. Specifically our pre-training technique: (i) leads to improved generalization over prior baselines the downstream tasks of object classification, object part and scene semantic segmentation: (ii) speeds up model convergence, in some cases, by up to 5x; (iii) maintains improvements the size of the labelled downstream dataset decreases; (iv) can be used for a variety of state-of-the-art point cloud models.
2 OCCLUSION COMPLETION
We now introduce Occlusion Completion (OcCo). Our approach is shown in Figure T Our main insight is that by continually occluding point clouds and learning a model c(-) to complete them, the weights of the completion model can be used initialization for downstream tasks (e.g., classification, segmentation). speeding up training and improving generalization over other initialization techniques.
Throughout we assume point clouds P are sets of points in 3D Buclidean space, P {pI, (p,,P2,... } where each point p; is a vector of coordinates (Ii, yi, z;) and features (e.g. color and normal). We begin by describing the components that make up our occlusion mapping o(-). Then we detail how to learn completion model c(-), giving pseudocode and the architectural details in appendix. Finally we discuss the criteria on validating the effectiveness of pre-training model for 3D point clouds.
2.1 GENERATING OCCLUSIONS
We first describe randomized occlusion mapping 0 P (where P is the space of all point clouds) from full point cloud an occluded point cloud P. We will do so by determining which points are occluded when the point cloud is viewed from particular camera position. This requires three steps: (1) projection of the point cloud (in a world reference frame) into the coordinates of camera reference frame; (2) Determining which points are occluded based on the camera view-points; (3) Mapping the points back from the camera reference frame to the world reference frame.
Viewing the point cloud from camera. A camera defines projection from 3D world reference frame into a distinctive 3D camera reference frame. It does so by specifying a camera model and camera view-point from which the projection occurs. The simplest camera model is the pinhole
2

---

Under review as a conference paper at ICLR 2021
camera, and view-point projection for it is given by a simple linear equation:
Team Veam Zeam
f
7
w/
r1
r2
r3
[1
0
8
h/ 2
F4
P5
t2
y
(1)
0
1
F7
T8
79
t3
1
2
intrinsic
nulation
translation
K
In the above, (I, 3/ z) are the original point cloud coordinates, the matrix including 7 and entries is the concatenation of 3D rotation matrix with a 3D translation vector, and the final matrix the left is the camera intrinsic matrix (f specifies the camera focal length, is the skewness between the and y axes in the camera, and w. h are the width and height of the camera image). Given these, the final coordinates (Feam, Meam; Feum) are the positions of the point in the camera reference frame. We will refer to the intrinsic matrix as and the rotation/translation matrix as [R[t].
Determining occluded points. We can think of the point (Zeam- Veam 2eam) ) in multiple ways: (a) as a 3D point the camera reference frame, (b) as a 2D pixel with coordinates (f camm Zcam= f yeam/ Zeam) with depth of 2cam- In this way, some 2D points resulting from the projection may be occluded by others if they have the same pixel coordinates, but appear at a larger depth. To determine which points occluded, we first Delaunay triangulation reconstruct polygon mesh from the points, and remove the points which belong to the hidden surfaces that are determined via z-buffering.
Mapping back from camera frame world frame. Once occluded points are removed, re- project the point cloud to the original world reference frame, via the following linear transformation:
P2
T3
f1
f/f
f/22
(
f2)/(2f2)
Feam
y'
F4
T5
T6
t2
0
1/f
h2(2
yeam
(2)
z
F7
78
Tg
f3
0
1
Zeam
0
0
0
1
0
0
0
1
1
[RIU]
K
Our randomized occlusion mapping o(-) is constructed follows. Fix initial point cloud P. Given a camera intrinsics matrix K, sample rotation/translation matrices [R]]t]].... di]..... [Ry[ty]] where V the number of views. For each view v [V], project into the camera frame of that view-point using eq. ( find occluded points and remove them, then map the rest back to the world reference
using eq. This yields the final occluded world frame point cloud for view-point v: P..
2.2 THE COMPLETION TASK
Given an occluded point cloud P produced by o(-), the goal of the completion task is to learn completion mapping c + from P to a completed point cloud P. We say that a completion mapping is accurate w.r.t. loss E(.,) if E.p ( ((c(P) P) + 0. The structure the completion model c(.) is an "encoder-decoder" network (Dai et al. 2017b: Yuan et al. 2018 Tchapmi et al. 2019 Wang et al., 2020a). The encoder maps an occluded point cloud to vector, and the decoder reconstructs the full shape. After pre-training, the encoder weights can be used as initialization for downstream tasks. appendix we gives pseudocode for OcCo and describes the architectures.
EXPERIMENTS
3.1 PRE-TRAINING AND DOWNSTREAM TRAINING DETAILS
We evaluate how OcCo improves the learning and generalization of a number of classification and segmentation tasks. Here we describe the details of training in each setting.
OcCo pre-training. For all experiments, will single pre-training dataset based Mod- elNet40 (Wu et al.) (013)). includes 12,311 synthesized objects from 40 object categories, di- vided into 9,843 training objects and 2,468 testing objects. To construct the pre-training dataset,

---

Under review as a conference paper at ICLR 2021
Car
Table
Person
Plane
Figure 2: Examples of self-occluded objects generated by method.
we generate occluded point clouds based on the training objects with a fixed camera intrinsics
{f 1000. 0, 1600, h 1200). 10 random selected viewpoints and zero translation.
Figure shows examples of the resulting occluded point clouds. Given these, we train an "encoder- decoder" style completion model c(-). For encoders, similar to prior completion models (Tchapmi
ct al. 2019 Wang et al. 2020al Wen et al. 2020a), we consider PointNet (Qi al. 2017a), PCN (Yuan
et al. 2018) and DGCNN Wang et al.) 2019b These networks encode an occluded point cloud into a dimeeniansi vector. We adapted the folding-based decoder from (Yuan et al. (001)) to complete the point clouds in two-stage procedure. We use the Chamfer Distance (CD) as loss function {(..). ). We use Adam (Kingma & Ba 2015) with an initial learning rate of le-4, decayed by 0.7 every 10 epochs to minimum value of le-6, for total of 50 epochs. We use batch size of 32 and set the momentum in the batch normalisation to be 0.9.
Few-shot learning. We use ModelNet40 and Syndey 10 (De Deuge et al. 2013) for "K "K-way N- shot" learning. During training. K classes are randomly selected and for each class we sample N random samples, then the model is tested on the same K classes As in Sharma & Kaul (020)), represent each object with 100 points. We use the same training settings as used in the next paragraph.
Object classification. We use three 3D object recognition benchmarks: ModelNet40, Scan- Net10 (Qnn et al., 2019) and ScanObjectNN (Uy et al., 2019); we describe them in the appendix. All objects are represented with 1024 points. We use the same training settings as the original works. Concretely, for PCN and PointNet, we use the Adam optimizer with an initial learning rate Ie-3, decayed by 0.7 every 20 epochs to a minimum value of 1e-5. For DGCNN, we use the SGD optimizer with a momentum of 0.9 and a weight decay of le-4. The learning rate starts from 0.1 and then reduces using cosine annealing Loshchilov & Hutter (2017) with minimum value of lc-3, We use dropout Srivastava et al. (2014) in the fully connected layers before the softmax output layer. The dropout rate of PointNet and PCN is set to 0.7, and is 0.5 for DGCNN. For all three models, we train them for 200 epochs with a batch size of 32. We report the results based on three runs.
Part segmentation. We use the ShapeNetPart (Armeni et al. 2016) benchmark for object part segmentation. This dataset contains 16,881 objects from 16 categories, and has 50 parts total. Each object is represented with 2048 points, and we use the same training settings as the original work.
Semantic segmentation. We the S3DIS benchmark (Armeni al. 2016) for semantic indoor scene segmentation. contains 3D scans collected via Matterport scanners in 6 different places, encompassing 271 rooms. Each point, described by a 9-dimensional vector (including coordinates, RGB values and normalised location), is labeled as one of 13 semantic categories (e.g. chair, table and floor). We use the same preprocessing procedures and training settings as the original work.
3.2 WHAT IS LEARNED FROM OcCo PRE-TRAINING
Alongside OcCo's ability to improve learning tasks we analyze the properties of the pre-trained representation itself. Here we describe the approaches we use for such analyses.
Visualisation of learned features. Feature visualisation OOla et al. 2017) is widely used to qualitatively understand the role of convolutional neural network unit. It links highly activated parts of a CNN channel with human concepts which have semantic meaning. Ideally the pre-training process leams disentangled features that useful to distinguish different parts of an object or scene. These learned features will be beneficial to not only few-shot learning, but also object recognition and part and scene segmentation.
4

---

Under review as a conference paper at ICLR 2021
Detection of semantic concepts. To quantitatively analyse the learned features of pre-training, adapt network dissection (Bau et al. 2017]] 2020) to determine the number of concept detectors in a pre-trained point cloud feature encoder. Specifically, for the k-th channel, first create binary activation mask M2 hased on highly activated point subsets. Since point cloud encoders usually learn each point feature either independently via neighborhood aggregation, the feature maps usually do not change in the vertical direction (see Figure 33. Therefore we can skip the retrieval step and directly quantify the alignment between an activation mask Mr and the n-th concept mask C, (i.e., object parts) via mean intersection of union (mloU) over a collection of point clouds Dp:
nCC
mioU (k.m) = Ep-Do
(3)
(M,(P) (P))
where ||
the set cardinality. mloU (A:a)) can be interpreted as how well unit i detects concept
Structural invariancelequivariance under SO(3) transformation. Pre-training should learn a representation that is robust under under rigid SO(3) transformations (i.e., rotation, translation, permutation). Although a single representation might vary after transformation, the cluster structures should be preserved. We use adjusted mutual information (AMI) (Nguyen al.. 2009) based on the clustering f and the ground truth label C, which prevents the score from monotonically increasing when the number of clusters increases,
I(9;C) E[(;C) AMI(2, C) = Ep (H( H(C))/2 [[[(2;C]]
(4)
where Q is the clustering determined by the learned embeddings Enc(-) and unsupervised clustering E,,
methods such as K-means. 1(S2; C) = E, P (wx nej)) log P(wc)P(c,) denotes the mutual information, H(-) is the entropy. AMI has a maximal of when two partitions are identical, and reaches a minimal of O if two clusters are total uncorrelated. calculated as:
P(wane))
== E=--pp.S-Sos [AMI(O, Enc(S(P)))
(5)
Once we finish the pre-training on ModelNet40,, first analyse the learned features and embeddings of the OcCo PointNet via the tests or probes deseribed above. Specifically, we examine the learned concepts of the pre-trained encoders that have top 20% highest values in the k-th unit of the feature, and the n-th concept mask C, is derived from the ground-truth annotations of the ni-th object parts. We ignore the object parts which have less than 100 points. We call k-th channel detector of concept when mloU (k.m) > 0.5.
SeapeNetPart. We assign activation mask Mr with points
We analyze the learned embeddings of the pre-trained encoders on ShapeNet10 and ScanObjectNN. We cluster the learned embeddings Enc(P) into Ω with K-means and calculate the AMI w.r.t labels. Since the encoders are permutation invariant, here we consider rotation, translation and jittering.
3.3 COMPLETION RESULTS AND PROBE TESTS
first visualize the features learned by OcCo PointNet on the objects from test split of Ned400 We visualize each learned feature by
Visualisation of learned features. In Figure 3
Points
T.Net ConviD
ReLU T-Mer (64)
MaxPool
Further Fine-Tuning
Part Segmentation
Input
(64)
Feat
Conv1D (128)
2
Corw1D
(1024)
Feat 3
Global
Scone Segmentation
Object ficttin
Few-Shot Learning
1024
128
1024
s3rd Feat
712"h Channel of Feat 3
F-NE Embedding
Figure 3: Visualisation on the learned features and embeddings of OcCo-initialised encoders. Above half illustrates the location of learned features in the architecture of PointNet (Qi al. 2017a).
5

---

Under review as a conference paper at ICLR 2021
4
Random Inibalisation
Feat1: 5
Feat2: 2
Feat3:
Jigsaw
Feat1: 14
Feat2: 14
Feal3: 16
4
Epochs
Feat1: 11
Feat2:
Feat3:
2
0
4
OcCo, 50 Epochs
Feat1:
Feat3:
2
0
Aero (4)
Bag (2)
Cap (2)
Car (4)
Chair (4)
Earphone (3)
Guitar (3)
Knlfe (2)
Lamp (4)
Laptop (2)
Motor (6)
Mug (2)
Pistal (3)
Rocket (4)
Skateboard (3)
Tabble (3)
Figure 4: Number of (unique) detected object parts in the feature maps of random, Jigsaw and OcCo-initialised PointNet. Digit in the bracket is the number of parts in that object category.
coloring the points according to their channel values. We find that, in early stage the encoder is able to learn low-level geometric primitives, i.e., planes, cylinders and cones, while later the network recognises more complex shapes like wings, leafs and upper bodies. We use t-SNE on embeddings of OcCo encoders based on ShapeNet10, distinguishable clusters are formed for different categories.
Number of concept detectors. In Figure we sketch the number of detected parts based on random, Jigsaw and OcCo (trained for 10 epochs and 50 epochs)-inittalised PointNet. We find that, while keeping the previously learned concepts, OcCo helps the encoder progressively detect more object parts as the training proceeds. We show that OcCo have outperformed prior methods in terms of total detected parts (numbers in legends). We provide visualisations in the appendix.
Table 1: Adjusted mutual information (AMI) under transformations. We reported the mean and std over 10 runs 'I 'T', 'R' stand for jittering, translation and rotation respectively.
Transformation
ShapeNetl0
ScanObjest
VEI
M2DP
Jigsaw
OcCo
VFII
M2DP
Jigsaw
OcCo
0.12 0.01
0.22.10.03
0.33100.4
0,51 0..03
0.05:10.02
0.18.10.22
0.2910.02
0.44:0.03
0.12
0.19110.2
0.32.10.22
0.45 L0.02
0.002
0.17.10.02
0.27:10.2
0.42 1..2
0.21100022
0.2910007
0.38
0.04:10.22
0.18100.3
0.2410000
0.07=0.03
0.20-:0.04
0.28=:0.3
0.35
0.04:00001
0.16::0.3
0.18::0.09
0.340.00
Dnvariance/qquivariinne under SO(3) transformation. We compare Jigsaw and OcCo- initialised PointNet encoder with two hand-crafted point cloud global descriptors: viewpoint feature histogram (VFH) (Rusu et al. 2010) and M2DP (He et al.) 2016) in Table [ Each point cloud is represented as vector, and use K-means for clustering, where K is set as the number of categories. We show that OcCo pre-training helps the networks to learn better embeddings of point cloud objects, especially when they are occluded and with outlier points (ScanObjeetNNN.
3.4 FEW-SHOT LEARNING
We use the same setting and train/test split as (Sharma & Kaul 2020)
(cTreee) and report the mean and standard deviation across 10 runs. The top half of the table reports results for eight randomly initialized point cloud models, while the bottom-half reports results on two models across three pre-training methods. We bold the best results (and those whose standard deviation overlaps the mean the best result). is worth mentioning (Sharma & Kaul 2020) pre-trained the encoders on both datasets before fine tuning, while only pre-trained once ModelNet40. The results show that models pre-traine reeetraieed with OcCo either outperform or have standard deviations that overlap with the best method in 7 out of settings.

---

Under review as a conference paper at ICLR 2021
Table 2: Few-shot classification accuracy
ModelNer00
Sydney10
Baseline
5-way
10 way
5. way
T0-way
10-shot
20-shor
10-abot
20-shot
10-shot
M0-shor
10-shor
3D-GAN
55,8:10.7
65 85.8999
40.3=66.5
48.41:566
54.2::466
58.8
36.016.22
45.3:779
FoldingNet
33.4 1:1.11
35.81:8.2
18.6::655
15.416.8
58.91:5.6
71.2:16.0
42.6133.4
63.5.1399
Latent- GAN
41.6.116.9
46.24119.7
32.9-9.2
25.549999
79.84344
50.54:3.0
PointCapsNet
42.3+174
13.0=18.77
38.02:44.3
27.24:149
69.446.3
70,5=488
44.1+22.0
60.3:4:999
PointNet++
38.5+16..0
42.4+142
23.1=77.0
18.8+5.
79.9+-6.8
85.0=5.3
55.442.22
63.4422.8
PoinCCNN
65.4:18.9
68.617.0
46.61:488
50.017.2
75.8:77.7
83.424.4
56.34:2.44
73.1::4.1
PointNet, Rand
52.0:12.2
57.8113.5
46.6:113.5
35.24:15.3
74.247..
82.2=3.1
$1441:3
$8.3::266
PointNet, <Tree
63.2+40.7
68.9+94
49.2-6.1
50.1+5.00
76.5+6.3
83.7=4.0
58.5+2.3
64.0-22.4
PointNet, OcCo
89.7+6.1
9244499
8399
89,7+4.66
77.748800
84.9-4499
60.9+3.77
65,5::5.5
DGCNN, Rand
31.6
4082:14
1999
16.9448
58.3266.6
76.1=13
48.12
76.12:30
DGCNN, Jigsaw
34.31-4.1
42.2+11100
26.0=7..5
29.9+8.2
62.5+6666
79.6=6.0
52.74333
69.1+22
DOCNN, <Thee
60.0:18.9
65.71884
48.5
53.0:24.11
86.2:14.44
90,9
66.2:12288
81.5:223
DGCNN, OcCo
92.5.6.0
86.517.11
86.4.1.77
63.312.77
77.6.139
3.5 OBJECT CLASSIFICATION RESULTS
We now compare OcCo against prior initialization approaches on object classification tasks. Table 3 compares DeCo-initialization random (Rand) and (Sauder & Sievers (019)'s (Jigsaw) initialization on various object classification datasets among different encoders. "MN40", "ScN10" and "SO15" stand for ModelNet40. ScanNet10 and ScanObjectNN respectively. Recall that O-Co-initializatton is pre-trained only on occlusions generated from the train split of ModelNet40. We color blue the best results for each encoder and bold in black the overall best result (and those whose standard deviation overlaps the mean of the best result) for each dataset. We show that OcCo DcCooinitialized models outperform all baselines. These results demonstrate that the OcCo-initialized models have strong transfer capabilities on out-of-domain datasets. We make more comparisons in the appendix.
Table 3: Comparison between OcCo, Jigsaw and Rand initialization on 3D object recognition
benchmarks. After confirming the scores from (Qi et al. 2017a Wang al.] 2019b; Uy al. 2019 Sauder & Sievers 2019) are reproducible, we reported the mean and standard error over three runs.
PointNet ligsaw
PCN
DOCNN Jigsaw
Dutaset
Jigsaw
MN40
89.2=0.1
89.620.1
90.140.1
89.320.1
89.6:1022
90,3102
92.5:10.4
92.31033
93.0:10.2
ScN10
76.9
77.2.10.2
78.0:10.2
77.0.1.33
77.910.3
78.2.10.3
76.1.10.7
77.8.10.5
78.
S015
73.5=0.5
76.3:00.4
80.0::0.2
78.3:0.3
78.2::0.1
80.4:50.2
82.4:0.44
52.74:088
83,9400
3.6 OBJECT PART SEGMENTATION RESULTS
Table[4compann OcCo- -ntiiiiiiiaiiztiinn to random and (Sauder & Sievers 2019)'s 's (Jigsaw) initialization on object part segmentation task. We show that OcCo-inttalized models outperform or match others in terms of accuracy and loU in all three encoders, demonstrating representations derived from completing occluded ModelNer40 improves the performance of part segmentation.
Table 4: Overall point prediction accuracy (mAcc) and mean intersection of union (mloU) on ShapeNetPart. We reported the mean and standard error based on three runs.
PointNet
PCN
DOCNN
Rand
Jissaw
OcCo
Rand
Jigsaw
OcCo
Rand
Jigsaw
OcCo
92.8.10.5
93.110033
93.4:044
92.3410.6
92.64055
92.21055
92.4005
94.410.4
mlol
62.2.114 1.4
8..2.116
83.4.1.
81.3115
81.11177
82.114
84.400.7
84.3
85.0
3.7
SEMANTIC SEGMENTATION
Here we compare random, Jigsaw and OcCo initialization on semantic segmentation task. We follow the same design of PointNet and DGCNN, use k-folld train-test procedure in (Armeni al. 2016). The results are reported in Table OcCo-initialized models outperform random and jigsaw- initialized ones, demonstrating that the pre-trained representations derived from completing occluded ModelNet40 brings improvements on segmenting indoor scenes which consist of occluded objects
We noticed that the random initialised/pre-trrii trriiid model in (Sauder & Sievers 2019 1oloU=40.3/41.22 did not achieve the similar results the original DGCNN (mloU=56.1)). They consider transductive setting which not directly comparable ours, so here stick the supervised setting and report reproduced scores.
7

---

Under review as a conference paper at ICLR 2021
ModelNet40, PointNet
ModelNett40, PCN
ScanObjectNN [OBJ_B)), DGCNN
Accuracy on Testt Set
0.00
Accuracy on Test Set
Accuracy, on Test Set
e.
OcCo
Jagsaw Random
Pasaw Random
84
Jgaa Random
76
16
Epochs
Epochs
Epochs
S3DIS, PointNet
ModelNeet0- 10%, DGCNN
ModelNet40- -0% DGCNN
Accuracy on Test Set
Accuracy on Tast Set
Accuracy on Tast Set
0.075
OcCo
0.312
Jigsaw Random
350
Random
Randam
Epochs
Epochs
Epochs
Figure 5: Learning curves of random, Jigsaw and OcCo, '10%' is the portion of used training data
Table Overall point prediction accuracy (mAce) and mean class intersection of union (mloU) on the S3DIS averaged across 6-cv-fold over three runs. OcCo encoders pre-trained on ModelNet40.
PointNer
PCN
DGONN
Rand
Jigsaw
OcCo
Rand
Jigsaw
OcCo
Rand
Jigsaw
OcCo
mAce
78.2.40.4
80. 100.7
820:00
82.::05
83,7404
85.1
837204
$4.1404
$4.6
miol.
47.0.10.8
1.66.1.1
54.9.06.
51.1114
52.21111
534.1.2
549112
58.6.10.8
58.01 1.0
3.8 LEARNING CURVES
We plot the learning curves for classification and segmentation tasks in Figure We observe that the models with OcCo initialization converge faster to better test accuracy than the random and sometimes Jigsaw-initialized models. For example, on ModelNet40 with PCN encoder, the OcCo-initialized model takes around 10 epochs converge, while the randomly initialized model takes around 50 epochs. Similarly, for ScanObjectNN with DGCNN encoder, the OcCo-initializee model converges around 20 epochs and better test accuracy than the random and Jigsaw-initializee initialized model.
RELATED WORK
4.1 DEEP MODELS FOR POINT CLOUDS
Work deep models for point clouds can largely be divided into three different structural approaches: (a) pointwise-basedd networks, (b) convolation-base networks, and (c) graph-based networks. We call the networks that independently process each point, before aggregating these point representations:
pointwise- networks (Qi et al. 2017ab
Seeph-Rivvii et al., 2019 Duan et al. 2019; Zhao al. 2019 Yang et al. 2019c Lin et al. 2019). One well-known method, PointNet, devises novel neural network that is designed to respect the permutation invariance of point clouds. Each point is independently fed into multi-layer perceptron, then outputs aggregated using permutation- invariant function (e.g., max-pooling) to obtain a global point cloud representation. Another class of
methods are convolution-basel networks (Hua et al. 2018, Su et al. 2018 Li al. 2018b] Atzmon
et al. 2018: Landrieu & Simonovsky 2018 Hermosilla et al. 2018) Groh et al. 2018 Rao et al. 2019 These works map point clouds to regular grid structures and extend the classic convolution operator to handle these grid structures. A representative model, PCNN (Atzmon et al.) 2018), defines two operators, extension and restriction, for mapping point cloud functions volumetric functions and vise versa. The third class of models is graph-based networks (Simonovsky & Komodakis 2017 Wang et al. 2019b) Shen et al.) 2018 Wang et al., 2018 Zhang & Rabbat] 2018 Chen al. 2019). These networks regard each point as a vertex of graph and generate edges based on spatial information and node similarities. A popular method is DGCNN (Wang et al.) 2019b). which introduces new operation, EdgeConv, to aggregate local features and graph update module to learn dynamic graph relations from layer to layer. NRS (Cao et al.) 2020) uses a neural random subspace method based the encoded embeddings to further improve the model performance.
8

---

Under review as a conference paper at ICLR 2021
4.2 PRE-TRAINING FOR POINT CLOUDS
Pre-training models on unlabelled data are gaining popularity recently due to its success on wide range of tasks, such natural language understanding (Mikolov al., 2013 Devlin al., 2018), object detection (He et al. 2020 Chen et al., (020)) and graph representations (Hu et al. 2020c d). The representations learned from these pre-trained models can be used as good initializer in downstream tasks, where task-specific annotated samples are scarce. The three most common pre-training objectives for point clouds are based on: (i) generative adversarial networks (GAN), (ii) autoencoders, and (iii) spatial relation (Sauder & Sievers 2019; [Sharma & Kaul 2020). However, GANs for point clouds are limited to non-point-set inputs, i.c., voxelized representations (Wu et al. 2016). 2D depth images of point clouds (Han et al. 2019), and latent representations from autoencoders (Achiioptas et al., 2018), as sampling point sets from a neural network is non-trivial. Thus these of point-sets. Autoencoders (Yang) ct al. 2018 Li et al. 2018a Hassami & Haley] 2019; Shi et al.] 2020) learn to encode point clouds into a latent space before reconstructing these point clouds from their latent representation. Similar to these methods, generative models based on normalizing flow ( (Yang et al. 2019b) and approximate convex decomposition (Gadelha et al., (020)) have been shown effective for the unsupervised learning on point clouds. However, both GAN and autoencoder-basee pre-training methods have been recently outperformed on downstream tasks by the pre-training technique of Sauder & Sievers (2019)
GAN approaches cannot leverage the natural
or Sharma & Kaul (2020) in few-shot setting.
These methods are based on spatial relation reconstruction, which aims to reconstruct points clouds given rearranged point clouds as input. To this end. [Sauder & Sievers (2019) equally split the 3D space into 3.3 voxels, rearrange 1.3 voxels and train a model to predict the original voxel label for each point. However, these random permutations destroy all spatial information that the model could have used to predict the true point cloud. Inspired by cover-trees (Beygelzimer et al.) 2006), /Sharma & Kaul (2020) utilised ball covers for hierarchical partitioning of points. They then train model to classify each point to their assigned clusters. However, the selection of the ball centroids is somewhat random and they need pre-train from scratch for each fine-tuning task. Instead,
method creates spatially realistic occlusions that a completion model learns to reconstruct. As such, this model learns how to naturally encode 3D object shape and contextual information. Recently there is new method called PointC (Xie et al. 2020b) which mainly uses contrastive learning for pre-training indoor segmentation models. Our method is more general and transferable compared with theirs.
Point cloud completion (Yuan et al. 2018) has received attentions in recent years. Most works aim at achieving lower reconstruction loss by incorporating 1) better encoder (Xie et al. 2020a
Huang et al. 2020 2) a better decoder (Tchapmi et al. 2019 Wen et al. 2020b); 3) cascaded
rehnement Wang al., 2020a) and 4) multi-viewed consistency Hu al. 2020b).
Completing 3D shapes for model initialisation has been considered before. Schönberger et al. (0118)
used scene completion (Song et al. 2017 Dai et al. 2020 Hou et al. 2020) as an auxiliary task to initialise 3D voxel descriptors for visual localisation. They generated nearly-complete and partial voxelised scenes based on depth images and trained a variational autoencoder for completion. They have showed that the pre-trained encoder is more robust under different viewpoints and weather conditions. We adapt this idea to pre-training for point clouds. We have shown that our initialisation is better than random and prior methods in terms of 1) object understanding; 2) invariance under transformations; and 3) downstream task performance.
5 DISCUSSION
In this work, we have demonstrated that why and how the Occlusion Completion (OcCo) learns the representations on point clouds that more transformation invariant, more accurate in few-shot learning, and in various classification and segmentation fine tuning tasks, compared to prior work. In future, would be interesting to design a completion model that is explicitly aware the view-point of the occlusion. A model like this would likely converge even quicker, and require fewer parameters, as this knowledge could act as a stronger inductive bias during learning. In general, we advocate for structuring deep models using graphical constraints as an inductive bias to improve learning.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas J. Guibas. Learning representa- tions and generative models for point clouds. In International Conference on Machine Learning. (ICML), 2018
Antonio Alliegro, Davide Boscaini, and Tatiana Tommasi. Joint supervised and self-supervised learning for 3d real-world challenges. arXiv preprint arXiv:2004.07339,, 2020. 2020.
Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, loannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In The IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2016.
Matan Atzmon, Haggai Maron, and Yaron Lipman. Point convolutional neural networks by extension operators. ACM Transactions on Graphics (TOG), 37(4):71:1-71:12, 2018.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role individual units deep neural network. Proceedings of the National Academy of Sciences. 2020.
Dimitri Bertsekas. distributed asynchronous relaxation algorithm for the assignment problem. In The IEEE Conference on Decision and Control, 1985.
Alina Beygelzimer, Sham Kakade, and John Langford. Cover trees for nearest neighbor. In Proceed- ings of the 23rd international conference on Machine learning (ICMI.), 2006.
Yun-Hao Cao, Jianxin Wu, Hanchen Wang, and Joan Lasenby. Neural random subspace. arXiv preprint arXiv:1911.0784 2020.
Chao Chen, Guanbin Li, Ruijia Xu, Tianshui Chen, Meng Wang, and Liang Lin. Clusternet: Deep hierarchical cluster network with rigorously rotation-invariant representation for point cloud analysis. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International Conference Machine Learning (ICML), 2020.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017a.
Angela Dai, Charles Ruizhongtai Qi, and Matthias Nießner. Shape completion using 3d-encoder- predictor cnns and shape synthesis. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017b.
Angela Dai, Christian Diller, and Matthias Nießner. Sg-nn: Sparse generative neural networks for self-supervised scene completion of rgb-d scans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Mark De Deuge, Alastair Quadros, Calvin Hung, and Bertrand Douillard. Unsupervised feature learning for classification of outdoor 3d scans. Australasian Conference on Robitics and Automation, 2013.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Yueqi Duan, Yu Zheng, Jiwen Lu, Jie Zhou, and Qi Tian. Structural relational reasoning of point clouds. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
10

---

Under review as a conference paper at ICLR 2021
Matheus Gadelha, Aruni RoyChowdhury, Gopal Sharma, Evangelos Kalogerakis, Liangliang Cao, Erik Learned-Miller, Rui Wang. and Subhransu Maji. Label-efficien learning on point clouds using approximate convex decompositions. In European conference computer vision (ECCV), 2020.
Fabian Groh, Patrick Wieschollek, and Hendrik Lensch. Flex-convolution (million-scall point-cloud learning beyond grid-worlds). arXiv preprint arXiv:1803.077 072880000079.,.. 2018.
Daniel Guo, Bernardo Ávila Pires, Bilal Piot, Mohammad Gheshlaghi Azar. Bootstrap latent- preettpredictive representations for multitask reinforce- ment learning. In International Conference Machine Learning (ICML), 2020.
Grill, Florent Altché, Rémi Munos, and
Timo Hackel, N. Savinov, L. Ladicky, Jan D. Wegner, K. Schindler, and M. Pollefeys. SEMAN- TIC3D.NET: A new large-scale point cloud classification benchmark. In ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, volume IV-1-W1, pp. 91-98, 2017.
Zhizhong Han, Mingyang Shang, Yu-Shen Liu, and Matthias Zwicker. View inter- inter-prediction gan: Unsupervised representation learning for 3d shapes by learning global shape memories to support local view predictions. In Proceedings of the AAAI Conference on Artificial Inielligence, 2019.
Kaveh Hassani and Mike Haley. Unsupervised multi-task feature learning on point clouds. In The IEEE International Conference Computer Vision (CVPR), 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Li He, Xiaolong Wang, and Hong Zhang. M2dp: A novel 3d point cloud descriptor and its application in loop closure detection. In JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2016.
Pedro Hermosilla, Tobias Ritschel, Pere-Pau Vázquez, Alvar Vinacua, and Timo Ropinski. Monte carlo convolution for learning on non-uniformly sampled point clouds. ACM Transactions on Graphics (TOG), 37(6):1-12, 2018.
Ji Hou, Angela Dai, and Matthias Nießner. Revealnet: Seeing behind objects in rgb-d scans. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
Qingyong HU, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020a.
Tao Hu, Zhizhong Han, and Matthias Zwicker. 3d shape completion with multi-view consistent inference. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020b.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovee. Strategies for pre-training graph neural networks. In International Conference Learning Representations (ICLR), 2020c.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2020d.
Binh-Son Hua, Minh-Khoi Tran, and Sii-Kit Yeung. Pointwise convolutional neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Zitian Huang, Yikuan Yu, Jiawen Xu, Feng Ni, and Xinyi Le. Pf-net: Point fractal network for 3d point cloud completion. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
11

---

Under review as a conference paper at ICLR 2021
Mor Joseph-Rivlin, Alon Zvirin, and Ron Kimmel. Momen (e) t: Flavor the moments in learning to classify shapes. In The IEEE International Conference on Computer Vision Workshops (CVPRW), 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with su- perpoint graphs. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Alex Lang. Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpil- lars: Fast encoders for object detection from point clouds. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self-organizing network for point cloud analysis. In The IEEE conference on computer vision and pattern recognition (CVPR), 2018 8a.
Yangyan Li, Rui Bu, Mingchao Sun, and Baoquan Chen. Pointenn. Advances in neural information processing systems (NeurIPS), 2018b.
Hongxin Lin, Zelin Xiao, Yang Tan, Hongyang Chao, and Shengyong Ding. Justlookup: One millisecond deep feature extraction for point clouds by lookup tables. In The IEEE International Conference on Multimedia and Expo (ICME), 2019.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. arXiv preprint 13X1v33301.3700 2013.
Xuan Vinh Nguyen, Julien Epps, and James Bailey. Information theoretic measures for clusterings comparison: is correction for chance necessary? In International Conference Machine Learning (ICML), 2009.
Christopher Olah, Ludwig Schubert, and Alexander Mordvintsev. Feature visualization. Distill, 2017.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In The North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (NAACL-HLT), 2018.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. The IEEE conference on computer vision and pattern recognition (CVPR), 2017a
Charles Ruizhongtai Qi, Li Yi, Hao Su. and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances neural information processing systems (NeurIPS), 2017b.
Can Qin, Haoxuan You, Lichen Wang, C.-C. Jay Kuo, and Yun Fu. Pointdan: A multi-scale 3d domain adaption network for point cloud representation. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Yongming Rao, Jiwen Lu, and Jie Zhou. Spherical fractal convolutional neural networks for point cloud recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Radu Bogdan Rusu and Steve Cousins. 3D) is here: Point Cloud Library (PCL.. In The IEEE International Conference Robotics and Automation (ICRA), 2011.
Radu Bogdan Rusu, Gary Bradski, Romain Thibaux, and John Hsu. Fast 3d recognition and pose using the viewpoint feature histogram. In /EEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2010.
12

---

Under review as a conference paper at ICLR 2021
Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by reconstructing space. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Johannes L Schönberger, Mare Pollefeys, Andreas Geiger, and Torsten Sattler. Semantic visual localization, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Charu Sharma and Manohar Kaul. Self-supervised few-shot learning on point clouds. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Yiru Shen, Chen Feng, Yaoqing Yang, and Dong Tian. Mining point cloud local structures by kernel correlation and graph pooling. In The IEEE conference on computer vision and pattern recognition (CVPR), 2018.
Yi Shi, Mengchen Xu, Shuaihang Yuan, and Yi Fang. Unsupervised deep shape descriptor with point distribution learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Martin Simonovsky and Nikos Komodakis. Dynamic adge-conditionel filters convolutional neural networks graphs. In The IEEE conference computer vision and pattern recognition (CVPR), 2017.
Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):192--) 1958, 2014.
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsoo Hsuan Yang, and Jan Kautz. Splatnet: Sparse lattice networks for point cloud processing. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Lyne P. Tchapmi, Vineet Kosaraju, Hamid Rezatofighi, Ian Reid, and Silvio Savarese Topnet: Struc- tural point cloud decoder. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In IEEE International Conference on Computer Vision (ICCV), 2019.
Bernic Wang, Virginia Wu, Bichen Wu, and Kurt Keutzer. Latte: accelerating lidar point cloud anno- tation via sensor fusion, one-click annotation, and tracking. In The IEEE Intelligent Transportation Systems Conference (ITSC), 2019a.
Chu Wang, Babak Samari, and Kaleem Siddiqi. Local spectral graph convolution for point set feature learning. European conference on computer vision (ECCV), 2018
Xiaogang Wang, Marcelo H. Ang, and Gim Hee Lee. Cascaded refinement network for point cloud completion. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020a.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay Sarma, Michael M Bronstein, and Justin Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions on Graphics (TOG), 38(5): 1-12, 2019b.
Yue Wang, Alireza Fathi, Abhijit Kundu, David Ross, Caroline Pantofaru, Tom Funkhouser, and Justin Solomon. Pillar-based object detection for autonomous driving. In European Conference on Computer Vision (ECCV), 2020b.
Xin Wen, Tianyang Li, Zhizhong Han, and Yu-Shen Liu. Point cloud completion by skip-attention network with hicrarchical folding. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020a.
13

---

Under review as a conference paper at ICLR 2021
Xin Wen, Tianyang Li, Zhizhong Han, and Yu-Shen Liu. Point cloud completion by skip-attention network with hierarchical folding. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020b.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning probabilis- tic latent space of object shapes via 3d generative-adversaiial modeling. In Advances in neural information processing systems (NeurIPS), 2016.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In the IEEE conference computer vision and pattern recognition (CVPR), 2015.
Haozhe Xie, Hongxun Yao, Shangchen Zhou, Jiageng Mao, Shengping Zhang, and Wenxiu Sun. Grnet: Gridding residual network for dense point cloud completion. In European conference computer vision (ECCV), 2020a.
Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas J Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. European conference on computer vision (ECCV), 2020b.
Bo Yang. Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, and Niki Trigoni. Learning object bounding boxes for 3d instance segmentation on point clouds. In Advances in Neural Information Processing Systems (NeurIPS), 2019a.
Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019b.
Jiancheng Yang, Qiang Zhang, Binghing Ni, Linguo Li, Jinxian Liu, Mengdie Zhou, and Qi Tian. Modeling point clouds with self-attentnn and gumbel subset sampling. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019c.
Yaoqing Yang, Chen Feng. Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 206-215, 2018.
Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pen: Point completion network. 2018 International Conference on 3D Vision (3DV), pp. 728-737. IEEE, 2018.
Yingxue Zhang and Michael Rabbat. A graph-enn for 3d point cloud classification. In The IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2018.
Hengshuang Zhao, Li Jiang, Chi-Wing Fu, and Jiaya Jia. Pointweb: Enhancing local neighborhood features for point cloud processing. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Qian-Y Yi Zhou and Ulrich Neumann. Complete residential urban area reconstruction from dense aerial lidar point clouds. Graphical Models, 75(3):118 18- 2013.
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
SM Zolanvari, Susana Ruano, Aakanksha Rana, Alan Cummins, Rogerio Eduardo da Silva, Morteza Rahbar, and Aljosa Smolic. Dublincity: Annotated lidar point cloud and its applications. In British Machine Vision Conference (BMVC), 2019.
14

---

Under review as a conference paper at ICLR 2021
A DESIGN OF THE COMPLETION MODEL
Previous point completion models Dai et al. 2017b] Yuan et al. 2018; Tehapmi et al. 2019 Wang al. 2020a) all use an "encoder-decoder" architecture. The encoder maps partial point cloud to vector of a fixed dimension, and the decoder reconstructs the full point cloud.
In the OcCo experiments, remaining architecture as the encoder to map partial point cloud into a 1024- -24-niimnns vector. We adapt the folding-basse based decoder design from PCN, which is a two-stage point cloud generator that produces coarse and fine-grained output point cloud (Ycoarse; Yine) for each input. We removed all the batch normalisation layers in the folding-based decoder since we find brings negative effects in the completion process in terms of Chamfer distance loss and convergent speed. On the basis of prior self-supervised learning methods, SimCLR (Chen al. 2020), MoCo (He et al., 2020) and BYOL (Guo et al. 2020), we find the batch normalisation important in the encoder yet harmful for our decoder. Also, we find the L2 normalisation in the Adam optimiser is undesirable for completion training but brings improvements the downstream fine-tuning tasks.
exclude the last few MLPs of PointNet and DGCNN, and use the
The predicted coarse point cloud Yeaarse, which represents the global geometry of shape, is generated via a set of fully connected layers. folding- buding-bassed generator is used to predict the local fine structures of each point in Yeoarse: this results in Yjine- The folding based structures is proved to be good at approximating a smooth surface which reflects the local geometry. During training, Y Ycoorse and Y fine are generated via randomly sampling 1024 and 16384 points from the mesh, respectively.
We use either Chamfer Distance (CD) or Earth Mover Distance (EMD) as the loss function for the completion model. We use a normalised and symmetric (thus commutative) version of Chamfer Distance (CD) to quantify the differences between two point clouds P and P:
CD(P, P)
Σ SEP
min l ZEP
x|12 +
min || Σ
2112-
(6)
IPI
PI
ZEP BEP
Note that it is no need that the two point cloud P and P have the same size. But when calculating the Earth Mover Distance (EMD), P and P are usually required to have the same number of points:
EMD(P (P, P) = min P REP
Σ 118-- -
where á is bijection between points in P and P. Note that EMD is not commutative. Since finding the optimal mapping é is quite time consuming, we use its approximation form Bertsekas (1985).
The loss of the completion task is adaptive weighted
of coarse and fine generations:
d1 (Y Yeoarse; Yccarse) + a * fiiens Yfine),
where the step-wise trade-off coefficient a incrementally grows during training. In our experiments, we find that even with approximation, it is still suboptimal to use EMD for d2, since it is inefficient to solve the approximate bijection mapping for over 16k point pairs. We evaluate both 'EMD+CD' and 'CD+CD' combinations for the loss I. We have found that OcCo with 'EMD+CD' loss has achieved comparable performance yet longer time in the downstream classification tasks compared with the "CD+CD'. We use 'CD+CD' as the loss function in the OcCo pre-training process described in Section. 3.1 in terms simplicity and efficiency.
B QUALITATIVE RESULTS FROM OcCo
In this section, we show some qualitative results of OcCo pre-training by visualising the input, coarse output, fine output and ground truth at different training epochs and encoders. In Figure. Figure. 7 and Figure. 8 we notice that the trained completion models are able to complete even difficult occluded shapes such as plants and planes. In Figure. 9 we plot some failure examples of completed shapes, possibly due to their complicated fine structures, while is worth mentioning that the completed model still completed these objects under the same category.
15

---

Under review as a conference paper at ICLR 2021
Ther
glu soo
cone
on shoa
gwadnn
ga anen
Figure 6: OcCo pre-training with PCN encoder on occluded ModelNet40.
epc
cana aou
the Mpa
geaanan
WETH
Figure 7: OcCo pre-training with PointNet encoder on occluded ModelNet40,
16

---

Under review as a conference paper at ICLR 2021
Vhen
Tworal
Figure 8: OcCo pre-training with DGCNN encoder occluded ModelNet40.
the
Tee OON
Tel
Figure 9: Failure completed examples during OcCo pre- mraiiinnn
17

---

Under review as a conference paper at ICLR 2021
C
VISUALISATION OF PART ACTIVATION MAP AND GROUND TRUTH
brtth activation map actruatton
53 Unit Feat1, Part: 14, mloU 0.627
41 Unit in Featl, Part: 12, mloU 0.606
2nd Unit Feat1, Part: 26, mlou 0.667
31a Unit in Feat1, Part: 39, mloU 0.635
63dd Unit in Feat2, Part: 17, mloU = 0. 584
10th Unit in Feat2, Part: 20, mlaU = 0.603
actoetton mer
99211 sotivalion 
657 Unit in Feat3, Part: 9, mloU 0. 510
1018th Unit in Feat3, Part: 44, mloU = 0. 619
Figure 10: Learned features OcCo pre-training
D ANALYSIS ON THE EFFECTS OF OcCo PRE-TRAINING DATASETS
We compare the occluded datasets based on ModelNet40 and ShapeNet8 for the OcCo pre-training We construct the ModelNet Occluded using the methods described in Section 2 and for ShapeNet Occluded we directly use the data provided in the PCN, whose generation method are similar but not exactly the same with ours. Basic statistics of these two datasets are reported in Table 6 Compared with the ShapeNet Occluded dataset which is publicized by PCN and used all Sillo-uppsTehapmtt et al. 2019 Wang et al. 2020a), categories, more view-points, more points per object and therefore is more challenging. We believe such differences will help the encoder models lear more comprehensive and robust representation which is transferable to downstream tasks. To support our idea, we perform OcCo pre-training on these datasets respectively, and test their performance ModelNet40 and ShapeNet Occluded classification benchmarks.
occluded ModelNet dataset has more object
Table 6: Statistics of occluded datasets for OcCo pre-trainng
Name
of Class
#f Object
# Views
of Pointss/OObject
ShapeNet Occluded (PCN)
30974
1045
NetelNet Occluded (OcCo)
40
12304
10
20085
18

---

Under review as a conference paper at ICLR 2021
The reason of choosing these two datasets for benchmarking is, ShapeNet Occluded is the out-of- domain data for the models pre-trained on ModelNet Occluded, and vice versa. We believe will give sufficient information which occluded dataset should be preferred the OcCo pre-training The Results are shown Table 7
Table 7: Performance of OcCo pre-trained models with different pre-trained datasets
OcCo Settings
Classification Accuracy
Encoder
Pre- Trained Dataset
ModelNet Oc
ShapeNet Oc
ShapeNet Oc
81.0
94.1
PointNet
ModelNet Oc
85.6
95.0
ShapeNet Oc
81.6
94.4
PCN
ModelNet Oc
85.1
95.1
ShapeNet Oc
86.7
94.5
DGCNN
ModelNet Oc
89.1
95.1
we see that the OcCo models pre-trained on ShapeNet Occluded do not perform as well as the ones pre-trained on ModelNet Occluded in most cases. Thus in our experiments, we reports the results pre-trained on ModelNet Occluded.
From Table 7
By visualising the objects from the ShapeNet Occluded (in Figure. 11 we believe this performance deficiency in downstream fine-training of pre-trained models is due the quality of the generated occluded point clouds (in comparison with generated dataset shown in Figure. 2 Further, think our dataset is more challenging task for all the present completion models.
Airplane
Chair
Car
Figure 11: Examples from ShapeNet Occluded which fail to depict the underlying object shapes
E DETAILED RESULTS ON TRAINING A LINEAR SVM FOR CLASSIFICATION
We first described the benchmark datasets that are used for classification in Table g
Table 8: Statistics of classification datasets
Name
Type
# Class
Training
Testing
(MN40) ModelNet40 (Wu et al. 20155) synthesized
40
9,843
2,468
(SNI0) ScanNet10 (Dai al. (2017))
real scanned
10
6,110
1,769
(SO15) ScanObjectNN (Uy et al. 2019)
real scanned
2,304
576
To make a comprehensive and convineing comparison, we follow the similar procedures from (Achlioptas et al. 2018 Han et al. 2019) Sauder & Sievers 2019 Wu et al., (2116] Yang et al. 2018)) OcCo encoders that are pre-trained on occluded objects from ModelNet40, For all six classification datasets, fit a linear SVM on the output (024-dimensional embeddings of the train split and evaluate it on the test split. Since Sauder & Sievers (209) have already proven their methods are better than the prior, here we only systematically compare with theirs. We report the results in Table 9 we can that all OcCo models achieve superior results compared the randomly sundomly-inttalizzed initialized counterparts, demonstrating that OcCo pre-training helps the generalisation both in-domain and cross-domain.
to train línear Support Vector Machine (SVM) to examine the generalisation
"I our implementation, we also provide an alternative to use grid search to find the optimal set of parameters for SVM with a Radial Basis Function (RBF) kernel. In this setting, all the OcCo pre-trained models have outperformed the random initialised and Jigsaw pre-trained by large margin well.
19

---

Under review as a conference paper at ICLR 2021
Table linear SVM on the output embeddings from random, Jigsaw and OcCo initialised encoders
PointNct
PCN
DGCNN
Dataset
Rand
Jigsaw
OcCo
Rand
Jigsaw
OcCo
Rand
Jigsaw
OcCo
ShapeNet10
91.3
91.1
93.9
88.5
91.8
94.6
90.6
91.5
94.5
ModelNet40
70.6
87.5
88.7
60.9
73.1
88.0
66.0
84,9
89.2
ShapeNet Oc
79.1
86.1
91.1
72.0
87.9
90.3
78.3
87,8
91.6
ModelNet
65.2
70.3
55.3
65.6
83.3
72.8
82.2
ScanNet10
64,8
64.1
67.7
62.3
66.3
75.5
61.2
69.4
71.2
ScanObjectNN
45.9
55.2
69.5
39.9
49.7
72.3
43.2
59.5
78.3
F RE-IMPLEMENTATION DETAILS OF "JIGSAW" PRE-TRAINING METHODS
In this section, we describe how we reproduce the 'Jigsaw' pre-training methods from (Sauder & Sievers, 2019). Following their description, we first separate the objects/chopped indoor scenes into 3'= = 27 small cubes and assign each point label indicting which small cube it belongs to. We then shuffle all the small cubes, and train a model to make a prediction for each point. We reformulate this task as 27-class semantic segmentation, for the details the data generation and model training, please refer to our released code.
G MORE COMPARISONS
In Table TO we compare OcCo with prior point-cloud-specific pre-training methods (Alliegro et al. 2020). Our method obtains the best results all settings. These results confirm that the inductive bias learned by reconstructing occluded point clouds is stronger than one based in reconstructing permuted clouds (Alliegro al. 2020 Sauder & Sievers 2019). Specifically,
believe that because OcCo does not rearrange object parts but instead creates point clouds that resemble real-world 3D sensor occlusions, the initialization better encodes realistic object shape and context.
Table 10: Accuracy comparison between OcCo and prior pre-training baselines Alliegro 2020) on 3D object recognition benchmarks. ModeINer40-20% means only 20% of training data are used.
Baseline
Dataset
Rand
Alliegro et (2020)
OcCo
ModelNet40
89.2
89.7
90.2
PointNet
ModeINet40-200
82.9
83.1
83.6
ScanObjectNN (OBJ BG)
73.7
71.3
80.2
H LABELLED SAMPLE EFFICIENCY
We investigate whether OcCo pre-training can improve the labelled sample efficiency of downstream tasks. Specifically, we reduce the labelled samples to 1%, 5%, 10% and 20% of the original training set for the ModelNet40 object classification task, and evaluate on the full test set. As shown in Table models,
I1 OcCo-initialized models achieve superior results compared to the randomly-initialie demonstrating that OcCo with in in-domain pre-training improves labelled sample efficiency.
Table 11: Sample efficiency with randomly-initialized and OcCo-initialized models.
PointNet Jigsaw
PCN Jigsaw
DGCNN
Baseline
Rand
OcCo
Rand
OcCo
Rand
Jigsaw
OcCo
1%
56.9
55.7
58.1
$7.8
$9.6
60.4
60.0
59.9
60.5
5%
73.9
74.3
74.9
73.2
75.8
76.7
79.4
79.2
79.7
10%
80.6
81.3
81.1
81.1
82.1
82.6
84.4
84.4
84.5
20%
83.6
84.2
84.2
83.6
84.2
84.4
86.5
86.7
87.2
20

---

Under review as a conference paper at ICLR 2021
I NETWORKS AND TRAINING SETTINGS OF PCN ENCODER
We sketch the network structures of PCN encoder and output layers for downstream tasks in Figure
PCN Encoder
Inpuu Points
Convio [128)
(256)
MaxPool
Tile
Conv10 151
RetU [Feat 4]
Feat
Foot
Goobal
ComvID [1024] [Fent5]
x 1024
MaxPool
DX3
x 128
X 512
CLS Output Layer
SemSeg Output Layer
MLP [312,
Global Global
ConviD (512) RetUI
Scores
F
n X Scores
Softax
Global
1280
Conv2D (k) Softmax
PartSeg Output Layer
Co RetU
Conv1D (256) RetU
Fest 5
Scores
Coeu1D (128)
5264
Global
Comv1D (2) Softmax
Figure 12: Encoder and Output Layers of PCN
J DETAILED RESULTS OF THE PART SEGMENTATION
Here in Table 12 we report the detailed scores on each individual shape category from ShapeNetPart, we bold the best scores for each class respectively. We show that for all three encoders, OcCo- initialisation has achieved better results over two thirds of these 15 object classes.
Table 12: Detailed Results on Part Segmentation Task on ShapeNetPart
PointNet
PCN
DGCNN
Shapes
Rand
Jigsaw
OcCo
Rand
Jigsaw
OcCo
Rand
Jigsaw
OcCo
mean (point)
83.7
83.8
84.4
82.8
82.8
83.7
85.1
85,5
Acro
83.4
83.0
82.9
81.5
82.1
82,4
84.2
84.1
84.4
Bag
78.7
79.5
77.2
72.3
74.2
79.4
83.7
84.0
77.5
Cap
82.5
82.4
85.5
67.8
86.3
84.4
83.4
Car
74.9
76.2
75.6
71.8
71.3
73.9
77.1
77.0
77.9
Chair
89.6
90.0
90.0
88.6
88.6
90.0
90.9
90.9
91.0
Earphone
73.0
69.7
74.8
69.2
69.1
68.8
78.3
75.2
Guitar
91.5
91.1
90.7
90.0
89.9
90.7
91.5
91.5
91.6
Knife
85.9
86.3
88.0
84.0
83.8
85.9
87.3
87.0
88.2
Lamp
80.8
80.7
81.3
78.5
78.8
80.4
82.9
83.2
83.5
Laptop
95.3
95.3
95.4
95.3
95.1
95.6
96.0
95.8
96.1
Motor
65.2
63.7
65.7
64.1
64.7
64.2
67.8
71.6
65.5
Mug
93.0
92.3
91.6
90.3
90.8
92.6
93.3
94.0
94.4
Pistol
81.2
80.8
81.0
81.0
81.5
81.5
82.6
82.6
79.6
Rocket
57.9
56.9
58.2
51.8
51.4
53.8
59.7
58.0
Skateboard
72.8
75.9
74.2
72.5
71.0
73.2
75.5
77.9
76.2
Table
80.6
80.8
81.8
81.4
81.2
81.2
82.0
81.8
82.8
21

---

Under review as a conference paper at ICLR 2021
K ALGORITHMIC DESCRIPTION OF OcCo
Algorithm Occlusion Completion (OcCo)
P: an initial point cloud
camera intrinsic matrix
V: number of total view points
loss: a loss function between point clouds
encoder-decoder complction model
p: cownstream prediction model
while V:
sample a random viex-point
R [random. rotation (),, random.transiatan translation ()
map point cloud to camera reference frame P_cam dot (K, dot (R_t, P))
create occluded point cloud P_cam_oc occlude (P_cam, alg='z=bufferingg
point cloud back to world frame Kiinv = [inv (K), zeros 1 zeres (), 1] R_t_inv transpose (RR_;; zercs (3,1), 1]) PLoc dot (R_t_inv, dot (Kvi P_cam_oc))
complete point cloud c.decoder (c.encoder (P_oC))
compute loss, update via gradient descent = [P_c, P)
1.backward
update (c. params) +
deanstream tasks, use pre-trained encoders D.initialize(c. p. www.eeee..aaaaaeeee p.train()
22


Reviews:
[{"review_id": "qFSx63oIHKN", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The idea of this paper is simple but fascinating. Actually there are many studies concerning the task of point cloud completion, but using it as the initialization approach to improve the other tasks is quiet novel. The experimental results seem solid and quantitatively prove the effectiveness of the OcCo-initialization.\n\nSince the idea itself is simple enough, the reviewer would not argue too much about the technical contribution of this paper, but concerns more about the analytic contributions. The reviewer’s concerns are as follows.\n1. In addition to verify the effectiveness of OcCo-initialization, more analysis on why this simple idea can take effect should also be given. For example, the author should go deeper to explain why OcCo-initialized PointNet can outperform the random initialized PointNet (e.g. by visualizing the learned features of the two kinds of PointNet, like Figure 3).\n2. The idea of OcCo-initialization can be concluded as some kind of task oriented initialization approach. Considering the simplicity of this idea, similar initialization strategy can be formulated, such as pre-training network on segmentation task and apply them on classification task. So why the author only chooses the completion task as the initialization strategy, or if task oriented initialization can be considered as a universal strategy in point cloud processing?\n3. Although the experimental results look solid, the reviewer still concerns if the proposed OcCo-initialization can achieve the SOTA results or close enough to the current SOTA. For example, PointCNN can achieve much better segmentation results compared to the methods in Table 3. So if the OcCo-initialization can still succeed in the PointCNN which has better ability of learning point cloud features?\n4. The author is advised to clarify the necessity of Sec 2.1, which is the main part of the model description. In reviewer’s opinion, a method to generate partial point cloud from single view is essentially not a technical contribution for this paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "X2KCXHb2f_v", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the problem of training networks for point cloud processing through a point cloud completion task. Given a point cloud, it is rendered from a set of viewpoints and for each viewpoint the set of visible points is determine. A network is then trained to generate the full point cloud from the partially observed point cloud for a given view. Here, an encoder-decoder architecture is used, where the encoder corresponds to the network that should be pre-trained. Experimental results show that the proposed method outperforms two baselines for three tasks (object classification, object part segmentation, and semantic segmentation), when using less training data, and that the pre-training on the occlusion task leads to faster convergence.\n\nOn the positive side, the occlusion completion (OcCo) task is clearly described and it should be fairly straightforward for a researcher to setup this task. The task requires no human annotation and is thus suitable for pre-training from large datasets captured in uncontrolled settings. The experiments cover a wide range of tasks and settings and show that the OcCo pre-training strategy outperforms random initialization and the approach from Saunders & Sievers. Here, the simplicity of the OcCo task coupled with its performance is clearly a major strength of the paper. In particular, Fig. 4 shows that the networks pre-trained on the OcCo task tend to converge much faster compared to the baselines.\n\nOn the negative side, I feel that the paper oversells the novelty of the OcCo task. The OcCo task is a variation of the (semantic) scene completion task that asks to complete a partial observation of a scene / object and is receiving attention in the computer vision community. Recent examples include [Dai et al.,  SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans, CVPR 2020] and [Hou et al., RevealNet: Seeing Behind Objects in RGB-D Scans, CVPR 2020], with older works including [Firman et al., Structured prediction of unobserved voxels from a single depth image, CVPR 2016]. [Schönberger et al., Semantic Visual Localization, CVPR 2018] use (semantic) scene completion as a proxy task to train a 3D descriptors for 3D-3D matching between models. Given a voxelized partial observation of a scene, they train an encoder-decoder architecture to predict the complete volume (potentially also predicting semantic labels for each voxel). The embedding in the latent space are then used as 3D descriptors (i.e., the decoder part of the network is not needed at test time). In other words, they use the OcCo task for training their networks. They show that the learned representation generalizes between datasets and sensor modalities (training on 3D data obtained from stereo images, tested on LiDAR data). Given this result, I see limited novelty in using the OcCo task for pre-training point cloud networks and it does not seem very surprising that pre-training on the OcCo task should result in meaningful representations.\n\nMy second main point of criticism is the level of detail of the experimental evaluation. While the experiments cover a wide range of tasks and settings, I feel that crucial information needed to understand the results are missing:\n1) Is the same dataset (ModelNet40) used to pre-train on the OcCo task also used to pre-train the JigSaw approach from Sauder & Sievers? Unfortunately, no details on the latter are provided in the main paper (or I was not able to find them), making it hard to understand how meaningful the comparison is.\n2) There are no details on how the networks are trained for the different tasks, e.g., for how many epochs are the network trained for the task at hand?, do the networks converge for all pre-training strategies?\n3) How significant are the improvements over the two baselines. For most considered settings, the improvements seem rather small, e.g., often less than 1 point compared to the Rand baseline in Tab. 2 and 3. Is this a meaningful improvment? Or would simply using a different random seed for training explain such a difference? Given that the paper claims that \"These results demonstrate that the OcCo-initialized models have strong transfer capabilities on out-of-domain datasets\" and that \"OcCo-initialized models achieve superior results compared to the randomly-initialized models\", this is an important question to answer.\nI think there is enough space in the paper to include this information. Specifically, I do not think that Alg. 1 is necessary in the main paper (but would be good to have as an appendix) as the text and Fig. 1 already describe the approach in sufficient detail. Similarly, the z-Buffer algorithm is a classic computer graphics technique that is covered in basic lectures and does not need to be discussed in detail (e.g., see [Pittaluga et al., Revealing Scenes by Inverting Structure from Motion Reconstructions, CVPR 2019] briefly mentioning z-Buffering and the use of Delaunay triangulation for determining visibility). I think this space could be spend on providing more details.\n\nIn the current form, I do not think the paper is ready for publication as the paper, in my opinion, overclaims its contributions, misses relevant work (see also below), and misses crucial details necessary to understand the experimental results. As such, I am currently recommending to reject the paper. I believe that these issues can be addressed, but I would base my final recommendation based on the authors' feedback.\n\nHere are additional detailed comments:\n* In Sec. 2.1, I do not understand the comment \"Our goal is to learn a randomized occlusion mapping o : P → P (where P is the space of all point clouds) from a full point cloud P to an occluded point cloud P\". As far as I can tell, the mapping is not learned but follows a fixed pipeline.\n* I don't understand how Eq. 2 \"most closely approximates the inverse of eq. (1)\". Eq. 2 is the inverse of Eq. 1. The only approximation that I could see if the 2D projection coordinates are rounded to the nearest integer.\n* The introduction teases with the statement \"Current 3D sensing modalities (i.e., 3D scanners, stereo cameras, lidars) have enabled the creation of large repositories of point cloud data (Rusu & Cousins, 2011; Hackel et al., 2017).\" However, only synthetic data is used for pre-training, which is a bit disappointing. I am not convinced that the synthetic datasets \"are qualitatively similar to point clouds in datasets where points are collected via 3D imaging devices such as handheld scanners (Dai et al., 2017a; Armeni et al., 2016) and lidar (Geiger et al., 2012).\" Based on my experience, handheld scanners (RGB-D cameras or (multi-view) stereo cameras) produce much more noisy measurements with outliers while lidar sensors, especially for autonomous vehicles, typically produce sparser point clouds.\n* Sec. 3.2 states that \"We observe that, in early stage the encoder is able to learn low-level geometric primitives, i.e., planes, cylinders and cones, while later the network recognises more complex shapes like wings, leafs and upper bodies (non-rigid).\" I am not sure how I see this in Fig. 3 since I don't know what the color-coding signifies.\n* Looking at Fig. 3, I am not sure whether the statement that \"clearly separable clusters are formed for different object classes\" is true. There seems to be quite some overlap between classes, but this is also a bit hard to tell given the small size of the figure.\n* [Yang et al., PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows, ICCV 2019] and [Gadelha et al., Label-Efficient Learning on Point Clouds using Approximate Convex Decompositions, ECCV 2020]  both propose generative models for point clouds. Both of them show that they can be used for unsupervised representation learning and they show competitive results for the task of only training an SVM classifier on top of the learned representation for ModelNet. Both should be discussed in the related work.\n\n### After rebuttal phase ###\nThe comments by the authors and the revised version of the paper successfully address my concerns. I thus recommend to accept the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1mCH5gO3d3O", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose completing an occluded point cloud as a pretraining step for point cloud processing methods. Multiple occlusions are generated for the network to complete by simulating a camera perspective. \n\nPros: \n- First work analyzing this specific pretraining for point clouds\n\nCons:\n- Weak novelty\n- Limited experimental reliability\n\nOverall, the novelty of the paper seems rather weak. The only novel contribution is the idea of point cloud completion as a pretraining task. This idea is rather simple and similar techniques are well known and used in other fields such as NLP. Hence, it does not represent a significant methodological advancement.\nNevertheless, the paper would still be interesting if it showed extraordinary results in this particular field of application. Unfortunately, the experimental results seem weak as well. They show modest gains with respect to existing techniques and they lack any information on run-to-run variance. This makes it impossible to understand if the gains that are shown are statistically significant or just lucky runs with careful parameter tuning. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "-gdQfih288t", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a better pre-trained prior for a variety of downstream applications in point cloud analysis. The workflow of the pre-training mechanism is to first 1) generate occluded points that result from view occlusion and then 2) optimize the encoder to learn how to complete the occluded points from the partial point cloud. In downstream applications, the obtained encoder will be used as the initial weights in the network training. Empirical experiments have shown that such a pre-train mechanism can improve initialization over prior baselines and benefit a variety of tasks even with a large domain gap.\n\nPros:\n1. The experimental results have shown a steady improvement in performance by using the proposed pre-training approach in different encoder architectures and different downstream applications. That provides strong support for validating the effectiveness of the proposed approach.\n2. I also like the result that the initialization is only pre-trained on the occlusions generated from the ModelNet40 but still work in another dataset. And yet, the pre-training is done in a self-supervised manner. This is a great plus for this approach as it indicates that it could be a general-purpose booster for a wide range of applications without spending too much effort in collecting special-purpose dataset for pre-training. \n3. The paper is well written and presented.\n\nCons:\n1. The improvement, as shown in the statistics, is very incremental in most cases. I understand it is difficult to achieve better results on well-established benchmarks, but it somehow indicates the improvement is limited.\n2. Though the paper already stated some nice explanation of the idea behind this approach, I would appreciate it if a more in-depth analysis of why such a pre-training mechanism could work is provided. Specifically, I would like more analysis of why such a pre-training method can adapt to different datasets? What are the common features that OcCo captures across different datasets? \nSome visualization similar to Figure 3 would be helpful.\n\n---- Final Rating ----\n\nThe authors' response has resolved my concerns. I would keep my positive rating.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an unsupervised pretraining approach for 3D recognition, which is based on point cloud completion. The initial review receives a mixed rating, with two reviewers rate the paper below the bar and two above the bar. After the rebuttal, R3 changes the opinion from above the bar to a rejection recommendation. While several reviewers recognize the simplicity of the proposed method, R2 and R4 consider the proposed method a straightforward extension of known approaches for NLP and vision tasks. A lack of novelty was also pointed out as a weakness by R3 and R4. After consolidating the reviews and the rebuttal, the AC finds the weakness claims convincing and determines the paper is not ready for publication in the current form. 
Decision: Reject