Pre-Training by Completing Point Clouds

1 INTRODUCTION . Point clouds are a natural representation of 3D objects . Recently , there has been a flurry of exciting new point cloud models in areas such as segmentation ( Landrieu & Simonovsky , 2018 ; Yang et al. , 2019a ; Hu et al. , 2020a ) and object detection ( Zhou & Tuzel , 2018 ; Lang et al. , 2019 ; Wang et al. , 2020b ) . Current 3D sensing modalities ( i.e. , 3D scanners , stereo cameras , lidars ) have enabled the creation of large repositories of point cloud data ( Rusu & Cousins , 2011 ; Hackel et al. , 2017 ) . However , annotating point clouds is challenging as : ( 1 ) Point cloud data can be sparse and at low resolutions , making the identity of points ambiguous ; ( 2 ) Datasets that are not sparse can easily reach hundreds of millions of points ( e.g. , small dense point clouds for object classification ( Zhou & Neumann , 2013 ) and large vast point clouds for 3D reconstruction ( Zolanvari et al. , 2019 ) ) ; ( 3 ) Labelling individual points or drawing 3D bounding boxes are both more complex and timeconsuming compared with annotating 2D images ( Wang et al. , 2019a ) . Since most methods require dense supervision , the lack of annotated point cloud data impedes the development of novel models . On the other hand , because of the rapid development of 3D sensors , unlabelled point cloud datasets are abundant . Recent work has developed unsupervised pre-training methods to learn initialization for point cloud models . These are based on designing novel generative adversarial networks ( GANs ) ( Wu et al. , 2016 ; Han et al. , 2019 ; Achlioptas et al. , 2018 ) and autoencoders ( Hassani & Haley , 2019 ; Li et al. , 2018a ; Yang et al. , 2018 ) . However , completely unsupervised pre-training methods have been recently outperformed by the self-supervised pre-training techniques of ( Sauder & Sievers , 2019 ) and ( Alliegro et al. , 2020 ) . Both methods work by first voxelizing point clouds , then splitting each axis into k parts , yielding k3 voxels . Then , voxels are randomly permuted , and a model is trained to rearrange the permuted voxels back to their original positions . The intuition is that such a model learns the spatial configuration of objects and scenes . However , such random permutation destroys all spatial information that the model could have used to predict the final object point cloud . Our insight is that partial point-cloud masking is a good candidate for pre-training in point-clouds because of two reasons : ( 1 ) The pre-trained model requires spatial and semantic understanding of the input point clouds to be able to reconstruct masked shapes . ( 2 ) Mask-based completion tasks have become the de facto standard for learning pre-trained representations in natural language processing ( NLP ) ( Mikolov et al. , 2013 ; Devlin et al. , 2018 ; Peters et al. , 2018 ) . Different from random permutations , masking respects the spatial constraints that are naturally encoded in point clouds of real-world objects and scenes . Given this insight , we propose Occlusion Completion ( OcCo ) a self-supervised pre-training method that consists of ( a ) a mechanism to generate occluded point clouds , and ( b ) a completion task to reconstruct the occluded point cloud . Specifically , in ( a ) point clouds are generated by determining what part of objects would be occluded if the underlying object was observed from a particular view-point . In fact , many point clouds generated from a fixed 3D sensor will have occlusions exactly like this . Given an occluded point cloud , the goal of the completion task ( b ) is to learn a model that accurately reconstructs the missing parts of the point cloud . For a model to perform this task well , it needs to learn to encode localized structural information , based on the context and geometry of partial objects . This is something that is useful for any point cloud model to know , even if used only for classification or segmentation . We demonstrate that the weights learned by our pre-training method on a single unsupervised dataset can be used as initialization for models in downstream tasks ( e.g. , object classification , part and semantic segmentation ) to improve them , even on completely different datasets . Specifically our pre-training technique : ( i ) leads to improved generalization over prior baselines on the downstream tasks of object classification , object part and scene semantic segmentation ; ( ii ) speeds up model convergence , in some cases , by up to 5× ; ( iii ) maintains improvements as the size of the labelled downstream dataset decreases ; ( iv ) can be used for a variety of state-of-the-art point cloud models . 2 OCCLUSION COMPLETION . We now introduce Occlusion Completion ( OcCo ) . Our approach is shown in Figure 1 . Our main insight is that by continually occluding point clouds and learning a model c ( · ) to complete them , the weights of the completion model can be used as initialization for downstream tasks ( e.g. , classification , segmentation ) , speeding up training and improving generalization over other initialization techniques . Throughout we assume point clouds P are sets of points in 3D Euclidean space , P = { p1 , p2 , ... , pn } , where each point pi is a vector of coordinates ( xi , yi , zi ) and features ( e.g . color and normal ) . We begin by describing the components that make up our occlusion mapping o ( · ) . Then we detail how to learn a completion model c ( · ) , giving pseudocode and the architectural details in appendix . Finally we discuss the criteria on validating the effectiveness of a pre-training model for 3D point clouds . 2.1 GENERATING OCCLUSIONS . We first describe a randomized occlusion mapping o : P → P ( where P is the space of all point clouds ) from a full point cloud P to an occluded point cloud P̃ . We will do so by determining which points are occluded when the point cloud is viewed from a particular camera position . This requires three steps : ( 1 ) A projection of the point cloud ( in a world reference frame ) into the coordinates of a camera reference frame ; ( 2 ) Determining which points are occluded based on the camera view-points ; ( 3 ) Mapping the points back from the camera reference frame to the world reference frame . Viewing the point cloud from a camera . A camera defines a projection from a 3D world reference frame into a distinctive 3D camera reference frame . It does so by specifying a camera model and a camera view-point from which the projection occurs . The simplest camera model is the pinhole camera , and view-point projection for it is given by a simple linear equation : [ xcam ycam zcam ] = [ f γ w/2 0 f h/2 0 0 1 ] ︸ ︷︷ ︸ intrinsic [ K ] [ r1 r2 r3 t1 r4 r5 r6 t2 r7 r8 r9 t3 ] ︸ ︷︷ ︸ rotation | translation [ R | t ] xyz 1 . ( 1 ) In the above , ( x , y , z ) are the original point cloud coordinates , the matrix including r and t entries is the concatenation of a 3D rotation matrix with a 3D translation vector , and the final matrix to the left is the camera intrinsic matrix ( f specifies the camera focal length , γ is the skewness between the x and y axes in the camera , and w , h are the width and height of the camera image ) . Given these , the final coordinates ( xcam , ycam , zcam ) are the positions of the point in the camera reference frame . We will refer to the intrinsic matrix as K and the rotation/translation matrix as [ R|t ] . Determining occluded points . We can think of the point ( xcam , ycam , zcam ) in multiple ways : ( a ) as a 3D point in the camera reference frame , ( b ) as a 2D pixel with coordinates ( fxcam/zcam , fycam/zcam ) with a depth of zcam . In this way , some 2D points resulting from the projection may be occluded by others if they have the same pixel coordinates , but appear at a larger depth . To determine which points are occluded , we first use Delaunay triangulation to reconstruct a polygon mesh from the points , and remove the points which belong to the hidden surfaces that are determined via z-buffering . Mapping back from camera frame to world frame . Once occluded points are removed , we reproject the point cloud to the original world reference frame , via the following linear transformation : x ′ y′ z′ 1 = r1 r2 r3 t1r4 r5 r6 t2r7 r8 r9 t3 0 0 0 1 > ︸ ︷︷ ︸ [ R | t 0 | 1 ] > 1/f −γ/f 2 ( γh− fw ) / ( 2f2 ) 0 0 1/f −h/ ( 2f ) 0 0 0 1 0 0 0 0 1 ︸ ︷︷ ︸ [ K−1 0 0 1 ] xcamycamzcam 1 . ( 2 ) Our randomized occlusion mapping o ( · ) is constructed as follows . Fix an initial point cloud P . Given a camera intrinsics matrix K , sample rotation/translation matrices [ [ R1|t1 ] , . . . , [ RV |tV ] ] , where V is the number of views . For each view v ∈ [ V ] , project P into the camera frame of that view-point using eq . ( 1 ) , find occluded points and remove them , then map the rest back to the world reference using eq . ( 2 ) . This yields the final occluded world frame point cloud for view-point v : P̃v . 2.2 THE COMPLETION TASK . Given an occluded point cloud P̃ produced by o ( · ) , the goal of the completion task is to learn a completion mapping c : P → P from P̃ to a completed point cloud P . We say that a completion mapping is accurate w.r.t . loss ` ( · , · ) if EP̃∼o ( P ) ` ( c ( P̃ ) , P ) → 0 . The structure of the completion model c ( · ) is an “ encoder-decoder ” network ( Dai et al. , 2017b ; Yuan et al. , 2018 ; Tchapmi et al. , 2019 ; Wang et al. , 2020a ) . The encoder maps an occluded point cloud to a vector , and the decoder reconstructs the full shape . After pre-training , the encoder weights can be used as initialization for downstream tasks . In appendix we gives pseudocode for OcCo and describes the architectures . 3 EXPERIMENTS . 3.1 PRE-TRAINING AND DOWNSTREAM TRAINING DETAILS . We evaluate how OcCo improves the learning and generalization of a number of classification and segmentation tasks . Here we describe the details of training in each setting . OcCo pre-training . For all experiments , we will use a single pre-training dataset based on ModelNet40 ( Wu et al. , 2015 ) . It includes 12,311 synthesized objects from 40 object categories , divided into 9,843 training objects and 2,468 testing objects . To construct the pre-training dataset , we generate occluded point clouds based on the training objects with a fixed camera intrinsics { f = 1000 , γ = 0 , ω = 1600 , h = 1200 } , 10 random selected viewpoints and zero translation . Figure 2 shows examples of the resulting occluded point clouds . Given these , we train an “ encoderdecoder ” style completion model c ( · ) . For encoders , similar to prior completion models ( Tchapmi et al. , 2019 ; Wang et al. , 2020a ; Wen et al. , 2020a ) , we consider PointNet ( Qi et al. , 2017a ) , PCN ( Yuan et al. , 2018 ) and DGCNN ( Wang et al. , 2019b ) . These networks encode an occluded point cloud into a 1024-dimensional vector . We adapted the folding-based decoder from ( Yuan et al. , 2018 ) to complete the point clouds in a two-stage procedure . We use the Chamfer Distance ( CD ) as our loss function ` ( · , · ) . We use Adam ( Kingma & Ba , 2015 ) with an initial learning rate of 1e-4 , decayed by 0.7 every 10 epochs to a minimum value of 1e-6 , for a total of 50 epochs . We use a batch size of 32 and set the momentum in the batch normalisation to be 0.9 . Few-shot learning . We use ModelNet40 and Syndey10 ( De Deuge et al. , 2013 ) for “ K-way N - shot ” learning . During training , K classes are randomly selected and for each class we sample N random samples , then the model is tested on the same K classes . As in Sharma & Kaul ( 2020 ) , we represent each object with 100 points . We use the same training settings as used in the next paragraph . Object classification . We use three 3D object recognition benchmarks : ModelNet40 , ScanNet10 ( Qin et al. , 2019 ) and ScanObjectNN ( Uy et al. , 2019 ) ; we describe them in the appendix . All objects are represented with 1024 points . We use the same training settings as the original works . Concretely , for PCN and PointNet , we use the Adam optimizer with an initial learning rate 1e-3 , decayed by 0.7 every 20 epochs to a minimum value of 1e-5 . For DGCNN , we use the SGD optimizer with a momentum of 0.9 and a weight decay of 1e-4 . The learning rate starts from 0.1 and then reduces using cosine annealing Loshchilov & Hutter ( 2017 ) with a minimum value of 1e-3 . We use dropout Srivastava et al . ( 2014 ) in the fully connected layers before the softmax output layer . The dropout rate of PointNet and PCN is set to 0.7 , and is 0.5 for DGCNN . For all three models , we train them for 200 epochs with a batch size of 32 . We report the results based on three runs . Part segmentation . We use the ShapeNetPart ( Armeni et al. , 2016 ) benchmark for object part segmentation . This dataset contains 16,881 objects from 16 categories , and has 50 parts in total . Each object is represented with 2048 points , and we use the same training settings as the original work . Semantic segmentation . We use the S3DIS benchmark ( Armeni et al. , 2016 ) for semantic indoor scene segmentation . It contains 3D scans collected via Matterport scanners in 6 different places , encompassing 271 rooms . Each point , described by a 9-dimensional vector ( including coordinates , RGB values and normalised location ) , is labeled as one of 13 semantic categories ( e.g . chair , table and floor ) . We use the same preprocessing procedures and training settings as the original work .

Summary: This paper proposes a better pre-trained prior for a variety of downstream applications in point cloud analysis. The workflow of the pre-training mechanism is to first 1) generate occluded points that result from view occlusion and then 2) optimize the encoder to learn how to complete the occluded points from the partial point cloud. In downstream applications, the obtained encoder will be used as the initial weights in the network training. Empirical experiments have shown that such a pre-train mechanism can improve initialization over prior baselines and benefit a variety of tasks even with a large domain gap.