Pre-Training by Completing Point Clouds

1 INTRODUCTION . Point clouds are a natural representation of 3D objects . Recently , there has been a flurry of exciting new point cloud models in areas such as segmentation ( Landrieu & Simonovsky , 2018 ; Yang et al. , 2019a ; Hu et al. , 2020a ) and object detection ( Zhou & Tuzel , 2018 ; Lang et al. , 2019 ; Wang et al. , 2020b ) . Current 3D sensing modalities ( i.e. , 3D scanners , stereo cameras , lidars ) have enabled the creation of large repositories of point cloud data ( Rusu & Cousins , 2011 ; Hackel et al. , 2017 ) . However , annotating point clouds is challenging as : ( 1 ) Point cloud data can be sparse and at low resolutions , making the identity of points ambiguous ; ( 2 ) Datasets that are not sparse can easily reach hundreds of millions of points ( e.g. , small dense point clouds for object classification ( Zhou & Neumann , 2013 ) and large vast point clouds for 3D reconstruction ( Zolanvari et al. , 2019 ) ) ; ( 3 ) Labelling individual points or drawing 3D bounding boxes are both more complex and timeconsuming compared with annotating 2D images ( Wang et al. , 2019a ) . Since most methods require dense supervision , the lack of annotated point cloud data impedes the development of novel models . On the other hand , because of the rapid development of 3D sensors , unlabelled point cloud datasets are abundant . Recent work has developed unsupervised pre-training methods to learn initialization for point cloud models . These are based on designing novel generative adversarial networks ( GANs ) ( Wu et al. , 2016 ; Han et al. , 2019 ; Achlioptas et al. , 2018 ) and autoencoders ( Hassani & Haley , 2019 ; Li et al. , 2018a ; Yang et al. , 2018 ) . However , completely unsupervised pre-training methods have been recently outperformed by the self-supervised pre-training techniques of ( Sauder & Sievers , 2019 ) and ( Alliegro et al. , 2020 ) . Both methods work by first voxelizing point clouds , then splitting each axis into k parts , yielding k3 voxels . Then , voxels are randomly permuted , and a model is trained to rearrange the permuted voxels back to their original positions . The intuition is that such a model learns the spatial configuration of objects and scenes . However , such random permutation destroys all spatial information that the model could have used to predict the final object point cloud . Our insight is that partial point-cloud masking is a good candidate for pre-training in point-clouds because of two reasons : ( 1 ) The pre-trained model requires spatial and semantic understanding of the input point clouds to be able to reconstruct masked shapes . ( 2 ) Mask-based completion tasks have become the de facto standard for learning pre-trained representations in natural language processing ( NLP ) ( Mikolov et al. , 2013 ; Devlin et al. , 2018 ; Peters et al. , 2018 ) . Different from random permutations , masking respects the spatial constraints that are naturally encoded in point clouds of real-world objects and scenes . Given this insight , we propose Occlusion Completion ( OcCo ) a self-supervised pre-training method that consists of ( a ) a mechanism to generate occluded point clouds , and ( b ) a completion task to reconstruct the occluded point cloud . Specifically , in ( a ) point clouds are generated by determining what part of objects would be occluded if the underlying object was observed from a particular view-point . In fact , many point clouds generated from a fixed 3D sensor will have occlusions exactly like this . Given an occluded point cloud , the goal of the completion task ( b ) is to learn a model that accurately reconstructs the missing parts of the point cloud . For a model to perform this task well , it needs to learn to encode localized structural information , based on the context and geometry of partial objects . This is something that is useful for any point cloud model to know , even if used only for classification or segmentation . We demonstrate that the weights learned by our pre-training method on a single unsupervised dataset can be used as initialization for models in downstream tasks ( e.g. , object classification , part and semantic segmentation ) to improve them , even on completely different datasets . Specifically our pre-training technique : ( i ) leads to improved generalization over prior baselines on the downstream tasks of object classification , object part and scene semantic segmentation ; ( ii ) speeds up model convergence , in some cases , by up to 5× ; ( iii ) maintains improvements as the size of the labelled downstream dataset decreases ; ( iv ) can be used for a variety of state-of-the-art point cloud models . 2 OCCLUSION COMPLETION . We now introduce Occlusion Completion ( OcCo ) . Our approach is shown in Figure 1 . Our main insight is that by continually occluding point clouds and learning a model c ( · ) to complete them , the weights of the completion model can be used as initialization for downstream tasks ( e.g. , classification , segmentation ) , speeding up training and improving generalization over other initialization techniques . Throughout we assume point clouds P are sets of points in 3D Euclidean space , P = { p1 , p2 , ... , pn } , where each point pi is a vector of coordinates ( xi , yi , zi ) and features ( e.g . color and normal ) . We begin by describing the components that make up our occlusion mapping o ( · ) . Then we detail how to learn a completion model c ( · ) , giving pseudocode and the architectural details in appendix . Finally we discuss the criteria on validating the effectiveness of a pre-training model for 3D point clouds . 2.1 GENERATING OCCLUSIONS . We first describe a randomized occlusion mapping o : P → P ( where P is the space of all point clouds ) from a full point cloud P to an occluded point cloud P̃ . We will do so by determining which points are occluded when the point cloud is viewed from a particular camera position . This requires three steps : ( 1 ) A projection of the point cloud ( in a world reference frame ) into the coordinates of a camera reference frame ; ( 2 ) Determining which points are occluded based on the camera view-points ; ( 3 ) Mapping the points back from the camera reference frame to the world reference frame . Viewing the point cloud from a camera . A camera defines a projection from a 3D world reference frame into a distinctive 3D camera reference frame . It does so by specifying a camera model and a camera view-point from which the projection occurs . The simplest camera model is the pinhole camera , and view-point projection for it is given by a simple linear equation : [ xcam ycam zcam ] = [ f γ w/2 0 f h/2 0 0 1 ] ︸ ︷︷ ︸ intrinsic [ K ] [ r1 r2 r3 t1 r4 r5 r6 t2 r7 r8 r9 t3 ] ︸ ︷︷ ︸ rotation | translation [ R | t ] xyz 1 . ( 1 ) In the above , ( x , y , z ) are the original point cloud coordinates , the matrix including r and t entries is the concatenation of a 3D rotation matrix with a 3D translation vector , and the final matrix to the left is the camera intrinsic matrix ( f specifies the camera focal length , γ is the skewness between the x and y axes in the camera , and w , h are the width and height of the camera image ) . Given these , the final coordinates ( xcam , ycam , zcam ) are the positions of the point in the camera reference frame . We will refer to the intrinsic matrix as K and the rotation/translation matrix as [ R|t ] . Determining occluded points . We can think of the point ( xcam , ycam , zcam ) in multiple ways : ( a ) as a 3D point in the camera reference frame , ( b ) as a 2D pixel with coordinates ( fxcam/zcam , fycam/zcam ) with a depth of zcam . In this way , some 2D points resulting from the projection may be occluded by others if they have the same pixel coordinates , but appear at a larger depth . To determine which points are occluded , we first use Delaunay triangulation to reconstruct a polygon mesh from the points , and remove the points which belong to the hidden surfaces that are determined via z-buffering . Mapping back from camera frame to world frame . Once occluded points are removed , we reproject the point cloud to the original world reference frame , via the following linear transformation : x ′ y′ z′ 1 = r1 r2 r3 t1r4 r5 r6 t2r7 r8 r9 t3 0 0 0 1 > ︸ ︷︷ ︸ [ R | t 0 | 1 ] > 1/f −γ/f 2 ( γh− fw ) / ( 2f2 ) 0 0 1/f −h/ ( 2f ) 0 0 0 1 0 0 0 0 1 ︸ ︷︷ ︸ [ K−1 0 0 1 ] xcamycamzcam 1 . ( 2 ) Our randomized occlusion mapping o ( · ) is constructed as follows . Fix an initial point cloud P . Given a camera intrinsics matrix K , sample rotation/translation matrices [ [ R1|t1 ] , . . . , [ RV |tV ] ] , where V is the number of views . For each view v ∈ [ V ] , project P into the camera frame of that view-point using eq . ( 1 ) , find occluded points and remove them , then map the rest back to the world reference using eq . ( 2 ) . This yields the final occluded world frame point cloud for view-point v : P̃v . 2.2 THE COMPLETION TASK . Given an occluded point cloud P̃ produced by o ( · ) , the goal of the completion task is to learn a completion mapping c : P → P from P̃ to a completed point cloud P . We say that a completion mapping is accurate w.r.t . loss ` ( · , · ) if EP̃∼o ( P ) ` ( c ( P̃ ) , P ) → 0 . The structure of the completion model c ( · ) is an “ encoder-decoder ” network ( Dai et al. , 2017b ; Yuan et al. , 2018 ; Tchapmi et al. , 2019 ; Wang et al. , 2020a ) . The encoder maps an occluded point cloud to a vector , and the decoder reconstructs the full shape . After pre-training , the encoder weights can be used as initialization for downstream tasks . In appendix we gives pseudocode for OcCo and describes the architectures . 3 EXPERIMENTS . 3.1 PRE-TRAINING AND DOWNSTREAM TRAINING DETAILS . We evaluate how OcCo improves the learning and generalization of a number of classification and segmentation tasks . Here we describe the details of training in each setting . OcCo pre-training . For all experiments , we will use a single pre-training dataset based on ModelNet40 ( Wu et al. , 2015 ) . It includes 12,311 synthesized objects from 40 object categories , divided into 9,843 training objects and 2,468 testing objects . To construct the pre-training dataset , we generate occluded point clouds based on the training objects with a fixed camera intrinsics { f = 1000 , γ = 0 , ω = 1600 , h = 1200 } , 10 random selected viewpoints and zero translation . Figure 2 shows examples of the resulting occluded point clouds . Given these , we train an “ encoderdecoder ” style completion model c ( · ) . For encoders , similar to prior completion models ( Tchapmi et al. , 2019 ; Wang et al. , 2020a ; Wen et al. , 2020a ) , we consider PointNet ( Qi et al. , 2017a ) , PCN ( Yuan et al. , 2018 ) and DGCNN ( Wang et al. , 2019b ) . These networks encode an occluded point cloud into a 1024-dimensional vector . We adapted the folding-based decoder from ( Yuan et al. , 2018 ) to complete the point clouds in a two-stage procedure . We use the Chamfer Distance ( CD ) as our loss function ` ( · , · ) . We use Adam ( Kingma & Ba , 2015 ) with an initial learning rate of 1e-4 , decayed by 0.7 every 10 epochs to a minimum value of 1e-6 , for a total of 50 epochs . We use a batch size of 32 and set the momentum in the batch normalisation to be 0.9 . Few-shot learning . We use ModelNet40 and Syndey10 ( De Deuge et al. , 2013 ) for “ K-way N - shot ” learning . During training , K classes are randomly selected and for each class we sample N random samples , then the model is tested on the same K classes . As in Sharma & Kaul ( 2020 ) , we represent each object with 100 points . We use the same training settings as used in the next paragraph . Object classification . We use three 3D object recognition benchmarks : ModelNet40 , ScanNet10 ( Qin et al. , 2019 ) and ScanObjectNN ( Uy et al. , 2019 ) ; we describe them in the appendix . All objects are represented with 1024 points . We use the same training settings as the original works . Concretely , for PCN and PointNet , we use the Adam optimizer with an initial learning rate 1e-3 , decayed by 0.7 every 20 epochs to a minimum value of 1e-5 . For DGCNN , we use the SGD optimizer with a momentum of 0.9 and a weight decay of 1e-4 . The learning rate starts from 0.1 and then reduces using cosine annealing Loshchilov & Hutter ( 2017 ) with a minimum value of 1e-3 . We use dropout Srivastava et al . ( 2014 ) in the fully connected layers before the softmax output layer . The dropout rate of PointNet and PCN is set to 0.7 , and is 0.5 for DGCNN . For all three models , we train them for 200 epochs with a batch size of 32 . We report the results based on three runs . Part segmentation . We use the ShapeNetPart ( Armeni et al. , 2016 ) benchmark for object part segmentation . This dataset contains 16,881 objects from 16 categories , and has 50 parts in total . Each object is represented with 2048 points , and we use the same training settings as the original work . Semantic segmentation . We use the S3DIS benchmark ( Armeni et al. , 2016 ) for semantic indoor scene segmentation . It contains 3D scans collected via Matterport scanners in 6 different places , encompassing 271 rooms . Each point , described by a 9-dimensional vector ( including coordinates , RGB values and normalised location ) , is labeled as one of 13 semantic categories ( e.g . chair , table and floor ) . We use the same preprocessing procedures and training settings as the original work .

Summary: The paper considers the problem of training networks for point cloud processing through a point cloud completion task. Given a point cloud, it is rendered from a set of viewpoints and for each viewpoint the set of visible points is determine. A network is then trained to generate the full point cloud from the partially observed point cloud for a given view. Here, an encoder-decoder architecture is used, where the encoder corresponds to the network that should be pre-trained. Experimental results show that the proposed method outperforms two baselines for three tasks (object classification, object part segmentation, and semantic segmentation), when using less training data, and that the pre-training on the occlusion task leads to faster convergence.