Unpaired Point Cloud Completion on Real Scans using Adversarial Training

1 INTRODUCTION . Robust , efficient , and scalable solutions now exist for easily scanning large environments and workspaces ( Dai et al. , 2017a ; Chang et al. , 2017 ) . The resultant scans , however , are often partial and have to be completed ( i.e. , missing parts have to be hallucinated and filled in ) before they can be used in downstream applications , e.g. , virtual walk-through , path planning . The most popular data-driven scan completion methods rely on paired supervision data , i.e. , for each incomplete training scan , a corresponding complete data ( e.g. , voxels , point sets , signed distance fields ) is required . One way to establish such a shape completion network is then to train a suitably designed encoder-decoder architecture ( Dai et al. , 2017b ; 2018 ) . The required paired training data is obtained by virtually scanning 3D objects ( e.g. , SunCG Song et al . ( 2017 ) , ShapeNet Chang et al . ( 2015 ) datasets ) to simulate occlusion effects . Such approaches , however , are unsuited for real scans where large volumes of paired supervision data remain difficult to collect . Additionally , when data distributions from virtual scans do not match those from real scans , completion networks trained on synthetic-partial and synthetic-complete data do not sufficiently generalize to real ( partial ) scans . To the best of our knowledge , no point-based unpaired method exists that learns to translate noisy and incomplete point cloud from raw scans to clean and complete point sets . We propose an unpaired point-based scan completion method that can be trained without requiring explicit correspondence between partial point sets ( e.g. , raw scans ) and example complete shape models ( e.g. , synthetic models ) . Note that the network does not require explicit examples of real complete scans and hence existing ( unpaired ) large-scale real 3D scan ( e.g. , Dai et al . ( 2017a ) ; Chang et al . ( 2017 ) ) and virtual 3D object repositories ( e.g. , Song et al . ( 2017 ) ; Chang et al . ( 2015 ) ) can directly be leveraged as training data . Figure 1 shows example scan completions . As we show in Table 1 , unlike methods requiring paired supervision , our method continues to perform well even if the data distributions of synthetic complete scans and real partial scans differ . We achieve this by designing a generative adversarial network ( GAN ) wherein a generator , i.e. , an adaptation network , transforms the input into a suitable latent representation such that a discriminator can not differentiate between the transformed latent variables and the latent variables obtained from training data ( i.e. , complete shape models ) . Intuitively , the generator is responsible for the key task of mapping raw partial point sets into clean and complete point sets , and the process is regu- larized by working in two different latent spaces that have separately learned manifolds of scanned and synthetic object data . We demonstrate our method on several publicly available real-world scan datasets namely ( i ) ScanNet ( Dai et al. , 2017a ) chairs and tables ; ( ii ) Matterport3D ( Chang et al. , 2017 ) chairs and tables ; and ( iii ) KITTI ( Geiger et al. , 2012 ) cars . In absence of completion ground truth , we can not directly compute accuracy for the completed scans , and instead compare using plausibility scores . Further , in order to quantitatively evaluate the performance of the network , we report numbers on a synthetic dataset ( Dai et al. , 2017b ) where completed versions are available . Finally , we compare our method against baseline methods to demonstrate the advantages of the proposed unpaired scan completion framework . 2 RELATED WORK . Shape Completion . Many deep neural networks have been proposed to address the shape completion challenge . Inspired by CNN-based 2D image completion networks , 3D convolutional neural networks applied on voxelized inputs have been widely adopted for 3D shape completion task ( Dai et al. , 2018 ; 2017b ; Sharma et al. , 2016 ; Han et al. , 2017 ; Thanh Nguyen et al. , 2016 ; Yang et al. , 2018 ; Wang et al. , 2017 ) . As quantizing shapes to voxel grids lead to geometric information loss , recent approaches ( Yuan et al. , 2018 ; Yu et al. , 2018b ; Achlioptas et al. , 2018 ) operate directly on point sets to fill in missing parts . These works , however , require supervision in the form of partialcomplete paired data for training deep neural networks to directly regress partial input to their ground truth counterparts . Since paired ground truth of real-world data is rarely available such training data is generated using virtual scanning . While the methods work well on synthetic test data , they do not generalize easily to real scans arising from hard-to-model acquisition processes . Realizing the gap between synthetically-generated data and real-world data , Stutz & Geiger ( 2018 ) proposed to directly work on voxelized real-world data . They also work in a latent space created for clean and complete data but measure reconstruction loss using a maximum likelihood estimator . Instead , we propose a GAN setup to learn a mapping between latent spaces respectively arising from partial real and synthetic complete data . Further , by measuring loss using Hausdorff distance on point clouds , we directly work with point sets instead of voxelized input . Generative Adversarial Network . Since its introduction , GAN ( Goodfellow et al. , 2014 ) has been used for a variety of generative tasks . In 2D image domain , researchers have utilized adversarial training to recover richer information from low-resolution images or corrupted images ( Ledig et al. , 2017 ; Wang et al. , 2018 ; Mao et al. , 2017 ; Park et al. , 2018 ; Bulat et al. , 2018 ; Yeh et al. , 2017 ; Iizuka et al. , 2017 ) . In 3D context , Yang et al . ( 2018 ) ; Wang et al . ( 2017 ) combine 3D-CNN and generative adversarial training to complete shapes under the supervision of ground truth data . Gurumurthy & Agrawal ( 2019 ) treats the point cloud completion task as denoising AE problem , utilizing adversarial training to optimize on the AE latent space . We also leverage the power of GAN for reasoning the missing part of partial point cloud scanning . However , our method is designed to work with unpaired data , and thus can directly be applied to real-world scans even when real-world and synthetic data distributions differ . Intuitively , our GAN-based approach directly learns a translation mapping between these two different distributions . Deep Learning on Point clouds . Our method is built upon recent advances in deep neural networks for point clouds . PointNet Qi et al . ( 2017a ) , the pioneering work on this topic , takes an input point set through point-wise MLP layers followed by a symmetric and permutation-invariant func- tion to produce a compact global feature , which can then be used for a diverse set of tasks ( e.g. , classification , segmentation ) . Although many improvements to PointNet have been proposed ( Su et al. , 2018 ; Li et al. , 2018b ; Qi et al. , 2017b ; Li et al. , 2018a ; Zaheer et al. , 2017 ) , the simplicity and effectiveness of PointNet and its extension PointNet++ make them popular for many other analysis tasks ( Yu et al. , 2018a ; Yin et al. , 2018 ; Yu et al. , 2018b ; Guerrero et al. , 2018 ) . In the context of synthesis , Achlioptas et al . ( 2018 ) proposed an autoencoder network , using a PointNet-based backbone , to learn compact representations of point clouds . By working in a reduced latent space produced by the autoencoder , they report significant advantages in training GANs , instead of having a generator producing raw point clouds . Inspired by this work , we design a GAN to translate between two different latent spaces to perform unpaired shape completion on real scans . 3 METHOD . Given a noisy and partial point set S = { si } as input , our goal is to produce a clean and complete point setR = { ri } as output . Note that although the two sets have the same number of points , there is no explicit correspondence between the sets S and R. Further , we assume access to clean and complete point sets for shapes for the object classes . We achieve unpaired completion by learning two class-specific point set manifolds , Xr for the scanned inputs , and Xc for clean and complete shapes . Solving the shape completion problem then amounts to learning a mapping Xr → Xc between the respective latent spaces . We train a generator Gθ : Xr → Xc to perform the mapping . Note that we do not require the noise characteristics in the two data distributions , i.e. , real and synthetic , to be the same . In absence of paired training data , we score the generated output by setting up a min-max game where the generator is trained to fool a discriminator Fχ , whose goal is to differentiate between encoded clean and complete shapes , and mapped encodings of the raw and partial inputs . Figure 2 shows the setup of the proposed scan completion network . The latent space encoder-decoders , the mapping generator , and the discriminator are all trained as detailed next . 3.1 LEARNING LATENT SPACES FOR POINT SETS . The latent space of a given set of point sets is obtained by training an autoencoder , which encodes the given input to a low-dimension latent feature and then decodes to reconstruct the original input . We work directly on the point sets via these learned latent spaces instead of quantizing them to voxel grids or signed distance fields . For point sets coming from the clean and complete point sets P , we learn an encoder network Ecη that maps P from the original parameter space R3N , defined by concatenating the coordinates of the N ( 2048 in all our experiments ) points , to a lower-dimensional latent space Xc . A decoder network Dcφ performs the inverse transformation back to R3N giving us a reconstructed point set P̃ with also N points . The encoder-decoders are trained with reconstruction loss , LEMD ( η , φ ) = EP∼pcompleted ( P , Dcφ ( Ecη ( P ) ) ) , ( 1 ) where P ∼ pcomplete denotes point set samples drawn from the set of clean and complete point sets , d ( X1 , X2 ) is the Earth Mover ’ s Distance ( EMD ) between point sets X1 , X2 , and ( η , φ ) are the input point set completion without HL completion with HL Figure 3 : Effect of unpaired scan completion without ( Equation 5 ) and with HL term ( Equation 6 ) . Without the HL term , the network produces a clean point set for a complete chair , that is different in shape from the input . With the HL term , the network produces a clean point set that matches the input . learnable parameters of the encoder and decoder networks , respectively . Once trained , the weights of both networks are held fixed and the latent code z = Ecη ( X ) , z ∈ Xc for a clean and complete point setX provides a compact representation for subsequent training and implicitly captures the manifold of clean and complete data . The architecture of the encoder and decoder is similar to Achlioptas et al . ( 2018 ) ; Qi et al . ( 2017a ) : using a 5-layer MLP to lift individual points to a deeper feature space , followed by a symmetric function to maintain permutation invariance . This results in a kdimensional latent code that describes the entire point cloud ( k=128 in all our experiments ) . More details of the network architecture can be found in the appendix . As for the point set coming from the noisy-partial point sets S , one can also train another encoder Erγ : S → Xr and decoder Drψ : Xr → S̃ pair that provides a latent parameterization Xr for the noisy-partial point sets , with the definition of the reconstruction loss as , LEMD ( γ , ψ ) = ES∼prawd ( S , Drψ ( Erγ ( S ) ) ) , ( 2 ) where S ∼ praw denotes point set samples drawn from the set of noisy and partial point sets . Although , in experiments , the latent space of this autoencoder trained on noisy-partial point sets works considerably well as the noisy-partial point set manifold , we found that using the latent space produced by feeding noisy-partial point sets to the autoencoder trained on clean and complete point sets yields slightly better results . Hence , unless specified , we set γ = η and ψ = φ in our experiments . The comparison of different choices to obtain the latent space for noisy-partial point sets is also presented in Section 4 . Next , we will describe the GAN setup to learn a mapping between the latent spaces of raw noisy-partial and synthetic clean-complete point sets , i.e. , Xr → Xc .

Summary: This paper proposes a new method for making 3D point clouds by automatically completing 3D scans. It does not require paired data samples for training which makes it possible to train it on real data instead of synthetic data. The authors use a generative adversarial network (GAN) to “generate” complete point clouds from noisy or partial point clouds obtained by 3D scanning. The generator learns to perform mapping from point set manifold of scanned noisy and partial input X_r to manifold of clean shapes X_c. The discriminator tries to tell between encoded clean shapes (synthetic data point clouds) and mappings of noisy input (point clouds from real-life data 3D scans).