DUAL ADVERSARIAL MODEL FOR GENERATING 3D POINT CLOUD

1 INTRODUCTION . Point clouds are widely used in many fields , such as vision , 3D object detection and so on . In particular , 3D point clouds can represent geometric details of the object , and suitable for simple geometric transformation . However , there exist significant disadvantages in the point cloud data . For instance , data points associated with objects in the distance are difficult to identify due to sparsity . Moreover , it is not easy to design suitable generative models because of the high-dimensionality of the observation space Gumhold et al . ( 2001 ) ; Daniels et al . ( 2007 ) ; Achlioptas et al . ( 2018 ) . Effectively and accurately synthesizing point cloud data is still a challenge task . A qualified point cloud data generation network should satisfy the following conditions : 1 ) It should be able to synthesize 3D objects that are both highly varied and vivid , which means that there need to be fine details in the generated examples Wu et al . ( 2016 ) . 2 ) The distribution of the data generated should be similar to that of the original data . If the dimension of the generated sample is higher than that of the raw data , the additional generated portion must follow the potential distribution of raw data points . In the past few years , many researchers have developed a lot of meaningful work in this direction , such as 3D modeling and synthesis Carlson ( 1982 ) ; Van Kaick et al . ( 2011 ) . However , many of these traditional methods synthesize new objects by borrowing components from existing CAD model libraries . Therefore , synthetic objects may look realistic , but they are not novel in concept in fact . In recent years , the development of deep learning has provided a new way to solve this problem . Deep learning can well learn the characteristics of data in a domain by self-updating . Girdhar et al . ( 2016 ) ; Qi et al . ( 2016 ) proposes voxelized-object based method in learning deep object representations , while Kalogerakis et al . ( 2017 ) ; Su et al . ( 2015 ) ( add ) from the perspective of view-based projections . Besides these , graph methods Henaff et al . ( 2015 ) ; Defferrard et al . ( 2016 ) ; Yi et al . ( 2017 ) ; Bruna et al . ( 2013 ) are also considered for processing algorithms for 3D data . The method of generating point clouds at this stage no longer retrieves them from the object database , instead , most of them synthesize new objects based on learning objects . It is a more difficult task compared to operating on 2D data because of the higher data dimension and fewer data features . Generative Adversarial Network ( GAN ) Goodfellow et al . ( 2014 ) is the most widely used model in the current generation domain and has achieved great success in 2D image generation field . Wu et al . ( 2016 ) first applied this network to the 3D point cloud generation field , and named it as 3DGAN and achieved great results . It introduces an adversarial discriminator to differentiate whether an object is synthesized or real , which may own the potential to capture the structural difference of two 3D objects . Although 3D-GAN has been proved to be successful , it still has some problems . Because of the fact that GAN training is difficult and unstable , its huge network parameters drag down the calculation speed Salimans et al . ( 2016 ) .Groueix et al . ( 2018 ) ; Li et al . ( 2019 ) by adding prior knowledge helping generating target data , Zhao et al . ( 2018 ) applied this learned prior to the generation of discrete data , and then Achlioptas et al . ( 2018 ) adopts the idea of adversarial in latent space , but the specific approach is completely different . The former proposes a workflow that first learns a representation by training an AE with a compact bottleneck layer , then trains a plain GAN in that fixed latent representation . The two parts are trained separately , and their sampling and coding representations are different thus they can ’ t be translated or interpreted within both spaces ; the latter has used the Adversarial Autoencoder ( AAE ) Makhzani et al . ( 2015 ) for 3D generation , having the distribution of latent space of autoencoder trained on point cloud data close to the prior distribution , such as standard gaussian distribution . The manifold hypothesis indicates that the real-world data in a high-dimensional observation space always tends to live in the vicinity of an intrinsic low dimensional manifoldRifai et al . ( 2011 ) . In this paper we propose a dual generation model , which can fully make use of the characteristics of data in low-dimensional latent space to help the network better learn and generate point cloud data . Our contributions are mainly from the following aspects : • Compared to the literature study on the latent space of raw data , our method , through reasonable theory explaining and experimental verification , proposes dual generate adversarial networks architecture , from which the two networks are divided into the primary and secondary part . The secondary network assists the primary network by learning that the data is compressed to features on different dimensions , and our results exceed the same type of articles that focus on hidden space learning Achlioptas et al . ( 2018 ) . • Our model consists of three parts and can be trained end-to-end , what ’ s more , each part interacts and restricts each other , which can be trained efficiently and steadily without using the pre-trained weights related to point cloud data and only adopt the normal random initialization method . The loss function is designed in the light of the function of each part in the network , and we managed to find the best strategy of each part ’ s iteration renewal through experiments . • The framework we proposed on learning different low-dimension channel can be trans- ferred to other vision tasks easily . We do experiment on the MNIST dataset to demonstrate its implementability on 2D image tasks . 2 RELATED WORK . In this section , we mainly introduce the necessary research backgrounds of this paper , including the characteristics of point cloud data and some related work . Point cloud The point cloud is a collection of massive points that form characteristic of the target ’ s surface represents the shape of the object in 3D space . It consists of many x , y , z coordinates points , which have a matrix structure of N × 3 , where N is the number of points in a set . Compared with image data , there are few information and extractable features as to point cloud data , so how to effectively learn and represent the features of point cloud data has become the focus of current research . Wu et al . ( 2015 ) took an input point cloud as the voxelized representation , other methods like Maturana & Scherer ( 2015 ) using 3D-CNN to calculate occupancy grid . Qi et al . ( 2017 ) proposes the PointNet architecture , it can solve the difficulty of dealing with unordered sets by introducing permutation-invariant function instead of convolutions . Deep learning for generating 3D data Learning and generating 3D object is an essential task in the graphics and vision community . In 3D generating field , Wu et al . ( 2016 ) proposes the 3D-GAN , which can generate 3D object from a probabilistic space by leveraging recent advances in volumetric convolutional networks and generative adversarial nets . Achlioptas et al . ( 2018 ) designed a generative model called latent-GAN which is composed of two procedures : first using the point cloud data as input to train an Autoencoder ( AE ) Rumelhart et al . ( 1988 ) , and then using the pre-trained encoder output as the label to train GAN , instead of training on the raw point cloud . Compared with 3D-GAN , its generative ability has improved , but the drawback of this model is that it is not an end-to-end one . Sampling and coding representations are different and can not be translated or interpreted within both spaces . Our model solves this shortcoming by proposing a dual GAN architecture and the result shows that it can largely improves the generation ability . 3 BACKGROUND . In this part we give the necessary introduction on some basic concepts which we will use in the rest of this paper , including point cloud metrics and some fundamental building blocks . Autoencoder Autoencoder ( AE ) is a multi-layer neural network that can learn the efficient representation of input data through unsupervised learning . It is composed of two parts : Encoder and Decoder . The networks ’ width of the former part gradually becomes shallower as the network becomes deeper , while the latter part is reversed , as is shown on the right . The Encoder learns a low-dimensional latent representation z of the input data x and the Decoder restores z to original x. Generative Adversarial Network Generative Adversarial Network ( GAN ) is the most widely used generative model today . It consists of a model generator ( G ) and a model discriminator ( D ) and establishes a min-max adversarial game between these two models . The generator tries to get the function G ( z ) that maps z sample from the prior p ( z ) to the data space . The discriminator tries to distinguish the real data between the generated output . The loss function of this game can be expressed as : min G max D Ex∼pdata [ logD ( x ) ] + Ez∼p ( z ) [ log 1−D ( G ( z ) ) ] ( 1 ) The model mainly includes two parts of training process:1 ) Train the discriminator to distinguish the true label between the fake data generated from sample noise by generator . 2 ) Train the generator to generate samples which aims to fool the discriminator . Point Cloud Metrics We adopt two permutation-invariant metrics to compare unordered point sets , which was proposed by Fan et al . ( 2017 ) . The Earth Mover ’ s distance ( EMD ) : Introduced in Rubner et al . ( 2000 ) , it is a metrics of a transpotation problem based on the minimal cost to transform one distribution into the other . For two equally sized subsets s1 ⊆ R3 , s2 ⊆ R3 , their EMD is defined by : dEMD ( S1 , S2 ) = min ∅ : S1→S2 ∑ x∈S1 ‖x− ∅ ( x ) ‖2 ( 2 ) where ∅ ( x ) is a bijection , EMD is differentiable almost everywhere as a loss . Chamfer pseudo-distance ( CD ) : It measures the squared distance between each point in one set to its nearest neighbor in another set , and the function is shown below : CD ( S1 , S2 ) = ∑ x∈S1 min y∈S2 ‖x− y‖22 + ∑ y∈S2 min x∈S1 ‖x− y‖22 ( 3 ) Compared to EMD , CD is more efficient to compute as a loss and is totally differentiable . 4 DUAL GENERATION MODEL FOR LEARNING REPRESENTATION . In this section we first describe the architecture of our model and the motivation why we design such a network . Then we give a brief introduction to some similar previous work on generating point cloud , and make a comparison of these models to our model . Finally we present a detail explanation of the loss function in our model and how to train it .

Summary: This paper proposed a new dual generation model for learning representation by combining GAN and Autoencoder for 3D point cloud data. The idea is to add a regularization term over the GAN loss, and the regularization term measures the distance between the output of the encoder of Autoencoder and that of GAN by two point cloud metrics: Earth Movers's distance (EMD) and Chamfer pseudo-distance (CD). The experiment shows better representation are learnt by this dual generation model. The paper is in a good writing, and easy to read.