Learning Localized Generative Models for 3D Point Clouds via Graph Convolution

1 INTRODUCTION . Convolutional neural networks are at the core of highly successful models in image generation and understanding . This success is due to the ability of the convolution operation to exploit the principles of locality , stationarity and compositionality that hold true for many data of interest . In particular , feature locality and weight sharing across the data domain greatly reduce the number of parameters in the model , simplifying training and countering overfitting . However , while images are defined on an underlying regular grid structure , several other types of data naturally lie on irregular or nonEuclidean domains ( Bronstein et al. , 2017 ) . Examples include problems in 3D models ( Boscaini et al. , 2016 ; Masci et al. , 2015 ) , computational biology ( Alipanahi et al. , 2015 ; Duvenaud et al. , 2015 ) or social network graphs ( Kipf & Welling , 2016 ) . Defining convolutional architectures on these domains is key to exploit useful priors on the data to obtain more powerful representations . Graph convolution is emerging as one of the most successful approaches to deal with data where the irregular domain can be represented as a graph . In this case , the data are defined as vectors on the nodes of a graph . Defining a convolution-like operation for this kind of data is not trivial , as even simple notions such as shifts are undefined . The literature has identified two main approaches to define graph convolution , namely spectral or spatial . In the former case ( Henaff et al. , 2015 ; Defferrard et al. , 2016 ; Kipf & Welling , 2016 ) , the convolution operator is defined in the spectral domain through the graph Fourier transform ( Shuman et al. , 2013 ) . Fast polynomial approximations ( Defferrard et al. , 2016 ) exist that allow an efficient implementation of the operation . This spectral approach has been successfully used in semi-supervised classification ( Kipf & Welling , 2016 ) and link prediction ( Schlichtkrull et al. , 2017 ) . However , the main drawback of these techniques is that the structure of the graph is supposed to be fixed and it is not clear how to handle the case where the graph structure varies . The latter class of methods ( Simonovsky & Komodakis , 2017 ; Wang et al. , 2018 ) defines the convolution operator using a spatial approach by means of local aggregations , i.e. , weighted combinations of the vectors restricted to a neighborhood . Since this kind of convolution is defined at a neighborhood level , the operation remains well defined even when the graph varies . Point clouds are a challenging data type due to the irregular positioning of the points and the fact that a point cloud is an unordered set of points , and therefore any permutation of its members , while changing the representation , does not change its semantic meaning . Some works have addressed supervised problems on point clouds such as classification or segmentation , either through voxelization ( Maturana & Scherer , 2015b ; Wu et al. , 2015 ) , where the irregular point structure is approximated with a regular 3D grid , or by networks like PointNet ( Qi et al. , 2017a ; b ) that address the problem of permutation invariance by processing each point identically and independently before applying a globally symmetric operation . The most recent approaches ( Simonovsky & Komodakis , 2017 ; Wang et al. , 2018 ) build graphs in the Euclidean space of the point cloud and use graph convolution operations . This approach has shown multiple advantages in i ) reducing the degrees of freedom in the learned models by enforcing some kind of weight sharing , ii ) extracting localized features that successfully capture dependencies among neighboring points . Generative models are powerful tools in unsupervised learning aiming at capturing the data distribution . However , so far little work has been done on generative models for point clouds . Generative models of point clouds can be useful for many tasks that range from data augmentation to shape completion or inpainting partial data thanks to the features learned by the model . Generative Adversarial Networks ( GANs ) have been shown on images to provide better approximations of the data distribution than variational autoencoders ( VAEs ) ( Larsen et al. , 2016 ) , being able to generate sharper images and to capture semantic properties in their latent space . For this reason , it is interesting to study them for unordered point sets . In the first work on the topic , Achlioptas et al . ( 2017 ) studied some GAN architectures to generate point clouds . Such architectures use the PointNet approach to deal with the permutation problem at the discriminator and employ a dense generator . However , this means that they are unable to learn localized features or exploit weight sharing . This paper studies a generative model for point clouds based on graph convolution . In particular , we focus on the GAN generator which is not well explored by the graph convolution literature . This poses a unique challenge : how can one apply a localized operation ( the graph convolution ) without knowing the domain ( the graph ) in advance because it is the very output of the generator ? We show that the proposed architecture learns domain and features simultaneously and promotes the features to be graph embeddings , i.e . representations in a vector space of the local dependencies between a point and its neighbors . Such localized features learned by the generator provide a flexible and descriptive model . Moreover , we address the problem of upsampling at the generator . While downsampling based on graph coarsening is a staple in ( semi- ) supervised problems using graph convolution , it is not obvious how to properly upsample the intermediate layers of a graph-convolutional GAN generator . We propose a method exploiting non-local self-similarities in the data distribution . 2 METHOD . 2.1 GRAPH-CONVOLUTIONAL GAN . GANs ( Goodfellow et al. , 2014 ) are state-of-the-art generative models composed of a generator and a discriminator network . The generator learns a function mapping a latent vector z to a sample x from the data distribution . In the original formulation , the discriminator worked as a classifier trained to separate real samples from generated ones . Recently , the Wasserstein GAN ( Arjovsky et al. , 2017 ) addressed the instability and mode collapse issues of the original formulation by modifying the loss function to be a dual formulation of an optimal transport problem using the Wasserstein metric : min G max �D�L≤1 Ex∼pdata [ D ( x ) ] − Ez∼pz [ D ( G ( z ) ) ] ( 1 ) with a discriminator D and a generator G. In this paper , we use the Wasserstein GAN with the gradient penalty method ( Gulrajani et al. , 2017 ) to enforce the Lipschitz constraint at the discriminator . In the proposed generative model , we use the Edge-Conditioned Convolution ( Simonovsky & Komodakis , 2017 ) which falls under the category of spatial approaches to graph convolution and is suitable for dealing with multiple arbitrary graphs . Given a layer l with N l feature vectors hlj ∈ Rd l of dimensionality dl , the convolution yields the feature vectors hl+1j ∈ Rd l+1 of the next layer by performing , for each node i of the graph , a weighted local aggregation of the feature vectors on the neighboring nodes j ∈ N li , where N li is the neighborhood of node i . The weights of the local aggregation are defined by a fully-connected network F l : Rdl → Rdl×dl+1 , which takes as input the difference between the features of two neighboring nodes and outputs the corresponding weight matrix Θl , ji = F lwl � hlj − hli � ∈ Rdl×dl+1 . Hence , the convolution operation is defined as : hl+1i =σ � � j∈N li F lwl � hlj − hli � hlj |N li | + hliW l + bl � = σ � � j∈N li Θl , jihlj |N li | � �� � neighborhood +hliW l � �� � node +bl � ( 2 ) where wl are the weights parametrizing network F l , Wl ∈ Rdl×dl+1 is a linear transformation of the node itself , bl a bias , and σ a non-linearity . It is important to note that the weights Θl , ij depend only on the difference between the features of the two nodes . This means that two pairs of nodes that have the same difference will have the same weight Θl , ij , even if they are in two different regions of the space . This creates weight sharing like in the classical CNNs and represents a strong advantage because it reduces the number of degrees of freedom of the model . GRAPH-BASED GENERATOR The focus of this paper is to design a GAN generator that uses localized operations in the form of graphs convolutions . Notice that such operations are able to deal with data in the form of unordered sets , such as points clouds , because they are by design invariant to permutations . However , there are some issues peculiar to the generative problem to be addressed . First , while in supervised problems ( Simonovsky & Komodakis , 2017 ; Wang et al. , 2018 ) or in unsupervised settings involving autoencoders ( Yang et al. , 2018 ) the point cloud is known in advance , the intermediate layers of the GAN generator do not know it in advance as it is the very result of the generation operation . It is therefore not obvious how to define an operation that is localized to neighborhoods of a graph that is not known in advance . The solution to this problem is to exploit the pairwise distances ( �hl−1j −hl−1i � ) between node features of the preceding layer to build a k-nearest neighbor graph . Figure 1 shows a block diagram of a graph-based generator where each graph convolution block uses the graph constructed from the input features of the block itself . The intuition behind this solution is that this promotes the features to become graph embeddings , i.e . representations in a high-dimensional metric space of relationships between points . Going through the generator network from the latent space towards the point cloud output , these embeddings are assembled hierarchically and their associated graphs represent better and better approximations of the graph of the output point cloud . According to the definition of graph convolution in ( 2 ) , the new features of a node are a weighted combination of the features of the node itself and of the neighbors as determined by the graph construction . Notice that this localized approach differs from the one in Achlioptas et al . ( 2017 ) where the generator of the r-GAN model is a fully-connected network , therefore unable to provide any localized interpretation of its hidden layers . It also differs from the PointNet ( Qi et al. , 2017a ) and PointNet++ ( Qi et al. , 2017b ) architectures . PointNet processes each point independently with the same weights and then aggregates them using a globally symmetric operation to deal with the permutation invariance problem . PointNet++ extends this work using some localized operations . However , the key difference with the work in this paper is that PointNet and PointNet++ are not generative models , but are used in supervised problems such as classification or segmentation . Other works explore likelihood-based generative models , typically in the form of variational autoencoders ( Fan et al. , 2017 ; Nash & Williams , 2017 ; Litany et al. , 2017 ) . The most similar approach to the method of this paper is the one in Grover et al . ( 2018 ) , with the key difference being that a distribution over adjacency matrices of graphs is learned using a spectral graph-convolutional VAE .

Summary: The authors present a method for generating points clouds with the help of graph convolution and a novel upsampling scheme. The proposed method exploits the pairwise distances between node features to build a NN-graph. The upsampling scheme generates new points via a slimmed down graph convolution, which are then concatenated to the initial node features. The proposed method is evaluated on four categories of the ShapeNet dataset. Resulting point clouds are evaluated via a qualitative and quantitative comparison to r-GAN.