Point Cloud GAN

Abstract: Generative Adversarial Networks (GAN) can achieve promising performance on learning complex data distributions on different types of data. In this paper, we first show that a straightforward extension of an existing GAN algorithm is not applicable to point clouds, because the constraint required for discriminators is undefined for set data. We propose a two fold modification to a GAN algorithm to be able to generate point clouds (PC-GAN). First, we combine ideas from hierarchical Bayesian modeling and implicit generative models by learning a hierarchical and interpretable sampling process. A key component of our method is that we train a posterior inference network for the hidden variables. Second, PC-GAN defines a generic framework that can incorporate many existing GAN algorithms. We further propose a sandwiching objective, which results in a tighter Wasserstein distance estimate than the commonly used dual form in WGAN. We validate our claims on the ModelNet40 benchmark dataset and observe that PC- GAN trained by the sandwiching objective achieves better results on test data than existing methods. We also conduct studies on several tasks, including generalization on unseen point clouds, latent space interpolation, classification, and image to point clouds transformation, to demonstrate the versatility of the proposed PC-GAN algorithm.

Reviews and author responses:

1. Yes, since the training takes time, we reported the preliminary results above, but we believe it should be enough to observe the trade-off when we increase/decrease the ratio toward the extremes (upper only and lower bound only). We will make the study more complete and revise the results in the revision. 

2. We do not have any quantitative results, but we do have some qualitative picture proofs. We can add that in appendix to the camera ready.  

3. Yes, we agree with reviewer that our framework is related to Neural Statistician and would discuss it  in the camera ready. 

4. We do have some results in the paper. For example, in Table 2, we can observe increasing number of points brings marginal improvements. Same situation is also observed in ShapeNet55 experiments, where increasing number of points can lead to <0.5% improvements. 

5. The results depends on the pooling function. For max pooling, it’s more robust to non-uniform decimation while the mean pooling suffers more. If the decimation results in significant change of the objects (e.g. remove half of the objects), both PC-GAN and AAE  (Achlioptas et al., 2017) cannot results in valid reconstruction since it is very different from training data. We will make this study more thorough in the revision.


Thanks for your response.
1) Do you plan to add the results under point 4 above to the paper?
2) Do you plan to add any experimental evidence to support your claim under point 5 above to the paper?

Do you plan to address the other points I raised in my review on 
3) relation to neural statistician
4) Experimental evaluation of auto-encoding using a variable number of input points
5) How auto encoding performs when non-uniform decimation of the input

I like the idea of the paper, but I think it can be strengthened by addressing these issues.

Summary:
This paper introduces a generative model for 3D point clouds. Authors aim at theoretically showing the difficulties of using existing generative models to learn distributions of point clouds, and propose a variant that supposedly solves the issues.

Pros:
+ The problem of designing generative models for 3D data is important.

Cons: 
- Paper is often hard to follow, and contains a significant number of typos. 
- Authors claim to identify a fundamental problem with the existing generative models for point clouds, yet Section 2 tries to show that a _specific version_ that uses DeepSet does not satisfy theoretical guarantees. What if we use e.g. a recurrent network instead? As is, the counter example proof itself is quite confusing: it would really help if the proof was more formal.
- Jointly learning an inference network (Q) has certainly been done before, and I am not sure authors provide an elaborate enough explanation of what is the difference with adversarially learned inference /  adversarial feature learning.
- It is not clear why authors did not follow the evaluation protocol of [Achlioptas’17] or [Wu’16] more closely. In particular, evaluation for the classification task should be compatible with the proposed model, which would give a much better picture of the learned representations.


Thanks for the elaborate response. Authors addressed some of my concerns, thus I slightly updated the rating. I am still not very convinced by the argument about the DeepSet: there certainly are multiple different ways to do point cloud classification, thus the relevance of this proof is not clear. 

We thank reviewer for his insightful comments.

1. We agree with the reviewer that sinkhorn iteration is a way to obtain an upper bound on Wasserstein distance. However, based on the original paper, they solve the Sinkhorn divergen with T iterations, later when they solve the generator based on the estimated distance, the gradient has to backpropagate through those T iterations, which is expensive and infeasible. We also note that there is new work, IPOT (Xie et al., 2018), which can get rid of backpropagating through the T iterations as what we adopted (Bertsekas, 1985) in the paper. Combining PC-GAN with IPOP or other future works could be an interesting future work. 

2.  The variance of the sandwiched estimator can be higher, but we are more concerned about bias in this work, which can be treated as a bias-variance trade-off. 

3. The 20:1 mixture used in practice do not directly correspond to s in theory, because the distances we compute are not scaled. For example, if the f_\phi, the discriminator of GAN, is k-Lipschitz, the lower bound estimate should be divided by k. However, k is unknown in practice. Therefore, we just numerically did a coarse grid search and find the best mixture ratio.  Also, we try different ratios as we replied to R2 above. 

Ratio                   D2F (Distance to Face)       Coverage
1:0                               6.03E+00                        3.36E-01
40:1                              6.06E+00                       3.41E-01
20:1                              5.77E+00                       3.47E-01
10:1                              6.85E+00                       3.56E-01
0 :1                               9.19E+00                       3.67E-01

4. We do not consider W_s to be very close from W_U. As can be seen from Figure 6, for the aeroplane examples, W_U fails to capture aeroplane tires while W_s can. Similarly for Chair example, W_s recovers better legs than W_U. Quantitatively, we highlight that W_s outperforms W_U consistently as shown in Table 1. Thus, we consider both W_U and W_L is needed to generate good quality point clouds. 




1. We apologize for typos and if any term is not defined at the appropriate places. We  fixed all the typos and define the abbreviation for IPM at the first occurrence. Please check the revision. P and G, the elements of the divergence D(P||G) that appears in the first paragraph of section 3, is defined in the subsequent two sentences in the same paragraph. By the notation G_theta(u) \sim p(theta), we mean that we want to train the generator G_theta such that when fed a random variable u \sim p(u), the distribution of G_theta(u) matches that of p(theta). Sorry if it is confusing, but G_theta is not parameterized by theta, it just indicates that its the generator for theta. (Like G_x indicates that it is the generator for x).

2. The training of G_theta is described in the subsection titled “Hierarchical Sampling”. As correctly pointed out by the reviewer, that G_theta does not appear in the objective function (4). Using (4), we train G_x and Q networks. After training G_x and Q, we use trained Q to collect inferred Q(X), for each point cloud X. Then we train the generator G_theta using ordinary WGAN formulation to produce samples from same distribution as that of the samples Q(X) for each point cloud X. In addition to such two step training, a joint training also works, but is slower computationally, thus we report only the two step training in the paper.

3. Quantitative evaluation of generative modeling performance is unfortunately very hard for real world problems like point clouds, which is the probable cause for it being missing from much of GAN literature. Thus, to provide some quantitative results for generation, we resorted to the toy problem. In the toy problem, we can accurately gauge the generation capabilities as can be seen from Figure 5. (We did not explicitly provide numbers like KL divergence, as it is evident from the Figure that PC-GAN would be significantly better than AAEs if we evaluate the numbers.) The same protocol can be extended for measuring the quality of the final hierarchical sampling. 

4. To showcase the effect of varying s, we chose the reasonable sized ModelNet10 dataset and ran for s=0, s=1, and three values s_1<s_2<s_3 in between. The results are as follows:


                   D2F (Distance to Face)       Coverage
s=0                        6.03E+00                     3.36E-01
s1                          6.06E+00                     3.41E-01
s2                          5.77E+00                     3.47E-01
s3                          6.85E+00                     3.56E-01
s=1                        9.19E+00                     3.67E-01

4. Yes the model nicely captures simple topological features of the object, like presence of holes versus being one solid object. Even in the latent space, objects with hole group together.



1. The purpose of the counterexample is only to show that there exists some spurious solutions to GANs with general DeepSets-style discriminator for point clouds. We agree that setup we selected is destined to fail, but it was done on purpose to illustrate the presence of spurious solutions. A good generator and discriminator would definitely be a solution as well. However, solutions during optimization might not always correspond to such good solutions and can also correspond to the demonstrated spurious solutions. We found empirically that GAN with simple DeepSet-like discriminator most of the times fails to learn to generate point clouds even after converging, however, it does sometimes results in reasonable generations (although worse than proposed PC-GAN). So, we do not consider the argument to be unrealistic as we often observe the degeneracy. So the message here is that we need additional constraints for GANs with simple DeepSet-like discriminator to exclude such bad solutions and lead to a more stable training. Other architectures like RNN might work, but they are not permutation invariant, which is a desirable property for set data like point clouds. More comparisons between using RNN and DeepSets for other tasks on set data can refer to Zaheer et al., (2017).

2. As we discussed in the end of Section 3, ALI and BiGan’s goal is to match (z, G(z)) and (Q(X), X), which aims to infer the random noise z and enforce the latent code to follow noise  distribution (e.g. Gaussian). On the other hand, we do not enforce Q(X) to follow from Gaussian. Instead, we train the other G_theta(u) to match Q(X), which is more similar to AAE-like works (Engel et al., 2017; Kim et al., 2017; Achlioptas et al. 2017). The difference of the interpretation between PC-GAN and  those AAE-like work is also explained in the second paragraph of Sec 4.

3. We followed the same protocol that we trained on ShapeNet55 and tested on ModelNet40 testing set. Please check Table 3 in the revision. PC-GAN achieves 86.9% accuracy which is better than AAE (84.5%),  3D-GAN (83.3%) and other unsupervised learning approach.


Dear reviewers, 

We thank for your comments. We polished the paper to fix typos. Also, we added addition experiments for classification with Shapenet55. Please check Table 3 in the revision.  We will respond to each reviewer’s comment below the review. 

Summary:
This paper proposes a generative point cloud model based on adversarial learning and definitti’s representation theorem of exchangeable variables.
The main focus in experiments and the exposition is on 3D point clouds representing object shapes (seems the surface, but could also be the interior of objects, please clarify). 
The main idea is to represent a point cloud using a global latent variable that captures the overall shape, and a collection of local latent variables that code for the position of a point on the shape.
The model consists of thee components:
(i) an “encoder” that takes a point cloud as input and maps it to a (point estimate of) the global latent variable of the shape represented by the input cloud, a point-net architecture is used here
(ii) a “decoder” that takes the estimated global latent variable, and a local latent variable, and maps it to an “output” point in the cloud to be produced by the model. 
(iii) a “discriminator” network that aims to distinguish points from a *given* shape, and the points produced by pipe-lining the encoder and decoder. Critically different from conventional GANs, the discriminator is optimized *per shape*, ie each point cloud is considered as a *distribution* over R^3 specific to that shape. 
(iv) a “shape prior” that, once the encoder-decoder model from above is trained, is used to model the distribution over the global latent variables. This model is trained, presumably, in a conventional GAN style using the global latent variable representations inferred across the different training point clouds.

As compared to prior work by Achiloptas et al (2017), the proposed approach has the advantage to allow for sampling an arbitrary number of points from the target shape, rather than a fixed pre-defined number. 

In addition, the authors propose to minimize a weighted average of a lower bound and upper bound on the Wasserstein distance between the distributions of points corresponding to given shapes. 
This approach translates to improved quantitative evaluation measures, 

Experiments are conducted on a simple toy data set, as  a proof of concept, and on data from ModelNet10 and ModelNet40. 
Two performance metrics are introduced to assess the auto-encoding ability of the model: to what extent does the encoder-decoder pipeline result in point clouds similar to the shape from which the input point-cloud is generated. 

Overall I find the idea of the paper interesting and worth publishing, but the exposition of the paper is less than ideal and needs further work. 
The experimental validation of the proposed approach can also be further improved, see more specific comments below. 

Specific comments:

- The counter example at the bottom of page 2 is limited, in the sense that the oracle assumption seems highly non-realistic, casting doubt on the relevance of the argument.

- The notation in section 3 (before 3.1) is rather sloppy. 
For example, 
- please define P and G, the elements of the divergence D(P||G) that appears in the first paragraph of section 3.
- it is not defined in which space theta lives, it is not clear what the authors intend with the notation G_theta(u) \sim p(theta). 
- what prior distributions p(z) and p(u) are used? What is the choice based on?

- abbreviation IPM is referred several times in the paper, but remains undefined in the paper until end of page 4, please define earlier. 

- The model G_theta does not appear in the training objective function (4), how is this module trained precisely?

- Lack of clarity in the following passage: “In our setting, each point xi in the point cloud can be considered to correspond to single images when we train GANs over images”

- The notion of divergence D(P|G) is not made concrete in section 3 and 3.1, which makes the notation of rather little use.

- The following paper merits a discussion in the related work section: 
“TOWARDS A NEURAL STATISTICIAN”, ICLR’17, https://openreview.net/pdf?id=HJDBUF5le

- The manuscript contains many typos. For example
 “vedio” op page 4, “circile” on page 5, “condct” on page 8, etc.
Please proof read your paper and fix these.
The refenence to  Bengio 2018 is incomplete: what do you refer to precisely?

- There seems to be no mention of the dimension of the “local” latent variables z_i. 
Please comment on the choice, and its impact on the behavior of the model.

- The quantitative evaluation in table 1 is interesting and useful. 
It is limited, however, in the sense that it (only) measures auto-encoding capabilities: to what extent can the shape be reproduced given a sample point cloud from the given shape. 
Quantitative evaluation of generative modeling performance is unfortunately missing from this paper, as it is in much of the GAN literature. 
Could you please comment on how this can/will be fixed?

- The toy data set experiments could be dropped  to make room for experiments suggested below.

- An experimental study of the effect of the mixing parameter “s” would be useful to include. 
For example, by taking s on a grid from 0 to 1, one could plot the coverage and distance-to-face measures.

- Experimental evaluation of auto-encoding using a variable number of input points is interesting to add: ie how do the two evaluation measures evolve as a function of the number of points in the input point cloud?

- Similar, it is interesting to evaluate how auto encoding performs when non-uniform decimation of the input cloud is performed, eg what happens if we “chop off” part of the input point cloud (eg the legs of the chair), does the model recover and add the removed parts? This is potentially useful to practitioners which have to deal with incomplete point clouds acquired by range scanners. 

- Analysis of shapes with different genus and dimensions would be interesting. 
Does the model manage to capture that some shapes have holes, or consists of a closed 2D surface (ball) vs an open surface (disk),  despite a simple prior on the local latent variables z?



Authors provide a variant of WGAN, called PC-GAN, to generate 3D point clouds. The drawback of a vanilla GAN with a DeepSet classifier is analyzed. The rationality that decoupling the point generator with the object generator is also discussed. 
A sandwiching objective function is proposed to achieve a better estimation of Wasserstein distance. 
Compared with AAE and the simplified variants of the proposed PC-GAN, the proposed PC-GAN achieves incremental results on point cloud generation.

Comments:
1. Authors calculate W_U in a primal form via solving an assignment programming problem. Have authors ever tried Sinkhorn iteration? To my knowledge, sinkhorn iteration is a very popular method to solve OT problem effectively. It would be nice if authors can provide some reasons and comparisons for their choice on the optimizer of W_U. 

2. Authors proved that the sandwiching object W_s is closer to the real Wasserstein distance, but it increases the variance of the loss function. Specifically, the dynamics of W_U, and W_L, according to lemma1, is (epsilon2-epsilon1)*w(P, G) while the dynamics of W_s is 2*epsilon1 * w(P, G), and 2epsilon1 > epsilon2 - epsilon1 (according to the assumption in lemma 1). Does it mean that the W_s is not as stable as W_L or W_U during training?  Additionally, authors combined W_U with W_L with a mixture 20:1, i.e., the s in Eqs(6, 13, 14) is smaller than 0.05. In such a situation, both the value and the dynamics of W_s will be very close to that of W_U. Does it mean that W_L is not so important as W_U? Authors should analyze the stability of their method in details.

Essentially, the proposed method is a variant of WGAN, which estimates Wasserstein distance with lower bias but may suffer from worse stability. In the experiments, both the setting and the experimental results show that the proposed W_s will be very close to W_U. As a result, the improvement caused by the proposed method is incremental compared with its variants. 

Typos:
- The end of the 2nd line of lemma 1: P, G should be \mathbb{P}, \mathbb{G}
- The 3rd line of lemma 1: epsilon1 -> epsilon_1
- Page 14, Eq(14), \lambda should be s
- Page 14, Eqs(13, 14), w(\mathbb{P}, \mathbb{G}) should appear on the right.


We thank you for raising an excellent question about counter example. The purpose of the counter example is only to show that there exists some spurious solutions to GANs with general DeepSets-style discriminator for point clouds. We agree that setup we selected is destined to fail, but it was done on purpose to illustrate the presence of spurious solutions. A good generator and discriminator would definitely be a solution as well. However, solutions during optimization might not always correspond to such good solutions and can also correspond to the demonstrated spurious solutions. This is precisely the point made by counter example argument, that simple DeepSets classifier does not ensure proper generative model. Moreover, we found empirically that GAN with simple DeepSet-like discriminator most of the times fails to learn to generate point clouds even after converging. However, sometimes it does results in reasonable generations (although worse than proposed PC-GAN), which may correspond to the case you describe. So the message here is that we need additional constraints for GANs with simple DeepSet-like discriminator to exclude such bad solutions and lead to a more stable training.


I don't think the counter example presented at the bottom of page 2 supports the conclusion that: "simply using DeepSets classifier without any constraints in simple GAN in order to handle sets does not lead to a valid generative model."

Specifically, introducing the deterministic transformation T({x_i}) artificially precludes the discriminator from observing vital information about how points are distributed within each set of points passed to the discriminator. By construction, this transform removes all information from the point sets aside from the object parameter(s) theta which could have generated each point in the set. But, to determine if the sets of points faithfully match the conditionals p(x|theta) in the true joint p(x, theta)=p(theta)p(x|theta), the discriminator requires finer-grained information about the points in these sets.

I agree that, with access only to the "source object" theta for each point in a set, the discriminator can only force the generator to generate point sets that (i) include points which are mapped to the same theta by T({x_i}), and (ii) match the true marginal p(theta) over objects. I disagree that this failing similarly affects a general DeepSets-style discriminator that isn't constrained to observe the point sets through this T({x_i}). At least, this point is not made by the argument as presented.

We thank you for pointing out some of the recent results of applying deep learning on point cloud for classification and segmentation by leveraging graph convolution. Main focus of our work is to design a generative model for point cloud and thus many of these papers were purposefully not included in related works for being concise. However, it might be a good idea to provide a more exhaustive background of on geometric deep learning and we would definitely discuss the related papers you mention above in next revision of the paper.

Moreover, some of the architectures presented in the mentioned papers, like the point cloud autoencoder with graph convolution may be an alternative choice of Q in addition to DeepSets in the proposed PC-GAN framework, but again is not the focus of our paper. We want to develop a generic framework for generation of point clouds or set data in general, thus employed the most simplistic components like DeepSets. Using the more advanced architectures (e.g. Wang et al., (2018) or PointNet++) should bring in improvements and can be an interesting future direction. We will discuss these aspects in the next revision of the paper. 


The authors should be aware of a large amount of works done on the intersection of ML, CG and CV fields on geometric deep learning on graphs and manifolds that are directly related to their work. In [1], the first intrinsic CNN-like architecture was proposed for 3D shapes, further extended in [2-4]. In [5], the first intrinsic autoencoder for 3D shapes was proposed. Most recent works applying graph-based deep learning techniques to point clouds are [7-8]. Finally, the authors may refer to a review paper [8] on geometric deep learning methods. 


1. Geodesic convolutional neural networks on Riemannian manifolds, ICCV Workshops 2015. 

2. Learning shape correspondence with anisotropic convolutional neural networks, NIPS 2016. 

3. Anisotropic diffusion descriptors, Computer Graphics Forum 35(2):431-441, 2016.

4. Geometric deep learning on graphs and manifolds using mixture model CNNs, CVPR 2017. 

5. Deformable shape completion with graph convolutional autoencoders, CVPR 2018.

6. Dynamic Graph CNN for learning on point clouds, arXiv:1712.00268

7. Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs, 1711.09869

8. Geometric deep learning: going beyond Euclidean data, IEEE Signal Processing Magazine, 34(4):18-42, 2017


Meta-review: Reviewers mostly recommended to reject after engaging with the authors, however since not all author answers have been acknowledged by reviewers, I am not sure if there are any remaining issues with the submission. I thus lean to recommend to reject and resubmit. Please take reviewers' comments into consideration to improve your submission should you decide to resubmit.


Decision: rejected-papers