DUAL ADVERSARIAL MODEL FOR GENERATING 3D POINT CLOUD
Authors: Yuhang Zhang
Zhenwei Miao
Tiebin Mi
Robert Caiming Qiu
Venue:  (2020)
Keywords: point cloud
generative
latent space

Abstract: Three-dimensional data, such as point clouds, are often composed of three coordinates with few featrues. In view of this, it is hard for common neural networks to learn and represent the characteristics directly. In this paper, we focus on latent space’s representation of data characteristics, introduce a novel generative framework based on AutoEncoder(AE) and Generative Adversarial Network(GAN) with extra well-designed loss. We embed this framework directly into the raw 3D-GAN, and experiments demonstrate the potential of the framework in regard of improving the performance on the public dataset compared with other point cloud generation models proposed in recent years. It even achieves state of-the-art performance. We also perform experiments on MNIST and exhibit an excellent result on 2D dataset.

Under review as a conference paper at ICLR 2020
DUAL ADVERSARIAL MODEL FOR GENERATING 3D POINT CLOUD
Anonymous authors Paper under double-blind review
ABSTRACT
Effectively and accurately synthesizing point clouds is yet challenge task due to the high-dimensionality of the observation space. In this article, we focus on the generative model to capture the informative intrinsic structure in the latent low-dimensional space and the diversity in the ambient space. We propose a new dual adversarial model which combines Generative Adversarial Network (GAN) and AutoEncoder (AE) with well well-designed loss. Extensive experiments demon- strate the effectiveness of the framework on 3D point clouds synthesis. The dual adversarial model achieves the ttate-of-the-arr performance comparing with sev- eral well-known point clouds generation models. We also perform experiments on MNIST dataset that exhibit the competitive performance on 2D data.
INTRODUCTION
Point clouds are widely used in many fields, such as vision, 3D object detection and so on. In particular, 3D point clouds represent geometric details of the object, and suitable for simple geometric transformation. However, there exist significant disadvantages in the point cloud data. For instance, data points associated with objects in the distance are difficult identify due sparsity. Moreover, it is not easy to design suitable generative models because of the high-dimensionality of the observation space Gumhold al. 2001); Daniels et al. (2007); Achlioptas et al. (2018). Effectively and accurately synthesizing point cloud data is still challenge task.
qualified point cloud data generation network should satisfy the following conditions: 1) It should be able to synthesize 3D objects that are both highly varied and vivid, which means that there need to be fine details the generated examples Wu et al. (2016). 2) The distribution of the data generated should be similar to that of the original data. If the dimension of the generated sample is higher than that of the raw data, the additional generated portion must follow the potential distribution of raw data points. In the past few years, many researchers have developed lot of meaningful work in this direction, such as 3D modeling and synthesis Carlson| ; Van Kaick et al. (201T). However, many of these traditional methods synthesize objects by borrowing components from existing CAD model libraries. Therefore, synthetic objects may look realistic, but they are not novel in concept in fact.
In recent years, the development of deep learning has provided a new way to solve this problem. Deep learning can well learn the characteristics of data in domain by self-updating. Girdhar et al. 2016); Qi et al. (0016) proposes voxelized-object based method in learning deep object represen- tations, while Kalogerakis et al. (017);; Su al. (2015) from the perspective of view-based projections. Besides these, graph methods Henaff et al. 2015); Defferrard et al. (2016); Yi et al. Bruna Bruna al.] (2013) are also considered for processing algorithms for 3D data. The method of generating point clouds at this stage no longer retrieves them from the object database, instead, most of them synthesize new objects based on learning objects. It is a more difficult task compared to operating 2D data because of the higher data dimension and fewer data features.
Generative Adversarial Network (GAN) Goodfellow et al. (2014) is the most widely used model in the current generation domain and has achieved great success in 2D image generation field. Wu et al. (2016) first applied this network to the 3D point cloud generation field, and named it as 3D- GAN and achieved great results. It introduces adversarial diseriminator to differentiate whether an object is synthesized or real, which may own the potential to capture the structural difference of two 3D objects.

---

Under review as a conference paper at ICLR 2020
Although 3D-GAN has been proved to be successful, it still has some problems. Because of the fact that GAN training is difficult and unstable, its huge network parameters drag down the calculation speed [Salimans et al. (0116) Groueix al. (2018); Li et al. (2019) by adding prior knowledge helping generating target data, iZhao et al. (2018) applied this learned prior to the generation of discrete data, and then AAchiiotas al. (2018) adopts the idea of adversarial in latent space, but the specific approach is completely different. The former proposes a workflow that first learns a representation by training an AE with compact bottleneck layer, then trains a plain GAN in that fixed latent representation. The two parts are trained separately, and their sampling and coding representations are different thus they can't 't be translated or interpreted within both spaces; the latter has used the Adversarial Autoencoder (AAE) Makhzani et al. (2015) for 3D generation, having the distribution latent space of autoencoder trained on point cloud data close to the prior distribution, such as standard gaussian distribution.
The manifold hypothesis indicates that the real-world data in high-dimensional observation space always tends to live in the vicinity of an intrinsic low dimensional manifold al. (201I). In this paper we propose a dual generation model, which can fully make use of the characteristics of data in -iwdesensional latent space to help the network better learn and generate point cloud data. Our contributions are mainly from the following aspects:
Compared the literature study on the latent space of raw data, our method, through rea- sonable theory explaining and experimental verification, proposes dual generate ad adversarial networks architecture, from which the two networks are divided into the primary and sec- ondary part. The secondary network assists the primary network by learning that the data is compressed to features on different dimensions, and our results exceed the same type of articles that focus on hidden space learning Achlioptas et al. (2018).
Our model consists of three parts and can be trained end-to-end, what's more, each part interacts and restricts each other, which can be trained efficiently and steadily without us- ing the pre-trained weights related to point cloud data and only adopt the normal random initialization method. The loss function is designed in the light of the function of each part in the network, and we managed to find the best strategy of each part's iteration renewal through experiments.
The framework we proposed on learning different low low-dimension channel can be trans- ferred to other vision tasks easily. We do experiment on the MNIST dataset to demonstrate its implementability on 2D image tasks.
RELATED WORK
In this section, we mainly introduce the necessary research backgrounds of this paper, including the characteristics of point cloud data and some related work.
Point cloud The point cloud is a collection of massive points that form characteristic of the tar- get's ss surface represents the shape of the object in 3D space. It consists of many æ. y. = coordinates points, which have a matrix structure of N x 3, where N is the number of points in a set. Compared with image data, there are few information and extractable features to point cloud data, how to effectively learn and represent the features of point cloud data has become the focus of current re- search. Wu et al. (2015) took input point cloud the voxelized representation, other methods like Maturana & Scherer (2015) using 3D-CNN to calculate occupancy grid. QQ et al. (2077) proposes the PointNet architecture, can solve the difficulty of dealing with unordered sets by introducing permutation-invarian function instead of convolutions.
Deep learning for generating 3D data Learning and generating 3D object is an essential task in the graphics and vision community. In 3D generating field, Wu et al. (2016) proposes the 3D-GAN, which can generate 3D object from probabilistic space by leveraging recent advances volumetric convolutional networks and generative adversarial nets. [Achiioptas et al. (2018) designed gener- ative model called latent-GAN which is composed of two procedures: first using the point cloud data as input train an Autoencoder (um/Rumelhart et al. (1988), and then using the pre-trained encoder output as the label to train GAN, instead of training on the raw point cloud. Compared with 3D-GAN, its generative ability has improved, but the drawback of this model is that it is not an
2

---

Under review as a conference paper at ICLR 2020
end-to-end one. Sampling and coding representations different and not be translated inter- preted within both spaces. Our model solves this shortcoming by proposing a dual GAN architecture and the result shows that it can largely improves the generation ability.
BACKGROUND
In this part we give the necessary introduction on some basic concepts which we will use in the rest of this paper, including point cloud metrics and some fundamental building blocks.
Autoencoder Autoencoder (AE) is multi-layer neural network that can learn the efficient representation of input data through unsu- pervised learning. It composed of two parts: Encoder and Decoder. The networks' width of the former part gradually becomes shallower the network becomes deeper, while the latter part is reversed, is shown on the right. The Encoder learns a low-dimensional latent representation of the input data x and the Decoder restores z original
Encoder
decodar
Generative Adversarial Network Generative Adversarial Net- work (GAN) is the most widely used generative model today. consists of model generator (G) and a model discriminator (D) and establishes min-max adversarial game between these two models. The generator tries get the function G(z) that maps sample from the prior p(z) the data space. The discriminator tries to distinguish the real data between the generated output. The loss function of this game can be expressed as:
min max Pam-pista [log Dxx)] Fesmptz) [log ((G()))
The model mainly includes two parts of training process: Train the discriminator to distinguish the true label between the fake data generated from sample noise by generator. Train the generator to generate samples which aims to fool the discriminator.
Point Cloud Metries
We adopt two permutation-invann invariant metrics compare unordered point sets, which was proposed by Fan al. (2017).
The Earth Mover's distance (EMD): Introduced in Rubner et al. 2000). it is metrics of transpotation problem based on the mini- mal cost to transform one distribution into the other. For two equally sized subsets $1 C R3 $2 C R², their EMD is defined by:
sapfi
$
D
real data
dEMD S2)
min 0:S1
11r-0(0)
(2)
2ES;
where ((x) is bijection, EMD is differentiable almost everywhere as a loss.
Chamfer pseudo-distance (CD): It measures the squared distance between each point in set to its nearest neighbor in another set, and the function is shown below:
CD(S,, = Σ min
Σ min |
yl12
(3)
vESa
ZES,
FES,
DES,
Compared to EMD. CD is more efficient compute loss and is totally differentiable.
DUAL GENERATION MODEL FOR LEARNING REPRESENTATION
In this section we first describe the architecture of our model and the motivation why we design such network. Then give brief introduction some similar previous work on generating point cloud, and make a comparison of these models to our model. Finally we present detail explanation of the loss function in model and how to train it.
3

---

As the figure shows above, the top of figure is the generator architecture, dimensional transformation occurs at the dotted line. x represents the raw data while D stands for discriminator, & (i=1, 2, ....) stands for the hidden output derived from each dimension of the generator. According to the above hypothesis, we may make an assumption that & (i=1 or 2 or 3,...) is an effective compress of the raw data, which will be evaluated at the next section, then can achieve high level on each Under review as a conference paper at ICLR 2020
4.1 OUR ARCHITECTURE
The dataset we use to present our result is ShapeNet Chang et al. (2015). the dimension of each sample is 2048 x 3. Inspired by the article AAchiiottas et al. (2018) which proposes that using the latent codes instead of raw point data to train 3D-GAN can achieve more precise metrics and less network computing consumption, is sensible make a hypothesis that if learning the latent codes can achieve better result, the generator, in the process of generating images from noise, must have certain dimension that the feature map generated in this dimension is the effective encoding of the raw data. In other words, the function of the latter multiple layers is simply to decode this feature.
Noise
2,
2,
2.
x
D
Figure 1: The architecture of our assumption
indicator by adding an adversarial branch to the 3D-GAN in appropriate dimension and precisely designing loss function and updating strategy of each module. From what has been discussed above,
g;
81
Notse
2,
1
d
d2
Figure 2: The architecture of our approach
we propose a dual generative model which is presented in Figure 2. To make it easier to illustrate, the modules in the structure are named separately as 91, 92. d1, d2 and E. gi and di are combined as generative adversarial model G and they are against each other in latent space. The raw data are trained as the label of G1 after being encoded by module E while 91, G2. d2 are combined as the 3D-GAN model named G2 as figure shows above. To expressly state it, name the lower part of the structure as E which is similar to the model encoder order to be easily understood. fact, there is no real decoder in this model form the autoencoder. During the training process, G2 will
4

---

Under review as a conference paper at ICLR 2020
provide auxiliary training for G2, consider G1 regularization for G2, since it helps G2 more fully trained and better capture the distribution of the raw data.
Our structure combines the advantages of many classic structures in forward propagation, if focus on module E' and 92. this branch like autoencoder; we pay attention to 91, 92 and d2, we will get the structure of raw GAN; if we only 01. d1 and E, they make up latent-gan proposed by AAhhioptas et al. 2018). In other words, our model can be transformed into the three frameworks mentioned above by some tricks, so we can achieve better performance their basis.
4.2 LOSS DESIGN AND TRAINING PROCEDURE
In this section, we will show our design of the error function, as well as the update strategy of each part. We use the loss function LGAN and CAE referred to in Section 3 to compose our loss. The following statements are details of the loss function in each module. We use leaky-ReLU Maaas et al. (2013) as our active function and there is batch-norm.
E: Since the module E is to make an effective dimension reduction for the raw point cloud data, we use the dEMD (X Xreat) or CD(X reaa,, Xyeat) to update its parameters. Xreat is the output of G, so the update process of this module is in cooperation with the auxiliary generator training. The loss function module E is named as LE.
G1: The purpose of module G1 is constrain the output of the hidden layer in the entire model, so that the output distribution of the hidden layer close to the output distribu- tion after the dimension reduction which is operated by module E. We loss function LGAN(noise +2) to train this module.
G2: G2 is a typical 3D-GAN structure, and its loss function must contain CGAN (noise--Xreal). Inspired by Makhzani al. (2015) Larsen et al. (0166), the difference between the generated data and the raw data can be considered as con- straint while updating generation network. Therefore, the loss function of this module is Lg =GAAN =niiinn +XC <>< 1).
The flowchart of the updating process is as follows:
Algorithm Training the Dual Generation Model
initialize parameters of each module 0G,,0a, Fg repeat
Xreat f- random mini-batch from dataset
noise + sampled from prior distribution N(0, /)
2 E(Xreal)
& - G1 (innise)
Xreat & (gnnoise)
LE &d (NEMp(&rent (Xreat: Xreat) or CD((reat, Xyeat)
update parameter OF + Vog
LG Lciniien(noisee>>>
update parameter 0G, + Bci (cAN(noise-2)) V
update parameter fG., Voozz(GGNN XLE)
until algorithm convergence
5 EVALUATION
In this section we will present our results and the enhancement on 2D and 3D data. In 3D case, want evaluate how well the fake data generated by model matches the given set, need comparison to evaluate the faithfulness and diversity of generative model, referring to Achlioptas et al. 2018). will use the following metries:
Coverage is defined as the fraction the point clouds in B, from which we first find its closest neighbor in A. It has two forms: COV-CD and COV-EMD, depending which metrics in Section
5

---

Under review as a conference paper at ICLR 2020
compute point-set distance.The value of coverage will highly increase if most of the point clouds in are more matched to point clouds in A.
M Minimumm Matching Distance MDD Unlike COV, which only focus the closest point cloud and isn't the representative of fidelity, we consider MMD as a metric to report the average of distance in the matching. Just like COV, we can use either of the structural distance, there are MMD-CD and MMD-EMD. The lower the value of MMD is, the stronger the model generation ability will be.
Jensen Shamnon Divergence(JSDD The Lensen-Shannon Divergence (JSD) is used to mea- sure the difference between two empirical distributions. Assume that we have two distribution P and Q, JSD is defined as:
P(x) + Q(x)
P(#) + Q(x)
JSD(PIIQ) = 2 KL P(a)
+ KL ((x))
2
(4)
2
2
where (L(P(x) [[Q(x)) means the KL-Divergence between and Q Kullback & Leibler (1951)
5.1
OTHER WORK FOR CONTRAST
Referring to what was described in Achlioptas et al. (2018) use the following algorithm frame- work as comparison:
Raw-GAN: It use the basic architecture of GAN which directly generates the point cloud data using noise sampled from prior distribution (0,
Instead operating on raw point cloud input, passes the data through pre- trained autoencoder, and each object class is trained separately with EMD or CD loss. The dimensional of bottleneck variable of AE is 128.
I-GAN:
5.2 EVALUATING THE LATENT SPACE REPRESENTATION
We the linear model latent space which is generated by our model E, and its performance can reflect the quality of the latent representation. We train our module E across all shape categories. To obtain features for an imput of 3D shape, we feed the point-cloud into module E and then extract the 128-dimensional bottleneck layer vector. The features are processed by linear classification SVM trained on the 3D classification benchmark of ModelNet Wu al. (2015). Table shows the comparative results.
Dataset
SPH
LFD
3D-GAN
I-GNN
3D-AAE
ours-CD
MNTO
0.798
0.799
0.910
0.954
0.932
MN40
0.682
0.755
0.833
0.845
0.847
0.840
Table 1: The result of classification on ModelNet40 and ModelNet10, from left to right we refer to Kazhdan et al.) (2003): Chen et al. (2003): Wu al. (2016): Achhiiopas al. (2018)
We can clearly see from the table that our method's performance is hetween the performance of I-GAN and 3D-GAN. This result is very consistent with previous inferences. The results of I-GAN's latent space will be directly used for the generation of point cloud images, but the repre- sentation of our latent space is only used as assistance for the main framework. Our results, better than 3D-GAN, also prove that the generation framework, after adding the adversarial branch, has better representation in the latent space.
5.3 EVALUATING THE RECONSTRUCTION
In this section, evaluate the reconstruction capabilities of the proposed autoencoder using unseen test examples. We confront the reconstruction results obtained by the model and use G2 model decoder with approaches to examine the influence of prior regularization reconstruction quality. In Table we show the MMD-CD and MMD-EMD value between reconstruction and corresponding ground-truth in the dataset of chair object class.

---

Under review as a conference paper at ICLR 2020
Method
(tr)(tr)
(SD(te)
MMD-CD
MMD-EMD
AE-CD 0.0028
0.0067
0.0004
0.0527
ours-E
0.0026
0.0067
0.0011
0.0562
Table 2: The result of latent space reconstruction ability (tr:train-spiit, te:test-split)
The result in Table 2 shows that our end-to-end model can approximate result from I-GAN when trained by step. This encouraging result indicates that the generation ability of this network will probably exceed the result of I-GAN step-by-step generation. We will introduce the experiment of this part in the next section.
5.4 EVALUATING THE GENERATION
We train and compare total of five generative models on the point clouds data of car category. The model E' is established with 128-dimensional bottleneck, and trained with CD or EMD loss respectively. The training procedure will be stopped once our model's performance on the valida- tion dataset is poor. According to the loss used by module E, we have two models of evaluating, which are DG-CD and DG-EMD. We also train 3D-GAN directly on the raw point cloud data as comparison.
Method
JSD
MMD-CD
MMD-EMD
COV-EMD
COV-CD
R-GAN
0.1764
0.0020
0.1230
19.0
52.3
I-GAN(CD)
0.0486
0.0020
0.0796
32.2
59.4
I-GAN(EMD)
0.0227
0.0019
0.0660
66.9
67.6
DG-CD
0.0135
0.0017
0.0632
56.3
61.7
DG-EMD
0.0080
0.0013
0.0581
68.2
68.5
Table 3: Evaluating generators on test-split of car dataset, for our model we did five experiments and reported its average value
In two different sets of experiments, measure how well the distribution of the generated samples resembles both the train and test splits of the ground truth distribution, by using our models to generate set of synthetic point clouds and employing the metrics which is introduced above compare with the train or test set distributions respectively. All experimental results are shown in Table and can seen that our method has achieved the best result on all five indicators, which means that in the algorithm based on generative adversarial network structure, our method has achieved the-state-of-ar performance. Figure with the results of the R-GAN in generating point cloud data. We can clearly see when generating the same dimensional point cloud data, our method more concentrated and its contour more obvious. Figure 4 shows some other examples and results.
comparison of the results of our method
84
84
00
00
-34
-04
-24-22 00
00
84
Figure Left: Image generated by R-GAN
Right: Image generated by our method

---

Under review as a conference paper at ICLR 2020
Figure 4: Some results generated by our method
5.5
2D RESULT
To prove the mobility of this structure, did some experiments on the 2D image dataset. We use our method to generate MNIST dataset and compare with the result of Goodfellow et al. (2014), that the edges of the images generate are sharper and the texture of the picture is more clear.
7
5
7
3
<
2
3
9
7
7
1
9
4
4
2
/ 8
2
2
0
8
9
6
L
2
7
3
2
4 2 7
9
3
7
7
9 8
0
5
7
9
/ 3
8
7
3
3
8
5
7
2
3
O
/
3
3 0 8 4 6
4 9 2 9
6
4 3 3
2
2
8
9
7
7 7 6
4
5 7
3
6
5
8
7
4
9
7
0
4
O
4 4
2
7
3
0
7
6
8
4
6
8
{
8
5 5
4
8
4
7
2
8
Figure 5: Left: Image generated by GAN
Right: Image generated by our method
Model
Test Error @ samples
Real Data
3.1%
GAN
6. 2.28%
Ours
3. 3.23%
Table 4: Nearest neighbor classification results
We evaluated the model using nearest neighbor classifier comparing real data to a set of generated conditional samples, we can see from Table 4 that our approach outperforms the original GAN,
CONCLUSION
In this paper propose a dual generation architecture for learning and generating 3D point cloud data, and our experiments show that learning the characteristics of raw input data in different coding- and-low dimension indeed help the GAN to generate object. Compared with other generative frameworks, our model an end-to-end architecture which has better performance and higher effi- ciency. Although experiment result may not be sufficient due to limited time, the insight of our work is worth for further study and discussion, what's more, the framework based on our learning and representing latent space feature will surely achieve better result in the future.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for 3d point clouds. In International Conference on Machine Learn- ing. pp. 40-49, 2018.
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv: 2013.
Wayne Carlson. An algorithm and data structure for 3d object synthesis using surface patch intersections. ACM SIGGRAPH Computer Graphics, 16(3):255-22) 263,,, 1982.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, and Su Hao. Shapenet: An information-rich 3d model repository. Computer Science, 2015.
Ding- Yun Chen, Xiao-Pei Tian, Yu-Te Shen, and Ming Oubyoung. On visual similarity based 3d model retrieval. In Computer graphics forum, volume 22, pp. 223-232. Wiley Online Library, 2003.
Joel II Daniels, Linh K Ha, Tilo Ochotta, and Claudio T Silva. Robust smooth feature extraction from point clouds. In JEEE International Conference Shape Modeling and Applications 2007, pp. 123-136. IEEE, 2007.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances Neural Information Processing Systems, pp. 3844-383300 3852, 2016.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object recon- struction from a single image. IEEE Conference on Computer Vision and Pattern Recognition, pp. 605 613, 2017.
Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning a predictable and generative vector representation for objects. In European Conference on Computer Vision, pp- 484-499. Springer, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.
Thibault Groueix, Matthew Fisher, Vladimir Kim, Bryan Russell, and Mathieu Aubry. 3d- coded: 3d correspondences by deep deformation. In European Conference Computer Vision, pp. 230-246, 2018.
Stefan Gumhold, Xinlong Wang, and Rob Macleod. Feature extraction from point clouds. In 10 th International Meshing Roundtable. Citeseer, 2001.
Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured data. arXiv preprint 30Xi::5506.0163 2015.
Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 3d shape segmentation with projective convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3779-3788. 2017.
Michael Kazhdan, Thomas Funkhouser, and Szymon Rusinkiewicz. Rotation invariant spherical harmonic representation of 3d shape descriptors. In Symposium on geometry processing, vol- ume 6, pp. 156,164, 2003.
Solomon Kullback and Richard Leibler. On information and sufficiency. The Annals of Mathe- matical Statistics, 22(1):79-88) 1951.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using a learned similarity metric. In International Conference on Machine Learning, pp. 1558-1566, 2016.

---

Under review as a conference paper at ICLR 2020
Chun-Liang Li, Tomas Simon, Jason Saragih, Barnabás Póczos, and Yaser Sheikh. Lbs autoencoder: Self-supervised fitting of articulated meshes to point clouds. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 11967-11976, 2019.
Andrew I. Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing. Citeseer, 2013.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey, Adversarial autoencoders. International Conference on Learning Representations, 2015.
Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Sys- tems, pp. 922-928. IEEE, 2015.
Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas Guibas. Volumetric and multi-view cnns for object classification on 3d data. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 5648-5656, 2016.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for classification and segmentation. IEEE Conference on Computer Vision and Pattern Recognition, pp. 652-660, 2017.
Salah Rifai, Yann N Dauphin, Pascal Vincent, Yoshua Bengio, and Xavier Muller. The manifold tangent classifier. Advances in Neural Information Processing Systems, pp. 2294-2302, 2011.
Yossi Rubner, Carlo Tomasi, and Leonidas Guibas. The earth mover's distance as metric for image retrieval. International Journal of Computer Vision, 40(2):99-11) 121, 2000.
David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, al. Learning representations by pook-propaggting errors. Cognitive modeling, 5(3):1, 1988.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pp. 2234-2242, 2016.
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convolu- tional neural networks for 3d shape recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 945-953, 2015.
Oliver Van Kaick, Hao Zhang, Ghassan Hamarne,, and Daniel Cohen-Or, A survey on shape cor- respondence. In Computer Graphics Forum, volume 30, pp. 1681-1707. Wiley Online Library, 2011.
Jiajun WI, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning prob- abilistic latent space of object shapes via 3d generative-adverseee adversarrall modeling. In Advances in Neural Information Processing Systems, pp. 82-90, 2016.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianx- iong Xiao. 3d shapenets: A deep representation for volumetric shapes. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1912-1920, 2015.
Li Yi, Hao Su, Xingwen Guo, and Leonidas Guibas. Synespecenn: Synchronized spectral cnn for 3d shape segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 2282 2290, 2017.
Junbo Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M Rush, Yann LeCun, et al. Adversarially regularized autoencoders. In International Conference on Machine Learning, pp. 5897-5906, 2018.
10

---

Under review as a conference paper at ICLR 2020
A How À AFFECT THE RESULT
In section 4.2, we denoted LG =LGAN(noise = <X< 1). Different will lead to different results figure shows below.
04
02
03
-04
#97686204
20
02
chietal m2
Figure 6: À from left to right 0.95.0.1.0001.000001
From the figure shows above we can see that when À is large, shown in the leftmost column above, the generated image edge is clear enough and is of less diversity; when À is small, shown in rightmost column, we observe that the generated image is slightly distorted. So we adopt À = 0.01 in our experiments.
11


Reviews:
[{"review_id": "SkxxhWRZqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper focuses on designing a generative framework for 3-D point data clouds. These point clouds correspond to objects shapes in 3-dimensions. According to the paper, previous approaches for generating such 3-D point clouds involved autoencoder and GANs used separately. The authors propose a framework combining both autoencoder and GAN in a single network. The authors claim that part of the network learns effective latent space embedding for 3-d point clouds corresponding to different objects and thus the entire network is more effective in generating 3-d point cloud. Experimental results are presented to support the claims for efficient embeddings, and better generation of 3-d point clouds.\n\nDecision\n\nThe paper has some lacking in explanation. I am particularly interested in the answers to the following questions:\n1.\tThe encoder module (denoted as E in the paper) uses a loss function that apparently does not involve the encoder output (z in the paper). How the module weights can be updated this way is unclear.\n2.\tG1 is updated twice in each iteration, according to algorithm 1 in paper (once during update of 𝜽G1 and 𝜽G). How the generator G1’s output manages to converge to output z from encoder E is unclear.\n3.\tThe authors claim that the proposed model has both better performance and efficiency. Although experimental results are provided to support claim for better performance, none have been offered for efficiency claim. (calculation of EMD distance seems to be very expensive computationally)\n\nFeedback\n\nFor section 5.5, a quantitative analysis might make the claim for portability of the framework stronger. In paragraph 1 of introduction, disadvantage of 3d point cloud data can be better explained.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJl0wPBLqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new dual generation model for learning representation by combining GAN and Autoencoder for 3D point cloud data. The idea is to add a regularization term over the GAN loss, and the regularization term measures the distance between the output of the encoder of Autoencoder and that of GAN by two point cloud metrics: Earth Movers's distance (EMD) and Chamfer pseudo-distance (CD). The experiment shows better representation are learnt by this dual generation model. The paper is in a good writing, and easy to read.\n\n``My suggestions:\n\n1) Although this paper is about 3D-point cloud, the idea itself is also feasible for 2D data. In the experiment, a small section is on 2D image data. In order not to confuse readers, I would suggest writing this paper as a general framework on both 2D and 3D data. Is there any particular reason for this work focusing on 3D point cloud data?\n\n2) In 2D data, how to measure the point-cloud distance?still using EMD and CD?\n\n3) In Figure 2, are you trying to minimize the distance between intermedia representation z and \\bar{z} from encoder and GAN respectively as well? I do not find corresponding step in Algorithm 1.\n\n4) How is the lambda changing the results? The regularization is the main idea for this paper, and thus it would be interesting to see how the regularization trades off with the loss.\n\n5) how is EMD's performance compared with CD?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xtcdeTFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper propose a dual generator approach for learning to generate point clouds.  The proposed algorithm is a adversarial autoencoder with learned prior generation of point clouds. From this point of view, the novelty of the paper is limited, also, the authors failed to discuss all the related works properly. \n\n1. The problem of the point cloud generation of Achlioptas et al., (2018) is it can't generate arbitrarily many number of points. This issue has been discussed and addressed in those works \n\n* Li et al., Point Cloud GAN, ICLR workshop, 2019.\n* Yang et al., FoldingNet: Point Cloud Auto-encoder via Deep Grid Deformation, CVPR, 2018\n* Groueix et al., AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation, CVPR 2018\n* Yang et al., PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows, CVPR 2019\n\nWith stronger prior in the generation process is even studied\n* Groueix et al., 3d-coded: 3d correspondences by deep deformation, ECCV, 2018\n* Li et al., LBS Autoencoder: Self-supervised Fitting of Articulated Meshes to Point Clouds, CVPR 2019\n\n2. The two generator approach is nothing new.  The first generator g_1 is the same as the \"learned prior\". There are many related works,  e.g.\n* Tomczak et al., VAE with a VampPrior, AISTATS 2018\n* Zhao et al., Adversarially Regularized Autoencoders, ICML 2018\n\n3. The paper is not well written, there are several wrong formats of citations and some typos, especially on math symbols, which hinders the reading.  (e.g. G is undefined in Algorithm 1)", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new method for 3d point cloud generation based upon auto encoders and GANs.

Two reviewers voted for accept and one reviewer for outright reject. Both authors and reviewers posted thorough responses. Based upon these it is judged best to not accept the paper in the present. The authors should take the feedback into account in a an updated version of the paper.

Rejection is recommended.  
Decision: Reject