Rethinking Sampling in 3D Point Cloud Generative Adversarial Networks
Authors: He Wang
Zetian Jiang
Li Yi
Kaichun Mo
Hao Su
Leonidas Guibas
Venue:  (2021)
Keywords: 3D point cloud
GAN
sampling pattern
evaluation metrics
discriminator

Abstract: In this paper, we examine the long-neglected yet important effects of point sam- pling patterns in point cloud GANs. Through extensive experiments, we show that sampling-insensitive discriminators (e.g. PointNet-Max) produce shape point clouds with point clustering artifacts while sampling-oversensitive discriminators (e.g. PointNet++, DGCNN, PointConv, KPConv) fail to guide valid shape generation. We propose the concept of sampling spectrum to depict the different sampling sensitivities of discriminators. We further study how different evaluation metrics weigh the sampling pattern against the geometry and propose several perceptual metrics forming a sampling spectrum of metrics. Guided by the proposed sampling spectrum, we discover a middle-point sampling-aware baseline discriminator, PointNet-Mix, which improves all existing point cloud generators by a large margin on sampling-related metrics. We point out that, given that recent research has been focused on the generator design, the discriminator design needs more attention. Our work provides both suggestions and tools for building future discriminators. We will release the code to facilitate future research.

Under review as a conference paper at ICLR 2021
RETHINKING SAMPLING IN 3D POINT CLOUD GENERATIVE ADVERSARIAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we examine the long-neglected yet important effects of point sam- pling patterns point cloud GAÑs. Through extensive experiments, we show that sampling-insensitvee discriminators (e.g. PointNet-Max) produce shape point clouds with point clustering artifacts while ceampling-oversennitee (e.g. PointNet++, DGCNN, PointConv, KPConv) fail to guide valid shape genera- tion. We propose the concept of sampling spectrum to depict the different sampling sensitivities of discriminators. We further study how different evaluation metrics weigh the sampling pattern against the geometry and propose several perceptual metrics forming a sampling spectrum of metrics. Guided by the proposed sam- pling spectrum, discover middle-point sampling-aware baseline discriminator, PointNet- which improves all existing point cloud generators by a large margin sampling-relate metries. We point out that, given that recent research has been focused on the generator design, the discriminator design needs more attention. Our work provides both suggestions and tools for building future discriminators. We will release the code to facilitate future research.
discriminators
INTRODUCTION
Point cloud, as the most common form of 3D sensor data, has been widely used a variety of 3D vision applications due to its compact yet expressive nature and its amenability to geometric manipulations. It is natural consider how to generate point cloud through deep learning approaches, which has been a popular research topic recently. The previous research efforts in the community have been mainly devoted to conditional generation of point clouds with 3D supervision. The condition
could either be images (Fan et al. 2017 Groueix al. 2018 Park al. 2019) partial point
clouds (Li et al. 2019 Yang et al.
2018).
fake
real
SamptingSpectrum
a
(b)
(c)
(d)
sampling-insensitive
(e.g. PointNet-Max)
sampling-aware
(e.g. PointNet-Mix/Attenniee
sumpling--warrcnssiive
(e.g. DoinNNet=,DDCNNN
Figure 1: We visualize the behavior of different discriminators when judging four different chair point clouds. (a) and (b) are generated results, (c) and (d) are point clouds sampled using FPS and uniform sampling from a real chair surface. When training the discriminators on data like (d), different discriminators make distinct decisions the point cloud realness forming sampling spectrum ranging from caapllinginnensitivee sampling-awar to campling-oversensittie aware discriminators in the middle of the spectrum, which provide good guidance for fixing geometric flaws (a) and big sample artifacts (b) and tolerate subtle sampling differences between (c) and (d).
We advocate sampling-

---

Under review as a conference paper at ICLR 2021
Generating 3D point clouds with GANs in an unsupervised manner is important but less explored problem. 3D point cloud GAN learns to transform random latent code into a 3D surface point cloud
by playing an adversarial game. Its development is still in an early stage compared with 2D image
GANs. While existing works such (Achlioptas al. 2018 Valsesia et al.] 2018 [Shu al., 2019)
have developed a variety of generators, they all use PointNet (Qi et al. 201 with max pooling (PointNet-Mane as their discriminator. PointNet, which is essentially a pointwise MLP followed by
global pooling operation, is too limited in capturing shape details for successful GAN. However, advanced networks, e.g. PointNet++ et al. 2017b). DGCNN(WWNg et al. 22119) KPCony (Thomas) et al., 2019), PointConv (Wu et al., 2019), which leverage relative positions between points and hierarchical feature extraction, may not help. From our empirical study, we find they all fail to be a functioning discriminator. Understanding their failure mechanism and improving discriminator design are hence important and urgent.
To design a better discriminator, we first need to answer the following question: what should the discriminator examine for improving the generation quality? Or, even more fundamentally, what does it mean by the quality of generated point clouds?
Since a shape point cloud are the points sampled from an object surface, its quality should be evaluated from two perspectives: the depicted surface geometry and the point sampling. Arguably, geometry plays decisive role and should be the main focus of a discriminator. However, when the generated point clouds have a good shape, there still a full spectrum on how much a discriminator cares about the sampling patterns. We introduce the concept of Sampling Spectrum to depict the discriminator
sampling sensitivity of discriminators, illustrated in Figure [ A campling-insensitte (e.g. PointNet-Maz -Ma) may ignore the point density variations as long as it perceives a good overall shape. Such discriminator could identify big geometric flaws shown in Figure (a). but turns a blind eye to highly non-uniform density distribution, e.g. point clusters in Figure (b). On the other extreme, a sampling-overseee oversensitive discriminator (e.g. PointNet++, DGCNN, KPConv) can even tell the subtle difference in sampling patterns, e.g. between furthest point sampling (FPS) in Figure (c) and uniform sampling in Figure [ (d), and hence can be very narrow- minded about what a real point cloud should look like. A sampling-awer in the middle of the spectrum, is able to identify density-related artifacts such as point clusters in Figure (b), while not being too sensitive to different sampling patterns, such as Figure I (c) and (d).
discriminator, e.g. PointNet-Mixt Mix/Attention, which lies
Resembling the sampling spectrum of discriminators, we also examine the existing point-cloud GAN evaluation metries from the perspective of sampling sensitivity and propose several perceptual metries forming a Sampling Spectrum of evaluation metrics. Understanding how the metrics weigh between sampling and geometry is prerequisite for evaluating point cloud GANs. Many of the existing coaplingg-nnensitive metrics only evaluate the geometry factor of the generated point clouds shapes, which are blind to the obvious point clustering artifacts and uneven point density. We propose novel compling-sensitivy metrics to further complete the spectrum of point-cloud GAN evaluation metrics.
Guided by the proposed sampling spectrum of discriminators and evaluation metrics, experiments show that different discriminators in the spectrum could provide very different suggestions to improve a generator according to its sampling sensitivity. Max, is unaware of point density variations and hence its generated point cloud inevitably suffer from clustering artifacts, while sampling-oversensiin KPCony, PointConv, simply fail to function the discriminators and can generate much degraded point cloud shapes. We design a diagnostic "no generator" experiment to factor out the impact from generators and reveal that the gradients of sampling-oversensiing sampling patterns over producing better shape geometry. Picking a middle- -oiit on the sampling spectrum, we discover a simple yet effective sampling-awar discriminator, PointNet-Mix, and find that it can supervise both shape generation and point density uniformity. It improves all existing generators by large margin on sampling-related metrics. Surprisingly, we find that even the most naive fully-connected generator, coupled with PointNet-Mix, simply beats all the start-of the-art point cloud GANs. This discovery conveys an important message to the community: instead of focusing on the generator design. people should invest more time into discriminator and seek for more powerful sampling-awan discriminators.
coapling-insensiiie discriminators. e.g. PointNet-
discriminators, e.g. PointNet+++ DGCNN,
discriminators prioritize adjusting
2 POINT CLOUD GAN LANDSCAPE
In this section, we review the current state of point cloud GAN covering the generators, the discrimi- nators, and the evaluation metrics we are examining in this work.
2

---

Under review as a conference paper at ICLR 2021
2.1 POINT CLOUD GAN GENERATORS
Recent point cloud GAN works primarily focus on the generator design. The generator takes input a random noise & ~ (0, 1) and outputs a point cloud p E RN The existing generators can be categorized into two classes: fully-connected (FC) generators and graph convolutional generators. The first point cloud GAN, r-GAN (Achlioptas et al. 2018), simply uses FC network as its generator. GraphCNN-GAN GAN (Valsesia et al. 2018) and TreeGAN (Shu et al. 2019) are the rest two published works in this field that use graph convolutional generators. The two methods are very similar in principle. The main difference lies in how they build graphs. GraphCNN-GAN a dynamic k-nn graph based upon feature space distance while TreeGAN enforces a tree structure throughout its sequential graph expansion and the messages can only be passed from ancestors vertices to descendants vertices.
builds
Deformation-based decoders (Groueix et al., 2018 Yang et al. 2018) are widely used in the point cloud auto-encoder networks for 3D shape reconstruction. The decoders leverage Multiple-layer Perceptrons (MIP) to deform template surfaces into shape surfaces taking as inputs the concatenation of template point coordinates and the latent shape feature vectors. Though the decoders can truly act as generators for point cloud GANs, they have not yet been used in unconditioned point cloud GAN literature. Recently, Mo et al. (Mo et al. (2020) use deformation-bens -based decoder as part generators for structure-conditioned point cloud GAN.
2.2 POINT CLOUD GAN DISCRIMINATORS
All existing works on unconditioned point cloud GANs use PointNet with max max-pooling (PointNet- Max) as their discriminators. The discriminator takes input point cloud p E Nx3 and outputs score from O to 1. PointNet learns function h that maps each point p; in the point cloud to per-point feature h(p:) E Ied and then extracts permutation-invarian pooling the per-point features across all points using a symmetric function g. which can be max pooling, average pooling, etc. Namely, we have F = (((h(11))h(p)) ... h)px))).
global feature F E Rd by
Though g can be any symmetric function, most existing works use max- pooling. In PointNet, the authors show that max-pooling outperforms average-pooling on 3D shape classification tasks. They further show that the global feature F obtained from max-pooling is determined by only a sparse subset of the points, namely critical points Cg. bringing PointNet-Max with robustness against small data perturbation and corruption. However, this property may limit its discriminative power on telling the density variations and classifying different sampling patterns. To investigate how different aggregation operations affect the sampling sensitivity and generation quality, we study the two common choices of the symmetric function g, max-pooling, and average-pooling in this paper. We will show in Sec. 3 that using different aggregation operations makes huge differences when adapting PointNet-based networks point cloud GAN discriminators.
PointNet-Max/Avg,
PointNet-Mix By simply concatenating the max-pooling feature and the average pool feature, we obtain another germutation-invariant feature. We name this PointNet-Mix. Formally, Fmix = [max (h(p)), ...,,h(p }}; avg (h())) ),... h(px)]] E Rddd The mix-pooling operation is special choice of the symmetric function g. We will show in our experiments that PointNet-Mix, though simple, surprisingly improves the performance for most point cloud GANs by a large margin on sampling-awar metrics.
PointNet-A ttention. A recent point cloud upsampling work (Li et al., 2019) incorporates self- attention module into PointNet for its discriminator and shows improved point density in the upsam- pled point cloud. We denote this discriminator as PointNet-Attention The self-attention module learns three separate MLPs to transform each h(p,) into (p.), 1(p:), k(p,) E Rd correspondingly. Then an (Ix) N A weighted features is then obtained through w(p:) = h(pp)) + WTpp). The fi- nal aggregated features of PointNer-Attention is max-pooling of the weighted feature, namely Fanention = max (w(p))..., (p Note that PointNet-Attention allows the per- point features to communicate with each other according to their similarity, which is more sensitive to sampling patterns than PointNet-Max. We investigate this strategy in our paper as well.
attention weight matrix is formed by W = SoftMax
E
Discriminators beyond PointNet. Recently, there have been many works (Qi et al. 2017aj
et al. 2018: Wang et al.) 2019) Hermosilla et al.) 2018 TThoas et al. (209)) extending PointNet to more advanced 3D deep learning architectures point clouds. They improve PointNet by extracting more local or hierarchical geometric features via point cloud convolutions or point cloud graph learning. Though proven to be effective on shape classification and segmentation tasks, no published work examines adapting them as point cloud GAN discriminators. In this paper, we investigate four
3

---

Under review as a conference paper at ICLR 2021
exemplar PrrttN-P discriminators: PointNet++ (Qi et al. 2017b), DGCNN (Wang et al.
2019). KPConv (Thomas et al. 2019). PointConv (Wu al. 2019).
2.3 POINT CLOUD GAN EVALUATION METRICS
Achlioptas et al. (Achlioptas al. 2018 introduces two distance metrics in the Euclidean space for evaluating point cloud GAN: the coverage scores (COV) computing the fraction of the point clouds in B that are closed to the point clouds in A using either Earth Mover's 's distance (EMD) or Chamfer distance (CD), and the Minimum matching distance (MMD) scores measuring the fidelity of with respect to B using either EMD or CD. In the field of 2D image GAN, is common to use perceptual metrics, such the Frechét distance (Heusel et al.) 2017) between real and fake Gaussian measures in the feature spaces, for evaluating the generated image results. Formally,
Frechet Distance = I|,e u 12 + TrE, + nn 2 ,Σ,,,,) 1/3) where μ and Σ are the mean vector and the covariance matrix the features calculated from either real or fake data distribution, and is the matrix trace. For point clouds, Shu et al. (Shu et al.) 2019) proposes Frechét Point Cloud Distance (FPD), which uses the features extracted from a pre-trained PointNer-Max model. In this paper, we position the existing metries on sampling spectrum according to their sampling sensitivity and propose novel sampling-awan metrics to augment the spectrum.
SAMPLING SPECTRUM OF DISCRIMINATORS
While recent works propose many advanced generator improvements for point cloud GANs, we find that designing a good discriminator is of equal importance, if not more. In this section, we introduce the sampling spectrum of discriminators, on which we thoroughly examine the positions of different discriminators that explain their behaviors when training point cloud GANs.
3.1 SAMPLING SENSITIVITY OF DISCRIMINATORS
The sampling sensitivity of a discriminator depicts how much responds to a change in the point density or sampling pattern of an input point cloud. We find it extremely hard to quantitatively measure this sensitivity given the difficulty of measuring changes in the sampling patterns. Naively using Euclidean metrics (e.g. CD or EMD) to measure the distance between two sampled point sets is not a solution, since given the same distance budget, the discriminator's 's responses can be dramatically different depending on how the sampled points move. Instead, we can landmarks in the continuous spectrum of sampling sensitivity by examining the discriminative power of the discriminators against different sampling patterns under series of experiments from easy to hard. Specifically, we design two experiments to test whether discriminator could tell clustering artifacts in point clouds and whether it could distinguish between FPS and uniform sampling patterns. Accordingly, we divide the spectrum into three regimes: sampling-
-ww.r//overseensiiie as shown in Fig
sampling-insenae insensitive discriminator does not respond to local point density changes if the overall shape remains roughly the same. This kind of discriminators can't 't tell clustering artifacts, i.e. Fig (b), and thus may cause the non-uniform density in the generated point clouds.
sampling-aware discriminator can notice the significant point -oifforfory and is hence capable of supervising the generator to enforce a similar sampling distribution to the training data, while being ignorant to subtle changes when the sampling is already uniform in an intermediate scale. Such discriminator will judge Fig (b) as fake but can't 't tell the difference between Fig (c)) and (d).
A
discriminator can tell very subtle difference in the sampling patterns, e.g. between FPS and uniform sampling, even if the underlying shapes are the same (Fig (c) and (d)).
3.2 SAMPLING SENSITIVITY EXAMINATION RESULTS
We design diagnostic experiments to categorize point cloud GAN discriminators on the sampling spectrum: -aaplingginsensitvvee (PointNet- Maant,ee-Max,,,,, sampling- aware (PointNet-Avg, Poin PointNet-Attention (Li et al. 2019),, and sampling-oversensitive (PointNet++, DGCNN, KPConv, PointConv).
The Discriminating Power against Clustering Artifacts. To examine whether the discriminators can tell clustering artifacts or not, we create diagnostic classification dataset using the models from ShapeNet (Chang et al. 2015). Taking 100 chairs, we uniformly sample 2048 points from each shape, forming set of real point clouds. To form fake of point clouds, for each chair, we first uniformly sample 1024 points, and then densely sample another 1024 points around a random position on the chair within a 0.1 radius. The real/fake point clouds are used as the labeled training dataset. We repeat the same process to generate test dataset using different set of 100 chairs.
4

---

Under review as a conference paper at ICLR 2021
Experiment
PN-Max
PN-Avg
PN-Mix
PN-AIt
PN++
DGCNN
PoineCony
KPConv
Clustering art. FPS vs. uniform ESS
5699/33%
9894/99%
979/66
9498/933
100%/98%
1009/999%
100%/99%% 100%/98%
1078/99%
50%//50%
50.3%
51.5%
50. 1%/500%
100%/97%
100%/966
100%/99%
0.46
0.95
0.95
0.97
0.99
Table I: Evaluating the discriminating power of the discriminators against clustering artifacts and sutble change in sampling patterns along with their empirical sampling sensitivity (ESS). In each cell of the top two rows, the left shows the training accuracy while the right shows the test accuracy.
We supervisely train each discriminator to classify the real/fake point clouds. We train them 200 epochs until convergence. The training and test accuracies are shown in the first row of Table Despite the huge density variation and the remarkable clustering artifacts, DointNet-Max is just barely better than random guess while the rest of the discriminators are very successful in telling the fake from the real. This indicates that PointNer- -ax is campling-insensiing power of network towards certain artifacts is maximized under such supervised training scheme. A network will fail to identify such artifacts in an adversarial training scheme if it fails in supervised training scheme. We will see in Sec. 5.2 point clouds generated by GAN using PointNet- Max as the discriminator indeed suffer from non- con-uniform density artifacts.
Note that the discriminating
Our insight why PointNet-Avg/Mi can tell the artifacts but PointNet-Max fails is that the average pooling feature computes the center of the mass of the points in feature space and is hence aware to certain global non-uniform density distributions. PointNet-Attention leverages learnable weighted averaging and is hence capable identify the difference.
Distinguishing between FPS and Uniform Sampling. We construct another diagnostic dataset with real and fake data which are the same in their shapes but only differ in their sampling patterss. Specifically, we perform uniform sampling to generate real data and use FPS to generate fake data from 100 chairs. Figure [ (c) and (d) illustrate the different sampling pattern outcomes.
We present the training and test accuracies in the second row of Table I and show that PointNet++, DGCNN, KPConv, and PointConv can perfectly distinguish the subtle difference in sampling patterns while the rest discriminators make no progress even on the training set. The experiment indicates that SoaulNee-Av//Mi//Ateeeee PointConv are sampling cornpliig-oversensiiiee oversensitive.
are sampling-aware while PointNet++, DGCNN, KPConv and
We believe, for PointNet+++, DGCNN, KPConv, and PointConv, their capability of distinguishing FPS from uniform sampling owes to their usage of relative point positions or edge information, which are highly sensitive to any change in sampling. We will show in Sec. [53 that their remarkable discriminating power on sampling patterns actually leads to the failures as functioning discriminators to train point cloud GANs.
Empirical Definition of Sampling Sensitivity Given the difficulty of measuring the change in sampling pattern, we would like to provide an empirical definition of sampling sensitivity of one discriminator according to its performance on the two experiments above: when the discriminator fails in telling clustering artifacts and telling the difference between FPS and uniform sampling pattern, its sampling sensitivity should be closed to O; when the discriminator can tell clustering artifacts but can't distinguish between FPS and uniform sampling, sensitivity closed to 0.5; when the discriminator can do both, we want to assign a sampling sensitivity closed to to it. Based on this, we define empirical sampling sensitivity ESS E [0,1] as:
want assign a sampling
ESS ATel clustering art.) + AcelTell FPS from uniform sampling) 1,
where Acc denotes the test accuracy of an experiment. See Table I for results. Using this metric, we would form a continuous sampling sensitivity spectrum.
4 SAMPLING SPECTRUM OF EVALUATION METRICS
Similar to the discriminator design, it is very important to understand how different evaluation metrics weigh the differences in sampling patterns against geometry quality. Thus, we introduce the sampling spectrum of evaluation metrics, which exactly resembles the sampling spectrum of discriminators introduced in Sec. 3 On the spectrum, we have sampling-insensitive metrics that measure only the shape geometry and are ignorant of the sampling patterns, and sampling-sensitiv metrics that measure both at the same time.
For perceptual metrics, f.e. Frechét distances in feature spaces, the sampling sensitivity of the metric purely depends on the sampling sensitivity of its feature extractor. Frechét distance measured in
5

---

Under review as a conference paper at ICLR 2021
Data
FPD-Mix
FPD-Max
PGD|
MMD-EL
MMD-CL
COV-E"
COV-C*
Unitorm re-sampling
0.1153
0.0926
0.8141
0.1104
0.00145
70.69
72.16
Farthest point
0.1200
0,1558
1,8833
0.1064
0.00137
67.74
69.36
Riased sampling 2.8631
0.3524
9.6719
0.2469
0.00145
23.12
71.28
Table 2: Examining sampling sensitivity of evaluation metries.
different feature spaces may respond very differently to changes in point density and sampling patterns. In this work, we examine three Frechét distance metries that extract features using PointNet- Max, Poi,, and DGCNN, respectively. We denote them as Frechét PointNet-Max Distance (FPD-Max), Frechét PointNet-Mix Distance (FPD-Mix), and Frechét DGCNN Distance (FGD). We pretrain all the three feature extraction networks on ModelNet40 (Wu et. 2015) shape classification.
To examine the sampling sensitivity of FPD-Mix/Max and FGD, we create several copies of the training split of our ShapeNet chair dataset (see Sec each of which uses different sampling strategy obtain the shape point clouds. The reference one used the ground truth is using uniform sampling. Then we consider 1) uniform sampling with a different random seed; 2) FPS; and 3) biased sampling with clustered artifacts (as described in Sec. 3.2). We use all the available metrics to evaluate their distances to the ground truth data. The results are shown in Table 2
We observe that the Frechét distance metrics share the same sampling sensitivity their correspond- ing discriminators. For example, since PointNet-Max is campling-insensitvve FPD-Max remains very low even on biased sampling data, hence FPD-Max serves as a perceptual geometry metric, which is ignorant to sampling patterns. Similarly, we find that FPD-Mix is sampling-aware since it clearly detects the biased sampling patterns while not being able to distinguish the uniform sampling in that can tell apart FPS and uniform sampling.
and FPS, while FGD is aampling-vversensee
For Euclidean distance metrics, results in Table 2 show that COV-EMD and MMD-EMD are sampling- aware, which is intuitively reasonable since EMD enforces one-to-one matching and is aware of the point density, while COV-CD and MMD-CD are
EXPERIMENTS
Aware of both the sampling spectrums, we conduct experiments to further evaluate the performance of point cloud GANs under various evaluation metrics. We show that the point cloud GANs using campling-insensitee
discriminators may produce point clustering artifacts. while sampling- oversensitive discriminators fail to supervise point cloud GAN training at all. We further devise a diagnostic no-generatorr experiment that factors out the generators to better illustrate discoveries on discriminators. More interesting, we find that the simple PointNet-Mix paired of any generator, even with the most naive fully-connected one, achieves the state-of-the performance.
5.1 SETTING AND DATASETS
We provide a thorough comparison of all the discriminators investigated in combining with all the available generators in the published literature, including the FC generator proposed in r- GAN (Achlioptas et al. 2018), and graph convolutional generators used in TreeGAN (Shu al., 2019). We also add a deformation-basee generator into the comparison given its popularity for supervised point cloud reconstruction (Groueix et al. 2018 Yang et al. 2018).
We use two datasets to evaluate the GANs. One is single-categon sampled from all 6,778 chair meshes in ShapeNet (Chang et al. 2015). The other is a multi-category dataset combining shapes from airplane, car, chair, rifle, sota, table, vessel categories in ShapeNet. The multi-category dataset contains 34,313 shapes in total. We uniformly sample 2048 points from each shape to form the two datasets. We follow the 85%/5% Train/vllidation/ees split in (Achlioptas al. (2018) and use WGAN-gp (Gulrajani et al. 2017) for the GAN training, similar to previous works (Achlioptas et al.) 2018 Valsesia et al.) 2018 Shu et al. 2019).
dataset containing point clouds
5.2 EVALUATING POINTNET-BASER DISCRIMINATORS WITH VARIOUS GENERATORS
We report the performance for point cloud GANs that combine inators and Ceteefooo//reeGA//GGii PointNet- as the discriminator outperform the ones using DointNet-Max/AAttntio across all differ- ent generators. On aaaeiing-aarr/eessiivvve metries (i.e. FPD-Mix, FGD, MMD-EMD, COV-EMD), PointNet-Mix is always significantly better than PointNet- and is better than PointNet-Attention mainly on FPD-Mix and FGD. Regarding geometry quality evaluated using sampling-insensiivv
discrim- generators in Table 3] We observe that GANs using

---

Under review as a conference paper at ICLR 2021
Dataset
Generator
Pooling
FPD-Mix
FPD-Max
FGDJ
MMD-EL
MMD-CL
COV-E1
COV-C1
PC
Max
1,571
0.211
7,030
0.1017
0.00164
23.56
72.75
FC
Mix
0.184
0.209
2.124
0.0674
0.00196
73.64
74.96
4.971
Deform
Max
0.913
0.268
5.602
0.0908
0.00201
68.55
72.61
Chair
Deform
Deform
Attention
0.096
0.755
2.987
0,1141
0.00160
68.77
69.36
TreeGAN
Max
1.442
0.654
7.808
0,0962
0.00191
24,74
73.49
TreeGAN
Mix
0.293
0.134
4.032
0.0704
0.00211
74.82
78.79
Graph- CNN
Max
1,034
0.981
7.494
0,0812
0.00191
48.90
63.18
PointFlow
1.782
1,405
3.256
0,1335
0,00190
71.76
71.73
Oracle
0.088
0.093
0.814
0.0594
0.00165
79.56
79.69
PC
Max
1.553
0.354
6.981
0.0842
0.00153
35.16
64.16
PC
Mix
0.255
0.285
2,550
0.0656
0.00184
73.5
72.16
Deform
Max
1,072
0,633
3.845
0.0799
0:00179
62.5
64.5
Muli
Deform
0.614
2.451
-Cat
Deform
Attention
0.616
0.720
2.531
0.1113
0.00141
72.04
69.60
TreeGAN
Max
1.714
0.437
6.342
0,1093
0.00158
25.33
67.0
TroGAN
Mix
0.388
0.420
4,300
0.0699
0.00184
72,66
71.0
Oraele
0.131
0.177
0.673
0,06012
0.00124
77.14
77.57
Table 3: Evaluating PointNet-based discrininators with different generators. We observe that PointNet-Mix can significantly improve all mampling-aware/sesiiin metric, including FPD-Mix, FGD, MMD-EMD, COV-EMD for all the generators. When paired with FC generator, PointNet-Mix achieves the best performance. We include PointFlow Yang et al. 2019) and Oracle for a comparison.
metrics (iee FPD-Max, MMD-CD, COV-CD), PointNet-Mix always significantly better than PointNet-Attention on FPD-Max and COV-CD while remaining on par with PointNet-Max.
In Figure 2 present the generated point clouds for all the experiments with color-coding for the local point density. We see that the generators trained using PointNet-Max usually suffer from non-uniform density, except for the deformation generator. On the chair class, points are usually clustering around the seat area while being sparse the back. On the contrary, PointNet-Mix enforces a globally uniform point density and hence greatly improves the visual quality of the generated point clouds. PointNet-Attention is in the between.
With Moin we observe that the most naive FC generator works the best outperforming the previously
generators on almost of metrics. This showcases the importance of being sampling-ave aware but not sampling- campling-insensitvveed iissenssiiveoveersensttvve as a discriminator. It also suggests that future works may focus on designing more powerful sampling camling--wwre aware discriminators.
5.3 DIAGNOSING FAILURES FOR VARIOUS DISCRIMINATORS
Observing that the discriminator forms the bottleneck of the point cloud GAN, it is important to examine whether advanced networks, e.g. PointNet++ (Qi al. 2017b), DGCNN (Wang al. 2019), KPConv (Thomas et al. 2019), PointConv (Wu et al. 2019) can outperform PointNet-based discriminator.
Surprisingly, we fail observe any of them ever be successful train a GAN in our extensive experiments, which we perform a systematic search, including varying generator architecture, changing hyperparameters (e.g. the number of layers and parameters in discriminator, learning rate), whether use batch norm. etc. During GAN training, we observe the four advanced discriminator behave very similarly: they are very diseriminative in the sense that the gaps between the real and fake scores remain significantly large, but the quality of the generated point clouds stays very bad. The contradictory behaviors seem unreasonable at the first glance: how can discriminator very discriminative but teach nothing to the generator? Thus, we design a diagnostic "no generator" experiment to examine the underlying reasons.
No Generator Experiments. During GAN training, the gradients from discriminator output scores pack-propagate to the generated point clouds and then further back-propagate to the weights of the generator update the generator. We design diagnostic experiment that removes the generator and examines whether the discriminator gradients are informative enough for supervising point cloud GANs.
Concretely, we take M real chairs real data and randomly initialize M learnable point clouds Nx3 with i.dd Gaussian noises (0, 0.1). For each training iteration, we sample B point clouds from the real dataset the real data and sample B point clouds from the learnable point
7

---

Under review as a conference paper at ICLR 2021
GT
FC-Max
FC-Mix
IC- Attention
Deform- Max
Deform- Mix
Deform- Attention
Tree-Max
Tree-Mis
Muld- Calegory
Figure Visualization of point clouds generated by different methods. We show generated point clouds in few exemplar shapes for fair and easier comparison regarding their sampling quality and geometry quality. We color-code the local point density that ranges from sparse (dark blue) to dense (light yellow).
PN++
DGGNN
PN-Max
PN-Mix
PN-Ave
KPConv
PaieaNes-Max
PrlssNesAis
PaieeNes-Arg
Wassersteln ostimate
4.2
20k
40k
60k
80k
trnining iterations
Figure 3: The diagnostic "no generator" experiment results. On the left, we show exemplar point clouds generated by different discriminators. On the right, we plot the training curves for the experiments. The r- and y-axis respectively represent training iterations and Wasserstein estimates.
Discriminator
FPD-Mix
FPD-Max_
FGDL
MMD-EL
MMD-CL
COV-ET
COV-CT
PointNet-Max Max
3.828
2.456
19.279
0.1039
0.00287
26
56
PointNet-Avg
39.793
0.00642
PointNet-Mix
1.835
2.208
13.197
0.0898
0.00331
50
51
SainNeet-Attetiing
1,350
1.642
12.359
0.1622
0.09271
52
48
PointNet++
15.081
14.187
33.958
0.1155
0.00503
19
37
DGCNN
10.629
9.939
14,882
0.0932
0.00396
46
48
KPConv
41.15
49.96
57.32
0.2695
0.01
2.5
PointConn
27.69
27.05
46.82
0.2832
0.00721
8
Table 4:
Quantitative
evaluation for the diagnostic "no generator" experi-
ments.
PoimtNet-Max//ix//ttennoo are successful while We see that
PointNet-
fail.
clouds as the fake data. Then, we conduct adversarial training where the discriminator gradients directly update the location of each points in the learnable point clouds. We adopt the training protocol and losses as experiments Sec. [.22 The goal of this "no generator" experiment is to see whether the discriminator can modify the point clouds to approximate the real shapes, or in other words, factorize the impact from generator architectures and only focus whether the gradients
8

---

Under review as a conference paper at ICLR 2021
from discriminators guide shape generation. Here M 100, N 2, 048, and B 32. We train each experiment for 10,000 epoches until convergence.
Results and Analysis. summariisss the results and Figure (left) shows exemplar generated point clouds. We clearly see that only Dointeet-axxMiixAAteenoo can successfully modify the learn- able point clouds to get high quality shapes, while
Avg produce much worse results. Figure S (right) presents the training eurves of the Wasserstein estimates, which essentially describe the gap between the real and fake scores. The large score gaps for PointNet++, DGCNN and KPConv throughout the training indicate the two models are very discriminative in telling apart the fake samples from the real data. However, their gradients simply don't help improve the learned point clouds. Note that both of them leverage relative point positions/edge information during their feature extraction, which leads to huge amount of gradients flowing along the surface direction focusing on changing sampling patterns instead of supervising shape geometry. As a suggestion, for developing future discriminators, people may need to avoid using the relative position features. For Avi, the failure is simply because the discriminating power not sufficient, evident from the low Wasserstein estimates. Note that training such "no generator" experiment shares a similar flavor to the SGD sampling in introspective Jin et al. 2017).
CONCLUSION AND SUGGESTIONS FOR FUTURE DISCRIMINATOR DESIGN
In this work. study the importance of sampling for 3D point cloud GAN design. We propose the sampling spectrum of discriminators and evaluation metrics that provide insights on the behaviors of point cloud GANs. We propose several empirical experiments for identifying the sampling sensitivity of discriminator employed, a sampling-insensitiee discriminator, c.g. the commonly used PointNet-Mart will produce point cloud shapes with non-uniform density and clustering artifacts, while sampling-oversensiing discriminator (e-g. PointNet++, DGCNN, PointConv, KPConv) will lead disastrous failures when adapting them to training point cloud GANs. Interestingly, a simple PointNet-Mix baseline coupled with the most naive fully-connected generator achieves the best performance, indicating that the current bottleneck of point cloud GAN on the discriminator side. For future discriminators, we suggest they should be more discriminative in shape and aware but not oversensitive to sampling patterns. For the sanity check of any novel discriminators, one leverage our proposed "no generator" experiment.
evaluation metric. Experiments indicate that, no matter what generator is

---

Under review as a conference paper at ICLR 2021
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds, 2018. htt.:/tttss:/ooeeeeeee wttp://wweweee.eeeeeeeeeeeee forum??
id Bd-BJInE
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03011,, 2015.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605-613, 2017.
Thibault Groueix, Matthew Fisher, Vladimir G Kim. Bryan C Russell, and Mathieu Aubry. A papier-mâché approach learning 3d surface generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 216-224, 2018.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in neural information processing systems, pp. 5767-5777, 2017.
Pedro Hermosilla, Tobias Ritschel, Pere-Pau Vázquez. Alvar Vinacua, and Timo Ropinski. Monte carlo convolution for learning on non-unifomlly sampled point clouds. In SIGGRAPH Asia 2018 Technical Papers. pp. 235, ACM, 2018.
Martin Heusel, Hubert Ramsauer. Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626 063666637,20 2017.
Long Jin, Justin Lazarow, and Zhuowen Tu. Introspective classification with convolutional nets. In Advances in Neural Information Processing Systems, pp. 823-833, 2017.
Ruihui Li, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. Pu-gan: point cloud upsampling adversarial network. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7203-7212, 2019.
Yangyan Li, Rui Bu, Mingehao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on x- x-transformed points. In Advances in neural information processing systems, pp. 820-830, 2018.
Kaichun Mo, He Wang, Xinchen Yan, and Leonidas Guibas. PT2PC: Learning to generate 3d point cloud shapes from part tree conditions. arXiv preprint arXiv::0033 086224, 2020.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 174. 2019.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 652-660, 2017a
Charles Ruizhongtai Qi, Li Yi, Hao Su. and Leonidas Guibas. Pointnet++: Deep hicrarchical feature learning on point sets in metric space. In Advances in neural information processing systems, pp. 5099-5108, 2017b.
Dong Wook Shu, Sung Woo Park, and Junseok Kwon. 3d point cloud generative adversarial network based on tree structured graph convolutions. In Proceedings of the IEEE International Conference Computer Vision, pp. 1859-3868. 2019.
Hugues Thomas, Charles R Qi. and Leonidas Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6411-6420, 2019.
Deschaud, Beatriz Marcotegui, François Goulette,
Diego Valsssia, Giulia Fracastoro, and Enrico Magli. Learning localized generative models for 3d point clouds via graph convolution. 2018.
10

---

Under review as a conference paper at ICLR 2021
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions on Graphics (TOG), 38(5): 146, 2019.
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9621 9630, 2019.
Zhirong Wu, Shuran Song. Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4541-4550, 2019.
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian, Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 206-215, 2018.


Reviews:
[{"review_id": "TG2dxMPnPMb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:\nThis studies how different discriminator architectures are sensitive to sampling strategy of point clouds.\nThe paper proposes three benchmarks that measures how sensitive the architecture to the sampling strategy of point clouds, which provides insights to future research about how to choose architectures for both discriminator and for feature extraction.\nEmpirically, the authors find out that using a discriminator architecture that’s just reasonably sensitive to sampling could create better generative results, regardless of the generator.\nFinally, the paper also provides a novel experiment setting (i.e. “no-generator” experiment) that could compare gradient quality of discriminators.\n\n\nStrength:\nThis work is the first one (that I’m aware of) that studies how the point sampling scheme affects the results and evaluation of the generation. The paper also tried to verify such results with a wide ranges of architectures and the results seemed rather self-consistent.\n\nWeakness:\n1. Discreteness of the “sampling spectrum”. While the idea to study how sensitive the network is to the sampling schema is certainly good, but the way to characterize the spectrum in this paper is too empirical and discrete. This might not be too useful when the sampling method of interested in certain application doesn’t fall into the category the author mention. It would be nice if the author could provide a way to quantify the difference between different sampling methods (e.g. expectation of EMD on the same surface area), and verify how different architecture falls into such continuous spectrum.\n2. Disentangling the effects of geometries v.s. the sampling. I agree with the authors that the point clouds’ quality can be thoughts as outputs of the geometries and the sampling, and those two factors goes hand-in-hand. But this paper seemed to emphasize solely on the architecture’s capacity to detach different sampling techniques, and tries to correlate this factor with the generation quality. My concern here is that could the network’s capacity toward geometries a confounding factors in the experiment results? For example, could it be the case that the network that’s over-sensitive to the sampling patterns actually fails to tell apart geometric difference, which leads to bad generative results. In that case, could it be the case that the network’s ability to discriminate geometries differences the main factor, regardless of whether the network is over-sensitive or under-sensitive to the sampling? Following this line of thoughts, if the authors’ claim holds, then there might be some fundamental difficulty for the network to be both capable of telling apart geometries and being over/under-sensitive to the sampling pattern, which doesn’t seem to be established in the paper concretely (with theory or empirical results). Maybe the authors could point us to evidence that show that the networks used in the paper all have about the same capacity in telling apart geometries, which is an alternative to address this concern.\n\n\nJustifications:\nThe paper is the first in my knowledge that studies how sampling strategy affects point cloud generation and evaluation. In this case, the paper has its value and novelty in the literature. But the results and insights are organized under the framework of “sampling spectrum”, which might still need improvement with more theories or results. WIth that, I will vote for borderline + for now and will be happy to hear from the authors.\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OaWXw9-p1P6", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper experimentally examine a number of generator and discriminator network choices for point cloud GAN. It is shown that the best generator choice would be a PointNet with a mixture between max-pooling and average pooling, and that an attention-based PointNet framework performs the best in terms of discriminators. Multiple metrics were evaluated and a comprehensive experiment was done on the metrics capabiliities w.r.t. the sampling procedure.\n\nPositive:\n\n-- Comprehensive experiments, multiple metrics, network structures tested on both the generator and the discriminator side.\n\nNegative:\n\n-- I don't get why the authors would always stick to existing generators. It has been shown that convolutional approaches such as KPConv and PointConv perform much better than PointNet in discriminative settings. Hence, it also makes sense that they could perform well in a generative setting. The tested graphCNN generator is not necessarily as powerful as KPConv/PointConv. From an intuitive point of view, convolutional architectures should be helpful in the generative models as well. Besides, using a convolutional discriminator may not be able to generate gradients good enough for a non-convolutional generator. But if the generator and discriminator match in terms of architecture presumably the performance could be better.\n\n-- The other concern I had is w.r.t. detailed settings of the \"no generator\" experiment in Table 4. Is WGAN-GP-type cost-term and penalty used in this experiment? It's well-known that discriminators that are very confident would not give proper gradient to generative models (e.g. from the WGAN papers, I also remember reading somewhere this paper \"A good GAN requires a bad discriminator\" although somehow I couldn't find this reference anymore), unless they are heavily constrained e.g. by WGAN-GP kind terms. Hence this detail can be important in deciding whether the conclusions from the paper would be credible.\n\nThe main reason I want to nitpick on these seemingly small items is that the conclusion of the paper might change significantly from those details. The paper tends to get to a strong conclusion that one needs a bad discriminator for a good GAN, although that's generally a reasonable assumption, its conclusions on inability to generate anything with a KPConv/PointConv discriminator could have far-fetching implications and hence I just want to apply a bit of additional scrutiny here.\n\nMinor: For completeness, it's worth writing down what is the input to the point cloud GAN. This is very unclear in the current paper. I assume that the input is a random point cloud as in (Achlioptas et al. 2018), but it's worth stating that clearly early on, e.g. Sec. 2.1.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "O7yJ0341KsE", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper conducts experiments to examine the effects of point sampling patterns in point cloud GANs. And, experimental results reveal the reason why current strong discriminators(PointNet++, etc) fail to train a reasonable point cloud generator. By proposing  a sampling spectrum, the authors introduce a middle point sampling-aware baseline discriminator--PointNet-Mix. The paper proposes a new improvement direction for Point Cloud GAN, which might have a strong impact in the community. The paper is written concisely and the illustrations are clear.\n\nHowever, there are still some concerns that need to be addressed:\n\n1. One of the major contributions claimed in the paper is the term “Sampling Spectrum”. However, the paper does not provide the exact definition of “Sampling Spectrum”. The paper mainly divides thes spectrum into three regimes: sampling-insensitive/-aware/oversensitive, which is oversimplified, and too broad. The authors should give more justification and background intuition on why the “Sampling Spectrum” should be in this form.\n\n2. Another concern to this paper is the “no generator” experiments. In Figure 3, the authors show some visualizations of the point clouds generated by different discriminators. It is reasonable to generate some weird Chair shape from PointNet++, and DGCNN. However, the generated point clouds from KPConv and PointConv are quite surprising. To make the experiment results convincing, it is better that the authors should provide more details on the experiments of KPConv and PointConv, and provide more explanation on why KPConv and PointConv give almost random point clouds. Because the PointNet++, DGCNN, KPConv, and PointConv all use relative point positions/edge information according to the paper. There must be more reasons on why the results look very different.\n\n3. More technique details on the “no generator”  experiments should be introduced. For example, what is the loss function used(both the “no generator” experiments and the rest GAN experiments)? Is it the improved WGAN?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5XYuEn24yor", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the task of 3d shape generation. More precisely, it embraces the point cloud GAN strategy for generation and offers an extensive comparative analysis of the existing architectures of point cloud generators and discriminators with a focus on the variations of the latter component.\n\nA series of synthetic experiments considering several point cloud sampling schemes is proposed and used to define a spectrum of sensitivity to sampling artifacts (various deviations from uniform sampling) for a range of considered discriminators. The same is performed with the existing metrics evaluating performance for point cloud generation.\n\nPresented results suggest that existing PointNet based discriminators with max-pooling point feature aggregation mechanism are insensitive to severe artifacts in point density, which leads to non-uniform samples from the generators. On the other hand, existing improvements to PointNet, which perform better in discriminative tasks are over-sensitive to sampling and manage to capture even the smallest deviations from the uniform sampling which leads to poor training signals (in terms of gradients) from those discriminators. As alternatives for both extremes the authors propose to use «sampling-aware» PointNet based discriminators, but either use a concatenation of the max-pooled and average-pooled point features, or self-attention mechanism as an aggregation function.\n\nIn the experimental section the authors compare numerous combinations of considered discriminators with several existing generators to confirm that the proposed «sampling-aware» discriminators yield superior performance across all the generators.\n\nPros:\n1) The paper is well-written and is easy to follow.\n2) Overall rigorousness of the experiments in the paper is impressive, and the conclusions drawn are logical and insightful.\n3) Additional inputs on the sampling sensitivity spectrum for evaluation metrics are also valuable.\n\nCons:\n1) The main drawback of the paper is the self-contained nature of the presented material. In its current form, it misses any validation by comparison to any external generative results. Thus, it is not clear, where the obtained improvements for point cloud GANs put them on the performance quality list, considering all the generative models for point clouds.\n\nOverall, this is an impressive work, which I think may be accepted even as it is. However, its value can be extended outside point cloud GAN community to general 3d shape generation task audience with a proper comparison of the best results with current state of the art. There are at least two recent works based on likelihood training [1, 2], which can be considered for comparison, or at least should be mentioned. Given that my score is only moderate.\n\n#################################################\n\nAdditional comments:\n\nSection 1, paragraph 2:\nImproper citation caused artifacts in the text.\n\nSection 5.1:\nBoth CD and EMD (also possibly FPD) are sensitive to the scale of the input point clouds, thus it is important to indicate the scale of the input point clouds (how they were normalized). For example for now, I am not sure that it is possible to directly compare your results from Table 3 with the results of [1, 2], since your values of MMD-CD/EMD and COV-CD/EMD significantly differ from the values reported there. This might be due to the different input data scales.\n\nSection 5.2:\nIn my experience, l-GANs from [3] gave very unstable performance with respect to even small variations in the model architectures (e.g. a change in the dimensionality of the latent space). To obtain reasonable performance I needed to tune hyperparameters related to training objective and optimization separately for different configurations and could not use a single set of hyperparameters for all the configurations.\nIn your case, you alter different network parts completely. Could you verify somehow that the relative performance of the considered configurations can not be attributed to the choice of the hyperparameters, which are best for one configuration but not optimal for others? In other worlds, have you tried to tune the hyperparameters separately for different configurations?\n\nI appreciated the PointNet-Max-2048 experiment in the supplementary materials, since I had the same question and found the answer there.\n\nSection 5.3, No generator experiments:\n«…to supervised the generator training.» -> to supervise\n\nFigure 2 is reported to contain «random selected» (-> randomly selected) samples. I do not understand how they can be randomly selected and at the same time all present approximately the same object in each row. I do not think you can claim random selection rather than cherry picking in that case, which is not a problem, since you wanted to demonstrate the differences in generations of approximately the same object.\n\nTable 4, caption:\n«Min» -> Mix\n\n[1] Yang, G., Huang, X., Hao, Z., Liu, M.Y., Belongie, S., Hariharan, B.: PointFlow: 3D point cloud generation with continuous normalizing flows. In ICCV’19.\n\n[2] Klokov, R., Boyer, E., Verbeek, J.: Discrete Point Flow Networks for Efficient Point Cloud Generation. In ECCV’20.\n\n[3] Achlioptas, P., Diamanti, O., Mitliagkas, I., Guibas, L.: Learning representations and generative models for 3D point clouds. In ICML’18.\n\n___\n\nAfter reading other reviews, authors' comments, and checking the revised manuscript I decided to slightly improve my rating for two reasons. Firstly, my concerns were answered during discussions, secondly, I do not agree that the concerns raised by other reviewers could justify a rejection. I believe this is an exemplary empirical study presenting novel information and insights about sampling sensitivity of point cloud encoders and point cloud evaluation metrics.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2eQcoIRGaXy", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies an interesting problem, i.e., point sampling, in 3D point cloud generation through GAN. It shows that the currently used discriminator in this framework is sampling insensitive, thus the learned generator is prone to sampling artifacts. On the contrary, the sampling sensitive point-cloud CNNs are not suitable for acting as a discriminator according to the authors’ experiments that all of them fail to generate reasonable point clouds. Observing this, the authors proposed sampling-aware discriminator and achieved better results. Especially, along with the proposed sampling-related metrics, the proposed method shows advantages over existing methods.\n\nStrength:\n1. The finding that sampling plays an important role in GAN based point cloud generation is helpful to the community.\n2. The proposed method by simply concatenating an average pooled feature to the max-pooled PointNet feature is shown to be effective in the proposed metrics.\n\n\nWeakness:\nOverall, there are many unclear issues need further explanation. The rationality of the proposed method is also unconvincing.\n1. Intuitively, as the sampling insensitive discriminator leads to generated point clouds unaware of the sampling artifacts, the advanced sampling sensitive discriminator could address this problem since it focuses more on this aspect. However, the results did not support this guess and the paper does not give strong reason/analysis to explain such failures.\n2. The proposed sampling aware discriminator is a concatenation of max pooling and avg pooling. Considering PointNet-MIX and PointNet-MAX only differs by an avg, does it mean that avg is sampling-aware? then, why not directly using avg? Unfortunately, the results shown in the paper (e.g., table 4) demonstrate that using avg also does not perform well. Therefore, it is quite hard to understand the underlying rationality for the success of PointNet-MIX.\n3. To show the proposed mix strategy is general enough, it is required to show with other baselines. For example, how about using the mix strategy in aggregating weighted feature in PN-attention? Does it still work?\n4. PN-attention improves PN by incorporating communication between points, however, why other networks such as PN++, PointConv that also uses points interactions can not improve on PN in GAN? This is worthy to analysis and give a reasonable explanation in this paper.\n5. About the proposed sampling-related metrics, the core is F distance based on various feature extractors. Since F distance is based on features, while the used features are depended on extractors, so this observation in P5 “the Frechét distance metrics share the same sampling sensitivity of their corresponding discriminators.” is obvious. In other words, it actually heavily relies on the used extractor, i.e., with a sampling insensitive extractor, the metric is tend to be sampling insensitive. Therefore, such metrics may be meaningless to reflect the property of different methods/generators.\n\nOther comments:\nIt is hard to understand what is “learnable point clouds” in No Generator Experiments. More details about the setup and network architectures about this experiment is helpful for reader to understand.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides empirical evidence that the sampling strategy used in point cloud GANs can drastically impact the generation quality of the network. Specifically, the authors show that discriminators that are not sensitive to sampling have clustering artifact errors, while those that are sensitive to sampling do not produce reasonable looking point clouds. They also provide a simple way (i.e. including AVG feature pooling) to improve generation quality for insensitive discriminator GAN setups. The reviewers agree that this is an interesting insight into the problem and this insight can help the community.

Based on the reviewers' comments and subsequent discussions, it becomes clear that the paper would be stronger and more compelling if the underlying hypothesis (i.e. the idea of sampling spectrum) is more rigorously defined (e.g. ideally with a theoretical grounding) and the claims/analyses are tied in with this definition. Such a grounded and precise setup would help in analyzing future generation discriminators that may not simply fall into the two discrete groups defined in the paper (i.e. sampling over-sensitive and sampling-insensitive). The results have promise, so the authors are encouraged to take into consideration the reviewer discussions to produce a stronger future submission. 
Decision: Reject