Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper experimentally examine a number of generator and discriminator network choices for point cloud GAN. It is shown that the best generator choice would be a PointNet with a mixture between max-pooling and average pooling, and that an attention-based PointNet framework performs the best in terms of discriminators. Multiple metrics were evaluated and a comprehensive experiment was done on the metrics capabiliities w.r.t. the sampling procedure.

Positive:

-- Comprehensive experiments, multiple metrics, network structures tested on both the generator and the discriminator side.

Negative:

-- I don't get why the authors would always stick to existing generators. It has been shown that convolutional approaches such as KPConv and PointConv perform much better than PointNet in discriminative settings. Hence, it also makes sense that they could perform well in a generative setting. The tested graphCNN generator is not necessarily as powerful as KPConv/PointConv. From an intuitive point of view, convolutional architectures should be helpful in the generative models as well. Besides, using a convolutional discriminator may not be able to generate gradients good enough for a non-convolutional generator. But if the generator and discriminator match in terms of architecture presumably the performance could be better.

-- The other concern I had is w.r.t. detailed settings of the "no generator" experiment in Table 4. Is WGAN-GP-type cost-term and penalty used in this experiment? It's well-known that discriminators that are very confident would not give proper gradient to generative models (e.g. from the WGAN papers, I also remember reading somewhere this paper "A good GAN requires a bad discriminator" although somehow I couldn't find this reference anymore), unless they are heavily constrained e.g. by WGAN-GP kind terms. Hence this detail can be important in deciding whether the conclusions from the paper would be credible.

The main reason I want to nitpick on these seemingly small items is that the conclusion of the paper might change significantly from those details. The paper tends to get to a strong conclusion that one needs a bad discriminator for a good GAN, although that's generally a reasonable assumption, its conclusions on inability to generate anything with a KPConv/PointConv discriminator could have far-fetching implications and hence I just want to apply a bit of additional scrutiny here.

Minor: For completeness, it's worth writing down what is the input to the point cloud GAN. This is very unclear in the current paper. I assume that the input is a random point cloud as in (Achlioptas et al. 2018), but it's worth stating that clearly early on, e.g. Sec. 2.1.


review_title: Why not convolutional generator?

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary of the paper:
This studies how different discriminator architectures are sensitive to sampling strategy of point clouds.
The paper proposes three benchmarks that measures how sensitive the architecture to the sampling strategy of point clouds, which provides insights to future research about how to choose architectures for both discriminator and for feature extraction.
Empirically, the authors find out that using a discriminator architecture that’s just reasonably sensitive to sampling could create better generative results, regardless of the generator.
Finally, the paper also provides a novel experiment setting (i.e. “no-generator” experiment) that could compare gradient quality of discriminators.


Strength:
This work is the first one (that I’m aware of) that studies how the point sampling scheme affects the results and evaluation of the generation. The paper also tried to verify such results with a wide ranges of architectures and the results seemed rather self-consistent.

Weakness:
1. Discreteness of the “sampling spectrum”. While the idea to study how sensitive the network is to the sampling schema is certainly good, but the way to characterize the spectrum in this paper is too empirical and discrete. This might not be too useful when the sampling method of interested in certain application doesn’t fall into the category the author mention. It would be nice if the author could provide a way to quantify the difference between different sampling methods (e.g. expectation of EMD on the same surface area), and verify how different architecture falls into such continuous spectrum.
2. Disentangling the effects of geometries v.s. the sampling. I agree with the authors that the point clouds’ quality can be thoughts as outputs of the geometries and the sampling, and those two factors goes hand-in-hand. But this paper seemed to emphasize solely on the architecture’s capacity to detach different sampling techniques, and tries to correlate this factor with the generation quality. My concern here is that could the network’s capacity toward geometries a confounding factors in the experiment results? For example, could it be the case that the network that’s over-sensitive to the sampling patterns actually fails to tell apart geometric difference, which leads to bad generative results. In that case, could it be the case that the network’s ability to discriminate geometries differences the main factor, regardless of whether the network is over-sensitive or under-sensitive to the sampling? Following this line of thoughts, if the authors’ claim holds, then there might be some fundamental difficulty for the network to be both capable of telling apart geometries and being over/under-sensitive to the sampling pattern, which doesn’t seem to be established in the paper concretely (with theory or empirical results). Maybe the authors could point us to evidence that show that the networks used in the paper all have about the same capacity in telling apart geometries, which is an alternative to address this concern.


Justifications:
The paper is the first in my knowledge that studies how sampling strategy affects point cloud generation and evaluation. In this case, the paper has its value and novelty in the literature. But the results and insights are organized under the framework of “sampling spectrum”, which might still need improvement with more theories or results. WIth that, I will vote for borderline + for now and will be happy to hear from the authors.





review_title: novel studies on how point clouds’ sampling strategy affects generation quality and evaluation, needs work on the “sampling spectrum” framework to organize the results

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper studies an interesting problem, i.e., point sampling, in 3D point cloud generation through GAN. It shows that the currently used discriminator in this framework is sampling insensitive, thus the learned generator is prone to sampling artifacts. On the contrary, the sampling sensitive point-cloud CNNs are not suitable for acting as a discriminator according to the authors’ experiments that all of them fail to generate reasonable point clouds. Observing this, the authors proposed sampling-aware discriminator and achieved better results. Especially, along with the proposed sampling-related metrics, the proposed method shows advantages over existing methods.

Strength:
1. The finding that sampling plays an important role in GAN based point cloud generation is helpful to the community.
2. The proposed method by simply concatenating an average pooled feature to the max-pooled PointNet feature is shown to be effective in the proposed metrics.


Weakness:
Overall, there are many unclear issues need further explanation. The rationality of the proposed method is also unconvincing.
1. Intuitively, as the sampling insensitive discriminator leads to generated point clouds unaware of the sampling artifacts, the advanced sampling sensitive discriminator could address this problem since it focuses more on this aspect. However, the results did not support this guess and the paper does not give strong reason/analysis to explain such failures.
2. The proposed sampling aware discriminator is a concatenation of max pooling and avg pooling. Considering PointNet-MIX and PointNet-MAX only differs by an avg, does it mean that avg is sampling-aware? then, why not directly using avg? Unfortunately, the results shown in the paper (e.g., table 4) demonstrate that using avg also does not perform well. Therefore, it is quite hard to understand the underlying rationality for the success of PointNet-MIX.
3. To show the proposed mix strategy is general enough, it is required to show with other baselines. For example, how about using the mix strategy in aggregating weighted feature in PN-attention? Does it still work?
4. PN-attention improves PN by incorporating communication between points, however, why other networks such as PN++, PointConv that also uses points interactions can not improve on PN in GAN? This is worthy to analysis and give a reasonable explanation in this paper.
5. About the proposed sampling-related metrics, the core is F distance based on various feature extractors. Since F distance is based on features, while the used features are depended on extractors, so this observation in P5 “the Frechét distance metrics share the same sampling sensitivity of their corresponding discriminators.” is obvious. In other words, it actually heavily relies on the used extractor, i.e., with a sampling insensitive extractor, the metric is tend to be sampling insensitive. Therefore, such metrics may be meaningless to reflect the property of different methods/generators.

Other comments:
It is hard to understand what is “learnable point clouds” in No Generator Experiments. More details about the setup and network architectures about this experiment is helpful for reader to understand.
review_title: review for ICLR2021 submission "rethinking sampling in 3D point cloud generative adversarial networks"

final_score: rating: 7: Good paper, accept
review_content: This paper considers the task of 3d shape generation. More precisely, it embraces the point cloud GAN strategy for generation and offers an extensive comparative analysis of the existing architectures of point cloud generators and discriminators with a focus on the variations of the latter component.

A series of synthetic experiments considering several point cloud sampling schemes is proposed and used to define a spectrum of sensitivity to sampling artifacts (various deviations from uniform sampling) for a range of considered discriminators. The same is performed with the existing metrics evaluating performance for point cloud generation.

Presented results suggest that existing PointNet based discriminators with max-pooling point feature aggregation mechanism are insensitive to severe artifacts in point density, which leads to non-uniform samples from the generators. On the other hand, existing improvements to PointNet, which perform better in discriminative tasks are over-sensitive to sampling and manage to capture even the smallest deviations from the uniform sampling which leads to poor training signals (in terms of gradients) from those discriminators. As alternatives for both extremes the authors propose to use «sampling-aware» PointNet based discriminators, but either use a concatenation of the max-pooled and average-pooled point features, or self-attention mechanism as an aggregation function.

In the experimental section the authors compare numerous combinations of considered discriminators with several existing generators to confirm that the proposed «sampling-aware» discriminators yield superior performance across all the generators.

Pros:
1) The paper is well-written and is easy to follow.
2) Overall rigorousness of the experiments in the paper is impressive, and the conclusions drawn are logical and insightful.
3) Additional inputs on the sampling sensitivity spectrum for evaluation metrics are also valuable.

Cons:
1) The main drawback of the paper is the self-contained nature of the presented material. In its current form, it misses any validation by comparison to any external generative results. Thus, it is not clear, where the obtained improvements for point cloud GANs put them on the performance quality list, considering all the generative models for point clouds.

Overall, this is an impressive work, which I think may be accepted even as it is. However, its value can be extended outside point cloud GAN community to general 3d shape generation task audience with a proper comparison of the best results with current state of the art. There are at least two recent works based on likelihood training [1, 2], which can be considered for comparison, or at least should be mentioned. Given that my score is only moderate.

#################################################

Additional comments:

Section 1, paragraph 2:
Improper citation caused artifacts in the text.

Section 5.1:
Both CD and EMD (also possibly FPD) are sensitive to the scale of the input point clouds, thus it is important to indicate the scale of the input point clouds (how they were normalized). For example for now, I am not sure that it is possible to directly compare your results from Table 3 with the results of [1, 2], since your values of MMD-CD/EMD and COV-CD/EMD significantly differ from the values reported there. This might be due to the different input data scales.

Section 5.2:
In my experience, l-GANs from [3] gave very unstable performance with respect to even small variations in the model architectures (e.g. a change in the dimensionality of the latent space). To obtain reasonable performance I needed to tune hyperparameters related to training objective and optimization separately for different configurations and could not use a single set of hyperparameters for all the configurations.
In your case, you alter different network parts completely. Could you verify somehow that the relative performance of the considered configurations can not be attributed to the choice of the hyperparameters, which are best for one configuration but not optimal for others? In other worlds, have you tried to tune the hyperparameters separately for different configurations?

I appreciated the PointNet-Max-2048 experiment in the supplementary materials, since I had the same question and found the answer there.

Section 5.3, No generator experiments:
«…to supervised the generator training.» -> to supervise

Figure 2 is reported to contain «random selected» (-> randomly selected) samples. I do not understand how they can be randomly selected and at the same time all present approximately the same object in each row. I do not think you can claim random selection rather than cherry picking in that case, which is not a problem, since you wanted to demonstrate the differences in generations of approximately the same object.

Table 4, caption:
«Min» -> Mix

[1] Yang, G., Huang, X., Hao, Z., Liu, M.Y., Belongie, S., Hariharan, B.: PointFlow: 3D point cloud generation with continuous normalizing flows. In ICCV’19.

[2] Klokov, R., Boyer, E., Verbeek, J.: Discrete Point Flow Networks for Efficient Point Cloud Generation. In ECCV’20.

[3] Achlioptas, P., Diamanti, O., Mitliagkas, I., Guibas, L.: Learning representations and generative models for 3D point clouds. In ICML’18.

___

After reading other reviews, authors' comments, and checking the revised manuscript I decided to slightly improve my rating for two reasons. Firstly, my concerns were answered during discussions, secondly, I do not agree that the concerns raised by other reviewers could justify a rejection. I believe this is an exemplary empirical study presenting novel information and insights about sampling sensitivity of point cloud encoders and point cloud evaluation metrics.
review_title: Insightful take on the generative capabilities of various point cloud GANs.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper conducts experiments to examine the effects of point sampling patterns in point cloud GANs. And, experimental results reveal the reason why current strong discriminators(PointNet++, etc) fail to train a reasonable point cloud generator. By proposing  a sampling spectrum, the authors introduce a middle point sampling-aware baseline discriminator--PointNet-Mix. The paper proposes a new improvement direction for Point Cloud GAN, which might have a strong impact in the community. The paper is written concisely and the illustrations are clear.

However, there are still some concerns that need to be addressed:

1. One of the major contributions claimed in the paper is the term “Sampling Spectrum”. However, the paper does not provide the exact definition of “Sampling Spectrum”. The paper mainly divides thes spectrum into three regimes: sampling-insensitive/-aware/oversensitive, which is oversimplified, and too broad. The authors should give more justification and background intuition on why the “Sampling Spectrum” should be in this form.

2. Another concern to this paper is the “no generator” experiments. In Figure 3, the authors show some visualizations of the point clouds generated by different discriminators. It is reasonable to generate some weird Chair shape from PointNet++, and DGCNN. However, the generated point clouds from KPConv and PointConv are quite surprising. To make the experiment results convincing, it is better that the authors should provide more details on the experiments of KPConv and PointConv, and provide more explanation on why KPConv and PointConv give almost random point clouds. Because the PointNet++, DGCNN, KPConv, and PointConv all use relative point positions/edge information according to the paper. There must be more reasons on why the results look very different.

3. More technique details on the “no generator”  experiments should be introduced. For example, what is the loss function used(both the “no generator” experiments and the rest GAN experiments)? Is it the improved WGAN?


review_title: An interesting view in Point Cloud GAN. However, the work lacks theoretical background support and some technique details.

Meta-review: The paper provides empirical evidence that the sampling strategy used in point cloud GANs can drastically impact the generation quality of the network. Specifically, the authors show that discriminators that are not sensitive to sampling have clustering artifact errors, while those that are sensitive to sampling do not produce reasonable looking point clouds. They also provide a simple way (i.e. including AVG feature pooling) to improve generation quality for insensitive discriminator GAN setups. The reviewers agree that this is an interesting insight into the problem and this insight can help the community.

Based on the reviewers' comments and subsequent discussions, it becomes clear that the paper would be stronger and more compelling if the underlying hypothesis (i.e. the idea of sampling spectrum) is more rigorously defined (e.g. ideally with a theoretical grounding) and the claims/analyses are tied in with this definition. Such a grounded and precise setup would help in analyzing future generation discriminators that may not simply fall into the two discrete groups defined in the paper (i.e. sampling over-sensitive and sampling-insensitive). The results have promise, so the authors are encouraged to take into consideration the reviewer discussions to produce a stronger future submission. 
Decision: Reject