Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new method for making 3D point clouds by automatically completing 3D scans. It does not require paired data samples for training which makes it possible to train it on real data instead of synthetic data. The authors use a generative adversarial network (GAN) to “generate” complete point clouds from noisy or partial point clouds obtained by 3D scanning. The generator learns to perform mapping from point set manifold of scanned noisy and partial input X_r to manifold of clean shapes X_c. The discriminator tries to tell between encoded clean shapes (synthetic data point clouds) and mappings of noisy input (point clouds from real-life data 3D scans). 

An encoder-decoder network similar to those in Achlioptas et al. (2018) and Qi et al. (2017a) is trained to transform original point clouds to a low-dimensional latent space prior to training the GAN. The authors find that using the encoder-decoder trained on clean shape data even for noisy input yields better results. 
One of the issues of completing noisy and partial scans is that the desired complete scan can have a very different shape compared to the noisy input. The generator can map latent vectors to any points on the target manifold which allows it to generate shapes that are far different from the original inputs. In order not to generate random clean shapes, the authors add a reconstruction loss to the generator which encourages it to preserve the partial shape of the input end reconstruct it in the completed clean shape. The choice of Hausdorff distance for reconstruction loss is sound and the ablation study confirms it. 

The authors perform rather extensive experimental evaluation of their proposed method. They perform qualitative and quantitative analysis on several datasets, both real-life and synthetic ones. The proposed method outperforms existing methods in real-life data scenario. On the synthetic dataset (3D-EPN), the quantitative results are not as good as those of PCN (which is a supervised method, unlike the proposed method), but the qualitative results look plausible and comparable to PCN results. In terms of plausibility score, the proposed method outperforms existing methods in all experiments, which is probably thanks to its objective - map the input into the latent space of clean and complete shapes. However, plausible looking point clouds do not necessarily have to precisely match the input, which is the objective of 3D scan point cloud completion. 
The ablation study also confirms the effectiveness of individual parts in the proposed method. 
The main contribution of this paper is training with unpaired data, which enables training on real-life data, leading to better results on real-life scans. While I understand the difficulties, I believe it would be better to try to focus more on real-life data in the evaluation. There is only one experiment with quantitative analysis on real-life data in the paper. Supporting that with a visual Turing test would have been great. 
 
Questions raised:

In 4.3, you say that “ground truth complete scans are not available for training.” How do you then train the supervised methods? Are they trained on 3D-EPN? In that case, is your proposed method also trained on 3D-EPN? If your method is the only one trained on your synthetic data (dataset D), which is also used for testing, then I do not think you could claim that your method is better at dealing with different data distributions. Please make clear in the comments what data you use for training in this experiment. 

The meaning of section 4.4 is not very clear to me. If my understanding is correct, when completing noisy or partial point clouds, low diversity in results is better than high diversity because it is a task of repairing the input, which only has one correct result. Are you trying to say in this section that your method yields more consistent results with lower diversity than the method from previous work? If that is the case, you should consider rewriting that part to make it clearer to readers. 

The PCN paper (Yuan et al., 2018) shows that PCN can generalize and complete point clouds of objects unseen during training very well. Unfortunately, this paper does not discuss performance on objects of unseen classes. Were such experiments considered? It would be beneficial to do a qualitative analysis of performance on unseen objects. 

The description of the left table in Table 1 first says that it shows performance on real-life scans, but performance on synthetic data also appears there. Shouldn’t it rather say that it is plausibility comparison on real-life scans and synthetic data?

Summary:

The proposed method is easy to understand, exploits recent progress in generative models, and allows training on real-life scans as it does not require paired training data. 
The paper does not contain much algorithmical novelty and mostly combines existing methods to solve the problem of obtaining clean and complete point clouds from real 3D scans. However, the ablation study shows that adding a good reconstruction loss to the generator is crucial for the performance. 
The main strengths of this paper are extensive experimental evaluation using both quantitative and qualitative analysis, and significant improvement in performance on real-life data over previous works. 
The main weakness is limited novelty in terms of the techniques used in the proposed method. 

Additional comments that do not affect the review decision:

Citations in the text have to be revised and correctly put into parentheses where necessary. E.g., on page 2: “Since its introduction, GAN Goodfellow et al. (2014) has been used...” should be rewritten to “Since its introduction, GAN (Goodfellow et al., 2014) has been used...”

Table 2 is the only table where the best results are not highlighted by bold text. It is also the only table where the proposed method is not the best performing method as it loses to PCN. I apologize if it is just a pure coincidence but it seems as if the authors did not want to draw attention to the fact that their proposed method loses to an existing method in that experiment. I believe that you should be fair and highlight best results in all tables. 

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The paper addresses the task of point cloud completion within an unpaired setting, where explicit correspondences between the partial and the complete shapes is not given. The setting represents significant interest in practice, e.g. in autonomous driving applications, where the precise completions of scanned objects, e.g. surrounding cars, are not necessary. 

The authors propose to use three models, wherein two models are point autoencoders in the style of [2], obtaining the two spaces of latent codes for the partial and the complete shapes, respectively. The third model learns a mapping between the two latent spaces in an adversarial way. While the idea of doing the unpaired shape completion has been known since the introduction unpaired image-based methods (e.g., [1]), the application is novel and the methods are formulated using the language known in the point cloud learning literature (e.g., EMD losses and point-based autoencoders).

Regarding the evaluation procedure, the authors demonstrated convincingly that the proposed approach is feasible. However, I believe for shape completions with ground-truth labels more quality measures may be used, such as the Chamfer distance, Hausdorff distance, or Earth Mover’s distance (which is actually optimized by the authors), as shown in literature on point cloud upsampling [3], which is a related task. 

Overall, I believe that the paper does a good job of combining the established components into something new and useful, particularly, the problem is well-defined, the method is intuitive and extends the state-of-the-art, and the evaluation looks convincing. The paper is well-written and easy to follow, too. 


[1] Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision (pp. 2223-2232).
[2] Achlioptas, P., Diamanti, O., Mitliagkas, I., & Guibas, L. (2018, July). Learning Representations and Generative Models for 3D Point Clouds. In International Conference on Machine Learning (pp. 40-49).
[3] Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. Pu-net: Point cloud upsampling network. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2790–2799, 2018b.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This manuscript focuses on reconstructing 3D shapes from point clouds, with applications for instance to 3D scanners. The contribution builds on an adversarial formulation of the reconstruction, as in GANs. The method uses an encoder to map the observed noisy set of points into a lower-dimensional latent space and a decoder for the inverse mapping. The training loss is based on an Earth Mover's Distance between points. The training is done with an adversarial (min-max) strategy, that seeks to align the behavior of the encoder / decoder across a clean complete dataset and a partially-observed noisy one, with a Hausdorff distance loss, to cater for partial matching. The method is benchmarked on simulated and real data. It outperforms the state of the art for unsupervised settings (no known reconstructions) but for supervised settings it is slightly below the PCN approach.

I am not an expert of the application setting, hence I do not have many comments. The methods are well formulated and make sense. The experiments show the interest of the contributed method.

One thing that I do wonder is: given that the supervised approaches work better, but that there is not always data available for supervising, could a transfer learning approach be developed, adapting a supervised problem to a non supervised problem.

One question: how are the hyper-parameters set in the unsupervised setting? Appendix D details the model selection procedure: using a validation set and select the model that gives the best f1 score. I do not understand how this can be done without ground truth.
review_title: Official Blind Review #1

Meta-review: This paper presents an unsupervised method for completing point clouds obtained from real 3D scans based on GAN. Generally, the paper is well-organized, and its contributions and experimental supports are clearly presented, from which all reviewers got positive impressions.
Although the technical contribution of the method seems marginal as it is essentially a combination of established methods, it well fits in a novel and practical application scenario, and its useful is convincingly demonstrated in intensive experiments. We conclude that the paper provides favorable insights covering the weakness in technical novelty, so I’d like to recommend acceptance. 

Decision: Accept (Poster)