Unpaired Point Cloud Completion on Real Scans using Adversarial Training
Authors: Xuelin Chen
Baoquan Chen
Niloy J. Mitra
Venue:  (2020)
Keywords: point cloud completion
generative adversarial network
real scans

Abstract: As 3D scanning solutions become increasingly popular, several deep learning setups have been developed for the task of scan completion, i.e., plausibly filling in regions that were missed in the raw scans. These methods, however, largely rely on supervision in the form of paired training data, i.e., partial scans with corresponding desired completed scans. While these methods have been successfully demonstrated on synthetic data, the approaches cannot be directly used on real scans in absence of suitable paired training data. We develop a first approach that works directly on input point clouds, does not require paired training data,  and hence can directly be applied to real scans for scan completion. We evaluate the approach qualitatively on several real-world datasets (ScanNet, Matterport3D, KITTI), quantitatively on 3D-EPN shape completion benchmark dataset, and demonstrate realistic completions under varying levels of incompleteness.


Published as a conference paper at ICLR 2020
UNPAIRED POINT CLOUD COMPLETION ON REAL SCANS USING ADVERSARIAL TRAINING
Xuelin Chen
Baoquan Chen Peking University
Niloy J. Mitra University College London Adobe Research London
Shandong University University College London
ABSTRACT
As 3D scanning solutions become increasingly popular, several deep learning se- tups have been developed for the task of scan completion, i.e., plausibly filling regions that were missed in the raw scans. These methods, however, largely rely on supervision the form of paired training data, i.e., partial scans with corre- sponding desired completed scans. While these methods have been successfully demonstrated on synthetic data, the approaches cannot be directly used on real scans in the absence of suitable paired training data. We develop first approach that works directly on input point clouds, does not require paired training data, and hence directly be applied to real scans for scan completion. We evaluate the approach qualitatively on several real-world datasets (ScanNet, Matterport3D, KITTI), quantitatively on 3D-EPN shape completion dataset, and demonstrate re- alistic completions under varying levels of incompleteness.
INTRODUCTION
Robust, efficient, and scalable solutions now exist for easily scanning large environments and workspaces (Dai al., 2017a Chang al., 2017). The resultant scans, however, are often par- tial and have to be completed (i.e., missing parts have to be hallucinated and filled in) before they can be used downstream applications, e.g., virtual walk-throughh, path planning.
The most popular data-driven scan completion methods rely on paired supervision data, i.e., for each incomplete training sean, a corresponding complete data (e.g., voxels, point sets, signed distance fields) is required. One way to establish such a shape completion network is then to train suitably designed encoder-decoder architecture (Dai et al. 2017b 2018). The required paired training data is obtained by virtually scanning 3D objects (e.g., SunCG Song et al. (017)) ShapeNet Chang al. (2015) datasets) to simulate occlusion effects. Such approaches, however, are unsuited for real scans where large volumes of paired supervision data remain difficult to collect. Additionally, when data distributions from virtual scans do not match those from real seans, completion networks trained on wynthetic-partic and synthetic-comples data do not sufficiently generalize real (partial) seans. To the best of our knowledge, no point-based unpaired method exists that learns to translate noisy and incomplete point cloud from raw scans clean and complete point sets.
We propose an unpaired point-based scan completion method that can be trained without requiring explicit correspondence between partial point sets (e.g., raw scans) and example complete shape models (e.g., synthetic models). Note that the network does not require explicit examples of real complete scans and hence existing (unpaired) large-scale real 3D scan (e.g.. Dai al. (2017a); Chang et al. (2017)) and virtual 3D object repositories (e.g..[Song et al. (2017): Chang et al. (2015) can directly be leveraged as training data. Figure | shows example sean completions. As we show in Table [ unlike methods requiring paired supervision, the data distributions of synthetic complete scans and real partial scans differ.
method continues to perform well even
We achieve this by designing a generative adversarial network (GAN) wherein generator, i.e., an adaptation network, transforms the input into a suitable latent representation such that a discrimi- nator cannot differentiate between the transformed latent variables and the latent variables obtained from training data (i.e., complete shape models). Intuitively, the generator is responsible for the key task of mapping raw partial point sets into clean and complete point sets, and the process is regu-

---

Published as a conference paper at ICLR 2020
Matterport3D raw scan
KITTI
EPN test scan
raw scan
raw scan
Figure 1: We present a point-based shape completion network that can be directly used on raw scans without requiring paired training data. Here we show a sampling results from the ScanNet, Matterport3D, 3D-EPN, 3D-EEN,
and KITTI datasets.
larized by working in two different latent spaces that have separately learned manifolds of scanned and synthetic object data.
We demonstrate our method on several publicly available real-world scan datasets namely (i) Scan- Net (Dai et al. 2017a) chairs and tables; (ii) Matterport3D (Chang et al., 2017) chairs and tables; and (iii) KITTI (Geiger et al. 2012) cars. In absence of completion ground truth, we cannot directly compute accuracy for the completed scans, and instead compare using plausibility scores. Further, in order to quantitatively evaluate the performance of the network, we report numbers on synthetic dataset (Dai et al., (2017b) where completed versions are available. Finally, we compare our method against baseline methods to demonstrate the advantages of the proposed unpaired scan completion framework.
RELATED WORK
Shape Completion. Many deep neural networks have been proposed address the shape com- pletion challenge. Inspired by CNN-based 2D image completion networks, 3D convolutional neural networks applied on voxelized inputs have been widely adopted for 3D shape completion task (Dai et al. 2018: 2017b Sharma et al. 2016 Han et al. 2017 Thanh Nguyen et al.) 2016 Yang et al., 2018 Wang et al. 2017). As quantizing shapes to voxel grids lead to geometric information loss,
recent approaches Yuaan al. 2018 Yu et al. 201 Achlioptas al. 2018) operate directly point sets to fill in missing parts. These works, however, require supervision in the form of partial- complete paired data for training deep neural networks directly regress partial input to their ground truth counterparts. Since paired ground truth of real-world data is rarely available such training data is generated using virtual scanning. While the methods work well on synthetic test data, they do not generalize easily to real scans arising from hard-to-model acquisition processes.
Realizing the gap between synthetically-geneaaie data and real-world data, /Stutz & Geiger (2018) proposed to directly work on voxelized real-world data. They also work in a latent space created for clean and complete data but measure reconstruction loss using a maximum likelihood estimator. Instead, we propose a GAN setup to learn a mapping between latent spaces respectively arising from partial real and synthetic complete data. Further, by measuring loss using Hausdorff distance on point clouds, we directly work with point sets instead of voxelized input.
Generative Adversarial Network. Since its introduction, GAN (Goodfellow et al. 2014) has
been used for variety of generative tasks. In 2D image domain, researchers have utilized adver- sarial training to recover richer information from low-resolution images or corrupted images (Ledig
et al. 2017 Wang al.) 2018 Mao et al.. 2017 Park et al. 2018 Bulat et al.) 2018 Yeh al. 2017 lizuka et al. (017). In 3D context, Yang et al. (2018); Wang et al. (2017) combine 3D-CNN and
generative adversarial training complete shapes under the supervision of ground truth data. Guru- murthy & Agrawal 2019) treats the point cloud completion task as denoising AE problem, utilizing adversarial training to optimize on the AE latent space. We also leverage the power of GAN for rea- soning the missing part of partial point cloud scanning. However, our method is designed to work with unpaired data, and thus can directly be applied to real-world scans even when real-world and synthetic data distributions differ. Intuitively, our GAN-based approach directly learns translation mapping between these two different distributions.
Deep Learning Point clouds. Our method is built upon recent advances in deep neural net- works for point clouds. PointNet Qi et al. (2017)). the pioneering work on this topic, takes an input point set through point-wise MI. layers followed by symmetric and permutation-invarti func-
2

---

Published as a conference paper at ICLR 2020
scon
P
P
P
3
E
D f
S
complem PCD
Zec.
dsee and compans
dean
complex
Ex
GAN
neal or lake
2c
S
S
D
2
Zc
R
nsay
G#
D s
partal PCD
completion
Figure 2: Unpaired Scan Completion Network.
tion to produce a compact global feature, which can then be used for a diverse set of tasks (c.g.. classification, segmentation). Although many improvements to PointNet have been proposed (Su et al. 2018 ei al. 2018b Qi et al. 2017b [i al., 2018a Zaheer et al. (0077). the simplicity and
effectiveness of PointNet and its extension PointNet++ make them popular for many other analysis
tasks (Yu et al. 2018a Yin et al. 2018 Yu et al.
2018b Guerrero et al. 2018).
In the context of synthesis, AAhhiippta al. (2018) proposed autoencoder network, using PointNet-based backbone, to learn compact representations of point clouds. By working in a reduced latent space produced by the autoencoder, they report significant advantages training GANs, stead of having generator producing raw point clouds. Inspired by this work, we design GAN to translate between two different latent spaces perform unpaired shape completion on real scans.
METHOD
Given a noisy and partial point set S {s:} } as input, our goal is to produce a clean and complete point set R {r,} as output. Note that although the two sets have the same number of points, there is no explicit correspondence between the sets S and R. Further, we assume access to clean and complete point sets for shapes for the object classes. We achieve unpaired completion by learning two class-specific point set manifolds, X, for the scanned inputs, and X for clean and complete shapes. Solving the shape completion problem then amounts to learning a mapping X, X. between the respective latent spaces. We train a generator Go X, + X to perform the mapping. Note that we do not require the noise characteristics in the two data distributions, i.e., real and synthetic, to be the same. In absence of paired training data, we score the generated output by setting up a min-max game where the generator is trained to fool a discriminator Fx, whose goal is to differentiate between encoded clean and complete shapes, and mapped encodings of the raw and partial inputs. Figure 2 shows the setup the proposed scan completion network. The latent space encoder-decoden the mapping generator, and the discriminator are all trained as detailed next.
3.1 LEARNING LATENT SPACES FOR POINT SETS
The latent space of given set of point sets is obtained by training autoencoder, which encodes the given input to www-dimension latent feature and then decodes to reconstruct the original input. We work directly the point sets via these learned latent spaces instead of quantizing them voxel grids or signed distance fields.
For point sets coming from the clean and complete point sets P. learn encoder network E; that maps P from the original parameter space 3N defined by concatenating the coordinates the N (2048 in all our experiments) points, to a lower-dimensional latent space X,. A decoder network D performs the inverse transformation back to R giving reconstructed point set P with also N points. The encoder-decoders are trained with reconstruction loss,
EMD (y, Φ) = Ep d(P, (E,
where P ~ Peomplete denotes point set samples drawn from the set clean and complete point sets, d( 6(X,, X2) is the Earth Mover's Distance (EMD) between point X1, X2, and (n, Φ) are the
3

---

Published as a conference paper at ICLR 2020
Figure Effect of unpaired scan completion without (Equation 5) and with HL term (Equa- tion (). Without the HL term, the network pro- duces clean point set for complete chair, that is different shape from the input. With the HL term, the network produces clean point set that matches the input.
Input point set
compleson HL
comptetion With
learnable parameters of the encoder and decoder networks, respectively. Once trained, the weights of both networks are held fixed and the latent code = = ( ; X for a clean and complete point set X provides compact representation for subsequent training and implicitly captures the manifold of clean and complete data. The architecture of the encoder and decoder is similar to Achlioptas et al. (2018); : Qi et al. (2017a): using 5-layer MLP to lift individual points to a deeper feature space, followed by a symmetric function to maintain permutation invariance. This results in a k- dimensional latent code that describes the entire point cloud (A=128 in all our experiments). More details of the network architecture can be found in the appendix.
As for the point set coming from the noisy-partial point sets S, one also train another encoder E S + Xr and decoder D X, + S pair that provides latent parameterization X, for the noisy-partial point sets, with the definition of the reconstruction loss as,
C FMD (7, v²) ES-p... d(s, (E,)))),
where S Praw denotes point set samples drawn from the set of noisy and partial point sets.
Although, in experiments, the latent space of this autoencoder trained on noisy-partial point sets works considerably well as the noisy-partial point set manifold, we found that using the latent space produced by feeding poiis-partial point sets to the autoencoder trained on clean and complete point sets yields slightly better results. Hence, unless specified, we set y = 7) and ai = Ó in our experi- ments. The comparison of different choices obtaín the latent space for noisy-partial point sets is also presented in Section 4 Next, we will descrihe the GAN setup to learn a mapping between the latent spaces of raw noisy-partial and synthetic clean-complete point sets, i.c., X,- %x.
3.2 LEARNING MAPPING BETWEEN LATENT SPACES
We set up min-max game between generator and a discriminator to perform the mapping between the latent spaces. The generator Go is trained to perform the mapping X, - X, such that the discriminator fails to reliably tell if the latent variable comes from original X_ or the remapped X,-
The latent representation of a noisy and partial scan 2, (S) is mapped by the generator to L (2 Then, the task of the discriminator Fg distinguish between latent representations E. and 2e (P)) We train the mapping function using GAN. Given training examples of clean latent variables 2, and remapped-non noisy latent variables Ecs we seek to optimize the following adversarial loss over the mapping generator Ge and discriminator Ex:
min max Enw coupons [log (Ex + Ey Par-Pasiy-aatt
E (G)(E,))))
(3)
In our experiments, we found the least square GAN Mao et al. (0166) to be easier to train and hence minimize both the discriminator and generator losses defined as,
Cx(x) Ear--ptanncnttt
(Fx(() - 1 + Ey Prrre puntsl (Fx((((E(()))]]
(4)
(o) Eyropaas porial
-1
(5)
The above setup encourages the generator to perform the mapping X, + X. resulting in (E.))) to be a clean and complete point cloud R. However, the generator is free to map a noisy latent vector to any point on the manifold of valid shapes in X,, including shapes that are far from the original partial scan S. As shown in Figure 3 the result is complete and clean point cloud that can be dissimilar in shape to the partial scanned input. To prevent this, we add a reconstruction loss term Creoon to the generator loss:
(0) Oz LeE.y--Pactygaan (Go(EE - 1]2 + BLtth (S, (FF(((()))))
(6)
4

---

Published as a conference paper at ICLR 2020
where C denotes the Hausdorff distance loss I (HL) from the partial input point set the comple- tion point set, which encourages the predicted completion point set to match the input only partially. Note that, it is crucial to use HL as Frecon- since the partial input can only provide partial supervision when no ground truth complete point set is available. In contrast, using EMD as frecom forces the network to reconstruct the overall partial input leading to worse completion results. The comparison of these design choices is presented in Section 4 Unless specified, we set the trade-off parameters as a 0.25 and 0.75 in all our experiments.
EXPERIMENTAL EVALUATION
We present quantitative and qualitative experimental results several noisy and partial datasets. First, we present results on real-world datasets, demonstrating the effectiveness of our method on unpaired raw scans. Second, 3D-EPN dataset, which contains simulated partial scans and corresponding ground truth for full evaluation. Finally, we derive a synthetic noisy-partial scan dataset based on ShapeNet, on which we can evaluate the performance degradation of applying supervised methods to test data of different distribution and the performance of our method under varying levels of incompleteness. A set of ablation studies is also included to evaluate our design choices.
thoroughly compare our method to various baseline methods on
Datasets. (A) Real-world dataset comes from three sources, First, a dataset of 5550 chairs and 550 tables extracted from the ScanNet dataset split into 90%-10% train-test sets. Second, a dataset of 20 chairs and 20 tables extracted from the Matterport3D dataset. Note that we train our method only on the ScanNet training split, and use the trained model to test on the Matterport3D data to evaluate generalization to new data sources. Third, a dataset containing cars from the KITTI Velo- dyne point clouds. (B) 3D-EPN dataset provides simulated partial scans with corresponding ground truth. Scans are represented as Signed Distance Field (SDF). We only use the provided point cloud representations of the training data, instead of using the SDF data which holds richer information. (C) Clean and complete point set dataset contains virtually scanned point sets of ShapeNet models covering categories, namely boat, car, chair, dresser, lamp, plane, sofa, and table. We use this dataset for learning the clean-complete point set manifold in all our experiments. (D) Synthetic dataset provides different incomplete scan distribution and at different levels of incompleteness. Ground truth complete scan counterparts are available for evaluation.
Evaluation measures. We assess completion quality using the following measures. (A) Accuracy measures the fraction of points in Peomp that are matched by Pgt. where and P Peomp denote the com- pletion point set and Pat denote the ground truth point set. Specifically, for each point e E Poomp: we compute D(v, Pat) e = 0.03, we count it as a correct match. The fraction of matched points is reported as the accuracy in percentage. (B) Completeness reports the fraction of points in Pgt that are within distance thresh- old e of any point in Peomp-
minn
q |l, E }. If D(v, Pgt) is within distance threshold
(C) Fl score is defined as the harmonic average of the accuracy and the completeness, where F1 reaches its best value at 1 (perfect accuracy and completeness) and worst at 0. (D) Plausibility of the completion is evaluated as the classification accuracy in percentage pro- duced by PointNet+++, SOA point-based classification network. To avoid bias on ShapeNet point clouds, we trained the classification network on the ModelNet40 dataset. We mainly used plausi- bility score for real-world data completions, where no ground truth data is available for calculating accuracy, completeness, or Fl scores.
In the following, we show all experimental and evaluation results. We trained separate networks for each category. More details are in the appendix.
4.1 EXPERIMENTAL RESULTS ON REAL- WORLD DATA
Our method works directly on real- wollw data where no paired data available. We train and test our network moisy-partial chairs and tables extracted from the ScanNet dataset. We further test the network trained on ScanNet dataset on chairs and tables extracted from the Matterport3D dataset, to show how well our network can generalize to definitely unseen data. We present qualitative results of our method in Fig Our method consistently produces plausible completions for the SeanNet and Matterport3D data.
Unless specified, we use unidirectional Hausdorff distance, from the partial input its completion.
5

---

Published as a conference paper at ICLR 2020
ScanNet
Matterport3D
KITTI
Figure 4: Qualitative comparisons real-world data, which includes partial scans of ScanNet chairs and tables, Matterport3D chairs and tables, and KITTI cars. We show the partial input in grey and the corresponding completion in gold on the right.
In the absence of ground truth completions real data, compare method quantitatively against others based on the plausibility of the results. The left sub-table of Table shows that our method is superior those supervised methods, namely 3D-EPN and PCN. Directly applying PCN trained on simulated partial data to real-world data leads to completions that have low plausibility, while our method consistently produces results with high plausibility. 3D-EPN trained simulated partial data failed to complete the real-world partial scans. In (..cion|4.2ann Section 4.3, we present more in in-depth comparisons on 3D-EPN and our synthetic dataset, where the ground truth is available for computing accuracy, completeness, and F1 of the completions.
Table 1: Completion plausibility synthetic scans and real-world scans and effects of data distribution discrepancy. (Left) Plausibility comparison on synthetic scans and real-world scans. Synthetic scans includes test data from 3D-EPN, real-world scans includes ScanNet and Matter- port3D test data. 3D-EPN failed to produce good completions on real-world data. (Right) On our synthetic data, supervised methods trained on other simulated partial scans produce worse results on partial scans with different data distribution.
input
3D-EPN
PCN
3I-EPN
PCN
Ours
chair table
73.1
70.3
85.0
91.5
modee
noc.
comp.
F1
acc.
comp.
P1
acc.
comp.
Syntheii
52.5
71,2
72.0
80.6
chair
29.6
61.8
48.2
49.3
76.0
59.8
80.3
80.8
80.8
Real-world
chair
table
71.4
47.8
7.1
4.4
78.6
94,3
plane
Gable
cur
36.6
43.8
17.1
61.0
62.3
45.8
51.4
63.2
62.3
80.6
$1.4
70.3
71.2
83.1
82.6
84.5
80.7
83.8
81.7
Completing the car observations from KITTI is extremely challenging, as each car instance only receives few data points from the Lidar scanner. D the qualitative results of our method on completing sparse point sets of KITTI cars, we can see that our network can still generate highly plausible cars with such sparse inputs.
We also use a point-based object part segmentation network (Qi et al. 2017b) to indirectly evaluate completions of real-world data. Due to the absence of ground truth segmentation, calculate the approximate segmentation accuracy for each completion. Specifically, for the completion of a chair, count the predicted segmentation label of each point to be correct long the predicted label falls into the set of 4 parts (i.e., seat, back, leg, and armrest) chair class. Our completion results have much higher approximate segmentation accuracy compared to the real-world raw input (chair: 77.2% vs. 24.8%; table: 96.4% vs. 83.5%; and car: 98.0% vs. 5.2%, as segmentation accuracy on our completions versus on original partial input), indicating high completion quality.
4.2 COMPARISON WITH BASELINES ON 3D-EPN DATA
We compare our method to several baseline methods and present both quantitative and qualitative comparisons on the 3D-EPN test set:
Autoencoder (AE), which is trained only with clean and complete point sets.
3D-EPN (Dai et al. 2017b), supervised method that requires SDF input and is trained with paired data. We convert its Distance Field representation results into surface meshes, from which uniformly sample N points for calculating our point-based measures.

---

Published as a conference paper at ICLR 2020
Table Comparison with baselines on the 3D-EPN dataset. Note that 3D-EPN and PCN require paired supervision data, while ours does not. Ours outperforms 3D-EPN and achieves comparable results PCN. Furthermore, after adapted to leverage the ground truth data as well, our method achieves similar performance to PCN.
(fully supervised)
PCN (Tully supersise))
Ours (unsapervised)
(upperised))
model
F1
comp.
Fl
F1
comp.
F1
comp.
F1
bost
89.6
81.4
K5.3
62.4
S1.4
81.9
92.6
93.4
93.0
96.6
84.7
85.6
39.8
92.0
90.9
81.3
75.3
97.3
68.9
88.2
93.1
79.9
61.7
76.9
68.5
91.1
90.6
90.9
78.7
77.4
78.0
83.3
dresser
68.9
64.2
66.5
58.4
72.7
64.8
93.5
91.5
92.5
75.8
76.5
76.2
87.4
91.5
89.1
lamp
75.9
79.6
77.7
60.8
67.8
64.1
82.9
88.3
85.5
71.3
80.2
75.5
76.6
86.3
81.3
78.1
98.3
08.2
sofa
80.3
64.0
71.2
65.0
72.6
68.6
91.5
90.8
91.1
68.2
72.3
70.2
81.0
87.0
83.9
table
82.8
72.5
77.3
56.8
75.1
64.7
93,4
89.2
91.2
82.2
77.8
80.0
81.2
81
81.3
PCN (Yuan et al. 2018), which completes partial inputs in a hierarchical manner, receiving supervision from both sparse and dense ground truth point clouds.
Ourst, which is an adaption of our method for training with paired data, show that our method can be easily adapted to work with ground truth data, improving the completion. Specifically, set 0 and use EMD loss as Lrecon More details and discussion about adapting our method to train with paired data can be found the appendix.
Table 2 shows quantitative results 3D-EPN test split and summarizes the comparisons: although our network is only trained with unpaired data, our method outperforms 3D-EPN method and achieves comparable results PCN. Note that both 3D-EPN and PCN require paired data. Fur- thermore, after adapting our method to be supervised by the ground truth, the performance of our method (Ours+) improves, achieving similar performance PCN. Note that simple autoencoder network trained with only clean-complete data can produce quantitatively good results, especially when the input is rather complete. Thus, we also evaluate the performance of AE on our synthetic data with incompleteness control in Section 4.3 to show that AE performance declines dramatically as the incompleteness of the input increases. Additional comparisons are included in the appendix.
4.3 EFFECT OF DATA DISTRIBUTION DISCREPANCY AND VARYING INCOMPLETENESS
Supervised methods assume that simulated partial scans share the same data distribution as the test data. We conduct quantitative experiments to show that applying 3D-EPN and PCN to our synthetic data, which is of different data distribution its training data and in which the ground truth complete scans are not available for training, lead to performance degradation. The right sub-table of Table shows that our method continues to produce good completions on our synthetic data, as we do not require paired data for training. The visual comparison is presented in the appendix.
To evaluate our method under different levels of input incompleteness, we conduct experiments on our synthetic data, in which we can control the fraction of missing points. Specifically, train our network with varying levels of incompleteness by randomizing the amount missing points during training. and afterwards fix the amount of missing points for testing. Table 3 the performance of our method on different classes under increasing amount of incompleteness and the comparison to AE. We can see that AE performance declines dramatically the incompleteness of the input increases, while our method can still produce completions with high plausibility and F1 score.
Table 3: Effect of varying incompleteness. Performance of AE and with increasing incom- pleteness (% the missing points). Our completions remain robust even with increasing incom- pleteness as our method restricts the completion via the learned latent shape manifolds.
Plansibility
Plansiilitt
Plansibiiit
Plausibility
incomp.
Curs model
model
AE
AE
model
Ours
Curs
10
96.3
99.4
94.9
85.8
88.0
91.01
88.4
87.7
69.3
74.01
90.0
90.2
88.9
91.0
96.6
96.5
20
95.5
40
30
85.4
96.1
98.2
71.8
79.7
44.0
89.4
52.5
70.8
76.9
45.1
55.2
71.8
69.6
77.3
80.1
plane
85.0
89.0
84.9
94.1
92.8
50
58.6
96.4
63.4
72.5
38.0
83.5
33.5
71.8
32.5
73,3
62.1
74.5
80.0
90.7
80.6
91.11

---

Published as a conference paper at ICLR 2020
4.4 DIVERSITY OF THE COMPLETION RESULTS
Our network is encouraged to partially match the input, alleviating the mode-collapse issue, which often occurs in GAN. Unlike traditional generative model problem, where high diversity in generated results is always better, the diversity our completion results should match that of the ground truth. Although this can be qualitatively accessed, see Fig. s..5ppm Appendix, in order to quantitatively quantify the divergence, we compute the Sensen-Shannon Divergence (JSD) between the marginal distribution of ground truth point sets and that of our completions proposed in Achliootas et al. 2018). As a reference, we simulate extremely mode-collapsed point cloud sets by repeating randomly selected point cloud, then report the JSD between ground truth point sets and the simulated extremely mode-collapsed point sets. The JSD scores lower is better highlight the diversity of 3D-EPN completions and the divergence between diversity and that of the ground truth using the extreme mode-collapse results as reference (the former ours): 0.06 vs. 0.46 on cars, 0.05 vs. 0.61 chairs, 0.04 0.53 planes and 0.04 0.59 on tables.
4.5 ABLATION STUDY
Ours with partial AE, uses encoder EXy and decoder D's that trained reconstruct partial point sets for the latent space of partial input.
Ours with EMD loss, uses EMD as the reconstruction loss.
Ours without GAN, "switch off" the GAN module by simply setting 0 and / 1, to verify the effectiveness of using adversarial training in our network.
Ours with reconstruction loss, removes the reconstruction loss term by simply setting and 0, to verify the effectiveness of the reconstruction loss term in generator loss.
Table 4 presents quantitative results for the ablation experiments, where demonstrate the impor- tance of various design choices and modules in our proposed network. We can see that our method has the best performance over all other variations.
Table Ablation study showing the importance various design choices in our proposed network.
Clurs w/ partial AE:
Ours w/ EMD
Our wio GAN
Our wio Recon.
Ours
soc.
ponigg
F1
acc.
comp.
F1
s0c.
pomp
F1
acc.
comp.
F1
600.
pomigg
F1
bosz
75.1
75.4
75.2
82.0
54.8
83.4
47.A
93.1
62.8
44.4
38.1
41.0
86.6
84.7
$5.6
88.9
87.6
88.2
76.0
70.8
76.4
46.2
88.3
60377
72.2
72.7
72.5
38.9
S1.7
$8.3
thair
64.1
66.7
65.4
78.6
76.4
77.5
41.3
79.8
54.4
75.6
75.1
75.3
78.7
77.4
75,0
67.4
71.4
72.3
74.4
55.4
21.4
lamp
64.0
74.8
89.0
69.9
79.0
74.2
28.6
84.7
42.8
15.6
22.2
18.5
71.3
80.2
75.5
plane
94.3
94.9
94.6
96.8
95.4
96.1
41.2
98.3
58.1
87.1
84.7
85.9
97.2
95.9
96.5
68.6
77.6
78.2
CONCLUSION
We presented point-based unpaired shape completion framework that be applied directly on raw partial seans to obtain clean and complete point clouds. At the core of the algorithm an adap- tation network acting a generator that transforms latent code encodings of the raw point scans, and maps them to latent code encodings of clean and complete object scans. The two latent spaces reg- ularize the problem by restricting the transfer problem to respective data manifolds. We extensively evaluated our method on real scans and virtual scans, demonstrating that our approach consistently leads to plausible completions and perform superior to other methods. The work opens up the pos- sibility of generalizing our approach to scene-level scan completions, rather than object-specific completions. Our method shares the same limitations as many of the supervised counterparts: does not produce fine-scale details and assumes input to be canonically oriented. Another interesting future direction will be to combine point- and image-feature, to apply the completion setup to both geometry and texture details.
8

---

Published as a conference paper at ICLR 2020
6 ACKNOWLEDGEMENTS
We thank all the anonymous reviewers for their insightful comments and feedback. This work is supported in part by grants from National Key R&D Program of China (2019YFF0302900) China Scholarship Council, National Natural Science Foundation of China (No.61602273), ERC Starting Grant, ERC PoC Grant, Google Faculty Award, Royal Society Advanced Newton Fellowship, and gifts from Adobe.
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for 3d point clouds. In International Conference on Machine Learn- ing (ICML), pp. 40 40-49, 2018.
Adrian Bulat, Jing Yang, and Georgios Tzimiropoulos. To learn image super-resolution, gan to learn how to do image degradation first. In European Conference on Computer Vision (ECCV), pp. 185-200, 2018.
Angel Chang, Angela Dai. Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv: 1709.06158 06188, 2017.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.0301 [cs.GR], Stanford University Princeton University Toyota Technological Institute Chicago, 2015.
Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. 1996.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießnnr. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Conference Com- puter Vision and Pattern Recognition (CVPR), 2017a.
Angela Dai, Charles Ruizhongtai Qi, and Matthias NieBner. Shape completion using 3d-encoder- predictor cnns and shape synthesis. In International Conference on Computer Vision (ICCV), pp- 5868-5877, 2017b.
Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, and Matthias NieBner. Scan- complete: Large-scale scene completion and semantic segmentation for scans. Conference Computer Vision and Pattern Recognition (CVPR), 2018
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Pro- cessing Systems, pp. 2672-2680, 2014.
Paul Guerrero, Yanir Kleiman, Maks Ovsjanikov, and Niloy J Mitra. Pepnet learning local shape properties from raw point clouds. In Computer Graphics Forum, volume 37, pp. 75-85, 2018.
Swaminathan Gurumurthy and Shubham Agrawal. High fidelity semantic shape completion for point clouds using latent optimization. In 2019 IEEE Winter Conference on Applications of Com- puter Vision (WACV), pp. 1099-11088 IEEF, 2019.
Xiaoguang Han, Zhen Li, Haibin Huang, Evangelos Kalogerakis, and Yizhou Yu. High-resolution shape completion using deep neural networks for global structure and local geometry inference. In International Conference on Computer Vision (ICCV), pp. 85-93, 2017.

---

Published as a conference paper at ICLR 2020
Satoshi lizuka, Edgar Simo-Serre and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (TOG), 36(4):107, 2017.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. In Conference on Computer Vision and Puttern Recognition (CVPR), pp. 4681 1-6900 2017.
Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self-organizing network for point cloud analysis. In Conference on Computer Vision and Puttern Recognition (CVPR), pp. 9397-9406, 2018a.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolu- tion on x-transformed points. In Advances in Neural Information Processing Systems, 20181 8b.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, and Zhen Wang. Multi-class generative adversarial networks with the L2 loss function. CoRR, abs/161 1.04 104076, 2016.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In International Conference on Computer Vision (ICCV), pp.
2017.
Seong-Jin Park, Hyeongseok Son, Sunghyun Cho, Ki-Sang Hong, and Seungyong Lee. Srfeat: Single image super-resolution with feature discrimination. In European Conference on Computer Vision (ECCV), pp. 439-455, 2018.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 652-660, 2017a.
Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgb-d data. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 918-927, 2018.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hicrarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems, pp. 5099-5108, 2017b.
Abhishek Sharma, Oliver Grau, and Mario Fritz. Vconv-dae: Deep volumetric shape learning with- out object labels. In European Conference on Computer Vision (ECCV), pp. 236-250, 2016.
Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from single depth image. Conference Computer Vision and Pattern Recognition (CVPR), 2017.
David Stutz and Andreas Geiger. Learning 3d shape completion from laser scan data with weak supervision. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1955-1964, 2018.
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, and Jan Kautz. Splatnet: Sparse lattice networks for point cloud processing. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2530-2539, 2018.
Due Thanh Nguyen, Binh-Son Hua, Khoi Tran, Quang-Hieu Pham, and Sai-Kit Yeung. field model for repairing 3d shapes. In Conference Computer Vision and Pattern Recognition (CVPR), pp. 5676-5684, 2016.
Weiyue Wang, Qiangui Huang, Suya You, Chao Yang, and Ulrich Neumann. Shape inpainting using 3d generative adversarial network and recurrent convolutional networks. In International Conference on Computer Vision (ICCV), pp. 2298-2306, 2017.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In European Confer- ence on Computer Vision (ECCV), pp. 63-79. Springer, 2018
10

---

Published as a conference paper at ICLR 2020
Bo Yang, Stefano Rosa, Andrew Markham, Niki Trigoni, and Hongkai Wen. 3d object dense recon- struction from single depth view. arXiv preprint arXiv:1802.00411, 1(2)6, 2018.
Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do. Semantic image inpainting with deep generative models. In Conference on Computer Vision and Puttern Recognition (CVPR), pp. 5485-5493, 2017.
Kangxue Yin, Hui Huang, Daniel Cohen-Or, and Hao Zhang. P2p-net: bidirectional point displace- ment net for shape transform. ACM Transactions on Graphics (TOG), 37(4):152, 2018.
Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. Ec-net: an edge- aware point set consolidation network. In European Conference Computer Vision (ECCV), pp- 482-402, 2018 8a.
Lequan Yu, Xianzhi Li, Chi- Bing Fu, Daniel Cohen-Or, and Pheng-Amn Heng. Pu-net: Point cloud upsampling network. In Conference on Computer Vision and Pattern Recognition (CVPR), pp- 2790-2799, 2018b.
Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pen: Point completion network. 2018 International Conference on 3D Vision (3DV), pp. 728-737, 2018.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems, 2017.

---

Published as a conference paper at ICLR 2020
A DETAILS OF DATASETS
Clean and Complete Point Sets are obtained by virtually scanning the models from ShapeNet. We subset of 8 categories, namely boat, car, chair, dresser, lamp, plane, sofa and table, in experiments. To generate clean and complete point set of a model, we virtually scan the models by performing mny-intersection test from cameras placed around the model to obtain the dense point set, followed by a down-sampling procedure to obtain relatively sparser point set of N points. Note that we use the models without any pose and scale augmentation.
This dataset used for training to learn the clean-complete point set manifold in all our experiments. The following datasets of different data distributions serve as different noisy-partial input data.
Real-world Data comes from three sources. The first one is derived from ScanNet dataset which provides many mesh objects that have been pre-segmented from its surrounding environment. For the purpsse of training and testing our network, we extract 5550 chair objects and ~550 table objects from ScanNet dataset, and manually align them be consistently orientated with models in ShapeNet dataset. We also split these objects into 90%/1 00% train/test sets.
The second one consists of 20 chairs and 20 tables from the Matterport3D dataset, which the same extraction and alignment as is done in ScanNet dataset is also applied. Note that we train our method only on ScanNet training split, and use the trained model to test on Matterport3D data, to show how our method can generalize to absolutely unseen data. For both ScanNet and Matterport3D datasets, we uniformly sample N points on the surface mesh of each object to obtain the input point sets.
Last, we extract car observations from the KITTI dataset using the provided ground truth bounding boxes for training and testing our method. We use KITTI Velodyne point clouds from the 3D object detection benchmark and the split of of et al. (2018). We filter the observations such that each car observation contains at least 100 points to avoid overly sparse observations.
3D-EPN Dataset provides partial reconstructions of ShapeNet objects (8 categories) by using vol- umetric fusion method Curless & Levoy] (1996) to integrate depth maps scanned along a virtual scanning trajectory around the model. For each model, a set of trajectories is generated with differ- ent levels of incompleteness, reflect the real-world scanning with a hand-held commodity RGB-D sensor. The entire dataset covers 8 categories and total of 25590 object instances (the test is composed of 5384 models). Note that, in the original 3D-EPN dataset, the data is represented in Signed Distance Field (SDF) for training data and Distance Field (DF) for test data. As our method works on pure point sets, we only use the point cloud representations of the training data provided by the authors, instead of using the SDF data which holds richer information and is claimed in Dai et al. (0177b) to be crucial for completing partial data.
Synthetic Data serves the purpose of having another dataset of different incomplete sean distri- bution and controlling the incompleteness of the input. We use ShapeNet to generate a synthetic dataset, in which we can control the incompleteness of the synthetic partial point sets. For the mod- els in each one of the 4 categories (car, chair, plane, and table), we split them into 90%/10% train/test sets. For each model, from which a clean and complete point set has been scanned (as described earlier in this subsection), we randomly pick point and remove its N x r (r [0, 1)) nearest neighbor points. The parameter r controls the incompleteness of the sunhetically-generaae Furthermore,
input. add Gaussian noise N N(u, o²) each point (==0 and a=0.01 for all our experi- ments). Last, we duplicate the points in the resulting point sets to generate point sets with an equal number of N points.
B NETWORK ARCHITECTURE DETAILS
In this section, we describe the details of the encoder, decoder, generator and discriminator in our network implementation.
B. AE ARCHITECTURE DETAILS
Encoder consists of 5 1-D convolutional layers which are implemented 1-D convolutions with ReLU and batch normalization, with kernel size of and stride of 1, to lift the feature of each point to high dimensional feature space independently. In all experiments, we use encoder with 64,
12

---

Published as a conference paper at ICLR 2020
128, 128, 256 and A 128 filters in each of its layers, with i being the latent code size. The output of the last convolutional layer is passed to a feature-wise maximum to produce a k-dimensional latent code.
Decoder transforms the latent vector using fully connected layers with 256, 256, and x neurons each, the first two having ReLUs, reconstruct N output.
B.2 GAN ARCHITECTURE DETAILS
Since the generator and discriminator of GAN directly operate on the latent space, the architecture for them is significantly simpler. Specifically, the generator is comprised of two fully connected layers with 128 and 128 neurons each, to map the latent code of noisy and incomplete point sets to that of clean and complete point sets. The discriminator consists of 3 fully connected layers with 256, 512 and neurons each, to produce a single scalar for each latent code.
C TRAINING DETAILS
To make the training of the entire network trackable, we pre-train the AEs used for obtaining the latent spaces, After that we retain the weights AEs, only the weights of the generator and discrim- inator are updated through the back-propagation during the GAN training. The following training hyper-paramend are used in all our experiments.
For training the AE, use Adam optimizer with an initial learning rate of 0.0005, 31 0.9 and batch size of 200 and train for a maximum of 2000 epochs.
For training the generator and discriminator the latent spaces, we Adam optimizer with an initial learning rate of 0.0001, 31 0.5 and a batch size of 24 and train the generator and discriminator alternately for maximum of 1000 epochs.
D QUALITATIVE RESULTS ON 3D-EPN DATASET
We present qualitatively comparisons in Fig where we show the partial input, AE, 3D-EPN, PCN, Ours, Ours+ result and the ground truth point We that, although our method is not quantitatively the best, our results are very qualitatively plausible, as the generator is restricted to generate point sets from learned clean and complete shape manifolds.
E VISUAL COMPARISON ON TEST DATA WITH DISTRIBUTION DIFFERENT TO TRAINING DATA
We present the visual comparison 3D-EPN, PCN and our method our synthetic data, which dif- fers from 3D-EPN and PCN training data. In this experiment, the ground truth of our synthetic data only available for evaluation. Fig 6 we can see that our method keeps producing high-quality completions, as our method does not require paired data for training hence can still be trained when no ground truth available. 3D-EPN and PCN produce much worse results as the data distribution of its training data and our synthetic data differ.
F VARIATIONS OF LEVERAGING GROUND TRUTH SUPERVISION
To adapt our network for training with ground truth point sets, we first change the reconstruction loss term in the generator loss from HD (Hausdorff Distance) to EMD (Earth Mover's Distance), as the ground truth point set is complete and thus contains full information for supervising the completion. Note that HD is superior when the ground truth is unavailable for training, shown in Table 4 of Section 4 Moreover, we present the comparison of different decisions on whether to adopt the adversarial training in network for training with the ground truth.
13

---

Published as a conference paper at ICLR 2020
Input
AE
3D
PCN
Durs
Ourst
GT
Input
AE
3D-EPN
PCN
Ours
Ours+
GT
Figure 5: Qualitative comparison on 3D-EPN dataset.
Figure 6: Effect of data distribution discrepancy and qualitative comparison synthetic dataset.
14

---

Published as a conference paper at ICLR 2020
Ours (GT+EMD), which is also denoted as Ours+ in the paper, removes the adversarial training in the network by simply setting a = and not updating the discriminator weights, hence there is only EMD reconstruction loss for the generator.
Ours (GT+EMD+GAN), in contrast, retains the adversarial training.
Table shows the quantitative comparison results, we can see that, when the ground truth point sets are available, Ours (GT+EMD) produces better results than Ours (GT+EMD+GAN), Our explana- tion for why adopting adversarial training here leads to worse results is that: when the ground truth is available, which is complete and contains all information for supervising the network, adding adver- sarial training will make the network much harder train, the network always gets punished by failing to fool the discriminator when is actually transforming current output closer to the ground truth.
Table 5: Removal of adversarial training when training with ground truth leads to significant im- provement.
Ours (GT+EMD)
Ours (GT+EMD+GAN)
model
acc.
comp.
F1
acc.
comp.
Fl
car
93.5
92.8
93.1
80.5
77.9
79.2
chair
82.3
83.3
82.8
51.5
58.1
54,6
plane
95.6
94.8
95.2
91.4
86.3
88.8
table
81.2
81.4
81.3
37.9
39.3
38.6
G MORE STATISTICS FRO THE BASELINE COMPARISON
For the baseline methods comparison on 3D-EPN dataset, we also report the Chamfer distance (CD), Earth Mover's Distance (EMD) and Hausdorff Distance (HD, maximum of the two directional distances) between the ground truth and the completion in Table
Table
AE
EPN
PCN
Ours
OBrs
model
CD
EMD
HD
CD)
EMD
HD
CD
EMD
HD
CD
EMD
HD
CD
EMD
HD
boal
0.012
0.0530
0.0864
6.0009
0.0500
0.0562
1.0006
0.8437
0.8635
0.0111
0.5133
0.0857
0.0008
0.0455
0.0799
0.0019
0.3688
0.1093
0.0124
0.0744
0.098)
0.0005
0.0418
0.8648
0.0100
00034
0.0763
0.0007
0.1393
010677
0.0744
0.0155
0.0619
0.0037
0.0985
0.1295
6.8027
0.073
0.8963
0
0.0545
0.0199
00588
0.0833
0.0011
0.042
0.0734
lamp
0.0226
0.0857
0.1092
0.0038
0.0366
0.1154
0.0013
0.0692
0.690
0.0025
0.0448
0.1073
0.0188
0.0729
0.1002
plane
0.0004
0,0346
0.0591
0,0050
0.0943
0.1255
0.9602
0.0308
0.0394
0.0004
0.0338
0.0545
0.0055
0.0405
0.0685
0.0680
0.1093
0/0555
0.0012
0.05%6
0.0958
Iable
0.0044
0,0896
0.1518
0.00]4
0.0681
0.9975
0.0010
0.0603
0.0968
0.0026
0.06$415
0.1071
0.0021
0.068]
0.1224
H USER STUDY ON HEAL-WORLD DATA COMPLETION
We also conducted a user study on the completion results of the real-world scans, where given partial input users are required to pick the most preferable completion among EPN, PCN and our results. In total, we received 1,000 valid user selections and report the preference (in percentage of the total selections) of each method in the user study. From Fig. 7 we can see that in over half (52%) of the selections, our completion results are selected as the best completion, while PCN completion results are better in 45% of the selections.
Interestingly, although our method outperforms other methods in the user study, our method clearly does not hold dominant position in this user study. After more in-depth analysis of the user selections, we found that users intend to pick the completion in which the partial input is embedded, which can be formulated as the Hausdorff distance from the partial input to the completion, and the supervised method PCN preserves the input point cloud in its completion output as this always minimizes the distance loss. The plausibility of the completion result is usually neglected by users,
15

---

Published as a conference paper at ICLR 2020
Preference in percentage
40
20
EPN
PCN
Figure 7
Input
PCN
Ours
Figure 8
while the plausibility and the Hausdorff distance from the partial input to the completion are both considered as two trade-off terms in the objective function of our completion generator. Fig. gives typical example, where the PCN completion without clear chair structure is often picked as better completion while our completion tries to trade-off between the HL and the plausibility.
I GENERALIZATION TO UNSEEN CLASSES
Input
Completion
GT
Figure 9
Our method does generalize to unseen objects from the same category in the test which is demon- strated in the experimental results section. However, our method intuitively should not generalize to classes that not during the training, as the autoencoder, which is fundamental component
16

---

Published as a conference paper at ICLR 2020
in our network, does not generalize to unseen classes. We present the qualitative results of applying our table completion network on chair and airplane class. We can see that from Fig. 9 on the unseen chair class, which shares similar structure with table, table completion network some reasonable structures, but is unable to complete with a seat back as the autoencoder does not have such capability; the unseen airplane class, which is rather dissimilar to table class, table completion network failed to complete the partial airplanes.
produce
17


Reviews:
[{"review_id": "rklENFSJqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript focuses on reconstructing 3D shapes from point clouds, with applications for instance to 3D scanners. The contribution builds on an adversarial formulation of the reconstruction, as in GANs. The method uses an encoder to map the observed noisy set of points into a lower-dimensional latent space and a decoder for the inverse mapping. The training loss is based on an Earth Mover's Distance between points. The training is done with an adversarial (min-max) strategy, that seeks to align the behavior of the encoder / decoder across a clean complete dataset and a partially-observed noisy one, with a Hausdorff distance loss, to cater for partial matching. The method is benchmarked on simulated and real data. It outperforms the state of the art for unsupervised settings (no known reconstructions) but for supervised settings it is slightly below the PCN approach.\n\nI am not an expert of the application setting, hence I do not have many comments. The methods are well formulated and make sense. The experiments show the interest of the contributed method.\n\nOne thing that I do wonder is: given that the supervised approaches work better, but that there is not always data available for supervising, could a transfer learning approach be developed, adapting a supervised problem to a non supervised problem.\n\nOne question: how are the hyper-parameters set in the unsupervised setting? Appendix D details the model selection procedure: using a validation set and select the model that gives the best f1 score. I do not understand how this can be done without ground truth.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rkgx7edm5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the task of point cloud completion within an unpaired setting, where explicit correspondences between the partial and the complete shapes is not given. The setting represents significant interest in practice, e.g. in autonomous driving applications, where the precise completions of scanned objects, e.g. surrounding cars, are not necessary. \n\nThe authors propose to use three models, wherein two models are point autoencoders in the style of [2], obtaining the two spaces of latent codes for the partial and the complete shapes, respectively. The third model learns a mapping between the two latent spaces in an adversarial way. While the idea of doing the unpaired shape completion has been known since the introduction unpaired image-based methods (e.g., [1]), the application is novel and the methods are formulated using the language known in the point cloud learning literature (e.g., EMD losses and point-based autoencoders).\n\nRegarding the evaluation procedure, the authors demonstrated convincingly that the proposed approach is feasible. However, I believe for shape completions with ground-truth labels more quality measures may be used, such as the Chamfer distance, Hausdorff distance, or Earth Mover’s distance (which is actually optimized by the authors), as shown in literature on point cloud upsampling [3], which is a related task. \n\nOverall, I believe that the paper does a good job of combining the established components into something new and useful, particularly, the problem is well-defined, the method is intuitive and extends the state-of-the-art, and the evaluation looks convincing. The paper is well-written and easy to follow, too. \n\n\n[1] Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision (pp. 2223-2232).\n[2] Achlioptas, P., Diamanti, O., Mitliagkas, I., & Guibas, L. (2018, July). Learning Representations and Generative Models for 3D Point Clouds. In International Conference on Machine Learning (pp. 40-49).\n[3] Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. Pu-net: Point cloud upsampling network. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2790–2799, 2018b.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyguezJ3cr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method for making 3D point clouds by automatically completing 3D scans. It does not require paired data samples for training which makes it possible to train it on real data instead of synthetic data. The authors use a generative adversarial network (GAN) to “generate” complete point clouds from noisy or partial point clouds obtained by 3D scanning. The generator learns to perform mapping from point set manifold of scanned noisy and partial input X_r to manifold of clean shapes X_c. The discriminator tries to tell between encoded clean shapes (synthetic data point clouds) and mappings of noisy input (point clouds from real-life data 3D scans). \n\nAn encoder-decoder network similar to those in Achlioptas et al. (2018) and Qi et al. (2017a) is trained to transform original point clouds to a low-dimensional latent space prior to training the GAN. The authors find that using the encoder-decoder trained on clean shape data even for noisy input yields better results. \nOne of the issues of completing noisy and partial scans is that the desired complete scan can have a very different shape compared to the noisy input. The generator can map latent vectors to any points on the target manifold which allows it to generate shapes that are far different from the original inputs. In order not to generate random clean shapes, the authors add a reconstruction loss to the generator which encourages it to preserve the partial shape of the input end reconstruct it in the completed clean shape. The choice of Hausdorff distance for reconstruction loss is sound and the ablation study confirms it. \n\nThe authors perform rather extensive experimental evaluation of their proposed method. They perform qualitative and quantitative analysis on several datasets, both real-life and synthetic ones. The proposed method outperforms existing methods in real-life data scenario. On the synthetic dataset (3D-EPN), the quantitative results are not as good as those of PCN (which is a supervised method, unlike the proposed method), but the qualitative results look plausible and comparable to PCN results. In terms of plausibility score, the proposed method outperforms existing methods in all experiments, which is probably thanks to its objective - map the input into the latent space of clean and complete shapes. However, plausible looking point clouds do not necessarily have to precisely match the input, which is the objective of 3D scan point cloud completion. \nThe ablation study also confirms the effectiveness of individual parts in the proposed method. \nThe main contribution of this paper is training with unpaired data, which enables training on real-life data, leading to better results on real-life scans. While I understand the difficulties, I believe it would be better to try to focus more on real-life data in the evaluation. There is only one experiment with quantitative analysis on real-life data in the paper. Supporting that with a visual Turing test would have been great. \n \nQuestions raised:\n\nIn 4.3, you say that “ground truth complete scans are not available for training.” How do you then train the supervised methods? Are they trained on 3D-EPN? In that case, is your proposed method also trained on 3D-EPN? If your method is the only one trained on your synthetic data (dataset D), which is also used for testing, then I do not think you could claim that your method is better at dealing with different data distributions. Please make clear in the comments what data you use for training in this experiment. \n\nThe meaning of section 4.4 is not very clear to me. If my understanding is correct, when completing noisy or partial point clouds, low diversity in results is better than high diversity because it is a task of repairing the input, which only has one correct result. Are you trying to say in this section that your method yields more consistent results with lower diversity than the method from previous work? If that is the case, you should consider rewriting that part to make it clearer to readers. \n\nThe PCN paper (Yuan et al., 2018) shows that PCN can generalize and complete point clouds of objects unseen during training very well. Unfortunately, this paper does not discuss performance on objects of unseen classes. Were such experiments considered? It would be beneficial to do a qualitative analysis of performance on unseen objects. \n\nThe description of the left table in Table 1 first says that it shows performance on real-life scans, but performance on synthetic data also appears there. Shouldn’t it rather say that it is plausibility comparison on real-life scans and synthetic data?\n\nSummary:\n\nThe proposed method is easy to understand, exploits recent progress in generative models, and allows training on real-life scans as it does not require paired training data. \nThe paper does not contain much algorithmical novelty and mostly combines existing methods to solve the problem of obtaining clean and complete point clouds from real 3D scans. However, the ablation study shows that adding a good reconstruction loss to the generator is crucial for the performance. \nThe main strengths of this paper are extensive experimental evaluation using both quantitative and qualitative analysis, and significant improvement in performance on real-life data over previous works. \nThe main weakness is limited novelty in terms of the techniques used in the proposed method. \n\nAdditional comments that do not affect the review decision:\n\nCitations in the text have to be revised and correctly put into parentheses where necessary. E.g., on page 2: “Since its introduction, GAN Goodfellow et al. (2014) has been used...” should be rewritten to “Since its introduction, GAN (Goodfellow et al., 2014) has been used...”\n\nTable 2 is the only table where the best results are not highlighted by bold text. It is also the only table where the proposed method is not the best performing method as it loses to PCN. I apologize if it is just a pure coincidence but it seems as if the authors did not want to draw attention to the fact that their proposed method loses to an existing method in that experiment. I believe that you should be fair and highlight best results in all tables. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an unsupervised method for completing point clouds obtained from real 3D scans based on GAN. Generally, the paper is well-organized, and its contributions and experimental supports are clearly presented, from which all reviewers got positive impressions.
Although the technical contribution of the method seems marginal as it is essentially a combination of established methods, it well fits in a novel and practical application scenario, and its useful is convincingly demonstrated in intensive experiments. We conclude that the paper provides favorable insights covering the weakness in technical novelty, so I’d like to recommend acceptance. 

Decision: Accept (Poster)