SlowGuess/NeurIPS_2023_OCR huggingface dataset:
which encourages the generator to produce completions with dissimilar mix-pooled features for different style codes. Rather than directly using the discriminator's mix-pooled feature, we perform pooling only over the set of point features that are not in partially observed regions. This helps avoid penalizing a lack of diversity in the partially observed regions of our completions.
We additionally make use of a partial reconstruction loss at each output level on both completions:
$$
\mathcal {L} _ {\text {p a r t}} = \sum_ {z \in \{z _ {1}, z _ {2} \}} \sum_ {i = 0} ^ {3} d ^ {U H D} \left(X _ {P}, \mathcal {G} _ {i} \left(X _ {P}, z\right)\right) \tag {5}
$$
where  $d^{UHD}$  stands for the unidirectional Hausdorff distance from partial point cloud to completion. Such a loss helps ensure that our completions respect the partial input for any style code  $z$ .
To ensure that the completion set covers the ground truth completions in the training set, we choose to always set random complete shape  $X_{1} = X_{GT}$  and sample  $z_{1} \sim E_{S}(z|X_{GT})$ , where  $X_{GT}$  is the corresponding ground truth completion to the partial input  $X_{P}$ . This allows us to provide supervision at the output of each upsampling layer via Chamfer Distance (CD) for one of our style codes:
$$
\mathcal {L} _ {\text {c o m p}} = \sum_ {i = 0} ^ {3} d ^ {C D} \left(X _ {G T}, \mathcal {G} _ {i} \left(X _ {P}, z _ {1}\right)\right) \tag {6}
$$
Our full loss that we use in training our generator is then:
$$
\mathcal {L} = \lambda_ {G} \mathcal {L} _ {G} + \lambda_ {\text {c o m p}} \mathcal {L} _ {\text {c o m p}} + \lambda_ {\text {p a r t}} \mathcal {L} _ {\text {p a r t}} + \lambda_ {\text {d i v}} \mathcal {L} _ {\text {d i v}} \tag {7}
$$
We set  $\lambda_{G} = 1, \lambda_{comp} = 0.5, \lambda_{part} = 1, \lambda_{div} = 5$ , which we found to be good default settings across the datasets used in our experiments.
In this section, we evaluate our method against a variety of baselines on the task of multimodal shape completion and show superior quantitative and qualitative results across several synthetic and real datasets. We further conduct a series of ablations to justify the design choices of our method.
Implementation Details Our model takes in  $N_P = 1024$  points as partial input and produces  $N = 2048$  points as a completion. For training the generator, the Adam optimizer is used with an initial learning rate of  $1 \times 10^{-4}$  and the learning rate is linearly decayed every 2 epochs with a decay rate of 0.98. For the discriminator, the Adam optimizer is used with a learning rate of  $1 \times 10^{-4}$ . We train a separate model for each shape category and train each model for 300 epochs with a batch size of 56. All models are trained on two NVIDIA Tesla V100 GPUs and take about 30 hours to train.
Datasets We conduct experiments on several synthetic and real datasets. Following the setup of [8], we evaluate our approach on the Chair, Table, and Airplane categories of the 3D-EPN dataset [58]. Similarly, we also perform experiments on the Chair, Table, and Lamp categories from the PartNet dataset [59]. To evaluate our method on real scanned data, we conduct experiments on the Google Scanned Objects (GSO) dataset [60]. For GSO, we share quantitative and qualitative results on the Shoe, Toys, and Consumer Goods categories. A full description is presented in the supplementary.
Metrics We follow [8] and evaluate with the Minimal Matching Distance (MMD), Total Mutual Difference (TMD), and Unidirectional Hausdorff Distance (UHD) metrics. MMD measures the fidelity of the completion set with respect to the ground truth completions. TMD measures the completion diversity for a partial input shape. UHD measures the completion fidelity with respect to the partial input. We evaluate metrics on  $K = 10$  generated completions per partial input. Reported MMD, TMD, and UHD values in our results are multiplied by  $10^{3}$ ,  $10^{2}$ , and  $10^{2}$ , respectively.
Table 1: Results on the 3D-EPN dataset. * indicates metric is not reported.
Method	MMD ↓	TMD ↑	UHD ↓
Chair	Plane	Table	Avg.	Chair	Plane	Table	Avg.	Chair	Plane	Table	Avg.
SeedFormer [7]	0.45	0.17	0.65	0.42	0.00	0.00	0.00	0.00	1.69	1.27	1.69	1.55
KNN-latent [8]	1.45	0.93	2.25	1.54	2.24	1.13	3.25	2.21	8.94	9.54	12.70	10.39
cGAN [8]	1.61	0.82	2.57	1.67	2.56	2.03	4.49	3.03	8.33	9.59	9.03	8.98
IMLE [9]	*	*	*	*	2.93	2.31	4.92	3.39	8.51	9.55	8.52	8.86
Ours	1.16	0.59	1.45	1.07	3.26	1.53	5.14	3.31	4.02	3.40	4.00	3.81
Figure 4: Qualitative comparison of multi-modal completions on the 3D-EPN dataset.
Table 2: Results on the PartNet dataset. * indicates that metric is not reported. † indicates methods that use an alternative computation for MMD and TMD.
Method	MMD ↓	TMD ↑	UHD ↓
Chair	Lamp	Table	Avg.	Chair	Lamp	Table	Avg.	Chair	Lamp	Table	Avg.
SeedFormer [7]	0.72	1.35	0.71	0.93	0.00	0.00	0.00	0.00	1.54	1.25	1.48	1.42
KNN-latent [8]	1.39	1.72	1.30	1.47	2.28	4.18	2.36	2.94	8.58	8.47	7.61	8.22
cGAN [8]	1.52	1.97	1.46	1.65	2.75	3.31	3.30	3.12	6.89	5.72	5.56	6.06
IMLE [9]	*	*	*	*	2.76	5.49	4.45	4.23	6.17	5.58	5.16	5.64
ShapeFormer [10]	*	*	*	1.32	*	*	*	3.96	*	*	*	*
Ours	1.50	1.84	1.15	1.49	4.36	6.55	5.11	5.34	3.79	3.88	3.69	3.79
PVD [10] †	1.27	1.03	1.98	1.43	1.91	1.70	5.92	3.18	*	*	*	*
Ours †	1.34	1.55	1.12	1.34	5.27	7.11	5.84	6.07	*	*	*	*
Figure 5: Qualitative results on the PartNet dataset.
Baselines We compare our model against three direct multi-modal shape completion methods: cGAN [8], IMLE [9], and KNN-latent which is a baseline proposed in [8]. We further compare with the diffusion-based method PVD [10] and the auto-regressive method ShapeFormer [13]. We also share quantitative results against the deterministic point cloud completion method SeedFormer [7].
Results on the 3D-EPN dataset are shown in Table 1. SeedFormer obtains a low UHD implying that their completions respect the partial input well; however, their method produces no diversity as it is deterministic. Our UHD is significantly better than all multi-modal completion baselines, suggesting that we more faithfully respect the partial input. Additionally, our method outperforms others in terms of TMD and MMD, indicating better diversity and completion quality. This is also reflected in the qualitative results shown in Figure 4, where KNN-latent fails to produce plausible completions, while completions from cGAN contain high levels of noise.
In Table 2, we compare against other methods on the PartNet dataset. For a fair comparison with PVD, we also report metrics following their protocol (denoted by  $\dagger$ ) [10]. In particular, under their protocol TMD is computed on a subsampled set of 1024 points and MMD is computed on the subsampled set concatenated with the original partial input. Once again our method obtains the best diversity (TMD) across all categories, beating out the diffusion-based method PVD and the auto-regressive method ShapeFormer. Our method also achieves significantly lower UHD and shows competitive performance in terms of MMD. Some qualitative results are shown in Figure 5. We find that our method produces cleaner surface geometry and obtains nice diversity in comparison to other methods.
Table 3: Results on Google Scanned Objects dataset. * indicates metric is not reported. † indicates methods that use an alternative computation for MMD and TMD.
Method	MMD ↓	TMD ↑	UHD ↓
Shoe	Toys	Goods	Avg.	Shoe	Toys	Goods	Avg.	Shoe	Toys	Goods	Avg.
SeedFormer [7]	0.42	0.67	0.47	0.52	0.00	0.00	0.00	0.00	1.49	1.69	1.55	1.58
cGAN [8]	1.00	2.75	1.79	1.85	1.10	1.87	1.95	1.64	5.05	6.61	6.35	6.00
Ours	0.85	1.90	0.99	1.25	1.71	2.27	1.89	1.96	2.88	3.84	4.68	3.80
PVD [10] †	0.66	2.04	1.11	1.27	1.15	2.05	1.44	1.55	*	*	*	*
Ours †	0.90	1.72	1.04	1.22	2.42	2.88	2.57	2.62	*	*	*	*
Figure 6: Qualitative comparison of diverse completions on the Google Scanned Objects dataset.
Figure 7: Multi-modal completions (blue) of partial point clouds (gray) produced by our method.
Additionally, we compare our method on real data from the Google Scanned Objects dataset. In Table 3, we show that our method obtains better performance across all the metrics for all three categories. We present a qualitative comparison of completions on objects from the Google Scanned Objects dataset in Figure 6. Completions by cGAN [8] are noisy and lack diversity, while completions from PVD [10] have little diversity and suffer from non-uniform density. Alternatively, our method produces cleaner and more diverse completions with more uniform density.
We further demonstrate the ability of our method to produce diverse high-quality shape completions in Figure 7. Even with varying levels of ambiguity in the partial scans, our method can produce plausible multi-modal completions of objects. In particular, we find that under high levels of ambiguity, such as in the lamp or shoe examples, our method produces more diverse completions. On the other hand, when the object is mostly observed, such as in the plane example, our completions exhibit less variation among them. For more qualitative results, we refer readers to our supplemental.
Finally, we find that our method is capable of inference in near real-time speeds. To produce  $K = 10$  completions of a partial input on a NVIDIA V100, our method takes an average of  $85~\mathrm{ms}$  while cGAN and KNN-latent require  $5\mathrm{ms}$ . PVD requires 45 seconds which is 500 times slower than us.
Table 4: Ablation on style code dim.
Style Code	MMD ↓	TMD ↑	UHD ↓
512-dim	1.51	3.41	3.98
128-dim	1.54	3.30	4.17
32-dim	1.59	3.89	4.04
16-dim	1.55	3.96	3.97
8-dim	1.51	3.94	3.97
4-dim	1.51	4.00	3.93
8-dim + noise (Ours)	1.50	4.36	3.79
Table 5: Ablation on style code generation.
Method	MMD ↓	TMD ↑	UHD ↓
Gaussian Noise	1.57	3.71	4.59
Mapping Network	1.45	4.03	3.72
Style Encoder (Ours)	1.50	4.36	3.79
Table 6: Ablation on completion network design.
Method	MMD ↓	TMD ↑	UHD ↓
SF	1.45	3.21	3.37
SF + PE	1.56	3.74	3.83
SF + PE + PCI (Ours)	1.50	4.36	3.79
Table 7: Ablation on discriminator architecture.
Method	MMD ↓	TMD ↑	UHD ↓
Single-scale	1.58	2.05	4.27
Multi-scale (Ours)	1.50	4.36	3.79
Table 8: Ablation on loss functions.
MMD ↓	TMD ↑	UHD ↓
w/o Lcomp	1.62	4.61	4.58
w/o Lpart	1.81	5.97	13.03
w/o Ldiv	1.70	0.41	3.57
Ours	1.50	4.36	3.79
In Table 4 we examine the dimensionality of our style codes. Our method obtains higher TMD when using smaller style code dimension size. We additionally find that adding a small amount of noise to sampled style codes during training further helps boost TMD while improving UHD. We believe that reducing the style code dimension and adding a small amount of noise helps prevent our style codes from encoding too much information about the ground truth shape, which could lead to overfitting to the ground truth completion. Furthermore, in Table 5, we present results with different choices for style code generation. Our proposed style encoder improves diversity over sampling style codes from a normal distribution or by using the mapping network from StyleGAN [46]. Despite having slightly worse MMD and UHD than StyleGAN's mapping network, we find the quality of completions at test time to be better when training with style codes sampled from our style encoder (see supplementary).
In our method, we made several changes to the completion network in SeedFormer [7]. We replaced the encoder from SeedFormer, which consisted of point transformer [61] and PointNet++ [62] set abstraction layers, with our proposed partial encoder as well as replaced the inverse distance weighted interpolation with PointConv interpolation in the SeedFormer decoder. To justify these changes, we compare the different architectures in our GAN framework in Table 6. We compare the performance of the original SeedFormer encoder and decoder (SF), our proposed partial encoder and SeedFormer decoder  $(\mathrm{PE} + \mathrm{SF})$ , and our full architecture where we replace inverse distance weighted interpolation with PointConv interpolation in the decoder  $(\mathrm{PE} + \mathrm{SF} + \mathrm{PCI})$ . Our proposed partial encoder produces an improvement in TMD for slightly worse completion fidelity. Further, we find using PointConv interpolation provides an additional boost in diversity while improving completion fidelity.
The importance of our multi-scale discriminator is shown in Table 7. Using a single discriminator/diversity penalty only at the final output resolution results in a drop in completion quality and diversity when compared with our multi-scale design.
Finally, we demonstrate the necessity of our loss functions in Table 8. Without  $\mathcal{L}_{comp}$ , our method has to rely on the discriminator alone for encouraging sharp completions in the missing regions. This leads to a drop in completion quality (MMD). Without  $\mathcal{L}_{part}$ , completions fail to respect the partial input, leading to poor UHD. With the removal of either of these losses, we do observe an increase in TMD; however, this is most likely due to the noise introduced by the worse completion quality. Without  $\mathcal{L}_{div}$ , we observe TMD drastically decreases towards zero, suggesting no diversity in the completions. This difference suggests how crucial our diversity penalty is for preventing conditional mode collapse. Moreover, we observe that when using all three losses, our method is able to obtain good completion quality, faithfully reconstruct the partial input, and produce diverse completions.
In this paper, we present a novel conditional GAN framework that learns a one-to-many mapping between partial point clouds and complete point clouds. To account for the inherent uncertainty present in the task of shape completion, our proposed style encoder and style-based seed generator enable diverse shape completion, with our multi-scale discriminator and diversity regularization preventing mode collapse in the completions. Through extensive experiments on both synthetic and real datasets, we demonstrate that our multi-modal completion algorithm obtains superior performance over current state-of-the-art approaches in both fidelity to the input partial point clouds and completion diversity. Additionally, our method runs in near real-time speed, making it suitable for applications in robotics such as planning or active perception.
While our method is capable of producing diverse completions, it considers only a segmented point cloud on the object. A promising future research direction is to explore how to incorporate additional scene constraints (e.g. ground plane and other obstacles) into our multi-modal completion framework.
This work was supported in part by ONR award N0014-21-1-2052, ONR/NAVSEA contract N00024-10-D-6318/DO#N0002420F8705 (Task 2: Fundamental Research in Autonomous Subsea Robotic Manipulation), NSF grant #1751412, and DARPA contract N66001-19-2-4035.
[55] He Wang, Zetian Jiang, Li Yi, Kaichun Mo, Hao Su, and Leonidas Guibas. Rethinking Sampling in 3D Point Cloud Generative Adversarial Networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop Report, 2021.
[56] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214–223. PMLR, 2017.
[57] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pages 3481-3490. PMLR, 2018.
[58] Angela Dai, Charles Ruizhongtai Qi, and Matthias Nießner. Shape completion using 3d-encoder-predictor cnns and shape synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5868-5877, 2017.
[59] Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[60] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pages 2553-2560. IEEE, 2022.
[61] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16259-16268, 2021.
[62] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.