Diverse Shape Completion via Style Modulated Generative Adversarial Networks
Authors: Wesley Khademi
Li Fuxin
Venue: NeurIPS 2023 poster (2023)
Keywords: multimodal shape completion
point cloud completion
3d shape generation
generative modeling
generative adversarial networks

Abstract: Shape completion aims to recover the full 3D geometry of an object from a partial observation. This problem is inherently multi-modal since there can be many ways to plausibly complete the missing regions of a shape. Such diversity would be indicative of the underlying uncertainty of the shape and could be preferable for downstream tasks such as planning. In this paper, we propose a novel conditional generative adversarial network that can produce many diverse plausible completions of a partially observed point cloud. To enable our network to produce multiple completions for the same partial input, we introduce stochasticity into our network via style modulation. By extracting style codes from complete shapes during training, and learning a distribution over them, our style codes can explicitly carry shape category information leading to better completions. We further introduce diversity penalties and discriminators at multiple scales to prevent conditional mode collapse and to train without the need for multiple ground truth completions for each partial input. Evaluations across several synthetic and real datasets demonstrate that our method achieves significant improvements in respecting the partial observations while obtaining greater diversity in completions.

Diverse Shape Completion via Style Modulated Generative Adversarial Networks
Wesley Khademi Oregon State University khademmiwoorregonsttteee. edu
Li Fuxin Oregon State University Lifooregonstate edu
Abstract
Shape completion aims to recover the full 3D geometry of an object from partial observation. This problem inherently multi-modal since there can be many ways to plausibly complete the missing regions of a shape. Such diversity would be indicative of the underlying uncertainty of the shape and could preferable for downstream tasks such as planning. In this paper, propose a novel conditional generative adversarial network that can produce many diverse plausible completions of partially observed point cloud. To enable network to produce multiple completions for the same partial input, we introduce stochasticity into our network via style modulation. By extracting style codes from complete shapes during training, and learning a distribution over them, our style codes can explicitly carry shape category information leading to better completions. We further introduce diversity penalties and discriminators at multiple scales to prevent conditional mode collapse and train without the need for multiple ground truth completions for each partial input. Evaluations across several synthetic and real datasets demonstrate that method achieves significant improvements in respecting the partial observations while obtaining greater diversity in completions.
Figure 1: Given a partially observed point cloud (gray), our method is capable of producing many plausible completions (blue) of the missing regions.
1 Introduction
With the rapid advancements in 3D sensing technologies, point clouds have emerged as a popular representation for capturing the geometry of real-world objects and scenes. Point clouds come from sensors such as LiDAR and depth cameras, and can find applications in various domains such as robotics, computer-aided design, augmented reality, and autonomous driving. However, the 3D geometry produced by such sensors is typically sparse, noisy, and incomplete, which hinders their effective utilization in many downstream tasks.
This motivates the task 3D shape completion from partially observed point cloud, which has seen significant research in the past few years [1] 2
Many early point cloud completion works mostly focus on generating single completion that matches the ground truth in the training set, which does not take into account the potential uncertainty underlying the complete point cloud given the partial view. Ideally, an approach should correctly characterize such uncertainty generating mostly similar completions when most of the object has been observed, and less similar completions when less of the object has been observed. good characterization of uncertainty would be informative for downstream tasks such as planning or active perception to aim to reduce such uncertainty.
5 6
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The task of completing 3D point cloud with the shape uncertainty in mind is called multi-modal shape completion [ 9. which aims to generate diverse point cloud completions (not to be confused with molti-modality of the input, e.g. text+image). A basic idea is to utilize the diversity coming from generative models such as generative adversarial networks (GAN), where diversity, or the avoidance of mode collapse to always generate the same output, has been studied extensively. However, early works 8 often obtain diversity at a cost of poor fidelity to the partial observations due to their simplistic completion formulation that decodes from single global latent vector. Alternatively,
recent diffusion-basee methods [T0] D 2 and rego-seeeese methods 13 [4]
have shown greater
generation capability, but suffer from slow inference time.
In this paper, we propose an approach balance the diversity of the generated completions and fidelity to the input partial points. Our first novelty comes from the introduction of a style encoder to encode the global shape information of complete objects. During training, the ground truth shapes are encoded with this style encoder so that the completions match the ground truth. However, multiple ground truth completions are not available for each partial input; therefore, only using the ground truth style is not enough to obtain diversity in completions. To overcome this, we randomly sample style codes to provide diversity in the other generated completions. Importantly, we discovered that reducing the capacity of the style encoder and adding noise to the encoded ground truth shape leads to improved diversity of the generated shapes. We believe this avoids the ground truth from encoding too much content, which may lead the model to overfit to only reconstructing ground truth shapes.
Besides the style encoder, we also take inspiration from recent work SeedFormer 7 adopt a coarse-to-fine completion architecture. SeedFormer has shown high-quality shape completion capabilities with fast inference time, but only provides deterministic completions. In our work, we make changes to the layers of SeedFormer, making it more suitable for the multi-modal completion task. Additionally, we utilize discriminators at multiple scales, which enable training without multiple ground truth completions and significantly improves completion quality. We further introduce multi-scale diversity penalty that operates in the feature space of our discriminators. This added regularization helps ensure different sampled style codes produce diverse completions.
With these improvements. we build molli- point cloud completion algorithm that outperforms state-of- state-of-the-art in both the fidelity to the input partial point clouds as well as the diversity in the generated shapes. Our method capable of fast inference speeds since it does not rely on any iterative procedure, making it suitable for real-time applications such as in robotics.
Our main contributions be summarized follows:
We design novel conditional GAN for the task of diverse shape completion that achieves greater diversity along with higher fidelity partial reconstruction and completion quality.
We introduce a style-based seed generator that produces diverse coarse shape completions via style modulation, where style codes are leamed from distribution of complete shapes.
We propose multi-scale discriminator and diversity penalty for training our diverse shape completion framework without access to multiple ground truth completions per partial input.
2 Related work
2.1 3D shape generation
The goal 3D shape generation is to learn a generative model that can capture the distribution of 3D shapes. In recent years, generative modeling frameworks have been investigated for shape generation using different 3D representations, including voxels, point clouds, meshes, and neural fields.
One of the most popular frameworks for 3D shape generation has been generative adversarial networks (GANs). Most works have explored point cloud-bass GANs [5]] have shown that GANs can be trained on neural representations 20
18 19. while recent works 23 To avoid the training instability and mode collapse GANs, variational autoencoders have been used for learning 3D
16
17
21
22
shapes 24 25| 26). while other works have made use of Normalizing Flows 27 28 29 Recently, diverse shape generation has been demonstrated by learning denoising diffusion models on point clouds [30] latent representations
31 32), or neural fields [33].

---

2.2 Point cloud completion
Point cloud completion aims recover the missing geometry of shape while preserving the partially observed point cloud. PCN [ was among the earliest deep learning-base -based methods that worked directly on point clouds using PointNet [3 for point cloud processing. Since then, other direct point cloud completion methods [35] 36 4 37 38 have improved completion results by using local-to-globe feature extractors and by producing completions through hierarchical decoding.
Recently, SeedFormer 7 has achieved state-of-the-an performance in point cloud completion. Their Patch Seeds representation has shown be more effective than global shape representations due to carrying learned geometric features and explicit shape structure. Furthermore, their transformer-basee upsampling layers enable reasoning about spatial relationships and aggregating local information in coarse-to-fing manner, leading to improved recovery of fine geometric structures.
GAN-based completion networks have also been studied to enable point cloud completion learning in unpaired [39] 40, 41 or unsupervised [42 settings. To enhance completion quality, some works have leveraged adversarial training alongside explicit reconstruction losses
4
2.3 Multimodal shape completion
Most point cloud completion models are deterministic despite the ill-posed nature of shape comple- tion. To address this, Wu et al. [8] proposed GAN framework that learns stochastic generator, conditioned on a partial shape code and noise sample, to generate complete shape codes the latent space of a pre- pre-trained autoencoder. Arora et al. [9] attempt to mitigate the mode collapse present in g by using implicit maximum likelihood estimation. These methods can only represent coarse geometry and struggle to respect the partial input due to decoding from global shape latent vector.
ShapeFormer (13) and AutoSDF [14 explore auto-regressive approaches for probabilistic shape completion. Both methods propose compact discrete 3D representations for shapes and learn an auto-regressive transformer to model the distribution of object completions on such representation. However, these methods have a costly sequential inference process and rely on voxelization and quantization steps, potentially resulting in a loss of geometric detail.
Zhou et al. I0 propose a conditional denoising diffusion model that directly operates on point clouds to produce diverse shape completions. Alternatively, DiffusionSDF (11) and SDFusion [12] first learn a compact latent representation of neural SDFs and then learn a diffusion model over this latent space. These methods suffer from slow inference times due the iterative denoising procedure, have an additional costly dense querying of the neural SDF for extracting a mesh.
while 11 12
2.4 Diversity in GANs
Addressing diversity in GANs has also been extensively studied in the image domain. In particular, style-based generators have shown impressive capability high-quality diverse image generation where several methods have been proposed for injecting style into generated images via adaptive instance normalization 44
45
46
47
or weight modulation 4d [x]. In the conditional setting,
diversity has been achieved by enforcing invertibility between output and latent codes [49 or by
regularizing the generator to prevent mode collapse [50 51 52
3 Method
In this section, we present our conditional GAN framework for diverse point cloud completion. The overall architecture of our method is shown in Figure 2
Our generator is tasked with producing high-quality shape completions when conditioned on a partial point cloud. To accomplish this, first introduce new partial shape encoder which extracts features from the partial input. We then follow SeedFormer [7] and utilize a seed generator to first propose a sparse set of points that represent coarse completion of shape given the extracted partial features. The coarse completion is then passed through series of upsampling layers that utilize transformers with local attention further refine and upsample the coarse completion into dense completion.
To obtain diversity in our completions, we propose a style-based seed generator that introduces stochasticity into completion network at the coarsest level. Our style-based seed generator
3

---

Style-based based Seed Generator
Patch Seeds Fi groxiung
Upsample Transformer
Partil Encoder
MLP
Ussample layer
Upsample Layer
Upsample Layer
Style Modulator
Su
9:
Dg
D:
Dz
D2
Style Encoder
Duplicat
KL
Output at lovel:
-Daptminator
Figure 2: Overview of our diverse shape completion framework. A partial encoder is used to extract information from partial point cloud. During training, a style encoder extracts style codes from complete point clouds, and inference time style codes randomly sampled from normal distribution. Sampled style codes are injected into the partial information lo produce diverse Patch Seeds in our style-based seed generator. The generated Patch Seeds are then upsampled into a dense completion through upsampling layers. Furthermore, discriminators and diversity penalties are used at every upsampling layer to train our model.
modulates the partial shape information with style codes before producing a sparse set of candidate points, enabling diverse coarse shape completions that propagate to dense completions through upsampling layers. The style codes used in modulating partial shape information are lcarned from an object category's 's complete shapes via style encoder. Finally, diversity penalties at multiple scales to train our model to produce diverse high-quality completions without having access to multiple ground truth completions that correspond to partial observation.
introduce discriminators and
3.1 Partial shape encoder
The goal of our partial encoder is to extract shape information in a local-to-giobal fashion, extracting local information that will be needed in the decoding stage to reconstruct fine geometric structures, while capturing global information needed make sure globally coherent shape is generated. An overview of the architecture for our proposed partial encoder is shown in Figure 3a
Our partial encoder takes in partially observed point cloud Xp and first applies MLP to obtain set of point-wise features Fo. To extract shape information in a local-to-global fashion, L consecutive downsampling blocks are applied to obtain set of downsampled points X with local features Fz- In each downsampling block, a grid downsampling operation is performed followed by a series of PointConv (53) layers for feature interpolation and aggregation. Following the downsampling blocks, a global representation of the partial shape is additionally extracted by an MLP followed by max pooling, producing partial latent vector fr.
3.2 Style encoder
To produce multi-modal completions, we need to introduce randomness into our completion model. One approach is to draw noise from a Gaussian distribution and combine it with partial features during the decoding phase. Another option is to follow StyleGAN 46 [54] and transform noise samples through a non-linear mapping to a latent space W before injecting them into the partial features. However, these methods rely on implicitly learning connection between latent samples and shape information. Instead, we propose to learn style codes from an object category's 's distribution of complete shapes and sample these codes to introduce stochasticity in completion model. Our style codes explicitly carry information about ground truth shapes, leading to higher quality and more diverse completions.
To do so, we leverage the set of complete shapes we have access to during training. We introduce an encoder E that maps complete shape X E Vx3 global latent vector via a 4-layer MLP followed by max pooling. We opt for a simple architecture as we would like the encoder to capture high level information about the distribution of shapes rather than fine-grained geometric structure.
Instead of assuming we have complete shapes to extract style codes from at inference time, we learn a distribution over style codes that we can sample from. Specifically, we define our style encoder as a learned Gaussian distribution (={= = (=]pp( ((=]](((X))) (EE())) by adding two fully connected layers, and a, to the encoder E to predict the mean and standard deviation of Gaussian to sample style code from. Since our aim to learn style codes that convey information about complete shapes
that is useful for generating diverse completions, we train our style encoder with guidance from our

---

2
Partial Encoder
(Xa Fi) Mooor in
W1
Downsample Bock (14)
Hi Max Pool
Hod
Damad
Downsampiing Block
X,e 2
12/4
PaniCon
A
Mod
Demod
Coow
64
(b)
Figure 3: (a) Architecture of our partial shape encoder. (b) Overview of our style modulator network. For cach style-modulated convolution (gray box), and 4. are learned weights and biases of convolution, w. are the weights after the modulation and demodulation process, and A is learned Affine transformation.
completion network's 's losses. To enable sampling during inference, we minimize the KL-divergence between [x|xx and normal distribution during training. We additionally find that adding noise to our sampled style codes during training leads to higher fidelity and more diverse completions.
3.3 Style-based seed generator
We make use of the Patch Seeds representation proposed in SeedFormer 77,, which enables faithfully completing unobserved regions while preserving partially observed structures. Patch Seeds are defined as a set of seed coordinates S E N and seed features FF N produced by a seed generator. In particular, set of upsampled features F',, E xx are generated from the partial local features (XL, FL) via an Upsample Transformer [77] Seed coordinates S and features F are then produced from upsampled features Fup concatenated with partial latent code fr via an MLP.
However, the described seed generator is deterministic, prohibiting the ability to generate diverse Patch Seeds that can then produce diverse completions through upsampling layers. We propose to incorporate stochasticity into the seed generator by injecting stochasticity into the partial latent vector fp. We introduce style modulator network M (fp, 2). shown in Figure 3b that injects a style code into partial latent vector fr to produce a styled partial shape latent vector fc. Following [54], weight modulation inject style into the activation outputs of network layer, where the demodulated weights w used in each convolution layer are computed as:
A(=), mod: ways = s; Wijk, demod: whye
wlyk
W + Σ c.ke
(I)
where an Affine transformation, and w is the original convolution weights with i, corre- sponding to the input channel, output channel, and spatial footprint of the convolution, respectively.
3.4 Coarse-to-fine decoder
Our decoder operates in a coarse-to foneee fashion, which has been shown to be effective in producing shapes with fine geometric structure for the task of point cloud completion [36 38 4. We treat our Patch Seed coordinates S our coarsest completion Go and progressively upsample by each upsampling stage i, a set seed
factor r to produce denser completions 9. (i 1, 3). 3). features are first interpolated from the Patch Seeds. Seed features along with the previous layer's points and features are then used by an Upsample Transformer (7) where local self-attention is performed produce new set of points and features upsampled by factor r. We replace the inverse distance weighted averaging used to interpolate seed features from Patch Seeds in SeedFormer with a PointConv interpolation. Since the importance of Patch Seed information may vary across the different layers, we believe a PointConv interpolation is more appropriate than a fixed weighted interpolation as it can learn the appropriate weighting of Patch Seed neighborhoods for each upsampling layer.
Unlike the fully-connected decoders in 91, the coarse-to-fine decoder used our method can reason about the structures of local regions, allowing to generate cleaner shape surfaces. A coarse- to-fine design provides us with the additional benefit having discriminators and diversity penalties at multiple resolutions, which have found to lead to better completion fidelity and diversity.
5

---

3.5 Multi-scale discriminator
During training, we employ adversarial training assist in learning realistic completions for any partial input and style code combination. We introduce a set of discriminators D, for i = {0..... 3]. to discriminate against real and fake point clouds at each output level of our generator. Each discriminator follows PointNet-Mix architecture proposed by Wang et al. [55]. In particular, an MLP first extracts a set of point features from a shape, which are max-pooles and average- pooled to produce Jma and faeg, respectively. The features are then concatenated to produce mix mix-pooled feature fii = [maa= Savg] before being passed through a fully-connected network to produce a final score about whether the point cloud is real or fake.
We also explored more complex discriminators that made use of PointConv or attention mechanisms, but we were unable to successfully train with any such discriminator. This is in line with the findings in [55]. suggesting that more powerful discriminators may not guide the learning of point cloud shape generation properly. Thus, we instead use weaker discriminator architecture but have multiple of them that operate at different scales to discriminate shape information at various feature levels.
For training, we use the WGAN loss [56] with RI gradient penalty (57 and average over the losses at each output level i. We let X, (X p,)) be the completion output level for partial input Xp and sampled style code z ~ E2 Xx) and let X; be real point cloud of same resolution. Then our discriminator loss LD and generator loss LG are defined as:
LD
Σ
Ex- - 2 3 (IX- (D((tt)
(2)
1
1 Lo 4
Σ
(3)
where is a hyperparameter (7 = 1 in our experiments), P(X) is the distribution of generated shapes, and P(X) is the distribution real shapes.
3.6 Diversity regularization
Despite introducing stochasticity into the partial latent vector, it is still possible for the network to learn to ignore the style code z, leading to mode collapse to a single completion. To address this, we propose a diversity penalty that operates in the feature space of our discriminator. Our key insight is that for a discriminator to be able properly discriminate between real and fake point clouds, its extracted features should have learned relevant structural information. Then our assumption is that if two completions are structurally different, the discriminator's global mpolee features should be dissimilar as well, which we try to enforce through our diversity penalty.
Specifically, at every training iteration we sample two style codes ž Es(z|X() and Es(=|Xx from random complete shapes X1 and X2. For a single partial input X we produce two different completions (1, and (Xpp22))
We treat our discriminator D, as feature extractor and extract the mixed-pooled feature for both completions at every output level i. We denote the mixed-pooled feature corresponding to a completion conditioned on style code 2 at output level i by fiize then minimize:
Laiv
(4)
miz;
which encourages the generator to produce completions with dissimilar mix-pooled features for different style codes. Rather than directly using the discriminator's mix-pooled feature, perform pooling only over the set of point features that are not in partially observed regions. This helps avoid penalizing lack of diversity in the partially observed regions of our completions.
We additionally make use of partial reconstruction loss each output level both completions:
Epart
Σ
d (Xp, (((Xp,)))
(E( Km0
where HDDDD stands for the unidirectional Hausdorff distance from partial point cloud to completion. Such loss helps ensure that our completions respect the partial input for any style code 2.

---

To ensure that the completion set covers the ground truth completions in the training set, we choose to always set random complete shape X1 Xer and sample 21 (x where Xer is the corresponding ground truth completion to the partial input Xp. This allows us provide supervision the output of upsampling layer via Chamfer Distance (CD) for one of our style codes:
Ciamy Σ
(Xor, (c(Xp,21))
(6)
2=0
Our full loss that we use in training our generator is then:
L = Ag + Acompoomm + Aport + Adso Cati
(7)
We set Àg = 1, Acomp = 0.5, Apart = 1, Adiv = 5, which we found to be good default settings across the datasets used in experiments.
4 Experiments
In this section, we evaluate our method against a variety of baselines on the task of multimodal shape completion and show superior quantitative and qualitative results across several synthetic and real datasets. We further conduct a series of ablations to justify the design choices of our method.
Implementation Details Our model takes in Np 1024 points partial input and produces N 2048 points as a completion. For training the generator, the Adam optimizer is used with an initial learning rate of x 10-4 4 and the learning rate linearly decayed every epochs with decay rate of 0.98. For the discriminator, the Adam optimizer is used with a learning rate of 1 x 10 10-4, We train separate model for each shape category and train each model for 300 epochs with batch size of 56. All models are trained on two NVIDIA Tesla V100 GPUs and take about 30 hours to train.
Datasets We conduct experiments on several synthetic and real datasets. Following the setup of [ we evaluate our approach on the Chair, Table, and Airplane categories of the 3D-EPN dataset [58] Similarly, also perform experiments
the Chair, Table, and Lamp categories from the PartNet dataset [59]. To evaluate our method on real scanned data, we conduct experiments on the Google Scanned Objects (GSO) dataset [60]. For GSO, share quantitative and qualitative results on the Shoe, Toys, and Consumer Goods categories. A full description is presented the supplementary.
Metrics We follow [ and evaluate with the Minimal Matching Distance (MMD), Total Mutual Difference (TMD), and Unidirectional Hausdorff Distance (UHD) metrics. MMD measures the fidelity of the completion set with respect to the ground truth completions. TMD measures the completion diversity for partial input shape. UHD measures the completion fidelity with respect to the partial input. We evaluate metrics on K 10 generated completions per partial input. Reported MMD, TMD, and UHD values our results are multiplied by 10 102, and 102, respectively.
Table
Results on the 3D-EPN dataset. * indicates metrie is not reported.
MMD
TMD
UHD
Method
Chair
Plane
Table
Avg.
Chair
Plane
Table
Avg.
Chair
Plane
Table
Avg.
SeedFormer
0.45
0.17
0.65
0.42
0.00
0.00
0.00
0.00
1.69
1.27
1.69
1.55
KNN-Iatent
g
1.45
0.93
2.25
1.54
2.24
1.13
3.25
2.21
8.94
9.54
12.70
10.39
cGAN
g
1.61
0.82
2.57
1.67
2.56
2.03
4.49
3.03
8.33
9.59
9.03
8.98
IMLE
2.93
2.31
4.92
3.39
8.51
9.55
8.52
8.86
Ours
1.16
0.59
1.45
1.07
3.26
1.53
5.14
3.31
4.02
3.40
4.00
3.81
Partial
KNN
cGAN
Ours
Partil
KNN
iGAN
Ours
Partial
KNN
cGAN
Ours
Figure 4: Qualitative comparison of multi-modal completions on the 3D-EPN dataset.
7

---

Table 2: Results the PartNet dataset. iiidicctess that metric is not reported. that use an alternative computation for MMD and TMD.
indicates methods
MMD 4
TMD
UHD
Method
Chair
Lamp
Table
Avg.
Chair
Lamp
Table
Avg.
Chair
Lamp
Table
Avg.
SeedFormer
0.72
1.35
0.71
0.93
0.00
0.00
0.00
0.00
1.54
1.25
1.48
1.42
KNN-Iatent
8
1.39
1.72
1.30
1.47
2.28
4.18
2.36
2.94
8.58
8.47
7.61
8.22
cGAN
8
1.52
1.97
1.46
1.65
2.75
3.31
3.30
3.12
6.89
5.72
5.56
6,06
IMLE
9
2.76
5.49
4.45
4.23
6.17
5.58
5.16
5.64
ShapeFormer
6
1.32
3.96
Ours
1.50
1.84
1.15
1.49
4.36
6.55
5.11
5.34
3.79
3.88
PVD
10
1.27
1.03
1.98
1.43
1.91
1.70
5.92
3.18
Ours
1.34
1.55
1.12
1.34
5.27
7.11
5.84
6.07
H
PVD
Puraal
cGAN
Figure 5: Qualitative results on the PartNet dataset.
Baselines We compare our model against three direct multi-modal shape completion methods: cGAN 81. IMLE (9), and KNN-latent which is a baseline proposed [8]. We further compare with the diffusion--aar based method PVD (T0) and the auto-regressive method ShapeFormer [13]. We also share
quantitative results against the deterministic point cloud completion method SeedFormer 7
4.1 Results
Results on the 3D-EPN dataset are shown in Table SeedFormer obtains low UHD implying that their completions respect the partial input well: however, their method produces no diversity is deterministic. Our UHD is significantly better than all multi-modal completion baselines, suggesting that more faithfully respect the partial input. Additionally, our method outperforms others in terms of TMD and MMD, indicating better diversity and completion quality. This is also reflected in the qualitative results shown in Figure where KNN- fails produce plausible completions, while completions from cGAN contain high levels of noise.
In Table 2 compare against other methods the PartNet dataset. For fair comparison with PVD, we also report metrics following their protocol (denoted by 110. In particular, under their protocol TMD is computed on subsampled set of 1024 points and MMD is computed set concatenated with the original partial input. Once again our method obtains the best diversity (TMD) across all categories, beating the diffusion-based method PVD and the auto-regressive method ShapeFormer. Our method also achieves significantly lower UHD and shows competitive performance in terms of MMD. Some qualitative results are shown in Figure 5 We find that our method produces cleaner surface geometry and obtains nice diversity in comparison to other methods.
the subsampled
Table 3: Results Google Scanned Objects dataset. * indicates metric is not reported. indicates methods that use an alternative computation for MMD and TMD.
MMD
TMD f
UHD J
Method
Shoe
Toys
Goods
Avg.
Shoe
Toys
Goods
Avg.
Shoe
Toys
Goods
Avg.
SeedFormer
0.42
0.67
0.47
0.52
0.00
0.00
0.00
0.00
1.69
1.58
cGAN 8
1.00
2.75
1.79
1.85
1.10
1.87
1.95
1.64
5.05
6.61
6.35
6.00
Ours
0.85
1.90
0.99
1.25
1.71
2.27
1.89
1.96
2.88
3.84
4.68
3.80
PVD
0.66
2.04
1.11
1.27
1.15
2.05
1.55
8
*
Ours
0.90
1.72
1.04
1.22
2.42
2.88
2.57
2.62
8.
*
8

---

Pantal
cGAN
PVD
Ours
Pantal
cGAN
PVD
Our
Figure 6: Qualitative comparison of diverse completions on the Google Scanned Objects dataset.
Figure 7: Multi-modal completions (bluc) of partial point clouds (gray) produced by our method.
Additionally, we compare our method on real data from the Google Scanned Objects dataset. In Table 3] we show that our method obtains better performance across all the metrics for all three categories. We present a qualitative comparison
completions on objects from the Google Scanned Objects dataset in Figure Completions by cGAN [8 are noisy and lack diversity, while completions from PVD [0] have little diversity and suffer from non-uniform density. Alternatively, our method produces cleaner and more diverse completions with more uniform density.
We further demonstrate the ability of our method produce diverse high-quality shape completions in Figure 7 Even with varying levels of ambiguity in the partial scans, our method can produce plausible multi-modal completions of objects. particular, as in the lamp or shoe examples, our method produces more diverse completions. On the other hand, when the object is mostly observed, such in the plane example, our completions exhibit less variation among them. For more qualitative results, we refer readers to our supplemental.
find that under high levels of ambiguity, such
Finally, find that method is capable of inference in near real-time speeds. To produce K 10 completions of a partial input on a NVIDIA V100, our method takes an average of 85 ms while cGAN and KNN-latent require 5 ms. PVD) requires 45 seconds which is 500 times slower than
Table 5: Ablation style code generation.
Table 4: Ablation style code dim.
Method
MMD
TMD
UHD
Gaussian Noise
1.57
3.71
4.59
Style Code
MMD + TMD
UHD
Mapping Network
1.45
4.03
3.72
512-dim
1.51
3.41
3,98
Style Encoder (Ours)
1.50
4.36
3.79
128-dim
1.54
3.30
4.17
32-dim
1,59
3.89
4.04
Table 6: Ablation completion network design.
16-dim
1.55
3.96
3.97
8-dim
1.51
3.94
3.97
4-dim
1.51
4.00
3.93
Method
MMD | TMD 1 UHD |
8-dim noise (Ours)
1.50
4.36
3.79
SF
1.45
3.21
3.37
SF + PE
1.56
3.74
3.83
SF PE PCI (Ours)
1.50
4.36
3.79

---

speed, making it suitable Table 8: Ablation on loss functions.
Table 7: Ablation on discriminator architecture.
MMD
TMD t UHD 4
Method
MMD 4 TMD f UHD 4
w/o Lcomp
1.62
4.61
4.58
Single-scale
1.58
2.05
4.27
w/o Lpart
1.81
5.97
13.03
Multi-scale (Ours)
1.50
4.36
3.79
w/o Liivo
1.70
0,41
3.57
Ours
1.50
4.36
3.79
4.2
Ablation studies
In Table we examine the dimensionality of our style codes. Our method obtains higher TMD when using smaller style code dimension size. We additionally find that adding small amount of noise to sampled style codes during training further helps boost TMD while improving UHD. We believe that reducing the style code dimension and adding a small amount of noise helps prevent our style codes from encoding too much information about the ground truth shape, which could lead to overfitting to the ground truth completion. Furthermore, in T we present results with different choices for style code generation. Our proposed style encoder improves diversity over sampling style codes from a normal distribution or by using the mapping network from StyleGAN [46). Despite having slightly worse MMD) and UHD) than StyleGAN's N mapping network, we find the quality of completions at test time to be better when training with style codes sampled from our style encoder (see supplementary).
In our method, we made several changes to the completion network in SeedFormer (7]. We replaced the encoder from SeedFormer, which consisted of point transformer [6] and PointNet++ [62] set abstraction layers, with our proposed partial encoder as well as replaced the inverse distance weighted interpolation with PointConv interpolation in the SeedFormer decoder. To justify these changes, we compare the different architectures in our GAN framework in Table O We compare the performance of the original SeedFormer encoder and decoder (SF), our proposed partial encoder and SeedFormer decoder (PE SF), and our full architecture where we replace inverse distance weighted interpolation with PointConv interpolation in the decoder (PE + SF + PCT). Our proposed partial encoder produces an improvement in TMD for slightly worse completion fidelity. Further, we find using PointConv interpolation provides an additional boost in diversity while improving completion fidelity.
The importance of our multi-scale discriminator is shown in Table 7 tor/diversity penalty only the final output resolution results in a drop in completion quality and diversity when compared with our multi-scale design.
Using single discrimina-
Finally, we demonstrate the necessity of our loss functions in Table 8 Without fromps our method has to rely on the discriminator alone for encouraging sharp completions in the missing regions. This leads to a drop in completion quality (MMD). Without Larrt. completions fail respect the partial input, leading to poor UHD. With the removal of either of these losses, we do observe an increase in TMD; however, this is most likely due the noise introduced by the worse completion quality, Without Cdiv. we observe TMD drastically decreases towards zero, suggesting no diversity in the completions. This difference suggests how crucial our diversity penalty for preventing conditional mode collapse. Moreover, we observe that when using all three losses, our method is able to obtain good completion quality, faithfully reconstruct the partial input, and produce diverse completions.
Conclusion
In this paper, we present a novel conditional GAN framework that learns a one-to-many mapping between partial point clouds and complete point clouds. To account for the inherent uncertainty present in the task of shape completion, our proposed style encoder and style-based seed generator enable diverse shape completion. with multi-scale discriminator and diversity regularization preventing mode collapse in the completions. Through extensive experiments on both synthetic and real datasets, we demonstrate that our multi-modal completion algorithm obtains superior performance over current sfate-of-the-an approaches in both fidelity to the input partial point clouds and completion diversity. Additionally, our method runs in near for applications in robotics such as planning or active perception.
While our method is capable of producing diverse completions, it considers only segmented point cloud on the object. promising future research direction is to explore how to incorporate additional scene constraints (c.g. ground plane and other obstacles) into our multi-mode completion framework.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by ONR award N0014-21-1-2052, ONR/NAVSEA contract N00024- 10-D-63 (B-D6333//DD440000220077777 (Task 2: Fundamental Research in Autonomous Subsea Robotic Manipulation), NSF grant #1751412, and DARPA contract N66001-19-2-4035.
References
Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pen: Point completion network. 2018 international conference on 3D vision (3DV), pages 728-737. IEEE, 2018.
[2] Lyne Tcchapmi, Vineet Kosaraju, Hamid Rezatofighi, Ian Reid, and Silvio Savarese. Topnet: Structural point cloud decoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 383-392, 2019.
[3] Minghua Liu, Lu Sheng, Sheng Yang. Jing Shao, and Shi-Min Hu. Morphing and sampling network for dense point cloud completion. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11596-1 1603, 2020.
[4] Xiaogang Wang, Marcelo Ang Jr, and Gim Hee Lec. Cascaded refinement network for point cloud completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 790-799, 2020.
[5] Haozhe Xie, Hongxun Yao, Shangchen Zhou. Jiageng Mao, Shengping Zhang, and Wenxiu Sun. Grnet: Gridding residual network for dense point cloud completion. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX, pages 365-381. Springer, 2020.
[6] Xumin Yu, Yongming Rao, Ziyi Wang. Zuyan Liu, Jiwen Lu, and Jie Zhou, Pointr: Diverse point cloud completion with geometry-awan transformers. In Proceedings of the IEEF/CVE international conference on computer vision, pages -25977 2021.
Haoran Zhou, Yun Cao, Wenqing Chu, Junwei Zhu, Tong Lu, Ying Tai, and Chengjie Wang. Seedformer: Patch seeds based point cloud completion with upsample transformer. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part III. pages 416 432. Springer, 2022.
[8] Rundi Wu, Xuelin Chen, Yixin Zhuang, and Baoquan Chen. Multimodal shape completion via conditional generative adversarial networks. In The European Conference on Computer Vision (ECCV), August 2020.
[9] Himanshu Arora, Saurabh Mishra, Shichong Peng, Ke Li, and Mahdavi- Aandavi-Amiri. Multimodal shape completion via implicit maximum likelihood estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, pages 2958 2967, June 2022.
[10] Linqi Zhou, Yilun Du and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3826-5835, October 2021.
[11] Gene Chou, Yuval Bahat, and Felix Heide. Diffusion-sdf: Conditional generative modeling of signed distance functions. In The IEEE International Conference on Computer Vision (ICCV), 2023.
[12] Yen-Chi Cheng, Hsin- Hsin-Ying Lee, Sergey Tuyakov, Alex Schwing, and Liangyan Gui. SDFusion: Multimodal 3d shape completion, reconstruction, and generation. arXiv, 2022.
[13] Xingguang Yan, Liqiang Lin, Niloy J. Mitra, Dani Lischinski, Danny Cohen Or, and Hui Huang. Shape- former: Transformer based shape completion via sparse representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[14] Paritosh Mittal, Yen-Chi Cheng. Maneesh Singh, and Shubham Tulsiani. AutoSDF: Shape priors for 3d completion, reconstruction and generation. In CVPR, 2022.
[15] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference machine learning, pages 40- PMLR, 2018.
[16] Chun- Liang Li, Manzil Zaheer, Yang Zhang, Barnabas Poczos, and Ruslan Salakhutdinov. Point cloud gan. arXiv preprint arXiv:1810.05777 2018.

---

[17] Dong Wook Shu, Sung Woo Park, and Junseok Kwon. 3d point cloud generative adversarial network based tree structured graph convolutions. In Proceedings of the IEEE/CVF internationll conference on computer vision, pages 3859-3868. 2019.
[18] Ruihui Li, Xianzhi Li, Ka-Hei Hui, and Chi-Wing Fu. Sp-gan: Sphere-guided 3d shape generation and manipulation. ACM Transactions on Graphics (TOG), 40(4): 2021.
[19] Yingzhi Tang. Yue Qian, Qijian Zhang, Yiming Zeng. Junhui Hou, and Xuefei Zhe. Warpinggan: Warping multiple uniform priors for adversarial 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6397-6405, 2022.
[20] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1939-5945 2019.
[21] Marian Kleineberg. Matthias Fey, and Frank Weichert. Adversarial generation of continuous implicit shape representations. arXiv preprint wwxiiv2002.003444 2020.
[22] Moritz lbing, Isaak I Lim, and Leif Kohhelt. 3d shape generation with grid-hased implicit functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13559- 13568, 2021.
[23] X Zheng. Yang Liu, P Wang, and Xin Tong. Sdf-stylegan: Implicit sdf-based stylegan for 3d shape generation. In Computer Graphics Forum, volume 41, pages 52-63. Wiley Online Library, 2022.
[24] Jinwoo Kim, Jachoon Yoo, Juho Lee, and Seunghoon Hong. Setvae: Learning hicrarchical composition for generative modeling of sel-structured data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15059-15068, 2021.
[25] Alireza Makhzani, Jonathon Shlens, Navdcep Jaitly, Ian Goodfellow, and Brendan Frcy, Adversarial autoencoders. arXiv preprint arXiv:151.00644 2015.
[26] Matheus Gadelha, Rui Wang, and Subhransu Maji. Multiresolution networks for 3d point cloud processing. In Proceedings of the European Conference on Computer Vision (ECCV), pages 103-118, 2018.
[27] Guandao Yang, Xun Huang. Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. Proceedings of the IEEE/CVF international conference on computer vision, pages 4541-4550, 2019.
[28] Roman Klokov, Edmond Boyer, and Jakob Verbeek. Discrete point flow networks for efficient point cloud generation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16. pages 694-710. Springer, 2020.
[29] Janis Postels, Mengya Liu, Riccardo Spezialetti, Luc Van Gool, and Federico Tombari. Go with the flows: Mixtures of normalizing flows for point cloud generation and reconstruction. In 2021 International Conference on 3D Vision (3DV), pages 1249-1258. IEEE, 2021.
[30] Shitong Luo and Wei Hu. Diffiusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2837-2845, 2021.
[31] Xiaohui Zeng. Arash Vahdat, Francis Williams. Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint arXiv:22 10.06978, 2022,
[32] Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero. 3d-Idm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint arXiv:2212.00842, 2022.
[33] Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein, 3d neural field generation using triplane diffusion. arXiv preprint arXiv:2211.16677 2022.
[34] Charles Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[35] Xin Wen, Tianyang Li, Zhizhong Han, and Yu-Shen Liu. Point cloud completion by skip- attention network with hierarchical folding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patter Recognition, pages 1939-1948, 2020.
[36] Zitian Huang, Yikuan Yu, Jiawen Xu, Feng Ni, and Xinyi Le. Pf-net: Point fractal network for 3d point cloud completion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 7670- 2020.
12

---

[37] Xin Wen, Peng Xiang, Zhizhong Han, Yan-Pei Cao. Pengfei Wan, Wen Zheng, and Yu-Shen Liu. Pmi-nne:: Point cloud completion by learning multi-step point moving paths. In Proceedingy of the IEEE/CVF conference on computer vision and pattern recogmition, pages 7443-7452, 2021.
[38] Peng Xiang, Xin Wen, Yu-Shen Liu, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Zhizhong Han. Snowflak- enet: Point cloud completion by snowflake point deconvolution with skip transformer. In Proceedings of the IEEE/CVF international conference on computer vision. pages
2021.
[39] Xin Wen, Zhizhong Han, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Yu- Yu-Shen Liu. Cycle4completion: Unpaired point cloud completion using cycle transformation with missing region coding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13080-13089, 2021.
[40] Zhen Cao, Wenxiao Zhang, Xin Wen, Zhen Dong, Yu-shen Liu, Xiongwu Xiao, and Bisheng Yang. Ktnet: Knowledge transfer for unpaired 3d shape completion. arXiv e-prints, pages MiXiv-2111) 2021.
[41] Xuelin Chen, Baoquan Chen, and Niloy J Mitra. Unpaired point cloud completion on real scans using adversarial training. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[42] Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, and Chen Change Loy. Unsupervised 3d shape completion through gan inversion. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1768-1777, 2021.
[43] Chulin Xie, Chuxin Wang, Bo Zhang, Hao Yang, Dong Chen, and Fang Wen. Style-based point generator with adversarial rendering for point cloud completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4619-4628, 2021.
[44] Ting Chen, Mario Lucic, Neil Houlsby, and Sylvain Gelly, On self modulation for generative adversarial networks. arXiv preprint arXiv:/810.011366 2018
[45] Tacsung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2337-2346, 2019.
[46] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4401 1401-4410, 2019.
[47] Ari Heljakka, Yuxin Hou, Juho Kannala, and Arno Solin. Deep automodulators. Advances in Neural Information Processing Systems, 33:13702-13713, 2020.
[48] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. arXiv preprint 10Xv:22003.1042, 2021.
[49] Jun- Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexii Efros, Oliver Wang, and Eli Shechtman. Toward multimodal smage-to-image translation. Advances in neural information processing systems, 30, 2017.
[50] Augustus Odena, Jacob Buckman, Catherine Olsson, Tom Brown, Christopher Olah, Colin Raffel, and lan Goodfellow. Is generator conditioning causally related to gan performance? In International conference on machine learning, pages 3849-3858. PMLR, 2018.
[51] Dingdong Yang, Seunghoon Hong, Yunscok Jang, Tianchen Zhao, and Honglak Lee. Diversity-sensitive conditional generative adversarial networks. arXiv preprint arXiv: 100111990.00.. 2019.
[52] Qi Mao, Hsin-Ying Lee, Hung-Yu Yu Tseng, Siwei Ma, and Ming Hsuan Yang. Mode seeking generative adversarial networks for diverse image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1429-1437, 2019.
[53] Wenxuan Wu, Zhongang Qi. and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 9621-9633 2019.
[54] Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110 8119, 2020.
13

---

[55] He Wang, Zetian Jiang, Li Yi, Kaichun Mo, Hao Su, and Leonidas Guibas. Rethinking Sampling in 3D Point Cloud Generative Adversarial Networks. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR) Workshop Report, 2021.
[56] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR. 2017.
[57] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pages 3481 3490. PMLR, 2018.
[58] Angela Dai, Charles Ruizhongtai Qi, and Matthias NieBner. Shape completion using 3d-emcoder-predictoo cnns shape synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5868-5877, 2017.
[59] Kaichun Mo, Shilin Zhu, Angel Chang, Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. PartNet: A large-scale benchmark for fine grained and hierarchical part-level 3D object understanding, In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[60] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high high-quality dataset of 3d scanned household items. In 2022 International Conference Robotics and Automation (ICRA), pages 2553 2560. IEEE, 2022.
[61] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVE international conference on computer vision, pages 16259-16268 2021.
[62] Charles Ruizhongtai Qi. Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets a metric space, Advances in neural information processing systems, 30, 2017.
14


Reviews:
[{"review_id": "4WBTWpq2Ma", "reviewer": "Reviewer_A3pQ", "summary": "This work proposes a novel GAN for diverse shape completion from partial point clouds. To enable diverse completions, a style-based generator is introduced that leverages style codes from a learned distribution of complete shapes for style modulation. Further, a multi-scale discriminator and a diversity penalty are proposed for better diversity.", "questions": "Please also see Weaknesses.\n\nHow can diffusion-based methods, e.g. LION, be extended for point cloud completion? Would it be possible to train a variant that was conditioned on the partial point cloud (i.e. similar to 2D image inpainting with diffusion models) and report this as a further baseline?\nWhile diffusion models are limited wrt to inference speed, it would be interesting to include them as a baseline due to their strong generative performance.", "limitations": "The limitations were discussed but a broader impact session should be added.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow. The experimental evaluation is thorough and supports the main claims in the paper. Both quantitative and qualitative results demonstrate the efficiency of the proposed approach over existing works.\n", "weaknesses": "While the supplementary includes an ablation with different diversity penalties, it should also contain an experiment without the diversity penalty for completeness. As the diversity penalty is one of the contributions of this work, I would consider moving the study to the main paper. \n\nIt would further be interesting to report the nearest neighbors in the training set for the completed regions in the results because the style encoder learns a distribution over the completed shapes.\n\nA discussion of failure cases should be added to the supplementary.\n\nFor completeness, the missing values in Table 3 for “Ours \\dagger” should be added.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer A3pQ for their valuable and constructive comments.\n\n**Diversity penalty ablation**  \nIn our global response we have provided a new table where we train our method without diversity penalty $L_{div}$. Please see our global response for further comments on this ablation. We'll adopt our diversity penalty ablations to the main paper in the final version.\n\n**Report nearest neighbors in the training set for completions**  \nIn Figure 1 of our attached rebuttal PDF we have included some examples of the nearest neighbor in the training set to a completion. Please see our global response for further comments on this. We will add this to our supplementary material in our final version.\n\n**Discussion of failure cases**  \nWe have included a few examples of failure cases in Figure 3 of our attached PDF for our rebuttal. Similar to any other generative model, our method occasionally fails to produce plausible completions. We observe that the occasional failures that happen are usually either due to missing thin structures or some noisy artifacts. We will add this to our supplemental in our final version.\n\n**Missing values in Table 3**  \nNote that for the methods denoted by $\\dagger$ in Table 3, we evaluate using the protocol of PVD [10]. In particular, we produce a completion with 2048 points, subsample 1024 of them, and then directly concatenate the subsampled 1024 points to the partial input. In this case, it is not useful to evaluate UHD as it will always be 0 and thus we simply mark it with an asteriks.\n\n**Extending LION [31] to point cloud completion as a baseline to compare against diffusion models**  \nWhile modifying unconditional shape generation methods like LION to handle point cloud completion could be done, such a modification would require design/architectural choices on how to incorporate the partial point cloud input. This would most likely take a significant amount of effort to produce a reasonable baseline. Instead, we compare our method to Point-Voxel Diffusion (PVD) [10] which is a diffusion-based method that can perform diverse shape completion. We show our method outperforms PVD quantitatively and qualitatively, meanwhile being 500 times faster than it, in Table 2, Table 3, and Figure 5 of our main paper as well as Figure 3 of our supplementary material.\n\n**Broader impacts**  \nAs a point cloud completion method, it has the potential to be used in robotics applications such as planning. As of right now, we do not foresee immediate negative social impacts until our method is integrated into real robots. However, incorporating such a completion method into robotics applications is a scope for future work."}, {"review_id": "90nxZLRuQI", "reviewer": "Reviewer_iNun", "summary": "The goal of multimodal shape completion is to generate many different plausible completions of an incomplete shape. Based on the conditional GAN, this paper introduces two key concepts to improve the diversity and accuracy of multimodal completion. One is to use style codes instead of random noise, which means better distribution of complete shapes. The other is the use of multiscale discriminators to refine the predicted shapes from coarse to fine. The method is evaluated on both synthetic and real data sets and shows significant improvements over comparable methods based on standard metrics.", "questions": "1. Is this method applicable to unseen categories?", "limitations": "The limitation is well discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The results of multimodal completion look very clean and diverse. The numerical results show a new SOTA.\n\n2. The use of the learned style codes instead of Gaussian noise is reasonable since the former is more similar to the distribution of complete shapes.", "weaknesses": "1. Ablation study with and without $\\mathcal L_{comp}$ and $\\mathcal L_{div}$ is not provided.\n\n2. The flat and thin structures of the shapes generated by the proposed method look good, while other methods are more noisy. For the task of multimodal completion, this is quite impressive.\nAccording to Figure 5 in the supplementary file, the main reason is probably the use of multiscale discriminators, since the results of a single scale are noisy. Is this true? And why are 4 scales used? If more than 4 scales are used, is the quality continuously improved?\n\n3. The distribution of style codes should better reflect the actual shape distribution. To better verify this, a visual analysis comparing the distribution of the different conditional codes is helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer iNun for their valuable and constructive comments.\n\n**Ablation study on $L_{comp}$ and $L_{div}$**  \nIn our global response we have provided a new table where we train our method with each of $L_{comp}$, $L_{div}$, and $L_{part}$ set to 0. Please see our global response for further comments on this ablation.\n\n**Is the multi-scale discriminator the reason for sharp/clean completions? Why are 4 scales used?**  \nYes, we find that when using a single global discriminator it is not able to effectively discriminate against real and fake point clouds and hence we use several of them at different point cloud resolutions/scales. We tried out more complicated discriminators involving PointConv and attention mechanisms but were not able to successfully train with such methods, which is in line with the findings of Wang et al. [55] (see also lines 214-218 of our paper).  \n\nWe also find our loss functions $L_{comp}$ and $L_{part}$ to be important for producing sharper completions (see our new table and comments on this presented in our global response).  \n\nWe use 4 scales as our method makes use of the upsampling procedure from the SOTA deterministic point cloud completion method SeedFormer [7], which also uses 4 layers (an initial coarse completion + 3 upsampled completions). Note we start with a coarse completion of 256 points and upsample by a factor of 2 at each upsampling layer to produce a final completion of 2048 points. We did not explore using more than 4 scales since if we introduce more scales past 2048 points we would not be able to fairly compare against all the other multimodal shape completion methods which produce 2048 points as output.  \n\n**Visualization of distribution of style codes**    \nIn Figure 2 of our attached rebuttal PDF we share a plot of our learned style codes projected into 2D space using PCA. By querying random 2D points and visualizing the corresponding ground truth shapes in the neighborhood we find that the shapes in each cluster/neighborhood tend to share some characteristic/style. Such a result suggests that our style encoder is effectively learning to extract styles from the distribution of complete shapes.\n\n**Is method applicable to unseen categories?**  \nNo, our method is trained per-category similar to cGAN [8], IMLE [9], and PVD [10].\n"}, {"review_id": "c5IjnVKFyR", "reviewer": "Reviewer_ohCS", "summary": "The paper proposes a diverse shape completion method by extracting style codes from complete shapes and learning a distribution over them. Moreover, diversity penalties and discriminators at multiple scales are introduced as well to prevent conditional modal collapse to generate various object shapes. To verify the effectiveness of the method, various experiments are conducted, and promising results are observed.  ", "questions": "None.", "limitations": "Some limitations are discussed in detail. To make a more comprehensive comparison with other SOTA works, possible memory consumption or time efficiency could be illustrated.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and organized.\n2. The methods are interesting and promising results are obtained. ", "weaknesses": "1. According to line 156, “style codes” carrying category information are learnt from the distribution of complete shapes, however, the goal is to generate diverse shapes not only among different classes but also within the same class. Hence, how to get the diverse information for a single class seems missing in the paper. \n2. What is the difference of style encoder technique compared to VAE? According to the illustration in Sec. 3.2, it looks the same to VAE, which potentially results in limited novelty of the proposed style encoder. \n3. It would be better to also show some ablation studies on the network architectures by removing specific components to see how different parts contribute to the overall performances, which could be done by settings specific loss's weight to 0.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer ohCS for their valuable and constructive comments.\n\n**Diverse information for a single class seems to be missing.**  \nOur method is trained per category similar to other multimodal shape completion works such as cGAN [8], IMLE [9], and PVD [10]. Thus, the \"style codes\" we learn carry different style information from complete shapes within a category rather than across categories. In Figure 2 of our attached rebuttal PDF we project style codes to 2D space using PCA and show that neighborhoods in 2D space contain shapes with similar characteristics/styles.\n\n**What is the difference of style encoder technique from VAE? It looks similar to VAE and thus there is potentially limited novelty of the style encoder.**  \nIn principle, the style encoder uses similar loss functions and sampling tricks as VAE. However, there is a significant difference as we seek to make the reconstruction capability from the style encoder **worse** by using less dimensions and an inefficient PointNet encoder. The reason is that the input of the style encoder is the ground truth shape and if one reconstructs it directly from the style encoder, then the encoding from the partial input would have little purpose, which hurts generalization and diversity. Table 4 shows our finding that by making the style encoder **worse** by using less dimensions and adding noise, we were able to take more information from the other partial input encoder and achieve better diversity and reconstruction error during test time. We believe this is a novel finding.\n\n**Would be better to show some ablations on network architecture by removing components/losses**  \nIn our global response we have provided a new table where we train our method with each of $L_{comp}$, $L_{div}$, and $L_{part}$ set to 0. Please see our global response for further comments on this ablation.\n\n**Would be nice to show memory consumption or time efficiency**  \nIn the last paragraph of Section 4.1 (lines 297-299) of our main paper we provide a comparison to other methods of average inference speed for producing 10 completions of a single partial input. We find that our method is capable of near real-time speeds and is 500 times faster than diffusion-based method PVD [10]."}, {"review_id": "OhqZd5hfls", "reviewer": "Reviewer_R39m", "summary": "The paper proposes to reconstruct partial point cloud inputs using a multi-modal process, where the generator can output multiple plausible shapes. The key idea is to have a separate network (StyleEncoder) that extracts style from an input, in addition to a separate network (PartialEncoder) that extracts structural information, that are then combined to generate the final reconstruction. Another contribution is a multi-scale discriminator that checks real and fake pairs at multiple generator output stages. The entire pipeline is essentially a conditional GAN network for generating 3D shapes. ", "questions": "1.\tOn line 256, it is stated the input contains 1024 points and the output contains 2048 points. Why is there a mismatch in these numbers? \n2.\tIf the input is meant to be a partial point cloud, have the authors considered the effect of point density?  \n3.\tAre selected datasets benchmark datasets for partial shape reconstruction?\n4.\tIs the goal of the multi-scale discriminator to address partial input (lack of ground truth for each structure/style pair, or is it to address structure at different scales)? Given the name and design, the naming and purpose is confusing.\n", "limitations": "The paper proposed clever modifications on an existing network (SegFormer) to allow it to generate multiple completions given a single partial input. The idea is general, but it does not appear that the method is well-validated outside of the specific architecture considered, and thus, its impact is not clear. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes to address an important problem where there could be multiple potential reconstructions given a partial input. Limitations of recent work (SeedFormer [7]) are addressed with a clever network design strategy. The method is evaluated on three public datasets (3D-EPN, GSO, and Part-Net) with promising quantitative results.", "weaknesses": "The methodology section is difficult to follow, and I found it hard to understand the training process. In addition, it seems that the proposed technique is heavily based on previous work (e.g., SegFormer). It may help to explain the key technical contributions with respect to these previous works and defer the reader to the other papers for understanding technical details. \n\nIn addition, the validation of the proposed technique is rather limited. The training procedure, and validation steps are not clearly described. Is the method sensitive to shape orientation? What type of data augmentation is used during training, and is it trained on a per-category level (thus learning certain shape priors) or category-agonostic level? \n\nI would have expected to see ablation experiments with same methodology but different network architecture. Finally, the authors evaluate on PartNet, yet, never quantify part level accuracy.\n\nIn Tables 5 and 6, ablation studies of the style code and discriminator have very similar MMD metric but large UHD metric. I did not understand the choice and motivation behind these metrics. Why not use standard Hausdorff distance instead?\n\nFinally, the method is geared toward generative multiple outputs given a single input, yet this capability is never evaluated. I would have expected to see additional examples where multiple plausible reconstructions an input can have, followed by a user study or a qualitative evaluation of the results. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer R39m for their valuable and constructive comments.\n\n**Paper is difficult to follow. Helps to state key contributions.**    \nWe'd like to clarify any confusions you've had. Meanwhile, note that 2 other reviewers found the paper well-written. First, we want to note that our work is not related to SegFormer, but is related to the deterministic point cloud completion method SeedFormer [7].\n\nOur main contributions are outlined in lines 63-69 of our paper. In particular, we propose:\n- A diverse seed generator which produces diverse completions via style modulation of partial shape features (see lines 179-182)\n- To learn the style codes used in style modulation from the distribution of complete shapes via a style encoder (see lines 151-158)\n- Discriminators at multiple scales to enable shape completion without access to multiple ground truth completions per partial input (see lines 205-207 and 217-218)\n- A diversity penalty at multiple scales which prevents conditional mode collapse (see lines 226-232)\n\nNone of these exist in SeedFormer, which is completely deterministic in the completion it generates.\n\n**Validation is limited. Training and validation is not clearly described. Data augmentation performed?**  \nWe beg to differ about the claim that the validation is limited. We have quantitatively evaluated on 3 different datasets, compared against major baselines including a diffusion model, and performed multiple ablation studies. Training procedure, datasets, and metrics used can be found in lines 256-273 of our paper. We provided full dataset information and mathematical definitions of metrics in Sections 3 and 4 of our supplemental.  \n\nWe did not perform any data augmentation to fairly compare against prior methods which also did not perform any augmentation. Our sensitivity to shape orientation is hence similar to prior work.  \n\nOur method is trained per category similar to other diverse shape completion works we compare against [8, 9, 10].\n\n**Expected to see ablation experiments on architecture. Part level accuracy was not quantified for PartNet**  \nWe provide additional ablations on network design in our supplemental, where we compare our PointConv-based partial encoder with SeedFormer's partial encoder in Table 1 and compare our proposed diversity penalty with an alternative baseline diversity penalty in Table 2.  \n\nWe have included a new ablation on our losses in the table presented in our global response.  \n\nWe do not evaluate part level accuracy in PartNet because for the task of multimodal shape completion, it is not required to completely reconstruct all the ground truth shapes. Also, it is not possible to evaluate bidirectional metrics such as Chamfer distance or MMD as completions are not predicted at the part level; thus there is no notion of what part a point belongs to. Additionally, one-side metrics such as UHD would make incorrect penalizations as not all parts need to exist in the completions to be considered valid. For example, a ground truth chair may have arm rests; however, if a partial point cloud of the chair only contains its legs, completions of the chair without arm rests may still be valid completions. Also, note such a metric is not computed in any of the works we compare against [8, 9, 10, 13].\n\n**Why do ablation results have similar MMD but large UHD? What is the motivation of these metrics? Why not standard Hausdorff distance?**  \nThe UHD we used is the standard one-sided Hausdorff distance. MMD performs an average over distance to nearest neighbors while UHD perform a max over them and thus is more susceptible to large variations in the presence of noisy completions (see Section 4 of supplemental for mathematical definitions of metrics). \n\nMMD measures completion quality and coverage of the test set and UHD measures fidelity to the partial input. These are standard evaluation metrics for the task of diverse shape completion [8, 9, 10, 13].  \n\n**Method is geared toward generating multiple outputs but capability is never evaluated.**  \nNote that the TMD metric quantitatively evaluates the diversity of generated outputs. Besides, qualitative examples of diverse completions are shown in Figures 1, 4 and 5 of our main paper. In the supplemental, we shared more visual examples of our method producing diverse completions per partial input. We will add a sentence to our main paper mentioning that more visual examples can be found in our supplemental.\n\n**Why does input have 1024 points and output have 2048 points?**  \nWe follow the standard convention for multimodal shape completion which is to use a 1024 point partial input and produce a 2048 point completion as output [8, 9, 10, 13].\n\n**Is input point density considered?**  \nWe make no assumption on point density as it can vary heavily across shapes and viewpoints. 3DEPN and GSO inputs are generated by lifting depth maps into 3D and thus exhibit non-uniform density. For PartNet, we uniformly sample points in the parts kept; however, the number of parts and size of each part can still vary.  \n\n**Benchmark datasets for diverse completion task?**  \n3DEPN and PartNet datasets are benchmark datasets for diverse shape completion that previous works have evaluated on [8, 9, 10, 13]. We also evaluated on Google Scanned Objects to see how our method performs when trained with real data.\n\n**Goal of multi-scale discriminator?**  \nWe introduce adversarial learning to address lack of ground truth for each structure/style pair (lines 205-207 of our paper).\n\nThe goal of our multi-scale discriminator is to discriminate completions at different output resolutions. At coarser levels, points represent the skeletal structure of the shape, with each point being more important to whether the shape looks realistic. We find that discriminating at these coarse levels along with the finer levels helps with shape completion. We show a completion result using single vs. multiple discriminators in Figure 5 of our supplemental."}, {"review_id": "IQvgLkak4K", "reviewer": "Reviewer_Ahkf", "summary": "This paper proposes a new conditional generative network that can produce diverse completions of a partially observed point cloud. The stochasticity is introduced via style modulation. A style code is learned to explicitly carry shape category information leading to better completions. Moreover, diversity penalties and discriminators at multiple scales are also set to prevent conditional mode collapse. Experiments show that the proposed framework can achieve significant improvements in respecting the partial observations while obtaining greater diversity in completions.", "questions": "Please answer the questions in the weakness section.\nMoreover, can this method be utilized into the point cloud completion of scenes? Like ScanNet dataset. ", "limitations": "Further considering point cloud completion with text conditions can help to achieve open-world and diverse synthesis effects, which can improve the contribution of this paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The results on different datasets show the effectiveness of the proposed method.", "weaknesses": "1.\tThe main target of this paper is to achieve diverse synthesis. However, no such visual samples are provided. The authors should provide the diverse synthesis for different categories in the main paper.\n\n2.\tHow to decide the optimal values for loss weights in Eq. 7? Although the authors claim that the same loss weight setting can lead to good results on all experimental datasets, the experimental datasets only contain several categories. More categories should be considered into analysis.\n\n3.\tThe idea of this paper is derived from the 2D StyleGAN. The difference about the idea should be clearly indicated.\n\n4.\tThe method has not compared with SOTA methods that use the diffusion model, like [30, 31]. And this method should also support unconditional point cloud synthesis?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Ahkf for their valuable and constructive comments.\n\n**No visual samples of diverse synthesis are shown**  \nQualitative comparisons to other methods were included in Figures 1, 4 and 5 in our main paper. In each of these figures we show 3 completions produced for the same partial input. We additionally shared more visual examples in our supplemental material. We will add a sentence to our main paper mentioning that more visual examples can be found in our supplementary material.\n\n**How did you decide optimal values for loss weights? More categories should be considered for analysis.**  \nThe loss weights were set by initially setting them all to a value of 1. We found this led to overfitting to the single ground truth completion and less diverse completions, thus we decreased the weight of our completion loss and increased the weight of our diversity penalty, which led to more satisfactory results.  \n\nIn terms of the categories chosen, we follow previous works (cGAN [8], IMLE [9], PVD [10], ShapeFormer [13]) and evaluate on the chair, table, and airplane categories of the 3DEPN dataset and the chair, table, and lamp categories of the PartNet dataset. To further validate our method we have performed additional experiments on the shoe, toys, and consumer goods categories of the Google Scanned Objects dataset.\n\n**Idea is derived from 2D StyleGAN [54], the difference should be clearly indicated.**  \nNote that the only idea we use from StyleGAN is weight modulation as a way to inject style codes into our features. 2D StyleGAN is focused on unconditional 2D image generation while our work is on diverse 3D shape completion. StyleGAN starts from a learned constant tensor while our work starts from a set of features extracted from a partial shape. Additionally, in StyleGAN, style codes start from random noise and are mapped to style space through a mapping network while in our work we extract style codes during training from complete shapes. Please refer to lines 151-158 in our paper for a discussion on StyleGAN's style code versus ours as well as lines 180-185 where we mention partial input features and learned style codes as input to our style modulation.\n\n**Method does not compare against SOTA diffusion models [30, 31]. Does the method support unconditional point cloud synthesis?**  \nThe works by Luo et al. [30] and Zeng et al. [31] are SOTA methods for the task of unconditional 3D shape generation while our work is focused on the task of diverse point cloud completion. Their methods do not accept a partial point cloud as input and hence cannot be compared on the task we're trying to solve.  \n\nWe have compared against Point-Voxel Diffusion (PVD) by Zhou et al. [10] which is a diffusion based method that can perform diverse shape completion. We show our method outperforms PVD quantitatively and qualitatively in Table 2, Table 3, and Figure 5 of our main paper as well as Figure 3 of our supplemental. We also find our method runs 500 times faster than PVD.   \n\nOur method does not directly support unconditional shape generation as our seed generator expects partial shape features as one of its inputs. The seed generator could be modified to be conditioned on random noise instead of partial shape features; however, that is outside the scope of this paper which is point cloud completion.\n\n**Can the method be used on datasets like ScanNet?**  \nOur method works on segmented point clouds and thus it is possible to work on ScanNet if segmented properly. However, our method would need to be modified to avoid producing completions that intersect with other occupied points in the scene, which we mentioned as future work in lines 328-329 of our main paper. "}]

Author rebuttal:
We thank the reviewers for taking the time to read and review our work. We have tried our best to answer and address any questions and clarifications in each individual reviewer response. In the rest of our global response, we discuss a common ablation that was requested across several reviewers and share some comments on some additional figures in the PDF we have attached for our rebuttal.

--------------------------
### Ablation of loss functions:

|     | MMD ($\downarrow$) | TMD ($\uparrow$)     | UHD ($\downarrow$)    |
| :---        |    :----:   |    :----:   |         ---: |
| w/o $L_{comp}$      | 1.62         | 4.61      |4.58    |
| w/o $L_{part}$    | 1.81      | 5.97       | 13.03       |
| w/o $L_{div}$    | 1.70         | 0.41        | 3.57      |
| Ours   | 1.50       | 4.36        | 3.79       |


As requested by several reviewers, we have performed an ablation on our loss functions in the table above. Without $L_{comp}$ our method has to rely on the discriminator alone for encouraging sharp completions in the missing regions. This leads to a drop in completion quality (MMD). It also hurts partial reconstruction quality (UHD) as our completion loss $L_{comp}$ provides further gradient signal to the partially observed regions as well. Without $L_{part}$, completions fail to respect the partial input, leading to poor UHD. We observe that failing to respect the partial input also leads to a general degradation in completion quality (worse MMD). With the removal of either of these losses, we do observe an increase in TMD; however, we note that TMD is trivial to increase with worse completion quality as noise can be simply introduced. Without $L_{div}$, we observe TMD drastically decreases towards 0, suggesting no diversity in the completions. This difference suggests how crucial our diversity penalty is for preventing conditional mode collapse. We observe that when using all three losses, our method is able to obtain good completion quality, faithfully reconstruct the partial input, and produce diverse completions. We'd include this study in the final version.

--------------------------
### Visualizations in attached PDF:
In Figure 1 of our attached PDF, we share several completions (in blue) of a partial input and each completions nearest neighbor (in yellow) to a ground truth complete shape in the training set. Note that our method produces a different nearest neighbor for each completion of a partial input, showing that our method can overcome conditional mode collapse. Additionally, each nearest neighbor is similar to the partially observed region and varies more in the missing regions, suggesting that our method is capturing plausible diversity in our completions that matches with variance in the ground truth shape distribution.

In Figure 2 of our attached PDF, we plot our learned style codes from shapes in the training set by projecting them into 2D using principal component analysis (PCA). To better understand whether our style encoder is learning to extract style from the shapes, we visualize the corresponding shapes in random neighborhoods/clusters of our projected data. We find that the shapes contained in a neighborhood have a shared style or characteristic. For example, the chairs in the brown cluster all have backs whose top is curved while the black cluster has chairs that all have thin slanted legs.

Meta-review: The reviews of the paper is overall positive. This paper proposes a novel GAN-based method to generate diverse completion from partial point clouds. The diversity is demonstrated to outperform prior methods on multiple datasets. The rebuttal has addressed most of the comments. Thus I recommend acceptance. Also, I encourage the authors to revise the paper based on reviewers' suggestions and as promised in the rebuttal.
Decision: Accept (poster)