Title: Diverse Shape Completion via Style Modulated Generative Adversarial Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: multimodal shape completion

point cloud completion

3d shape generation

generative modeling

generative adversarial networks
Abstract: Shape completion aims to recover the full 3D geometry of an object from a partial observation. This problem is inherently multi-modal since there can be many ways to plausibly complete the missing regions of a shape. Such diversity would be indicative of the underlying uncertainty of the shape and could be preferable for downstream tasks such as planning. In this paper, we propose a novel conditional generative adversarial network that can produce many diverse plausible completions of a partially observed point cloud. To enable our network to produce multiple completions for the same partial input, we introduce stochasticity into our network via style modulation. By extracting style codes from complete shapes during training, and learning a distribution over them, our style codes can explicitly carry shape category information leading to better completions. We further introduce diversity penalties and discriminators at multiple scales to prevent conditional mode collapse and to train without the need for multiple ground truth completions for each partial input. Evaluations across several synthetic and real datasets demonstrate that our method achieves significant improvements in respecting the partial observations while obtaining greater diversity in completions.

Diverse Shape Completion via Style Modulated Generative Adversarial Networks

Abstract

Shape completion aims to recover the full 3D geometry of an object from a partial observation. This problem is inherently multi-modal since there can be many ways to plausibly complete the missing regions of a shape. Such diversity would be indicative of the underlying uncertainty of the shape and could be preferable for downstream tasks such as planning. In this paper, we propose a novel conditional generative adversarial network that can produce many diverse plausible completions of a partially observed point cloud. To enable our network to produce multiple completions for the same partial input, we introduce stochasticity into our network via style modulation. By extracting style codes from complete shapes during training, and learning a distribution over them, our style codes can explicitly carry shape category information leading to better completions. We further introduce diversity penalties and discriminators at multiple scales to prevent conditional mode collapse and to train without the need for multiple ground truth completions for each partial input. Evaluations across several synthetic and real datasets demonstrate that our method achieves significant improvements in respecting the partial observations while obtaining greater diversity in completions.

1 Introduction

With the rapid advancements in 3D sensing technologies, point clouds have emerged as a popular representation for capturing the geometry of real-world objects and scenes. Point clouds come from sensors such as LiDAR and depth cameras, and can find applications in various domains such as robotics, computer-aided design, augmented reality, and autonomous driving. However, the 3D geometry produced by such sensors is typically sparse, noisy, and incomplete, which hinders their effective utilization in many downstream tasks.

This motivates the task of 3D shape completion from a partially observed point cloud, which has seen significant research in the past few years [1, 2, 3, 4, 5, 6, 7]. Many early point cloud completion works mostly focus on generating a single completion that matches the ground truth in the training set, which does not take into account the potential uncertainty underlying the complete point cloud given the partial view. Ideally, an approach should correctly characterize such uncertainty - generating mostly similar completions when most of the object has been observed, and less similar completions when less of the object has been observed. A good characterization of uncertainty would be informative for downstream tasks such as planning or active perception to aim to reduce such uncertainty.

Li Fuxin

The task of completing a 3D point cloud with the shape uncertainty in mind is called multi-modal shape completion [8, 9], which aims to generate diverse point cloud completions (not to be confused with multi-modality of the input, e.g. text+image). A basic idea is to utilize the diversity coming from generative models such as generative adversarial networks (GAN), where diversity, or the avoidance of mode collapse to always generate the same output, has been studied extensively. However, early works [8, 9] often obtain diversity at a cost of poor fidelity to the partial observations due to their simplistic completion formulation that decodes from a single global latent vector. Alternatively, recent diffusion-based methods [10, 11, 12] and auto-regressive methods [13, 14] have shown greater generation capability, but suffer from slow inference time.

In this paper, we propose an approach to balance the diversity of the generated completions and fidelity to the input partial points. Our first novelty comes from the introduction of a style encoder to encode the global shape information of complete objects. During training, the ground truth shapes are encoded with this style encoder so that the completions match the ground truth. However, multiple ground truth completions are not available for each partial input; therefore, only using the ground truth style is not enough to obtain diversity in completions. To overcome this, we randomly sample style codes to provide diversity in the other generated completions. Importantly, we discovered that reducing the capacity of the style encoder and adding noise to the encoded ground truth shape leads to improved diversity of the generated shapes. We believe this avoids the ground truth from encoding too much content, which may lead the model to overfit to only reconstructing ground truth shapes.

Besides the style encoder, we also take inspiration from recent work SeedFormer [7] to adopt a coarse-to-fine completion architecture. SeedFormer has shown high-quality shape completion capabilities with fast inference time, but only provides deterministic completions. In our work, we make changes to the layers of SeedFormer, making it more suitable for the multi-modal completion task. Additionally, we utilize discriminators at multiple scales , which enable training without multiple ground truth completions and significantly improves completion quality. We further introduce a multi-scale diversity penalty that operates in the feature space of our discriminators. This added regularization helps ensure different sampled style codes produce diverse completions.

With these improvements, we build a multi-modal point cloud completion algorithm that outperforms state-of-the-art in both the fidelity to the input partial point clouds as well as the diversity in the generated shapes. Our method is capable of fast inference speeds since it does not rely on any iterative procedure, making it suitable for real-time applications such as in robotics.

Our main contributions can be summarized as follows:

We design a novel conditional GAN for the task of diverse shape completion that achieves greater diversity along with higher fidelity partial reconstruction and completion quality.

We introduce a style-based seed generator that produces diverse coarse shape completions via style modulation, where style codes are learned from a distribution of complete shapes.

We propose a multi-scale discriminator and diversity penalty for training our diverse shape completion framework without access to multiple ground truth completions per partial input.

2 Related work

2.1 3D shape generation

The goal of 3D shape generation is to learn a generative model that can capture the distribution of 3D shapes. In recent years, generative modeling frameworks have been investigated for shape generation using different 3D representations, including voxels, point clouds, meshes, and neural fields.

One of the most popular frameworks for 3D shape generation has been generative adversarial networks (GANs). Most works have explored point cloud-based GANs [15, 16, 17, 18, 19], while recent works have shown that GANs can be trained on neural representations [20, 21, 22, 23]. To avoid the training instability and mode collapse in GANs, variational autoencoders have been used for learning 3D shapes [24, 25, 26], while other works have made use of Normalizing Flows [27, 28, 29]. Recently, diverse shape generation has been demonstrated by learning denoising diffusion models on point clouds [30], latent representations [31, 32], or neural fields [33].

2.2 Point cloud completion

Point cloud completion aims to recover the missing geometry of a shape while preserving the partially observed point cloud. PCN [1] was among the earliest deep learning-based methods that worked directly on point clouds using PointNet [34] for point cloud processing. Since then, other direct point cloud completion methods [35, 36, 4, 37, 6, 38, 7] have improved completion results by using local-to-global feature extractors and by producing completions through hierarchical decoding.

Recently, SeedFormer [7] has achieved state-of-the-art performance in point cloud completion. Their Patch Seeds representation has shown to be more effective than global shape representations due to carrying learned geometric features and explicit shape structure. Furthermore, their transformer-based upsampling layers enable reasoning about spatial relationships and aggregating local information in a coarse-to-fine manner, leading to improved recovery of fine geometric structures.

GAN-based completion networks have also been studied to enable point cloud completion learning in unpaired [39, 40, 41] or unsupervised [42] settings. To enhance completion quality, some works have leveraged adversarial training alongside explicit reconstruction losses [4, 43].

2.3 Multimodal shape completion

Most point cloud completion models are deterministic despite the ill-posed nature of shape completion. To address this, Wu et al. [8] proposed a GAN framework that learns a stochastic generator, conditioned on a partial shape code and noise sample, to generate complete shape codes in the latent space of a pre-trained autoencoder. Arora et al. [9] attempt to mitigate the mode collapse present in [8] by using implicit maximum likelihood estimation. These methods can only represent coarse geometry and struggle to respect the partial input due to decoding from a global shape latent vector.

ShapeFormer [13] and AutoSDF [14] explore auto-regressive approaches for probabilistic shape completion. Both methods propose compact discrete 3D representations for shapes and learn an auto-regressive transformer to model the distribution of object completions on such representation. However, these methods have a costly sequential inference process and rely on voxelization and quantization steps, potentially resulting in a loss of geometric detail.

Zhou et al. [10] propose a conditional denoising diffusion model that directly operates on point clouds to produce diverse shape completions. Alternatively, DiffusionSDF [11] and SDFusion [12] first learn a compact latent representation of neural SDFs and then learn a diffusion model over this latent space. These methods suffer from slow inference times due to the iterative denoising procedure, while [11, 12] have an additional costly dense querying of the neural SDF for extracting a mesh.

2.4 Diversity in GANs

Addressing diversity in GANs has also been extensively studied in the image domain. In particular, style-based generators have shown impressive capability in high-quality diverse image generation where several methods have been proposed for injecting style into generated images via adaptive instance normalization [44, 45, 46, 47] or weight modulation [46, 48]. In the conditional setting, diversity has been achieved by enforcing invertibility between output and latent codes [49] or by regularizing the generator to prevent mode collapse [50, 51, 52] .

3 Method

In this section, we present our conditional GAN framework for diverse point cloud completion. The overall architecture of our method is shown in Figure 2.

Our generator is tasked with producing high-quality shape completions when conditioned on a partial point cloud. To accomplish this, we first introduce a new partial shape encoder which extracts features from the partial input. We then follow SeedFormer [7] and utilize a seed generator to first propose a sparse set of points that represent a coarse completion of a shape given the extracted partial features. The coarse completion is then passed through a series of upsampling layers that utilize transformers with local attention to further refine and upsample the coarse completion into a dense completion.

To obtain diversity in our completions, we propose a style-based seed generator that introduces stochasticity into our completion network at the coarsest level. Our style-based seed generator

Style

Encoder

Partial

Encoder

Style-Based Seed Generator

- Discriminator at level

- Output at level

- Duplicate

- Concatenate

D

C

D

C

Upsample

Transformer

Upsample

Layer

Upsample

Layer

Upsample

Layer

MLP

Style

Modulator

modulates the partial shape information with style codes before producing a sparse set of candidate points, enabling diverse coarse shape completions that propagate to dense completions through upsampling layers. The style codes used in modulating partial shape information are learned from an object category's complete shapes via a style encoder. Finally, we introduce discriminators and diversity penalties at multiple scales to train our model to produce diverse high-quality completions without having access to multiple ground truth completions that correspond to a partial observation.

3.1 Partial shape encoder

The goal of our partial encoder is to extract shape information in a local-to-global fashion, extracting local information that will be needed in the decoding stage to reconstruct fine geometric structures, while capturing global information needed to make sure a globally coherent shape is generated. An overview of the architecture for our proposed partial encoder is shown in Figure 3a.

Our partial encoder takes in a partially observed point cloud X P and first applies a MLP to obtain a set of point-wise features F 0 . To extract shape information in a local-to-global fashion, L consecutive downsampling blocks are applied to obtain a set of downsampled points X L with local features F L . In each downsampling block, a grid downsampling operation is performed followed by a series of PointConv [53] layers for feature interpolation and aggregation. Following the downsampling blocks, a global representation of the partial shape is additionally extracted by an MLP followed by max pooling, producing partial latent vector f P .

3.2 Style encoder

To produce multi-modal completions, we need to introduce randomness into our completion model. One approach is to draw noise from a Gaussian distribution and combine it with partial features during the decoding phase. Another option is to follow StyleGAN [46, 54] and transform noise samples through a non-linear mapping to a latent space W before injecting them into the partial features. However, these methods rely on implicitly learning a connection between latent samples and shape information. Instead, we propose to learn style codes from an object category's distribution of complete shapes and sample these codes to introduce stochasticity in our completion model. Our style codes explicitly carry information about ground truth shapes, leading to higher quality and more diverse completions.

To do so, we leverage the set of complete shapes we have access to during training. We introduce an encoder E that maps a complete shape X ∈ R N × 3 to a global latent vector via a 4-layer MLP followed by max pooling. We opt for a simple architecture as we would like the encoder to capture high level information about the distribution of shapes rather than fine-grained geometric structure.

Instead of assuming we have complete shapes to extract style codes from at inference time, we learn a distribution over style codes that we can sample from. Specifically, we define our style encoder as a learned Gaussian distribution E S ( z | X ) = N ( z | µ ( E ( X )) , σ ( E ( X ))) by adding two fully connected layers, µ and σ , to the encoder E to predict the mean and standard deviation of a Gaussian to sample a style code from. Since our aim is to learn style codes that convey information about complete shapes that is useful for generating diverse completions, we train our style encoder with guidance from our

Partial Encoder

Downsample

Block (x4)

MLP

MLP +

Downsampling Block

Downsample

+

+

PointConv

PointConv

PointConv

(a)

Conv

Mod

+

+

Conv

Mod

(b)

completion network's losses. To enable sampling during inference, we minimize the KL-divergence between E S ( z | X ) and a normal distribution during training. We additionally find that adding noise to our sampled style codes during training leads to higher fidelity and more diverse completions.

3.3 Style-based seed generator

We make use of the Patch Seeds representation proposed in SeedFormer [7], which enables faithfully completing unobserved regions while preserving partially observed structures. Patch Seeds are defined as a set of seed coordinates S ∈ R N S × 3 and seed features F ∈ R N S × C S produced by a seed generator. In particular, a set of upsampled features F up ∈ R N S × C S are generated from the partial local features ( X L , F L ) via an Upsample Transformer [7]. Seed coordinates S and features F are then produced from upsampled features F up concatenated with partial latent code f P via an MLP.

However, the described seed generator is deterministic, prohibiting the ability to generate diverse Patch Seeds that can then produce diverse completions through upsampling layers. We propose to incorporate stochasticity into the seed generator by injecting stochasticity into the partial latent vector f P . We introduce a style modulator network M ( f P , z ) , shown in Figure 3b, that injects a style code z into a partial latent vector f P to produce a styled partial shape latent vector f C . Following [54], we use weight modulation to inject style into the activation outputs of a network layer, where the demodulated weights w ′′ used in each convolution layer are computed as:

where A is an Affine transformation, and w is the original convolution weights with i, j, k corresponding to the input channel, output channel, and spatial footprint of the convolution, respectively.

3.4 Coarse-to-fine decoder

Our decoder operates in a coarse-to-fine fashion, which has been shown to be effective in producing shapes with fine geometric structure for the task of point cloud completion [36, 38, 7, 4]. We treat our Patch Seed coordinates S as our coarsest completion G 0 and progressively upsample by a factor r to produce denser completions G i ( i = 1 , ..., 3) . At each upsampling stage i , a set of seed features are first interpolated from the Patch Seeds. Seed features along with the previous layer's points and features are then used by an Upsample Transformer [7] where local self-attention is performed to produce a new set of points and features upsampled by a factor r . We replace the inverse distance weighted averaging used to interpolate seed features from Patch Seeds in SeedFormer with a PointConv interpolation. Since the importance of Patch Seed information may vary across the different layers, we believe a PointConv interpolation is more appropriate than a fixed weighted interpolation as it can learn the appropriate weighting of Patch Seed neighborhoods for each upsampling layer.

Unlike the fully-connected decoders in [8, 9], the coarse-to-fine decoder used in our method can reason about the structures of local regions, allowing us to generate cleaner shape surfaces. A coarseto-fine design provides us with the additional benefit of having discriminators and diversity penalties at multiple resolutions, which we have found to lead to better completion fidelity and diversity.

3.5 Multi-scale discriminator

During training, we employ adversarial training to assist in learning realistic completions for any partial input and style code combination. We introduce a set of discriminators D i for i = { 0 , ..., 3 } , to discriminate against real and fake point clouds at each output level of our generator. Each discriminator follows a PointNet-Mix architecture proposed by Wang et al. [55]. In particular, an MLP first extracts a set of point features from a shape, which are max-pooled and average-pooled to produce f max and f avg , respectively. The features are then concatenated to produce mix-pooled feature f mix = [ f max , f avg ] before being passed through a fully-connected network to produce a final score about whether the point cloud is real or fake.

We also explored more complex discriminators that made use of PointConv or attention mechanisms, but we were unable to successfully train with any such discriminator. This is in line with the findings in [55], suggesting that more powerful discriminators may not guide the learning of point cloud shape generation properly. Thus, we instead use a weaker discriminator architecture but have multiple of them that operate at different scales to discriminate shape information at various feature levels.

For training, we use the WGAN loss [56] with R1 gradient penalty [57] and average over the losses at each output level i . We let ˆ X i = G i ( X P , z ) be the completion output at level i for partial input X P and sampled style code z ∼ E S ( z | X ) , and let X i be a real point cloud of same resolution. Then our discriminator loss L D and generator loss L G are defined as:

where γ is a hyperparameter ( γ = 1 in our experiments), P ( ˆ X ) is the distribution of generated shapes, and P ( X ) is the distribution of real shapes.

3.6 Diversity regularization

Despite introducing stochasticity into the partial latent vector, it is still possible for the network to learn to ignore the style code z , leading to mode collapse to a single completion. To address this, we propose a diversity penalty that operates in the feature space of our discriminator. Our key insight is that for a discriminator to be able to properly discriminate between real and fake point clouds, its extracted features should have learned relevant structural information. Then our assumption is that if two completions are structurally different, the discriminator's global mix-pooled features should be dissimilar as well, which we try to enforce through our diversity penalty.

Specifically, at every training iteration we sample two style codes z 1 ∼ E S ( z | X 1 ) and z 2 ∼ E S ( z | X 2 ) from random complete shapes X 1 and X 2 . For a single partial input X P , we produce two different completions G i ( X P , z 1 ) and G i ( X P , z 2 ) . We treat our discriminator D i as a feature extractor and extract the mixed-pooled feature for both completions at every output level i . We denote the mixed-pooled feature corresponding to a completion conditioned on style code z at output level i by f z mix i , then minimize:

which encourages the generator to produce completions with dissimilar mix-pooled features for different style codes. Rather than directly using the discriminator's mix-pooled feature, we perform pooling only over the set of point features that are not in partially observed regions. This helps avoid penalizing a lack of diversity in the partially observed regions of our completions.

We additionally make use of a partial reconstruction loss at each output level on both completions:

where d UHD stands for the unidirectional Hausdorff distance from partial point cloud to completion. Such a loss helps ensure that our completions respect the partial input for any style code z .

To ensure that the completion set covers the ground truth completions in the training set, we choose to always set random complete shape X 1 = X GT and sample z 1 ∼ E S ( z | X GT ) , where X GT is the corresponding ground truth completion to the partial input X P . This allows us to provide supervision at the output of each upsampling layer via Chamfer Distance (CD) for one of our style codes:

Our full loss that we use in training our generator is then:

We set λ G = 1 , λ comp = 0 . 5 , λ part = 1 , λ div = 5 , which we found to be good default settings across the datasets used in our experiments.

4 Experiments

In this section, we evaluate our method against a variety of baselines on the task of multimodal shape completion and show superior quantitative and qualitative results across several synthetic and real datasets. We further conduct a series of ablations to justify the design choices of our method.

Implementation Details Our model takes in N P = 1024 points as partial input and produces N = 2048 points as a completion. For training the generator, the Adam optimizer is used with an initial learning rate of 1 × 10 -4 and the learning rate is linearly decayed every 2 epochs with a decay rate of 0 . 98 . For the discriminator, the Adam optimizer is used with a learning rate of 1 × 10 -4 . We train a separate model for each shape category and train each model for 300 epochs with a batch size of 56. All models are trained on two NVIDIA Tesla V100 GPUs and take about 30 hours to train.

Datasets We conduct experiments on several synthetic and real datasets. Following the setup of [8], we evaluate our approach on the Chair, Table, and Airplane categories of the 3D-EPN dataset [58]. Similarly, we also perform experiments on the Chair, Table, and Lamp categories from the PartNet dataset [59]. To evaluate our method on real scanned data, we conduct experiments on the Google Scanned Objects (GSO) dataset [60]. For GSO, we share quantitative and qualitative results on the Shoe, Toys, and Consumer Goods categories. A full description is presented in the supplementary.

Metrics We follow [8] and evaluate with the Minimal Matching Distance (MMD), Total Mutual Difference (TMD), and Unidirectional Hausdorff Distance (UHD) metrics. MMD measures the fidelity of the completion set with respect to the ground truth completions. TMD measures the completion diversity for a partial input shape. UHD measures the completion fidelity with respect to the partial input. We evaluate metrics on K = 10 generated completions per partial input. Reported MMD, TMD, and UHD values in our results are multiplied by 10 3 , 10 2 , and 10 2 , respectively.

Partial

KNN

cGAN

Ours

Partial

KNN

cGAN

Ours

Partial

KNN

cGAN

Ours

4

MA7

X

MMD

↓

TMD

↑

UHD

↓

Method

Chair

Lamp

Table

Chair

Lamp

Table

Chair

Lamp

Table

SeedFormer [7]

0.72

1.35

0.71

0.93

0.00

0.00

0.00

0.00

1.54

1.25

1.48

1.42

KNN-latent [8]

1.39

1.72

1.30

1.47

2.28

4.18

2.36

2.94

8.58

8.47

7.61

8.22

cGAN [8]

1.52

1.97

1.46

1.65

2.75

3.31

3.30

3.12

6.89

5.72

5.56

6.06

IMLE [9]

*

*

*

*

2.76

5.49

4.45

4.23

6.17

5.58

5.16

5.64

ShapeFormer [10]

*

*

*

1.32

*

*

*

3.96

*

*

*

*

Ours

1.50

1.84

1.15

1.49

4.36

6.55

5.11

5.34

3.79

3.88

3.69

3.79

PVD [10]

†

1.27

1.03

1.98

1.43

1.91

1.70

5.92

3.18

*

*

*

*

Ours

†

1.34

1.55

1.12

1.34

5.27

7.11

5.84

6.07

*

*

*

*

Partial

KNN

cGAN

PVD

Ours

Partial

KNN

cGAN

PVD

Ours

Partial

KNN

cGAN

PVD

Ours

Baselines We compare our model against three direct multi-modal shape completion methods: cGAN [8], IMLE [9], and KNN-latent which is a baseline proposed in [8]. We further compare with the diffusion-based method PVD [10] and the auto-regressive method ShapeFormer [13]. We also share quantitative results against the deterministic point cloud completion method SeedFormer [7].

4.1 Results

Results on the 3D-EPN dataset are shown in Table 1. SeedFormer obtains a low UHD implying that their completions respect the partial input well; however, their method produces no diversity as it is deterministic. Our UHD is significantly better than all multi-modal completion baselines, suggesting that we more faithfully respect the partial input. Additionally, our method outperforms others in terms of TMD and MMD, indicating better diversity and completion quality. This is also reflected in the qualitative results shown in Figure 4, where KNN-latent fails to produce plausible completions, while completions from cGAN contain high levels of noise.

In Table 2, we compare against other methods on the PartNet dataset. For a fair comparison with PVD, we also report metrics following their protocol (denoted by † )[10]. In particular, under their protocol TMD is computed on a subsampled set of 1024 points and MMD is computed on the subsampled set concatenated with the original partial input. Once again our method obtains the best diversity (TMD) across all categories, beating out the diffusion-based method PVD and the auto-regressive method ShapeFormer. Our method also achieves significantly lower UHD and shows competitive performance in terms of MMD. Some qualitative results are shown in Figure 5. We find that our method produces cleaner surface geometry and obtains nice diversity in comparison to other methods.

Partial

cGAN

PVD

Ours

Partial

cGAN

PVD

Ours

112

Additionally, we compare our method on real data from the Google Scanned Objects dataset. In Table 3, we show that our method obtains better performance across all the metrics for all three categories. We present a qualitative comparison of completions on objects from the Google Scanned Objects dataset in Figure 6. Completions by cGAN [8] are noisy and lack diversity, while completions from PVD [10] have little diversity and suffer from non-uniform density. Alternatively, our method produces cleaner and more diverse completions with more uniform density.

We further demonstrate the ability of our method to produce diverse high-quality shape completions in Figure 7. Even with varying levels of ambiguity in the partial scans, our method can produce plausible multi-modal completions of objects. In particular, we find that under high levels of ambiguity, such as in the lamp or shoe examples, our method produces more diverse completions. On the other hand, when the object is mostly observed, such as in the plane example, our completions exhibit less variation among them. For more qualitative results, we refer readers to our supplemental.

Finally, we find that our method is capable of inference in near real-time speeds. To produce K = 10 completions of a partial input on a NVIDIA V100, our method takes an average of 85 ms while cGAN and KNN-latent require 5 ms. PVD requires 45 seconds which is 500 times slower than us.

4.2 Ablation studies

In Table 4 we examine the dimensionality of our style codes. Our method obtains higher TMD when using smaller style code dimension size. We additionally find that adding a small amount of noise to sampled style codes during training further helps boost TMD while improving UHD. We believe that reducing the style code dimension and adding a small amount of noise helps prevent our style codes from encoding too much information about the ground truth shape, which could lead to overfitting to the ground truth completion. Furthermore, in Table 5, we present results with different choices for style code generation. Our proposed style encoder improves diversity over sampling style codes from a normal distribution or by using the mapping network from StyleGAN [46]. Despite having slightly worse MMD and UHD than StyleGAN's mapping network, we find the quality of completions at test time to be better when training with style codes sampled from our style encoder (see supplementary).

In our method, we made several changes to the completion network in SeedFormer [7]. We replaced the encoder from SeedFormer, which consisted of point transformer [61] and PointNet++ [62] set abstraction layers, with our proposed partial encoder as well as replaced the inverse distance weighted interpolation with PointConv interpolation in the SeedFormer decoder. To justify these changes, we compare the different architectures in our GAN framework in Table 6. We compare the performance of the original SeedFormer encoder and decoder (SF), our proposed partial encoder and SeedFormer decoder (PE + SF), and our full architecture where we replace inverse distance weighted interpolation with PointConv interpolation in the decoder (PE + SF + PCI). Our proposed partial encoder produces an improvement in TMD for slightly worse completion fidelity. Further, we find using PointConv interpolation provides an additional boost in diversity while improving completion fidelity.

The importance of our multi-scale discriminator is shown in Table 7. Using a single discriminator/diversity penalty only at the final output resolution results in a drop in completion quality and diversity when compared with our multi-scale design.

Finally, we demonstrate the necessity of our loss functions in Table 8. Without L comp , our method has to rely on the discriminator alone for encouraging sharp completions in the missing regions. This leads to a drop in completion quality (MMD). Without L part , completions fail to respect the partial input, leading to poor UHD. With the removal of either of these losses, we do observe an increase in TMD; however, this is most likely due to the noise introduced by the worse completion quality. Without L div , we observe TMD drastically decreases towards zero, suggesting no diversity in the completions. This difference suggests how crucial our diversity penalty is for preventing conditional mode collapse. Moreover, we observe that when using all three losses, our method is able to obtain good completion quality, faithfully reconstruct the partial input, and produce diverse completions.

5 Conclusion

In this paper, we present a novel conditional GAN framework that learns a one-to-many mapping between partial point clouds and complete point clouds. To account for the inherent uncertainty present in the task of shape completion, our proposed style encoder and style-based seed generator enable diverse shape completion, with our multi-scale discriminator and diversity regularization preventing mode collapse in the completions. Through extensive experiments on both synthetic and real datasets, we demonstrate that our multi-modal completion algorithm obtains superior performance over current state-of-the-art approaches in both fidelity to the input partial point clouds and completion diversity. Additionally, our method runs in near real-time speed, making it suitable for applications in robotics such as planning or active perception.

While our method is capable of producing diverse completions, it considers only a segmented point cloud on the object. A promising future research direction is to explore how to incorporate additional scene constraints (e.g. ground plane and other obstacles) into our multi-modal completion framework.

Acknowledgments and Disclosure of Funding

This work was supported in part by ONR award N0014-21-1-2052, ONR/NAVSEA contract N0002410-D-6318/DO#N0002420F8705 (Task 2: Fundamental Research in Autonomous Subsea Robotic Manipulation), NSF grant #1751412, and DARPA contract N66001-19-2-4035.

References

Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pcn: Point completion network. In 2018 international conference on 3D vision (3DV) , pages 728-737. IEEE, 2018.

Lyne P Tchapmi, Vineet Kosaraju, Hamid Rezatofighi, Ian Reid, and Silvio Savarese. Topnet: Structural point cloud decoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 383-392, 2019.

Minghua Liu, Lu Sheng, Sheng Yang, Jing Shao, and Shi-Min Hu. Morphing and sampling network for dense point cloud completion. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 11596-11603, 2020.

Xiaogang Wang, Marcelo H Ang Jr, and Gim Hee Lee. Cascaded refinement network for point cloud completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 790-799, 2020.

Haozhe Xie, Hongxun Yao, Shangchen Zhou, Jiageng Mao, Shengping Zhang, and Wenxiu Sun. Grnet: Gridding residual network for dense point cloud completion. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX , pages 365-381. Springer, 2020.

Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-aware transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 12498-12507, 2021.

Haoran Zhou, Yun Cao, Wenqing Chu, Junwei Zhu, Tong Lu, Ying Tai, and Chengjie Wang. Seedformer: Patch seeds based point cloud completion with upsample transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part III , pages 416-432. Springer, 2022.

Rundi Wu, Xuelin Chen, Yixin Zhuang, and Baoquan Chen. Multimodal shape completion via conditional generative adversarial networks. In The European Conference on Computer Vision (ECCV) , August 2020.

Himanshu Arora, Saurabh Mishra, Shichong Peng, Ke Li, and Ali Mahdavi-Amiri. Multimodal shape completion via implicit maximum likelihood estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pages 2958-2967, June 2022.

Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 5826-5835, October 2021.

Gene Chou, Yuval Bahat, and Felix Heide. Diffusion-sdf: Conditional generative modeling of signed distance functions. In The IEEE International Conference on Computer Vision (ICCV) , 2023.

Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tuyakov, Alex Schwing, and Liangyan Gui. SDFusion: Multimodal 3d shape completion, reconstruction, and generation. arXiv , 2022.

Xingguang Yan, Liqiang Lin, Niloy J. Mitra, Dani Lischinski, Danny Cohen-Or, and Hui Huang. Shapeformer: Transformer-based shape completion via sparse representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. AutoSDF: Shape priors for 3d completion, reconstruction and generation. In CVPR , 2022.

Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning , pages 40-49. PMLR, 2018.

Chun-Liang Li, Manzil Zaheer, Yang Zhang, Barnabas Poczos, and Ruslan Salakhutdinov. Point cloud gan. arXiv preprint arXiv:1810.05795 , 2018.

Dong Wook Shu, Sung Woo Park, and Junseok Kwon. 3d point cloud generative adversarial network based on tree structured graph convolutions. In Proceedings of the IEEE/CVF international conference on computer vision , pages 3859-3868, 2019.

Ruihui Li, Xianzhi Li, Ka-Hei Hui, and Chi-Wing Fu. Sp-gan: Sphere-guided 3d shape generation and manipulation. ACM Transactions on Graphics (TOG) , 40(4):1-12, 2021.

Yingzhi Tang, Yue Qian, Qijian Zhang, Yiming Zeng, Junhui Hou, and Xuefei Zhe. Warpinggan: Warping multiple uniform priors for adversarial 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6397-6405, 2022.

Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5939-5948, 2019.

Marian Kleineberg, Matthias Fey, and Frank Weichert. Adversarial generation of continuous implicit shape representations. arXiv preprint arXiv:2002.00349 , 2020.

Moritz Ibing, Isaak Lim, and Leif Kobbelt. 3d shape generation with grid-based implicit functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1355913568, 2021.

X Zheng, Yang Liu, P Wang, and Xin Tong. Sdf-stylegan: Implicit sdf-based stylegan for 3d shape generation. In Computer Graphics Forum , volume 41, pages 52-63. Wiley Online Library, 2022.

Jinwoo Kim, Jaehoon Yoo, Juho Lee, and Seunghoon Hong. Setvae: Learning hierarchical composition for generative modeling of set-structured data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15059-15068, 2021.

Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:1511.05644 , 2015.

Matheus Gadelha, Rui Wang, and Subhransu Maji. Multiresolution tree networks for 3d point cloud processing. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 103-118, 2018.

Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4541-4550, 2019.

Roman Klokov, Edmond Boyer, and Jakob Verbeek. Discrete point flow networks for efficient point cloud generation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16 , pages 694-710. Springer, 2020.

Janis Postels, Mengya Liu, Riccardo Spezialetti, Luc Van Gool, and Federico Tombari. Go with the flows: Mixtures of normalizing flows for point cloud generation and reconstruction. In 2021 International Conference on 3D Vision (3DV) , pages 1249-1258. IEEE, 2021.

Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2837-2845, 2021.

Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint arXiv:2210.06978 , 2022.

Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero. 3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint arXiv:2212.00842 , 2022.

J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. arXiv preprint arXiv:2211.16677 , 2022.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 652-660, 2017.

Xin Wen, Tianyang Li, Zhizhong Han, and Yu-Shen Liu. Point cloud completion by skip-attention network with hierarchical folding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1939-1948, 2020.

Zitian Huang, Yikuan Yu, Jiawen Xu, Feng Ni, and Xinyi Le. Pf-net: Point fractal network for 3d point cloud completion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7662-7670, 2020.

Xin Wen, Peng Xiang, Zhizhong Han, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Yu-Shen Liu. Pmp-net: Point cloud completion by learning multi-step point moving paths. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7443-7452, 2021.

Peng Xiang, Xin Wen, Yu-Shen Liu, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Zhizhong Han. Snowflakenet: Point cloud completion by snowflake point deconvolution with skip-transformer. In Proceedings of the IEEE/CVF international conference on computer vision , pages 5499-5509, 2021.

Xin Wen, Zhizhong Han, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Yu-Shen Liu. Cycle4completion: Unpaired point cloud completion using cycle transformation with missing region coding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13080-13089, 2021.

Zhen Cao, Wenxiao Zhang, Xin Wen, Zhen Dong, Yu-shen Liu, Xiongwu Xiao, and Bisheng Yang. Ktnet: Knowledge transfer for unpaired 3d shape completion. arXiv e-prints , pages arXiv-2111, 2021.

Xuelin Chen, Baoquan Chen, and Niloy J Mitra. Unpaired point cloud completion on real scans using adversarial training. In Proceedings of the International Conference on Learning Representations (ICLR) , 2020.

Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, and Chen Change Loy. Unsupervised 3d shape completion through gan inversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1768-1777, 2021.

Chulin Xie, Chuxin Wang, Bo Zhang, Hao Yang, Dong Chen, and Fang Wen. Style-based point generator with adversarial rendering for point cloud completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4619-4628, 2021.

Ting Chen, Mario Lucic, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adversarial networks. arXiv preprint arXiv:1810.01365 , 2018.

Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatiallyadaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2337-2346, 2019.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Ari Heljakka, Yuxin Hou, Juho Kannala, and Arno Solin. Deep automodulators. Advances in Neural Information Processing Systems , 33:13702-13713, 2020.

Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. arXiv preprint arXiv:2103.10428 , 2021.

Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. Advances in neural information processing systems , 30, 2017.

Augustus Odena, Jacob Buckman, Catherine Olsson, Tom Brown, Christopher Olah, Colin Raffel, and Ian Goodfellow. Is generator conditioning causally related to gan performance? In International conference on machine learning , pages 3849-3858. PMLR, 2018.

Dingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, and Honglak Lee. Diversity-sensitive conditional generative adversarial networks. arXiv preprint arXiv:1901.09024 , 2019.

Qi Mao, Hsin-Ying Lee, Hung-Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking generative adversarial networks for diverse image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1429-1437, 2019.

Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages 9621-9630, 2019.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8110-8119, 2020.

He Wang, Zetian Jiang, Li Yi, Kaichun Mo, Hao Su, and Leonidas Guibas. Rethinking Sampling in 3D Point Cloud Generative Adversarial Networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop Report , 2021.

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning , pages 214-223. PMLR, 2017.

Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning , pages 3481-3490. PMLR, 2018.

Angela Dai, Charles Ruizhongtai Qi, and Matthias Nießner. Shape completion using 3d-encoder-predictor cnns and shape synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5868-5877, 2017.

Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA) , pages 2553-2560. IEEE, 2022.

Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision , pages 16259-16268, 2021.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems , 30, 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new conditional generative network that can produce diverse completions of a partially observed point cloud. The stochasticity is introduced via style modulation. A style code is learned to explicitly carry shape category information leading to better completions. Moreover, diversity penalties and discriminators at multiple scales are also set to prevent conditional mode collapse. Experiments show that the proposed framework can achieve significant improvements in respecting the partial observations while obtaining greater diversity in completions.\n\nSTRENGTHS:\nThe results on different datasets show the effectiveness of the proposed method.\n\nWEAKNESSES:\n1.\tThe main target of this paper is to achieve diverse synthesis. However, no such visual samples are provided. The authors should provide the diverse synthesis for different categories in the main paper.\n\n2.\tHow to decide the optimal values for loss weights in Eq. 7? Although the authors claim that the same loss weight setting can lead to good results on all experimental datasets, the experimental datasets only contain several categories. More categories should be considered into analysis.\n\n3.\tThe idea of this paper is derived from the 2D StyleGAN. The difference about the idea should be clearly indicated.\n\n4.\tThe method has not compared with SOTA methods that use the diffusion model, like [30, 31]. And this method should also support unconditional point cloud synthesis?\n\nQUESTIONS:\nPlease answer the questions in the weakness section.\nMoreover, can this method be utilized into the point cloud completion of scenes? Like ScanNet dataset.\n\nLIMITATIONS:\nFurther considering point cloud completion with text conditions can help to achieve open-world and diverse synthesis effects, which can improve the contribution of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to reconstruct partial point cloud inputs using a multi-modal process, where the generator can output multiple plausible shapes. The key idea is to have a separate network (StyleEncoder) that extracts style from an input, in addition to a separate network (PartialEncoder) that extracts structural information, that are then combined to generate the final reconstruction. Another contribution is a multi-scale discriminator that checks real and fake pairs at multiple generator output stages. The entire pipeline is essentially a conditional GAN network for generating 3D shapes.\n\nSTRENGTHS:\nThe paper proposes to address an important problem where there could be multiple potential reconstructions given a partial input. Limitations of recent work (SeedFormer [7]) are addressed with a clever network design strategy. The method is evaluated on three public datasets (3D-EPN, GSO, and Part-Net) with promising quantitative results.\n\nWEAKNESSES:\nThe methodology section is difficult to follow, and I found it hard to understand the training process. In addition, it seems that the proposed technique is heavily based on previous work (e.g., SegFormer). It may help to explain the key technical contributions with respect to these previous works and defer the reader to the other papers for understanding technical details. \n\nIn addition, the validation of the proposed technique is rather limited. The training procedure, and validation steps are not clearly described. Is the method sensitive to shape orientation? What type of data augmentation is used during training, and is it trained on a per-category level (thus learning certain shape priors) or category-agonostic level? \n\nI would have expected to see ablation experiments with same methodology but different network architecture. Finally, the authors evaluate on PartNet, yet, never quantify part level accuracy.\n\nIn Tables 5 and 6, ablation studies of the style code and discriminator have very similar MMD metric but large UHD metric. I did not understand the choice and motivation behind these metrics. Why not use standard Hausdorff distance instead?\n\nFinally, the method is geared toward generative multiple outputs given a single input, yet this capability is never evaluated. I would have expected to see additional examples where multiple plausible reconstructions an input can have, followed by a user study or a qualitative evaluation of the results.\n\nQUESTIONS:\n1.\tOn line 256, it is stated the input contains 1024 points and the output contains 2048 points. Why is there a mismatch in these numbers? \n2.\tIf the input is meant to be a partial point cloud, have the authors considered the effect of point density?  \n3.\tAre selected datasets benchmark datasets for partial shape reconstruction?\n4.\tIs the goal of the multi-scale discriminator to address partial input (lack of ground truth for each structure/style pair, or is it to address structure at different scales)? Given the name and design, the naming and purpose is confusing.\n\nLIMITATIONS:\nThe paper proposed clever modifications on an existing network (SegFormer) to allow it to generate multiple completions given a single partial input. The idea is general, but it does not appear that the method is well-validated outside of the specific architecture considered, and thus, its impact is not clear.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
The paper proposes a diverse shape completion method by extracting style codes from complete shapes and learning a distribution over them. Moreover, diversity penalties and discriminators at multiple scales are introduced as well to prevent conditional modal collapse to generate various object shapes. To verify the effectiveness of the method, various experiments are conducted, and promising results are observed.

STRENGTHS:
1. The paper is well written and organized.
2. The methods are interesting and promising results are obtained.

WEAKNESSES:
1. According to line 156, “style codes” carrying category information are learnt from the distribution of complete shapes, however, the goal is to generate diverse shapes not only among different classes but also within the same class. Hence, how to get the diverse information for a single class seems missing in the paper. 
2. What is the difference of style encoder technique compared to VAE? According to the illustration in Sec. 3.2, it looks the same to VAE, which potentially results in limited novelty of the proposed style encoder. 
3. It would be better to also show some ablation studies on the network architectures by removing specific components to see how different parts contribute to the overall performances, which could be done by settings specific loss's weight to 0.

QUESTIONS:
None.

LIMITATIONS:
Some limitations are discussed in detail. To make a more comprehensive comparison with other SOTA works, possible memory consumption or time efficiency could be illustrated.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
The goal of multimodal shape completion is to generate many different plausible completions of an incomplete shape. Based on the conditional GAN, this paper introduces two key concepts to improve the diversity and accuracy of multimodal completion. One is to use style codes instead of random noise, which means better distribution of complete shapes. The other is the use of multiscale discriminators to refine the predicted shapes from coarse to fine. The method is evaluated on both synthetic and real data sets and shows significant improvements over comparable methods based on standard metrics.

STRENGTHS:
1. The results of multimodal completion look very clean and diverse. The numerical results show a new SOTA.

2. The use of the learned style codes instead of Gaussian noise is reasonable since the former is more similar to the distribution of complete shapes.

WEAKNESSES:
1. Ablation study with and without $\mathcal L_{comp}$ and $\mathcal L_{div}$ is not provided.

2. The flat and thin structures of the shapes generated by the proposed method look good, while other methods are more noisy. For the task of multimodal completion, this is quite impressive.
According to Figure 5 in the supplementary file, the main reason is probably the use of multiscale discriminators, since the results of a single scale are noisy. Is this true? And why are 4 scales used? If more than 4 scales are used, is the quality continuously improved?

3. The distribution of style codes should better reflect the actual shape distribution. To better verify this, a visual analysis comparing the distribution of the different conditional codes is helpful.

QUESTIONS:
1. Is this method applicable to unseen categories?

LIMITATIONS:
The limitation is well discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a novel GAN for diverse shape completion from partial point clouds. To enable diverse completions, a style-based generator is introduced that leverages style codes from a learned distribution of complete shapes for style modulation. Further, a multi-scale discriminator and a diversity penalty are proposed for better diversity.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. The experimental evaluation is thorough and supports the main claims in the paper. Both quantitative and qualitative results demonstrate the efficiency of the proposed approach over existing works.\n\nWEAKNESSES:\nWhile the supplementary includes an ablation with different diversity penalties, it should also contain an experiment without the diversity penalty for completeness. As the diversity penalty is one of the contributions of this work, I would consider moving the study to the main paper. \n\nIt would further be interesting to report the nearest neighbors in the training set for the completed regions in the results because the style encoder learns a distribution over the completed shapes.\n\nA discussion of failure cases should be added to the supplementary.\n\nFor completeness, the missing values in Table 3 for “Ours \\dagger” should be added.\n\nQUESTIONS:\nPlease also see Weaknesses.\n\nHow can diffusion-based methods, e.g. LION, be extended for point cloud completion? Would it be possible to train a variant that was conditioned on the partial point cloud (i.e. similar to 2D image inpainting with diffusion models) and report this as a further baseline?\nWhile diffusion models are limited wrt to inference speed, it would be interesting to include them as a baseline due to their strong generative performance.\n\nLIMITATIONS:\nThe limitations were discussed but a broader impact session should be added.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}