A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion
Authors: Zhaoyang Lyu
Zhifeng Kong
Xudong XU
Liang Pan
Dahua Lin
Venue: ICLR 2022 Poster (2022)
Keywords: Point Cloud Completion
Denoising Diffusion Pobabilistic Model
Conditional Generation

Abstract: 3D point clouds are an important data format that captures 3D information for real world objects.  Since 3D point clouds scanned in the real world are often incomplete, it is important to recover the complete point cloud for many downstreaming applications. Most existing point cloud completion methods use the Chamfer Distance (CD) loss for training. The CD loss estimates correspondences between two point clouds by searching nearest neighbors, which does not capture the overall point distribution on the generated shape, and therefore likely leads to non-uniform point cloud generation. To tackle this problem, we propose a novel Point Diffusion-Refinement (PDR) paradigm for point cloud completion. PDR consists of a Conditional Generation Network (CGNet) and a ReFinement Network (RFNet). The CGNet uses a conditional generative model called the denoising diffusion probabilistic model (DDPM) to generate a coarse completion conditioned on the partial observation. DDPM establishes a one-to-one pointwise mapping between the generated point cloud and the uniform ground truth, and then optimizes the mean squared error loss to realize uniform generation. The RFNet refines the coarse output of the CGNet and further improves quality of the completed point cloud.  In terms of the architecture, we develop a novel dual-path architecture for both networks. The architecture can (1) effectively and efficiently extract multi-level features from partially observed point clouds to guide completion, and (2) accurately manipulate spatial locations of 3D points to obtain smooth surfaces and sharp details. Extensive experimental results on various benchmark datasets show that our PDR paradigm outperforms previous state-of-the-art methods for point cloud completion. In addition, with the help of the RFNet,  we can accelerate the iterative generation process of the DDPM by up to 50 times without much performance drop.

Published as a conference paper at ICLR 2022
A CONDITIONAL POINT CIFFUSION-REFINERR PARADIGM FOR 3D POINT CLOUD COMPLETION
Zhaoyang Lyu Zhifeng Kong? Xudong Xu' Liang Pan
Dahua Lin
"CUHK-Senseinng Time Joint Lab, The Chinese University of Hong Kong Shanghai Al Laboratory
University of California, San Diego
LS-Lab Nanyang Technological University
lyuzhaoyange inkk cuhk edu. hk, z4kongBeng. wusddedde xx0180ie fethk.edu.hh woangppn@nnt...mmm edu. sg, dhlinβie An cuhk. edu.
ABSTRACT
3D point cloud is an important 3D representation for capturing real world 3D ob- jects. However, real-scanned 3D point clouds often incomplete, and is im- portant to recover complete point clouds for downstream applications. Most exist- ing point cloud completion methods Chamfer Distance (CD) loss for training. The CD loss estimates correspondences between two point clouds by searching nearest neighbors, which does not capture the overall point density distribution on the generated shape, and therefore likely leads to non-uni non-uniform point cloud gener- ation. To tackle this problem, we propose a novel Point Iiffusion-Refinemn (PDR) paradigm for point cloud completion. PDR consists of a Conditional Gen- eration Network (CGNet) and ReF'inement Network (RFNet). The CGNet uses conditional generative model called the denoising diffusion probabilistic model (DDPM) to generate coarse completion conditioned on the partial observation. DDPM establishes a one-to-one pointwise mapping between the generated point cloud and the uniform ground truth, and then optimizes the mean squared error loss realize uniform generation. The RFNet refines the coarse output of the CGNet and further improves quality of the completed point cloud. Furthermore, we develop a novel dual-path architecture for both networks. The architecture can (1) effectively and efficiently extract multi-level features from partially observed point clouds to guide completion, and (2) accurately manipulate spatial locations of 3D points to obtain smooth surfaces and sharp details. Extensive experimental results on various benchmark datasets show that our PDR paradigm outperforms previous of-e-offthheart methods for point cloud completion. Remarkably, with the help of the RFNet, we can accelerate the iterative generation process of the DDPM by up to 50 times without much performance drop.
INTRODUCTION
With the rapid developments of 3D sensors, 3D point clouds are an important data format that cap- tures 3D information owing to their ease of acquisition and efficiency in storage. Unfortunately, point clouds scanned in the real world are often incomplete due to partial observation and self oc- clusion. It is important to recover the complete shape by inferring the missing parts for many down- stream tasks such as 3D reconstruction, augmented reality and scene understanding. To tackle this problem, many learning-based methods (Yuan et al., 2018; 8; Yang et al., 2018; Tchapmi al., 2019; Xie et al., 2020; Liu et al., 2020; Pan et al., 2021) are proposed, which are supervised by using either the Chamfer Distance (CD) or Earth Mover Distance (EMD) to penalize the discrepancies between the generated complete point cloud and the ground truth. However, CD loss is not sensitive to overall density distribution, and thus networks trained by CD loss could generate non-uniform point cloud completion results (See Figure 10 and 11 in Appendix). EMD is more distinctive to measure density distributions, but too expensive to compute in training. The absence of an effective and efficient training loss highly limits the capabilities of many existing point cloud completion networks.
* Equul Contribution.
Code is released at
in
Di f fuusi on_ooo____eeeeee Rement.

---

Published as a conference paper at ICLR 2022
Geussian Noise
Stepa
Coerse Shape
Rafined Shape
Ground Truth
Conditional Goneration Network (DDPM]
Refinement Networt
Conditionr: Incomplete Point Cloud
Our Conditional Point Refiimmeet (PDR) paradigm first moves Gaussian noise step by step towards a coarse completion of the partial observation through a diffusion model (DDPM). Then it refines the coarse point cloud by one step to obtain high quality point cloud.
Figure 1:
We find that denoising diffusion probabilistic models (DDPM) (Sohl- -Sohl-Dickstein et al., 2015; Ho et al., 2020) can potentially generate uniform and high quality point clouds with an effective and efficient loss function. It can iteratively move clean point cloud. DDPM defines a one-to-one pointwise mapping between two consecutive point clouds in the diffusion process, which enables it use simple mean squared error loss function for training. This loss function is efficient to compute and explicitly requires the generated point cloud to be uniform, as one-to-one point mapping is naturally established between the generated point cloud and the ground truth. Point cloud completion task can be treated as a conditional generation problem the framework of DDPM (Zhou et al., 2021; Luo & Hu, 2021). Indeed, we find the complete point clouds generated by a conditional DDPM often have good overall distribution that uniformly covers the shape of the object.
of Gaussian noise towards complete and
Nonetheless, due to the probabilistic nature of DDPM and the lack of a suitable network architecture to train the conditional DDPM for 3D point cloud completion in previous works, we find DDPM completed point clouds often lack smooth surfaces and sharp details (See Figure/ I and Appendix Figure 12), which is also reflected by their high CD loss compared with state-of-the-art point cloud completion methods in our experiments. Another problem with DDPM is its inefficiency in the inference phase. It usually takes several hundreds and even up to one thousand forward steps to generate a single point cloud. Several methods (Song al., 2020; Nichol & Dhariwal, 2021; Kong & Ping. 2021) are proposed to accelerate DDPM using jumping steps without retraining networks, which however, leads to an obvious performance drop when using a small number of diffusion steps.
In this work, we propose the Conditional Point Diffusion-Refinemeen Refinement (PDR) paradigm to generate both uniform and high quality complete point clouds. As shown in Figure 1, our PDR paradigm performs point cloud completion in coarse-to-fine fashion. Firstly, we use the Conditional Gen- eration Network (CGNet) to generate a coarse complete point cloud by the DDPM conditioned on the partial point cloud. It iteratively moves set of Gaussian noise towards a complete point cloud. Following, the ReFinement Network (RFNet) further refines the coarse complete point cloud gen- erated from the Conditional Generation Network with the help of partial point clouds. In addition, RFNet can be used to refine the low quality point clouds generated by an accelerated DDPM, so that we could enjoy an acceleration up to 50 times, while minimizing the performance drop. In this way, the completion results generated by our PDR paradigm demonstrate both good overall density distribution (i.e. uniform) and sharp local details.
Both CGNet and RFNet have a novel dual-path network architecture shown in Figure 2, which is composed of two parallel sub-networks, Denoise subnet and Condition Feature Extraction subnet for noisy point clouds and partial point clouds, respectively. Specifically, we propose Point Adaptive Deconvolution (PA-Deconv) operation for upsampling, which can effectively manipulate spatial locations of 3D points. Furthermore, we propose the Feature Transfer (FT) module to directly transmit encoded point features at different scales from the Condition Feature Extraction subnet to the corresponding hierarchy in the Denoise subnet. Extensive experimental results show that our PDR paradigm can provide new state-of-the- performance for point cloud completion.
Our Key contributions can be summarized as: 1) We identify conditional DDPM to be good model with an effective and efficient loss function to generate uniform point clouds point cloud comple- tion task. 2) By using RFNet to refine the coarse point clouds, our PDR paradigm can generate com- plete point cloud with both good overall density distribution (i.e. uniform) and sharp local details. 3) We design novel point learning modules, including PA-Deconv and Feature Transfer modules, for constructing CGNet in DDPM and RFNet, which effectively and efficiently utilizes lelei- features extracted from incomplete point clouds for point cloud completion. 4) With the help of our proposed RFNet, we can accelerate the generation process of DDPM up to 50 times without a significant drop in point cloud quality.
2

---

Published as a conference paper at ICLR 2022
Connections
Condition Feature
Extraction Subnet
Condition
Subsampling
Pointict
PointNet
Subsempling
Poirthax
Pointet
Pl. Decome
Pointet
Pi- Deconv
Pointet
Point
Cloud
2-5rage Porntket to Extract obal Feature
Feature Transfer Modules
Transder
Modules
Skip Connections
Diffusion Step
Noisy
Subsempiin miniing
PointNet
PointNed
Suhssmpling
Pointtte
Sahsampling
Point
Pie Decome
Pain
De-eeamm
Pointet
PA- Desonv
Pointee
Shared MLP
Output
Point
Posisional Encodingg & FC Layers
Cloud x²
"o(x''... c,t)
Denoise
Subnet
FC layers
Figure 2: Network architecture of the Conditional Generation Network (CGNet) and ReFinement Network (RFNet). It consists of the Condition Feature Extraction subnet and the Denoise subnet.
PROBLEM STATEMENT
In this paper, we focus on the 3D point cloud completion task. A 3D point cloud represented by N points in the 3D space: X = {æ, 1 S S JN, where each zj E R is the 3D coordinates of the j-th point. We assume the dataset is composed of . data pairs {Xx, C.) S S M). where X; is the i-th ground-truth point cloud, and C; is the incomplete point cloud from a partial observation of X, The goal is develop a model that completes the partial observation C, and outputs point cloud as close to the ground truth X; possible. For algebraic convenience, we let E R&N be vector form point cloud X, and similarly e be the vector form of C.
3 METHODOLOGY
We consider the point cloud completion task as conditional generation problem, where the incom- plete point cloud C serves as the conditioner. We use the powerful generative model called de- noising diffusion probabilistic models (DDPM) (Sohl-Dickstein et al., 2015; Ho et al., 2020; Kong al., 2020) first generate coarse completion of the partial observation. Then we use another network to refine the coarse point cloud to improve its visual quality. Our point cloud completion pipeline is shown Figure I. We first briefly introduce the theory of DDPM in Section 3.1, and then describe detailed architecture of the Conditional Generation Network (CGNet) and ReFinement Network (RFNet) in Section 3.2 and Section 3.3.
3.1 BACKGROUND ON CONDITIONAL DENOISING DIFFUSION PROBABILISTIC MODELS
We assume Pdata to the distribution of the complete point cloud x, in the dataset, and Platent = (OON I3N to be the latent distribution, where / is the Gaussian distribution. Then, the con- ditional DDPM consists of two Markov chains called the diffusion process and the reverse process. Both processes have length equal to T. We set T 1000 in this paper.
The Diffusion Process. The diffusion process is Markov process that adds Gaussian noise into the clean data distribution Pdata until the output distribution is close to Platent The diffusion process irrelevant of the conditioner, the incomplete point cloud c, Formally, let æ Pdata- We use the superseript to denote the diffusion step t. For conciseness, we omit the subscription i in the following discussion. The diffusion process from clean data x. to xT is defined
T
g(x!
x
I g( xr 1'- where q(="|(x--)) N(n'; / B,x ..T)..
The hyperparameters 3r are pre-defined, small positive constants (See details in Appendix Sec- tion A. According to Ho et al. (2020), there is closed form expression for g( (x.). We first de- fine constants a: a₁ I 0g. Then, have q( a:") - (ar; votx" T))
3

---

Published as a conference paper at ICLR 2022
Therefore, when T is large enough, a goes to 0, and q<>=1=0) becomes close the latent distri- bution Platent (x') TT)) ). Note that x' can be directly sampled through the following equation:
x' = Va + vI- āGE, where E a standard Gaussian noise.
We emphasize that ((="|<=-~)) can be through the equation x² vi Bia,-1 + 3.6. Therefore, the order of points in x is preserved in the diffusion process. However, it does matter what kind of order we input the points in æ². That is because when T' is large enough, x² will become Gaussian distribution. Every point in Gaussian distribution equivalent and there no way distinguish one point from another.
one-to-one pointwise mapping as x' can be sampled
The Reverse Process. The reverse process is a Markov process that predicts and eliminates the noise added the diffusion process. The reverse process conditioned on the conditioner, the incomplete point cloud c. Let x Platent be latent variable. The reverse process from latent x! to clean data x is defined as
T
pe(x", ,
c)
pe " (x',c), c), where pe (a (x' = " =N(x'' ; . c,t),,,;1 II
(3)
The mean s4 (x",, C, () is a neural network parameterized by 0 and the variance of is time-step de- pendent constant. To generate sample conditioned c, we first sample xx ~ ~O(g,, I3Nx3N), then draw x² ~ pe I|",, c) for t = T.T - 1, ... 1. and finally outputs x".
Training. DDPM is trained via variational inference. Ho et al. (2020) introduced certain pa- rameterization for μo that can largely simplify the training objective. The parameterization is vT-a, co(',,c,, where co is a neural net-
1- Br, and (z C, = Vai x 3. of = 1-6, work taking noisy point cloud x² ~ q(>1) 220) in equation (2), diffusion step f, and conditioner e as inputs. Then, the simplified training objective becomes
L( = (0-- ||e
colVājo! + vI - 11-51,,, c,,
(4)
where ([M) is the uniform distribution over {1, 21, .... M). The neural network co learns to pre- dict the noise added to the clean point cloud xt which can be used denoise the noisy point cloud
x Va,a" + /I O,6. Note that traditional CD loss or EMD loss is NOT present in Equation 4. The reason that we are able to use the simple mean squared error because DDPM naturally defines a one-to-one pointwise mapping between two consecutive point clouds in the diffusion process as shown in Equation Note that each training step, we not only need to sample a pair of point clouds 23, Ct but also diffusion step i and a Gaussian noise 6.
3.2 CONDITIONAL GENERATION NETWORK
In this section, we introduce the architecture of Conditional Generation Network (CGNet) Eg. The inputs of this network the noisy point cloud æ the incomplete point cloud c, and the diffusion step f. We can intuitively interpret the output of cg as per-point difference between x' and " (with some arithmetic ignored). In addition, Eø should also effectively incorporate multi-level infor- mation from e. The goal is to infer not only the overall shape but also the fine-grained details based on e. We design a neural network that achieves these features. The overall architecture is shown in Figure 2. It is composed of two parallel sub-networks similar to PointNet++ (Qi et al., 2017b), and they have the same hierarchical structure.
The upper subnet, which we refer as the Condition Feature Extraction subnet, extracts multi-level features from the incomplete point cloud e. The lower subnet, which we refer as the Denoise subnet, takes the noisy point cloud x² as input. We also add the diffusion step f, the global feature extracted from c. and multi-level features extracted by the Condition Feature Extraction subnet to the Denoise subnet. The diffusion step l is first transformed into 512-dimension step embedding vector through positional encoding and fully connected (FC) layers (See Appendix Section A.1 for details), and then inserted to every level of the Denoise subnet. Similarly, the conditioner is first transformed into a 1024-length global feature through a two-stage PointNet, and then inserted to every level of the Denoise subnet. The multi-level features extracted by the Condition Feature Extraction subnet
4

---

Published as a conference paper at ICLR 2022
MLP
Incomplete Point Cloud
Diffusion Step
Embbedding
Mies (dl
512
IC Layer
d,
Shared
Repeat
Nias x K x d,
Refined Point
Mia
Add
Shared PC
Globel Feature
1024 FC Laver
dz
Miss x dl
Repeat
Add
Shered TC
M1a
Upsampled Point
Noss x K x dips
Noisy Point Cloud
Coarse Point
(b)
(c)
Figure 3: (a) Insert information of the diffusion step embedding and the global feature to the shared MLP. (b) The Feature Transfer module maps features from the incomplete point cloud to the noisy point cloud. (c) Refine and upsample the coarse points at the same time.
inserted to every level the Denoise subnet through Feature Transfer modules. Finally, the Denoise subnet is connected to shared MLP and outputs 6a (x',, ,,, ().
Additionally, while Zhou et al. (2021) argues PointNet+++ cannot be used in a DDPM that generates point clouds, we find attaching the absolute position of cach point to its feature solves this problem. See Appendix Section A.3 for detailed analysis. We also improve the backbone PointNet++ so that it manipulates positions of points more accurately.
In the next paragraphs, we elaborate on the building blocks of the improved PointNet++: Set Ab- straction modules in the encoder, and Feature Propagation modules in the decoder, and Feature Transfer modules between the Condition Feature Extraction subnet and the Denoise subnet.
Set Abstraction (SA) Module. Similar to PointNet++, this module subsamples the input point cloud and propagates the input features. Assume the input is {a,|1 S j Ni), where x;j is the 3D coordinate of the j-th point and N; is the number of input points to the Set Abstraction module of level 1. Each point has a feature of dimension d: We concatenate these features with their corresponding 3D coordinates and group them together to form a matrix Fi of shape N₁ x (di + 3). The SA module first uses iterative farthest point sampling (FPS) subsample the input points to N1+1 points: [2x[1 < & < Nt+1). Then it finds K neighbors in the input {25|1 < < Ni for each yk- We denote the K neighbors of y/s as {ıylū E (y(y))). where (yk) is the index set of the K neighbors. See definition of neighbors in Appendix A.2. These neighbors and their features are grouped together to form a matrix G.n of shape N111 K x (d) + +3). Then a shared multi- layer perceptron (MLP) is applied to transform the grouped feature G,n to Gout. which is of shape N441 x K x di+1 and d1+1 is the dimension of the output feature. Finally, a max moolliooing is applied to aggregate features from the K neighbors {x,15 (y())) to l/k- We obtain the output of the SA module, the matrix F+41, which is of shape N441 X di+1-
Note that need to incorporate information of the diffusion step embedding and global feature extracted from the incomplete point cloud c to every SA module in the Denoise subnet as shown in Figure 2. We insert these information to the shared MLP that transforms Gin to Gout mentioned in the above paragraph. Specifically, we add them to the channel dimension of the intermediate feature maps in the shared MLP. Figure 3(a) illustrates this process in details. Inspired by the works (Pan et al., 2021; Zhao et al., 2020), we also replace the max mrax-pooling layer in the SA module with a self-attention layer. Feature at yı obtained by weighted sum of the features of its K neighbors {ıylū E B, (y=)} instead of max-pooling, and the weights are adaptively computed through the attention mechanism. See Appendix A.4 for details of this attention layer.
Feature Propagation (FP) Module. Similar to PointNet++,, this module upsamples the input point cloud and propagates the input features. In PointNet++, the features are upsampled from { < k < N4+1} to {2, S j < Ni) by three interpolation: Feature at ε is a weighted sum of the features of its three nearest neighbors in {yx |1 < k < N.+1). We think that the three interpolation operation is not suitable in our task, because the interpolation operation may lose some information about the accurate positions of the points. See detailed analysis in Appendix Section A.5.
5

---

Published as a conference paper at ICLR 2022
We propose use Point Adaptive Deconvolution (PA-Deconv) module to upsample the point features. In the SA module, the features are mapped from set {z,[1 S S Mi to {3k S & S N441). The key step is to find the neighbors {ıyl} E (vx)) E {xy[1 S yk- Features at {æyli E (yx))) are transformed and then aggregated to the point yı through either need to map features the other way around: from {2x|1 S & < N++1 to { S S. Mr.. We can achieve this goal through a similar method. We find the neighbors {yu]k: Ik [x,)) S {yx S k S M++1 for each Ty. Features at [3a,lk E By(r)) are transformed through a shared MLP, and then aggregated to the point zy through attention mechanism. Similar to SA modules, we insert information of the diffusion step embedding and the global feature extracted from the incomplete point cloud e to the shared MLP in every FP module in the Denoise subnet. Finally, same as the original FP module PointNet++, the upsampled features are concatenated with skip linked point features from the corresponding SA module, and then passed through a unit PointNet. The Feature Propagation module are applied four times and features are eventually propagated to the original input point cloud.
Ni for each
max-pooling or attention mechanism. Now in the FP module,
Feature Transfer (FT) Module. The FT module transmits information from the Condition Fea- ture Extraction subnet to the Denoise subnet. Assume the point cloud at level I in the Condition
Feature Extraction subnet is { I [ < Si}. where St is the number of points at level in the Condition Feature Extraction subnet. The FT module maps the features at points { [1 < r < S to points at the same level in the Denoise subnet, which are {ayl1 S S Nr). Then the mapped features are concatenated with the original features at {2r/11 j S NI). Next, the concatenated features are fed to the next level of the Denoise subnet. In this way. the Denoise subnet can utilize local features at different levels of the incomplete point cloud manipulate the noisy input point cloud to form a clean and complete point cloud. The key step in this process is to map features at {zr11 S r S 31} to {2 S j < Ni). We adopt similar strategy in the SA module. We find the neighbors {2y|r E (Z,))) C {2,|1 < r S Si} for each Tj.. Features at { E (z,)) are transformed through a shared MI.P, and then aggregated to the point xy through the attention mechanism, which is a weighted sum of the features at (zrlr E (1rr))
We set a small distance to define neighbors in low level FT modules, so that they only query the adjacent parts of the incomplete point cloud c to preserve local details in it. Large distances are set to define neighbors in high level FT modules. This makes high-level FT modules have large receptive fields, so that they can query large part of the incomplete point cloud to infer high level 3D structural relations. See detailed neighbor definitions in Appendix Section A.2.
3.3 REFINEMENT NETWORK
We denote the coarse point cloud generated by the Conditional Generation Network as U. We use another network of the same architecture shown in Figure 2 to predict a per-point displacement for U to refine it. The differences to insert the diffusion step embedding to the Denoise subnet. The predicted displacement are added to U obtain the refined point cloud V: te c), where n, u, e are the concatenated 3D) u +
that the input to the Denoise subnet becomes U and we do not need
coordinates of the point clouds V,U. U, C, respectively. 7 is a small constant and we set it to 0.001 in all our experiments. Ey the ReFinement Network. We use the Chamfer Distance (CD) loss between the refined point cloud V and ground truth point cloud X to supervise the network Ey:
(V x
1
min l 2112 NEV
Σ
min ||v VVV X
1
Σ xE
==1??
where |V] means number of points in V. we we also want to upsample points in U by factor of X, we can simply increase the output dimension of the network ey. In addition to predicting one 3D displacement of each point in U. we predict another X displacements. We consider each point in the refined point cloud V as the center of a group of X points the dense point cloud that we want to generate. The additional À displacements are added to every point in V to form dense point cloud. Figure 3(c) illustrates how we upsample every point in V by a factor of X = 8.
When training the ReFinement Network e,, parameters in the Conditional Generation Network eg are fixed. It is not practical to generate coarse point clouds U on the fly in the training process of e,, because the generation process of DDPM is slow. Instead, we generate and save the coarse point clouds in advance. Due to the probabilistic nature of DDPM, we generate 10 coarse point clouds for each incomplete point cloud in the dataset to increase diversity of training data.

---

Published as a conference paper at ICLR 2022
4 RELATED WORKS
Point cloud completion. Inspired by the pioneering work, PointNet (Qi et al., 2017a), researchers focus on learning global feature embeddings from 3D point clouds for completion (Yuan et al., 2018; Tchapmi et al., 2019), which however cannot predict local and thin shape structures. To ad- dress these challenges, following research works (Pan, 2020; Xie et al., 2020; Zhang et al., 2020; Wen al., 2021; Yu al., 2021; Pan et al., 2021) exploit multi-scale local point features re- construct complete point clouds with fine-grained geometric details. Recently, PointTr (Yu et al., 2021) and VRCNet (Pan et al., 2021) provide impressive point cloud complction results with the help of attention- based operations. Nonetheless, as a challenging conditional generation problem, point cloud completion has not been fully resolved.
DDPM for point cloud generation. Luo & Hu (2021) are the first to use DDPM for unconditional point cloud generation. They use Pointwise-net generate point clouds, which is similar a 2-stage PointNet used for point cloud part segmentation. However, the Pointwise-net could only receive global feature. It can not leverage fine-grained local structures in the incomplete point cloud. Zhou et al. (2021) further use conditional DDPM for point cloud completion by training a point-voxel CNN (Liu et al., 2019), but the way they use the incomplete point cloud c is different from ours. They directly concatenate with the noisy input æ', and feed them to single point- voxel CNN. This may hurt performance of the network, because the concatenated point cloud is very likely to be non-uniform. In addition, x' is very different from for large t's due to the large noise magnitude in x." Feeding two point clouds of very different properties to a single network at once could be quite confusing for the network. The other major difference is that they do not refine or upsample the coarse point cloud generated by DDPM like we do.
EXPERIMENTS
5.1 DATASETS
We conduct point cloud completion experiments on the following three datasets. MVP. The MVP dataset (Pan et al., 2021) has 62400 training partial-complete point cloud pairs and 41600 testing pairs sampled from ShapeNet (Chang et al., 2015). Every partial point cloud has 2048 points. In particular, MVP dataset provides ground truth point clouds with different resolutions, including 2048, 4096, 8192, and 16384 points. MVP-40. The MVP-40 dataset (Pan et al., 2021) consists of 41600 training samples and 64168 testing samples from 40 categories in ModelNet40 (Wu et al., 2015). Its partial point clouds are sampled from complete point clouds with a pre-defined missing ratio, i.e., 50%, 25% and 12.5% missing. Both the partial and complete point clouds have 2048 points. Completion3D. It (Tchapmi et al., 2019) consists of 28974 point cloud pairs for training and 1184 for testing from 8 object categories in ShapeNet. Both the partial and complete point clouds have 2048 points. We find some pairs of the incomplete point cloud and complete point cloud have inconsistent scales in the Completion3D dataset. We correct the scales and use the corrected dataset in our experiments. See details in Appendix Section B.4.
5.2 EVALUATION METRICS
We use the Chamfer Distance (CD), Earth Mover Distance (EMD), and Fl score to evaluate the quality of the generated point clouds. CD distance is defined in Equation 5.
Earth Mover Distance. Consider the predicted point cloud V and the ground truth point cloud X of cqual size N = |V| = X the EMD loss penalizes their shape discrepancy by optimizing transportation problem. It estimates a bijection ø V +- X between V and X:
LEMD (V, X) = d:V X
min Σl E1vu-A())
(6)
VEV
F1 score. To compensate the problem that CD loss can be sensitive to outliers, we follow previous
methods (Pan et al., 2021; Tatarchenko et al., 2019) and use Fl score to explicitly evaluates the distance between object surfaces, which is defined as the harmonic mean between precision Lp(p)
and recall C (p): LFI
2L(()) where Lp(p) = M Σ 2n())
min #EX
x u1] p LR(P)
ス x
min | w p and is a predefined distance threshold. We set = 10-4 for the
MVP and Completion3D datasets, and set p 10-3 3 for the MVP-40 dataset.
7

---

Published as a conference paper at ICLR 2022
Table I: Point cloud completion results on MVP, MVP-40 and Completion3D datasets at the resolution of 2048 points. CD loss is multiplied by 10 EMD loss is multiplied by 10" Scale factors of the two losses are the same in all the other tables. The two losses are the lower the better, while Fl score is the higher the better. Note that MVP-40 dataset has larger CD and EMD losses because objects in it have larger scales than the other two datasets. Results of MVP-40 dataset % missing ratio is complemented in Appendix Table 5.
MVP
MVP40 (50% missing)
MVP40 (12.5% missing)
Completion
Method
EMD
EMD
PCN (Yuan et al., 2018)
8.65
1.95
0.342
39.67
6.37
0.581
32.56
6.18
0.619
8.81
3.03
0.315
TopNet (Tchapmi et al. 2019)
10.19
2.44
0.299
48.52
8.75
0.506
40.12
9.08
0.542
11.56
3.69
0.257
FoldingNet (Yang ct al. 2018)
10.54
3.64
0.256
51.89
11.66
0.441
46.03
8.93
0.480
14.32
4,81
0.186
MSN (Liu al., 2020)
7,08
1.71
0.434
34.33
9.70
0.646
20.20
4.54
0.728
8.88
2.69
0.359
Cascade (Wang et sl., 2020)
6,83
2.14
0,436
34,16
15,40
0.635
26.73
5.71
0.657
7.31
2.70
0.408
ECG (Pan, 2020)
7,06
2.36
0.443
34.06
16.19
0.671
40.00
6.98
0.597
10.43
3.63
0.300
GRNet (Xie al., 2020)
7.61
2.36
0.353
35,99
12.33
0.589
22.04
6,43
0.646
8.54
2.87
0.314
PMPNet (Wen et al., 2021)
5.85
3.42
0.475
25.41
29.92
0.721
13.00
8.92
0.815
7.45
4.85
0.386
VRCNet (Pan ul., 2021)
5.82
2.31
0.495
25.70
18.40
0.736
14.20
5.90
0.807
6.69
3,57
0.433
PDR paradigm (Ours)
5.66
1.37
0.499
27.20
2.68
0.739
12.70
1.39
0.827
7.10
1.75
0.451
Table 2: Completion results on MVP dataset at the resolution of 4096, 8192, 16384 points.
Table 3: Comparison of different network structures in term of training the conditional generation network and refinement network.
4096
8192
16384
Points
CD
Task
Network
CD
EMD
F1
PCN
7.14
0.469
6.02
0.577
5.18
0.650
Pointwise-nel
11.99
1.63
0.265
TopNet
7.69
0.434
6.64
0.526
5.14
0.618
Generate
Concate x' & c
10.79
1.54
0.382
Not
8.76
0.351
6,90
0.433
6.98
0.464
Coarse
PointNet++
9.39
1,38
0.355
MSN
5.37
0.583
4.40
0.663
4.09
0.696
Points
PA-Deonv
8.81
1.34
0.379
Cascade
5.46
0.579
4.51
0.686
3,90
0,743
PA-Deonv & Att.
8.71
1.29
0.389
ECG
7.31
0.506
3,99
0.717
3.32
0.774
Pointwise-net
7.71
1.45
0.407
GRNet
5.73
0.493
4.51
0.616
3.54
0.700
Refine
Concate x' &
5.78
1.38
0.490
PoinTr
4.29
0.638
3,52
0.725
2.95
0.783
Coarse
PointNct++
6.03
L.40
0.480
VRCNet
4.62
0.629
3,39
0.734
2.81
0.780
Points
PA PA-DeonV
5.96
1.40
0.482
Ours
4.26
0.649
3.35
0.754
2.61
0.817
PA- PA-Deonv & AIL.
5.66
1.37
0.499
5.3 POINT CLOUD COMPLETION
We compare our point cloud completion method with previous state-of-the-arr point cloud comple- tion methods. The comparison is performed on MVP, MVP-40, and Completion3D datasets. Results are shown in Table 1. We also conduct multi-resolution experiments on the MVP dataset, and results are shown in Table 2. Detailed experimental setups are provided in Appendix Section B. We can see that our Conditional Point by large margin in terms of EMD loss, which is highly indicative of uniformness (Zhang et al., 2021). We also achieve the highest Fl score and very low CD loss, Although VRCNet sometimes has lower CD loss than ours, it tends to put more points in the parts that are known in the incomplete point clouds, while put less points in the missing part (See Figure 10 in Appendix). In this way, its CD loss could be very low, but this mon-uniformness is undesired and leads very high EMD loss. We compare our method with other baselines in terms of visual quality of completed point clouds in Figure 4. We can see that our method generally has better visual quality. More samples are provided in Figure 9 and Figure 11 in Appendix. We also find that our PDR paradigm demonstrate some diversity in completion results as discussed in Appendix B.8.
(PDR) paradigm outperforms other methods
Ablation Study. We study the effect of attention mechanism, Point Adaptive Deconvolution (PA- Deconv) module, and Feature Transfer (FT) module in term of training the Conditional Generation Network and the Refinement Network. The experiments are conducted on MVP dataset at the reso- lution of 2048 points and results are shown in Table 3. "PA-Deonv & Att." is our proposed complete network shown in Figure 2. "PA-Deonv" removes attention mechanism. "PointNet++" further re- moves PA-Decony module. "Concate x' & c" removes FT modules. It concatenates e with x' as Zhou et al. (2021) do, and feed them to a single PointNet++ with attention mechanism and PA- Deconv. "Pointwise-net" only utilizes a global feature extracted from the incomplete point cloud. We can see that these proposed modules indeed improve the networks' performance. Note that the conditional generation networks in Table 3 are trained without data augmentation. Complete exper- imental results with data augmentation are presented in Appendix Section B.6. All the refinement networks are trained using data generated by our proposed complete dual- network trained with data augmentation. If the other ablated networks use training data generated by themselves, they would have worse refinement results.
8

---

Published as a conference paper at ICLR 2022
Partial
MSN
PoinTr
VRCNet
Ours
Ground Truth
Figure Visual comparison of point cloud completion results the MVP dataset (16384 points).
Uifferent Plousble
Pleaaile
Bounding Boxes
Generated
Ground Truth
Bounding Boxes
Genernted
Ground Truth
Figure 5: Our method can be extended to controllable point cloud generation.
DDPM acceleration. Kong Ping (2021) propose to accelerate the generation process of DDPM by jumping steps in the reverse process. The method does need retraining of the DDPM. We directly apply their method to our 3D point cloud generation network. However, we observe considerable performance drop in the accelerated DDPM. On the MVP dataset (2048 points), the original 1000-step DDPM achieves 10.7 x 10 CD loss. CD losses of the accelerated 50-step and 20-step DDPMs increase 13.2 10 and 18.1 x 10 10-4 respectively. Fortunately, we can generate coarse point clouds using the accelerated DDPMs and use another Refinement Network to refine them. The refined point clouds of the 50-step and 20-step DDPMs bear CD losses of 5.68 x 10-4 and 5. 78 10-4 respectively. Compared with the original 1000-step DDPM, which has CD loss of 5.66 x 10 it's quite temping to accept slight drop in performance for an acceleration up to 50 times. Complete results of the acceleration experiment presented in Appendix Section B.7.
5.4
EXTENSION TO CONTROLLABLE GENERATION
Our conditional PDR paradigm can be readily extended controllable point cloud generation con- ditioned bounding boxes of every part of object. We sample points the surfaces of the bounding boxes and regard this point cloud as the conditioner, just like the incomplete point cloud serves the conditioner for point cloud completion. We conduct experiments on the chair category of PartNet (Mo et al., 2019) dataset. Two examples are shown Figure 5. It's interesting that our method can generate shape different from the ground truth in some details, but be still plausible.
CONCLUSION
In this paper, we propose the Conditional Point Diffusion-RRfiiemmenn (PDR) paradigm for point cloud completion. Our method effectively leverages the strong spatial correspondence between the adjacent parts of the incomplete point cloud and the complete point cloud through the proposed Feature Transfer module, which could also infer high-level 3D structural relations. We make im- provements of the backbone PointNet++ to make capable of accurately manipulating positions of input points. Our method demonstrate significant advantages over previous methods, especially in terms of the overall distribution of the generated point cloud. We also find that method has great potential to be applied in other conditional point cloud generation tasks such as controllable point cloud generation.

---

Published as a conference paper at ICLR 2022
7 ACKNOWLEDGEMENTS
This work is partially supported by General Research Fund (GRF) of Hong Kong (No. 14205719). The authors thank useful discussions with Quan Wang from SimeeTTime and Tong Wu from CUHK.
REFERENCES
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information- -itiionn-iihhhaa 3D Model Repository. Technical Report arXiv:1512.03013 [cs.GR], Stanford University Princeton University Toyota Technological Institute at Chicago, 2015.
Jonathan Ho. Ajay Jain, and Picter Abbeel. Denoising difftusion probabilistic models. arXiv preprint arXiv:2006.112.38 2020.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint arXhv:2106.00132 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0976. 2020.
Minghua Liu, Lu Sheng. Sheng Yang, Jing Shao, and Shi-Min Hu. Morphing and sampling net- work for dense point cloud completion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11596-1 1603, 2020.
Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel cnn for efficient 3d deep learning. arXiv preprint NXXv::907.0077555 2019.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2837-2845, 2021.
Kaichun Mo. Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas Guibas, and Hao Su. Partnet: A scale benchmark for fine-grained and hierarchical part-level 3d object under- standing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 909-918, 2019.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.09672, 2021.
Liang Pan. Ecg: Edge- point cloud completion with graph convolution. IEEE Robotics and Automation Letters, 2020.
Liang Pan, Xinyi Chen, Zhongang Cai, Junzhe Zhang, Haiyu Zhao Shuai Yi, and Ziwei Liu. Vari- ational relational point completion network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8524-8533, 2021.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep point sets in a metric space. In I. Guyon, U. V.
hierarchical feature learning Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017b. URL. d8bf w0w.eeeee0001122221111111 9b
neurips cc//paper/221//111 .df.
Prajit Ramachandran, Barret Zoph, and Quoc Le. Swish: a self-gated activation function. arXiv preprint 1r: 2017.
10

---

Published as a conference paper at ICLR 2022
Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. arXiv preprint arXiv:1503.0333 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
Maxim Tatarchenko, Stephan Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. What do single-view 3d reconstruction networks learn? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3405-34- 3414, 2019.
Lyne P Tehapmi, Vinect Kosaraju, Hamid Rezatofighi, Ian Reid, and Silvio Savarese. Topnet: Structural point cloud decoder. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 383-392, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Xiaogang Wang, Marcelo H Ang Jr, and Gim Hee Lee. Cascaded refinement network for point cloud completion. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 790-799, 2020.
Xin Wen, Peng Xiang, Zhizhong Han, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Yu-Shen Liu. Pmp-net: Point cloud completion by learning multi-step point moving paths. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7443-7452, 2021.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Yaqi Xia, Yan Xia, Wei Li, Rui Song, Kailang Cao, and Uwe Stilla. Asfm-net: Asymmetrical siamese feature matching network for point completion, 2021.
Haozhe Xie, Hongxun Yao, Shangchen Zhou, Jiageng Mao, Shengping Zhang, and Wenxiu Sun. Grnet: Gridding residual network for dense point cloud completion. arXiv preprint arXiv:2006.03766 2020.
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 206-215, 2018
Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-awan transformers. arXiv preprint arXiv:2108.0883/ 2021.
Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pen: Point completion network. In 2018 International Conference on 3D Vision (3DV), pp. 728-737. IEEE, 2018
Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yco, Bo Dai, and Chen Change Loy. Unsupervised 3d shape completion through gan inversion. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1768-1777, 2021.
Wenxiao Zhang, Qingan Yan, and Chunxia Xiao. Detail preserved point cloud completion via separated feature aggregation. arXiv preprint arXiv:2007.02377 2020.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. arXiv preprint arX1v:2012.09164, 2020.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through voxet diffusion. arXiv preprint 2rXiv:2104.0.: 2021.
11

---

Published as a conference paper at ICLR 2022
APPENDIX
A METHODOLOGY DETAILS
A.1 DETAILS THE DDPM
Hyperparameters 3,. We define hyperparameters B, in the diffusion process according to linear
schedule, We let 8 = x 10 and BT = 2 x 10 Then, we define B, [- (3T - -1),t
1,2....,T.
Diffusion step embedding. The network needs to output different cØ c, for different dif- fusion steps t E (1......3 .T). We first use positional encoding (Vaswani et al., 2017) to encode each t into a 2d, dimensional vector @emb (t) ((sin(((t) cos(1p(t)) where
4x0 10
2
(1) 10
10
We set de = 64 in experiments. Then, we use two fully- fully-connected (FC) layers to transform Øemb (t) into 512 dimensional embedding vector (Ho et al., 2020). The first FC layer has input dimension 2d, = 128 and output dimension 512. The second FC layer has input dimension 512 and output dimension 512. Both layers followed by the Swish activation function (Ramachandran et al., 2017).
A.2 NEIGHBOR DEFINITION
In the Set Abstraction module and the Feature Transfer module, the neighbors are defined as the points that are within specified distance to the center point. If a center point has more than K neighbors, we randomly select K neighbors from its neighbors. If center point has less than K neighbors, pad it with dummy neighbors that has the same position has features of zeros. These dummy neighbors are excluded from the max max-pooling operation. And in the attention mechanism, the weights of the dummy neighbors way, we can guarantee that a center point can always find neighbors. We set K = 32 in the Set Abstraction module and the Feature Transfer module. In the levels of the Set Abstraction modules, the neighboring distance are set to 0.1, 0.2, 0.4, 0.8, respectively. In the 9 Feature Transfer modules, the neighboring distance are set to 0.2, 0.4, 0.8, 1, 6, 0.8, 0.4, 0.2, 0.1, respectively. Coordinates of samples in all datasets are normalized to the range [-1,
the center point, and
manually set to In this
In the Point Adaptive Deconvolution (PA-Deconv) modules of the Feature Propagation modules, the K neighbors are defined as the K nearest neighbors of the center point, and we set K 8 for all Feature Propagation modules.
A.3 PROBLEMS OF VANILLA POINTNET++
Zhou al. (2021) argue that PointNet++ not be used to train DDPM. We observe the same phenomenon in our experiments. We find that this is because the density of the input noisy cloud x' is too low for large f's. Recall that x vā,x" + vI âre, where is a Gaussian noise. ār goes to 0 for large t's. This means æ' is close to a Gaussian noise when t is large. The density of Gaussian noise is much lower than the complete point cloud x". This because points in x concentrate on the surface of some object and x is normalized to the range 1, while points from a standard Gaussian distribution could fill the whole space the range of [-3,3].
PointNet++ is originally designed to process point clouds like x It's selection of distances to de- fine neighbors described in Appendix Section A.2 suitable for point clouds that have the same level of density as xº, but can not handle point clouds close to Gaussian noise. Indeed, we find the average number of neighbors for each point the four levels the Set Abstraction mod- ules are 22.3984, 29.9199..9 29.9133, 29. 29.3266, 27. 77.8375, respectively, for 10 random shapes sampled from the MVP dataset. In constrast, the average number of neighbors for each point in the four levels are 1.0864, 1. 1. for 10 random point clouds sampled from the Gaussian distribution. Note that each point itself is considered to be neighbor itself. This means most points do not have any neighbors besides itself in a Gaussian noise.
12

---

Published as a conference paper at ICLR 2022
PointNet++ only utilizes the relative positions of input points. The input feature of each point to the first Set Abstraction module is its relative position to the center point, which is subsampled from the original input points by farthest point sampling. No information can be extracted when points do not have neighbors. This is the reason why PointNet++ can not be directly used to train a DDPM.
Our solution is attach the absolute position of each point to its feature. This guarantees that point at least can utilize its own position to decide which direction to move when it does not have neighbors. Afterall, a point cloud with large magnitude noises does not have many meaningful structures. There is not much information in the relative positions of points. Another solution is to change the definition of neighbors: From points within a specified distance K-nearest neighbors. This guarantees that point always has K neighbors. We conduct experiments to compare these two solutions, and we find that their performances are basically the same, Therefore, we just stick to the original neighbor definitions in PointNet++.
A.4 ATTENTION MECHANISM
In Section 3.2 in the main text, we mentioned that we use the attention mechanism instead of max- pooling to aggregate features at the neighboring points to the center point. We take the Set Abstrac- tion module example to elaborate on the attention mechanism. Attention mechanism in the Feature Propagation module and Feature Transfer module is similarly designed.
Assume we want to propagate features from {zy|1 S j S N1 to {ue|1 S kks S N.+1). Each point in {z,|1 S S } has a feature of dimension d,. We concatenate these features with their corresponding 3D coordinates and group them together to form matrix Fi of shape N₂ x (di + 3). We finds K neighbors in the input set {az]1 I j S )₂ for each yk- These neighbors together with their features are grouped together form matrix Gin of shape N(+1 K x (dj + 3). Then a shared multi-layer perceptron (MLP) is applied to transform the grouped feature G. to G which is a matrix of shape N141 X K x di+1 and d 1 is the dimension of the output feature.
In our attention mechanism, Gin (shape N1+1 X K x (d + 3)) will act like keys, Gout (shape N141 K x d(41) will act like values, while the original features (ykl1 [1 S k S N4+13 will act like queries. Since {ur 1 S & S } is a subset of {z, S j < Ni). we can group the original features {ynl1 S k S Ni+1 to form a matrix Q. which of shape Ni41 x (di+3), Q is first repeated K times into a matrix of shape No+1 x K x (d: + 3). Then this matrix is passed through a shared MLP and transformed into a matrix Q', which is of shape N1+1 x K x dguerg- Next, we pass G.n. through shared MLP to transform it into new matrix G¹ which is of shape N141 X K x diey. We concatenate the query matrix Q' with the key matrix Gin along the feature dimension. We denote this matrix IQ', Gi,.l. which is of shape Ni+1 x K X (dquery + dkey)- is passed through a shared MLP to obtain the scores of all the K neighbors. We denote the scores matrix S of shape N X K x d1+1- Note that S has the same shape Gout- And the scores S of the K neighbors are adaptively computed according to the feature at the center point yk and features at its K neighbors {æ,í E (yx)). We apply softmax operation along the neighbor dimension (the second dimension) of S to obtain the weight matrix of all the K neighbors. We denote it as W, which is of shape N441 x K de41- Note that we manually set the weights of the padded dummy neighbors to 0 in W. Then the weight matrix W and the value matrix Gout are dot producted along the neighbor dimension (the second dimension)
form the output matrix F', which is of shape N1+1 x d(+1. Finally, F" is concatenated with the 3D coordinates of the set {yee1 < k < N1+1) to form output of the Set Abstraction module, F4+1, which a matrix of shape N441 x (d++ + 3).
A.5 PROBLEMS WITH THREF INTERPOLATION
The original PointNet++ uses three interpolation to upsample features in the Feature Propagation module. We think that the three interpolation operation is suitable for tasks like point cloud part segmentation, but not suitable in our task. Interpolation means that points close to each other have similar features. Points close to each other tend to have similar semantic labels in a clean point cloud, therefore it is meaningful to use interpolation operation to upsample features in the part segmentation task. However, in our task, the network need to predict a per-point displacement for all points in noisy point cloud and move it towards clean point cloud. Points close to each other
13

---

Published as a conference paper at ICLR 2022
do not need to move in similar direction in general. In fact, they may just need to move towards the opposite direction to form a smooth surface.
We also find that the three interpolation operation lacks the ability manipulate positions of points accurately in small scales. We first elaborate on the three interpolation operation used in the original +. Assume we want upsample features at {yx|1 I k M+11 to {ayl1 I Ni),
where {yx|1 11 & S N is a subset {zy J Σ Ni)-
For each zj. assume its three nearest neighbors in {yu & k z N4+1) } are {ynlk E By Then feature at æÿ is obtained through the following equation:
[1xz) EkEBya(a,) #f) (yss
1
where w(yk, = 1luk x3 xTyl?
EkEBy,(aR)) 3(),
[wx) and f(2z) are features y and Tj,, respectively. We see that the value of f(x) is determined by the relative distances between itself and its three nearest neighbors. However, in 3D space, the point that has specific relative distances to three fixed points not unique. In fact, the point can move freely on a curve.
Let's take very simple example, assume the three nearest neighbors of zy forms regular triangle. If we move 2 along the straight line that passes the center of the triangle and is perpendicular to the plane determined by the triangle, then 2; will always have the same relative distances from the three points, which means Ij will always have the same interpolated feature, as long as its movement is small enough that its three nearest neighbors do not change.
This property makes the three interpolation operation not able to distinguish some specific points in small scale, since these points could have the same interpolated value. Therefore, three interpolation operation is not suitable for our task, as we need to accurately manipulate positions of points to make them form a meaningful shape with smooth surfaces and sharp details.
B EXPERIMENT
B. DETAILED EXPERIMENTAL SETUP
In experiments, we use the Adam optimizer with learning rate of x 10-1, -, For experiments of PDR paradigm in Table I and Table 2 in the main text, we use data augmentation described in Appendix Section B.3. We train our Conditional Generation Network for 340 epochs, 200 epochs, and 500 epochs on the MVP, MVP-40, and Completion3D datasets, respectively. We save check- point and evaluate the network's performance on both the training set and the test set every 20 epochs. Since the generation process of DDPM is very slow, we randomly select 1600 samples from the training set and test set respectively for evaluation. (Test set of the Completion3D dataset has less than 1600 samples. Therefore, we all samples in the test for evaluation.) The checkpoint with the lowest CD loss is chosen as the best network. It is used to generate training data for the Refinement Network. We train the Refinement Network for 100 epochs, 150 epochs and 200 epochs on the MVP, MVP-40, and Completion3D datasets, respectively.
Note that we subsampling the test set only when we try to choose best checkpoint in the training process of the conditional generation network in DDPM. After choosing the best checkpoint, we use it to generate training data to train the refinement network. However, when we evaluate the whole PDR paradigm (composed of the conditional generation network and refinement network) and compare with previous methods, we evaluate them on the complete test set. Therefore, the comparison result in Table 1 and Table 2 in the main text is reliable and fair.
For ablation studies in Table 3, the Conditional Generation Networks are trained without data aug- mentation for 300 epochs. All the Refinement Networks are trained the same data generated by our proposed Conditional Generation Network trained with data augmentation. The Refinement Networks trained for 100 epochs.
All baseline methods are rerun under the data augmentation described in Appendix Section B.3 according to their open source codes. And CD loss is chosen to train all the baseline methods.
14

---

Published as a conference paper at ICLR 2022
Condition Feature
Extraction Subnet
128
Feature Dimension
Ski Connections
Condition Feature
Extraction Subnet
Condition
mabsamaling
Pointict
PointNet Subse maling
colnammiing
.dttampiing
Pairehir
Ph-Berom
Pa-Sa Swrrrrm
PoinNit
Pi- Decamm
Paintint
Pk Decan
Pointet
Point
Cloud
2-Stage Pointivet to Eatrast Global Feature
Feature Trandfer Modules
Transfer
Modules
skip Connections
Diffusion Stop
Noley Point
Subsampling
PointNet
Subsampling
Pointlee
Subrsmpling
Pointtee
Pamblet
PA Docona
Decomy
PoinNet
Deconv
Paintet
Shared
Output Co(x', c, t)
Positionall Encoding & FC Layers
Cloud x
Denoise
Subnet
KC layers
Num of Points
2048
1024
256
64
16
64
256
1024
2048
Denoise Subnet
Feature Dimension
32
64
256
512
256
256
128
128
3
Figure 6: Detailed network structure.
B.2 DETAILS THE NETWORK STRUCTURE.
Detailed network structure is shown in Figure 6. We present the number of points and feature dimension in each level of the Feature Extraction network and the Denoise network. The distances to define neighbors are provided in Appendix Section A.2.
15

---

Published as a conference paper at ICLR 2022
B.3 DATA AUGMENTATION
We use rotation, mirror, translation, and scaling as data augmentation methods during training. Ro- tation is performed along the upward direction of the shapes. The upward direction is the y-axis in MVP dataset and Completion3D dataset, while upward direction in the MVP-40 dataset is the 2- axis. And the rotation angle is uniformly sampled from the interval a. a]. where a is a predefined hyper-parameter that controls the magnitude of the rotation.
Mirror operation performed with respect to the two planes that are parallel to the upward direction: 0 plane and 0 plane for MVP dataset and Completion3D dataset, x = 0 plane and 0 plane for MVP-40 dataset. The mirror operation is performed with a probability of m/2 with respect to the two planes, respectively, m is predefined hyper-parameter the controls the probability of the mirror operation.
Same as some previous works (Wang al., 2020; Xia et al., 2021), we observe that most objects in the MVP dataset and Completion3D dataset have reflection symmetry with respect to the zy plane. Therefore, mirror the partial input with respect to this plane and concatenate the mirrored points with the original partial input for these two datasets. We subsample this concatenated point cloud from 4096 points 3072 points by farthest point sampling to obtain uniform point cloud. We label the original points with 1 and the mirrored points with -1. This concatenated point cloud is feed to both the Conditional Generation Network and the Refinement Network, so that they could learn whether an object has reflection symmetry and determine whether to utilize the mirrored points according to their 1 label.
Translation is achieved by adding a randomly sampled 3D vector to every point in the incomplete point cloud and the complete point cloud. Each component of the 3D translation vector is sampled from Gaussian distribution with zero mean and standard deviation of a, where a is a predefined hyper-parameter the controls the magnitude of the translation operation.
We also randomly samples scaling factor uniformly from the interval [Stow: Snägh) when loading training pair. The scaling factor is multiplied to the coordinates of all points in the incomplete point cloud and the complete point cloud.
We observe that data augmentations can prevent the network from overfitting on the training set, but could also lead to performance drop on the test set. Therefore, we use different data augmentation schemes to train the Conditional Generation Network in the DDPM and the Refinement Network. When training the Conditional Generation Network, we hope the network does not overfit the training set, because we need to generate training samples to train the Refinement Network. There- fore, data augmentations of large magnitudes to train the Conditional Generation Network. However, when training the Refinement Network, high performance is the top priority. Therefore, we use data augmentations of small magnitudes train the Refinement Network. We also use data augmentation to train baseline methods. The data augmentations are the same ones that we use to train the Refinement Network. The details of the data augmentation is shown Table 4.
Table 4: Data augmentations used in MVP, MVP-40 and Completion3D dataset by the conditional generation network, refinement network, and all baselines.
Conditional Generation Network
Refinement Network and other Baselines
Rotation
Mirror
Translation
Scaling
Rotation
Mirror
Translation
Scaling
MVP
90"
0.5
0.1
[1/1.2, 1.2]
a= 32
0.5
0.005
[1/1.01, 1.01]
MVP-40
= 0"
m
0.5
0 0
[1/1.2, 1.2]
a= 3"
m
0.5
= 0.005
[1/1.01, 1.01]
Completion3D
10a
0.2
0.01
[0.66, 1]
3
0.1
0.005
00.66, 1]
B.4
ISSUE OF THE COMPLETION3D DATASET
We find that many pairs of incomplete-comple point clouds have inconsistent scales in the Com- Pletion3D dataset. A few inconsistent examples are shown in Figure 7. Ideally, the incomplete point should overlap with the complete point cloud in 3D space, but many incomplete-comple pairs in the Completion dataset cannot overlap with each other due inconsistent scales, which misleads
16

---

Published as a conference paper at ICLR 2022
Points of Incomplete Point Cloud
Points of Complete Point Cloud
Inconsistent Pairs
Corrected
Pairs
Figure 7: The first row shows some inconsistent pairs of the incomplete point cloud and complete point cloud from the Completion3D dataset. The second row are the corrected pairs by minimizing one-side CD loss.
the network training. Moreover, the scale-inconsistency issue also gives rise unreliable evaluation results, as we expect the network to predict a complete point cloud of a consistent scale with the incomplete point cloud. Therefore, it is necessary to correct the scales of these pairs before using the dataset.
We leverage the one-side CD loss identify and correct these pairs. For a consistent pair of the incomplete point cloud C' and complete point cloud X, the one-side CD loss should be very low:
LiSid CD CC, X)
Σ
min ||c z/|2. =EX
C
We find the correct scale of the incomplete point cloud by optimizing the following problem min Li Side (D X).
This optimization problem solved by using the python package scipy.optimize.fie for every pair of point clouds in the Completion3D dataset. We consider the pairs with scale factor å greater than 1.05 or less than 0.95 as inconsistent pairs, and then correct its scale inconsistency by multiplying the scale factor d to these incomplete point clouds. In the training set, we find 2.81% pairs are inconsistent. The inconsistent pairs are also discovered in the validation set. We can not verify the test set because the ground truth complete point cloud is not released.
We did not use the online Completion3D benchmark server to evaluate our method and previous methods for the following reasons: 1) the website server was out of service, as it gave feedback for any submissions at the time we conduct this work; 2) the ground truth complete point clouds test set of the Completion3D dataset are not released, and hence we can not verify whether this inconsistency problem is also present in the test set. Therefore, we use the test set provided in the work (Wang et al., 2020), which contains 1200 pairs of ceooplete-compleer point clouds for testing. It contains the same set of objects as the test set of the original Completion3D dataset. After correcting inconsistent pairs in this test set, we evaluate our method and previous methods on this revised test set to achieve fair and reliable comparisons.
B.5 COMPLETE EXPERIMENT RESULTS FOR MVP-40 DATASET
In Table I in the main text, we only present the completion result at the missing ratio of 50% and 12.5% for the MVP-40 dataset. We present the complete experiment result on MVP-40 dataset including result at the 25% missing ratio in Table below.
17

---

Published as a conference paper at ICLR 2022
Table 5: Complete Point cloud completion results MVP-40 dataset. The missing ratio is 50%,
25% and 12.5%, respectively, CD loss is multiplied by 10*. EMD loss is multiplied by 102.
MVP40 (50% missing)
MVP40 (25% missing)
MVP40 (12.5% missing)
Method
CD
FMD
FMD
FMD
PCN (Yuan et al., 2018)
39.67
6.37
0.581
34.40
6.21
0.606
32.56
6.18
0.619
TopNet (Tchapmi al., 2019)
48.52
8.75
0.506
42.39
10.25
0.520
40.12
9.08
0.542
FoldingNet (Yang al., 2018)
51.89
11.66
0.441
45.99
9.85
0.475
46.03
8.93
0.480
(Liu et al., 2020)
34.33
9.70
0.646
23.14
6.59
0.712
20.20
4.54
0.728
Cascade (Wang et al., 2020)
34.16
15.40
0.635
29.13
8.16
0.647
26.73
5.71
0.657
ECG (Pan, 2020)
34.06
16.19
0.671
28.01
10.79
0.717
16.90
6.20
0.774
GRNet (Xie et al., 2020)
35.99
12.33
0.589
25.84
8.43
0,626
22.04
6,43
0.646
PMPNet (Wen et al., 2021)
25.41
29.92
0.721
15.73
16.08
0.815
13.00
8.92
0.815
VRCNet (Pan et al., 2021)
25.70
18.40
0.736
18.28
10.96
0.776
14.20
5.90
0.807
PDR paradigm (Ours)
27.20
2.68
0.739
16.54
1.68
0.800
12.70
1.39
0.827
B.6 COMPLETE EXPERIMENT RESULTS FOR NETWORK ABLATION STUDY
Table Comparison of coarse point clouds generated by conditional generation networks of differ- ent structures MVP dataset at the resolution of 2048 points. Experiments conducted under two circumstances: with and without data augmentation. The networks without data augmentation trained for 600 epochs. The
trained for 300 epochs, and networks with data augmentation data augmentation we use is specified in Table for the MVP dataset. We report the networks' the overfitting problem is largely
performance both the training set and the test set. We mitigated by data augmentation.
Data
CD
EMD
F1
Model
Augmentation
Train
Test
Train
Test
Train
Test
Pointwise-net
X
6.96
11.99
0.88
1.63
0.328
0.265
Concate x' &
X
9.96
10.79
1.57
1.54
0.397
0.382
PointNet++
X
6.44
9.39
0.84
1.38
0.397
0.355
PA-Deconv
X
5.85
8.81
0.73
1.34
0.425
0.379
PA-Deconv & Att.
X
5.56
8.71
0.70
1.29
0.443
0.389
Pointwise-net
11.30
12.69
1.33
1.61
0.262
0.246
Concate x'
14.52
16.31
1.90
1.99
0.414
0.395
PointNet++
9.86
11.18
1.43
1.74
0.402
0.376
PA-Decony
9.22
10.78
1.24
1.50
0.407
0.381
PA-Decony & Att.
7.98
9.24
1.03
1.32
0.436
0.409
In Section 5.3 in the main text, conduct ablation study of our proposed network architecture, and results are shown in Table 3. Note that the conditional generation networks in DDPM in Table 3 are trained without data augmentation. However, it is actually very important train the conditional generation networks with data augmentation, because we need to prevent it from overfitting on the training set, so that they can generate coarse point clouds of consistent distribution on the training set and test set to train the refinement network. Therefore, we provide the training results with data augmentation in Table 6. The data augmentation is specified in Table 4 for MVP dataset.
Same as Table 3 in the main text, "PA-Deonv & Att." is our proposed complete network shown in Figure 2. "PA-Deonv" our network without attention mechanism. "PointNet+4" further removes the PA-Deconv module. "Concate x' & c removes FT modules. It concatenates e with æf as Zhou et al. (2021) do, and feed them to a single PointNet++ with attention mechanism and PA-Deconv. "Pointwise-ne -ee" only utilizes global feature extracted from the incomplete point cloud. We can that these proposed modules indeed improve the networks' performance. Our proposed networks achieve superior results both with and without data augmentation.
18

---

We also observe that networks generally achieve better performance on both the training set and test set without data augmentation, but they tend to overfit on the training set. This is undesirable because need these conditional generation networks to generate training data for the refinement networks. is very important for them to generate coarse point clouds that have consistent distributions on the that the overfitting problem is largely mitigated in Published as a conference paper at ICLR 2022
training set and the test set. Indeed, we the presence of data augmentation.
B.7 COMPLETE EXPERIMENT RESULTS FOR DDPM ACCELERATION
The complete experiment results the DDPM acceleration shown in Table We can see that the quality of coarse point clouds generated by the accelerated DDPMs has dropped considerably. How- ever, with the help of the Refinement Network, the performance drop of the final refined point clouds slight. This demonstrates the strong refinement capability of our proposed network architecture shown in Figure 2 in the main text.
Table Refine coarse point clouds generated by the accelerated DDPMs on the MVP dataset at the resolution of 2048 points. We can performance drop is slight for the refined point clouds. We also report the average generation time of a single point cloud evaluated on one NVIDIA GEFORCE RTX 2080 Ti GPU for DDPM of different reverse steps.
Number of
Average
CD
EMD
F1
Reverse Steps
Generation Time
Coarse
Refined
Coarse
Refined
Coarse
Refined
1000 (Original)
16.86
10.69
5.66
1.46
1.37
0.400
0.499
50
0.78 s
13.19
5.68
1.65
1.47
0.341
0.493
20
0.32 s
18.12
5.78
1.99
1.56
0.255
0.474
19

---

Published as a conference paper at ICLR 2022
Coarse
The landing gear is different.
The ship bow is different.
Refined
Coarse
The curve of the back is different.
The antenna is different.
Refined
Figure 8: Our PDR paradigm demonstrates diversity in the completion results. For each object, the two images in the first row are coarse completion results from a trained DDPM generated in two trials for the same incomplete point cloud. The two images in the second row refined results for the two coarse point clouds, respectively. We can see that some diversity is preserved after the refinement.
B.8 GENERATION DIVERSITY OF THE PDR PARADIGM
In this section, we discuss whether the PDR Paradigm can generate diverse completion results for the same incomplete point cloud. Although there is no stochasticity in the refinement network, find our PDR paradigm still demonstrates some kind of diversity in the completion results.
We know that DDPM itself is probabilistic model and can generate diverse completion results. The refinement network receives a coarse completion from the DDPM and then refines it according to the condition point cloud, i.e., the incomplete point cloud. The final refined result surely depends on the condition point cloud, but also depends on the coarse point cloud received from the DDPM. The refinement network can only refine the coarse point cloud in small scale, because we multiply the output of the refinement network by a small constant 7 0.001 as described in Section 3.3 in the main text. Therefore, the overall sketch of the coarse point cloud will be preserved after the refinement. This explains why the PDR paradigm still bears low EMD loss the DDPM, even though we use CD loss train the refinement network, because the refinement network does not change the overall distribution of the coarse shape generated by DDPM.
Back to the diversity issue, the coarse completion results from DDPM demonstrate diversity for the same incomplete point cloud, the refined point clouds will also demonstrate some diversity because the inputs the refinement network are different. Figure 8 shows some examples where the PDR paradigm demonstrate diversity in the completion results.
20

---

Published as a conference paper at ICLR 2022
B.9 MORE VISUALIZATION RESULTS
Partial
TopNet
FoldingNet
PCN
ECG
GRNet
Cascade
MSN
PoinTr
VRCNet
Ours
Ground Truth
Figure 9: Visual comparison our method and other baselines. Samples are from the MVP dataset at the resolution of 16384 points. We can see that point clouds generated by our method generally have better visual quality.
21

---

Published as a conference paper at ICLR 2022
Partial
VRCNet
Ours
Ground Truth
Figure 10: Visual comparison of our method and VRCNet. Samples are from the MVP dataset at the resolution of 16384 points. We can that VRCNet sometimes tend predict more points to the parts that are known in the incompelte point cloud, while put less points at the missing part. This could effectively reduce CD loss, but leads to large EMD loss. Compared with VRCNet, our method generally generates more uniform point clouds.
22

---

Published as a conference paper at ICLR 2022
Partial
PoinTr
Ours
Ground Truth
Figure 11: Visual comparison of our method and PoinTr. Samples from the MVP dataset at the resolution of 16384 points. We can see that PoinTr sometimes tend to predict more points at the skeleton of objects, while points surfaces sparse. Compared with PoinTr, our method generally generates more uniform point clouds.
23

---

Published as a conference paper at ICLR 2022
Partial
Coarse
Refined
Ground Truth
Figure 12: Visual comparison of coarse point clouds generated by the Conditional Generation Net- work in DDPM and point clouds after refinement. Samples are from the MVP dataset at the reso- that coarse point clouds generated by the Conditional Generation
lution of 2048 points. We Network basically uniformly cover the overall shape of objects, but tend to be noisy. After refine- ment, point clouds demonstrate both good overall density distribution and sharp local details.
24


Reviews:
[{"review_id": "XLMdjfVhrZY", "reviewer": "Reviewer_bEQt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n1. The manuscript is overall clearly written.\n2. The authors propose a novel pipeline for point cloud completion.\n\n**Weaknesses**\n\n1. The figures in the manuscript are not clear enough. For example, the relationship between CGNet and DDPM is not clearly represented in Figure 1. The relationship of CGNet, RFNet, CFENet, and Denoise Net are not well represented. This unclarity makes me wonder that both CGNet and RFNet have CFENet and Denoise Net, but why do they have different names?\n2. The authors claim that the generated results are more uniform than other methods. Please provide evidence.\n3. It seems that the results of the Completion3D benchmark in Table 1 are not consistent with the online results [here](https://completion3d.stanford.edu/results). Could you explain the reason?\n4. There are lots of methods in Tables 1-3. However, there are only a few methods in Figure 4. Please provide more qualitative results in Figures 4.\n5. Please provide the running time comparison with other methods because you incoporate lots of time-comsuming operations.", "summary_of_the_paper": "In this paper, the authors propose a novel Point Diffusion-Refinement (PDR) paradigm for point cloud completion. The proposed method effectively and efficiently extracts multi-level features from partially observed point clouds to guide completion. Moreover, it accurately manipulates spatial locations of 3D points to obtain smooth surfaces and sharp details. Experimental results demonstrate its state-of-the-art performance on the mainstream datasets and benchmarks.", "summary_of_the_review": "I'm leaning to reject this paper (Borderline Reject) *at this time* due to the following reasons:\n\n1. Some claims in the manuscript lacks references. \n2. The results of the Completion 3D benchmark are not consistent with the online results.\n\n**Update (11/26/2021)**\n\nThe authors address all of my concerns in the rebuttal. Therefore, I agree to accept the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ugsi0PDYafl", "reviewer": "Reviewer_qA73", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \nState-of-the-art performer for point cloud completion on various datasets.\nI am impressed by the comprehensiveness of the paper. All the key details and experiments are included and well-presented.\nFurthermore, since the specific problem of point cloud completion using a DDPM is a relatively under-studied problem, there are many design decisions that need to be made and justified. The paper is very good in this respect.\nVarious components in the original PointNet++, such as up- and down-sampling modules, are carefully analyzed of their inner workings, taylored to the proposed setting, and evaluated with ablation studies.\n\nQuestions:\n1. In some sense, the refinement network can be considered as one extra step to the DDPM generator. I wonder if you have any insight on how does the proposed two-stage generation + refinement paradigm compare with using a single stage DDPM, but with a larger model?\n\n2. in section B.1, the author mentiond that the reported results are evaluated on a subset due to the slow evaluation speed. Since the confidence intervals of the evaluation metrics w.r.t. subsampling are not reported, I wonder if subsampling the test set can potentially lead to unreliable and unfair comparison with previous methods?\n\n3. Regarding evaluation speed, I wonder how long does it take to evaluate one point cloud?\n\n4. Regarding Table 5, is the refinement network retrained for experiments with different DDPM steps?\n\n**Update:**\nThank the author for providing additional insights on how the 2-staged model can improve the result -- indeed, minimizing ELBO does not imply minimizing Chamfer distance. I am also glad to learn that the comparisons with baselines are fair. I thus retain my rating of accept.\n", "summary_of_the_paper": "The paper tackles the problem of shape-level point cloud completion in a fully supervised setting. The big picture of the proposed method is to use a conditional DDPM to generate a noisy but complete point cloud given incomplete input, and then use another refinement network, also conditional on the incomplete input, to further refine the noisy point cloud. Both networks adopt a novel, dual-path network architecture, based on PointNet++, to allow localized guidance from the incomplete point cloud. Thanks to the 2-staged approach, up to 50x acceleration of evaluation speed is achievable using step jumping, with moderate performance drop. The model is evaluated on MVP, MVP-40 and Completion3D datasets.", "summary_of_the_review": "Good performance. Novel approach. Comprehensive paper. Contributions are well justified. some concern on the reliability of the reported numbers due to subsampling the test set.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "60DBNpJT8ZU", "reviewer": "Reviewer_prns", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I thoroughly enjoyed this paper. Though the paper casts itself as a \"point cloud completion\" paper, I find that the architecture presented is more general and more convincingly powerful than previous DDPM models proposed for general 3D point cloud generation (Luo et al, CVPR'21 and Zhou et al, ICCV'21). \n\nLuo's approach, which was a best paper finalist for CVPR, is comparatively much weaker in its conditionalization strategy, whereas this proposed work utilizes a multi-scale dual U-Net type approach which allows multi-scale conditioning information to propagate through the network, which seems to add to additional fidelity. Zhou's approach uses point-voxel CNN's, making the (mostly empirical) argument that some kind of point ordering (i.e. voxelization) is necessary for diffusion and so something like PointNet++ cannot be used. This argument never sat well with me, so I was happy to see the authors in this paper also rebut Zhou's reasoning, and additionally present an easy and elegant fix-- to add the absolute coordinates to the PointNet feature (similar in spirit to the CoordConv approach of Liu et al, NeurIPS'18).\n\nThe paper is well-presented and easy-to-read, with a robust supplementary. All design choices were reasonably discussed, and I was happy to see that the authors stressed how Chamfer distance losses could be avoided using the DDPM design, a point which isn't always well-explained in the point cloud diffusion literature (even though the authors end up using CD-based loss anyway in the refinement module). I like and agree with the proposed minor modifications to the PointNet++ modules, as well as the use of self-attention instead of pooling. I'm not 100% sold on the refinement network-- I think the training procedure of generating 10 coarse models seems a bit ad hoc-- but I do think it represents an interesting trade-off, and I'm glad the authors included it.\n\nHere's my wishlist for this paper:\n1. Direct comparisons against Point-Voxel diffusion completion results. This is the most apples-to-apples comparison you could make. \n2. Multimodal completions. My guess is that the refinementNet probably hurts your ability to get diverse completion results? Perhaps with some stochasticity in the feature transfer module could be added to produce multimodal completions? If your method is unable to provide multimodal completions, you should discuss this limitation.\n3. Representation learning results. Recently it was shown that point cloud completion can be a powerful pretext task for representation learning on point clouds (see Wang et al, \"Unsupervised Point Cloud Pre-training via Occlusion Completion\" ICCV'21). I would be very curious what would happen if you attempted to extract a representation from your trained feature transfer modules and use it for a different downstream supervised task. My guess is that it could very well outperform previous art. \n4. \"In the wild\" completion results: for example, training on shapenet/modelnet cars and then conditioning from bounding box crops of partially occluded car point clouds in KITTI\n5. Point cloud generation with depth map conditioning modality (like as done in Point-Voxel diffusion)\n\nI caught a few typos. Please comb over the paper one more time. For example, from the first couple pages I see: \"for many downstreaming applications\" and \"it can iteratively moves a set of Gaussian noise\" and \"we propose the feature transform (FT) module to directly transmits encoded point features\"", "summary_of_the_paper": "This paper proposed a novel way to utilize DDPM's for point cloud completion. They use a conditional DDPM to generate a plausible \"complete\" point cloud from pure Gaussian noise, conditional on a partial subset of the point cloud points. Their main contribution is is in the design of the conditional feature extraction and denoising subnet, which essentially form dual path connected U-Net type structures with internal modules based on an improved PointNet++ design. Additionally, the authors propose a refinement network to mitigate the computational burden of needing so many diffusion steps, which seems to offer an acceptable performance tradeoff for increases computational efficiency. The method convincingly demonstrates superior performance over its competitors, with crisp details preserved on the datasets tested.", "summary_of_the_review": "Overall, I really like this paper. I would like to see the authors address the items in my wishlist (specifically iterms #1-3), but the paper stands pretty well on its own.\n\nI also think that diffusion models and 3D point clouds is a nice and timely area of research, and this paper will likely be a nice reference point for others trying to combine DDPM's with 3D point cloud applications.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach based on conditional denoising diffusion models for point cloud completion. The reviewers have recognized the significance of contributions, the clarity of presentation, and the comprehensivity of experiments. I am happy to recommend this paper for presentation at ICLR.
Decision: Accept (Poster)