A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion

1 INTRODUCTION . With the rapid developments of 3D sensors , 3D point clouds are an important data format that captures 3D information owing to their ease of acquisition and efficiency in storage . Unfortunately , point clouds scanned in the real world are often incomplete due to partial observation and self occlusion . It is important to recover the complete shape by inferring the missing parts for many downstream tasks such as 3D reconstruction , augmented reality and scene understanding . To tackle this problem , many learning-based methods ( Yuan et al. , 2018 ; Yang et al. , 2018 ; Tchapmi et al. , 2019 ; Xie et al. , 2020 ; Liu et al. , 2020 ; Pan et al. , 2021 ) are proposed , which are supervised by using either the Chamfer Distance ( CD ) or Earth Mover Distance ( EMD ) to penalize the discrepancies between the generated complete point cloud and the ground truth . However , CD loss is not sensitive to overall density distribution , and thus networks trained by CD loss could generate non-uniform point cloud completion results ( See Figure 10 and 11 in Appendix ) . EMD is more distinctive to measure density distributions , but it is too expensive to compute in training . The absence of an effective and efficient training loss highly limits the capabilities of many existing point cloud completion networks . We find that denoising diffusion probabilistic models ( DDPM ) ( Sohl-Dickstein et al. , 2015 ; Ho et al. , 2020 ) can potentially generate uniform and high quality point clouds with an effective and efficient loss function . It can iteratively move a set of Gaussian noise towards a complete and clean point cloud . DDPM defines a one-to-one pointwise mapping between two consecutive point clouds in the diffusion process , which enables it to use a simple mean squared error loss function for training . This loss function is efficient to compute and explicitly requires the generated point cloud to be uniform , as a one-to-one point mapping is naturally established between the generated point cloud and the ground truth . Point cloud completion task can be treated as a conditional generation problem in the framework of DDPM ( Zhou et al. , 2021 ; Luo & Hu , 2021 ) . Indeed , we find the complete point clouds generated by a conditional DDPM often have a good overall distribution that uniformly covers the shape of the object . Nonetheless , due to the probabilistic nature of DDPM and the lack of a suitable network architecture to train the conditional DDPM for 3D point cloud completion in previous works , we find DDPM completed point clouds often lack smooth surfaces and sharp details ( See Figure 1 and Appendix Figure 12 ) , which is also reflected by their high CD loss compared with state-of-the-art point cloud completion methods in our experiments . Another problem with DDPM is its inefficiency in the inference phase . It usually takes several hundreds and even up to one thousand forward steps to generate a single point cloud . Several methods ( Song et al. , 2020 ; Nichol & Dhariwal , 2021 ; Kong & Ping , 2021 ) are proposed to accelerate DDPM using jumping steps without retraining networks , which however , leads to an obvious performance drop when using a small number of diffusion steps . In this work , we propose the Conditional Point Diffusion-Refinement ( PDR ) paradigm to generate both uniform and high quality complete point clouds . As shown in Figure 1 , our PDR paradigm performs point cloud completion in a coarse-to-fine fashion . Firstly , we use the Conditional Generation Network ( CGNet ) to generate a coarse complete point cloud by the DDPM conditioned on the partial point cloud . It iteratively moves a set of Gaussian noise towards a complete point cloud . Following , the ReFinement Network ( RFNet ) further refines the coarse complete point cloud generated from the Conditional Generation Network with the help of partial point clouds . In addition , RFNet can be used to refine the low quality point clouds generated by an accelerated DDPM , so that we could enjoy an acceleration up to 50 times , while minimizing the performance drop . In this way , the completion results generated by our PDR paradigm demonstrate both good overall density distribution ( i.e . uniform ) and sharp local details . Both CGNet and RFNet have a novel dual-path network architecture shown in Figure 2 , which is composed of two parallel sub-networks , a Denoise subnet and a Condition Feature Extraction subnet for noisy point clouds and partial point clouds , respectively . Specifically , we propose Point Adaptive Deconvolution ( PA-Deconv ) operation for upsampling , which can effectively manipulate spatial locations of 3D points . Furthermore , we propose the Feature Transfer ( FT ) module to directly transmit encoded point features at different scales from the Condition Feature Extraction subnet to the corresponding hierarchy in the Denoise subnet . Extensive experimental results show that our PDR paradigm can provide new state-of-the-art performance for point cloud completion . Our Key contributions can be summarized as : 1 ) We identify conditional DDPM to be a good model with an effective and efficient loss function to generate uniform point clouds in point cloud completion task . 2 ) By using RFNet to refine the coarse point clouds , our PDR paradigm can generate complete point cloud with both good overall density distribution ( i.e . uniform ) and sharp local details . 3 ) We design novel point learning modules , including PA-Deconv and Feature Transfer modules , for constructing CGNet in DDPM and RFNet , which effectively and efficiently utilizes multi-level features extracted from incomplete point clouds for point cloud completion . 4 ) With the help of our proposed RFNet , we can accelerate the generation process of DDPM up to 50 times without a significant drop in point cloud quality . 2 PROBLEM STATEMENT . In this paper , we focus on the 3D point cloud completion task . A 3D point cloud is represented by N points in the 3D space : X = { xj |1 ≤ j ≤ N } , where each xj ∈ R3 is the 3D coordinates of the j-th point . We assume the dataset is composed of M data pairs { ( Xi , Ci ) |1 ≤ i ≤ M } , where Xi is the i-th ground-truth point cloud , and Ci is the incomplete point cloud from a partial observation ofXi . The goal is to develop a model that completes the partial observation Ci and outputs a point cloud as close to the ground truth Xi as possible . For algebraic convenience , we let x ∈ R3N be the vector form of a point cloudX , and similarly c be the vector form of C . 3 METHODOLOGY . We consider the point cloud completion task as a conditional generation problem , where the incomplete point cloud C serves as the conditioner . We use the powerful generative model called denoising diffusion probabilistic models ( DDPM ) ( Sohl-Dickstein et al. , 2015 ; Ho et al. , 2020 ) to first generate a coarse completion of the partial observation . Then we use another network to refine the coarse point cloud to improve its visual quality . Our point cloud completion pipeline is shown in Figure 1 . We first briefly introduce the theory of DDPM in Section 3.1 , and then describe detailed architecture of the Conditional Generation Network ( CGNet ) and ReFinement Network ( RFNet ) in Section 3.2 and Section 3.3 . 3.1 BACKGROUND ON CONDITIONAL DENOISING DIFFUSION PROBABILISTIC MODELS . We assume pdata to be the distribution of the complete point cloud xi in the dataset , and platent = N ( 03N , I3N×3N ) to be the latent distribution , whereN is the Gaussian distribution . Then , the conditional DDPM consists of two Markov chains called the diffusion process and the reverse process . Both processes have length equal to T . We set T = 1000 in this paper . The Diffusion Process . The diffusion process is a Markov process that adds Gaussian noise into the clean data distribution pdata until the output distribution is close to platent . The diffusion process is irrelevant of the conditioner , the incomplete point cloud ci . Formally , let x0 ∼ pdata . We use the superscript to denote the diffusion step t. For conciseness , we omit the subscription i in the following discussion . The diffusion process from clean data x0 to xT is defined as q ( x1 , · · · , xT |x0 ) = T∏ t=1 q ( xt|xt−1 ) , where q ( xt|xt−1 ) = N ( xt ; √ 1− βtxt−1 , βtI ) . ( 1 ) The hyperparameters βt are pre-defined , small positive constants ( See details in Appendix Section A.1 ) . According to Ho et al . ( 2020 ) , there is a closed form expression for q ( xt|x0 ) . We first define constants αt = 1− βt , ᾱt = ∏t i=1 αi . Then , we have q ( x t|x0 ) = N ( xt ; √ ᾱtx 0 , ( 1− ᾱt ) I ) . Therefore , when T is large enough , ᾱt goes to 0 , and q ( xT |x0 ) becomes close to the latent distribution platent ( xT ) . Note that xt can be directly sampled through the following equation : xt = √ ᾱtx 0 + √ 1− ᾱt , where is a Gaussian noise . ( 2 ) We emphasize that q ( xt|xt−1 ) can be seen as a one-to-one pointwise mapping as xt can be sampled through the equation xt = √ 1− βtxt−1 + βt . Therefore , the order of points in x0 is preserved in the diffusion process . However , it does not matter what kind of order we input the points in x0 . That is because when T is large enough , xT will become a Gaussian distribution . Every point in a Gaussian distribution is equivalent and there is no way to distinguish one point from another . The Reverse Process . The reverse process is a Markov process that predicts and eliminates the noise added in the diffusion process . The reverse process is conditioned on the conditioner , the incomplete point cloud c. Let xT ∼ platent be a latent variable . The reverse process from latent xT to clean data x0 is defined as pθ ( x 0 , · · · , xT−1|xT , c ) = T∏ t=1 pθ ( x t−1|xt , c ) , where pθ ( xt−1|xt , c ) = N ( xt−1 ; µθ ( xt , c , t ) , σ2t I ) . ( 3 ) The mean µθ ( xt , c , t ) is a neural network parameterized by θ and the variance σ2t is a time-step dependent constant . To generate a sample conditioned on c , we first sample xT ∼ N ( 03N , I3N×3N ) , then draw xt−1 ∼ pθ ( xt−1|xt , c ) for t = T , T − 1 , · · · , 1 , and finally outputs x0 . Training . DDPM is trained via variational inference . Ho et al . ( 2020 ) introduced a certain parameterization for µθ that can largely simplify the training objective . The parameterization is σ2t = 1−ᾱt−1 1−ᾱt βt , and µθ ( x t , c , t ) = 1√αt ( xt − βt√ 1−ᾱt θ ( x t , c , t ) ) , where θ is a neural network taking noisy point cloud xt ∼ q ( xt|x0 ) in equation ( 2 ) , diffusion step t , and conditioner c as inputs . Then , the simplified training objective becomes L ( θ ) = Ei∼U ( [ M ] ) , t∼U ( [ T ] ) , ∼N ( 0 , I ) ‖ − θ ( √ ᾱtx 0 i + √ 1− ᾱt , ci , t ) ‖2 , ( 4 ) where U ( [ M ] ) is the uniform distribution over { 1 , 2 , · · · , M } . The neural network θ learns to predict the noise added to the clean point cloud x0 , which can be used to denoise the noisy point cloud xt = √ ᾱtx 0 + √ 1− ᾱt . Note that traditional CD loss or EMD loss is NOT present in Equation 4 . The reason that we are able to use the simple mean squared error is because DDPM naturally defines a one-to-one pointwise mapping between two consecutive point clouds in the diffusion process as shown in Equation 1 . Note that at each training step , we not only need to sample a pair of point clouds xi , ci , but also a diffusion step t and a Gaussian noise .

Summary: In this paper, the authors propose a novel Point Diffusion-Refinement (PDR) paradigm for point cloud completion. The proposed method effectively and efficiently extracts multi-level features from partially observed point clouds to guide completion. Moreover, it accurately manipulates spatial locations of 3D points to obtain smooth surfaces and sharp details. Experimental results demonstrate its state-of-the-art performance on the mainstream datasets and benchmarks.