A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion

1 INTRODUCTION . With the rapid developments of 3D sensors , 3D point clouds are an important data format that captures 3D information owing to their ease of acquisition and efficiency in storage . Unfortunately , point clouds scanned in the real world are often incomplete due to partial observation and self occlusion . It is important to recover the complete shape by inferring the missing parts for many downstream tasks such as 3D reconstruction , augmented reality and scene understanding . To tackle this problem , many learning-based methods ( Yuan et al. , 2018 ; Yang et al. , 2018 ; Tchapmi et al. , 2019 ; Xie et al. , 2020 ; Liu et al. , 2020 ; Pan et al. , 2021 ) are proposed , which are supervised by using either the Chamfer Distance ( CD ) or Earth Mover Distance ( EMD ) to penalize the discrepancies between the generated complete point cloud and the ground truth . However , CD loss is not sensitive to overall density distribution , and thus networks trained by CD loss could generate non-uniform point cloud completion results ( See Figure 10 and 11 in Appendix ) . EMD is more distinctive to measure density distributions , but it is too expensive to compute in training . The absence of an effective and efficient training loss highly limits the capabilities of many existing point cloud completion networks . We find that denoising diffusion probabilistic models ( DDPM ) ( Sohl-Dickstein et al. , 2015 ; Ho et al. , 2020 ) can potentially generate uniform and high quality point clouds with an effective and efficient loss function . It can iteratively move a set of Gaussian noise towards a complete and clean point cloud . DDPM defines a one-to-one pointwise mapping between two consecutive point clouds in the diffusion process , which enables it to use a simple mean squared error loss function for training . This loss function is efficient to compute and explicitly requires the generated point cloud to be uniform , as a one-to-one point mapping is naturally established between the generated point cloud and the ground truth . Point cloud completion task can be treated as a conditional generation problem in the framework of DDPM ( Zhou et al. , 2021 ; Luo & Hu , 2021 ) . Indeed , we find the complete point clouds generated by a conditional DDPM often have a good overall distribution that uniformly covers the shape of the object . Nonetheless , due to the probabilistic nature of DDPM and the lack of a suitable network architecture to train the conditional DDPM for 3D point cloud completion in previous works , we find DDPM completed point clouds often lack smooth surfaces and sharp details ( See Figure 1 and Appendix Figure 12 ) , which is also reflected by their high CD loss compared with state-of-the-art point cloud completion methods in our experiments . Another problem with DDPM is its inefficiency in the inference phase . It usually takes several hundreds and even up to one thousand forward steps to generate a single point cloud . Several methods ( Song et al. , 2020 ; Nichol & Dhariwal , 2021 ; Kong & Ping , 2021 ) are proposed to accelerate DDPM using jumping steps without retraining networks , which however , leads to an obvious performance drop when using a small number of diffusion steps . In this work , we propose the Conditional Point Diffusion-Refinement ( PDR ) paradigm to generate both uniform and high quality complete point clouds . As shown in Figure 1 , our PDR paradigm performs point cloud completion in a coarse-to-fine fashion . Firstly , we use the Conditional Generation Network ( CGNet ) to generate a coarse complete point cloud by the DDPM conditioned on the partial point cloud . It iteratively moves a set of Gaussian noise towards a complete point cloud . Following , the ReFinement Network ( RFNet ) further refines the coarse complete point cloud generated from the Conditional Generation Network with the help of partial point clouds . In addition , RFNet can be used to refine the low quality point clouds generated by an accelerated DDPM , so that we could enjoy an acceleration up to 50 times , while minimizing the performance drop . In this way , the completion results generated by our PDR paradigm demonstrate both good overall density distribution ( i.e . uniform ) and sharp local details . Both CGNet and RFNet have a novel dual-path network architecture shown in Figure 2 , which is composed of two parallel sub-networks , a Denoise subnet and a Condition Feature Extraction subnet for noisy point clouds and partial point clouds , respectively . Specifically , we propose Point Adaptive Deconvolution ( PA-Deconv ) operation for upsampling , which can effectively manipulate spatial locations of 3D points . Furthermore , we propose the Feature Transfer ( FT ) module to directly transmit encoded point features at different scales from the Condition Feature Extraction subnet to the corresponding hierarchy in the Denoise subnet . Extensive experimental results show that our PDR paradigm can provide new state-of-the-art performance for point cloud completion . Our Key contributions can be summarized as : 1 ) We identify conditional DDPM to be a good model with an effective and efficient loss function to generate uniform point clouds in point cloud completion task . 2 ) By using RFNet to refine the coarse point clouds , our PDR paradigm can generate complete point cloud with both good overall density distribution ( i.e . uniform ) and sharp local details . 3 ) We design novel point learning modules , including PA-Deconv and Feature Transfer modules , for constructing CGNet in DDPM and RFNet , which effectively and efficiently utilizes multi-level features extracted from incomplete point clouds for point cloud completion . 4 ) With the help of our proposed RFNet , we can accelerate the generation process of DDPM up to 50 times without a significant drop in point cloud quality . 2 PROBLEM STATEMENT . In this paper , we focus on the 3D point cloud completion task . A 3D point cloud is represented by N points in the 3D space : X = { xj |1 ≤ j ≤ N } , where each xj ∈ R3 is the 3D coordinates of the j-th point . We assume the dataset is composed of M data pairs { ( Xi , Ci ) |1 ≤ i ≤ M } , where Xi is the i-th ground-truth point cloud , and Ci is the incomplete point cloud from a partial observation ofXi . The goal is to develop a model that completes the partial observation Ci and outputs a point cloud as close to the ground truth Xi as possible . For algebraic convenience , we let x ∈ R3N be the vector form of a point cloudX , and similarly c be the vector form of C . 3 METHODOLOGY . We consider the point cloud completion task as a conditional generation problem , where the incomplete point cloud C serves as the conditioner . We use the powerful generative model called denoising diffusion probabilistic models ( DDPM ) ( Sohl-Dickstein et al. , 2015 ; Ho et al. , 2020 ) to first generate a coarse completion of the partial observation . Then we use another network to refine the coarse point cloud to improve its visual quality . Our point cloud completion pipeline is shown in Figure 1 . We first briefly introduce the theory of DDPM in Section 3.1 , and then describe detailed architecture of the Conditional Generation Network ( CGNet ) and ReFinement Network ( RFNet ) in Section 3.2 and Section 3.3 . 3.1 BACKGROUND ON CONDITIONAL DENOISING DIFFUSION PROBABILISTIC MODELS . We assume pdata to be the distribution of the complete point cloud xi in the dataset , and platent = N ( 03N , I3N×3N ) to be the latent distribution , whereN is the Gaussian distribution . Then , the conditional DDPM consists of two Markov chains called the diffusion process and the reverse process . Both processes have length equal to T . We set T = 1000 in this paper . The Diffusion Process . The diffusion process is a Markov process that adds Gaussian noise into the clean data distribution pdata until the output distribution is close to platent . The diffusion process is irrelevant of the conditioner , the incomplete point cloud ci . Formally , let x0 ∼ pdata . We use the superscript to denote the diffusion step t. For conciseness , we omit the subscription i in the following discussion . The diffusion process from clean data x0 to xT is defined as q ( x1 , · · · , xT |x0 ) = T∏ t=1 q ( xt|xt−1 ) , where q ( xt|xt−1 ) = N ( xt ; √ 1− βtxt−1 , βtI ) . ( 1 ) The hyperparameters βt are pre-defined , small positive constants ( See details in Appendix Section A.1 ) . According to Ho et al . ( 2020 ) , there is a closed form expression for q ( xt|x0 ) . We first define constants αt = 1− βt , ᾱt = ∏t i=1 αi . Then , we have q ( x t|x0 ) = N ( xt ; √ ᾱtx 0 , ( 1− ᾱt ) I ) . Therefore , when T is large enough , ᾱt goes to 0 , and q ( xT |x0 ) becomes close to the latent distribution platent ( xT ) . Note that xt can be directly sampled through the following equation : xt = √ ᾱtx 0 + √ 1− ᾱt , where is a Gaussian noise . ( 2 ) We emphasize that q ( xt|xt−1 ) can be seen as a one-to-one pointwise mapping as xt can be sampled through the equation xt = √ 1− βtxt−1 + βt . Therefore , the order of points in x0 is preserved in the diffusion process . However , it does not matter what kind of order we input the points in x0 . That is because when T is large enough , xT will become a Gaussian distribution . Every point in a Gaussian distribution is equivalent and there is no way to distinguish one point from another . The Reverse Process . The reverse process is a Markov process that predicts and eliminates the noise added in the diffusion process . The reverse process is conditioned on the conditioner , the incomplete point cloud c. Let xT ∼ platent be a latent variable . The reverse process from latent xT to clean data x0 is defined as pθ ( x 0 , · · · , xT−1|xT , c ) = T∏ t=1 pθ ( x t−1|xt , c ) , where pθ ( xt−1|xt , c ) = N ( xt−1 ; µθ ( xt , c , t ) , σ2t I ) . ( 3 ) The mean µθ ( xt , c , t ) is a neural network parameterized by θ and the variance σ2t is a time-step dependent constant . To generate a sample conditioned on c , we first sample xT ∼ N ( 03N , I3N×3N ) , then draw xt−1 ∼ pθ ( xt−1|xt , c ) for t = T , T − 1 , · · · , 1 , and finally outputs x0 . Training . DDPM is trained via variational inference . Ho et al . ( 2020 ) introduced a certain parameterization for µθ that can largely simplify the training objective . The parameterization is σ2t = 1−ᾱt−1 1−ᾱt βt , and µθ ( x t , c , t ) = 1√αt ( xt − βt√ 1−ᾱt θ ( x t , c , t ) ) , where θ is a neural network taking noisy point cloud xt ∼ q ( xt|x0 ) in equation ( 2 ) , diffusion step t , and conditioner c as inputs . Then , the simplified training objective becomes L ( θ ) = Ei∼U ( [ M ] ) , t∼U ( [ T ] ) , ∼N ( 0 , I ) ‖ − θ ( √ ᾱtx 0 i + √ 1− ᾱt , ci , t ) ‖2 , ( 4 ) where U ( [ M ] ) is the uniform distribution over { 1 , 2 , · · · , M } . The neural network θ learns to predict the noise added to the clean point cloud x0 , which can be used to denoise the noisy point cloud xt = √ ᾱtx 0 + √ 1− ᾱt . Note that traditional CD loss or EMD loss is NOT present in Equation 4 . The reason that we are able to use the simple mean squared error is because DDPM naturally defines a one-to-one pointwise mapping between two consecutive point clouds in the diffusion process as shown in Equation 1 . Note that at each training step , we not only need to sample a pair of point clouds xi , ci , but also a diffusion step t and a Gaussian noise .

Summary: The paper tackles the problem of shape-level point cloud completion in a fully supervised setting. The big picture of the proposed method is to use a conditional DDPM to generate a noisy but complete point cloud given incomplete input, and then use another refinement network, also conditional on the incomplete input, to further refine the noisy point cloud. Both networks adopt a novel, dual-path network architecture, based on PointNet++, to allow localized guidance from the incomplete point cloud. Thanks to the 2-staged approach, up to 50x acceleration of evaluation speed is achievable using step jumping, with moderate performance drop. The model is evaluated on MVP, MVP-40 and Completion3D datasets.