ComPC: Completing a 3D Point Cloud with 2D Diffusion Priors

## Abstract

Abstract 3D point cloud completion is designed to recover complete shapes from partially observed point clouds. Conventional completion methods typically depend on extensive point cloud data for training with their effectiveness often constrained to object categories similar to those seen during training. In contrast, we propose a zero-shot framework aimed at completing partially observed point clouds across any unseen categories. Leveraging point rendering via Gaussian Splatting, we develop techniques of Point Cloud Colorization and Zero-shot Fractal Completion that utilize 2D priors from pre-trained diffusion models to infer missing regions. Experimental results on both synthetic and real-world scanned point clouds demonstrate that our approach outperforms existing methods in completing a variety of objects without any requirement for specific training data.

###### Keywords:

## 1 Introduction

3D point clouds have always been an important representation for the physical 3D world with extensive use in various applications such as SLAM [] or 3D detection [, ]. However, point clouds are often captured from specific camera viewpoints [, ] in real applications, which may lead to the incompleteness of collected points due to the self-occlusion. Effective and robust completion for partial point clouds can greatly reduce the cost for data collection, and are also useful for subsequent perception of the 3D world.

As illustrated in Fig. 1-(a), most existing completion methods [, , , ] adopt well-designed deep neural networks to directly generate complete point clouds from partial ones. These methods are usually trained on specific point cloud datasets [, ] and demonstrate outstanding performances on their respective test sets. However, their effectiveness is often compromised by the lack of diversity in the training datasets, leading to challenges in handling test time data that differs significantly, *e.g*. unseen object categories or real-world scans. This limitation significantly hinders the practical deployment of these point cloud completion methods.

Leveraging the impressive capabilities of 2D diffusion models [, , ], Kasten et al. [] propose a test-time point cloud completion methods utilizing text-to-3D generative models [, ].
As shown in Fig. 1-(b), this method optimizes a NeuS surface [] guided by Score Distillation Sampling (SDS) of the text-conditioned Stable Diffusion [].
The NeuS surface is modeled as a Signed Distance Field (SDF) to incorporate geometric details from the partial points by setting their SDF values to zero.
The completed points are then generated from the optimized surface for assessment.
By leveraging the extensive 2D knowledge provided by diffusion models, Kasten et al. [] manage to achieve significantly improved robustness in point cloud completion compared to previous network-based methods.
However, a notable limitation of the method proposed by Kasten et al. [] is its dependency on manually created text prompts for each point cloud to guide the completion. This requirement faces a challenge in real-world applications, where providing detailed and accurate text descriptions for incomplete point clouds is not always feasible. Furthermore, the process of optimizing the NeuS surfaces that is integral to this method is time-intensive.

In view of the above-mentioned issues, we propose a novel zero-shot point cloud completion framework that eliminates the need for any extra manually provided information such as text descriptions.
As discussed in PCN [] and Kasten et al. [], existing completion methods concentrate mainly on point clouds incomplete due to self-occlusion. These point clouds would be almost the same as the completed point clouds when observed from a certain reference viewpoint, such as the viewpoint of the LiDAR or depth scanner.
Motivated by the amodal perception [, ], we aim to complete a point cloud with its observation from a reference viewpoint in this work.

As illustrated in Fig. 1-(c), we propose Point Cloud Colorization to estimate this viewpoint and acquire a reference image of the partial point clouds using 3D Gaussian Splatting (GS).
Inspired by the capability of Zero 1-to-3 [] to infer global geometry from a single viewpoint, we use this reference image as a condition for guidance from the diffusion model of Zero 1-to-3 [] to optimize 3D Gaussians.
The missing parts are then completed by the optimization of 3D Gaussians. We propose Preservation Constraint to maintain the geometric integrity of partial point clouds. The completed point clouds are then finally acquired from the 3D Gaussian centers.

Our main contributions can be summarized as below:

- •
We present a zero-shot framework for completing point clouds, leveraging 2D priors from the diffusion model through the technique of 3D Gaussian Splatting;
- •
We propose the Point Cloud Colorization operation to generate a reference image for partial points, which is observed from an estimated reference viewpoint;
- •
Based on the reference image, we develop the Zero-shot Fractal Completion operation to complete missing regions of point clouds;
- •
Through comprehensive evaluation across various data, we demonstrate that our approach surpasses conventional completion methods in handling both synthetic and real-world scanned point clouds.

## 2 Related Works

### 2.1 3D Generation via 2D priors

Since the notable success of 2D diffusion models in text-to-image generation [, , ], text-to-3D and image-to-3D generation have attracted the attention of an increasing number of researchers. However, existing 3D naive diffusion models [, , , , ] are limited by the lack of large-scale 3D datasets, which can usually only be applied to objects from one or a few categories.
In comparison, the abundance and accessibility of 2D data make it feasible to train more unversal diffusion models.
To achieve robust and generalizable 3D generation, several methods propose to lift 2D priors for 3D generation [, , , ].
These works usually optimize specific 3D representations by guidance from 2D diffusion models under different viewpoints, where the guidance is calculated with Score Distillation Sampling (SDS) [] through rendered images.

Zero 1-to-3 [] achieve superior 3D generation quality by using SDS guidance from their pre-trained novel view synthesis diffusion model explicitly conditioned on the reference image and camera transformation. However, its reliance on NeRF representation leads to prolonged optimization times.
Dreamgaussian [] offers a solution by optimizing 3D Gaussian Splatting (GS) [] representations through SDS from Zero 1-to-3, achieving a balance between high-quality outputs and acceptable optimization durations.
Motivated by Dreamgaussian, we recognize the potential of GS to refine 3D coordinates of Gaussian centers using guidance from 2D diffusion models. This insight presents an opportunity to apply 2D priors to tasks related to 3D point clouds.

### 2.2 Point Cloud Completion

Point cloud completion aims to recover completed point clouds from partial input point clouds.
Ever since PCN [] first applied deep neural networks to predict complete point clouds from partial inputs, numerous advancements [, , , , , , ] have been made to enhance the accuracy of point cloud completion by altering network architectures. For example, GRNet [] converts point clouds into grid formats and employs 3D CNNs for predicting the completed structures while PFNet [] adopts a fractal approach to specifically regenerate the missing sections of point clouds. RFNet [] utilizes a differentiable layer to merge new geometrical details into existing partial point clouds. More recent approaches like PoinTr [], Snowflakenet [], Seedformer [], and AdaPoinTr [] integrate transformers to improve completion accuracy by considering broader geometric relationships.

However, performances of these point cloud completion methods are significantly dependent on the diversity of training data, making them less effective for point clouds from unseen categories. Kasten et al. [] proposed an optimization-based completion framework that employs NeuS [] for rendering, drawing on priors from pre-trained text-to-image diffusion models []. This approach maintains the original shapes by constraining the Signed Distance Field (SDF) values of the partial inputs.
Yet, this strategy’s reliance on text-to-image diffusion models for guidance necessitates well-defined text prompts for each partial point cloud, which may not be practical in real-world applications. Moreover, the rendering based on NeuS is relatively time-consuming.

In this study, we propose to leverage 3D Gaussian Splatting (GS) [] to bridge point clouds with 2D priors from diffusion models.
By generating a reference image of the partial point cloud to serve as a condition for guidance from Zero 1-to-3 [], our method can extract uniform and completed point clouds from the 3D Gaussian centers. Since our method exclusively utilizes information gathered from the incomplete point cloud for completion, it eliminates the need for any additional manually specified prompts for each point cloud.

Figure: Figure 2: Illustration of our framework. In Point Cloud Colorization, Reference Viewpoint Estimation estimates a camera pose $V_{p}$ where $P_{in}$ can be most completely observed. $G_{in}$ includes 3D Gaussians initialized from $P_{in}$. We render depth map $D_{in}$ and image $I_{in}$ under $V_{p}$ following DreamGaussian [], where $G_{in}$ is optimized through $I_{in}$ with SDS [] from depth-conditioned ControlNet $f_{c}$ [] conditioned on $D_{in}$. In Zero-shot Fractal Completion, $G_{m}$ begins with an initialization using noisy $P_{N}$ and undergoes optimization guided by view-dependent guidance from Zero-1-to-3 $f_{Z}$ [] based on a randomly chosen camera pose $V_{i}$. Additionally, it incorporates a Preservation Constraint computed with respect to $V_{p}$. The frozen $G_{in}$ is mixed with $G_{m}$ to form $G_{all}$, introducing the partial geometry from $P_{in}$. After optimization, surface points $P_{surf}$ are derived from centers of $G_{all}$, and converted into uniform $P_{out}$ with Grid Pulling module.
Refer to caption: x2.png

## 3 Our Methodology

As shown in Fig. 2, the whole completion process is composed of Point Cloud Colorization and Zero-shot Fractal Completion. For the given partial point cloud $P_{in}$, we first transform it into colorized reference image $I_{in}$ and 3D Gaussians $G_{in}$ with Point Cloud Colorization.
Subsequently, $I_{in}$ and $G_{in}$ are introduced to Zero-shot Fractal Completion to generate completed point cloud $P_{out}$. We use $I_{in}$ to guide the optimization of 3D Gaussians $G_{m}$ for completion, and finally acquire the completed point clouds $P_{out}$ from the centers of Gaussians $G_{in}$ and $G_{m}$. More details are presented below.

### 3.1 Point Cloud Colorization

To introduce 2D priors from pre-trained diffusion models, we use 3D Gaussian Splatting (GS) to achieve differentiable rendering from 3D point clouds to 2D images.
In Point Cloud Colorization, we firstly estimate a reference camera pose $V_{p}$ with the Reference Viewpoint Estimation. 3D Gaussians $G_{in}$ are initialized from $P_{in}$, and then optimized through Depth-conditioned Colorization under camera pose $V_{p}$. Note that the centers of $G_{in}$ are fixed to preserve the shape of $P_{in}$. We only conduct the colorization in this section as the depth-conditioned ControlNet [] cannot infer missing regions.

Modification for 3D Gaussians.
To better satisfy the requirements to colorize a point cloud, we make a few modifications to the original 3D Gaussians: 1) Considering point clouds are observed as multiple equal size spheres, we set the scaling of all 3D Gaussians to a single shared scalar value to keep the shape of Gaussians consistent as points;
2) The opacity for all 3D Gaussians within $G_{in}$ is set to a constant value of 1. This step ensures that Gaussians representing all partial points are nearly opaque and clearly visible during the rendering process.

Reference Viewpoint Estimation.
For any point cloud to be completed, we first determine a reference camera pose $V_{p}$ that captures the most completed observation of the point cloud. The completion process then builds upon this observation. Since the incomplete point cloud $P_{in}$ typically spans across a surface, its most complete view is characterized by minimal self-occlusion and closeness to the camera.

Considering the potential occlusion of rear Gaussians by those in the foreground during rendering,
we implement a filter $h(G_{in},V_{n})$ to identify the indices of the frontmost 3D Gaussians in $G_{in}$ from the camera pose $V_{n}$.
Given that the centers of $G_{in}$ are anchored to $P_{in}$, we can estimate $V_{p}$ by minimizing:

$$ $V_{p}=\arg\min_{V_{n}}\operatorname{CD}(P_{in}[h(G_{in},V_{n})],P_{in})+w_{0} \cdot\operatorname{Dist}(P_{in},V_{n}),$ (1) $$

where $\operatorname{CD}(\cdot,\cdot)$ is the Chamfer Distance [] to measure shape differences between two point clouds. $\operatorname{Dist}(P_{in},V_{n})$ calculates the mean distance of $P_{in}$ from the camera at pose $V_{n}$, and $w_{0}$ is a weighting factor to ensure balance. For this study, we estimate $V_{p}$ by examining 5,000 camera positions uniformly distributed around the partial point cloud.

Depth-conditioned Colorization.
Upon estimating the reference camera pose $V_{p}$, we render a depth map $D_{in}$ and a reference image $I_{in}$ from the 3D Gaussians $G_{in}$. The color $G^{f}_{in}$ and scaling $G^{s}_{in}$ parameters of $G_{in}$ are then optimized using SDS from Depth-conditioned ControlNet $f_{c}$ [] with the Gaussian centers kept fixed.
Denoting $\epsilon_{f_{c}}$ as the noise predicted by the 2D diffusion model $f_{c}$, and with $t$ and $\epsilon$ representing the time step and standard noise, respectively, the SDS process is formulated as follows:

$$ $\nabla_{G_{in}}L_{SDS}=\mathbb{E}_{t,\epsilon}[(\epsilon_{f_{c}}(I_{in};D_{in} ,t)-\epsilon)\frac{\partial I_{in}}{\partial G_{in}}].$ (2) $$

Additionally, we introduce a regularization term for the optimized scaling $G^{s}_{in}$:

$$ $L_{inreg}=w_{1}\cdot|G^{s}_{in}|,$ (3) $$

where $w_{1}$ is the weighting factor.

### 3.2 Zero-shot Fractal Completion

Zero-shot Fractal Completion (ZFC) aims to reconstruct complete and uniform point clouds $P_{out}$ from the colorized reference image $I_{in}$ and 3D Gaussians $G_{in}$. As illustrated in Fig. 2, ZFC optimizes 3D Gaussians $G_{m}$ for completion and is guided by the View-dependent Guidance and the Preservation Constraint.
After optimization, we extract point cloud $P_{surf}$ from centers of 3D Gaussians $G_{all}$ with Gaussian Surface Extraction. Finally, uniform $P_{out}$ is resampled from $P_{surf}$ by the Grid Pulling module.

Modification for 3D Gaussians.
1) We also adjust the scaling of 3D Gaussians as discussed in Sec. 3.1 to ensure the shapes of 3D Gaussians are consistent as points in point clouds.
2) Furthermore, as demonstrated in Fig. 3, the original approach to opacity can lead to a dispersion of Gaussian centers around the actual surface due to the range of opacities $0\prec opacity<1$ used in rendering. To address this problem, we apply a differentiable quantization [] for Gaussian opacity to binarize the values. For 3D Gaussians $G_{m}$ with original opacity $G^{o}_{m}$, the binarization is implemented as follows:

$$ $G^{o}_{m}=f_{stop}(\operatorname{round}(G^{o}_{m})-G^{o}_{m})+G^{o}_{m},$ (4) $$

where

$$ $\operatorname{round}(G^{o}_{m})=\begin{cases}1&\text{if }G^{o}_{m}>0.5,\\ \delta&\text{otherwise}.\end{cases}$ $$

with $f_{stop}(\cdot)$ designed to halt gradient propagation. Here, the forward propagation result of Eq. 4 is $\operatorname{round}(G^{o}_{m})$, while the gradient during backpropagation is calculated based on $G^{o}_{m}$. The parameter $\delta$ is a predefined small constant set to 0.01 in this work because lower opacity may make the Gaussians hard to optimize.
Consequently, 3D Gaussians with $G^{o}_{m}\rightarrow 1$ cluster near the surface as shown in Fig. 3, while those with $G^{o}_{m}\rightarrow 0$ are considered noise and excluded during the Gaussian Surface Extraction process.

Figure: Figure 3: Differences between our binarized opacity and original continuous opacity. $\prec$ denotes smaller but not approaching.
Refer to caption: x3.png

View-dependent Guidance.
To complete the missing regions, we leverage 2D priors from Zero 1-to-3 [] due to its capability to deduce the unseen regions based on available imagery. As illustrated in Fig. 2, we utilize the reference image $I_{in}$ from Point Cloud Colorization to derive the SDS guidance [] based on image $I_{i}$ rendered with Gaussian splatting in a randomly selected viewpoint $V_{i}$, referred to as View-dependent guidance. Defining $\epsilon_{f_{Z}}$ as the noise anticipated by the 2D diffusion model $f_{Z}$ with $t$ and $\epsilon$ indicating the time step and standard noise, respectively, the SDS guidance is calculated as:

$$ $\nabla_{G_{all}}L_{SDS}=\mathbb{E}_{t,\epsilon}[(\epsilon_{f_{Z}}(I_{i};I_{in} ,V_{i},t)-\epsilon)\frac{\partial I_{i}}{\partial G_{all}}].$ (5) $$

For the task of point cloud completion, we adopt a fractal approach as discussed in PFNet [], focusing on optimizing only $G_{m}$ within $G_{all}$ for reconstructing missing regions, while $G_{in}$ remains unchanged to conserve the original geometric characteristic of the partial point clouds $P_{in}$.
Additionally, to manage the scaling $G^{s}_{m}$ of 3D Gaussians $G_{m}$ during optimization, we implement a regularization with a weighting factor of $w_{2}$:

$$ $L_{mreg}=w_{2}\cdot|G^{s}_{m}|.$ (6) $$

Preservation Constraint.
To maintain the geometric shapes of the initial partial point clouds, we introduce Preservation Constraint aimed at reducing the shape differences between the partial point cloud $P_{in}$ and Gaussian center coordinates $P_{pre}$ acquired from the partial observation of 3D Gaussians $G_{all}$ under $V_{p}$.
Utilizing the surface filter $h(\cdot,\cdot)$ presented in Sec. 3.1, and considering $G_{all}$ as the combined set of $G_{m}$ and $G_{in}$ with $P_{G}[\cdot]$ representing the centers of $G_{all}$, the Preservation Constraint is formulated as:

$$ $L_{p}=w_{3}\cdot\operatorname{CD}(P_{G}[h(G_{all},V_{p})],P_{in}),$ (7) $$

where $\operatorname{CD}(\cdot,\cdot)$ is the Chamfer Distance []. $w_{3}$ is the weighting factor. This Preservation Constraint ensures the alignment of $G_{all}$ with $P_{in}$ when observed from $V_{p}$.

Gaussian Surface Extraction.
The centers of the 3D Gaussians can lie both on and below the surface of the shape after optimization.
As a result, it is unsatisfactory to directly use these centers as the complete point cloud.
To address this issue, we introduce a Gaussian Surface Extraction process to select surface points $P_{surf}$ from the centers of 3D Gaussian $G_{all}$.
This procedure is detailed in Alg. 1. By adjusting the opacity of all 3D Gaussians to either $\delta$ or 1, we note that Gaussians with minimal opacity $\delta$ hardly contributes to the rendering process. Consequently, our initial step involves filtering $G_{all}$ based on opacity as outlined in Alg. 1. To this end, $G_{all}$ is examined from several uniformly distributed camera positions and $h(\cdot,\cdot)$ is employed to extract the centers of the frontmost visible Gaussians as the surface points $P_{surf}$.

Figure: Algorithm 1 Gaussian Surface Extraction

Grid Pulling.

Figure: Figure 4: Illustration of the Grid Pulling module. $g(\cdot)$ is a MLP-based SDF learned from the completed point cloud $P_{surf}$. Merge denotes merge layer introduced in []. Given the 3D grids $\mathcal{G}$, $r$ is the diagonal length of a unit grid. Sampled points would be $P_{s}=\{p\mid g(p)<0.5r,p\in\mathcal{G}\}$.
Refer to caption: x4.png

It is evident in Fig. 2 that the density of points in the completed regions of $P_{surf}$ can significantly differ from that in the original partial point clouds. Ideally, we aim for a consistently dense and uniform distribution of points across the entire shape.
Direct attempts to enhance point density within the Zero-shot Fractal Completion (ZFC) framework would lead to a substantial increase in computational cost. Inspired by NeuralPull [], we introduce a Grid Pulling (GP) module designed to resample points uniformly from initially non-uniform point clouds.

NeuralPull [] employs a Signed Distance Field (SDF) $g(\cdot)$ to pull randomly sampled points $P_{sam}$
that are often generated by adding noise to $P_{gt}$ as $P_{sam}=P_{gt}+N(0,\sigma_{0}^{2})$ towards the surface defined by the original point cloud $P_{gt}$.
$\sigma_{0}$ is the standard deviation for normal distribution $N(0,\sigma_{0}^{2})$.
The pulling operation is defined as: $P_{pull}=P_{sam}-g(P_{sam})\cdot\nabla g(P_{sam})/\|g(P_{sam})\|_{2}$. The optimization of $g(\cdot)$ is guided by the Chamfer Distance (CD) as a measure of the distance between $P_{gt}$ and the adjusted points:

$$ $L_{pull}(P_{sam},P_{gt})=\operatorname{CD}(P_{pull},P_{gt}).$ (8) $$

Leveraging $P_{surf}$ obtained from Gaussian Surface Extraction, GP module learns an SDF $g(\cdot)$ to align uniformly sampled points around $P_{surf}$ with its surface. Unlike NeuralPull, which optimizes using only noised point clouds, our approach trains $g(\cdot)$ with both noised point clouds $P_{near}=P_{surf}+N(0,\sigma_{0}^{2})$, and $P_{far}$ being randomly sampled within the 3D bounding box encompassing $P_{surf}$. The loss functions are defined as $L_{far}=L_{pull}(P_{far},P_{surf})$ and $L_{near}=L_{pull}(P_{near},P_{surf})$.

Additionally, we utilize a merge layer as suggested in [] to incorporate geometric details from $P_{in}$ into $P_{pull}$.
Given the distances from $P_{pull}$ points to their nearest neighbors in $P_{in}$ as $dist=\min_{x\in P_{pull},\forall y\in P_{in}}\|x-y\|_{2}$, and corresponding neighbor indexes $idx=\arg\min_{x\in P_{pull},\forall y\in P_{in}}\|x-y\|_{2}$, the merge layer $g_{m}$ outputs a set of merged points:

$$ $g_{m}(P_{pull},P_{in})=e^{-\frac{dist}{\sigma}}P_{in}[idx]+(1-e^{-\frac{dist}{ \sigma}})P_{pull},$ (9) $$

where $\sigma$ is a small optimizable variable to decide how much to merge. The corresponding loss would be $L_{mer}=L_{pull}(g_{m}(P_{pull},P_{in}),P_{surf})+w_{4}\cdot\|\sigma\|_{2}$, where $w_{4}$ is the weighting factor for the regularization of $\sigma$.
The overall training loss for $g(\cdot)$ is then:

$$ $L_{g}=L_{far}+L_{near}+L_{mer}.$ (10) $$

As depicted in Fig. 4, we initialize a $128^{3}$ 3D grid $\mathcal{G}$ according to the bounding box of $P_{surf}$. Uniform points $P_{s}$ would be selected by $P_{s}=\{p\mid g(p)<0.5r,p\in\mathcal{G}\}$. $P_{s}$ is then pulled to the surface of $P_{surf}$ and combined with $P_{in}$ through merge layer.
The final output point clouds is $P_{out}=g_{m}(g(P_{s}),P_{in})$.

Figure: Figure 5: Qualitative comparison on synthetic data.
Refer to caption: x5.png

**Table 1: Quantitative comparison on synthetic data. Bold marks the best results.**
| Object | Horse | MaxPlanck | Armadillo | Cow | Homer | Teapot | Bunny | Nefertiti | Bimba | Ogre | Aver |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metrics | CD/EMD | CD/EMD | CD/EMD | CD/EMD | CD/EMD | CD/EMD | CD/EMD | CD/EMD | CD/EMD | CD/EMD | CD/EMD |
| PoinTr | 2.75/4.47 | 6.34/6.84 | 3.51/6.07 | 3.13/4.25 | 1.90/4.19 | 3.81/5.12 | 6.39/8.03 | 4.29/5.50 | 5.53/6.73 | 3.41/5.06 | 4.10/5.63 |
| Snowflake | 3.34/5.26 | 6.33/7.18 | 3.03/6.13 | 2.61/4.27 | 1.89/3.53 | 3.22/4.69 | 6.99/8.81 | 4.15/5.94 | 5.36/7.11 | 3.18/5.46 | 4.01/5.84 |
| SeedFormer | 3.24/5.30 | 6.91/7.62 | 3.28/6.21 | 3.11/4.00 | 2.04/3.52 | 3.41/4.94 | 6.92/9.10 | 4.25/5.78 | 5.63/7.09 | 3.31/5.73 | 4.21/5.93 |
| AdaPoinTr | 4.88/5.45 | 8.60/8.51 | 5.14/5.95 | 3.48/4.53 | 2.28/3.34 | 3.92/4.56 | 9.33/8.87 | 5.54/6.14 | 8.16/7.64 | 4.53/5.41 | 5.59/6.04 |
| Ours | 1.32/1.84 | 1.57/1.98 | 1.90/3.30 | 2.83/2.00 | 1.81/2.35 | 1.02/1.39 | 1.50/1.93 | 1.95/2.30 | 1.92/2.34 | 2.70/3.83 | 1.75/2.33 |

## 4 Experiments

To assess the efficacy of our method, we conduct comparisons on objects derived from both synthetic datasets and real-world scans. For synthetic datasets, we generate partial point clouds by sampling from various viewpoints around completely modeled objects from established sources [, , , ]. For the real world scans, we follow Kasten et al. [] for the settings of Redwood dataset []. Single scans are used as partial input while the ground truths are adopted by composing multiple scans. We standardize all point clouds following the processing of PCN [].

We compare our approach with state-of-the-art methods such as Snowflakenet [], PoinTr [], Seedformer [], and AdaPoinTr []. The evaluation metrics include the Chamfer Distance (CD) and Earth Mover’s Distance (EMD) [] that measure the similarity between the reconstructed point clouds and the ground truths. All metrics are multiplied with $10^{2}$ in subsequent comparisons.
These comparisons are all performed at a resolution of 16,384 points following PCN [].

Figure: Figure 6: Qualitative comparison on real scans from Redwood dataset [, ].
Refer to caption: x6.png

**Table 2: Quantitative comparison on Redwood dataset [, ]. Bold marks the best results.**
|  | In Domain | Out Domain |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Object | Table | Exe-Chair | Out-Chair | Old-Chair | Average | Vase | Off Can | Vespa | Tricycle | Average |
| Metrics | CD/EMD | CD/EMD | CD/EMD | CD/EMD | CD/EMD | CD/EMD | CD/EMD | CD/EMD | CD/EMD | CD/EMD |
| PoinTr | 3.56/7.42 | 1.91/4.50 | 0.67/1.41 | 2.48/6.28 | 2.16/4.90 | 3.84/6.55 | 3.76/5.83 | 1.84/3.94 | 5.91/11.63 | 3.84/6.99 |
| Snowflake | 3.65/7.54 | 2.00/4.83 | 0.80/1.58 | 2.39/5.08 | 2.21/4.76 | 3.82/6.15 | 3.33/5.75 | 2.29/4.47 | 1.77/3.50 | 2.80/4.96 |
| SeedFormer | 3.38/7.01 | 1.90/4.55 | 0.76/1.44 | 2.72/5.16 | 2.19/4.54 | 3.99/6.36 | 3.88/6.11 | 2.38/4.38 | 2.10/3.38 | 3.09/5.06 |
| AdaPoinTr | 5.02/6.23 | 2.58/4.79 | 0.82/1.38 | 3.62/5.61 | 3.01/4.50 | 5.14/6.48 | 4.47/6.32 | 1.94/3.52 | 1.83/3.67 | 3.34/4.98 |
| Ours | 1.73/3.27 | 1.23/1.89 | 1.37/2.09 | 1.97/2.84 | 1.57/2.52 | 2.88/4.29 | 3.17/3.67 | 1.62/2.52 | 2.66/2.51 | 2.58/3.25 |

### 4.1 Comparison on Synthetic Point Clouds

In this section, we conduct an evaluation on synthetic point clouds. The quantitative and qualitative results are presented in Table 1 and Fig. 5, respectively. We can see that our method significantly outperforms other methods. As shown in Fig. 5, our method can generate accurate and reasonable completed results, which may benefit from the abundant 2D priors from diffusion models.
An interesting case is that our method learns to complete an appropriate handle for the teapot in the first row of Fig. 5, even without any prompts and related geometrical structures. It confirms that the pipeline can actually percept the actual categories of completed objects instead of simply inferring a shape to fill in the missing regions.

Figure: Figure 7: Comparison between our methods with or without colorization. $I_{in}$ and $P_{out}$ denote the colorized reference image and completed point clouds, respectively.
Refer to caption: x7.png

Figure: Figure 8: Ablation study for Zero-shot Fractal Completion (ZFC). Surf, Pres, and GP denote Gaussian surface extraction, Preservation Constraints, and Grid Pulling module in ZFC, respectively.
Refer to caption: x8.png

Figure: Figure 9: Ablation study for Grid Pulling module. Near, and Merge denote the $L_{near}$, and merge layer $g_{m}(\cdot)$, respectively. W/O Near means only $L_{far}$ is adopted for optimization.
Refer to caption: x9.png

Figure: Figure 10: Ablation Study for the 3D Gaussian modifications. Scaling and Opacity denotes the parameter-shared scalar scaling and binary opacity operations mentioned in Sec. 3.2, respectively.
Refer to caption: x10.png

Figure: Figure 11: Ablation Study for the fractal completion strategy. W/O Fractal and W/ Fractal denote using and not using Fractal completion strategy, respectively.
Refer to caption: x11.png

### 4.2 Comparison on Real Scans

We follow the settings of Kasten et al. [] for comparison on real scans from Redwood [].
Following Kasten et al. [], scans are divided into the "in domain" categories included in the training datasets of existing completion networks [, , , ], and "out domain" categories never seen during their training.
The qualitative and quantitative comparison results are illustrated in Fig. 6 and Table 2, respectively.
As shown in Table 2, our method outperforms other methods on both "in domain" and "out domain" models, which further confirms the effectiveness and genelizability of our method.

### 4.3 Effect of Colorization

To validate the effect of colorization in the whole completion framework, we conduct an ablation study by removing the colorization from the framework. Removing the colorization means that we directly use the depth image $D_{in}$ as the reference image $I_{in}$ without colorization from depth-conditioned ControlNet [].
The results are presented in Fig. 7. We can see that defects show up in circled regions after removing the colorization. The reason may be that the original depth images lack of textures, where the colorization with pre-trained depth-conditioned ControlNet adds clearer textures to the initial view. It is beneficial for subsequent view dependent guidance in ZFC.

### 4.4 Ablation Study for Zero-shot Fractal Completion

Abation for components in ZFC. Zero-shot Fractal Completion consists of view dependent guidance, Gaussian surface extraction, Preservation Constraint, and Grid Pulling module.
From Fig. 8, we can see that our ZFC with all components have uniform and reasonable completed results.
GP obviously generates quite uniform point clouds from the non-uniform completed ones directly acquired from 3D Gaussians.
The Preservation Constraint avoid redundant shapes by introducing strict constraints between partially observed points and existing partial point clouds.
We can also see that the Gaussian Surface Extraction operation effectively extracts the surface from relatively disorganized Gaussian centers.

Ablation for Grid Pulling module
Grid Pulling (GP) module is proposed to resample uniform and regular point clouds from non-uniform $P_{surf}$. As claimed in Sec. 1, $L_{far}$ and $L_{near}$ are used to optimize an continuous surface presented by MLP while Merge layer is introduced to merge output point clouds with partial input.
From results in Fig. 9, we can observe that $L_{far}$ and $L_{near}$ contribute to overall contours and local shapes, respectively. Nonetheless, they are still limited to the over-smoothed results.
The merge layer helps preserve local geometrical details in the circled regions from the partial input point cloud.

### 4.5 Ablation Study for Modifications of 3D Gaussians

As presented in Sec. 3.2, we make a few modifications to the original 3D Gaussian Splatting [] including the parameter-shared scalar scaling definition and binary opacity estimation. In this section, we conduct a few experiments to validate the effectiveness of these proposed operations.
To better illustrate their performances, we conduct comparisons based on $P_{surf}$ directly acquired from the Gaussian centers in Fig. 10.

We can see that adopting a shared scalar scaling helps in revealing more defined geometric details in the point cloud completion task.
The original settings of separate scaling across different 3D Gaussians tend to produce blurring edges and lose finer details.
In addition, the binary opacity operation obviously reduce the noises in $P_{surf}$. With the original opacity settings, a considerable number of 3D Gaussians with moderate opacity values would scatter around the actual surfaces, blurring the distinction between the object and its surroundings.
The binary opacity method effectively eliminates this issue, ensuring a cleaner bounding and more accurate surface representation.

### 4.6 Effect of the Fractal Completion Strategy

As illustrated in Fig. 2, we introduce the fractal completion strategy in ZFC by optimizing 3D Gaussians $G_{m}$ together with frozen 3D Gaussians $G_{in}$ initialized from partial point clouds. In this section, we conduct experiments to verify the effect of this strategy. A few visualized examples are presented in Fig. 11.
When not using fractal completion strategy, we directly optimize 3D Gaussians $G_{m}$ for all structures without concatenation with $G_{in}$.
We can see that completions without fractal completion strategy tend to ignore some shape details in existing partial point clouds.

## 5 Conclusion

In this work, we propose a zero-shot point cloud completion framework making use of abundant 2D priors from diffusion models [, ] through 3D Gaussian Splatting rendering.
Unlike the text-driven completion methods [], our method does not have any requirements for extra manual prompts.
The whole completion is composed of Point Cloud Colorization and Zero-shot Fractal Completion (ZFC). In Point Cloud Colorization, we propose Reference Viewpoint Estimation and Depth-conditioned Colorization to estimate a reference image of the partial point cloud.
Subsequently, we introduce ZFC to complete missing regions of the partial point cloud by optimizing 3D Gaussians through the View-dependent Guidance conditioned on the reference image.
Finally, we extract the completed point cloud from 3D Gaussians with Gaussian surface extraction, and resample it with Grid Pulling module to a uniform result.
According to our experiments, our method achieves better performances than existing network-based completion methods, with great robustness on both synthetic and real scanned point clouds.

## 6 Limitation

While our approach successfully completes point clouds from unseen categories, it requires a separate optimization process for each point cloud to integrate 2D priors from diffusion models. This makes the completion process slower than existing network-based methods, which is a common issue for all optimization-based methods such as the one mentioned in Kasten et al. [].
However, this challenge may be resolved by training a genelizable network from 2D priors. We will explore it in our future work.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - As noted in the **Weaknesses** section, this method incorporates the input point cloud across all stages of the pipeline. It would be beneficial to examine how the method performs under varying occlusion levels and perturbation intensities.
- Following the above, could the method leverage multiple reference viewpoints during the ZFC step? Would incorporating additional viewpoints lead to more accurate completion of point clouds, particularly in cases with high noise levels and significant occlusions?
- Could this method benefit from adopting 3D scene representations such as VolSDF [1], as used in SDS-complete, which implicitly encodes the SDF of the underlying surface?
- Clarity: 
  - It’s important to clarify that the SDS-complete method leverages VolSDF [1], rather than NeuS, to represent the 3D scene.
  - Adding a horizontal separator line after the Metrics row in Tables 1, 2, and 9 would improve readability. 
  - Please also add the SeedFormer method in line 418.

[1] L. Yariv et al., “Volume Rendering of Neural Implicit Surfaces”, In Proc. NeurIPS, 2021.
soundness: 3
strengths: The proposed framework generates complete point clouds that accurately capture the underlying surface, entirely bypassing the need for training on extensive 3D shape categories. By leveraging a 2D diffusion model to synthesize novel views from a specific input viewpoint, it also removes the need for manual text prompts. Furthermore, this approach achieves superior or comparable performance to other test-time frameworks (e.g., SDS-complete) while significantly reducing computational time. Finally, the adaptation of the NeuraPull approach, using near and far points from the 3D bounding box of the $P_{surf}$ point cloud, while also incorporating geometric details from the input partial point cloud, is able to produce uniformly-sampled point clouds.
summary: This paper addresses the problem of 3D point cloud completion from partial and noisy data, specifically in an open-set setting, where point clouds from unseen categories are completed. The proposed approach introduces a novel, training-free framework that leverages 3D Gaussian Splatting (3DGS) representations and pre-trained 2D diffusion models for guiding the completion process.

First, a Partial Gaussian Initialization (PGI) module estimates a reference viewpoint, $V_p$, that maximizes coverage of the input partial point cloud, $P_{in}$. This viewpoint is used to initialize partial 3D Gaussians, $G_{in}$, and render an image $I_{in}$. The Zero-shot Fractal Completion (ZFC) module then combines $G_{in}$ with another set of 3D Gaussians, $G_m$, which is generated from a noisy variant of the input point cloud, $P_m$. Using the Zero-1-to-3 diffusion model, $G_m$ is refined via Score Distillation Guidance based on the reference viewpoint image $I_{in}$, which optimizes the Gaussians based on a randomly rendered image $I_i$.  A geometric preservation constraint is also introduced by measuring the Chamfer Distance between the input point cloud and the predicted point cloud, $P_{pre}$the predicted one $P_{pre}$ under the reference viewpoint $V_p$. 

Finally, the Point Cloud Extraction (PCE) module extracts surface points, $P_{surf}$, from the 3D Gaussian centers, $G_{all} = {G_{in}, G_m}$, excluding those with low opacity and selecting the ones that are visible from a set of uniform camera poses. To address the non-uniform density of $P_{surf}$, a Grid Pulling step is applied that promotes uniform alignment of $P_{surf}$ along the underlying surface. The framework is evaluated on synthetic and real-world point clouds for the point cloud completion task, achieving superior or comparable performance w.r.t. existing methods.
weaknesses: Although this method claims to rely solely on 3D coordinates, as required by the point cloud completion task, it implicitly introduces additional information by incorporating the 3D normal of each point to encode the color map $G_{in}^c$. This additional input signal, which is not utilized by other methods such as PCN or SDS-complete, may introduce an advantage, as evidenced by its significant impact on performance shown in Table 3 and Figure 7. Furthermore, the method heavily depends on the input partial point cloud throughout all stages: in the ZFC module, the SDS guidance is conditioned on the reference viewpoint image derived from the input partial point cloud, and a geometric preservation constraint is based on the original point cloud. In the PCE module, the initial point cloud is merged with $P_{pull}$ in the Grid Pulling step, further reinforcing reliance on the input partial point cloud.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: All of my questions are listed in the weaknesses section, and I may adjust the rating if they are well addressed.
soundness: 3
strengths: a) The problem of completing partial point clouds in a generalizable way is important, as existing methods mostly yield good results only on in-domain samples. b) 2D/3D priors from diffusion models are well-suited for achieving generalization. c) The paper is well-written, the problem is clearly specified and the solution is well presented.
summary: The paper introduces a test-time point cloud completion framework that leverages 2D diffusion priors. Unlike previous methods, which require manually provided information such as text prompts, the proposed method conditions the diffusion model on a reference image from the partial observation. After initializing the partial Gaussians and rendering the reference image, the framework optimizes the parameters of the Gaussians representing the 'missing' parts and then extracts the completed point cloud. The approach is validated on synthetic and real-world scanned point clouds.
weaknesses: a) As the incompleteness of the partial point cloud increases, the diffusion model may produce different completion results at each optimization. However, the multi-modal nature of the approach is not analyzed. For a small set of shapes, multiple runs can be performed on the same partial input, and the resulting completions can be evaluated using multi-modal metrics such as TMD, UHD, MMD. b) The experiment in A.5 could be performed on observations with varying levels of incompleteness, rather than artificially eliminating points, to better demonstrate the method’s actual performance. For instance, depth maps could be used to obtain incomplete observations by merging points from a different number of views. c) The paper does not include a comparison with recent related work such as [1, 2]. The authors are suggested to compare ComPC with also these recent baselines to better evaluate its completion and generalization performance.  

[1] Chu, R., Xie, E., Mo, S., Li, Z., Nießner, M., Fu, C. W., & Jia, J. (2024). Diffcomplete: Diffusion-based generative 3d shape completion. Advances in Neural Information Processing Systems, 36.
[2] Yan, X., Lin, L., Mitra, N. J., Lischinski, D., Cohen-Or, D., & Huang, H. (2022). Shapeformer: Transformer-based shape completion via sparse representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6239-6249).

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: It might be interesting to compare the density of completed point clouds and compare it with GT density. What is the reason for the density difference? In the section 4.4. of ablation study we can see that point cloud extraction causes a kind of separate clusters. What is the reason?

It is recommended to provide a possibility to test the pipeline on a custom dataset. Moreover, it is recommended to split Figure 2 into two-three separate pictures with clarified steps description. The united scheme makes understanding the pipeline hard.
soundness: 3
strengths: Learning-based methods perform not good when tested out of the domain they were trained on. The strong point of the proposed method is that authors propose a framework that performs out of the training domain as good as in it. Moreover, the model does not require any additional modalities, so the possibility of application is maximal. Lastly, there are empirical experiments done that prove that proposed method outperforms overviewed analogical approaches. The ablation study is well done: this part of the paper is split into categories and each one is devoted to the specific part of the pipeline.
summary: This paper proposes a new pipeline for point cloud completion. The proposed pipeline consists of three steps. The pipeline initializes a single optimal viewpoint for initial render of point cloud. This render is made so the point cloud is most completely observed. The Gaussian Splatting is used to initialize a frozen set of 3D gaussians by utilizing an optimal viewpoint. In the next step the set 3D gaussians are prepared in the optimal way to cover missing point cloud regions. Further Zero 1-to-3 approach is used to complete missing parts of the point cloud. On the final step the point cloud is extracted from the 3D gaussians distributed uniformly (unlike in the start of pipeline) on the shape surface.

Despite the weaknesses, I recommend accepting this work because the authors definitely introduced a new strong pipeline that incorporates many existing methods. Authors proved that the proposed sequence of methods is successfully used for the important complicated task.

The main reason to accept this work is good results that superior existing point cloud completion methods based on single-modality input data. The method shows good performance in the training domain and out of the training domain. In some cases (depicted in the paper) the density of points seems to be more uniform than in GT point clouds. This feature can be useful in tracking and segmentation tasks.
weaknesses: Despite the strengths of the work, there are a number of questions about it. There is a method of missing parts completion in section 3.2. Many side methods like Zero 1-to-3, SDS guidance, PFNet are referenced in this section while there are very few particular discussions that are taken from these works. For example, what is the fractal approach discussion taken from PFNet? The second question is: what is the point of applying differentiable quantization to set the gaussians' opacity? It seems that there is a quite simple solution. Another weakness of the proposed work is the absence of the possibility to test the pipeline. The source code that aggregates the mentioned methods can clarify the approach that authors suggest.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. I can't see how your work can generate a surface like SDS-complete?
soundness: 3
strengths: They continue works that complete point clouds in the wild and challenge the current weakness of long-running time and the need for textual description.
The work exploits advances in 3D generation for further improvement in point cloud completion.
summary: The work tries to complete a general partial point cloud using  zero123, which gets an object's image and outputs images of the exact object from any unseen direction.
They first model the partial point cloud as gaussians, and render them from the view direction, where the object seems full.
than through SDS iterations using zero123 they complete the object from any noval direction.
weaknesses: good work

Meta-review: This paper introduces an approach for point cloud completion. Instead of data-driven approaches based on manually organized training sets, the proposed approach utilizes 2D diffusion priors. The proposed approach is not affected by the conventional 3D dataset, which is often manually organized or biased toward certain categories. The 3D point clouds are represented with 3D Gaussians and completed with 2D diffusion priors, and the final point clouds are extracted uniformly. Given the compelling results and detailed description of the proposed approach, AC confirms that this paper can shed light on the possibility of the practical use of diffusion priors for dense prediction tasks. However, handling non-object-centric point clouds or recovering detailed surfaces seems to be a future work of this approach.

Decision: Accept (Poster)