Point Cloud Completion with Pretrained Text-to-Image Diffusion Models
Authors: Yoni Kasten
Ohad Rahamim
Gal Chechik
Venue: NeurIPS 2023 poster (2023)
Keywords: Point Cloud
Text
3D

Abstract: Point cloud data collected in real-world applications are often incomplete. This is because they are observed from partial viewpoints, which capture only a specific perspective or angle, or due to occlusion and low resolution. Existing completion approaches rely on datasets of specific predefined objects to guide the completion of incomplete, and possibly noisy, point clouds. However, these approaches perform poorly with Out-Of-Distribution (OOD) objects, which are either absent from the dataset or poorly represented. In recent years, the field of text-guided image generation has made significant progress, leading to major breakthroughs in text guided shape generation. We describe an approach called SDS-Complete that uses a pre-trained text-to-image diffusion model and leverages the text semantic of a given incomplete point cloud of an object, to obtain a complete surface representation. SDS-Complete can complete a variety of objects at test time optimization without the need for an expensive collection of 3D information. We evaluate SDS-Complete on incomplete scanned objects, captured by real-world depth sensors and LiDAR scanners, and demonstrate that is effective in handling objects which are typically absent from common datasets.

Point-Cloud Completion with Pretrained Text-to-image Diffusion Models
Yoni Kasten
Ohad Rahamim²
Gal Chechik!:2 1,2
NVVIDI Research
Bar--llan University
Abstract
Point-cloud data collected in real-world applications are often incomplete, be- cause objects are being observed from specific viewpoints, which only capture one perspective. Data can also be incomplete due occlusion and low-resolution sampling. Existing approaches to completion rely on training models with datasets of predefined objects to guide the completion of point clouds. Unfortunately, these approaches fail to generalize when tested on objects or real-world setups that are poorly represented in their training set. Here, we leverage recent advances in text-guided 3D shape generation, showing how to use image priors for gen- erating 3D objects. We describe approach called SDS Cos-Coeelete that uses pre-trained text-to-image diffusion model and leverages the text semantics of a given incomplete point cloud of an object, to obtain complete surface representa- tion. SDS DDS-Compeete can complete a variety objects using test-time optimization without expensive collection of 3D data. We evaluate SDS-Complete lection of incomplete scanned objects, captured by real-world depth sensors and LiDAR scanners. We find that it effectively reconstructs objects that absent from common datasets, reducing Chamfer loss by about 50% on average compared with current methods. Project page: hhtps://sds-comples cotplete..geeebu.bbiie. iol
col-
SDS Complete
SDS Complete
Partial point- cloud
Partial point- cloud
"A Chair"
"A teapot"
We present SDS-Complete: A test-time optimization method for completing point clouds captured by depth sensors, leveraging pre-trained text-to-image diffusion model. The inputs to our method are an incomplete point cloud (blue) along with a textual description the object. The output is a complete surface (gray) that is consistent with the input points (blue). The method works well on a variety of objects captured by real-world point-cloud sensors.
Figure 1:
1 Introduction
Modeling 3D objects and scenes is becoming a central part machine perception. Most 3D data is collected using sensors that capture the 3D structure of various objects, like LIDAR depth scanners.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

When used in the real world, various factors may cause scanners to capture incomplete partially sampled 3D objects. First and foremost, objects are often captured from specific camera viewpoints, collecting points from only "one side" of an object (Figure 1). Reconstruction may also suffer from self-occlusions and low sensor resolution. To fully understand the three-dimensional world, one must deal with partial data and missing parts.
Current approaches for completing partial point clouds [58 50 30
56 49 operate as follows. 30 53
They first gather extensive training datasets of complete 3D objects ge.g. 9); then, they extract partial point clouds from the 3D objects as training data; finally, they train deep models to predict completed objects using their known ground -ruth geometry. These methods are commonly evaluated on test partitions derived from their training data and consistently show high accuracy. However, they tend to generalize poorly to real data, due to several reasons. First, the training data primarily consists of Computer-Aided Designed (CAD) models, which differ from real-world objects [.] Second, points are sampled in an artificial way, which does not accurately simulate real- world capture processes. Finally, the training data is mostly aligned [5T], which can pose challenges when dealing with real-world data that is typically not perfectly aligned with the training data. The performance of these models deteriorates even further for objects and shapes that were not observed during training (label shift). This is severe problem because the diversity of shapes in existing datasets of 3D objects is very limited. Recent work (56) attempted to address this limitation by expanding the range of object categories. However, the quality of completed objects is significantly decreased when applied to real-world data, as we illustrate below. The poor generalization of these methods to real data limits their practical use in real-world situations that demand 3D perception, such indoor-scene reconstruction and autonomous driving.
Here, we address the challenge of completing 3D objects in the wild from real-world partial point clouds. This is achieved by leveraging priors about object shapes that are encoded in pretrained text-to-image diffusion models. Our key idea is that since text-to-image diffusion models were trained on a vast number of diverse objects, they contain a strong prior about the shape and texture of objects, and that prior can be used for completing object missing parts. For example, given a partial point cloud, knowing that it corresponds to a chair can guide the completion process, because objects from this class are expected to exhibit some types of symmetries and parts that are captured in 2D images.
similar intuition has been used for generating 3D objects "from scratch" (DreamFusion) 139 DreamFusion uses the SDS loss, which measures agreement between 2D model prior and renderings of the 3D shape. Unfortunately, naively applying the SDS loss to our problem of point cloud completion fails. This is because, as show below, it does not combine well the hard constraints implied by the points collected from the sensor with the prior embedded in the diffusion model.
To address these challenges, we introduce SDS-Complete: a method to complete a given partial point cloud using several considerations. First, we use Signed Distance Function (SDF) surface representation [37] I8 5 (55), and constrain the zero level set of the SDF to go through the input points. Second, we use information about areas with no collected points, to rule out object parts in these areas. Third, we use prior about camera position and orientation and curriculum of out- out-painting when sampling camera positions. Finally, we use the SDS loss 39 to incorporate prior guided by the class of an object on the rendered images.
We demonstrate that SDS-Complete generates completions for various objects with different shape types from two real-world datasets: the Redwood dataset [12). which contains various incomplete real-world depth camera partial scans, and the KITTI dataset [8, which contains object LiDAR scans from driving scenarios. In both cases, SDS DDS-Complete outperforms the current state-of-the-art methods.
In summary, this paper makes the following contributions: (1) A formulation of point cloud com- pletion as test-time optimization problem, avoiding the cost of collecting large datasets of 3D) geometries and training models. (2) A new approach to PC completion, which combines an empirical point-cloud, with image priors using an SDF surface representation. approach to completing and preserving real-world captured 3D content from various depth sensors, (LiDAR or depth camera) all while incorporating prior knowledge of camera poses through well- structured camera curriculum. (4) We demonstrate state-of-the-art completion results for diverse in-the-wild objects, captured by real-world sensors.
(3) A practical and unified
2

---

2 Related work
Surface Completion from Point Clouds. Over the last years approaches based on deep-networks
588
50
30
38 have demonstrated remarkable capabilities in reconstructing objects from incomplete
46 used voxel grid representations of 3D geometries due to their straightforward processing with off-the-shelf 31) convolutional layers. While voxels proved to be useful, they suffer from a space complexity issue, as their representation grows cubically. Consequently, these methods can only generate shapes with limited resolution. In contrast, point cloud representations 15 2 geometries using neural networks. Several methods [7) completed point cloud given a partial input point cloud. However, to obtain a surface representation, surface reconstruction technique [24] needs to be applied as post-pun posseprocessing step, which can introduce additional errors. Recently, an alternative approach has emerged where the output surface represented using neural representations (30 37). The advantage of these representations lies in their ability to represent the surface continuously without any discretization. (30) (3) trained deep
or partial inputs. Early attempts with neural networks 13 14
19
have been leveraged to model higher-resolutin
such techniques for predicting the
neural networks and latent conditioned implicit neural representations on dataset of predefined object classes [ to perform point cloud completion.
While most deep methods for surface completion train a different model per object class, very recent methods have focused on training multi-class models, allowing for better generalization [53] [5).. PoinTr [56] uses transformer encoder-decodee architecture for translating given input point cloud into a set of point proxies. These point proxies are then converted into completed point clouds using FoldingNet [54]. ShapeFormer [53] directly reconstructs surfaces from incomplete point clouds using a transformer. Recently, []] combined point cloud and image inputs for completing the object shape. loss on the rendered output.
They further use the input image for applying
Other recent works I 27 34] show progress in the task of shape completion given a partial surface, where (34] uses a transformer and autoregressive modeling, and 27 employ diffusion processes that allow controlling the completion with text. However, these methods require a surface as input and cannot handle incomplete point clouds. Furthermore, their applicability is limited to the domain they are trained on.
In contrast to the test-time optimization process using pre-trained available diffusion models, does not rely on any collection of 3D shapes for training, and works on much broader domains.
methods, our method performs point cloud completion as
3D models from text using 2D supervision. Several approaches used large vision-and-language models like CLIP 40 to analyze and synthesize 3D objects. Text2Mesh [32], CLIP-Mesh (25] and present approaches for editing meshes, generating 3D models, and synthesizing NeRFs 33 respectively, based on input text prompts. The methods employ differentiable renderers
DreamFields 23
to generate images while maximizing their similarity with the input text in CLIP space.
More directly relevant to the current paper are methods that build on diffusion models. Text-guided [6] generate images based text prompts, enabling control over the generated visual content. These 2D models can then be used to guide 3D object generation, an
image diffusion models 43 44
6
approach first presented by DreamFusion [39] and then further improved (28) Latent-NeRF (31 enables DreamFusion to run with higher-resolution images by optimizing NeRF with diffusion model features instead of RGB colors. TEXTure [42] and Text2Tex (10) use depth-aware text-to-image diffusion models to synthesize textures for meshes. Other recent works predict shapes directly from 2D images [41 4 29]. In contrast, our method uses the input text for completing partial point clouds,
rather than editing or synthesizing 3D content.
3 Method
3.1
Problem Setup
We address the problem of completing a surface given incomplete point cloud measurements, captured by real point cloud sensor. In contrast to previous works for object completion [56] 53 that first trained one feed-forward model on a large dataset of 3D shapes, we operate in sost-iime-pptimzaaiion and solve each object separately from scratch without using any 3D dataset for pre-training.
3

---

Inputs
Representation
Losses
Sensor-compatibility Paints loss (p.)
Neural Surface (Signed Distance Function) Represented using Co. Optimize
Absence Loss (P. Distance Loss L, fb)
Incomplete Point cloud
fo
Im
Stable D
"A sofa"
Imt
Img
View selector Radiance field
Diffusion
Text-compatibility
Text prompt
Lus (y, Jm.)
rendering
Figure SDS-Complete optimizes two neural functions: signed distance function fo representing the surface and volumetric coloring function Cp. Together, (Cpp fo) define radiance field, which used to render novel image views /mo..... Im, The SDS loss is applied to the renderings to encourage them to be compatible with the input text y. Three sensor-compatibiliyy losses verify that reconstructed surface compatible with the sensor observations in various aspects.
Inputs and components of our system. The overall scheme for method is depicted in Fig. Two inputs are expected, (Fig. Ieftt): a set of 3D input points = {P1: P2..... Px measured relative to the sensor's location, and text description embedding of the incomplete object. As in previous methods [53] [6], we assume that the point cloud is segmented out from the original scan, namely, that all the points in P belong to single object.
Our task find the 3D surface (Fig. center) of the complete object that is consistent with both the input points P and the text prompt y. For any given point cloud, our method optimizes for the complete object surface represented by a neural signed distance function So RR > IR, and a neural color function C.p R3 R3, where and represent the learned parameters of the neural functions. As shown [55], these two functions form a neural radiance field (33 and can optimized using the rendered images of the 3D volumetric functions. More background details are given in the appendix.
3.2 Overview of the Optimization Process
Our main goal is reconstruct surface that consistent with the partial input point cloud P. typically capturing only "one side" of the object. Clearly, constraining the surface to be consistent with the observed input point cloud is not sufficient for determining the surface on the "other side" of the object, and some prior knowledge should be used. Traditionally, such prior knowledge is learned by training a model over a large dataset of 3D shapes [53] | [56). Here, we instead use pre-trained text-to-image diffusion model, applying SDS loss [39] to rendered images of the object (Fig 2 bottom). To ensure that we correctly reconstruct all sides of an input object, we use two types of compatibility losses: (1) censor-compaatiblliy losses (Fig. 2 top-right) (2) Text-compatibility loss (Fig. 2 bottom-right). These losses are described below in more detail in Sec 3.3
Applying and combining these compatibility losses is far from trivial and we now discuss several critical issues when using them. First, unlike DreamFusion (39,, where each object generated "from scratch", the pose of a generated object in our setup is determined by the input points P. As result, camera positions must be sampled in a way that compatible with the image prior, namely, that they render the object in natural poses.

---

Second, we find that simply rendering the object from the "other side" (unobserved side) and applying the SDS loss tends to generate content that is compatible with the input text but not with the input point cloud. To address this, we define "curriculum" of sampling camera poses. We start from the known original viewpoint of the sensor that captured the points and gradually increase the range of views that we sample from around the original view (details in the appendix). With this sampling protocol, generated completions are continuously kept compatible with both the description text and the point cloud generated so fär, until the entire object is completed successfully. As shown in our ablation study, this camera sampling protocol is key for producing high-quality object completions.
We next describe in more detail the compatibility losses and the camera- samera-sampling process.
3.3 Training Losses
Sensor-compatibility losses. The sensor compatibility losses are used for constraining the output surface to be compatible with the input point cloud P. The surface is defined as the zero-level set of our optimized neural SDF fo. Therefore, for constraining the surface go through the point cloud, we encourage the function to be zero at these points.
Σ 1 (p:)- Cp N
1
We note that the points in P are produced by subset of sensor rays. Sensor rays that do not produce any points in define constraints on the boundary of the object. We use this to define an additional sensor-compatibility loss fm- For each sensor ray i, we denote its opacity by M; {0, 1} where M, = if the ray produces a point P and M, = 0 otherwise. We further denote its rendered opacity by Mi E [0, 1]. The absence loss for the mask M is defined by:
Lm
M4
M
K
1=1
where K is the number of sensor rays.
Lp constrains the location of the surface, but it does not constrain the sign of the values around the surface which indicates on which side the interior of the object is located. For that, we use the fact that each point in P is measured relative to the sensor location. We denote the distance of each point from the sensor by D; and the rendered distance of the ray that produced this point by D, Our distance loss is defined by:
N
D. - D.11
La
(3)
N
E1
Text-coopatibilitty loss. We use a pre-trained text-to-image diffusion model, +; to provide a semantic prior for predicting the unobserved parts, such that any rendered image of the object would be compatible with the input text embedding y. To this end, we render random object views using radiance field and apply the SDS loss with the input text embedding to optimize fo and cp (Fig. bottom-right). More details about the SDS loss are given the appendix.
Regularization losses. introduced in [8
To constrain fo to form valid SDF, we apply the Eikonal loss regularization
Σ 11V (p:) 1
Leikonal
Peik
pEP.
where Pesk contains both P and uniformly sampled points from the region of interest.
Finally, we use the known world plane to further prevent the surface from drifting below the ground:
Cylane
max (- (-fo (p),
(5)
where Puniform is a set of uniformly sampled 3. points below the plane in the region of interest.
5

---

Input
GT
Ours
ShapeFormer
PoinTr
Figure 3: Qualitative results for the Redwood dataset. A qualitative comparison between our method and state-of-the-are methods. SDS-Complete produces more accurate completions.
Our total loss is:
Lioal 6m SaLa Ceskona + Splane Splane + LSDS.
(6)
where 6m dd, Sp, deskonal and ópione are the coefficients that define the weights of the different loss terms relative the SDS loss and were selected by hyperparameters search. The same constant coefficients were used in all of our experiments. See appendix, for more implementation details.
3.4 Handling Camera Positions
As diseussed above, the protocol for sampling camera views has large impact the quality of final completions. Let C. = (Ro. to) be the original camera-to-world pose of the sensor. To prevent rendering flipped unrealistically rotated images of the object, define the azimuth and elevation deviation from Co relative to the world plane. Specifically, let n E 3 be the normal to the world plane define the azimuth rotation update to be Razimuth R(m, Tarimuh), where R(n, 7) is the Rodrigues' rotation formula for rotation around the unit vector with degrees. Similarly, let ag be the normalized principal axis direction of Co. we define the elevation rotation update by Relovation = R(nj x "elevation) Assuming that the origin is located at the center of the object, an updated camera, Cupdates for Tazimmth and Televation degrees, given by:
Cupdate
(Razimuth Retevation Ro, Razimuth Retevation t.)-
During training, we start by applying the SDS loss to rendered images from Co pose, and then we gradually increase the sampling range of the deviation angles until the entire object is covered. More implementation details are given in the appendix.
4 Experiments
When considering evaluation, our primary goal to evaluate our SDS-Complete and baseline method in real-world scenarios. This is in contrast to evaluating test splits from the synthetic datasets that were used for training the baseline methods. To achieve relevant evaluation datasets, we based the evaluation partial real-world point clouds obtained from depth images and LiDAR scans.
Datasets.
For depth images, we used the Redwood dataset [12 that contains diverse depth images from 14 representative objects with ground truth 360 reconstructions which enable quantitative evaluation. We further tested our model on the KITTI LiDAR dataset 7 [7], which contains incomplete point clouds of objects in real-world scenes captured by LiDAR sensors. Both, our method and the baselines, require a segmented point cloud inputs. Therefore as a preprocessing. segmented out and centralized the main object from each scene (see more technical details in the
objects. We used

---

Table 1: Chamfer loss (lower is better) for chair and table categories from the Redwood dataset. All baselines were trained on chairs and tables.
Object
Shape Former
PoinTr cGAN
Sinv
SDS Complete
Full
Simple
Old chair
23.2
34.1
33.2
36.7
19.3
18.9
Outside chair
25.9
29.6
42.8
28.7
22.6
22.4
One lag table
39.7
21.6
99.4
24.9
20.3
18.1
Executive chair
33.6
43.9
208
20.6
23.7
22.0
Average
30.6
32.3
95.8
27.7
21.5
20.4
Table Chamfer loss (lower is better) for general objects from the Redwood dataset. Most of the object categories new for all methods and were not observed during training.
Object
Shape Former
PoinTTr
SDS-Complete Simple
Full
Trash
136.4
137
36.4
43.1
Plant in a vase
60.8
41
29.5
27.4
Vespa
79.4
70.3
57.6
35.7
Tricycle
65.2
60.4
39
41.3
Couch
43.9
87.4
36.5
50.1
Office trash
68.8
49.7
20.5
18.7
Plant in vase
31.3
37.6
28.1
26.3
Park trash can
130
119.9
33.8
26.4
Bench
29
32.6
55.4
98
Sofa
106.6
129.3
40.6
43.2
Average
75.1
76.5
37.8
41.0
appendix). For each input object, the world's ground plane, E ps that our method uses for camera sampling (Sec. (Sec.3.44 is extracted from each scene automatically by robust fitting. We also use the same world plane for the baseline methods part of the point cloud to dataset alignment procedure that they require (see appendix for details).
Baselines: compare our method to state-of-the-ard point-cloud completion approaches: PoinTr [56). ShapeFormer [53), cGAN [50] and Sinv (58). Poin'Tr and ShapeFormer trained one model on multiple classes; cGAN and Sinv trained per-class models for chairs and tables classes.
4.1 Results on the Redwood dataset
As our quality metric, we measure the Chamfer distances in millimeters to quantify the dissimilarity between the generated completions and their corresponding ground-truth shapes. Results are presented in Tables [ Table compares all methods on objects from two categories, tables and chairs;
Table compares our method to PoinTr [56) and ShapeFormer (53) on the remaining Redwood objects. Both tables demonstrate that SDS pos-CCeeee obtains state-of of-tth--rtttt results on the task of
point cloud completion from real point clouds.
In addition to the full variant discussed above, we also show results with simplified model we name "SDS- Complete Simple". This simplified variant omits the RGB network ce, and uses gray-shaded rendering images of the optimized geometry for the SDS-loss. It is more efficient but has higher average We further show qualitative results for general objects in Fig. Qualitative results for chairs and tables are presented in the appendix.
Completion with various text descriptions. Our approach operates by combining a partial input point cloud with text description that guides the model when completing missing parts of the object.
tested the effect of changing the text prompt while keeping the same input point cloud. Figure shows results for completing Redwood's ''s scan (08754"(Teapo)) of partially captured teapot (Fig. 11. Completing the point cloud with other text descriptions demonstrates how the text controls the shape.

---

Input points
"A teapot"
"A coffee pot"
"A watering can"
"A pitcher"
"A cup"
Figure 4: The effect of text prompt. Completion of the input, with different descriptions. Results obtained with our method for the partial point cloud of scan "08754". While the handle and the top part of the object are constrained by the input point cloud, the model completes the other side of the object according to the input text.
(a) Input points
(b) "An object"
(c) "A thing"
(d) "A plant"
(c) "A plant in large vase"
Figure 5: Generic specific text prompts: qualitative results. Results are shown for reconstructing scan "06127" (Plant in a vase) from the Redwood dataset. (a) The input point cloud. (b, c) Completion using two generic texts. Completion quality is poor. (d) Completion using the object class name. (e) Completion using a detailed textual description.
Generic specific Text Prompts. To evaluate the contribution of selecting an appropriate text prompt per object, we repeated reconstruction experiments of the 14 objects that we evaluate in Tables [nd 2 but varied the text prompts. Specifically, First, for fully generic prompt (class agnostic), we tested two alternatives: "An object", and "A thing". Second, we used the class name as the prompt ("A <class name>"). Finally, we used more detailed description. The full-text prompts are given in the supplementary.
used three levels of description specificity,
Table shows the Chamfer distances between reconstruction and the ground truth for all prompts. Using generic text yields inferior reconstructions. Adding specific details did not provide a significant improvement over using the class name. A qualitative comparison is shown in Fig.
Ablation study. To evaluate the contribution of each component of our method present quali- tative and quantitative ablation studies in Fig. 6 As can be seen, without the SDS loss, our model has no understanding of object characteristics like the fact that the chair has four legs and straight back-side. Without the SDF representation, it is not possible to apply the point cloud constraints directly the surface which results in inferior ability follow the partial input. Finally, can be seen that our camera sampling "curriculum" that described in Sec. 3.4 improves the completion compared random camera sampling ("Naīve camera sampling") by preserving the consistency of the generated content with the existing sensor measurements and by verifying that the diffusion model does not see any "unnatural" pose of the object.
4.2 Results on the KITTI Dataset
We compare method to ShapeFormer 53 and PoinTr [56] on subset of 15 real object scans from the Semantic KITTI Dataset [ which consists of 5 cars, motorcycles, and 5 trucks. We present qualitative comparisons in Fig. 7 Notably, our method shows better completion results, particularly with motorcycle objects which are less frequent the Shapenet dataset.
User Study. We conducted user study to evaluate the various methods on KITTI dataset. Specifically, gathered a group of 11 participants to rank the quality of each completed surface and faithfulness the input partial point cloud. For each object, the participants were given three anonymous shapes produced by the three methods: SDS- DDS-Complete, ShapeFormer (53), and Poin

---

Text Prompt
"An object"
"A thing"
"A <class name>"
Full text
Chamfer
52.0
52.7
33.8
33.1
Table 3: Generic specific text prompts: Chamfer distances (lower is better). Columns 1 and
2: two generic configurations where a global text used for all objects. Column 3: Only the class name is used e.g. both "executive chair" and "outside chair" are reconstructed with the text "A chair". Column 4: The results our method with the full text prompts (provided in the supplementary).
Input points
Naïve camera
No SDS loss
No SDF representation
Full method
sampling
Quantitative (Avg. Chamfer)
42.3
46.9
75.0
33.1
Figure 6: Ablation study. We demonstrate the contribution of each part of our method. Naïve camera sampling: running without our camera handling that is described in Sec.B..4 No SDS loss: using all losses but the SDS loss. No SDF representation: running with density function as in [33]. Below, we quantitatively compare the average Chamfer distance over the evaluated Redwood seans. We show extended ablations in the appendix.
[56]. While the outputs of SDS-S CoS-Ceeeee and ShapeFormer surfaces, PoinTTr only outputs point cloud. Therefore, we applied Screened Poisson Surface Reconstruction (24] to each output of PoinTr base the user study comparisons on surface representations. The participants were instructed to choose the best shape, while the order of the methods was shuffled for each object. The best completion method for each input case is selected by the majority vote. The results of the user study are presented in Table showing that our method got the highest number of wins (14 out of 15).
Quantitative evaluation on KITTY. For a quantitative metric, we followed PCN [57] and caleu- lated the Minimal Matching Distance (MMD). MMD is the Chamfer Distance (CD) between the output surface and the surface from ShapeNet that is closest the input point cloud in terms of CD. We calculated this metric on the surfaces that were evaluated in our user study from two categories: car and motorcycle. These are the only categories that have associated Shapenet subsets which is necessary condition for calculating the MMD metric. The mean MMD over the motorcycle and car shapes are presented in Table showing that our approach improves over the baselines.
We further computed the CLIP R-Precision metric (39 on all of our evaluated KITTI categories: "car". "truck" and "motorcycle". This metric checks the accuracy of classifying rendered image by choosing the class that maximizes the cosine similarity score between the image and the text: "a rendering of <class name >" among all classes. We evaluated the output geometries of the different methods, each rendered from 360 degrees with azimuth gaps of 2 degrees (180 images for each surface). We report the mean accuracy in Tab. Here again, our approach is substantially better.
4.3 Limitations
As in previous work (53)). for extracting point clouds from depth images, we need the internal parameters of the depth camera. Our test time optimization method slow compared to the feedfor-
Table 4: Quantitative results on
Shape Former
PoinTr SDS-Complete (ours)
KITTI. User-study wins refer to the percentage of cases in which method was selected by raters as the best method. Our method outper- forms the baseline methods in all metrics.
User-study wins [%]t
7.0
0.0
93
Average MMD
0.036
0.052
0.027
CLIP R-Precision t
0.61
0.43
0.76

---

Input
SDS-Complete
ShapeFormer
PoinTr
Figure 7: Qualitative completion results on the KITTI dataset.
Redwood Tables and Chairs
Redwood all
Ours Simple
Ours Simple
Ours Full
Ours Full
35
ShapeFormer
ShapeFormer
PoinTi
PoinTT
Chanter
Chanferr
30
25
20
500
1000
1300
2000
200
1000
1500
2000
Spochs
Spochs
Figure 8: Shorter Training. The Chamfer error when running our method for fewer epochs. For reference, also include the (inference time) numbers of the baselines. Left: our convergence the table and chair categories. Right: our convergence when considering all 14 Redwood models.
ward baseline methods that were pre-trained on large datasets of 3D shapes. The major factor for the running time the SDS-loss which needs many sampling views. The number of training epochs can be shortened by reducing the number of iterations. In Tab, & we show the effect of reducing our runtime. As we show, our method outperforms the baselines terms average accuracy after 5% of the training time and keeps improving when more training time is given. In the appendix, we show failure cases our method and list additional implementation details.
Conclusions
We presented SDS-Complete, novel test time optimization approach for 3D completion using text-to-2D pre-trained model. For handling point cloud inputs, we incorporated an SDF representation and constrained the surface to lie the input points. We successfully applied the SDDS loss images rendered from novel views and completed the missing part of the object by aligning the images with an input textual description. By handling the camera sampling carefully we maintained the consistency of the completed part with the input captured part. This enabled us to produce superior results even previously unconsidered objects for completion. Future work includes improving running times by combining recent NeRF techniques (e.g. (35)), and supporting scene completion from incomplete point clouds of multiple objects.
10

---

6 Acknowledgments
We thank Lior Yariv, Dolev Ofri, Or Perel and Haggai Maron for their insightful comments. We thank Lior Bracha and Chen Tessler for helping with the user study. This work was funded by grant to GC from the Israel Science Foundation (ISF 737/2018) 18), and by an equipment grant to GC and Bar-Ilan University from the Israel Science Foundation (ISF 2332/18). OR is supported by a PhD fellowship from Bar-Ilan data science institute (BIU DSI).
References
[1] I. Achituve, H. Maron, and G. Chechik. Self-supervised learning for domain adaptation point clouds. In Proceedings of the IEEE/CVF winter conference applications computer vision, pages 123-133, 2021.
[2] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas. Learning representations and generative models for 3d point clouds. In International conference machine learning, pages 40 49. PMLR, 2018.
[3] E. Aiello, D. Valsesia, and E. Magli. Cross-modal learning for image-guided point cloud shape complction. Advances in Neural Information Processing Systems, 35:37349-3783 2022.
[4] K. S. Arun, T. S. Huang, and S. D. Blostein. Least-squares fitting of two 3-d point sets. IEEE Transactions Pattern Analysis and Machine Intelligence, PAMMI-9698--700, 1987.
[5] M. Atzmon and Y. Lipman. Sal: Sign agnostic learning of shapes from raw data. In JEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), June 2020.
[6] Y. Balaji. S. Nah, X. Huang. A. Vahdat, J. Song. Q. Zhang, K. Kreis, M. Aittala, T. Aila, S. Laine, B. Catanzaro, T. Karras, and M.-Y. Liu. ediff-i: Text-to-image diffuusio models with ensemble of expert denoisers. arXiv preprint arXiv:2211 120::2211101322 2022.
[7] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, J. Gall, and C. Stachniss. Towards 3D LiDAR- based semantic scene understanding of 3D point cloud sequences: The SemanticKITTI Dataset. The International Journal on Robotics Research, 40(8-9):959-967 2021.
[8] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall. Semantickiti:: A dataset for semantic conference on computer vision, pages 9297-9307, 2019.
understanding of lidar sequences. In Proceedings of the IEEE/CVF international
[9] A. X. Chang, T. Funkhouser. L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich model repository. arXiv preprint arXiv:1512.03011 2015.
[10] D. Z. Chen, Y. Siddiqui, H.-Y. Lee. S. Tulyakov, and M. NieBner. Text2tex: Text-driven texture synthesis diffiusion models. arXiv preprint arXiv:2303.
2023.
[11] Y.C.. Cheng, H.-Y. Lec, S. Tulyakov, A. G. Schwing, and L.-Y. Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4456-4465, 2023.
[12] S. Choi, Q.-y Zhou, S. Miller, and V. Koltun. A large dataset of object scans. arXiv preprint vrXiv:1602.000 1602002481 2016.
[13] C. B. Choy, D. Xu, Gwak, K. Chen, and S. Savarese. 3d-12n2: A unified approach for single and multi- view object reconstruction. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 228-644. Springer, 2016.
-d-encoder-predictor [14] A. Dai, C. Ruizhongtai Qi, and M. Nießner. Shape completion using 3d-encoder predictor cans and shape synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1868-5877,22 2017.
[15] H. Fan, H. Su, and L. J. Guibas, A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 605-613, 2017.
[16] R. Gal, Y. Alaluf, Y. Atzmon, O. Patashnik, A. H. Bermano, G. Chechik, and D. Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022.
[17] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3354 3361, 2012.
11

---

[18] A. Gropp, L. Yariv, N. Haim, M. Atzmon, and Y. Lipman. Implicit geometric regularization for learning shapes. International Conference on Machine Learning, pages 3789-3799. PMLR, 2020.
[19] C. Häne, S. Tulsiani, and J. Malik. Hierarchical surface prediction for 3d object reconstruction. In 20/7 International Conference on 3D Vision (3DV), pages 412-420. IEEE, 2017.
[20] R. Hartley and A. Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
[21] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778. 2016.
[22] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
[23] A. Jain, B. Mildenhall, T. Barron, Abbeel, and B. Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 867-876, 2022.
[24] M. Kazhdan and H. Hoppe. Screened poisson surface reconstruction. ACM Transactions on Graphics (ToG), 32(3):1-13 2013.
[25] N. M. Khalid, T. Xie. E. Belilovsky, and T. Popa. CLIP-mesh: Generating textured meshes from text using pretrained image-text models. SIGGRAPH Asia 2022 Conference Pupers. ACM, nov 2022.
[26] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint ar Xivi::1422.690 2014.
[27] M. Li. Y. Duan, J. Zhou, and J. Lu. Diffusion--dff: Text-to-shape via voxelized diffusion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 12642-12651, 2023.
[28] C.-H. Lin, J. Gao, L. Tang. T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin. Magic3d: High-resolution text-to-3d content creation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[29] L.. Melas-Kyriazi, I. Laina, C. Rupprecht, and A. Vedaldi. Realfusion: 360deg reconstruction of any object from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8446-8455. 2023.
[30] L. Mescheder, M. Occhsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4460-4470, 2019.
[31] G. Metzer, F. Richardson, O.
Patashnik, R. Giryes, and D. Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 12663-12673, 2023.
[32] O. Michel, Bar-On, Liu, Benaim, and R. Hanocka. Text2mesh: Text-driven neural stylization for meshes, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13492-13502, 2022.
[33] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron. R. Ramamoorthi, and R. Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In Computer Eision-ECCH 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 405-421. Springer, 2020.
[34] P. Mittal, Y.-. Cheng, M. Singh, and Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 305-315, 2022.
[35] T. Müller, A. Fvans, C. Schied, and A. Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Trans. Graph., 11:42:102:1-022:55 July 2022.
[36] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[37] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
12

---

[38] S. Peng, M. Niemeyer. L. Mescheder. M. Pollefeys, and A. Geiger. Convolutional occupancy networks. In Computer VisionEECCV 2020: 16th European Conference, Glaxgow, UK. August 23-28, 2020, Proceedings. Part III 16. pages 523-540. Springer, 2020.
[39] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
[40] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[41] A. Raj. S. Kaza, B. Poole, M. Niemeyer, N. Ruiz, B. Mildenhall, S. Zada, K. Aberman, M. Rubinstein, Barron, et al. Dreambooth3d: Subject-driven text-to-3d generation. arXiv preprint arXiv:2303.13500 2023.
[42] E. Richardson, Metzer, Y. Alaluf, R. Giryes, and D. Cohen-Or. Texture: Text-guided texturing 3d shapes. arXiv preprint arXiv:2302.0177 2023.
[43] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent difftsson models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pages 10684-10695, 2022.
[44] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances Neural Information Processing Systems, volume 35, pages 36479-36494. Curran Associates, Inc., 2022.
[45] J. Song. C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[46] D. Stutz and A. Geiger. Learning 3d shape completion under weak supervision. International Journal of Computer Vision, 162-5):1122-1181 act 2018.
[47] J. Tang. Stable-dreamfusion:: Text-to-3d with stable-diffusion, 2022. https//github.ccm/sssawwkkkkkkk dreamfusion.
[48] J. Tang, T. Wang, B. Zhang. T. Zhang, R. Yi, L. Ma, and D. Chen. Make-it-3d: High-fidelity 3d creation from single image with diffusion prior. arXiv preprint arXiv:2303. 14184, 2023.
[49] F. Williams, Z. Gojeic, S. Khamis, D. Zorin, J. Bruna, S. Fidler, and O. Litany. Neural fields as learnable kernels for 3d reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1155001 June 2022.
[50] R. Wu, X. Chen, Y. Zhuang, and B. Chen. Multimodal shape completion via conditional generative adversarial networks. Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 281-296. Springer, 2020.
[51] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[52] P. Xiang, X. Wen, Y.-S. Liu. Y.P. Cao, P. Wan, W. Zheng, and Z. Han. Snowilakenet: Point cloud completion by snowflake point deconvolution with skip kiiptrrasformmer In Proceedings of the IEEE/CVF international conference on computer vision, pages 5499-5509, 2021.
[53] X. Yan, L. Lin, N. J. Mitra, D. Lischinski, D. Cohen Or, and H. Huang. Shapeformer: Transformer based shape completion via sparse representation. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 6239-6249, 2022.
[54] Y. Yang, C. Feng, Y. Shen, and D. Tian. Foldingnet: Interpretable unsupervised learning on 3d point clouds. arXiv preprint arXiv:1712.07262. 2(3):5, 2017.
[55] L. Yariy, J. Gu, Y. Kasten, and Y. Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805 4815, 2021.
[56] X. Yu, Y. Rao, Z. Wang, Z. Liu, J. Lu, and Zhou. Pointr: Diverse point cloud completion with geometry- aware transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12498 12507, 2021.
13

---

[57] W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert. Pen: Point completion network. In 2018 international conference 3D vision (3DV), pages 728 -377. IEEE, 2018.
[58] J. Zhang, X. Chen, Z. Cai, I.. Pan, H. Zhao, S. Yi, C. K. Yeo, B. Dai, and C. C. Loy. Unsupervised 3d shape completion through gan inversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 1768-1777, 2021.
14

---

A Implementation Details
Running Time We run our method for 2000 epochs, where each epoch uses 100 iterations. That takes about 1950 and 1380 minutes for Redwood and KITTI scans respectively. The optimization time for our simplified model takes about 1256 and 924 minutes for Redwood and KITTI scans respectively. All times are measured when running our method on NVIDIA RTX A6000. We note that many scans need much fewer iterations for converging (see Fig. 8 but to complete the fine details, e.g. the chair's legs, many iterations are needed.
Network Architecture For our optimized coloring function C., we use 4 linear layers with 96 channels, where the two intermediate ones are incorporated in ResNet [2I] blocks, as implemented by [47]. with SiLU activations [22]. For the SDF network fa. use 4 linear layers with 96 channels and ReLU activations. fo is initialized to a sphere [5] with radius lengths of 0.5 and 0.9 for Redwood and KITTI scans respectively. For both, C,p and fo we use Positional Encoding with 6 levels [33
For extracting density from fe (Equations 12 and 13
we use 100, 3 10
SDS-loss Implementation Details We base our code on the implementation of 447 During training, for each iteration, we randomly sample a background color, to prevent the model from "changing" the geometry by just coloring it with the background color. For Redwood cases, we render 80 x 80 images for the SDS-loss using the sampled camera and the known internal parameters of the sensor. For KITTI, at initialization time we first project the object's ' LiDAR points to 2D spherical projection [8), with height and width of 64 and 1024 pixels respectively. We use the projected 2D mask to select the 2D bounding box area of 64 pixels height, where the width is determined by the min and max horizontal coordinates of the object =5 pixels. The LiDAR rays that define this selected bounding box are used to render the object during training, where a novel camera pose is defined by rotating these rays around the object's 's centroid. As a text-to-image diffusion model we use Stable Diffusion v2 43
Training Details We optimize the networks using the Adam optimizer [26] with a learning rate 10-1. The coefficients for our loss for all the experiments are 8m 10 8, = 10 6p = 105, deskonal 10a, Öplane Leskonat. For Lm, La, at each iteration, we randomly sample 2000 pixels for Redwood cases, whereas for KITTI, we render the entire bounding box.
10a At each iteration we sample 1000 uniform points for Cptane and
Camera Sampling As described in Sec 3.4, during training, start by applying the SDS loss on the rendered image from Co pose, and then we gradually increase the sampling range of the deviation angles until the entire object is covered. In more detail, gradually increase the sampling range of the azimuth angles: Tazimuth ~ (-v,, v), starting from P = 0 to v = 180. Specifically, we set 30, 45, 60, 90, 180 at epochs 20, 50, 80, 100, 120 respectively. Televation is set to 0 for 20 epochs and then uniformly sampled according to: Velevation ~ Ui( (oo 0) for Redwood scans, where Eu the elevation of Co from the plane in degrees. For KITTI scans (after epoch 20) we use Televation ~ U(-5o. Eo) since the original viewpoint is usually low, and we also scale the distance from the source to the object uniformly by ~ 21(1, 2) after epoch 20. As in [39, we augment the input text according to the viewing direction, with text that describes the viewpoint orientation. Specifically, as in 47 we use "*, front view". "* side view". "* back view". "* overhead view" and "e, bottom view", where denotes the input text. Unlike [39,, the orientation of the object is determined by the input points. Therefore, we use an extra input from the user of 7m which explains the original viewpoint, e.g. "Oainun = 90 if the object is viewed from the side. Then, during training, we use "Yominm and Tarimuth to calculate the azimuth with respect to the object, and "elevation to compute the elevation with respect to the plane I. These orientations are used to augment the text with the corresponding view direction description. In our simplified version ("SDS-Complete Complete Simple") " we omit the text augmentation and hence do not need this extra input from the user of Y0
Object Centralization Given the input points we centralize them at the origin. This is done in general by subtracting their center of mass. When the object's largest dimension is aligned with the viewing axis, the center of mass is usually biased toward the camera. To handle this, we extract an oriented 3D bounding box for the input points and measure the ratio between the largest distance to the smallest distance from the center of mass to any bounding box point. If this ratio is above 1.7 we
15

---

use the bounding box center as our centroid instead of using the center of mass. In the KITTI dataset, which mostly includes non-isotropic objects, we always use the bounding box center as our centroid. We then scale the points such that the largest point norm is 0.5.
Baseline Runnings For running the baseline methods, we tried locate the input points as much as possible according to the method's 's expectations to prevent them from failing. This includes using our knowledge about the world plane I and the object orientation with respect the camera "0rrman For ShapeFormer, each time we took the best shape out of the 5 that it outputs.
Data Processing For the Redwood dataset, we segmented out the foreground object manually. For the evaluation only, as a preprocessing, we manually aligned the GT scan with the partial point cloud and applied ICP for refinement 44. Each KITTI sean that we used, is the aggregation The segmentation map for KITTI is given by [8. For both, KITTI and Redwood datasets and for each scan, the plane is segmented out from the original point cloud using RANSAC 20)
timestamps.
B
Preliminaries
B.1 Volume Rendering
Neural Radiance Field A neural radiance field [33] is a pair two functions: о R - R+ and c: (RR (8) + RR, each represented by Multilayer Perceptron (MLP). The function maps 3D point x E R into a density value, and the function c maps 3D point x and a view direction v E S into an RGB color. A neural radiance field can represent the geometric and appearance properties of a 3D object and is used as a differentiable renderer of 2D images from the 3D scene. Let be an image with camera center t TR3 the pixel coordinate 3D ray Tu. starting at and going through the pixel u with a direction v E 82, Let (41, (A.,...,, ,NN. be sample distances from t on the ray Tu. then the densities and colors of the radiance field are alpha composited from the camera center through the ray. The RGB image color ((u, v) is calculated by:
(u, v) RR2 is backprojected into
N= I(u,v)
Σ w
(8)
where w; = a: a 0z) is the color contribution of the jth segment to the rendered pixel, and
a ===1- 1
exp (-o ( (4)) is the opacity of segment Eq. (8) is differentiable with respect to the learned parameters of с and a and therefore, is used to train the neural radiance field.
Let be the ground truth image, then the MSE loss is used to train the neural radiance field:
LMSE = ((())
(9)
where n is the number pixels in the batch.
Volume Rendering of Neural Implicit Surfaces While the neural radiance field shows impressive performances in synthesizing novel views, extracting object geometries from a trained radiance field not trivial. Defining the surface by simply thresholding the density results in noisy and inaccurate geometry. We adopt the solution proposed by [55]. Let Q c R3 be the space occupied by the object, and M denotes the boundary of the surface. Then the SDF f R3 + R is defined by
f(x) (=1)) min ||x yll
(10)
YEM
[ x E Ω 0 otherwise
where 1x(x)=
Given f, the surface M is defined by its zero level set, i.c.
M ex f(x) 0}
(11)
A signed distance function can be utilized for defining neural radiance field density. Let x E R and f. 3 + IR be a 3D point and an SDF respectively, the density (xx) defined by:
(xx) at a( (x))(x))
(12)
16

---

where V ( is the Cumulative Distribution Function (CDF) of the Laplace distribution with zero mean and 8 scale:
2 exp ( - exp
S 0
(13)
>>0 0
and and 3 are parameters that can be learned during training (in case, we set them to be constant), then possible to train a neural radiance field, defined by the SDF f and the neural color function c, using the loss function defined by Eq. (9)
B.2 Score Distillation Sampling (SDS)
generates image samples from Gaussian noise image, by inverting the process of gradually adding noise to an image. This process is defined follows: at time t 1, ................................ T, Gaussian noise (O, I) is added the image: 1; = V + /1 y1-ārt, âte, where ā. II ag a 1 3r and 3 E (0, 1) defines the amount of added noise. A denoising neural network & = (Ix; t) is trained to predict the added noise é given the noisy image I and the noise level The diffusion models are trained on large image collections C for minimizing the loss
Diffusion Models A diffusion model 36 45
LD E Iec
[| v1 -,e; - =
(14)
Given pretrained Φ image sample is generated by sampling Gaussian noise image IT 1), and gradually denoising it using $
Diffusion models can be extended to be conditioned on additional inputs. Text-to-image image diffusion models [43] condition on a textual prompt embedding input y. and train 4 t, y). Therefore, they can generate images given text and random Gaussian noise.
DreamFusion 39 uses a pretrained, and fixed, text condition diffusion model (I t, y) and uses to train a NeRF model from scratch, given textural description embedding Yo- In each iteration, camera view is sampled and used to render an image In from the NeRF model. In is differentiable with respect to the learned parameters of the NeRF model (ONeRF), and used as an input to Φ To; v). The Score Distillation Sampling (SDS) loss then applied:
Oruas LsDs (T)) E.,- [(w(())(V (Vonlo + vI 04e; L, vo) V)V DNERF Jo]
(15)
Note that V Ve LSDS is the gradient with respect to ONERF of Eq. (4). where the Jacobian of & is omitted for stability and efficiency. Intuitively, if lo looks like a natural image, and is compatible with yo. then the pretrained diffusion model predicts the added noise successfully, resulting low values for Lp. By updating the NeRF's weights according to Eq. (15), Lp is reduced, and as result, the rendered images become more compatible with yo.
C Limitations Examples
(b)
(c)
(d)
Figure 9: Failure cases of our method. (a).(b Input points and surface completion respectively, for Redwood sean "05492" (standing sign). (c).(d Input points and surface completion respectively for Redwood scan "01373" (picnic table).
17

---

Input Points
Input points +
Rendered Image (low-res)
Output surface
Figure 10: Intermittent patterns in the legs of the chair. The supervision signal for the legs, coming from the thin lines of input points (left) is not enough for reconstructing the chair's legs correctly (middle). The SDS loss sees valid rendered images this case and therefore does not fix (right).
Failure Cases Fig. shows failure examples. general, our method does reconstruct well thin surfaces. We hypothesize that the initialization of the SDF to sphere [5], prevents the model from minimizing the occluded part early training stages. Then, the SDS loss usually tries to paint this redundant content according to the text prompt, instead of removing it. Different initializations to the SDF, or other regularizations, need be explored and left as future work.
Intermittent patterns Fig. TO demonstrates a case where the thin structure of the input points is too weak constrain the surface of the outside chair. Unfortunately, the SDS loss does not help in making the surface thicker since the rendering of the thin surface looks valid. This is due to our rendering process that uses VOLSDF which defines a smooth mapping from surface to density (Equation (12). ) and since the entire leg has SDF values that are close to 0, the legs get densities that produce valid low-rcsolution renderings.
D Additional Results
Reconstruction with incorrect prompts. We further check the sensitivity of our method to wrong text prompts. Specifically, we used the text: "A table" for chair, and the text "A chair" for table.
The visualizations are presented in Fig. [1
Supplementary Redwood Results
Qualitative results for table and chairs categories from Redwood dataset are presented in Fig. 14. Additional qualitative comparisons for other objects from the Redwood dataset are presented in Fig We can see that our method completes the shapes better
than the baselines.
We applied our method additional Redwood cases of various object types with no available ground truth. Qualitative results, including RGB renderings, are shown in Fig. 13
Video Results We attach to the supplementary folder, 360 video visualizations of our reconstructed objects for both, KITTI and Redwood datasets.
Extended Ablation We show extended ablation study in Tab. 5 that shows a quantitative evaluation of the importance of the depth and mask losses. We further show separate numbers for the table and chair categories and the rest of the categories.
Reduced input points We tested our "Full" method on the Redwood dataset with 50% and 10% of the original input points, on the evaluation set from the Redwood dataset. The locations of the removed points were selected randomly. We found that there is significant difference in the results (up to 1.5 mm difference in terms of average Chamfer distance) when excluding the "Bench" case. For the "Bench" case there is no significant difference when using 50% of the points, but when using 10% of the points we see that the Chamfer distance increases from 55.4 to 149, These results indicate that in general, our method is robust to the number of input points.
18

---

Input points
Incorrect text
Conrect text
Figure The effect of incorrect textual descriptions. Each row corresponds to different object. Left: The partial scans that are given as input to our model. Midd dle: Completion performed using incor- rect text descriptions. Right: the com- pletion results of our method with our final text prompts. In the first two rows, the complction is inferior when given the wrong text. In the bottom row, with an incorrect text ("A table") the model still completes the chair correctly. This is because the input provides strong con- straints. To make the shape to a table, the method still needs to recon- struct the missing leg.
chair"
leg
square table"
"An outside chair"
similar
"A table"
old chair
input
Ours
ShapeFormer
PoinTr
Figure 12: Additional qualitative comparisons on the Redwood dataset.
19

---

Naive camera sampling
No SDS
No SDF
No Depth
No Mask
Full
Redwood T & C
36.6
44.0
59.9
28.0
58.0
21.5
Redwood general
44.6
48.1
81.0
45.1
99.1
37.8
Redwood all
42.3
46.9
75.0
40.2
87.4
33.1
Table 5: An ablation study for demonstrating the contribution of each part of our method. Naïve
camera sampling: running without camera handling that described in Sec. [3.4 No SDS loss: using all losses but the SDS loss. No SDF representation: running with a density function as in [34]. No Depth: using all losses but the Depth loss. No Mask: using all losses but the Mask loss. The 3 rows present the average numbers for the table and chair categories (Redwood T & C), for all categories except table and chairs (Redwood general), and for the complete evaluation (Redwood All).
Ours+Points
Render
Input
Ourst
Render
Figure 13: Qualitative outputs of our method, when applied on Redwood cases with no available 360 GT scans for quantitative comparison. The figure is arranged columns of different objects, where for each object we show (from left to right): the input point cloud, the completed surface, the completed surface together with the input points, and image rendering of our optimized coloring function c.
20

---

Input
GT
Ours
ShapeFormer
Sinv
cGAN
PoinTr
Figure 14: Qualitative results for the table and chair categories from the Redwood dataset. Red represents methods that output point clouds.
21


Reviews:
[{"review_id": "vdrtWl8oA5", "reviewer": "Reviewer_adEs", "summary": "This work proposes to use text-to-image diffusion model for OOD point-cloud completion. Similar to DreamFusion3D that trains NeRF accompanied by an SDS loss for 3D generation, this work applies the idea to point-cloud completion. Experiments show that the performance is good on both Redwood dataset and KITTI dataset. ", "questions": "If I understand correctly, the method requires to overfit per-object via NeRF, on the other hand, since it requires inference pre-trained SD for each iteration, it seems that the training process is pretty slow. Can the authors provide the training details?", "limitations": "It seems that this work can only work for the cases of single objects since it heavily relies on the capability of pre-trained stable diffusion. I wonder whether it can achieve scene-level completion.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the proposed idea is simple yet novel. The results also show the effectiveness of the proposed method. The paper is also easy to follow.", "weaknesses": "I notice in both figure 3 and figure 5, although the patterns/shape of the completed point-cloud is correct, it seems that they are not smooth which even makes it worse than ShapeFormer in Figure 3. This makes the performance of the proposed idea less attractive, despite the paper targets at OOD point-cloud completion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the comments and the feedback from the reviewer.\n\nW1: Generated completions are not smooth in Figure 3 and Figure 5  \n\nOverall our method produces better outputs qualitatively (see more examples in the SM, Figure 4,5 ),  and quantitatively where our method’s Chamfer distance is lower by 50% overall, and by 30% for Shapenet classes (see Table 1 )  compared to ShapeFormer. Regarding Figure 5, in the user study (SM, Section 3),  for 73.4% of the cases, the participants preferred our outputs over ShapeFormer’s outputs, which demonstrates that our method outputs better completions in terms of quality.\n\n\nQ1: Provide details on running time:\n\nSee supplementary, Section 5, “Running time”. Our test time optimization method is slow compared to the feedforward baseline methods,  but works much better on real point clouds, and does not require any dataset of 3D shapes for training as required by the baseline methods. We believe that it could be accelerated  e.g. by combining recent acceleration techniques such as Instant-NGP but this is left as a future work. \n\n\nL1: The method is limited to single-object:\n\nThe reviewer is correct. There are natural ways to extend this approach to scenes with multiple objects. For instance, starting with segmenting a scene, and splitting it into a disjoint set of partial point clouds. This tends to be easier if RGB appearance data is available. Then, our method can reconstruct each object separately, and the results can be merged back together to create the scene. Interestingly, our loss can be extended to take into account segmentation confidence. This is left for future work.   \n"}, {"review_id": "qKAD9tRMjg", "reviewer": "Reviewer_BnfZ", "summary": "This paper proposed a novel method for completing a 3D object from its incomplete input shape. It specifically focuses on the out-of-distribution objects and proposed a diffusion model based framework to generatively learns a SDF for the complete shapes. The idea is simple and easy to follow.", "questions": "See weaknesses", "limitations": "The limitations has been addresses in the draft.", "rating": 4, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "The idea of using cross-modal generative network to complete partial shape is novel.\nThe writing and organization is good.", "weaknesses": "1. The completion of out-of-distribution objects aims for practical usage. However, the proposed SDS-Complete requires up to 5 different inputs for completing a partial shape, which is contrary to the original intention for out-of-distribution shape completion. Specifically, according to the description in Sec.4, SDS-Complete requires: 1) depth image; 2) segmented point cloud; 3) segmented binary mask; 4) representation of text; 5) internal parameters of the sensor, and each of them is indispensable. Therefore, the proposed method is highly impractical for real-world scenario compared with image reconstruction or shape completion from single input.\n\n2. As for in-domain completion results visualized in Figure 3, the complete shape generated by PoinTr is highly suspicious and may be unfair. The chair is one of the most commonly used categories in shape completion experiments, and PoinTr has been referred by many previous studies to generate a robust prediction on this chair category. However, in Figure 3, PoinTr even fails to predict the missing chair legs, which in reviewer's opinion, is highly impossible if the training procedure is correct. \n\n3. According to the above discussion of visualization results, the quantitative results in Table 1 are insufficient to prove the effectiveness of the proposed method in terms of out-of-domain shape completion task.\n\n4. Moreover, in-domain comparison should at least be conducted on one of the popular completion benchmarks such as ShapeNet dataset, MVP dataset or PCN dataset, but none of them appear in the experiments.\n\n5. The quantitative results on KITTI dataset is missing, only 5 samples in Figure 5 are highly insufficient to verify the effectiveness of the proposed method on KITTI dataset.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the comments and the feedback from the reviewer.\n\n\nW1: The method requires 5 input types\n\nOur setup follows an existing protocol that is presented in ShapeFormer which uses real partial point clouds that are extracted from real-world sensors. In contrast to synthetic setups where each point cloud is artificially sampled from a single CAD model, the evaluated real point clouds are recorded by an active depth sensor (LiDAR or depth camera) that stores for each ray the distance of the point from the sensor location.  Therefore, depth information is not a limitation for real point clouds since it is inherently available in real PC data and does not limit the method in practice. The internal parameters for each sensor are available online. \nNote that these real sensors record a complete scene and therefore for isolating the relevant object from the scene (for any of the baseline methods), it is required to segment out the object from the scene.  Regarding text input, some other completion methods [3-6] use different trained models for each class (tables, chairs, …), thus they assume that the object class is known.  \n\nW2: Complete shape generated by PoinTr is highly suspicious and may be unfair:\n\nNote that PoinTr was trained and evaluated on synthetic data. Therefore its generalization to real data is limited. Moreover, PoinTr has some generalization issues even for Shapenet models e.g. Figure 4 in [7].\n\n W3: The quantitative results in Table 1 are insufficient to prove the effectiveness of the proposed method in terms of out-of-domain shape completion task \n\nAccording to Table 1, we improve over the baseline methods by 50% on real-world point clouds. Note that additional qualitative and quantitative examples are presented in the supplementary materials. \n\nW4: Missing comparisons on Shapenet/MVP/PCN\n\nOur work addresses the problem of completing point clouds captured ``in the wild”, rather than in close-world synthetic data. We expect methods trained to complete PCs of specific predefined classes on synthetic data to perform well in that setting, but as our paper demonstrates, they struggle when applied to open-world settings. We focus the experiments on real partial point cloud data captured by real sensors rather than on synthetic data sampled artificially from CAD models. Unlike the previous methods, our method was not trained on synthetic datasets so it would be unfair to base the comparisons on these datasets. Both, our method and the baselines, were not trained on real cases, so we find it fair to base the comparison on real-world data.\n\nW5 : Comparisons on KITTI are quantitative using only 5 samples\n\nWe conducted a user study on 15 sampled KITTI objects. The results are presented in the supplemental material, showing that our method outperformed the baseline methods in 73.4% of the cases. These results demonstrate that our method outputs better completions on KiTTI in terms of quality. Since GT shapes for KiTTI are not available, it is not possible to compute automated metrics. \n"}, {"review_id": "sYqGlmBt3G", "reviewer": "Reviewer_Fuyp", "summary": "This paper proposes a point cloud completion method with the help of text-to-image diffusion model and formulate the point cloud completion as a test-time optimization problem. It exploits the SDS loss proposed in Dreamfusion to generate 3D given text prompt, with a text-to-image diffusion model. Additionally, it also utilizes the supervision from the input partial point cloud (and also the depth and mask from sensor) for better regularization. And the authors also propose a camera sampling strategy to let the optimized shape follows the input point cloud. Since the model is built on top of the diffusion model's prior, it doesn't have the severe OOD problem as for previous train-test point cloud completion baselines.", "questions": "1.I'm confusing with the epoch-iteration training schedule and the proposed camera handling. In the total 200K iterations (2k epochs x 100 iterations per epoch), are the different camera angles being sampled in a loop manner or just sample one angle for many iterations and never optimized again? I think the loop manner is the correct one, but if the rendering angles are looped again and again, I think it means the motivation of camera handling isn't that important.\n\n2. I wonder what's the optimization time comparing with DreamFusion? Since in the point cloud completion task a stronger supervision or prior is given, I assume with shorter iterations it can get comparable geometry? (Although definitely more iterations the results can be better).\n\n3. Some results seems to be unexpected. Like the second row in Figure3. The chair leg with point cloud is intermittent, but the leg behind without point cloud supervision is perfect. Is it because the point cloud is sparse and sometimes the L_p can have bad influence?\n\n4. Is the low resolution 80x80 a bottleneck? I remeber it isn't the render resolution in DreamFusion.", "limitations": "Yes, the authors have a discussion of their limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "By forming the task as a test-time-optimization problem, the proposed SDS-complete won't rely on training with large-scale datasets, and instead using the prior from text-to-image models,  which greatly boost the performance on OOD classes. Different from uniform camera sampling in DreamFusion which is used to generate object from scratch, the authors also proposed a camera handling procedure with strong motivation, which helps the model to fit the input partial point cloud and prevents the diffusion model to effect this geometry.", "weaknesses": "I think the proposed method is lack of enough ablation study. I didn't find ablation study (maybe just qualitative) of the proposed camera handling procedure. And also there might be some paper organization problem, see the questions below. \n\nAnd I believe the time required is also a weakness comparing to train-test method, which I think should be discussed in paper, although currently hard to solve. \n\nSome typos. In L231 I think it should be \"multi-modal\". ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the comments and the feedback from the reviewer.\n\nW1:  Missing Ablation study for camera handling:\n\nFigure 6  in the main paper shows a table with a quantitative ablation study on the camera handling procedure (“Random camera: running without our camera handling that is described in Sec. 4.2”). It shows that without our camera sampling, the mean Chamfer error is significantly larger.\n\n\nW2:  Time required is also a weakness that should be discussed:\n\nSee the discussion of the limitations in L258. Our test time optimization method is slow compared to the feedforward models of the baseline methods but its quality is far superior on real point clouds, and does not require any dataset of 3D shapes for training as required by the baseline methods.  We will emphasize this point more clearly in the revised version\n\n\nSome typos In L231 \n\nWe will fix this in the revised paper. Thank you.\n\n\nQ1: Camera Sampling Missing Details:\n\nL76 in the SM gives more implementation details. At each iteration, the cameras are sampled randomly from a uniform distribution with a limited range. This range increases during training, where the full range is applied from epoch 120. That allows the color and the geometry to be optimized consistently and gradually, starting from the areas that are covered by the sensor, to the areas without any input points. See the ablation study in Figure 6 of the main paper for quantitative and qualitative ablation studies on camera sampling.   \n\n\nQ2 What is the optimization time compared to DreamFusion?\n\nMost of the time is consumed by pushing gradients for the SDS loss. \nBy default, DreamFusion is trained for 100 epochs. In our submitted version we let our method run for 2000 epochs. Following the reviewer’s question, we present in Table 2 (Rebuttal PDF) the effect of running our method for a shorter time. It can be seen that after 5% (100 epochs) of the total running time,  our method already outperforms the baseline in terms of average Chamfer distance.  Given more running time reduces our error rate even more. \n\n\nQ3: Intermittent patterns in the legs of the chair\n\nThe thin structure of the input points is too weak for constraining the surface of the outside chair presented in Figure 3.  Unfortunately, the SDS loss does not help in making the surface thicker since the rendering of the thin surface looks valid. This is due to our rendering process that uses VOLSDF which defines a smooth mapping from surface to density (Equation 5), and since the entire leg has SDF values that are close to 0, the legs get densities that produce valid low-resolution renderings. We demonstrate the issue in Figure 1 (Rebuttal PDF).   We will discuss this limitation in the revised version of the paper.  \n\n\nQ4: Is the low rendering resolution 80x80 a bottleneck? \n\nDreamFusion is trained with 64x64 rendered images. In general, higher resolution requires higher GPU memory where 64x64,80x80,128x128 require 16GB(Google Colab), 32GB(V100), and 80GB(A100)  memory respectively.  \n"}, {"review_id": "Gq5XQASj9s", "reviewer": "Reviewer_zxYf", "summary": "This paper tackles point cloud completion with SDS-Complete, which aims at improving Out-Of-Distribution results with a pre-trained Stable Diffusion model for score distillation sampling (SDS). Specifically, the author propose to learn the MLPs for SDF and color with rendering-based losses with respect to the depth and the mask from the sensor's perspective. For novel views, SDS-Complete guides the rendering results with SDS which enforces a visual-language semantic. Experiments are conducted on the Redwood 3D Scans dataset with state-of-the-art results for OOD objects. The author also provides qualitative results on the KITTI LiDAR dataset.", "questions": "- Will the performance be affected if fewer initial points are used as the input?\n- How much do the depth and mask inputs help in terms of performance?", "limitations": "The author commented on the limitations of SDS-Complete in terms of the huge number of sampling views and the topology of objects to be completed. It would be good if they can further specify the runtime and the number of sampling views.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Figures are clear and method descriptions are detailed.\n- Results of comparison experiments are encouraging.\n- The motivation is clear, and the relevant pipeline design is intuitive and effective.", "weaknesses": "- Limited technical contribution. Including pre-trained vision-language models to improve out-of-distribution results does not bring new insights to the readers and the approach to include it (score distillation sampling in this paper) is not new.\n- Limited evaluation. The experimental results are not enough to support this paper's claim. It would be good if the author can provide results on ShapeNet and other datasets.\n- Weak ablation study. How much does SDS help with OOD data? It would be good if the author can provide step-by-step ablation results and more analysis on improvements with respect to OOD data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the comments and the feedback from the reviewer.\n\nW1: Technical Contribution and Novelty:\n\nTo the best of our knowledge, and as stated by reviewer 1, our method is the first to use a pre-trained vision-language model for completing point clouds. That allows us to complete partial point clouds from unseen classes, and without collecting any 3D data, as required by previous approaches. Using the SDS-loss for point cloud completion is far from simple, as indicated by the ablation study in Figure 6 (main paper). \n\n\nW2: provide results on ShapeNet \n\nOur work addresses the problem of completing point clouds captured ``in the wild”, rather than in close-world synthetic data. We expect methods trained to complete PCs of specific predefined classes on synthetic data to perform well in that setting, but as our paper demonstrates, they struggle when applied to open-world settings. We focus the experiments on real partial point cloud data captured by real sensors rather than on synthetic data sampled artificially from CAD models. Unlike the previous methods, our method was not trained on synthetic datasets so it would be unfair to base the comparisons on these datasets. Both, our method and the baselines, were not trained on real cases, so we find it fair to base the comparison on real-world data.\n\n\nW3: How much does SDS help with OOD data?\n\nFigure 6 in the main paper provides a quantitative ablation study addressing this question.  It shows that without the SDS loss, the error is significantly higher.  Following the reviewer’s request we present in a new Table 1  (Rebuttal PDF) the numbers with separated columns for in-distribution and OOD objects.\n\n\nW4: Will the performance be affected if fewer initial points are used as the input?:\n\nTo address the reviewer’s question we run our method with 50% and 10% of the original input points for the evaluation set from the Redwood dataset. We found that there is no significant difference in the results (up to 1.5 mm difference in terms of average Chamfer distance). These results indicate that our method is robust to the number of input points. We will include this experiment in the revised paper.\n\n\nQ1: How much do the depth and mask inputs help in terms of performance?:\n\nFollowing the reviewer’s question we add to the ablation study two rows: “No Depth” and “No Mask”. The numbers are presented in Table 1 of the rebuttal PDF.  It can be seen that “No Mask” has a more significant impact on the accuracy, and that without the depth loss, the error is higher by about 10 percent.\n\n\nQ2: Further specifying the runtime and the number of sampling views:\n\nSee supplementary, Section 5, “Running time”. Our test time optimization method is slow compared to the feedforward models of the baseline methods, but works much better on real point clouds, and does not require any dataset of 3D shapes for training as required by the baseline methods.  \n"}, {"review_id": "ASXNAQ6O4O", "reviewer": "Reviewer_MGrG", "summary": "This paper proposes a optimization method for point cloud completion by leveraging a 2D diffusion model. By constructing a neural SDF field, the proposed method make sure that the represented object matches the input partial point cloud, while the rendering images fit a pre-trained Stable Diffusion by the guidance of SDS loss. The results show that the proposed method does not need large dataset like previous works do. ", "questions": "1. I notice that you still have a color MLP head for the object even though your task is just point cloud completion. Is it because rendering colored images is more favorable for Stable Diffusion? Is it possible to take off the color head and directly assign a fixed color to your neural SDF and add the corresponding color to the input text for Stable Diffusion to reduce the complexity of your neural SDF?\n2. Please respond to the weakness part.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method does not need large dataset for training class specific completion model, making it more practical.\n", "weaknesses": "1. Looks like the proposed method simply adds a new constraint (matching surface with input partial point cloud) on DreamFusion-like structure and the rest are just existing modules and techniques. Although I believe this is technically not easy, the novelty is still limited. \n2. In Figure 1, the author mentioned in distribution object and out-of-distribution object. However, this grouping method is meaningless for the proposed method since a pre-trained Stable Diffusion is utilized. The real out-of-distribution object for this model should be an object that even stable diffusion cannot generate a reasonable image for it, and clearly this situation is fatal for the proposed method. I think the author should not emphasize in or out-of-distribution here.\n3. The comparisons in this paper seem to be unfair since previous works are trained under a totally different setting. The authors should show results of the same dataset (ShapeNet) with the previous works, which I believe existing works would outperform the proposed method.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the comments and the feedback from the reviewer.\n\nW1: Technical Contribution and Novelty:\n\nTo the best of our knowledge, and as stated by reviewer 1, our method is the first to use a pre-trained vision-language model for completing point clouds. That allows us to complete partial point clouds from unseen classes, and without collecting any 3D data, as required by previous approaches. Using the SDS-loss for point cloud completion is far from simple, as indicated by the ablation study in Figure 6 (main paper).   \n\n\nW2: The author should not emphasize in or out-of-distribution in Figure 1:\n\nWe thank the reviewer for this comment, and we will update the figure accordingly.\n\n\nW3: The authors should show results on Shapenet:\n\nOur work addresses the problem of completing point clouds captured ``in the wild”, rather than in close-world synthetic data. We expect methods trained to complete PCs of specific predefined classes on synthetic data to perform well in that setting, but as our paper demonstrates, they struggle when applied to open-world settings. We focus the experiments on real partial point cloud data captured by real sensors rather than on synthetic data sampled artificially from CAD models. Unlike the previous methods, our method was not trained on synthetic datasets so it would be unfair to base the comparisons on these datasets. Both, our method and the baselines, were not trained on real cases, so we find it fair to base the comparison on real-world data.\n\n\nQ1: Is it possible to take off the color head?\n\nWe thank the reviewer for the interesting suggestion. We implemented this suggestion by setting a constant gray color to be the color of the object and changing the text prompt to contain the word “gray” in the object description. For rendering the object details we applied shading as suggested by [10]. The results are comparable to the results with the RGB branch in terms of Chamfer distance on the evaluation set (31.3mm compared to 30.5mm with the color MLP head). We conclude that such renderings are in the distribution of Stable Diffusion such that they can be used for completing the surface correctly given the signal from the point cloud. The color MLP head does help in understanding the semantic content that is being generated, especially in failure cases, but it can definitely be omitted for simplifying the network architecture if resources are limited. We will discuss this in the revised paper.  \n"}]

Author rebuttal:
We thank the reviewers for their insightful comments. Below we answer separately for each reviewer.  

Meta-review: The scores are mix. The AC carefully read all the review comments and the authors' responses, and think evenly the setting requires more inputs and heavy computing cost. While using 2D prior model for point completion task is very new, and the results are also promising. Thus, The AC vote to accept.
Decision: Accept (poster)