DiffComplete: Diffusion-based Generative 3D Shape Completion
Authors: Ruihang Chu
Enze Xie
Shentong Mo
Zhenguo Li
Matthias Nießner
Chi-Wing Fu
Jiaya Jia
Venue: NeurIPS 2023 poster (2023)
Keywords: 3d shape completion
conditional generation
diffusion models

Abstract: We introduce a new diffusion-based approach for shape completion on 3D range scans. Compared with prior deterministic and probabilistic methods, we strike a balance between realism, multi-modality, and high fidelity. We propose DiffComplete by casting shape completion as a generative task conditioned on the incomplete shape. Our key designs are two-fold. First, we devise a hierarchical feature aggregation mechanism to inject conditional features in a spatially-consistent manner. So, we can capture both local details and broader contexts of the conditional inputs to control the shape completion. Second, we propose an occupancy-aware fusion strategy in our model to enable the completion of multiple partial shapes and introduce higher flexibility on the input conditions. DiffComplete sets a new SOTA performance (e.g., 40% decrease on $l_1$ error) on two large-scale 3D shape completion benchmarks. Our completed shapes not only have a realistic outlook compared with the deterministic methods but also exhibit high similarity to the ground truths compared with the probabilistic alternatives. Further, DiffComplete has strong generalizability on objects of entirely unseen classes for both synthetic and real data, eliminating the need for model re-training in various applications.

Dif Diffusion-based Generative 3D Shape Completion
Ruihang Chu
Enze Xie Matthias Nießner
Shentong Mo
Zhenguo Li
Chi-Wing Fu!
Jiaya Jia
TTee Chinese University of Hong Kong Huuawei Noah's Ark Lab MBZUAI
*Technical University of Munich "SmartMore
https: attpe://rulhangchu. com/ di aiffcemplete htmll
Abstract
We introduce a new diffusion- approach for shape completion on 3D range scans. Compared with prior deterministic and probabilistic methods, we strike a balance between realism, molti--mddality, and high fidelity. We propose DiffCom- plete by casting shape completion as generative task conditioned on the incom- plete shape. Our key designs are two-fold. First, we devise a hierarchical feature aggregation mechanism to inject conditional features in a spatially sottially-oonsiteent man- ner. So, we can capture both local details and broader contexts of the conditional inputs to control the shape completion. Second, propose occupancy-awe fusion strategy our model to enable the completion of multiple partial shapes and introduce higher flexibility
the input conditions. DiftComplete sets new SOTA performance (e.g., 40% decrease on I error) on two large-scale 3D shape comple- tion benchmarks. Our completed shapes only have realistic outlook compared with the deterministic methods but also exhibit high similarity to the ground truths compared with the probabilistic alternatives. Further, DifiComplete has strong generalizability on objects of entirely unseen classes for both synthetic and real data, eliminating the need for model re-training in various applications.
Input
Output
(a)
(b)
Figure Our method is able to (a) produce realistic completed shapes from partial scans, (b) incorporate multiple incomplete scans (denoted by the plus symbol) to improve the completion accuracy, and (c) directly generalize to work on real objects of unseen classes without finetuning.
Introduction
The advent of affordable range sensors, such as the Microsoft Kinect and Intel RealSense, has spurred remarkable progress in 3D reconstruction [1], facilitating applications in content creation, mixed reality, robot navigation, and more. Despite the improved reconstruction quality [2-5]. typical scanning sessions cannot cover every aspect of the scene objects. Hence, the reconstructed 3D models often have incomplete geometry, thereby limiting their practical usage in downstream applications.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

To fully unlock the potential of 3D reconstruction for supporting assorted applications, it is essential to address the challenges of completing incomplete shapes.
Effectively, a shape completion should produce shapes that are realistic, probabilistic, and high- fidelity. First, the produced shapes should look plausible without visual artifacts. Second, considering the under-determined nature of complction, it is desirable for the model to generate diverse, multi- modal outputs when filling in the missing regions, to improve the likelihood of obtaining superior shape and enable creative use. Third, while multiple outputs encourage the model's generative ability, maintaining effective control over the completion results is crucial to ensure coherent reconstructions that closely resemble the ground-truth shapes.
Current approaches 3D shape completion fall into deterministic and probabilistic paradigms. The former class excels at aligning predictions with ground truths, owing to their determined mapping functions and full supervision. However, this can expose the models to higher risk of over-fitting, leading to undesirable artifacts, such as rugged surfaces, especially on unseen objects. On the other hand, probabilistic approaches formulate the shape completion as a conditional generation task to produce plausible results, paired with techniques like autoregressive model [6], adversarial training [7-9], or diffusion models [10-13]. These approaches mainly focus on cases that prioritize completion diversity, for instance, filling in large missing regions cropped out by 3D boxes, where the algorithm has more freedom to explore various plausible shapes. However, when completing shapes obtained from range scans/reconstructions. whose geometries are often contaminated with noise, having varying degrees of incompleteness (see row in Fig. 1), a promising goal is to recover the ground-truth scanned objects as accurately as possible. Therefore, relying solely on previous probabilistic approaches may compromise the completion accuracy and leads to low-fidelity results.
In this work, we focus on completing shapes acquired from range scans, aiming to produce realistic and high-fidelity completed shapes, while considering also the probabilistic uncertainty. We approach this task as a conditional generation with the proposed diffusion model named DifiComplete. To address the accuracy challenge typically in probabilistic models, we introduce key design that incorporates range scan features in a hierarchical and spatially-consiste manner to facilitate effective controls. Additionally, we propose novel accupancy-award fusion strategy that allows taking multiple partial shapes as input for more precise and controllable completions.
Specifically, we formulate the diffusion process on a truncated distance field in volumetric space. This representation allows us to encode both complete and incomplete shapes into multi-resolution structured feature volumes, thereby enabling their interactions at various network levels. At each level, features are aggregated based on voxel-to-voxel correspondence, enabling precise correlation of the difference in partialness between the shapes. By leveraging multi-level aggregation, the completion process can be controlled by both local details and broader contexts of the conditional inputs. It effectively improves the completion accuracy and network generalizability, as illusrated in Sec. 4.5.
To condition the completion on multiple partial shapes, our occupancy- fusion strategy adopts the occupancy mask as a weight to adaptively fuse all observed geometry features from various incomplete shapes, and then employ the fused features to control the completion process. For robustness, such operation is performed in a multi-scale feature space. With our novel design, switching between single and multiple conditions can be efficiently achieved by finetuning an MLP layer. As Fig. ((b) and 6 show, using multiple conditions for completion progressively refines the final shapes towards the ground truths.
DiffComplete sets new state-of-te oh-the-art performance on two large-scale shape completion bench- marks [14, 15] in terms of completion accuracy and visual quality. Compared to prior deterministic and probabilistic approaches, we not only generate multimodal plausible shapes with minimal artifacts but also present high shape fidelity relative to the ground truths. Further, DiffComplete can directly generalize to work on unseen object categories. Without specific zero-shot designs, it surpasses all existing approaches for both synthetic and real-world data, allowing us to eliminate the need for model re-training in various applications. To sum up, our contributions are listed follows:
We introduce a diffusion model to complete 3D shapes on range scans, enabling realistic. probabilistic, and high-fidelity 3D geometry.
We enhance the completion accuracy with two designs. For single condition, we propose a hierarchical feature aggregation strategy to control the outputs. For multiple conditions, we introduce occupancy- fusion strategy to incorporate more shape details.

---

We show SOTA performance on shape completion on both novel instances and entirely unseen object categories, along with an in-depth analysis on a range model characteristics.
2
Related Work
RGB-D reconstruction. Traditional methods rely geometric approaches for 3D reconstruction [16-
20]. A pioneering method [21] proposes a volumetric fusion strategy to integrate multiple range images into unified 3D model on truncated signed distance fields (TSDF), forming the basis for
many modern techniques like KinectFusion [22. 23]. VoxelHashing [24], and BundleFusion [2]. Recent learning-based approaches further improve the reconstruction quality with fewer artifacts [25-
28, 3], yet the intrinsic occlusions and measurement noise of 3D scans constrain the completeness of 3D reconstructions, making them still less refined than manually -ereatee assets.
3D shape completion is a common post-processing step to fill in missing parts in the reconstructed shapes. Classic methods mainly handle small holes and geometry primitives via Laplacian hole filling [29-31] or Poisson surface reconstruction [32, 33]. Another line exploits the structural regularities of 3D shapes, such as the symmetries, to predict the unobserved data [34-38].
The availability of large 3D data has sparked retrieval-based methods [39-42] and learning- based fitting methods [43, 44, 14, 45-49]. The former retrieves the shapes from a database that best match the incomplete inputs, whereas the latter minimizes the difference between the network-predice shapes and ground truths. 3D-EPN [14], for instance, proposes a 3D encoder-decoder architecture to predict the complete shape from partial volumetric data. Scan2Mesh [50] converts range seans into 3D meshes with direct optimization on the mesh surface. PatchComplete [15] further leverages local structural priors for completing shapes of unseen categories.
Generative methods, e.g., GANS [51, 9, 52, 7, 8] and AutoEncoders [53], offer an alternative to shape completion. While some generative models can generate diverse global shapes given partial input, they potentially allow for a high generation freedom and overlook the completion accuracy. IMLE [54], for instance, adopts an Implicit Maximum Likelihood Estimation technique to specially enhance structural variance among the generated shapes. Distinctively, we formulate a diffusion model for shape completion. Our method mainly prioritizes fidelity relative to ground truths while preserving output diversity. Compared to both generative and fitting- based paradigms, our method also effectively reduces surface artifacts, producing more realistic and natural 3D shapes. addition, we show superior generalization ability on completing objects of novel classes over SOTAs.
Diffusion models for 3D generation. Diffusion models [55-62] have shown superior performance in various generation tasks, outperforming GANs' sample quality while preserving the likelihood evaluation property of VAEs. When adopted in 3D domain, range of works [12, 63-65] focus point cloud generation. For more complex surface generation, some works [66, 13, 10, 67, 68, 11] adopt latent diffusion models to learn implicit neural representations and form final shapes via a decoder. For conditional shape completion, both DiffRF [10] and Diffusion-SDF [69] adopt a masked diffusion strategy to fill in missing regions cropped out by 3D boxes. However, their training processes do not leverage paired from accurately learning the completion rules. Contrarily, our method explicitly uses the scan pairs for conditional training, yielding outputs closely resembling the actual objects scanned. Also, we apply the diffusion process in explicit volumetric TSDFs, preserving more geometrical structures during the completing process.
ground truth, which may prevent them
3 Method
3.1 Formulation
To prepare the training data, we generate an incomplete 3D scan from depth frames using volumetric fusion [21] and represent the scan as truncated signed distance field (TSDF) in volumetric grid. Yet, to accurately represent a ground-truth shape, we choose to use a volumetric truncated unsigned distance field (TUDF) instead. This is because retrieving the sign bit from arbitrary 3D CAD models (e.g., some are not closed) is non-trivial. By using TUDE, we can robustly capture the geometric features of different objects without being limited by the topology.
3

---

Given such volume pair, ie., the incomplete scan and complete 3D shape zo, we formulate the shape completion as a generation task that produces Tn conditioned on c. We employ the probabilistic diffusion model as our generative model. In particular, it contains (i) a forward process (denoted as ((Fo:T) that gradually adds Gaussian noise to corrupt the ground-truth shape 2g into a random noise volume 2T, where T' is the total number of time steps; and (ii) backward process that employs a shape completion network, with learned parameters 0, to iteratively remove the noise from #T conditioned the partial scan and produce the final shape, denoted po po (Zo.T, c). As both the forward and backward processes are governed by a discrete-time Markov chain with time steps {0, ..., T}, their Gaussian transition probabilities can be formulated
((FO:T) (200) II g( g(=.(1=1) = NVVI -,,0--1,
(1)
T
and (2 c) = p(xT) II po(11_1/00 c) po(x)-1101) == N(uo((
(2)
In the forward process (Eq. (I)), the scalars B. E [0, control a variance schedule that defines the amount of noise added in each step t. In the backward process (Eq. (2)), p(zr) is a Gaussian prior in time step l, [ág represents the mean predicted from our network and of is the variance. As suggested in DDPM [60], predicting [i (1re c) can be simplified to alternatively predicting ea ca I, c). which should approximate the noise used to corrupt 14-1 i in the forward process, and o, can replaced by can optimize the network parameter & with a mean
the pre-defined B.. With these modifications, squared error loss to maximize the generation probability po (zo). The training objective is
arg min oo.zo: ca( L, e) IO,))
0
where is the applied noise to corrupt Ho into I3 and (N I) denotes a unit Gaussian distribution. We define all the diffusion processes in volume space of resolution 5 i.e., To.T, C, R where each voxel stores a scalar TSDF/TUDF value; 5=32 in our experiments. Compared to previous latent diffusion models [11, 13, 66. 10] that require shape embedding first, we directly manipulate the shape with a better preservation of geometric features. Doing so naturally enables our hierarchical feature aggregation strategy (see Sec. 3.2). Next, we will introduce how to predict Eg (Iz, t).
3.2 Shape Completion Pipeline
Overview. To enhance the completion accuracy, we encourage the incomplete scans to control completion behaviors. Inspired from the recent ControlNet [70], which shows great control ability given 2D conditions, we adopt a similar principle to train an additional control branch. To predict the noise eø EI,,,, c) in Eq. (3), we encode the corrupted ground-truth shape Ie by main branch and the partial shape by a control branch, where two branches have the same network structure without parameter sharing. Owing to the compact shape representation in 3D volume space, both complete and incomplete shapes are encoded into multi-resolution feature volumes with preserved spatial structures. Then, we hierarchically aggregate their features at each network level, as the sizes of two feature volumes are always aligned. In this work, we simply add up the two feature volumes to allow for a ppatially-consister are combined. Compared with frequently-used cross-attention technique [11, 1, 13, 57], doing can significantly reduce computation costs. By multi-scale feature interaction, the network can effectively correlate the difference in partialness between two shapes, both locally and globally, to learn the completion regularitics. The final integrated features are then used to predict the noise volume Eg.
feature aggregation, i.e., only features at the same 3D location
Network architecture. Fig. 2 provides an overview of our network architecture, which has main branch and a control branch respectively handle complete and incomplete shapes. The main branch (upper) is a 3D U-Net modified from 2D version [59]. It takes as input an corrupted complete shape 24, including (i) pre- pre-processing block (-) of two convolution layers to project 2; into a high-dimension space, (ii) N subsequent encoder blocks, denoted as {F) (-) to progressively encode and downsample the corrupted shape x, into a collection of multi-scale features. (iii) a middle block Mz (.) with a self-attention layer to incorporate non-local information into the encoded feature volume, and (iv) N decoder blocks (D& ()) that sequentially upsample features through an inversion convolution to produce feature volume of the same size as the input 2;. The term

---

(x)
Main branch
t
Encoder block 1
Middle block
Decoder block
co(XE, E)
Complete shope (Xi xx)
Pool Norm
& foomm
S (
(EN(
Binarization
Binanization
Encoder block
Middle
Projection loyer
Incomplete scon(s)
Control branch
Occupancy-awarc fusion
Given a corrupted complete shape Ty (diffused from xo) and an incomplete scan c, we first process them into (x2) and E<(c) to align the distributions. We employ main branch to forward Ex (2,), and a control branch to propagate their fused features f into deep layers. Multi-level features of f are aggregated into the main branch for hierarchical control in predicting the diffusion noise. To support multiple partial scans as condition, e.g., two scans {cl. C2 we switch on occupancy aecupancy-aware aware fusion (Sec. 3.3). This strategy utilizes the occupancy masks to enable weighted feature fusion for c1 and (2 by considering their geometry reliability before feeding them into the main branch.
Figure 2:
"eld)ncoder/middle block" denotes group of neural layers commonly adopted as unit in neural networks, e.g., "resnet-downsanple block.
We use a control branch (bottom) to extract the features incomplete scan(s). Likewise, we employ pre-processing block, N encoder blocks, and middle block extract multi-scale features from the conditional input, denoted as s.(-). (-)) and Ma, respectively. They mirror the architecture of the corresponding blocks in the upper branch, yet operating with non-shared parameters, considering the different input data regularities (e.g., complete and sparse). Unlike the upper branch, there are no decoders for computation efficiency. Instead, we append projection layer after each encoder/middle to forward multi-scale features and feed them into the decoder blocks of the main branch. As diffusson models are time-conditioned, we also convert the time step into an embedding via two MLPs and add it to the volume features in each network block (the blue box in Fig. 2).
block F Ft/M, /M,
Hierarchical feature aggregation. To integrate the features of the complete and incomplete shapes for conditional control, we fuse them across multiple network levels. In this process, we first consider single incomplete shape (denoted as and Ty (incomplete v.s. complete, TSDF v.s. TUDF), we employ the pre- mee-pooesssing layers (sx (-) and Ee(-) to first empirically project the two different fields into fuse them into feature volume f. i.e., f (1²) + Ec(c). After that, f is passed through the control branch better propagate the useful information to the deeper layers. In turn, we use multi-scale condition features from the control branch to guide each level of decoder blocks (D2()) in the main branch. The features that enter the i-th decoder block are denoted as
the condition. Given the significant disparity between c
compatible feature space. Then
d' [D (..))) + (F((()) f f= + +E((c)
where [ is the concatenation operation and of one x 1 convolution layer for feature projection. For simplicity, we use (#0) to denote the features of the after the i-th encoder block (.).). To not directly processed by (+): the same notation used for (f) and D Di-1 D The feature after the final decoder block is the network output (x c).
By such design, we can hierarchically incorporate conditional features, leveraging both their local and broader contexts to optimize the network outputs. Interestingly. we observe that adjusting the network level for feature aggregation can alter trade-off between completion accuracy and diversity. Specifically, if we only aggregate features at the low-resolution network layers, we might miss finer geometry details present in the higher-resolution layers. This could reduce completion accuracy, as the model has less contextual information to work with. In contrast, if aggregating features at all levels, the effective control might make completion results closely resemble the ground truths, yet lowering the diversity. Further ablation study is provided in Sec. 4.5.
5

---

3.3 Occupancy-aware Fusion
Our framework can also take multiple incomplete scans as inputs. This option not only provides richer information to constrain the completed shape geometry but also enhances the approach's s practicality, particularly scenarios that a single-pass scan may not fully capture the entire object. Our critical design is to effectively fuse multiple partial shape features. As averaging the original TSDF volumes is sensitive to registration noise, we first register multiple partial shapes (e.g., using Fast-Robuus Rost-Robst-ICCP [71]) and propose an occupancy- approach to fuse them in the feature space.
Given a set of incomplete shapes of the same object, denoted as {cl..., CMr). we individually feed them into the control branch to produce feature volumes {f2(f)),.... ("(( [f)) after the i-th encoder block. Before feeding them into the decoder D we refer to their occupancy masks perform weighted feature average. Concretely, we first compute the original occupancy mask for each partial shape based on the TSDF values, i.e., w = (abs(c,) < 7) E x for the j-th partial shape. is a pre-defined threshold that assigns the volumes near the object surface as occupied and the rest as unoccupied; B is a binary tensor. Then, we perform a pooling operation to resize wi into w to align the resolution with feature volume (fy) and normalize w by w/- = w (w/ + + whr) For the voxels with zero values across all oceupancy masks, we uniformly assign them 1e-2 value to avoid the division-by-zer issue. We rewrite Eq. (4), which is for single partial shape condition, as
d' = ( (I.). - 2 L (F
fj =-= + Ea(c))
where sb is MLP layer to refine the fused condition features, aiming to mitigate the discrepancies among different partial shape features, as validated in Sec. 4.4. The right part of Fig. 2 illustrates the above process with two incomplete scans the inputs.
3.4 Training and Inference
We first train the network with a single incomplete shape as the conditional input. this phase, all the network parameters, except the MLP layer v for occupancy-ave fusion (in Eq. (5)), are trained with the objective in Eq. (3). When the network converges, we lock the optimized network parameters and efficiently finetune the MIP layer ve with multiple incomplete shapes as input.
At the inference stage, we randomize a 3D noise volume as 2T from the standard Gaussian distribu- tion. The trained completion networks are then employed for I' iterations to produce zo from JT conditioned on partial shape(s) c. The occupancy-awand fusion is activated only for multi-condition completion. To accelerate the inference process, we adopt a technique from [61] to sub-sample set of time steps from [1...,[/10] during inference. After obtaining the generated shape volume Tu, we extract an explicit 3D mesh using the marching cube algorithm [72].
4 Experiment
4.1 Experimental Setup
Benchmarks. We evaluate on two large-scale shape completion benchmarks: 3D-EPN [14] and PatchComplete [15]. 3D-EPN comprises 25,590 object instances of eight classes in ShapeNet [73]. For each instance, six partial scans of varying completeness are created in the 32 TSDF volumes by virtual scanning; the ground-truth counterpart, represented by 32 TUDF, obtained by a distance field transform on 3D scanline method [74]. While using similar data generation pipeline, PatchComplete emphasizes completing objects of unseen categories. It includes both the synthetic data from ShapeNet [73] and the challenging real data from ScanNet [75]. For fair comparison, follow their data splits and evaluation metrics, i.e., mean ( error on the TUDF predictions across all voxels on 3D-EPN, and ( Chamfer Distance (CD) and Intersection over Union (IoU) between the predicted and ground- -ruth shapes on PatchComplete. As these metrics only measure the completion accuracy, we introduce other metrics in Sec. 4.3 to compare multimodal completion characteristics.
Implementation details. We first train our network using single partial scan as input by 200k iterations on four RTX3090 GPUs, taking around two days. If multiple conditions are needed, we finetune project layers t for additional 50k iterations. Adam optimizer [76] is employed with a learning rate of le-1 and the batch size is 32. On the 3D-EPN benchmark, we train specific

---

Table 1:
Quantitative shape completion results on objects of known categories [14].
li-err (J)
Avg. (C)
Chair
Table
Sofa
Lamp
Plane
Car
Dresser
Boat
3D-EPN [14]
0.374
0.418
0.377
0.392
0,388
0.421
0.259
0.381
0.356
SDF-StyleGAN [51]
0.278
0.321
0.256
0.289
0.280
0.295
0.224
0.273
0.282
RePaint-3D [78]
0.266
0.289
0.264
0.266
0.268
0.302
0.214
0.285
0.243
ConvONet [26]
0.220
0.210
0.247
0.254
0.234
0.185
0.195
0.250
0.184
AutoSDF [6]
0.217
0.201
0.258
0,226
0.275
0.184
0.187
0.248
0.157
cGCA [79]
0.185
0.174
0.212
0.179
0.239
0.170
0.161
0.204
0.143
ShapeFormer [80]
0.141
0.104
0.175
0.133
0.176
0.136
0.127
0.157
0.119
PVD [12]
0.114
0.097
0,122
0.154
0.128
0.093
0.087
0.127
0.107
PatchComplete [15]
0.088
0.134
0.095
0.084
0.087
0.061
0.053
0.134
0.058
DifiComplete (Ours)
0.053
0.070
0.073
0.061
0.059
0.015
0.025
0.086
0.031
Table Shape completion results on synthetic objects [73] of unseen categories. / means CD/IoU.
CDC->IoUUitt)DDEEP [14] Shot [F- [49] Auto-SDi [6] ConvONet [26] PatchComplete [1S]
Ours
Bag
5.01 / 73.8
8.00/56.1
69.8
5.81 56.3
5.10 70.8
3 77.6
3.86 78.3
Lamp
8.07 /47.2
15.1/25.4
5.70/50.8
6.57 39.1
5,42 52.6
4.68 56.4
4.80 57.9
Bathtub
4.21 57.9
7.05/45.7
4.72/55.0
5.17 41.0
4.96 60.4
66.3
3.52/ 68.9
58,4
10.0/39.6
5.34/60.7
6.01 /4.66
5..2/63.2
4.49 66.8
4.16 67.1
Basket
7..0/54.0
8.72/40.6
4.44/50.2
6.70/39.8
6.16/ 54.6
5.15/61.0
4,94 65.5
Printer
5.15/73.6
9.26/56.7
5.83/70.5
7.52 49.9
5.56
4
4,40 76.8
Laptop
3.90/62.0
10.4/31.3
6.47
4.81/
4.78/57.3
/
/
Bench
4..4/48.3
8.11/27.2
5.03/49.77
4.31/39.5
49.66
3.70/53.9
3,56 58.2
Avg.
5.58/ 59.4
9.58
5.29
5.86 45.2
5.26/ 60.1
4.27 /5.4
4.10 67.5
Table 3: Shape completion results on real-world objects [75] of unseen categories. // means CD/IoU.
CD(LY\oUU())3D-EEN [14] Few-Shot [81] IF-Ness [49] Auto-SDF [6] ConvONet [26] PutchComplete [15]
Ours
Bag
8.83 53.7
9.10 44.9
8.96 44,2
9.30 48.7
9.12 52.5
8.23 58.3
7.05 / 48.5
Lamp
14.3 20.7
11.9 19.6
10.2/24.9
11.2/ 24.4
9.83 / 20.3
9.42 / 28.4
6,84/ 30.5
Bathtub
7.56/41.0
7.77
3.1
7.84 36.6
7.93/41.2
6.77 /
8.22 / 48.5
Bed
7.76/47.8
9.07 /34.9
8.24/44.9
7.91 / 38.0
8.14/41.6
7.24 / 48.4
7,20 / 46.6
Basket
/6.5
/303
6.74/42.7
7.54 36.1
7.39/37.0
6.60 455.5
7.42 59.2
Printer
63.0
8.30 62.2
8.28 60.7
9.66 / 49.9
7.62 64.9
6,84 70.5
6,36 74.5
Avg.
9409
9.02/38.6
8.26
8.90 38.9
8.34 /4..9
7.52
/1.33
model for completing shapes of each known category; while on PatchComplete, we merge all object categories to optimize one model for promoting general completion learning. Due to the unknown class IDs at test time, no classifier-uuided [56] or classifier-free [77] sampling techniques are used in our diffusion model. On both two benchmarks, all training data are the same across the compared methods for fairness. The truncation distance in TSDF/TUDFs is set as voxel units. More details about network architecture and experiments available in the supplementary file. Unless otherwise specified, we report the results on single partial shape completion.
4.2 Main Results
Completion on known object categories. On the 3D-EPN benchmark, we compare DiffCom- plete against SOTA deterministic [14, 15, 51] and probabilistic [6. 78] methods in terms of completion accuracy (i.e., 1 errors). For probabilistic methods, we use the average results from five inferences, each with random initialization, account for multimodal outcomes. As shown in Table 1 and Fig. 3, DiffComplete improves over state of the arts by 40% on 1 error (0.053 v.s. 0.088), well as producing more realistic and high-fidelity shapes. Deterministic methods include 3D-EPN [14], ConvONet [26), and PatchComplete [15]. Unlike these learning a one-step map function for shape completion, we iteratively refine the generated shape, thus significantly mitigating the surface ar- tifacts; see visualization comparisons in Fig. 3. Compared to GAN-based SDF-StyleGAN [51] and Autoregressive--assee AutoSDF [6], our diffusion-basee generative model offers superior mode coverage and sampling quality.
7

---

Input
3D-EPN
AutoSDF
ConvONet
ShapeFormer
PVD
Ours
GT
Figure 3: Shape Completion various known object classes. We achieve the best completion quality.
Input
Patch- Complete
Ours
GT
Figure 4: Shape completion on synthetic (blue) and real (yellow) objects of entirely unseen classes. Our method produces the completed shapes in superior quality given both synthetic and real data.
Regarding diffusion-basse approaches, RePaint-3D is adapted from Repaint [78], 2D diffusion- based inpainting method that only involves partial shape conditions during the inference process. In contrast, our DiffComplete explicitly matches each partial shape with a complete counterpart at the training stage, thereby improving the output consistency with the ground truths. PVD [12] is originally point cloud diffusion model. Here, we adapt it to perform TSDF (TUDF) diffusion. A limitation PVD that retains noises of the partial input. Hence, when mixed with generated missing part, noise severely affects the final completion quality. Instead, we design two branches to separately process the partial and complete shapes. By doing so, the model can learn diffusion process from noise to clean shapes.
Completion unseen object categories. two datasets of the PatchComplete benchmark, compare the generalizability of DifiComplete against the state of the arts, including approaches partic- ularly designed for unseen-class completion [81, 15]. summarized in Table 2, our method exhibits the best completion quality on average for eight unseen object categories in the synthetic ShapeNet data, despite lacking zero-shot designs. The previous SOTA PatchComplete, mainly leverages the multi-scale structural information to improve the completion robustness. Our method inherently embraces this concept within the diffusion models. With our hierarchical feature aggregation, the network learns multi-scale local completion patterns, which could generalize to various object classes,

---

Input
Multi-modal outputs
11-err00.56
14-err-0.12
11-err00.08
11-err=0.04
Figure 5: Our method produces multimodal plau- sible results given the same partial shape.
Figure Our method incorporates multiple par- tial shapes to refine completion results (11 -er. 4)-
Table 4: Multimodal capacity.
Figure 7: TMD curve.
Table 5: Multi-condition results.
Method
MMD TMD 1 UHD 1
AutoSDF [6]
0.008
0.028
0.061
Cond. Num.
[1-err. 4 MMD
TMD
ShupeFormer (80)
0.007
0.024
0.055
3.0
0.07
0.002
0.025
RePaint-3D [78]
PVD [12]
0.007
0.029
0.027
0.053
2.0
80 90
two three
0.05
0.04
0.001
0.001
0.019
0.021
0.025
Completeness Degree (9))
Table 6: Effects of fusion choices Table 7: Feat. aggregation levels. Table 8: Ablation different fea-
for multiple conditional inputs.
1/
1/2
MMD
CD
ture aggregation mechanisms.
Stratggy
I--em. 4 CD
loUT
0.005
0.031
4.8
Operation
Bree..
MMD
TMD
simple average
0.13
4.56
63.3
0.004
0.028
cross-actn.
0.12
0.005
0,027
w/o MLP i
0.23
5.88
57.4
0.003
0.026
4.2
concut.
0.07
0.002
0.024
oce-aware (Ours)
0.05
0.002
0.025
addition (Ours)
0.002
0.025
as their local structures are often shared. Our ablation study in Sec. 4.5 further validates this benefit. Table 3 demonstrates method's superior performance with real-world scans, which are often
cluttered and noisy. As showcased in Fig. 4, the 3D shapes produced by DiffComplete stand out for their impressive global coherence and local details.
4.3 Multimodal Completion Characteristics
Quantitative evaluations. Probabilistic methods have the intriguing ability to produce multiple plausible completions on the same partial shape, known as multimodal completion. For the 3D-EPN chair class, generate ten results per partial shape with randomized initial noise æT, and employ metrics from prior works [7, 6], i.e., (i) MMD measures the completion accuracy against the ground truths, (ii) TMD for completion diversity, and (iii) UHD for completion fidelity Table 4 shows that our method attains much better completion accuracy and fidelity, while exhibiting moderate diversity. This aligns with our design choice of leveraging the control mechanism to prioritize completion accuracy over diversity. Yet, we can adjust this trade-off to improve shape diversity, as discussed in Sec. 4.5. Fig. presents multimodal outputs, all showing great realism.
partial input.
Effects of input completeness degree. To verify the influence of input completeness degree on diversity, we select ten chair instances from ShapeNet, due to chair's 's large structural variation. For each instance, we create six virtual scans of varying completeness, and for each scan, we generate ten diverse completions to compute their TMD. Fig. 7 presents negative correlation between the shape diversity (reflected by TMD) and input completeness degree, which is measured by the occupied voxel's 's ratio between the partial and complete GT shapes.
4.4 Multiple Conditional Inputs
Quantitative evaluations. As indicated in Table 5, DiffComplete consistently improves completion accuracy (lower 1 error) and reduces diversity (lower TMD) when more conditional inputs are added. Fig. 1(b) and 6 showcase the progression of completion results when gradually introduce more partial shapes of the same object (denoted by the plus symbol). Our model incorporates the local structures of all partial inputs and harmonizes them into coherent final shape with the lower 1 error.
Effects occupancy-awan fusion. In Table 6, we compare our design with alternatives on the 3D-EPN chair class and PatchComplete benchmark using two conditional inputs. Averaging fea-

---

tures without occupancy masks largely lowers the completion accuracy due to disturbances from non-informative free-space features. Removing the learnable MLP layer & hinders the network's adaptation from single to multiple conditions, also worsening the results. Instead, we adaptively aggregate multi-condition features and refine them to mitigate their discrepancy for reliable com- pletions. Note that directly fusing original scans in TSDF (TUDF) space might be vulnerable to registration errors, as compared with our strategy in the supplementary file.
4.5 Ablation Study
Effects of hierarchical feature aggregation. Table 7 shows the effects of aggregating features of the complete and incomplete shapes at different decoder levels (see Eq. (4)). First, increasing feature aggregation layers (from single to hierarchical) consistently boosts the completion accuracy (lower MMD), while decreasing yields better diversity (higher TMD). If connecting features only at the network layer with 1/8 resolution, we achieve the best TMD that surpasses other methods (see Table 4). Thus, the accuracy-diversity trade-off can be adjusted by altering the level of feature aggregation. Second, hierarchical feature aggregation facilitates unscen-class completion (lower CD, tested on PatchComplete benchmark). This improvement suggests that leveraging multi-scale structural information from partial inputs enhances the completion robustness.
Effects of feature aggregation mechanism. We ablate on the way aggregate F) and d' (F in Eq. (4), As Table 8 shows, direct addition achieves the best completion accuracy (the lowest 1 lerrr and MMD), as only combines features the same 3D location to precisely correlate their difference for completion.
disrupts this spatial consistency and yields less accurate results. Concatenation has a similar performance with addition, while the latter is more efficient.
Conclusion and Discussion
We presented DiffComplete, new diffusion-basse based approach to enable multimodal, realistic, and high-fidelity 3D shape completion, surpassing prior approaches on completion accuracy and quality, This success is attributed two key designs: hierarchical feature aggregation mechanism for effective conditional control and an occupancy-awer fusion strategy to seamlessly incorporate additional inputs for geometry refinement. Also, DiffComplete exhibits robust generalization to unseen object classes for both synthetic and real data, and allows an adjustable balance between completion diversity and accuracy suit specific needs. These features position DiffComplete as powerful tool in various applications in 3D perception and content creation. Yet, as with most diffusion models, DiffComplete requires additional computation due to its multi-step inference. comprehensive discussion on the limitation and broader impact is provided in the supplementary file.
10

---

Acknowledgement
This work is partially supported by the Research Grants Council under the Areas of Excellence scheme grant AoE/E-601/22-R, Shenzhen Science and Technology Program KQTD202 108 1090149095, and the Research Grants Council of the Hong Kong Special Administrative Region, China (CUHK 14206320).
References
[1] Michael Zollhöfer, Patrick Stotko, Andreas Görlitz, Christian Theobalt, Matthias Nießner, Reinhard Klein, and Andreas Kolb. State of the 3d reconstruction with rgb-d cameras. Computer Graphics Forum, 77:625-652, 05 2018.
[2] Angela Dai, Matthias NicBuer, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reinte- gration. ACM Trans. Graph., 36(3):24:1-24:11 2017. 1,3
[3] Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6290-6301, 2022.
[4] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[5] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33:2492 2204422-2502, 2020.
[6] Paritosh Mittal, Yen-Chi Cheng. Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 306-315, 2022. 2, 7,9
[7] Rundi Wu, Xuelin Chen, Yixin Zhuang, and Baoquan Chen. Multimodal shape completion via conditional generative adversarial networks. Computer AssionEECVV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 281-296
Springer, 2020. 2, 3,
[8] Edward Smith and David Meger. Improved adversarial systems for object generation and reconstruction. In Conference Robot Learning, pages 87-96. PMLR, 2017. 3
[9] Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, and Chen Change Loy. Unsupervised shape completion through gan inversion. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 2021. 3.3
[10] Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner. Diffrf: Rendering-guiee 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4328 3328-4338, 2023. 3,
[11] Yen-Chi Cheng, Hsin- Ying Lee, Sergey Tulyakov, Alexander Schwing, and Liangyan Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. arXiv preprint urXiv:2212.04499 2022. 3,
[12] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826-5835, 2021. 3, 7, 8,9
[13] Gene Chou, Yuval Bahat, and Felix Heide. Diffusion-sdf- Conditional generative modeling of signed distance functions. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 2262-2272, 2023, 3,

---

[14] Angela Dai, Charles Ruizhongtai Qi, and Matthias Nießner. Shape completion using ed-encoder- predictor enns and shape synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5868-5877, 2017. 2017, 2, 3, 6, 7, 17, 18
[15] Yuchen Rao, Yinyu Nie, and Angela Dai. Patchcomplete: Learning multi cellitresooltion patch priors for 3d shape completion on unseen categories. arXiv preprint arXiv:2206.0449 2022.
2, 3, 6, 7, 8, 18, 23, 24
[16] Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part II 13, pages 849. Springer, 2014.
[17] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions robotics, 31(5):1147-116; 2015.
[18] Thomas Whelan, Stefan Leutenegger, Renato F. Salas-Moreno, Ben Glocker, and Andrew J. Davison. Elasticfusion: Dense SLAM without A pose graph. In Robotics: Science and Systems XI, Sapienza University of Rome, Rome, Italy, July 13-17, 2015, 2015.
[19] R. Maier, K. Kim, Cremers, Kautz, and M. Nießner. Intrinsic3d: High-quality 3D reconstruction by joint appearance and geometry optimization with spatially-varying lighting. In International Conference on Computer Vision (ICCV), Venice, Italy, October 2017.
[20] Michael Zollhöfer, Angela Dai, Matthias Innmann, Chenglei Wu, Mare Stamminger, Christian Theobalt, and Matthias Nießner. Shading-based refinement on volumetric signed distance functions. ACM Transactions on Graphics (TOG), 2015. 3
[21] Brian Curless and Mare Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Téchniques, SIGGRAPH 1996, New Orleans, LA, USA, August 4-9, 1996, pages 303-312, 1996.
[22] Shahram Izadi, David Kim, Otmar Hilliges, David Molyncaux, Richard A. Newcombe, Push- meet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew J. Davison, and Andrew W. Fitzgibbon. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In Proceedings of the 24th Anmual ACM Symposium on User Interface Software and Technology, Santa Barbara, CA, USA, October 16-13 (9, 2011, pages 559-568, 2011.
[23] Richard A. Newcombe, Shahram Izadi, Otmar Hilliges. David Molyneaux, David Kim, An- drew Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew W. Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In 10th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2011, Basel, Switzerland, October 26-29,
2011, pages 127-136, 2011.3 3
[24] Matthias Nicßner, Michael Zollhöfer, Shahram Izadi, and Mare Stamminger. Real-time 3d reconstruction at scale using voxel hashing. ACM Transactions on Graphics (ToG), 12(6):1-11, 2013. 3
[25] Silvan Weder, Johannes Schonberger, Mare Pollefeyy, and Martin R Oswald. Routedfusion: Learning real-time depth map fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4887 4897, 2020.
[26] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Con- volutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, 2020. 7, 19
[27] Ang Li, Zejian Yuan, Yonggen Ling, Wanchao Chi, Chong Zhang, et al. A multi-scale guided cascade hourglass network for depth completion. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 32-40, 2020.
[28] Angela Dai, Yawar Siddiqui, Justus Thies, Julien Valentin, and Matthias Nießner. Spsg: Self- supervised photometric scene generation from rgb-d scans. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 1747-1756, 2021.
12

---

[29] Olga Sorkine and Daniel Cohen-Or. Least-squares meshes. In Proceedings Shape Modeling Applications, 2004., pages 191-199. IEEE, 2004.
[30] Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Mare Alexa. Laplacian mesh optimization. In Proceedings of the 4th international conference on Computer graphics and interactive techniques in Australasia and Southeast Asia, pages 381-389, 2006.
[31] Wei Zhao, Shuming Gao, and Hongwei Lin. A robust hole-filling algorithm for triangular mesh. The Visual Computer, 23:987 997, 2007.
[32] Michael M. Kazhdan, Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction. In Proceedings of the Fourth Eurographics Symposium on Geometry Processing, Cagliari, Sardinia, Italy, June 26-28, 2006, pages 61-70, 2006. 3
[33] Michael Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction. ACM Transac- tions on Graphics (ToG), 32(3):1-13, 2013. 3
[34] Sebastian Thrun and Ben Wegbreit. Shape from symmetry. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, volume 2, pages 1831. IEEE, 2005.
3
[35] Niloy J Mitra, Leonidas J Guibas, and Mark Pauly. Partial and approximate symmetry detection for 3d geometry, ACM Transactions Graphics (ToG), 25(3):560- 2006.
[36] Mark Pauly, Niloy Mitra, Johannes Wallner, Helmut Pottmann, and Leonidas J Guibas. Discovering structural regularity in 3d geometry. In ACM SIGGRAPH 2008 papers. 2008.
[37] Ivan Sipiran, Robert Gregor, and Tobias Schreck. Approximate symmetry detection in partial
3d meshes. In Computer Graphics Forum, 2014.
[38] Pablo Speciale, Martin R Oswald, Andrea Cohen, and Mare Pollefeys. A symmetry prior for convex variational 3d reconstruction. In Computer Aeiio-EECVV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part VIII 14, pages 313-328. Springer, 2016. 3
[39] Minhyuk Sung, Vladimir G Kim, Roland Angst, and Leonidas Guibas. Data-driven structural priors for shape completion. ACM Transactions on Graphics (TOG), 34(6):1-11, 2015.
[40] Yangyan Li, Angela Dai, Leonidas Guibas, and Matthias Nießner. Database-assistee object retrieval for real-time 3d reconstruction. In Computer graphics forum, 2015.
[41] Liangliang Nan, Ke Xie, and Andrei Sharf. A search-classify approach for cluttered indoor scene understanding. ACM Transactions on Graphies (TOG), 3106:1110, 2012.
[42] Young Min Kim, Niloy Mitra, Dong-Ming Yan, and Leonidas Guibas. Acquiring 3d indoor environments with variability and repetition. ACM Transactions on Graphics 310:),,1 2012. 3
[43] Duc Thanh Nguyen, Binh-Son Hua, Khoi Tran, Quang-Hice Pham, and Sii-Kit Yeung. A field model for repairing 3d shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5676-5684, 2016. 3
[44] Michael Firman, Oisin Mac Aodha, Simon Julier, and Gabriel Brostow. Structured prediction of unobserved voxels from a single depth image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5431-5440, 2016. 3
[45] Angela Dai, Christian Diller, and Matthias Nicßner. Sg-nn: Sparse generative neural net- works for self-supervised scene completion of rgb-d scans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 849-858, 2020.
[46] Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-aware transformers. In Proceedings of the IEEF/CVF international conference computer vision, pages 12507, 2021.
13

---

[47] Xiaoguang Han, Zhen Li, Haibin Huang, Evangelos Kalogerakis. and Yizhou Yu. High- resolution shape completion using deep neural networks for global structure and local geometry inference. In Proceedings of the IEEE international conference on computer vision, pages 85-93, 2017.
[48] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1746-1754, 2017.
[49] Julian Chibane. Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape reconstruction and completion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6970-6984 177006681,..1 2020. 3,7
[50] Angela Dai and Matthias Nießner. Scan2mesh: From unstructured range scans 3d meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5574-5583, 2019.
[51] X Zheng, Yang Liu, P Wang, and Xin Tong. Sdf-stylegan: Implicit sdf-based stylegan for 3d shape generation. In Computer Graphics Forum, 2022. 3,
[52] Xuelin Chen, Baoquan Chen, and Niloy J Mitra. Unpaired point cloud completion on real scans 2019.
using adversarial training. arXiv preprint
[53] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning repre- sentations and generative models for 3d point clouds. In International conference on machine learning, pages 40-49. PMLR, 2018.
[54] Himanshu Arora, Saurabh Mishra, Shichong Peng, Ke Li, and Ali Mahdavi-Amiri. Multimodal shape completion via implicit maximum likelihood estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1958-29651 2022. 3
[55] Jascha Dock-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[56] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021. 7
[57] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10684 10695, 2022. 4
[58] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021.
[59] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171, PMLR, 2021. 22
[60] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020. 4, 22
[61] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv
preprint arXiv:2010.02.00. 2020. 6. 22
[62] Abhishek Sinha, Jiaming Song. Chenlin Meng, and Stefano Ermon. D2c: Diffusion -ecooding models for few-shot conditional generation. Advances in Neural Information Processing Systems, 14:122333-2248, 2021.
[63] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint a7X.v:2210.06978, 2022.
14

---

[64] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2837-2845, 2021.
[65] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022.
[66] Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero. 3d-Idm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint arXiv:2212.00842 2022. 3,4
[67] Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: 3d shape representation for neural fields and generative diffusion models. arXiv preprint arXiv:2301.11445. 2023.
[68] Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, et al. Gaudi: A neural architect for immersive 3d scene generation. Advances in Neural Information Processing Systems, 15:25102-22116, 2022. 3
[69] Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: Text-to-shape via voxelized diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12642-12657 2023. 3, 22
[70] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint ar 2321v:2302.05543, 2023. 17, 21
[71] Juyong Zhang, Yuxin Yao, and Bailin Deng. Fast and robust iterative closest point. IEEE Transactions on Patlern Analysis and Machine Intelligence, 2021.
[72] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 21(4):163-169 1987, 6
[73] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 2015. 6, 7,
[74] John Amanatides, Andrew Woo, et al. fast voxel traversal algorithm for ray tracing. In Eurographics, 1987. 6
[75] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet:
3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-58888 088-5539,400 2017. 6,
[76] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 6
[77] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint 2.Xi:22077.1598. 2022.
[78] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pages 1 11461-11471, 2022. 7,8,9
[79] Dongsu Zhang, Changwoon Choi, Inbum Park, and Young Min Kim. Probabilistic implicit scene completion. In International Conference on Learning Representations, 2022.
[80] Xingguang Yan, Ligiang Lin, Niloy J Mitra, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Shapeformer: Transformer-based shape completion via sparse representation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 7,9
15

---

[81] Bram Wallace and Bharath Hariharan. Few-shot generalization for single-image 3d reconstrue- tion via priors. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1888-38277) 2019. 8
[82] Peng Xiang, Xin Wen, Yu-Shen Liu, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Zhizhong Han. Snowflake point deconvolution for point cloud completion and generation with skip-tr trip-rransformer. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022. 19
[83] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. arXiv preprint arXiv:2211. 1.111111444
2022. 22
[84] Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. CVPR, 2018. 22
[85] Tao Hu, Xiaogang Xu, Ruihang Chu, and Jiaya Jia. Trivol: Point cloud rendering via triple volumes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 22
[86] Peng-Shin Sonn-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yo Yu Sun, and Xin Tong. O-enn: Octree-based convolutional neural networks for 3d shape analysis. ACM Transactions On Graphics (TOG), 36(4):1-11, 2017. 22
16


Reviews:
[{"review_id": "IIeceV4bIc", "reviewer": "Reviewer_Ct9H", "summary": "The proposed method aims to tackle the TSDF shape completion problem using diffusion models. That is, given one or several incomplete TSDFs obtained from partial scans of a single object produced by range sensors, the method generates a TSDF of complete shape while trying to preserve the geometric details of the incomplete one. The method achieves this by using the same technique in ControlNet: adding a control branch to the vanilla UNet module in diffusion models. \nAlso, in order to better incorporate the partial scans from different views, the authors propose an \"occupancy-aware fusion\" module which performs a weighted feature fusion for the multi-view scans considering their geometric reliability.", "questions": "- Please show advantages over these baselines:\n(1) Convolutional Occupancy Network. Train ConvOccNet to map partial TSDF grid to complete grid. Then compare.\n(2) ShapeFormer. TSDF can be converted to point cloud. Train ShapeFormer to map point cloud to implicit TSDF fields.\n(3) TSDF diffusion with PVD style of completion sampling. That is, in the sampling time, always replace the noisy to the actual TSDF value in the place of partial scanned cells. This way, only the \"missing\" regions are denoised and completed.", "limitations": "Authors include a comprehensive discussion on method limitations and potential societal impact in the supplementary material.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "In general, the paper is easy to read and is technically sound.\nThe following are specific points:\n\n- Single probabilistic framework for all tasks. As more and more recent papers suggested, generative models are not only doing great in generation problems, but are also competitive in deterministic problems where usually a unique optimal solution is desired given the input. This method shows this point for the 3D completion problem: Even when the input shape has little to no ambiguity, the generated completion has better quality than previous deterministic approaches.\n\n- Incorporate multi-view scans. This special consideration is useful in real-life application, where the scanners are constantly capturing new scans and the small errors in registration will make it non-trivial to fuse features from different views.", "weaknesses": "I have concerns on baseline & references, unsatisfactory contricution and writing. Hence I believe this paper does not reach the NeurIPS bar and would like to rate \"Reject\".\nHere are detailed points:\n\n- Missing baselines and references. Comparison with multiple very related works are omitted. ConvOccNet can perform shape completion very well in higher resolution (64 and higher, see the Fig. 4 in their supplementary). The model is very good at deterministic completion. For multimodal setting, two very related work, ShapeFormer and 3DILG, are not mentioned or discussed. And the former work is especially designed for multimodal completion.\n\n- Contribution is not satisfactory. The proposed method seems to be just deploying ControlNet on the PatchComplete problem and benchmark. The used resolution (32) is also not impressive comparing to previous works like DiffusionSDF or 3DShape2VecSet. I would be skeptical that training diffusion models directly and keep the partial tsdf unchanged during sampling (as what PVD does for completion) can achieve better results.\n\n- Many minor writing problems. L99: It is confusing to classify AutoSDF to be a type of autoencoder. It is an autoregressive approach. \nL114: \"Due to the absence of meaningful ground truths, ...\". In the mentioned works, they all have meaningful ground truths shapes. It is unclear what is \"meaningful\" here.\nL325: \"Effects of feature aggregation manner\". A better way is to write: \"Effects of the manner in which features are aggregated\" or \"Effects of our feature aggregation mechanism\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Ct9H,\n\nIn the following, we address all comments in the review. Our additional results validate that our method achieves a clear improvement over the newly-suggested baselines. We are happy to include these experiments in the final revision of the paper.\n\n**Q1: Add comparisons with the suggested baselines.**\n\nA1: Comprehensive comparisons with ConvONet, ShapeFormer, and PVD are presented in **Tables R1**, **R2**, and **R3** within *the response to all reviewers*, located at the top of this webpage. These results show the superior performance of our method. Regarding the suggested 3D-ILG [1], it mainly focuses on the representation for shape generation rather than the specific completion task. As 3D-ILG also supports shape completion with a conditional generation paradigm, we compare it with our method with aligned data and experimental setups. As presented in **Table R4**, our method achieves much more accurate results compared to 3D-ILG. We'll include **Table R4** into our supplement in the revision.\n\n**Table R4:** Comparisons of completion accuracy on the 3D-EPN benchmark, evaluated by average $l_1$-error ($\\downarrow$) across eight object classes.\n| **Method**|**Avg. $l_1$-err. ($\\downarrow$)**|\n|-|:-:|\n|3D-ILG [1]|0.165|\n|DiffComplete (Ours)|**0.053**|\n\n**Q2: Is the proposed method a deployment of ControlNet on the PatchComplete benchmark?**\n\nA2: Although the paradigm of injecting conditional features takes inspiration from ControlNet, DiffComplete has critical distinctions to justify our contributions. \n\nFirst, DiffComplete differs from ControlNet in several key areas. **(i) Task:** ControlNet tackles the 2D text-to-image generation task, making it work on our 3D completion task is non-trivial. To this end, we design the appropriate volumetric representation and 3D networks. **(ii) Motivation:** The motivation of ControlNet is to finetune pretrained large diffusion models, while we aim to train a specific diffusion model. This leads to different training strategies. **(iii) Training Strategy:** ControlNet utilizes a \"trainable copy\" initialization, but our experiments found that training from scratch is the most effective way. Please see Table 9 in the supplement for details. **(iv) Design Details:** We discard \"zero convolution\", a critical component in ControlNet, as we do not require the pre-training process. We also directly embed the original 3D shape representation rather than operating in latent space.\n\nSecond, DiffComplete offers new features and insights beyond ControlNet. **(i)** Our method further supports multiple inputs to improve the completion accuracy. **(ii)** We delve deeper into the feature injection mechanism, observing that altering the feature aggregation level finely controls the trade-off between completion accuracy and diversity. This finding can be leveraged to adjust the model's performance, as described in Section 4.5 of the main paper.\n\n**Q3: The used resolution ($32^3$) is not impressive.**\n\nA3: We are able to scale up the resolution through various available strategies. For instance, we can apply the gradient accumulation technique to break the large batch into smaller chunks, ensuring each fits within the GPU's memory constraints. Such a technique effectively facilitates the training of DiffComplete at higher resolutions. Other beneficial options would be improving computation efficiency, such as employing a lightweight backbone network (*e.g.*, TriVol [2]) or leveraging the autoencoding approach (*e.g.*, from [3]) for shape compression. These approaches could be incorporated into our generic pipeline to complete high-resolution shapes even on smaller GPUs.\n\n**Q4: Will training diffusion models with PVD paradigm achieve better results than DiffComplete?**\n\nA4: DiffComplete attains much better performance than PVD, as shown in **Tables R1**, **R2**, and **R3** within *the response to all reviewers* with detailed analysis. In addition, our method can flexibly support multiple conditional inputs, while this may not be feasible for the PVD pipeline.\n\n**Q5: Minor writing problems.**\n\nA5: Thanks for pointing out these wording issues and typos. We will carefully polish and revise the paper.\n\n**(i)** Though AutoSDF employs an AutoEncoder, it should be classified as an autoregressive model. We'll rectify this. \n\n**(ii)** Our goal is to illustrate that certain 3D diffusion models (Diffusion-SDF [3] and DiffRF [4]) do not employ incomplete-to-complete ground-truth pairs during training. The related sentences (lines 112-117) will be revised as follows:\n\n\"For conditional shape completion, both DiffRF and Diffusion-SDF adopt a masked diffusion strategy to fill in missing regions cropped out by 3D boxes. However, their training processes do not leverage a paired incomplete-to-complete ground truth, which may prevent them from accurately learning the completion rules. Contrarily, our method explicitly uses the scan pairs for conditional training.\"\n\n**(iii)** Thanks for the suggestions. We've already fixed it.\n****\n**References**\n\n[1] Zhang, et al. 3DILG: Irregular Latent Grids for 3D Generative Modeling. In NeurIPS 2022.\n\n[2] Hu, et al. TriVol: Point Cloud Rendering via Triple Volumes. In CVPR 2023.\n\n[3] Li, et al. Diffusion-SDF: Text-to-Shape via Voxelized Diffusion. In CVPR 2023.\n\n[4] Muller, et al. DiffRF: Rendering-Guided 3D Radiance Field Diffusion. In CVPR 2023."}, {"review_id": "Uirvrhm80C", "reviewer": "Reviewer_KuxT", "summary": "The paper introduces a diffusion-based approach, DiffComplete, to generate complete shapes conditioned on partial 3D range scans. The condition is represented as volumetric TSDF (truncated signed distance function), while the complete shape is represented as volumetric TUDF (truncated signed distance function). Inspired by 2D ControlNet, the authors devise a hierarchical feature aggregation mechanism to inject conditional features in a spatially-consistent manner. Besides, they propose a fine-tuning strategy to adapt the model trained on a single condition to multiple conditions. The trade-off between multi-modality and high fidelity can be controlled through the network level for feature aggregation between conditions and denoised complete shapes. The results on 3D-EPN and PatchComplete show the superiority of DiffComplete. The authors also demonstrate its zero-shot ability.", "questions": "1. Is there a typo in Figure 7? Is it an MMD curve?\n2. For the ablation study on multiple conditions, can the authors first fuse the scans and compute TSDF, then extract features instead of averaging features extracted from individual scans?", "limitations": "The authors have adequately addressed the limitations (e.g., failure cases) in the appendix.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is clearly written and easy to follow.\n- The ControlNet-style design to inject features from the condition is reasonable.\n- Fine-tuning the network trained with a single incomplete shape for multiple incomplete shapes is a good strategy.\n- The generalizability looks good according to Fig. 4.", "weaknesses": "It is unclear why some baselines (especially point-cloud-based methods) are missing. For example, \"3D Shape Generation and Completion Through Point-Voxel Diffusion\" and \"SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer KuxT,\n\nThanks for the encouraging feedback. We are grateful to see your recognition of our methodology, model performance, and manuscript writing. We address the specific comments below.\n\n**Q1: Compare with the suggested baselines (especially point-cloud-based methods), e.g., PVD [1] and SnowflakeNet [2].**\n\nA1: In our rebuttal (see above), we've expanded comparisons with several suggested surface completion baselines, including an adapted version of PVD. Please refer to **Tables R1**, **R2**, and **R3** within *the response to all reviewers* for details, located at the top of this webpage. These additional comparisons further show the superior capability of our method and will be included in the final revision.\n\nRegarding methods on point cloud completion, we compare our DiffComplete with the leading approach SnowflakeNet (TPAMI 2022) as per your suggestion. To make the comparisons fair, we converted SnowflakeNet's outputs to meshes using the reconstruction technique from ConvONet [3]. As shown in **Table R5**, our method delivers much better completion accuracy. We'll include this table in the supplement of the revision.\n\n**Table R5:** Comparisons of completion accuracy on the 3D-EPN benchmark, evaluated by average $l_1$-error ($\\downarrow$) across eight object classes.\n| **Method**|**Output**|**Avg. $l_1$-err. ($\\downarrow$)**|\n|-|-|:-:|\n|SnowflakeNet [2]|point cloud|0.189|\n|DiffComplete (Ours)|volumetric TUDF|**0.053**|\n\n**Q2: Is there a typo in Figure 7? Is it an MMD curve?**\n\nA2: Figure 7 is a TMD curve to reveal the varying shape diversity. Yet, a typo in line 301 identifies it as an MMD curve. We have already corrected the issue and will do a careful wording pass.\n\n**Q3: For multiple conditional inputs, can the authors first fuse the scans and compute TSDF, then extract features instead of averaging features extracted from individual scans?**\n\nA3: We compare our strategy with this option in Section B.2 of the supplement. As indicated by **Table R6** (copy of Table 10 in the supplement), fusing the original scans might be vulnerable to registration errors, which can disrupt the final results. Instead, fusing scan features in hierarchical feature spaces shows greater resilience to simple noise at the TSDF level. For a more clear illustration of our design, we will move this ablation study to the main paper.\n\n**Table R6:** Choice of fusion space for multiple partial shapes. Directly fusing them in the original TSDF space significantly impairs the completion quality.\n|**Fusion Space**|**$l_1$-err. $\\downarrow$**|**CD $\\downarrow$**|**IoU $\\uparrow$**|\n|-|:-:|:-:|:-:|\n|TSDF|0.12|4.78|61.0|\n|feature|**0.05**|**3.97**|**68.3**|\n****\n**References**\n\n[1] Zhou, et al. 3D Shape Generation and Completion Through Point-Voxel Diffusion. In ICCV 2021.\n\n[2] Xiang, et al. Snowflake Point Deconvolution for Point Cloud Completion and Generation with Skip-Transformer. TPAMI 2022.\n\n[3] Peng, et al. Convolutional Occupancy Networks. In ECCV 2020."}, {"review_id": "08KNQyba0G", "reviewer": "Reviewer_DbPU", "summary": "The paper tackles the problem of probabilistic shape completion using diffusion models, learning from range scans. The main contribution of the paper comes from proposing 2 novel techniques: hierarchical feature aggregation for strong conditioning and occupancy-aware fusion technique. The method is tested on ShapeNet dataset and achieves better completions compared to the baselines. Also, the proposed method exhibits robust generatlization for out of distribution inputs. \n", "questions": "* Regarding the related work with diffusion based models, the authors mentioned that “. Due to the absence of meaningful ground truths in these completion scenarios, they could also face completion accuracy challenges like the above generative approaches” in line 114. Does that mean that the diffusion methods require the ground truth while the proposed method does not? To the best of my knowledge, the diffcomplete model requires ground truth TUDF for training. I think that the assumption that we can acquire the ground truth TUDF means that we can acquire ground truth point cloud and implicit function (mostly UDF based) as well. If so, then other baseline methods mentioned in the weakness section should be compared. ", "limitations": "Authors have addressed the limitations in the supplementary material.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* I enjoyed the idea of hierarchical feature aggregation. To best of my knowledge, conditioning a diffusion model is hard, and the ablation study in Tab. 7 clearly shows that the multi-level conditioning acts as intended.\n\n* Strong empirical results on completion on unseen categories. The results in Tab2, 3 shows that the method achieves strong completion results on unseen categories and it makes sense due to strong hierarchical conditioning.\n", "weaknesses": "* Similar works on 3D completion using diffusion models exists. As mentioned in the related sections, the difference compes from the representation used to represent the state in the diffusion model. For example, point-voxel uses point cloud, while this work uses TSDF (TUDF) representation. While I do not think that the novelty of the work degrades even there are other diffusion based 3D completion models, I want to hear in detail how the method differs from other diffusion based models. Please look at the question section. \n\n* My main concern is that some of the important recent works are not mentioned and compared. For determnisitc completion methods, convocc completes the given partial point cloud using occupancy fields. For multimodal completion methods, point-voxel [2] (although mentioned in the related-section) and GCA[3] completes point cloud in point cloud/sparse voxel representation. ShapeFormer[4] and cGCA[5] completes the point cloud in implicit neural representations.\nAlthough I realize that comparing all of these baselines requires alot of effort, these baselines should have been tested since the core contribution of the work stems from achieving SOTA results on an existing benchmark. \n\n[1] Peng et al. Convolutional Occupancy Networks. ECCV, 2020\n\n[2] Zhou et al. 3D Shape Generation and Completion Through Point-Voxel Diffusion. ICCV, 2021\n\n[3] Zhang et al. Learning to generate 3d shapes with generative cellular automata. ICLR, 2021\n\n[4] Yan et al. ShapeFormer: Transformer-based Shape Completion via Sparse Representation. CVPR, 2022\n\n[5]  Zhang et al. Probabilistic implicit scene completion. ICLR, 2022\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer DbPU,\n\nThanks for the valuable feedback. We are happy to address each specific comment below.\n\n**Q1: Explain more details about the differences from other diffusion-based models.**\n\nA1: DiffComplete is specifically designed for high-quality shape completion. As a result, it differs from recent 3D diffusion models in various design choices, as summarized in **Table R7**: (i) diffusion Space, (ii) training strategy, and (iii) condition injection mechanism. Below, we present the details about each of these aspects.\n\n**(i) Diffusion Space**: Different from many previous approaches, We perform the diffusion process on the original volumetric TUDF. Doing so helps to ensure a direct transition to final meshes via Marching Cubes. This choice is favored as it preserves the fidelity of shape information better than the latent codes and circumvents the need for post-processing required by point clouds. While radiance fields are suited for rendering tasks, they fall short when recovering the shape geometries.\n\n**(ii) Training Strategy**: DiffRF [1] and Diffusion-SDF [2] both perform a masked diffusion process to incorporate conditions only at the inference stage. They do not include any conditional inputs at the training stage, thus lacking paired incomplete-to-complete GT supervision to accurately learn the completion rules. In contrast, Diffusion-SDF [3], PVD [4], and our method utilize these scan pairs in training to enhance the completion accuracy. However, we design different mechanisms to inject the conditions, as we discuss next.\n\n**(iii) Condition Injection**: Utilizing a control branch for hierarchical condition injection is a key design of DiffComplete, which enables both accurate completion and effective controls. This strategy yields superior performance over both the cross-attention (see Table 8 of the main paper) and the PVD's injection mechanism (see **Tables R1**, **R2**, and **R3** in *the response to all reviewers*, located at the top of this webpage).\n\nIn the revision, we'll add these discussions into the supplement to clarify DiffComplete's unique attributes.\n\n**Table R7:** Comparisons with different 3D diffusion models. Note that [2] and [3] are different works.\n|**Method**|**Diffusion Space**|**Conditional Training**|**Condition Injection**|\n|-|-|-|-|\n|DiffRF [1]|radiance field|×|masked diffusion|\n|Diffusion-SDF [2]|latent patch|×|masked diffusion|\n|Diffusion-SDF [3]|latent vector|✓|cross-attention|\n|PVD [4]|point cloud|✓|single branch|\n|DiffComplete|volumetric TUDF|✓|two-branch hierarchical aggregation|\n\n\n**Q2: Confusions in Line 114 - \"Due to the absence of meaningful ground truths in these completion scenarios, they could also face completion accuracy challenges like the above generative approaches\".**\n\nA2: Similar to Diffusion-SDF [3] and PVD [4], our method requires incomplete-to-complete ground-truth shape pairs during training, as indicated in **Table R7**. For line 114, we intended to illustrate that certain 3D diffusion models (DiffRF [1] and Diffusion-SDF [2]) do not use these pairs for conditional training, which limits their completion accuracy. To avoid any confusion, the related sentences (lines 112-117) will be revised as follows:\n\n\"For conditional shape completion, both DiffRF and Diffusion-SDF adopt a masked diffusion strategy to fill in large missing regions cropped out by 3D boxes. However, their training processes do not leverage a paired incomplete-to-complete ground truth, which may prevent them from accurately learning the completion rules. Contrarily, our method explicitly uses the scan pairs for conditional training.\"\n\n**Q3: Comparisons with the suggested baselines.**\n\nA3: As suggested, we have extensively compared DiffComplete with ConvONet, ShapeFormer, cGCA, and diffusion-based PVD. The results are illustrated in **Tables R1**, **R2**, and **R3** within *the response to all reviewers* with detailed discussions, located at the top of this webpage. Notably, DiffComplete exhibits obvious advantages over these baselines across various benchmarks and evaluation metrics. As GCA and cGCA share a similar generation paradigm, we only compared DiffComplete with cGCA, which has superior performance as demonstrated in its paper.\n\n****\n**References**\n\n[1] Muller, et al. DiffRF: Rendering-Guided 3D Radiance Field Diffusion. In CVPR 2023.\n\n[2] Li, et al. Diffusion-SDF: Text-to-Shape via Voxelized Diffusion. In CVPR 2023.\n\n[3] Chou, et al. Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions. In ICCV 2023.\n\n[4] Zhou, et al. 3D Shape Generation and Completion Through Point-Voxel Diffusion. In ICCV 2021."}, {"review_id": "MLRGZAj6sr", "reviewer": "Reviewer_hw6P", "summary": "The authors present a diffusion-based neural network for 3D shape completion. They introduce a hierarchical and spatially consistent feature aggregation scheme to fuse partial and complete shapes, and an occupancy-aware fusion strategy to fuse multiple partial shapes. They demonstrate a new SOTA performance in shape completion for both new instances and completely unknown object categories.", "questions": "1. I wonder if the resolution can go up to 64^3 or even 128^3?\n\n2. The diversity of multimodal complement is worse than other methods. So is it because the diffusion model gives more deterministic results than others?", "limitations": "The limitation is not addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors successfully apply the diffusion-based model to the shape completion task and achieve a new SOTA performance.\n\n2. The stepwise addition of partial scans to predict a complete shape is interesting and makes completion tasks interactive and adaptable.\n\n3. The proposed model supports multimodal completions that can predict multiple plausible results.", "weaknesses": "1. The writing of this paper can be improved.\n\n2. The technical designs, such as the two feature aggregation schemes, are not well motivated. As a main contribution, I expect more explanations of the design ideas.\n\n3. The experimental design is not very clear. Does DiffComplete train on a single category? Since PatchComplete trains on all categories, DiffComplete should also train across categories to allow a fair comparison.\n\n4. The results of multiple conditional inputs are unpredictable. A better example is adding and editing a semantic part of the object from a partial input.\n\n5. Does the proposed model require much more time than its competitors?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer hw6P,\n\nThanks for the constructive comments. Please find our response to the specific points below.\n\n**Q1: Explain more design ideas about the two feature aggregation schemes.**\n\nA1: Our key design motivation is to enhance the completion accuracy and generalization ability. Then, we illustrate how each design choice benefits the goals.\n\n**(i) Two-branch architecture**: We utilize two branches to separately process the complete and incomplete scans, such that each branch has a specific focus. Given the same architecture of the two branches, the feature sizes of complete and incomplete shapes are always aligned. This alignment greatly simplifies the following feature aggregation process.\n\n**(ii) Hierarchical feature aggregation**: Leveraging the multilayer structure of CNNs, we aggregate two-branch features at various network levels. It brings two-fold benefits. First, the network can correlate the difference between two shapes at multiple abstraction levels. The learned completion regularities are multi-scale, thus being more accurate and robust. As the substructures of objects are more general than full shapes, *e.g.*, chairs and tables may share very similar legs, the learned local completion pattern could generalize to various object classes. Second, we found that altering the level of feature aggregation can control the trade-off between completion accuracy and diversity. Hence, we can leverage this feature to adjust the model's performance, as illustrated in Table 7 of the main paper.\n\n**(iii) Spatially-consistent feature aggregation**: We simply add up the two-branch features at the respective 3D locations. This operation precisely compares the corresponding parts of two scan pairs, contributing to yield more accurate completion than the widely-adopted cross-attention mechanism, as shown in Table 8 of the main paper.\n\nWe'll further clarify our design motivation in the revision.\n\n**Q2: Does DiffComplete train on a single category? Since PatchComplete trains on all categories, DiffComplete should also train across categories to allow a fair comparison.**\n\nA2: For fair comparisons, on the PatchComplete benchmark, we train DiffComplete across all object categories like PatchComplete. This has been described in lines 252-254 of the main paper. On the 3D-EPN benchmark, both PatchComplete and our method train a specific model for each object category. We will make the experimental settings more clear in the revision.\n\n**Q3: To show the results of multiple conditional inputs, a better example is adding and editing a semantic part of the object from a partial input.**\n\nA3: We have provided several visual examples in the attached **PDF** under *the response to all reviewers*. These examples will be integrated into our revised manuscript to better showcase the model's multiple-input capability.\n\n**Q4: Does the proposed model require much more time than its competitors?**\n\nA4: As our method belongs to the class of diffusion model, like other diffusion-based approaches, it's inherently slower than methods of other classes (*e.g.*, GAN) due to the iterative sampling process. In scenarios where the completion quality weighs more than inference time, DiffComplete is definitely a better choice. Additionally, we'd like to mention that the inference time can be reduced by employing fast sampling techniques (*e.g.*, [1]) for diffusion models.\n\nCompared with two other diffusion models [2] and [3], our method maintains similar inference times but significantly improves the completion quality, as shown in **Table R8**. \n\n**Table R8:** Comparisons of completion accuracy (Avg. $l_1$-err.) and average inference time on the 3D-EPN benchmark. It is tested on a single RTX 3090 GPU with a batch size of 1.\n| **Method**|**Paradigm**|**Avg. $l_1$-err. $\\downarrow$**|**Inference Time $\\downarrow$**|\n|-|-|:-:|:-:|\n|RePaint-3D [2]|diffusion model|0.374|34.1 s|\n|PVD [3]|diffusion model|0.114|**3.1 s**|\n|DiffComplete (Ours)|diffusion model|**0.053**|3.2 s|\n\n\n**Q5: Can the resolution go up to $64^3$ or even $128^3$?**\n\nA5: Yes. **(i)** It's feasible to train our DiffComplete with a resolution of $64^3$ on affordable NVIDIA 3090 GPUs. We provide visual examples of our outputs in the attached **PDF** under *the response to all reviewers*. We are happy to include these results in the supplement during revision.\n\n**(ii)** There are various available strategies to scale up resolutions further (*e.g.*, to $128^3$) even on smaller GPUs. For instance, we can apply the gradient accumulation technique to break the large batch into smaller chunks, ensuring each fits within the GPU's memory constraints. Other beneficial options would be improving computation efficiency, such as employing a lightweight backbone network (*e.g.*, TriVol [4]) or leveraging the autoencoding approach (*e.g.*, from [5]) for shape compression. These techniques could be incorporated into our generic pipeline to facilitate high-resolution shape completion.\n\n**Q6: For worse diversity, is it because diffusion models give more deterministic results than other methods?**\n\nA6: It's not about the diffusion model, as RePaint-3D, with the best diversity, is also diffusion-based. This is because our design choice prioritizes completion accuracy over diversity. Yet, the accuracy-diversity trade-off can be easily adjusted in DiffComplete. By altering the feature aggregation level, our method can achieve the best diversity, as shown in Table 7 of the main paper.\n****\n**Reference**\n\n[1] Zheng, et al. Fast Sampling of Diffusion Models via Operator Learning. In NeurIPS 2022 Workshop.\n\n[2] Lugmayr, et al. RePaint: Inpainting using Denoising Diffusion Probabilistic Models. In CVPR 2022.\n\n[3] Zhou, et al. 3D Shape Generation and Completion Through Point-Voxel Diffusion. In ICCV 2021.\n\n[4] Hu, et al. TriVol: Point Cloud Rendering via Triple Volumes. In CVPR 2023.\n\n[5] Li, et al. Diffusion-SDF: Text-to-Shape via Voxelized Diffusion. In CVPR 2023."}, {"review_id": "IMYGYugpc4", "reviewer": "Reviewer_9z1Q", "summary": "The paper tackles the shape completion task by introducing a diffusion-based technique. To do so, additional design choices have been made such as hierarchical feature aggregation and an occupancy-aware fusion strategy to better reproduce the shape and respect the details. Based on numerous quantitative and qualitative experiments, the proposed method outperforms others by a considerable margin.", "questions": "When you pro-process the input and the incomplete shape through a convolutional network and add them, why do you call it aligning distributions? \nAs the computation and time are two limitations of this method, would a part-based completion task be a better approach? In fact, my question is whether a part based approach in which a model is first broken into parts will save computation time and understand the geometry better?", "limitations": "The method is mostly trained category specific. Considering the time and computational cost of this approach, this makes it hard and expensive to be adopted and modified. easily. However, I weigh the quality of the results more than the time and computation so I am okay with this limitation. ", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper clearly outperforms other SOTA.\nThe visual quality of the completion is good. \nOccupancy-aware fusion and hierarchical feature aggregation that are the main task specific designs make sense and the authros have successfully shown their importance in the design through ablation studies.\nI like the adoption of control net in the design. I believe it makes sense to learn gradually and at different scales. \nThe fact that the data is from the actual scan and it is not synthesized by adding noise and removing parts make the method very useful and interesting. \n", "weaknesses": "In terms of writing, the paper could use simpler and more understandable and short sentences. For example, this sentence could be improved by making it shorter or breaking it into two sentences:\nTo improve the completion accuracy, we enable effective control from single-input condition by hierarchical and spatially-consistent feature aggregation, and from multiple ones by an occupancy-aware fusion strategy.\nThis happens many times throughout the paper. \nMissing reference:\nMultimodal shape completion with IMLE:\nhttps://openaccess.thecvf.com/content/CVPR2022W/DLGC/html/Arora_Multimodal_Shape_Completion_via_Implicit_Maximum_Likelihood_Estimation_CVPRW_2022_paper.html", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 9z1Q,\n\nThank you for the valuable comments and encouraging feedback. We are happy to respond to each question below.\n\n**Q1: The paper could use simpler, more understandable, and short sentences.**\n\nA1: In the revision, we will carefully polish the manuscript to ensure better clarity and conciseness. For instance, the suggested example will be rephrased as \"We enhance the completion accuracy with two key designs. For a single condition, we propose a hierarchical feature aggregation strategy to effectively control the outputs. For multiple conditions, we introduce an occupancy-aware fusion strategy to incorporate more shape details.\"\n\n**Q2: Add reference: Multimodal Shape Completion with Implicit Maximum Likelihood Estimation.**\n\nA2: Both IMLE and DiffComplete address multimodal shape completion, and we'll cite this paper in the revision. The relevant discussion will be inserted in line 102 of the paper and is shown as follows, with the new contents in bold.\n\n\"While some generative models can generate diverse global shapes given a partial input, they potentially allow for a high generation freedom and overlook the completion accuracy. **IMLE, for instance, adopts an Implicit Maximum Likelihood Estimation technique to specially enhance structural variance among the generated shapes.** Distinctively, we formulate a diffusion model for shape completion. Our method mainly prioritizes fidelity relative to ground truths while preserving output diversity.\"\n\n**Q3: Why do you call the pre-processing process of incomplete and complete shapes \"aligning distributions\"?**\n\nA3: The complete and incomplete shapes are represented by different data fields, *i.e.*, TUDF and TSDF, respectively. Directly fusing them could result in confusing representations. Hence, we use a pre-processing layer to empirically project the two different fields into a more compatible feature space for interaction. We previously referred to this as \"aligning distributions\". To avoid any misunderstanding, we will remove the term \"aligning distributions\" and incorporate the above explanations in the revision.\n\n**Q4: Will a part-based approach save computation time and help understand the geometry better?**\n\nA4: Decomposing 3D shapes into parts can help the model capture high-resolution details with low memory costs. We agree that this strategy is beneficial to our shape completion task. For instance, we can utilize the techniques of work [1] and [2] to first encode global volumetric TSDF (TUDF) into smaller structural patches. Then, our DiffComplete is capable of performing accurate part-level completion. Such an adaptation is feasible due to the versatility of DiffComplete's core designs. Future work will explore more efficient 3D representations (*e.g.,* patch-based) to enhance the capability of our method. We'll add these discussions to the future work section in our revised paper.\n****\n**References**\n\n[1] Li, et al. Diffusion-SDF: Text-to-Shape via Voxelized Diffusion. In CVPR 2023.\n\n[2] Hertz, et al. SPAGHETTI: Editing Implicit Shapes Through Part Aware Generation. TOG 2022."}]

Author rebuttal:
Dear all reviewers,

We sincerely thank your constructive comments and are grateful to see the recognition received:

- **Methodology:** Reviewer 9z1Q: "I like the adoption of ControlNet in the design"; Reviewer KuxT: "The ControlNet-style design is reasonable", "Fine-tuning the network for multiple incomplete shapes is a good strategy"; Reviewer DbPU: "I enjoyed the idea of hierarchical feature aggregation"; Reviewer Ct9H: "The paper is technically sound'.

- **Experimental Results:** Reviewer 9z1Q: "clearly outperforms other SOTAs", "shown their importance in the design through ablation studies"; Reviewer hw6P: "a new SOTA performance"; Reviewer KuxT: "The generalizability looks good"; Reviewer DBPU: "Strong empirical results on completion on unseen categories".

- **Practicality**: Reviewer 9z1Q: "Data is from the actual scan, making the method very useful and interesting"; Reviewer DbPU: "The stepwise addition of partial scans is interesting and makes completion tasks interactive and adaptable".

We first address the general concern of Reviewer Ct9H/KuxT/DbPU below. Then, we provide a detailed  response to each reviewer's comments and address all issues.

**Q: Performance comparisons with the suggested baselines. (To Reviewer Ct9H/KuxT/DbPU)**

A: The suggested baselines include both the deterministic approach (ConvONet [1]) and multimodal approaches (ShapeFormer [2], PVD [3], and cGCA [4]). We conducted experiments on both the 3D-EPN and PatchComplete benchmarks, showing the results in **Table R1** and **Table R2**, respectively, inside our rebuttal. Our DiffComplete surpasses all of these baselines by a significant margin with respect to completion accuracy and generalization ability. We provide a detailed discussion below.

 **(i)** ConvONet [1] extends the 3D-EPN structure (which is a baseline in our paper) by incorporating an implicit occupancy decoder. Both methods are deterministic by nature and hence cannot handle ambiguities of missing data. Thus, their completion accuracy and generalization ability are lower than our method.

 **(ii)** Our method differs from ShapeFormer [2] and cGCA [4] by the proposed generative model. The employed diffusion model offers superior sampling quality than the autoregressive model (ShapeFormer) and the Generative Cellular Automata model (cGCA). Compared with [2] and [4], which perform shape compression, we also preserve the fidelity of the original structures to enhance the completion details.
 
 **(iii)** PVD [3] is originally a point cloud diffusion model. Here, we adapt it to perform TSDF (TUDF) diffusion as suggested by Reviewer Ct9H. A limitation of PVD is that it retains noises of the partial input. Hence, when it is mixed with the generated missing part, noise severely affects the final completion quality. Instead, we design two branches to separately process the partial and complete shapes. By doing so, the model can effectively learn a diffusion process from noise to clean shapes.

For multimodal approaches, we further compare their multimodal capacity. **Table R3** shows that our method achieves significant improvements on both completion accuracy (MMD) and fidelity (UHD), while preserving a moderate (or even comparable) completion diversity (TMD). This result aligns with our design choice that prioritizes completion accuracy over diversity. We'd like to highlight that the trade off between accuracy and diversity can be easily adjusted in DiffComplete, as detailed in Section 4.5 of the main paper.

All experiments are conducted on the same data for a fair comparison. We'll include the additional comparison results from **Tables R1**, **R2**, and **R3** into Tables 1, 2, and 4 of the main paper, respectively, since they directly correlate. We will also release our code to facilitate reproducibility.

**Table R1:** Comparisons of completion accuracy on the 3D-EPN benchmark, evaluated by average $l_1$-error ($\downarrow$) across eight object classes.
| **Method**|**Paradigm**|**Avg. $l_1$-err. ($\downarrow$)**|
|-|-|:-:|
|ConvONet [1]|deterministic|0.220|
|ShapeFormer [2]|multi-modal|0.141|
|PVD [3]|multi-modal|0.114|
|cGCA [4]|multi-modal|0.185|
|DiffComplete (Ours)|multi-modal|**0.053**|

**Table R2:** Comparisons of generalization ability on the PatchComplete benchmark, evaluated by average CD ($\downarrow$) and IoU ($\uparrow$) across eight *unseen* object classes.
| **Method**|**Paradigm**|**Avg. CD ($\downarrow$)**|**Avg. IoU ($\uparrow$)**|
|-|-|:-:|:-:|
|ConvONet [1]|deterministic|5.26|60.1|
|ShapeFormer [2]|multi-modal|5.05|62.5|
|PVD [3]|multi-modal|4.94|62.8|
|cGCA [4]|multi-modal|5.09|61.7|
|DiffComplete (Ours)|multi-modal|**4.10**|**67.5**|

**Table R3:** Comparisons of multimodal capacity on Chair class of the 3D-EPN benchmark. 
| **Method**|**MMD ($\downarrow$)**|**TMD ($\uparrow$)**|**UHD ($\downarrow$)**|
|-|:-:|:-:|:-:|
|ShapeFormer [2]|0.007|0.024|0.055|
|PVD [3]|0.007|**0.027**|0.042|
|cGCA [4]|0.006|0.024|0.047|
|DiffComplete (Ours)|**0.002**|0.025|**0.032**|

**References**

[1] Peng, et al. Convolutional Occupancy Networks. In ECCV 2020.

[2] Yan, et al. ShapeFormer: Transformer-based Shape Completion via Sparse Representation. In CVPR 2022.

[3] Zhou, et al. 3D Shape Generation and Completion Through Point-Voxel Diffusion. In ICCV 2021.

[4] Zhang, et al. Probabilistic implicit scene completion. In ICLR 2022.

Meta-review: The paper proposed a framework for shape completion with a conditional 3D diffusion model utilizing a control-net style conditioning, with the improvement of using hierarchical feature aggregation, and occupancy aware fusion. Multi-modal results and generalization to unseen categories demonstrate the effectiveness of the proposed method. The rebuttal and following discussions successfully addressed the concerns of all the reviewers who all became positive after the rebuttal, thus, I follow the reviewers' suggestions on accepting this paper. 

As promised during the rebuttal and discussion, please add the additional comparisons with baselines provided during the rebuttal to the main paper, motivation and clarification on the feature aggregation schemes and exact experimental settings, and the discussion of extending to part-based generations.
Decision: Accept (poster)