Title: DiffComplete: Diffusion-based Generative 3D Shape Completion
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3d shape completion

conditional generation

diffusion models
Abstract: We introduce a new diffusion-based approach for shape completion on 3D range scans. Compared with prior deterministic and probabilistic methods, we strike a balance between realism, multi-modality, and high fidelity. We propose DiffComplete by casting shape completion as a generative task conditioned on the incomplete shape. Our key designs are two-fold. First, we devise a hierarchical feature aggregation mechanism to inject conditional features in a spatially-consistent manner. So, we can capture both local details and broader contexts of the conditional inputs to control the shape completion. Second, we propose an occupancy-aware fusion strategy in our model to enable the completion of multiple partial shapes and introduce higher flexibility on the input conditions. DiffComplete sets a new SOTA performance (e.g., 40% decrease on $l_1$ error) on two large-scale 3D shape completion benchmarks. Our completed shapes not only have a realistic outlook compared with the deterministic methods but also exhibit high similarity to the ground truths compared with the probabilistic alternatives. Further, DiffComplete has strong generalizability on objects of entirely unseen classes for both synthetic and real data, eliminating the need for model re-training in various applications.

DiffComplete: Diffusion-based Generative 3D Shape Completion

Abstract

We introduce a new diffusion-based approach for shape completion on 3D range scans. Compared with prior deterministic and probabilistic methods, we strike a balance between realism, multi-modality, and high fidelity. We propose DiffComplete by casting shape completion as a generative task conditioned on the incomplete shape. Our key designs are two-fold. First, we devise a hierarchical feature aggregation mechanism to inject conditional features in a spatially-consistent manner. So, we can capture both local details and broader contexts of the conditional inputs to control the shape completion. Second, we propose an occupancy-aware fusion strategy in our model to enable the completion of multiple partial shapes and introduce higher flexibility on the input conditions. DiffComplete sets a new SOTA performance ( e.g. , 40% decrease on l 1 error) on two large-scale 3D shape completion benchmarks. Our completed shapes not only have a realistic outlook compared with the deterministic methods but also exhibit high similarity to the ground truths compared with the probabilistic alternatives. Further, DiffComplete has strong generalizability on objects of entirely unseen classes for both synthetic and real data, eliminating the need for model re-training in various applications.

Input

Output

(a)

(b)

(c)

1 Introduction

The advent of affordable range sensors, such as the Microsoft Kinect and Intel RealSense, has spurred remarkable progress in 3D reconstruction [1], facilitating applications in content creation, mixed reality, robot navigation, and more. Despite the improved reconstruction quality [2-5], typical scanning sessions cannot cover every aspect of the scene objects. Hence, the reconstructed 3D models often have incomplete geometry, thereby limiting their practical usage in downstream applications.

To fully unlock the potential of 3D reconstruction for supporting assorted applications, it is essential to address the challenges of completing incomplete shapes.

Effectively, a shape completion should produce shapes that are realistic , probabilistic , and highfidelity . First, the produced shapes should look plausible without visual artifacts. Second, considering the under-determined nature of completion, it is desirable for the model to generate diverse, multimodal outputs when filling in the missing regions, to improve the likelihood of obtaining a superior shape and enable creative use. Third, while multiple outputs encourage the model's generative ability, maintaining effective control over the completion results is crucial to ensure coherent reconstructions that closely resemble the ground-truth shapes.

Current approaches to 3D shape completion fall into deterministic and probabilistic paradigms. The former class excels at aligning predictions with ground truths, owing to their determined mapping functions and full supervision. However, this can expose the models to a higher risk of over-fitting, leading to undesirable artifacts, such as rugged surfaces, especially on unseen objects. On the other hand, probabilistic approaches formulate the shape completion as a conditional generation task to produce plausible results, paired with techniques like autoregressive model [6], adversarial training [7-9], or diffusion models [10-13]. These approaches mainly focus on cases that prioritize completion diversity, for instance, filling in large missing regions cropped out by 3D boxes, where the algorithm has more freedom to explore various plausible shapes. However, when completing shapes obtained from range scans/reconstructions, whose geometries are often contaminated with noise, having varying degrees of incompleteness (see row 1 in Fig. 1), a promising goal is to recover the ground-truth scanned objects as accurately as possible. Therefore, relying solely on previous probabilistic approaches may compromise the completion accuracy and leads to low-fidelity results.

In this work, we focus on completing shapes acquired from range scans, aiming to produce realistic and high-fidelity completed shapes, while considering also the probabilistic uncertainty. We approach this task as a conditional generation with the proposed diffusion model named DiffComplete. To address the accuracy challenge typically in probabilistic models, we introduce a key design that incorporates range scan features in a hierarchical and spatially-consistent manner to facilitate effective controls. Additionally, we propose a novel occupancy-aware fusion strategy that allows taking multiple partial shapes as input for more precise and controllable completions.

Specifically, we formulate the diffusion process on a truncated distance field in a volumetric space. This representation allows us to encode both complete and incomplete shapes into multi-resolution structured feature volumes, thereby enabling their interactions at various network levels. At each level, features are aggregated based on voxel-to-voxel correspondence, enabling precise correlation of the difference in partialness between the shapes. By leveraging multi-level aggregation, the completion process can be controlled by both local details and broader contexts of the conditional inputs. It effectively improves the completion accuracy and network generalizability, as illusrated in Sec. 4.5.

To condition the completion on multiple partial shapes, our occupancy-aware fusion strategy adopts the occupancy mask as a weight to adaptively fuse all observed geometry features from various incomplete shapes, and then employ the fused features to control the completion process. For robustness, such an operation is performed in a multi-scale feature space. With our novel design, switching between single and multiple conditions can be efficiently achieved by finetuning an MLP layer. As Fig. 1(b) and 6 show, using multiple conditions for completion progressively refines the final shapes towards the ground truths.

DiffComplete sets a new state-of-the-art performance on two large-scale shape completion benchmarks [14, 15] in terms of completion accuracy and visual quality. Compared to prior deterministic and probabilistic approaches, we not only generate multimodal plausible shapes with minimal artifacts but also present high shape fidelity relative to the ground truths. Further, DiffComplete can directly generalize to work on unseen object categories. Without specific zero-shot designs, it surpasses all existing approaches for both synthetic and real-world data, allowing us to eliminate the need for model re-training in various applications. To sum up, our contributions are listed as follows:

We introduce a diffusion model to complete 3D shapes on range scans, enabling realistic, probabilistic, and high-fidelity 3D geometry.

We enhance the completion accuracy with two designs. For a single condition, we propose a hierarchical feature aggregation strategy to control the outputs. For multiple conditions, we introduce an occupancy-aware fusion strategy to incorporate more shape details.

We show SOTA performance on shape completion on both novel instances and entirely unseen object categories, along with an in-depth analysis on a range of model characteristics.

2 Related Work

RGB-Dreconstruction. Traditional methods rely on geometric approaches for 3D reconstruction [1620]. A pioneering method [21] proposes a volumetric fusion strategy to integrate multiple range images into a unified 3D model on truncated signed distance fields (TSDF), forming the basis for many modern techniques like KinectFusion [22, 23], VoxelHashing [24], and BundleFusion [2]. Recent learning-based approaches further improve the reconstruction quality with fewer artifacts [2528, 3], yet the intrinsic occlusions and measurement noise of 3D scans constrain the completeness of 3D reconstructions, making them still less refined than manually-created assets.

3D shape completion is a common post-processing step to fill in missing parts in the reconstructed shapes. Classic methods mainly handle small holes and geometry primitives via Laplacian hole filling [29-31] or Poisson surface reconstruction [32, 33]. Another line exploits the structural regularities of 3D shapes, such as the symmetries, to predict the unobserved data [34-38].

The availability of large 3D data has sparked retrieval-based methods [39-42] and learning-based fitting methods [43, 44, 14, 45-49]. The former retrieves the shapes from a database that best match the incomplete inputs, whereas the latter minimizes the difference between the network-predicted shapes and ground truths. 3D-EPN [14], for instance, proposes a 3D encoder-decoder architecture to predict the complete shape from partial volumetric data. Scan2Mesh [50] converts range scans into 3D meshes with a direct optimization on the mesh surface. PatchComplete [15] further leverages local structural priors for completing shapes of unseen categories.

Generative methods, e.g., GANs [51, 9, 52, 7, 8] and AutoEncoders [53], offer an alternative to shape completion. While some generative models can generate diverse global shapes given a partial input, they potentially allow for a high generation freedom and overlook the completion accuracy. IMLE [54], for instance, adopts an Implicit Maximum Likelihood Estimation technique to specially enhance structural variance among the generated shapes. Distinctively, we formulate a diffusion model for shape completion. Our method mainly prioritizes fidelity relative to ground truths while preserving output diversity. Compared to both generative and fitting-based paradigms, our method also effectively reduces surface artifacts, producing more realistic and natural 3D shapes. In addition, we show superior generalization ability on completing objects of novel classes over SOTAs.

Diffusion models for 3D generation. Diffusion models [55-62] have shown superior performance in various generation tasks, outperforming GANs' sample quality while preserving the likelihood evaluation property of VAEs. When adopted in 3D domain, a range of works [12, 63-65] focus on point cloud generation. For more complex surface generation, some works [66, 13, 10, 67, 68, 11] adopt latent diffusion models to learn implicit neural representations and form final shapes via a decoder. For conditional shape completion, both DiffRF [10] and Diffusion-SDF [69] adopt a masked diffusion strategy to fill in missing regions cropped out by 3D boxes. However, their training processes do not leverage a paired incomplete-to-complete ground truth, which may prevent them from accurately learning the completion rules. Contrarily, our method explicitly uses the scan pairs for conditional training, yielding outputs closely resembling the actual objects scanned. Also, we apply the diffusion process in explicit volumetric TSDFs, preserving more geometrical structures during the completing process.

3 Method

3.1 Formulation

To prepare the training data, we generate an incomplete 3D scan from depth frames using volumetric fusion [21] and represent the scan as a truncated signed distance field (TSDF) in a volumetric grid. Yet, to accurately represent a ground-truth shape, we choose to use a volumetric truncated unsigned distance field (TUDF) instead. This is because retrieving the sign bit from arbitrary 3D CAD models ( e.g. , some are not closed) is non-trivial. By using TUDF, we can robustly capture the geometric features of different objects without being limited by the topology.

Given such a volume pair, i.e. , the incomplete scan c and complete 3D shape x 0 , we formulate the shape completion as a generation task that produces x 0 conditioned on c . We employ the probabilistic diffusion model as our generative model. In particular, it contains (i) a forward process (denoted as q ( x 0: T ) ) that gradually adds Gaussian noise to corrupt the ground-truth shape x 0 into a random noise volume x T , where T is the total number of time steps; and (ii) a backward process that employs a shape completion network, with learned parameters θ , to iteratively remove the noise from x T conditioned on the partial scan c and produce the final shape, denoted as p θ ( x 0: T , c ) . As both the forward and backward processes are governed by a discrete-time Markov chain with time steps { 0 , ..., T } , their Gaussian transition probabilities can be formulated as

In the forward process (Eq. (1)), the scalars β t ∈ [0 , 1] control a variance schedule that defines the amount of noise added in each step t . In the backward process (Eq. (2)), p ( x T ) is a Gaussian prior in time step t , µ θ represents the mean predicted from our network and σ 2 t is the variance. As suggested in DDPM [60], predicting µ θ ( x t , t, c ) can be simplified to alternatively predicting ϵ θ ( x t , t, c ) , which should approximate the noise used to corrupt x t -1 in the forward process, and σ t can be replaced by the pre-defined β t . With these modifications, we can optimize the network parameter θ with a mean squared error loss to maximize the generation probability p θ ( x 0 ) . The training objective is

where ϵ is the applied noise to corrupt x 0 into x t and N (0 , I ) denotes a unit Gaussian distribution. We define all the diffusion processes in a volume space of a resolution S 3 , i.e. , x 0: T , c, ϵ ∈ R S × S × S , where each voxel stores a scalar TSDF/TUDF value; S =32 in our experiments. Compared to previous latent diffusion models [11, 13, 66, 10] that require shape embedding first, we directly manipulate the shape with a better preservation of geometric features. Doing so naturally enables our hierarchical feature aggregation strategy (see Sec. 3.2). Next, we will introduce how to predict ϵ θ ( x t , t, c ) .

3.2 Shape Completion Pipeline

Overview. To enhance the completion accuracy, we encourage the incomplete scans to control completion behaviors. Inspired from the recent ControlNet [70], which shows great control ability given 2D conditions, we adopt a similar principle to train an additional control branch. To predict the noise ϵ θ ( x t , t, c ) in Eq. (3), we encode the corrupted ground-truth shape x t by a main branch and the partial shape c by a control branch, where two branches have the same network structure without parameter sharing. Owing to the compact shape representation in 3D volume space, both complete and incomplete shapes are encoded into multi-resolution feature volumes with preserved spatial structures. Then, we hierarchically aggregate their features at each network level, as the sizes of two feature volumes are always aligned. In this work, we simply add up the two feature volumes to allow for a spatially-consistent feature aggregation, i.e. , only features at the same 3D location are combined. Compared with frequently-used cross-attention technique [11, 13, 57], doing so can significantly reduce computation costs. By multi-scale feature interaction, the network can effectively correlate the difference in partialness between two shapes, both locally and globally, to learn the completion regularities. The final integrated features are then used to predict the noise volume ϵ θ .

Network architecture. Fig. 2 provides an overview of our network architecture, which has a main branch and a control branch to respectively handle complete and incomplete shapes. The main branch (upper) is a 3D U-Net modified from a 2D version [59]. It takes as input an corrupted complete shape x t , including (i) a pre-processing block ε x ( · ) of two convolution layers to project x t into a high-dimension space, (ii) N subsequent encoder blocks, denoted as { F i x ( · ) } N i =1 , to progressively encode and downsample the corrupted shape x t into a collection of multi-scale features, (iii) a middle block M x ( · ) with a self-attention layer to incorporate non-local information into the encoded feature volume, and (iv) N decoder blocks { D i x ( · ) } N i =1 that sequentially upsample features through an inversion convolution to produce a feature volume of the same size as the input x t . The term

…

Incomplete scan(s)

𝜖

!

(𝑥

"

, 𝑐, 𝑡)

Encoder

block

𝑖

…

Middle

block

Decoder

block

𝑖

𝜀

#

(𝑥

"

)

𝐹

#

$

(𝑥

"

)

…

…

…

Encoder

block

𝑖

…

Middle

block

Projection

layer

𝑖

𝜀

%

(𝑐)

…

𝐹

%

$

(𝑓)

Complete shape

(𝑥

&

→ 𝑥

"

)

𝜙

$

𝐹

%

$

(𝑓)

Occupancy-aware fusion

𝜖 ∈ 𝒩(0, I)

+𝜖

Control branch

Main branch

…

𝑐

!

Binarization

Pool & Norm

MLP

𝜙

!

𝐹

"

!

(𝑓

#

)

𝜙

!

𝐹

"

!

(𝑓

$

)

𝑐

"

Binarization

Pool & Norm

𝑤

#

𝑤

$

𝑤

#

!

𝑤

$

!

4

'e(d)ncoder/middle block' denotes a group of neural layers commonly adopted as a unit in neural networks, e.g. , a 'resnet+downsample' block.

We use a control branch (bottom) to extract the features of incomplete scan(s). Likewise, we employ a pre-processing block, N encoder blocks, and a middle block to extract multi-scale features from the conditional input, denoted as ε c ( · ) , { F i c ( · ) } N i =1 , and M c , respectively. They mirror the architecture of the corresponding blocks in the upper branch, yet operating with non-shared parameters, considering the different input data regularities ( e.g. , complete and sparse). Unlike the upper branch, there are no decoders for computation efficiency. Instead, we append a projection layer after each encoder/middle block F i c /M c to forward multi-scale features and feed them into the decoder blocks of the main branch. As diffusion models are time-conditioned, we also convert the time step t into an embedding via two MLPs and add it to the volume features in each network block (the blue box in Fig. 2).

Hierarchical feature aggregation. To integrate the features of the complete and incomplete shapes for conditional control, we fuse them across multiple network levels. In this process, we first consider a single incomplete shape (denoted as c ) as the condition. Given the significant disparity between c and x t (incomplete v.s. complete, TSDF v.s. TUDF), we employ the pre-processing layers ( ε x ( · ) and ε c ( · )) to first empirically project the two different fields into more compatible feature space. Then we fuse them into feature volume f , i.e. , f = ε x ( x t ) + ε c ( c ) . After that, f is passed through the control branch to better propagate the useful information to the deeper layers. In turn, we use multi-scale condition features from the control branch to guide each level of decoder blocks { D i x ( · ) } N i =1 in the main branch. The features that enter the i -th decoder block are denoted as

where [ · , · ] is the concatenation operation and ϕ i is one 1 × 1 convolution layer for feature projection. For simplicity, we use F i x ( x 0 ) to denote the features of the x 0 after the i -th encoder block F i x ( · ) , as x 0 is not directly processed by F i x ( · ) ; the same notation is used for F i c ( f ) and D i -1 x ( x t ) . The feature after the final decoder block is the network output ϵ θ ( x t , t, c ) .

By such a design, we can hierarchically incorporate conditional features, leveraging both their local and broader contexts to optimize the network outputs. Interestingly, we observe that adjusting the network level for feature aggregation can alter a trade-off between completion accuracy and diversity. Specifically, if we only aggregate features at the low-resolution network layers, we might miss finer geometry details present in the higher-resolution layers. This could reduce completion accuracy, as the model has less contextual information to work with. In contrast, if aggregating features at all levels, the effective control might make completion results closely resemble the ground truths, yet lowering the diversity. Further ablation study is provided in Sec. 4.5.

3.3 Occupancy-aware Fusion

Our framework can also take multiple incomplete scans as inputs. This option not only provides richer information to constrain the completed shape geometry but also enhances the approach's practicality, particularly in scenarios that a single-pass scan may not fully capture the entire object. Our critical design is to effectively fuse multiple partial shape features. As averaging the original TSDF volumes is sensitive to registration noise, we first register multiple partial shapes ( e.g. , using Fast-Robust-ICP [71]) and propose an occupancy-aware approach to fuse them in the feature space.

Given a set of incomplete shapes of the same object, denoted as { c 1 , ..., c M } , we individually feed them into the control branch to produce feature volumes { F i c ( f 1 ) , ..., F i c ( f M ) } after the i -th encoder block. Before feeding them into the decoder D i x , we refer to their occupancy masks to perform a weighted feature average. Concretely, we first compute the original occupancy mask for each partial shape based on the TSDF values, i.e. , w j = ( abs ( c j ) < τ ) ∈ B S × S × S for the j -th partial shape. τ is a pre-defined threshold that assigns the volumes near the object surface as occupied and the rest as unoccupied; B is a binary tensor. Then, we perform a pooling operation to resize w j into w i j to align the resolution with feature volume F i c ( f j ) and normalize w i j by w i j = w i j / ( w i 0 + .. + w i M ) . For the voxels with zero values across all occupancy masks, we uniformly assign them a 1e-2 value to avoid the division-by-zero issue. We rewrite Eq. (4), which is for single partial shape condition, as

where ψ is an MLP layer to refine the fused condition features, aiming to mitigate the discrepancies among different partial shape features, as validated in Sec. 4.4. The right part of Fig. 2 illustrates the above process with two incomplete scans as the inputs.

3.4 Training and Inference

We first train the network with a single incomplete shape as the conditional input. In this phase, all the network parameters, except the MLP layer ψ for occupancy-aware fusion (in Eq. (5)), are trained with the objective in Eq. (3). When the network converges, we lock the optimized network parameters and efficiently finetune the MLP layer ψ with multiple incomplete shapes as input.

At the inference stage, we randomize a 3D noise volume as x T from the standard Gaussian distribution. The trained completion networks are then employed for T iterations to produce x 0 from x T conditioned on partial shape(s) c . The occupancy-aware fusion is activated only for multi-condition completion. To accelerate the inference process, we adopt a technique from [61] to sub-sample a set of time steps from [1,..., T /10] during inference. After obtaining the generated shape volume x 0 , we extract an explicit 3D mesh using the marching cube algorithm [72].

4 Experiment

4.1 Experimental Setup

Benchmarks. We evaluate on two large-scale shape completion benchmarks: 3D-EPN [14] and PatchComplete [15]. 3D-EPN comprises 25,590 object instances of eight classes in ShapeNet [73]. For each instance, six partial scans of varying completeness are created in the 32 3 TSDF volumes by virtual scanning; the ground-truth counterpart, represented by 32 3 TUDF, is obtained by a distance field transform on a 3D scanline method [74]. While using a similar data generation pipeline, PatchComplete emphasizes completing objects of unseen categories. It includes both the synthetic data from ShapeNet [73] and the challenging real data from ScanNet [75]. For a fair comparison, we follow their data splits and evaluation metrics, i.e. , mean l 1 error on the TUDF predictions across all voxels on 3D-EPN, and l 1 Chamfer Distance (CD) and Intersection over Union (IoU) between the predicted and ground-truth shapes on PatchComplete. As these metrics only measure the completion accuracy, we introduce other metrics in Sec. 4.3 to compare multimodal completion characteristics.

Implementation details. We first train our network using a single partial scan as input by 200k iterations on four RTX3090 GPUs, taking around two days. If multiple conditions are needed, we finetune project layers ψ for additional 50k iterations. Adam optimizer [76] is employed with a learning rate of 1 e -4 and the batch size is 32. On the 3D-EPN benchmark, we train a specific

model for completing shapes of each known category; while on PatchComplete, we merge all object categories to optimize one model for promoting general completion learning. Due to the unknown class IDs at test time, no classifier-guided [56] or classifier-free [77] sampling techniques are used in our diffusion model. On both two benchmarks, all training data are the same across the compared methods for fairness. The truncation distance in TSDF/TUDFs is set as 3 voxel units. More details about network architecture and experiments are available in the supplementary file. Unless otherwise specified, we report the results on single partial shape completion.

4.2 Main Results

Completion on known object categories. On the 3D-EPN benchmark, we compare DiffComplete against SOTA deterministic [14, 15, 51] and probabilistic [6, 78] methods in terms of completion accuracy ( i.e. , l 1 errors). For probabilistic methods, we use the average results from five inferences, each with random initialization, to account for multimodal outcomes. As shown in Table 1 and Fig. 3, DiffComplete improves over state of the arts by 40% on l 1 error (0.053 v.s. 0.088), as well as producing more realistic and high-fidelity shapes. Deterministic methods include 3D-EPN [14], ConvONet [26], and PatchComplete [15]. Unlike these learning a one-step map function for shape completion, we iteratively refine the generated shape, thus significantly mitigating the surface artifacts; see visualization comparisons in Fig. 3. Compared to GAN-based SDF-StyleGAN [51] and Autoregressive-based AutoSDF [6], our diffusion-based generative model offers superior mode coverage and sampling quality.

Input

3D-EPN

AutoSDF

Ours

GT

ConvONet

ShapeFormer

PVD

U

Input

Patch-

Complete

Ours

GT

Regarding diffusion-based approaches, RePaint-3D is adapted from Repaint [78], a 2D diffusionbased inpainting method that only involves partial shape conditions during the inference process. In contrast, our DiffComplete explicitly matches each partial shape with a complete counterpart at the training stage, thereby improving the output consistency with the ground truths. PVD [12] is originally a point cloud diffusion model. Here, we adapt it to perform TSDF (TUDF) diffusion. A limitation of PVD is that it retains noises of the partial input. Hence, when it is mixed with the generated missing part, noise severely affects the final completion quality. Instead, we design two branches to separately process the partial and complete shapes. By doing so, the model can learn a diffusion process from noise to clean shapes.

Completion on unseen object categories. In two datasets of the PatchComplete benchmark, we compare the generalizability of DiffComplete against the state of the arts, including approaches particularly designed for unseen-class completion [81, 15]. As summarized in Table 2, our method exhibits the best completion quality on average for eight unseen object categories in the synthetic ShapeNet data, despite lacking zero-shot designs. The previous SOTA PatchComplete, mainly leverages the multi-scale structural information to improve the completion robustness. Our method inherently embraces this concept within the diffusion models. With our hierarchical feature aggregation, the network learns multi-scale local completion patterns, which could generalize to various object classes,

Input

Multi-modal outputs

𝒍

𝟏

-err=0.56

𝒍

𝟏

-err=0.12

𝒍

𝟏

-err=0.08

𝒍

𝟏

-err=0.04

Figure 7: TMD curve.

20

40

60

80

90

Completeness Degree (%)

2.0

2.5

3.0

3.5

TMD (

10

2

)

as their local structures are often shared. Our ablation study in Sec. 4.5 further validates this benefit. Table 3 demonstrates our method's superior performance with real-world scans, which are often cluttered and noisy. As showcased in Fig. 4, the 3D shapes produced by DiffComplete stand out for their impressive global coherence and local details.

4.3 Multimodal Completion Characteristics

Quantitative evaluations. Probabilistic methods have the intriguing ability to produce multiple plausible completions on the same partial shape, known as multimodal completion. For the 3D-EPN chair class, we generate ten results per partial shape with randomized initial noise x T , and employ metrics from prior works [7, 6], i.e. , (i) MMD measures the completion accuracy against the ground truths, (ii) TMD for completion diversity, and (iii) UHD for completion fidelity to a partial input. Table 4 shows that our method attains much better completion accuracy and fidelity, while exhibiting moderate diversity. This aligns with our design choice of leveraging the control mechanism to prioritize completion accuracy over diversity. Yet, we can adjust this trade-off to improve shape diversity, as discussed in Sec. 4.5. Fig. 5 presents our multimodal outputs, all showing great realism.

Effects of input completeness degree. To verify the influence of input completeness degree on diversity, we select ten chair instances from ShapeNet, due to chair's large structural variation. For each instance, we create six virtual scans of varying completeness, and for each scan, we generate ten diverse completions to compute their TMD. Fig. 7 presents a negative correlation between the shape diversity (reflected by TMD) and input completeness degree, which is measured by the occupied voxel's ratio between the partial and complete GT shapes.

4.4 Multiple Conditional Inputs

Quantitative evaluations. As indicated in Table 5, DiffComplete consistently improves completion accuracy (lower l 1 error) and reduces diversity (lower TMD) when more conditional inputs are added. Fig. 1(b) and 6 showcase the progression of completion results when we gradually introduce more partial shapes of the same object (denoted by the plus symbol). Our model incorporates the local structures of all partial inputs and harmonizes them into a coherent final shape with the lower l 1 error.

Effects of occupancy-aware fusion. In Table 6, we compare our design with alternatives on the 3D-EPN chair class and PatchComplete benchmark using two conditional inputs. Averaging fea-

tures without occupancy masks largely lowers the completion accuracy due to disturbances from non-informative free-space features. Removing the learnable MLP layer ψ hinders the network's adaptation from single to multiple conditions, also worsening the results. Instead, we adaptively aggregate multi-condition features and refine them to mitigate their discrepancy for reliable completions. Note that directly fusing original scans in TSDF (TUDF) space might be vulnerable to registration errors, as compared with our strategy in the supplementary file.

4.5 Ablation Study

Effects of hierarchical feature aggregation. Table 7 shows the effects of aggregating features of the complete and incomplete shapes at different decoder levels (see Eq. (4)). First, increasing feature aggregation layers (from single to hierarchical) consistently boosts the completion accuracy (lower MMD), while decreasing it yields better diversity (higher TMD). If connecting features only at the network layer with 1/8 resolution, we achieve the best TMD that surpasses other methods (see Table 4). Thus, the accuracy-diversity trade-off can be adjusted by altering the level of feature aggregation. Second, hierarchical feature aggregation facilitates unseen-class completion (lower CD, tested on PatchComplete benchmark). This improvement suggests that leveraging multi-scale structural information from partial inputs enhances the completion robustness.

Effects of feature aggregation mechanism. We ablate on the way to aggregate F i x ( x t ) and ϕ i ( F i c ( f ) in Eq. (4). As Table 8 shows, direct addition achieves the best completion accuracy (the lowest l 1 -err and MMD), as it only combines features at the same 3D location to precisely correlate their difference for completion. Cross-attention disrupts this spatial consistency and yields less accurate results. Concatenation has a similar performance with addition, while the latter is more efficient.

5 Conclusion and Discussion

We presented DiffComplete, a new diffusion-based approach to enable multimodal, realistic, and high-fidelity 3D shape completion, surpassing prior approaches on completion accuracy and quality. This success is attributed to two key designs: a hierarchical feature aggregation mechanism for effective conditional control and an occupancy-aware fusion strategy to seamlessly incorporate additional inputs for geometry refinement. Also, DiffComplete exhibits robust generalization to unseen object classes for both synthetic and real data, and allows an adjustable balance between completion diversity and accuracy to suit specific needs. These features position DiffComplete as a powerful tool in various applications in 3D perception and content creation. Yet, as with most diffusion models, DiffComplete requires additional computation due to its multi-step inference. A comprehensive discussion on the limitation and broader impact is provided in the supplementary file.

References

Michael Zollhöfer, Patrick Stotko, Andreas Görlitz, Christian Theobalt, Matthias Nießner, Reinhard Klein, and Andreas Kolb. State of the art on 3d reconstruction with rgb-d cameras. Computer Graphics Forum , 37:625-652, 05 2018. 1

Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Trans. Graph. , 36(3):24:1-24:18, 2017. 1, 3

Dejan Azinovi´ c, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6290-6301, 2022. 3

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 165-174, 2019.

Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems , 33:2492-2502, 2020. 1

Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 306-315, 2022. 2, 7, 9

Rundi Wu, Xuelin Chen, Yixin Zhuang, and Baoquan Chen. Multimodal shape completion via conditional generative adversarial networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16 , pages 281-296. Springer, 2020. 2, 3, 9

Edward J Smith and David Meger. Improved adversarial systems for 3d object generation and reconstruction. In Conference on Robot Learning , pages 87-96. PMLR, 2017. 3

Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, and Chen Change Loy. Unsupervised 3d shape completion through gan inversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1768-1777, 2021. 2, 3

Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner. Diffrf: Rendering-guided 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4328-4338, 2023. 2, 3, 4

Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander Schwing, and Liangyan Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. arXiv preprint arXiv:2212.04493 , 2022. 3, 4

Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5826-5835, 2021. 3, 7, 8, 9

Gene Chou, Yuval Bahat, and Felix Heide. Diffusion-sdf: Conditional generative modeling of signed distance functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2262-2272, 2023. 2, 3, 4

Angela Dai, Charles Ruizhongtai Qi, and Matthias Nießner. Shape completion using 3d-encoderpredictor cnns and shape synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5868-5877, 2017. 2, 3, 6, 7, 17, 18

Yuchen Rao, Yinyu Nie, and Angela Dai. Patchcomplete: Learning multi-resolution patch priors for 3d shape completion on unseen categories. arXiv preprint arXiv:2206.04916 , 2022. 2, 3, 6, 7, 8, 18, 23, 24

Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part II 13 , pages 834-849. Springer, 2014. 3

Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics , 31(5):1147-1163, 2015.

Thomas Whelan, Stefan Leutenegger, Renato F. Salas-Moreno, Ben Glocker, and Andrew J. Davison. Elasticfusion: Dense SLAM without A pose graph. In Robotics: Science and Systems XI, Sapienza University of Rome, Rome, Italy, July 13-17, 2015 , 2015.

R. Maier, K. Kim, D. Cremers, J. Kautz, and M. Nießner. Intrinsic3d: High-quality 3D reconstruction by joint appearance and geometry optimization with spatially-varying lighting. In International Conference on Computer Vision (ICCV) , Venice, Italy, October 2017.

Michael Zollhöfer, Angela Dai, Matthias Innmann, Chenglei Wu, Marc Stamminger, Christian Theobalt, and Matthias Nießner. Shading-based refinement on volumetric signed distance functions. ACM Transactions on Graphics (TOG) , 2015. 3

Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1996, New Orleans, LA, USA, August 4-9, 1996 , pages 303-312, 1996. 3

Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard A. Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew J. Davison, and Andrew W. Fitzgibbon. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, Santa Barbara, CA, USA, October 16-19, 2011 , pages 559-568, 2011. 3

Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew W. Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In 10th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2011, Basel, Switzerland, October 26-29, 2011 , pages 127-136, 2011. 3

Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. Real-time 3d reconstruction at scale using voxel hashing. ACM Transactions on Graphics (ToG) , 32(6):1-11, 2013. 3

Silvan Weder, Johannes Schonberger, Marc Pollefeys, and Martin R Oswald. Routedfusion: Learning real-time depth map fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4887-4897, 2020. 3

Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , 2020. 7, 19

Ang Li, Zejian Yuan, Yonggen Ling, Wanchao Chi, Chong Zhang, et al. A multi-scale guided cascade hourglass network for depth completion. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 32-40, 2020.

Angela Dai, Yawar Siddiqui, Justus Thies, Julien Valentin, and Matthias Nießner. Spsg: Selfsupervised photometric scene generation from rgb-d scans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1747-1756, 2021. 3

Olga Sorkine and Daniel Cohen-Or. Least-squares meshes. In Proceedings Shape Modeling Applications, 2004. , pages 191-199. IEEE, 2004. 3

Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Marc Alexa. Laplacian mesh optimization. In Proceedings of the 4th international conference on Computer graphics and interactive techniques in Australasia and Southeast Asia , pages 381-389, 2006.

Wei Zhao, Shuming Gao, and Hongwei Lin. A robust hole-filling algorithm for triangular mesh. The Visual Computer , 23:987-997, 2007. 3

Michael M. Kazhdan, Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction. In Proceedings of the Fourth Eurographics Symposium on Geometry Processing, Cagliari, Sardinia, Italy, June 26-28, 2006 , pages 61-70, 2006. 3

Michael Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction. ACM Transactions on Graphics (ToG) , 32(3):1-13, 2013. 3

Sebastian Thrun and Ben Wegbreit. Shape from symmetry. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1 , volume 2, pages 1824-1831. IEEE, 2005. 3

Niloy J Mitra, Leonidas J Guibas, and Mark Pauly. Partial and approximate symmetry detection for 3d geometry. ACM Transactions on Graphics (ToG) , 25(3):560-568, 2006.

Mark Pauly, Niloy J Mitra, Johannes Wallner, Helmut Pottmann, and Leonidas J Guibas. Discovering structural regularity in 3d geometry. In ACM SIGGRAPH 2008 papers . 2008.

Ivan Sipiran, Robert Gregor, and Tobias Schreck. Approximate symmetry detection in partial 3d meshes. In Computer Graphics Forum , 2014.

Pablo Speciale, Martin R Oswald, Andrea Cohen, and Marc Pollefeys. A symmetry prior for convex variational 3d reconstruction. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14 , pages 313-328. Springer, 2016. 3

Minhyuk Sung, Vladimir G Kim, Roland Angst, and Leonidas Guibas. Data-driven structural priors for shape completion. ACM Transactions on Graphics (TOG) , 34(6):1-11, 2015. 3

Yangyan Li, Angela Dai, Leonidas Guibas, and Matthias Nießner. Database-assisted object retrieval for real-time 3d reconstruction. In Computer graphics forum , 2015.

Liangliang Nan, Ke Xie, and Andrei Sharf. A search-classify approach for cluttered indoor scene understanding. ACM Transactions on Graphics (TOG) , 31(6):1-10, 2012.

Young Min Kim, Niloy J Mitra, Dong-Ming Yan, and Leonidas Guibas. Acquiring 3d indoor environments with variability and repetition. ACMTransactions on Graphics (TOG) , 31(6):1-11, 2012. 3

Duc Thanh Nguyen, Binh-Son Hua, Khoi Tran, Quang-Hieu Pham, and Sai-Kit Yeung. A field model for repairing 3d shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5676-5684, 2016. 3

Michael Firman, Oisin Mac Aodha, Simon Julier, and Gabriel J Brostow. Structured prediction of unobserved voxels from a single depth image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5431-5440, 2016. 3

Angela Dai, Christian Diller, and Matthias Nießner. Sg-nn: Sparse generative neural networks for self-supervised scene completion of rgb-d scans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 849-858, 2020. 3

Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-aware transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 12498-12507, 2021.

Xiaoguang Han, Zhen Li, Haibin Huang, Evangelos Kalogerakis, and Yizhou Yu. Highresolution shape completion using deep neural networks for global structure and local geometry inference. In Proceedings of the IEEE international conference on computer vision , pages 85-93, 2017.

Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1746-1754, 2017.

Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape reconstruction and completion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6970-6981, 2020. 3, 7

Angela Dai and Matthias Nießner. Scan2mesh: From unstructured range scans to 3d meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5574-5583, 2019. 3

X Zheng, Yang Liu, P Wang, and Xin Tong. Sdf-stylegan: Implicit sdf-based stylegan for 3d shape generation. In Computer Graphics Forum , 2022. 3, 7

Xuelin Chen, Baoquan Chen, and Niloy J Mitra. Unpaired point cloud completion on real scans using adversarial training. arXiv preprint arXiv:1904.00069 , 2019. 3

Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning , pages 40-49. PMLR, 2018. 3

Himanshu Arora, Saurabh Mishra, Shichong Peng, Ke Li, and Ali Mahdavi-Amiri. Multimodal shape completion via implicit maximum likelihood estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2958-2967, 2022. 3

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015. 3

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021. 7

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022. 4

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR , 2021.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021. 4, 22

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020. 4, 22

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020. 6, 22

Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-decoding models for few-shot conditional generation. Advances in Neural Information Processing Systems , 34:12533-12548, 2021. 3

Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint arXiv:2210.06978 , 2022. 3

Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2837-2845, 2021.

Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751 , 2022. 3

Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero. 3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint arXiv:2212.00842 , 2022. 3, 4

Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. arXiv preprint arXiv:2301.11445 , 2023. 3

Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, et al. Gaudi: A neural architect for immersive 3d scene generation. Advances in Neural Information Processing Systems , 35:25102-25116, 2022. 3

Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: Text-to-shape via voxelized diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12642-12651, 2023. 3, 22

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 , 2023. 4, 17, 21

Juyong Zhang, Yuxin Yao, and Bailin Deng. Fast and robust iterative closest point. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021. 6

William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics , 21(4):163-169, 1987. 6

Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 , 2015. 6, 7, 19

John Amanatides, Andrew Woo, et al. A fast voxel traversal algorithm for ray tracing. In Eurographics , 1987. 6

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5828-5839, 2017. 6, 7

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. 6

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022. 7

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11461-11471, 2022. 7, 8, 9

Dongsu Zhang, Changwoon Choi, Inbum Park, and Young Min Kim. Probabilistic implicit scene completion. In International Conference on Learning Representations , 2022. 7

Xingguang Yan, Liqiang Lin, Niloy J Mitra, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Shapeformer: Transformer-based shape completion via sparse representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022. 7, 9

Bram Wallace and Bharath Hariharan. Few-shot generalization for single-image 3d reconstruction via priors. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3818-3827, 2019. 7, 8

Peng Xiang, Xin Wen, Yu-Shen Liu, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Zhizhong Han. Snowflake point deconvolution for point cloud completion and generation with skip-transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022. 19

Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. arXiv preprint arXiv:2211.13449 , 2022. 22

Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. CVPR , 2018. 22

Tao Hu, Xiaogang Xu, Ruihang Chu, and Jiaya Jia. Trivol: Point cloud rendering via triple volumes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 22

Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong. O-cnn: Octree-based convolutional neural networks for 3d shape analysis. ACM Transactions On Graphics (TOG) , 36(4):1-11, 2017. 22

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the shape completion task by introducing a diffusion-based technique. To do so, additional design choices have been made such as hierarchical feature aggregation and an occupancy-aware fusion strategy to better reproduce the shape and respect the details. Based on numerous quantitative and qualitative experiments, the proposed method outperforms others by a considerable margin.\n\nSTRENGTHS:\nThe paper clearly outperforms other SOTA.\nThe visual quality of the completion is good. \nOccupancy-aware fusion and hierarchical feature aggregation that are the main task specific designs make sense and the authros have successfully shown their importance in the design through ablation studies.\nI like the adoption of control net in the design. I believe it makes sense to learn gradually and at different scales. \nThe fact that the data is from the actual scan and it is not synthesized by adding noise and removing parts make the method very useful and interesting.\n\nWEAKNESSES:\nIn terms of writing, the paper could use simpler and more understandable and short sentences. For example, this sentence could be improved by making it shorter or breaking it into two sentences:\nTo improve the completion accuracy, we enable effective control from single-input condition by hierarchical and spatially-consistent feature aggregation, and from multiple ones by an occupancy-aware fusion strategy.\nThis happens many times throughout the paper. \nMissing reference:\nMultimodal shape completion with IMLE:\nhttps://openaccess.thecvf.com/content/CVPR2022W/DLGC/html/Arora_Multimodal_Shape_Completion_via_Implicit_Maximum_Likelihood_Estimation_CVPRW_2022_paper.html\n\nQUESTIONS:\nWhen you pro-process the input and the incomplete shape through a convolutional network and add them, why do you call it aligning distributions? \nAs the computation and time are two limitations of this method, would a part-based completion task be a better approach? In fact, my question is whether a part based approach in which a model is first broken into parts will save computation time and understand the geometry better?\n\nLIMITATIONS:\nThe method is mostly trained category specific. Considering the time and computational cost of this approach, this makes it hard and expensive to be adopted and modified. easily. However, I weigh the quality of the results more than the time and computation so I am okay with this limitation.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

comments: SUMMARY:
The authors present a diffusion-based neural network for 3D shape completion. They introduce a hierarchical and spatially consistent feature aggregation scheme to fuse partial and complete shapes, and an occupancy-aware fusion strategy to fuse multiple partial shapes. They demonstrate a new SOTA performance in shape completion for both new instances and completely unknown object categories.

STRENGTHS:
1. The authors successfully apply the diffusion-based model to the shape completion task and achieve a new SOTA performance.

2. The stepwise addition of partial scans to predict a complete shape is interesting and makes completion tasks interactive and adaptable.

3. The proposed model supports multimodal completions that can predict multiple plausible results.

WEAKNESSES:
1. The writing of this paper can be improved.

2. The technical designs, such as the two feature aggregation schemes, are not well motivated. As a main contribution, I expect more explanations of the design ideas.

3. The experimental design is not very clear. Does DiffComplete train on a single category? Since PatchComplete trains on all categories, DiffComplete should also train across categories to allow a fair comparison.

4. The results of multiple conditional inputs are unpredictable. A better example is adding and editing a semantic part of the object from a partial input.

5. Does the proposed model require much more time than its competitors?

QUESTIONS:
1. I wonder if the resolution can go up to 64^3 or even 128^3?

2. The diversity of multimodal complement is worse than other methods. So is it because the diffusion model gives more deterministic results than others?

LIMITATIONS:
The limitation is not addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of probabilistic shape completion using diffusion models, learning from range scans. The main contribution of the paper comes from proposing 2 novel techniques: hierarchical feature aggregation for strong conditioning and occupancy-aware fusion technique. The method is tested on ShapeNet dataset and achieves better completions compared to the baselines. Also, the proposed method exhibits robust generatlization for out of distribution inputs.\n\nSTRENGTHS:\n* I enjoyed the idea of hierarchical feature aggregation. To best of my knowledge, conditioning a diffusion model is hard, and the ablation study in Tab. 7 clearly shows that the multi-level conditioning acts as intended.\n\n* Strong empirical results on completion on unseen categories. The results in Tab2, 3 shows that the method achieves strong completion results on unseen categories and it makes sense due to strong hierarchical conditioning.\n\nWEAKNESSES:\n* Similar works on 3D completion using diffusion models exists. As mentioned in the related sections, the difference compes from the representation used to represent the state in the diffusion model. For example, point-voxel uses point cloud, while this work uses TSDF (TUDF) representation. While I do not think that the novelty of the work degrades even there are other diffusion based 3D completion models, I want to hear in detail how the method differs from other diffusion based models. Please look at the question section. \n\n* My main concern is that some of the important recent works are not mentioned and compared. For determnisitc completion methods, convocc completes the given partial point cloud using occupancy fields. For multimodal completion methods, point-voxel [2] (although mentioned in the related-section) and GCA[3] completes point cloud in point cloud/sparse voxel representation. ShapeFormer[4] and cGCA[5] completes the point cloud in implicit neural representations.\nAlthough I realize that comparing all of these baselines requires alot of effort, these baselines should have been tested since the core contribution of the work stems from achieving SOTA results on an existing benchmark. \n\n[1] Peng et al. Convolutional Occupancy Networks. ECCV, 2020\n\n[2] Zhou et al. 3D Shape Generation and Completion Through Point-Voxel Diffusion. ICCV, 2021\n\n[3] Zhang et al. Learning to generate 3d shapes with generative cellular automata. ICLR, 2021\n\n[4] Yan et al. ShapeFormer: Transformer-based Shape Completion via Sparse Representation. CVPR, 2022\n\n[5]  Zhang et al. Probabilistic implicit scene completion. ICLR, 2022\n\nQUESTIONS:\n* Regarding the related work with diffusion based models, the authors mentioned that “. Due to the absence of meaningful ground truths in these completion scenarios, they could also face completion accuracy challenges like the above generative approaches” in line 114. Does that mean that the diffusion methods require the ground truth while the proposed method does not? To the best of my knowledge, the diffcomplete model requires ground truth TUDF for training. I think that the assumption that we can acquire the ground truth TUDF means that we can acquire ground truth point cloud and implicit function (mostly UDF based) as well. If so, then other baseline methods mentioned in the weakness section should be compared.\n\nLIMITATIONS:\nAuthors have addressed the limitations in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a diffusion-based approach, DiffComplete, to generate complete shapes conditioned on partial 3D range scans. The condition is represented as volumetric TSDF (truncated signed distance function), while the complete shape is represented as volumetric TUDF (truncated signed distance function). Inspired by 2D ControlNet, the authors devise a hierarchical feature aggregation mechanism to inject conditional features in a spatially-consistent manner. Besides, they propose a fine-tuning strategy to adapt the model trained on a single condition to multiple conditions. The trade-off between multi-modality and high fidelity can be controlled through the network level for feature aggregation between conditions and denoised complete shapes. The results on 3D-EPN and PatchComplete show the superiority of DiffComplete. The authors also demonstrate its zero-shot ability.\n\nSTRENGTHS:\n- The paper is clearly written and easy to follow.\n- The ControlNet-style design to inject features from the condition is reasonable.\n- Fine-tuning the network trained with a single incomplete shape for multiple incomplete shapes is a good strategy.\n- The generalizability looks good according to Fig. 4.\n\nWEAKNESSES:\nIt is unclear why some baselines (especially point-cloud-based methods) are missing. For example, \"3D Shape Generation and Completion Through Point-Voxel Diffusion\" and \"SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer\".\n\nQUESTIONS:\n1. Is there a typo in Figure 7? Is it an MMD curve?\n2. For the ablation study on multiple conditions, can the authors first fuse the scans and compute TSDF, then extract features instead of averaging features extracted from individual scans?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations (e.g., failure cases) in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed method aims to tackle the TSDF shape completion problem using diffusion models. That is, given one or several incomplete TSDFs obtained from partial scans of a single object produced by range sensors, the method generates a TSDF of complete shape while trying to preserve the geometric details of the incomplete one. The method achieves this by using the same technique in ControlNet: adding a control branch to the vanilla UNet module in diffusion models. \nAlso, in order to better incorporate the partial scans from different views, the authors propose an \"occupancy-aware fusion\" module which performs a weighted feature fusion for the multi-view scans considering their geometric reliability.\n\nSTRENGTHS:\nIn general, the paper is easy to read and is technically sound.\nThe following are specific points:\n\n- Single probabilistic framework for all tasks. As more and more recent papers suggested, generative models are not only doing great in generation problems, but are also competitive in deterministic problems where usually a unique optimal solution is desired given the input. This method shows this point for the 3D completion problem: Even when the input shape has little to no ambiguity, the generated completion has better quality than previous deterministic approaches.\n\n- Incorporate multi-view scans. This special consideration is useful in real-life application, where the scanners are constantly capturing new scans and the small errors in registration will make it non-trivial to fuse features from different views.\n\nWEAKNESSES:\nI have concerns on baseline & references, unsatisfactory contricution and writing. Hence I believe this paper does not reach the NeurIPS bar and would like to rate \"Reject\".\nHere are detailed points:\n\n- Missing baselines and references. Comparison with multiple very related works are omitted. ConvOccNet can perform shape completion very well in higher resolution (64 and higher, see the Fig. 4 in their supplementary). The model is very good at deterministic completion. For multimodal setting, two very related work, ShapeFormer and 3DILG, are not mentioned or discussed. And the former work is especially designed for multimodal completion.\n\n- Contribution is not satisfactory. The proposed method seems to be just deploying ControlNet on the PatchComplete problem and benchmark. The used resolution (32) is also not impressive comparing to previous works like DiffusionSDF or 3DShape2VecSet. I would be skeptical that training diffusion models directly and keep the partial tsdf unchanged during sampling (as what PVD does for completion) can achieve better results.\n\n- Many minor writing problems. L99: It is confusing to classify AutoSDF to be a type of autoencoder. It is an autoregressive approach. \nL114: \"Due to the absence of meaningful ground truths, ...\". In the mentioned works, they all have meaningful ground truths shapes. It is unclear what is \"meaningful\" here.\nL325: \"Effects of feature aggregation manner\". A better way is to write: \"Effects of the manner in which features are aggregated\" or \"Effects of our feature aggregation mechanism\"\n\nQUESTIONS:\n- Please show advantages over these baselines:\n(1) Convolutional Occupancy Network. Train ConvOccNet to map partial TSDF grid to complete grid. Then compare.\n(2) ShapeFormer. TSDF can be converted to point cloud. Train ShapeFormer to map point cloud to implicit TSDF fields.\n(3) TSDF diffusion with PVD style of completion sampling. That is, in the sampling time, always replace the noisy to the actual TSDF value in the place of partial scanned cells. This way, only the \"missing\" regions are denoised and completed.\n\nLIMITATIONS:\nAuthors include a comprehensive discussion on method limitations and potential societal impact in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}