SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\psi$  is an MLP layer to refine the fused condition features, aiming to mitigate the discrepancies among different partial shape features, as validated in Sec. 4.4. The right part of Fig. 2 illustrates the above process with two incomplete scans as the inputs.
We first train the network with a single incomplete shape as the conditional input. In this phase, all the network parameters, except the MLP layer  $\psi$  for occupancy-aware fusion (in Eq. (5)), are trained with the objective in Eq. (3). When the network converges, we lock the optimized network parameters and efficiently finetune the MLP layer  $\psi$  with multiple incomplete shapes as input.
At the inference stage, we randomize a 3D noise volume as  $x_{T}$  from the standard Gaussian distribution. The trained completion networks are then employed for  $T$  iterations to produce  $x_{0}$  from  $x_{T}$  conditioned on partial shape(s)  $c$ . The occupancy-aware fusion is activated only for multi-condition completion. To accelerate the inference process, we adopt a technique from [61] to sub-sample a set of time steps from  $[1,\dots,T / 10]$  during inference. After obtaining the generated shape volume  $x_{0}$ , we extract an explicit 3D mesh using the marching cube algorithm [72].
**Benchmarks.** We evaluate on two large-scale shape completion benchmarks: 3D-EPN [14] and PatchComplete [15]. 3D-EPN comprises 25,590 object instances of eight classes in ShapeNet [73]. For each instance, six partial scans of varying completeness are created in the  $32^{3}$  TSDF volumes by virtual scanning; the ground-truth counterpart, represented by  $32^{3}$  TUDF, is obtained by a distance field transform on a 3D scanline method [74]. While using a similar data generation pipeline, PatchComplete emphasizes completing objects of unseen categories. It includes both the synthetic data from ShapeNet [73] and the challenging real data from ScanNet [75]. For a fair comparison, we follow their data splits and evaluation metrics, i.e., mean  $l_{1}$  error on the TUDF predictions across all voxels on 3D-EPN, and  $l_{1}$  Chamfer Distance (CD) and Intersection over Union (IoU) between the predicted and ground-truth shapes on PatchComplete. As these metrics only measure the completion accuracy, we introduce other metrics in Sec. 4.3 to compare multimodal completion characteristics.
Implementation details. We first train our network using a single partial scan as input by 200k iterations on four RTX3090 GPUs, taking around two days. If multiple conditions are needed, we finetune project layers  $\psi$  for additional 50k iterations. Adam optimizer [76] is employed with a learning rate of  $1e^{-4}$  and the batch size is 32. On the 3D-EPN benchmark, we train a specific
Table 1: Quantitative shape completion results on objects of known categories [14].
l1-err. (↓)	Avg. (↓)	Chair	Table	Sofa	Lamp	Plane	Car	Dresser	Boat
3D-EPN [14]	0.374	0.418	0.377	0.392	0.388	0.421	0.259	0.381	0.356
SDF-StyleGAN [51]	0.278	0.321	0.256	0.289	0.280	0.295	0.224	0.273	0.282
RePaint-3D [78]	0.266	0.289	0.264	0.266	0.268	0.302	0.214	0.285	0.243
ConvONet [26]	0.220	0.210	0.247	0.254	0.234	0.185	0.195	0.250	0.184
AutoSDF [6]	0.217	0.201	0.258	0.226	0.275	0.184	0.187	0.248	0.157
cGCA [79]	0.185	0.174	0.212	0.179	0.239	0.170	0.161	0.204	0.143
ShapeFormer [80]	0.141	0.104	0.175	0.133	0.176	0.136	0.127	0.157	0.119
PVD [12]	0.114	0.097	0.122	0.154	0.128	0.093	0.087	0.127	0.107
PatchComplete [15]	0.088	0.134	0.095	0.084	0.087	0.061	0.053	0.134	0.058
DiffComplete (Ours)	0.053	0.070	0.073	0.061	0.059	0.015	0.025	0.086	0.031
Table 2: Shape completion results on synthetic objects [73] of unseen categories.  $\cdot  / \cdot$  means CD/IoU.
CD(↓)/IoU(↑)	3D-EPN [14]	Few-Shot [81]	IF-Nets [49]	Auto-SDF [6]	ConvONet [26]	PatchComplete [15]	Ours
Bag	5.01 / 73.8	8.00 / 56.1	4.77 / 69.8	5.81 / 56.3	5.10 / 70.8	3.94 / 77.6	3.86 / 78.3
Lamp	8.07 / 47.2	15.1 / 25.4	5.70 / 50.8	6.57 / 39.1	5.42 / 52.6	4.68 / 56.4	4.80 / 57.9
Bathtub	4.21 / 57.9	7.05 / 45.7	4.72 / 55.0	5.17 / 41.0	4.96 / 60.4	3.78 / 66.3	3.52 / 68.9
Bed	5.84 / 58.4	10.0 / 39.6	5.34 / 60.7	6.01 / 44.6	5.42 / 63.2	4.49 / 66.8	4.16 / 67.1
Basket	7.90 / 54.0	8.72 / 40.6	4.44 / 50.2	6.70 / 39.8	6.16 / 54.6	5.15 / 61.0	4.94 / 65.5
Printer	5.15 / 73.6	9.26 / 56.7	5.83 / 70.5	7.52 / 49.9	5.56 / 72.1	4.63 / 77.6	4.40 / 76.8
Laptop	3.90 / 62.0	10.4 / 31.3	6.47 / 58.3	4.81 / 51.1	4.78 / 57.3	3.77 / 63.8	3.52 / 67.4
Bench	4.54 / 48.3	8.11 / 27.2	5.03 / 49.7	4.31 / 39.5	4.69 / 49.6	3.70 / 53.9	3.56 / 58.2
Avg.	5.58 / 59.4	9.58 / 40.3	5.29 / 58.1	5.86 / 45.2	5.26 / 60.1	4.27 / 65.4	4.10 / 67.5
Table 3: Shape completion results on real-world objects [75] of unseen categories.  $\cdot  / \cdot$  means CD/IoU.
CD(↓)/IoU(↑)	3D-EPN [14]	Few-Shot [81]	IF-Nets [49]	Auto-SDF [6]	ConvONet [26]	PatchComplete [15]	Ours
Bag	8.83 / 53.7	9.10 / 44.9	8.96 / 44.2	9.30 / 48.7	9.12 / 52.5	8.23 / 58.3	7.05 / 48.5
Lamp	14.3 / 20.7	11.9 / 19.6	10.2 / 24.9	11.2 / 24.4	9.83 / 20.3	9.42 / 28.4	6.84 / 30.5
Bathtub	7.56 / 41.0	7.77 / 38.2	7.19 / 39.5	7.84 / 36.6	7.93 / 41.2	6.77 / 48.0	8.22 / 48.5
Bed	7.76 / 47.8	9.07 / 34.9	8.24 / 44.9	7.91 / 38.0	8.14 / 41.6	7.24 / 48.4	7.20 / 46.6
Basket	7.74 / 36.5	8.02 / 34.3	6.74 / 42.7	7.54 / 36.1	7.39 / 37.0	6.60 / 45.5	7.42 / 59.2
Printer	8.36 / 63.0	8.30 / 62.2	8.28 / 60.7	9.66 / 49.9	7.62 / 64.9	6.84 / 70.5	6.36 / 74.5
Avg.	9.09 / 44.0	9.02 / 38.6	8.26 / 42.6	8.90 / 38.9	8.34 / 42.9	7.52 / 49.8	7.18 / 51.3
model for completing shapes of each known category; while on PatchComplete, we merge all object categories to optimize one model for promoting general completion learning. Due to the unknown class IDs at test time, no classifier-guided [56] or classifier-free [77] sampling techniques are used in our diffusion model. On both two benchmarks, all training data are the same across the compared methods for fairness. The truncation distance in TSDF/TUDFs is set as 3 voxel units. More details about network architecture and experiments are available in the supplementary file. Unless otherwise specified, we report the results on single partial shape completion.
Completion on known object categories. On the 3D-EPN benchmark, we compare DiffComplete against SOTA deterministic [14, 15, 51] and probabilistic [6, 78] methods in terms of completion accuracy (i.e.,  $l_{1}$  errors). For probabilistic methods, we use the average results from five inferences, each with random initialization, to account for multimodal outcomes. As shown in Table 1 and Fig. 3, DiffComplete improves over state of the arts by  $40\%$  on  $l_{1}$  error (0.053 v.s. 0.088), as well as producing more realistic and high-fidelity shapes. Deterministic methods include 3D-EPN [14], ConvONet [26], and PatchComplete [15]. Unlike these learning a one-step map function for shape completion, we iteratively refine the generated shape, thus significantly mitigating the surface artifacts; see visualization comparisons in Fig. 3. Compared to GAN-based SDF-StyleGAN [51] and Autoregressive-based AutoSDF [6], our diffusion-based generative model offers superior mode coverage and sampling quality.
Figure 3: Shape Completion on various known object classes. We achieve the best completion quality.
Figure 4: Shape completion on synthetic (blue) and real (yellow) objects of entirely unseen classes. Our method produces the completed shapes in superior quality given both synthetic and real data.
Regarding diffusion-based approaches, RePaint-3D is adapted from Repaint [78], a 2D diffusion-based inpainting method that only involves partial shape conditions during the inference process. In contrast, our DiffComplete explicitly matches each partial shape with a complete counterpart at the training stage, thereby improving the output consistency with the ground truths. PVD [12] is originally a point cloud diffusion model. Here, we adapt it to perform TSDF (TUDF) diffusion. A limitation of PVD is that it retains noises of the partial input. Hence, when it is mixed with the generated missing part, noise severely affects the final completion quality. Instead, we design two branches to separately process the partial and complete shapes. By doing so, the model can learn a diffusion process from noise to clean shapes.
Completion on unseen object categories. In two datasets of the PatchComplete benchmark, we compare the generalizability of DiffComplete against the state of the arts, including approaches particularly designed for unseen-class completion [81, 15]. As summarized in Table 2, our method exhibits the best completion quality on average for eight unseen object categories in the synthetic ShapeNet data, despite lacking zero-shot designs. The previous SOTA PatchComplete, mainly leverages the multi-scale structural information to improve the completion robustness. Our method inherently embraces this concept within the diffusion models. With our hierarchical feature aggregation, the network learns multi-scale local completion patterns, which could generalize to various object classes,
Figure 5: Our method produces multimodal plausible results given the same partial shape.
Figure 6: Our method incorporates multiple partial shapes to refine completion results  $(l_{1}$  -err.  $\downarrow$
Table 4: Multimodal capacity.
Method	MMD ↓	TMD ↑	UHD ↓
AutoSDF [6]	0.008	0.028	0.061
ShapeFormer [80]	0.007	0.024	0.055
PVD [12]	0.007	0.027	0.042
RePaint-3D [78]	0.007	0.029	0.053
Ours	0.002	0.025	0.032
Figure 7: TMD curve.
Table 5: Multi-condition results.
Cond. Num. | l1-err. ↓	MMD ↓	TMD ↑
one	0.07	0.002	0.025
two	0.05	0.001	0.021
three	0.04	0.001	0.019
Table 6: Effects of fusion choices Table 7: Feat. aggregation levels. Table 8: Ablation on different feafor multiple conditional inputs.  $1/8\quad 1/4\quad 1/2\quad 1\quad \mid \mathrm{MMD}\downarrow \quad \mathrm{TMD}\uparrow \mid \mathrm{CD}\downarrow$  ture aggregation mechanisms.
Strategy	l1-err. ↓	CD ↓	IoU ↑
simple average	0.13	4.56	63.3
w/o MLP ψ	0.23	5.88	57.4
occ-aware (Ours)	0.05	3.97	68.3
✓	0.005	0.031	4.8
✓	✓	0.004	0.028	4.4
✓	✓	✓	0.003	0.026	4.2
✓	✓	✓	0.002	0.025	4.1
Operation	l1-err. ↓	MMD ↓	TMD ↑
cross-attn.	0.12	0.005	0.027
concat.	0.07	0.002	0.024
addition (Ours)	0.07	0.002	0.025
as their local structures are often shared. Our ablation study in Sec. 4.5 further validates this benefit. Table 3 demonstrates our method's superior performance with real-world scans, which are often cluttered and noisy. As showcased in Fig. 4, the 3D shapes produced by DiffComplete stand out for their impressive global coherence and local details.
4.3 Multimodal Completion Characteristics
Quantitative evaluations. Probabilistic methods have the intriguing ability to produce multiple plausible completions on the same partial shape, known as multimodal completion. For the 3D-EPN chair class, we generate ten results per partial shape with randomized initial noise  $x_{T}$ , and employ metrics from prior works [7, 6], i.e., (i) MMD measures the completion accuracy against the ground truths, (ii) TMD for completion diversity, and (iii) UHD for completion fidelity to a partial input. Table 4 shows that our method attains much better completion accuracy and fidelity, while exhibiting moderate diversity. This aligns with our design choice of leveraging the control mechanism to prioritize completion accuracy over diversity. Yet, we can adjust this trade-off to improve shape diversity, as discussed in Sec. 4.5. Fig. 5 presents our multimodal outputs, all showing great realism.
Effects of input completeness degree. To verify the influence of input completeness degree on diversity, we select ten chair instances from ShapeNet, due to chair's large structural variation. For each instance, we create six virtual scans of varying completeness, and for each scan, we generate ten diverse completions to compute their TMD. Fig. 7 presents a negative correlation between the shape diversity (reflected by TMD) and input completeness degree, which is measured by the occupied voxel's ratio between the partial and complete GT shapes.
4.4 Multiple Conditional Inputs
Quantitative evaluations. As indicated in Table 5, DiffComplete consistently improves completion accuracy (lower  $l_{1}$  error) and reduces diversity (lower TMD) when more conditional inputs are added. Fig. 1(b) and 6 showcase the progression of completion results when we gradually introduce more partial shapes of the same object (denoted by the plus symbol). Our model incorporates the local structures of all partial inputs and harmonizes them into a coherent final shape with the lower  $l_{1}$  error.
Effects of occupancy-aware fusion. In Table 6, we compare our design with alternatives on the 3D-EPN chair class and PatchComplete benchmark using two conditional inputs. Averaging fea
tures without occupancy masks largely lowers the completion accuracy due to disturbances from non-informative free-space features. Removing the learnable MLP layer  $\psi$  hinders the network's adaptation from single to multiple conditions, also worsening the results. Instead, we adaptively aggregate multi-condition features and refine them to mitigate their discrepancy for reliable completions. Note that directly fusing original scans in TSDF (TUDF) space might be vulnerable to registration errors, as compared with our strategy in the supplementary file.
Effects of hierarchical feature aggregation. Table 7 shows the effects of aggregating features of the complete and incomplete shapes at different decoder levels (see Eq. (4)). First, increasing feature aggregation layers (from single to hierarchical) consistently boosts the completion accuracy (lower MMD), while decreasing it yields better diversity (higher TMD). If connecting features only at the network layer with  $1/8$  resolution, we achieve the best TMD that surpasses other methods (see Table 4). Thus, the accuracy-diversity trade-off can be adjusted by altering the level of feature aggregation. Second, hierarchical feature aggregation facilitates unseen-class completion (lower CD, tested on PatchComplete benchmark). This improvement suggests that leveraging multi-scale structural information from partial inputs enhances the completion robustness.
Effects of feature aggregation mechanism. We ablate on the way to aggregate  $F_{x}^{i}(x_{t})$  and  $\phi^i (F_c^i (f)$  in Eq. (4). As Table 8 shows, direct addition achieves the best completion accuracy (the lowest  $l_{1}$ -err and MMD), as it only combines features at the same 3D location to precisely correlate their difference for completion. Cross-attention disrupts this spatial consistency and yields less accurate results. Concatenation has a similar performance with addition, while the latter is more efficient.
We presented DiffComplete, a new diffusion-based approach to enable multimodal, realistic, and high-fidelity 3D shape completion, surpassing prior approaches on completion accuracy and quality. This success is attributed to two key designs: a hierarchical feature aggregation mechanism for effective conditional control and an occupancy-aware fusion strategy to seamlessly incorporate additional inputs for geometry refinement. Also, DiffComplete exhibits robust generalization to unseen object classes for both synthetic and real data, and allows an adjustable balance between completion diversity and accuracy to suit specific needs. These features position DiffComplete as a powerful tool in various applications in 3D perception and content creation. Yet, as with most diffusion models, DiffComplete requires additional computation due to its multi-step inference. A comprehensive discussion on the limitation and broader impact is provided in the supplementary file.
This work is partially supported by the Research Grants Council under the Areas of Excellence scheme grant AoE/E-601/22-R, Shenzhen Science and Technology Program KQTD20210811090149095, and the Research Grants Council of the Hong Kong Special Administrative Region, China (CUHK 14206320).
[81] Bram Wallace and Bharath Hariharan. Few-shot generalization for single-image 3d reconstruction via priors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3818-3827, 2019. 7, 8
[82] Peng Xiang, Xin Wen, Yu-Shen Liu, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Zhizhong Han. Snowflake point deconvolution for point cloud completion and generation with skip-transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. 19
[83] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. arXiv preprint arXiv:2211.13449, 2022. 22
[84] Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. CVPR, 2018. 22
[85] Tao Hu, Xiaogang Xu, Ruihang Chu, and Jiaya Jia. Trivol: Point cloud rendering via triple volumes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 22
[86] Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong. O-cnn: Octree-based convolutional neural networks for 3d shape analysis. ACM Transactions on Graphics (TOG), 36(4):1-11, 2017. 22