Decompositional Neural Scene Reconstruction with Generative Diffusion
  Prior
Authors: name: Junfeng Ni, name: Yu Liu, name: Ruijie Lu, name: Zirui Zhou, name: Song-Chun Zhu, name: Yixin Chen, name: Siyuan Huang

Decompositional reconstruction of 3D scenes, with complete shapes and
detailed texture of all objects within, is intriguing for downstream
applications but remains challenging, particularly with sparse views as input.
Recent approaches incorporate semantic or geometric regularization to address
this issue, but they suffer significant degradation in underconstrained areas
and fail to recover occluded regions. We argue that the key to solving this
problem lies in supplementing missing information for these areas. To this end,
we propose DP-Recon, which employs diffusion priors in the form of Score
Distillation Sampling (SDS) to optimize the neural representation of each
individual object under novel views. This provides additional information for
the underconstrained areas, but directly incorporating diffusion prior raises
potential conflicts between the reconstruction and generative guidance.
Therefore, we further introduce a visibility-guided approach to dynamically
adjust the per-pixel SDS loss weights. Together these components enhance both
geometry and appearance recovery while remaining faithful to input images.
Extensive experiments across Replica and ScanNet++ demonstrate that our method
significantly outperforms SOTA methods. Notably, it achieves better object
reconstruction under 10 views than the baselines under 100 views. Our method
enables seamless text-based editing for geometry and appearance through SDS
optimization and produces decomposed object meshes with detailed UV maps that
support photorealistic Visual effects (VFX) editing. The project page is
available at https://dp-recon.github.io/.