MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes
Authors: name: Ruijie Lu; name: Yixin Chen; name: Junfeng Ni; name: Baoxiong Jia; name: Yu Liu; name: Diwen Wan; name: Gang Zeng; name: Siyuan Huang
Published: 2024-12-16 05:23:45

Abstract: Repurposing pre-trained diffusion models has been proven to be effective for
NVS. However, these methods are mostly limited to a single object; directly
applying such methods to compositional multi-object scenarios yields inferior
results, especially incorrect object placement and inconsistent shape and
appearance under novel views. How to enhance and systematically evaluate the
cross-view consistency of such models remains under-explored. To address this
issue, we propose MOVIS to enhance the structural awareness of the
view-conditioned diffusion model for multi-object NVS in terms of model inputs,
auxiliary tasks, and training strategy. First, we inject structure-aware
features, including depth and object mask, into the denoising U-Net to enhance
the model's comprehension of object instances and their spatial relationships.
Second, we introduce an auxiliary task requiring the model to simultaneously
predict novel view object masks, further improving the model's capability in
differentiating and placing objects. Finally, we conduct an in-depth analysis
of the diffusion sampling process and carefully devise a structure-guided
timestep sampling scheduler during training, which balances the learning of
global object placement and fine-grained detail recovery. To systematically
evaluate the plausibility of synthesized images, we propose to assess
cross-view consistency and novel view object placement alongside existing
image-level NVS metrics. Extensive experiments on challenging synthetic and
realistic datasets demonstrate that our method exhibits strong generalization
capabilities and produces consistent novel view synthesis, highlighting its
potential to guide future 3D-aware multi-object NVS tasks.