MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation
Authors: name: Zehuan Huang; name: Yuan-Chen Guo; name: Xingqiao An; name: Yunhan Yang; name: Yangguang Li; name: Zi-Xin Zou; name: Ding Liang; name: Xihui Liu; name: Yan-Pei Cao; name: Lu Sheng
Published: 2024-12-04 18:52:40

Abstract: This paper introduces MIDI, a novel paradigm for compositional 3D scene
generation from a single image. Unlike existing methods that rely on
reconstruction or retrieval techniques or recent approaches that employ
multi-stage object-by-object generation, MIDI extends pre-trained image-to-3D
object generation models to multi-instance diffusion models, enabling the
simultaneous generation of multiple 3D instances with accurate spatial
relationships and high generalizability. At its core, MIDI incorporates a novel
multi-instance attention mechanism, that effectively captures inter-object
interactions and spatial coherence directly within the generation process,
without the need for complex multi-step processes. The method utilizes partial
object images and global scene context as inputs, directly modeling object
completion during 3D generation. During training, we effectively supervise the
interactions between 3D instances using a limited amount of scene-level data,
while incorporating single-object data for regularization, thereby maintaining
the pre-trained generalization ability. MIDI demonstrates state-of-the-art
performance in image-to-scene generation, validated through evaluations on
synthetic data, real-world scene data, and stylized scene images generated by
text-to-image diffusion models.