Imaginarium: Vision-guided High-Quality 3D Scene Layout Generation
Authors: name: Xiaoming Zhu, name: Xu Huang, name: Qinghongbing Xie, name: Zhi Deng, name: Junsheng Yu, name: Yirui Guan, name: Zhongyuan Liu, name: Lin Zhu, name: Qijun Zhao, name: Ligang Liu, name: Long Zeng

Generating artistic and coherent 3D scene layouts is crucial in digital
content creation. Traditional optimization-based methods are often constrained
by cumbersome manual rules, while deep generative models face challenges in
producing content with richness and diversity. Furthermore, approaches that
utilize large language models frequently lack robustness and fail to accurately
capture complex spatial relationships. To address these challenges, this paper
presents a novel vision-guided 3D layout generation system. We first construct
a high-quality asset library containing 2,037 scene assets and 147 3D scene
layouts. Subsequently, we employ an image generation model to expand prompt
representations into images, fine-tuning it to align with our asset library. We
then develop a robust image parsing module to recover the 3D layout of scenes
based on visual semantics and geometric information. Finally, we optimize the
scene layout using scene graphs and overall visual semantics to ensure logical
coherence and alignment with the images. Extensive user testing demonstrates
that our algorithm significantly outperforms existing methods in terms of
layout richness and quality. The code and dataset will be available at
https://github.com/HiHiAllen/Imaginarium.