SceneGen: Single-Image 3D Scene Generation in One Feedforward Pass
Authors: name: Yanxu Meng, name: Haoning Wu, name: Ya Zhang, name: Weidi Xie

3D content generation has recently attracted significant research interest
due to its applications in VR/AR and embodied AI. In this work, we address the
challenging task of synthesizing multiple 3D assets within a single scene
image. Concretely, our contributions are fourfold: (i) we present SceneGen, a
novel framework that takes a scene image and corresponding object masks as
input, simultaneously producing multiple 3D assets with geometry and texture.
Notably, SceneGen operates with no need for optimization or asset retrieval;
(ii) we introduce a novel feature aggregation module that integrates local and
global scene information from visual and geometric encoders within the feature
extraction module. Coupled with a position head, this enables the generation of
3D assets and their relative spatial positions in a single feedforward pass;
(iii) we demonstrate SceneGen's direct extensibility to multi-image input
scenarios. Despite being trained solely on single-image inputs, our
architectural design enables improved generation performance with multi-image
inputs; and (iv) extensive quantitative and qualitative evaluations confirm the
efficiency and robust generation abilities of our approach. We believe this
paradigm offers a novel solution for high-quality 3D content generation,
potentially advancing its practical applications in downstream tasks. The code
and model will be publicly available at: https://mengmouxu.github.io/SceneGen.