BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane
  Extrapolation
Authors: name: Zhennan Wu; name: Yang Li; name: Han Yan; name: Taizhang Shang; name: Weixuan Sun; name: Senbo Wang; name: Ruikai Cui; name: Weizhe Liu; name: Hiroyuki Sato; name: Hongdong Li; name: Pan Ji
Published: 2024-01-30 14:34:19

Abstract: We present BlockFusion, a diffusion-based model that generates 3D scenes as
unit blocks and seamlessly incorporates new blocks to extend the scene.
BlockFusion is trained using datasets of 3D blocks that are randomly cropped
from complete 3D scene meshes. Through per-block fitting, all training blocks
are converted into the hybrid neural fields: with a tri-plane containing the
geometry features, followed by a Multi-layer Perceptron (MLP) for decoding the
signed distance values. A variational auto-encoder is employed to compress the
tri-planes into the latent tri-plane space, on which the denoising diffusion
process is performed. Diffusion applied to the latent representations allows
for high-quality and diverse 3D scene generation. To expand a scene during
generation, one needs only to append empty blocks to overlap with the current
scene and extrapolate existing latent tri-planes to populate new blocks. The
extrapolation is done by conditioning the generation process with the feature
samples from the overlapping tri-planes during the denoising iterations. Latent
tri-plane extrapolation produces semantically and geometrically meaningful
transitions that harmoniously blend with the existing scene. A 2D layout
conditioning mechanism is used to control the placement and arrangement of
scene elements. Experimental results indicate that BlockFusion is capable of
generating diverse, geometrically consistent and unbounded large 3D scenes with
unprecedented high-quality shapes in both indoor and outdoor scenarios.