DimensionX: Create Any 3D and 4D Scenes from a Single Image with
  Controllable Video Diffusion
Authors: name: Wenqiang Sun; name: Shuo Chen; name: Fangfu Liu; name: Zilong Chen; name: Yueqi Duan; name: Jun Zhang; name: Yikai Wang
Published: 2024-11-07 18:07:31

Abstract: In this paper, we introduce DimensionX, a framework designed to
generate photorealistic 3D and 4D scenes from just a single image with video
diffusion. Our approach begins with the insight that both the spatial structure
of a 3D scene and the temporal evolution of a 4D scene can be effectively
represented through sequences of video frames. While recent video diffusion
models have shown remarkable success in producing vivid visuals, they face
limitations in directly recovering 3D/4D scenes due to limited spatial and
temporal controllability during generation. To overcome this, we propose
ST-Director, which decouples spatial and temporal factors in video diffusion by
learning dimension-aware LoRAs from dimension-variant data. This controllable
video diffusion approach enables precise manipulation of spatial structure and
temporal dynamics, allowing us to reconstruct both 3D and 4D representations
from sequential frames with the combination of spatial and temporal dimensions.
Additionally, to bridge the gap between generated videos and real-world scenes,
we introduce a trajectory-aware mechanism for 3D generation and an
identity-preserving denoising strategy for 4D generation. Extensive experiments
on various real-world and synthetic datasets demonstrate that DimensionX
achieves superior results in controllable video generation, as well as in 3D
and 4D scene generation, compared with previous methods.