VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting
  Generation with Flexible Pose and Multi-View Joint Modeling
Authors: name: Hyojun Go; name: Byeongjun Park; name: Hyelin Nam; name: Byung-Hoon Kim; name: Hyungjin Chung; name: Changick Kim
Published: 2025-03-20 05:26:09

Abstract: We propose VideoRFSplat, a direct text-to-3D model leveraging a video
generation model to generate realistic 3D Gaussian Splatting (3DGS) for
unbounded real-world scenes. To generate diverse camera poses and unbounded
spatial extent of real-world scenes, while ensuring generalization to arbitrary
text prompts, previous methods fine-tune 2D generative models to jointly model
camera poses and multi-view images. However, these methods suffer from
instability when extending 2D generative models to joint modeling due to the
modality gap, which necessitates additional models to stabilize training and
inference. In this work, we propose an architecture and a sampling strategy to
jointly model multi-view images and camera poses when fine-tuning a video
generation model. Our core idea is a dual-stream architecture that attaches a
dedicated pose generation model alongside a pre-trained video generation model
via communication blocks, generating multi-view images and camera poses through
separate streams. This design reduces interference between the pose and image
modalities. Additionally, we propose an asynchronous sampling strategy that
denoises camera poses faster than multi-view images, allowing rapidly denoised
poses to condition multi-view generation, reducing mutual ambiguity and
enhancing cross-modal consistency. Trained on multiple large-scale real-world
datasets (RealEstate10K, MVImgNet, DL3DV-10K, ACID), VideoRFSplat outperforms
existing text-to-3D direct generation methods that heavily depend on post-hoc
refinement via score distillation sampling, achieving superior results without
such refinement.