Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model
  Self-Distillation
Authors: name: Sherwin Bahmani, name: Tianchang Shen, name: Jiawei Ren, name: Jiahui Huang, name: Yifeng Jiang, name: Haithem Turki, name: Andrea Tagliasacchi, name: David B. Lindell, name: Zan Gojcic, name: Sanja Fidler, name: Huan Ling, name: Jun Gao, name: Xuanchi Ren

The ability to generate virtual environments is crucial for applications
ranging from gaming to physical AI domains such as robotics, autonomous
driving, and industrial AI. Current learning-based 3D reconstruction methods
rely on the availability of captured real-world multi-view data, which is not
always readily available. Recent advancements in video diffusion models have
shown remarkable imagination capabilities, yet their 2D nature limits the
applications to simulation where a robot needs to navigate and interact with
the environment. In this paper, we propose a self-distillation framework that
aims to distill the implicit 3D knowledge in the video diffusion models into an
explicit 3D Gaussian Splatting (3DGS) representation, eliminating the need for
multi-view training data. Specifically, we augment the typical RGB decoder with
a 3DGS decoder, which is supervised by the output of the RGB decoder. In this
approach, the 3DGS decoder can be purely trained with synthetic data generated
by video diffusion models. At inference time, our model can synthesize 3D
scenes from either a text prompt or a single image for real-time rendering. Our
framework further extends to dynamic 3D scene generation from a monocular input
video. Experimental results show that our framework achieves state-of-the-art
performance in static and dynamic 3D scene generation.