Generative View Stitching
Authors: name: Chonghyuk Song, name: Michal Stary, name: Boyuan Chen, name: George Kopanas, name: Vincent Sitzmann
Organization: Massachusetts Institute of Technology

Autoregressive video diffusion models are capable of long rollouts that are
stable and consistent with history, but they are unable to guide the current
generation with conditioning from the future. In camera-guided video generation
with a predefined camera trajectory, this limitation leads to collisions with
the generated scene, after which autoregression quickly collapses. To address
this, we propose Generative View Stitching (GVS), which samples the entire
sequence in parallel such that the generated scene is faithful to every part of
the predefined camera trajectory. Our main contribution is a sampling algorithm
that extends prior work on diffusion stitching for robot planning to video
generation. While such stitching methods usually require a specially trained
model, GVS is compatible with any off-the-shelf video model trained with
Diffusion Forcing, a prevalent sequence diffusion framework that we show
already provides the affordances necessary for stitching. We then introduce
Omni Guidance, a technique that enhances the temporal consistency in stitching
by conditioning on both the past and future, and that enables our proposed
loop-closing mechanism for delivering long-range coherence. Overall, GVS
achieves camera-guided video generation that is stable, collision-free,
frame-to-frame consistent, and closes loops for a variety of predefined camera
paths, including Oscar Reutersv\"ard's Impossible Staircase. Results are best
viewed as videos at https://andrewsonga.github.io/gvs.