Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using
  Real-Time Warped Noise
Authors: name: Ryan Burgert; name: Yuancheng Xu; name: Wenqi Xian; name: Oliver Pilarski; name: Pascal Clausen; name: Mingming He; name: Li Ma; name: Yitong Deng; name: Lingxiao Li; name: Mohsen Mousavi; name: Michael Ryoo; name: Paul Debevec; name: Ning Yu
Published: 2025-01-14 18:59:10

Abstract: Generative modeling aims to transform random noise into structured outputs.
In this work, we enhance video diffusion models by allowing motion control via
structured latent noise sampling. This is achieved by just a change in data: we
pre-process training videos to yield structured noise. Consequently, our method
is agnostic to diffusion model design, requiring no changes to model
architectures or training pipelines. Specifically, we propose a novel noise
warping algorithm, fast enough to run in real time, that replaces random
temporal Gaussianity with correlated warped noise derived from optical flow
fields, while preserving the spatial Gaussianity. The efficiency of our
algorithm enables us to fine-tune modern video diffusion base models using
warped noise with minimal overhead, and provide a one-stop solution for a wide
range of user-friendly motion control: local object motion control, global
camera movement control, and motion transfer. The harmonization between
temporal coherence and spatial Gaussianity in our warped noise leads to
effective motion control while maintaining per-frame pixel quality. Extensive
experiments and user studies demonstrate the advantages of our method, making
it a robust and scalable approach for controlling motion in video diffusion
models. Video results are available on our webpage:
https://vgenai-netflix-eyeline-research.github.io/Go-with-the-Flow. Source code
and model checkpoints are available on GitHub:
https://github.com/VGenAI-Netflix-Eyeline-Research/Go-with-the-Flow.