Taming generative video models for zero-shot optical flow extraction
Authors: name: Seungwoo Kim, name: Khai Loong Aw, name: Klemen Kotar, name: Cristobal Eyzaguirre, name: Wanhee Lee, name: Yunong Liu, name: Jared Watrous, name: Stefan Stojanov, name: Juan Carlos Niebles, name: Jiajun Wu, name: Daniel L. K. Yamins

Extracting optical flow from videos remains a core computer vision problem.
Motivated by the success of large general-purpose models, we ask whether frozen
self-supervised video models trained only for future frame prediction can be
prompted, without fine-tuning, to output flow. Prior work reading out depth or
illumination from video generators required fine-tuning, which is impractical
for flow where labels are scarce and synthetic datasets suffer from a
sim-to-real gap. Inspired by the Counterfactual World Model (CWM) paradigm,
which can obtain point-wise correspondences by injecting a small tracer
perturbation into a next-frame predictor and tracking its propagation, we
extend this idea to generative video models. We explore several popular
architectures and find that successful zero-shot flow extraction in this manner
is aided by three model properties: (1) distributional prediction of future
frames (avoiding blurry or noisy outputs); (2) factorized latents that treat
each spatio-temporal patch independently; and (3) random-access decoding that
can condition on any subset of future pixels. These properties are uniquely
present in the recent Local Random Access Sequence (LRAS) architecture.
Building on LRAS, we propose KL-tracing: a novel test-time procedure that
injects a localized perturbation into the first frame, rolls out the model one
step, and computes the Kullback-Leibler divergence between perturbed and
unperturbed predictive distributions. Without any flow-specific fine-tuning,
our method outperforms state-of-the-art models on real-world TAP-Vid DAVIS
dataset (16.6% relative improvement for endpoint error) and synthetic TAP-Vid
Kubric (4.7% relative improvement). Our results indicate that counterfactual
prompting of controllable generative video models is a scalable and effective
alternative to supervised or photometric-loss approaches for high-quality flow.